Python linting made easy. Also a casual yet honorific way to address individuals who have entered an organization prior to you.

Overview

pysen

What is pysen?

pysen aims to provide a unified platform to configure and run day-to-day development tools. We envision the following scenarios in the future:

  • You open any project and pysen run lint, pysen run format will check and format the entire codebase
  • Standardized coding styles are setup with a few lines in a single pyproject.toml file

pysen centralizes the code and knowledge related to development tools that teams have accumulated, most notably for python linters. You can make tasks that can be executed from both setup.py and our command-line tool. We currently provide tasks that manage setting files for the following tools:

  • linters
    • flake8
    • isort
    • mypy
    • black
  • utilities
    • (planned) protoc

What isn't pysen?

  • pysen is not a linting tool per se. Rather, pysen run lint orchestrates multiple python linting tools by automatically setting up their configurations from a more abstract setting for pysen.
  • pysen does not manage your depedencies and packages. We recommend using package managers such as pipenv or poetry to lock your dependecy versions, including the versions for the linting tools that pysen coordinates (i.e., isort, mypy, flake8, black). The supported versions for these tools can be found in the extra_requires/lint section in pysen's setup.py. You should not rely on pip install pysen[lint] to control the versions of your linting tools.
  • pysen is not limited to linting purposes or python. See the plugin section for details.

Install

PyPI

pip install "pysen[lint]"

Other installation examples

# pipenv
pipenv install --dev "pysen[lint]==0.9.0"
# poetry
poetry add -D pysen==0.9.0 -E lint

Quickstart: Set up linters using pysen

Put the following pysen configuration to pyproject.toml of your python package:

[tool.pysen]
version = "0.9"

[tool.pysen.lint]
enable_black = true
enable_flake8 = true
enable_isort = true
enable_mypy = true
mypy_preset = "strict"
line_length = 88
py_version = "py37"
[[tool.pysen.lint.mypy_targets]]
  paths = ["."]

then, execute the following command:

$ pysen run lint
$ pysen run format  # corrects errors with compatible commands (black, isort)

That's it! pysen, or more accurately pysen tasks that support the specified linters, generate setting files for black, isort, mypy, and flake8 and run them with the appropriate configuration. For more details about the configuration items that you can write in pyproject.toml, please refer to pysen/pyproject_model.py.

You can also add custom setup commands to your Python package by adding the following lines to its setup.py:

import pysen
setup = pysen.setup_from_pyproject(__file__)
$ python setup.py lint

We also provide a Python interface for customizing our configuration and extending pysen. For more details, please refer to the following two examples:

  • Example configuration from Python: examples/advanced_example/config.py
  • Example plugin for pysen: examples/plugin_example/plugin.py

How it works: Settings file directory

Under the hood, whenever you run pysen, it generates the setting files as ephemeral temporary files to be used by linters. You may want to keep those setting files on your disk, e.g. when you want to use them for your editor. If that is the case, run the following command to generate the setting files to your directory of choice:

$ pysen generate [out_dir]

You can specify the settings directory that pysen uses when you pysen run. To do so add the following section to your pyproject.toml:

[tool.pysen-cli]
settings_dir = "path/to/generate/settings"

When you specify a directory that already contains some configurations, pysen merges the contents. The resulting behavior may differ from when you don't specify settings_dir.

Also keep in mind that this option is honored only when you use pysen through its CLI. When using pre-commit or setuptools you need to specify settings_dir as arguments.

Tips: IDE / Text editor integration

vim

You can add errors that pysen reports to your quickfix window by:

:cex system("pysen run_files lint --error-format gnu ".expand('%:p'))

Another way is to set pysen to makeprg:

set makeprg=pysen\ run_files\ --error-format\ gnu\ lint\ %

Then running :make will populate your quickfix window with errors. This also works with vim-dispatch as long as you invoke :Make instead of :Dispatch (for this reason)

The result will look like the following:

pysen-vim

Emacs

Refer to the Compilation mode. The following is an example hook for python.

(add-hook 'python-mode-hook
    (lambda ()
        (set (make-local-variable 'compile-command)
            (concat "pysen run_files lint --error-format gnu  " buffer-file-name))))

VSCode

Refer to the example task setting. Running the task will populate your "PROBLEMS" window like so:

pysen-vscode

Note that this may report duplicate errors if you have configured linters like flake8 directly through your VSCode python extension. We do not currently recommend watching for file changes to trigger the task in large projects since pysen will check for all files and may consume a considerable amount of time.

Configure pysen

We provide two methods to write configuration for pysen.

One is the [tool.pysen.lint] section in pyproject.toml. It is the most simple way to configure pysen, but the settings we provide are limited.

The other method is to write a python script that configures pysen directly. If you want to customize configuration files that pysen generates, command-line arguments that pysen takes, or whatever actions pysen performs, we recommend you use this method. For more examples, please refer to pysen/examples.

pyproject.toml configuration model

Please refer to pysen/pyproject_model.py for the latest model.

Here is an example of a basic configuration:

[tool.pysen]
version = "0.9"

[tool.pysen.lint]
enable_black = true
enable_flake8 = true
enable_isort = true
enable_mypy = true
mypy_preset = "strict"
line_length = 88
py_version = "py37"
isort_known_third_party = ["numpy"]
isort_known_first_party = ["pysen"]
mypy_ignore_packages = ["pysen.generated.*"]
mypy_path = ["stubs"]
[[tool.pysen.lint.mypy_targets]]
  paths = [".", "tests/"]

[tool.pysen.lint.source]
  includes = ["."]
  include_globs = ["**/*.template"]
  excludes = ["third_party/"]
  exclude_globs = ["**/*_grpc.py"]

[tool.pysen.lint.mypy_modules."pysen.scripts"]
  preset = "entry"

[tool.pysen.lint.mypy_modules."numpy"]
  ignore_errors = true

Create a plugin to customize pysen

We provide a plugin interface for customizing our tool support, setting files management, setup commands and so on. For more details, please refer to pysen/examples/plugin_example.

Development

pipenv is required for managing our development environment.

# setup your environment
$ pipenv sync
# activate the environment
$ pipenv shell
  • Update depedencies in Pipfile.lock
$ pipenv lock --pre
  • Run all tests
$ pipenv run tox

Contributing

This repository serves only as a mirror of our main repository on our private repository. Therefore we do not plan to accept any pull requests. We encourage aspiring developers to make patches on their forked repositories.

Also, our resource limitations force us to prioritize development to fulfill our corporate-specific demands. As such we will keep Issues closed for the foreseeable future. With a heavy heart we direct all questions, troubleshooting, feature requests and bug reports to /dev/null.

Comments
  • Unfriendly error in the absence of `tool.pysen.lint` section

    Unfriendly error in the absence of `tool.pysen.lint` section

    When pyproject.toml does not contain a tool.pysen.lint section, running pysen run lint results in the following error:

    pysen run lint
    usage: pysen run [-h] [--error-format {gnu}] [--no-parallel] {} [{} ...]
    pysen run: error: argument targets: invalid choice: 'lint' (choose from )
    

    In our case I believe the best configuration is no configuration at all, so I suggest enabling black, isort, flake and mypy without explicit configuration by default. By doing so we will not need to directly address the superficial issue of the unfriendly error.

    opened by sergeant-wizard 8
  • `tool.pysen.lint.source` may not work

    `tool.pysen.lint.source` may not work

    Problem

    I could not exclude a specific path when I run mypy. I think I can configure this by setting tool.pysen.lint.source properly, but it didn't work. I checked the generated pyproject.toml which linters use, and found that the information I specified in tool.pysen.lint.source didn't exist there.

    How to reproduce

    1. Create a pyproject.toml following the example on https://github.com/pfnet/pysen#configuration-model
    [tool.pysen]
    version = "0.10"
    
    [tool.pysen.lint]
    enable_black = true
    enable_flake8 = true
    enable_isort = true
    enable_mypy = true
    mypy_preset = "strict"
    line_length = 88
    py_version = "py37"
    isort_known_third_party = ["numpy"]
    isort_known_first_party = ["pysen"]
    mypy_ignore_packages = ["pysen.generated.*"]
    mypy_path = ["stubs"]
    [[tool.pysen.lint.mypy_targets]]
      paths = [".", "tests/"]
    
    [tool.pysen.lint.source]
      includes = ["."]
      include_globs = ["**/*.template"]
      excludes = ["third_party/"]
      exclude_globs = ["**/*_grpc.py"]
    
    [tool.pysen.lint.mypy_modules."pysen.scripts"]
      preset = "entry"
    
    [tool.pysen.lint.mypy_modules."numpy"]
      ignore_errors = true
    
    1. Run pysen generate hoge

    Then, hoge/pyproject.toml looks like

    [tool.black] # automatically generated by pysen
    line-length = 88
    target-version = ["py37"]
    
    [tool.isort] # automatically generated by pysen
    default_section = "THIRDPARTY"
    ensure_newline_before_comments = true
    force_grid_wrap = 0
    force_single_line = false
    include_trailing_comma = true
    known_first_party = ["pysen"]
    known_third_party = ["numpy"]
    line_length = 88
    multi_line_output = 3
    use_parentheses = true
    

    and hoge/setup.cfg looks like

    [flake8]
    # automatically generated by pysen
    # e203: black treats : as a binary operator
    # e231: black doesn't put a space after ,
    # e501: black may exceed the line-length to follow other style rules
    # w503 or w504: either one needs to be disabled to select w error codes
    ignore = E203,E231,E501,W503
    max-line-length = 88
    select = B,B950,C,E,F,W
    
    [mypy]
    # automatically generated by pysen
    check_untyped_defs = True
    disallow_any_decorated = False
    disallow_any_generics = False
    disallow_any_unimported = False
    disallow_incomplete_defs = True
    disallow_subclassing_any = True
    disallow_untyped_calls = True
    disallow_untyped_decorators = False
    disallow_untyped_defs = True
    ignore_errors = False
    ignore_missing_imports = True
    mypy_path = stubs
    no_implicit_optional = True
    python_version = 3.7
    show_error_codes = True
    strict_equality = True
    strict_optional = True
    warn_redundant_casts = True
    warn_return_any = True
    warn_unreachable = True
    warn_unused_configs = True
    warn_unused_ignores = False
    
    [mypy-numpy]
    # automatically generated by pysen
    ignore_errors = True
    
    [mypy-pysen.generated.*]
    # automatically generated by pysen
    follow_imports = skip
    ignore_errors = True
    
    [mypy-pysen.scripts]
    # automatically generated by pysen
    disallow_untyped_calls = False
    disallow_untyped_defs = False
    warn_return_any = False
    

    In the above example, [tool.pysen.lint.source] information is lost.

    Environment

    • macos 12.0.1
    • pysen version 0.10.1
    $ python
    Python 3.8.5 (default, Sep  8 2020, 17:55:39) 
    [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    
    opened by xuzijian629 5
  • Force lint extras to use older click

    Force lint extras to use older click

    https://github.com/psf/black/issues/2964 にあるように最新のclickでは古いblackが動作しない問題への対処です。 問題になるのは pysen[lint] を使ってinstallしたときで、blackは古いversionが入る一方でclickは最新版が入ります (black側でclickの上限を制限していないため)。

    pysen[lint] でinstallするときにclickのversionを制限することで不整合が起きないようにしました。 別の解決策としてblackの制限を緩めるというのもありますが、コードの変更や動作確認の手間が最小になるこちらの方法をとりました。 blackあるいはclickの最新版を使いたいユーザーは extras_require を使わないという方法があるので大きな問題にはならないという認識です。

    chore 
    opened by Hakuyume 3
  • Update lint extras for py39

    Update lint extras for py39

    Since PFN uses py39 widely, pysen[lint] should work with py39. Current lint extras tries to install older black and it does not work with py39.

    (optional) Can we drop py36 and py37 support?

    opened by Hakuyume 2
  • Use 'release' event to trigger release workflow

    Use 'release' event to trigger release workflow

    There is a limitation that a workflow cannot trigger a new workflow. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow

    Currently, a release is created by release-drafter, and that's why the release workflow wasn't triggered for the 0.10.2 tag.

    As such, I configured the release workflow to use the workflow_dispatch to publish a new release. I also removed push.tags trigger as we can use the workflow_dispatch trigger for the manual release as well, but I don't have a strong opinion about that.

    chore 
    opened by bonprosoft 2
  • Issue of required black version

    Issue of required black version

    In setup.py, required version of black is,

    "black>=19.10b0,<=20.8",

    But it's written to install black==21.10b0 on README.md. And it cause error during executing black.

    opened by kazyam53 2
  • State only that linter versions for [lint] are

    State only that linter versions for [lint] are "confirmed" instead of "supported" in README

    The current wording is not accurate because the versions are those that are confirmed to work by the developers. Higher versions are not necessarily unsupported. I think the word "confirmed" is often used to imply such scenarios.

    documentation 
    opened by sergeant-wizard 0
  • Add pysen-test action

    Add pysen-test action

    I created pysen-test action. The action builds the Dockerfile on each CI job and runs tox on the built image.

    Initially I used the docker image hosted on quay.io. But actually it would be problematic for some PRs like #18 to update the docker image.

    chore 
    opened by bonprosoft 0
Releases(0.10.2)
  • 0.10.2(Apr 23, 2022)

    📜 Documentation

    • Add missing assets/src directory for rendering images (#6) @bonprosoft

    🧰 Maintenance

    • Bump version to 0.10.2 (#14) @bonprosoft
    • supports python 3.10 as a target version (#15) @hiro-o918
    • Force lint extras to use older click (#13) @Hakuyume
    • Sort files to make output more readable (#11) @grafi-tt
    • Update comment in settings (#10) @Hakuyume
    • Support tomlkit 0.10.0 (#9) @bonprosoft
    • Add workflow for releasing PyPI packages (#5) @bonprosoft
    • Add release drafter (#3) @sergeant-wizard
    • Modify pull_request trigger to the proper indent level (#4) @bonprosoft
    • Fix typo (#1) @sergeant-wizard
    • Add pysen-test CI (#2) @bonprosoft
    Source code(tar.gz)
    Source code(zip)
  • 0.10.1(Nov 25, 2021)

    We have released pysen 0.10.1!

    pysen 0.10.1 includes...

    • Latest linter version support
      • We have added support for the following versions since v0.9.1
        • mypy>=0.800
        • black>=21.4b
        • flake8>=4.0
      • Note that lint extra is still restricted to older versions. Please refer to the Install section in README.
    • mypy_preset renewal
      • entry: Recommended for newbies
      • strict: Recommended for projects with libraries which do not provide complete type annotations, such as opencv, pytorch, etc.
      • very_strict: For strict type checking. This preset may not be practical when used with libraries without complete type annotation. This corresponds to the strict preset before v0.9.1.
    • Option to use pysen.toml as configuration file name
    • mypy namespace packages support
    • Many bug fixes

    We have also added a “Frequently Asked Questions” section in README. Please look for solutions to your problem in this section.

    Source code(tar.gz)
    Source code(zip)
  • 0.9.1(Mar 29, 2021)

Owner
Preferred Networks, Inc.
Preferred Networks, Inc.
Web3 Ethereum DeFi toolkit for smart contracts, Uniswap and PancakeSwap trades, Ethereum JSON-RPC utilities, wallets and automated test suites.

Web3 Ethereum Defi This project contains common Ethereum smart contracts and utilities, for trading, wallets,automated test suites and backend integra

Trading Strategy 222 Jan 04, 2023
This discord bot will help you to control your target through PickleC2

PickleC2-Bot This discord bot will help you to control your target through PickleC2 WHAT's a PickleC2? PickleC2 is a simple C2 framework written in py

4 Jun 25, 2022
The Foursquare API client for Python

foursquare Python client for the foursquare API. Philosophy: Map foursquare's endpoints one-to-one Clean, simple, Pythonic calls Only handle raw data,

Mike Lewis 400 Dec 19, 2022
This repo provides the source code for "Cross-Domain Adaptive Teacher for Object Detection".

Cross-Domain Adaptive Teacher for Object Detection This is the PyTorch implementation of our paper: Cross-Domain Adaptive Teacher for Object Detection

Meta Research 91 Dec 12, 2022
Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and 4pm.

Weather Bot Besac Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and

Rgld_ 1 Nov 15, 2021
Minimal telegram voice chat music bot, in pyrogram.

VCBOT Fully working VC (user)Bot, based on py-tgcalls and py-tgcalls-wrapper with minimal features. Deploying To heroku: Local machine/VPS: git clone

Aditya 33 Nov 12, 2022
Notion API Database Python Implementation

Python Notion Database Notion API Database Python Implementation created only by database from the official Notion API. Installing / Getting started p

minwook 78 Dec 19, 2022
Python bindings for BigML.io

BigML Python Bindings BigML makes machine learning easy by taking care of the details required to add data-driven decisions and predictive power to yo

BigML Inc, Machine Learning made easy 271 Dec 27, 2022
This Bot Can Upload Video from Link Of Pdisk to Pdisk using its API. @PredatorHackerzZ

𝐏𝐝𝐢𝐬𝐤 𝐂𝐨𝐧𝐯𝐞𝐫𝐭𝐞𝐫 𝐁𝐨𝐭 Make short link by using 𝐏𝐝𝐢𝐬𝐤 API key Installation 𝐓𝐡𝐞 𝐄𝐚𝐬𝐲 𝐖𝐚𝐲 𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐝 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞

ρяє∂αтσя 25 Dec 02, 2022
Simple integrate of API udemy.com with python

Pyudemy Simple integrate of API udemy.com with python Quick start $ pip install pyudemy or $ python setup.py install Authentication To make any calls

Hudson Brendon 30 Jan 02, 2023
Python wrapper for the Intercom API.

python-intercom Not officially supported Please note that this is NOT an official Intercom SDK. The third party that maintained it reached out to us t

Intercom 215 Dec 22, 2022
1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能)

op-devops-api 1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能) 后端项目配套前端项目为:op-devops-ui jenkinsManager 一.插件python-jenkins bug修复 (1).插件版本 pyt

3 Nov 12, 2021
Instant messaging client in tkinter

Concord_client_tk Instant messaging client in tkinter Contributors : Ilade-s [https://github.com/Ilade-s] Doku [https://github.com/D0kuhebi] Descripti

Raphaël Merlet 2 Jun 15, 2022
Wrapper for the Swiss Parliament API for Python

swissparlpy This module provides easy access to the data of the OData webservice of the Swiss parliament. Table of Contents Installation Usage Get tab

Stefan Oderbolz 8 Jun 13, 2022
SC4.0 - BEST EXPERIENCE · HEX EDITOR · Discord Nuker · Plugin Adder · Cheat Engine

smilecreator4 This site is for people who want to hack or want to learn it! Furthermore, this program does not work without turning off Antivirus or W

1 Jan 04, 2022
A simple bot discord in PY with moderation controls

Voila un bot discord en py avec les commandes simples de modération tout simplement faut changer les lignes 70 vous mettez votre token de votre bot 53

Ethan 1 Nov 20, 2021
A maintained fork of Danny's discord.py

Nextcord A modern, easy-to-use, feature-rich, and async-ready API wrapper for Discord written in Python. Fork notice This is a fork of discord.py, whi

977 Jan 05, 2023
Free and Open Source Channel/Group Voice chat music player for telegram ❤️ with button support Heroku Commands

ZeusMusic Requirements 📝 FFmpeg NodeJS nodesource.com Python 3.7 or higher PyTgCalls MongoDB 2nd Telegram Account (needed for userbot) 🧪 Get SESSION

ZeusNetwork 4 Jan 03, 2022
Unarchive Bot for Telegram

Telegram UnArchiver Bot UnArchiveBot: 🇬🇧 Bot that allows you to extract supported archive formats in telegram. 🇹🇷 Desteklenen arşiv biçimleri tele

Hüzünlü Artemis [HuzunluArtemis] 25 May 07, 2022
A multipurpose Telegram Bot writen in Python for mirroring files

Deepak Clouds Mirror Deepak Clouds Torrent is a multipurpose Telegram Bot writen in Python for mirroring files on the Internet to our beloved Google D

MR.SHAGGY 0 Dec 19, 2021