Pre-commit hook for upgrading type hints

Overview

supported python versions code coverage pre-commit

PEP585 Upgrade

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

This will work for any Python version above 3.7, though if you're not using 3.9 you will need to run the hook with futures-imports=true.

A complete type list is shown below.

See the complete list
Used to be Will be upgraded to
typing.Tuple tuple
typing.List list
typing.Dict dict
typing.Set set
typing.FrozenSet frozenset
typing.Type type
typing.Deque collections.deque
typing.DefaultDict collections.defaultdict
typing.OrderedDict collections.OrderedDict
typing.Counter collections.Counter
typing.ChainMap collections.ChainMap
typing.Awaitable collections.abc.Awaitable
typing.Coroutine collections.abc.Coroutine
typing.AsyncIterable collections.abc.AsyncIterable
typing.AsyncIterator collections.abc.AsyncIterator
typing.AsyncGenerator collections.abc.AsyncGenerator
typing.Iterable collections.abc.Iterable
typing.Iterator collections.abc.Iterator
typing.Generator collections.abc.Generator
typing.Reversible collections.abc.Reversible
typing.Container collections.abc.Container
typing.Collection collections.abc.Collection
typing.Callable collections.abc.Callable
typing.AbstractSet collections.abc.Set
typing.MutableSet collections.abc.MutableSet
typing.Mapping collections.abc.Mapping
typing.MutableMapping collections.abc.MutableMapping
typing.Sequence collections.abc.Sequence
typing.MutableSequence collections.abc.MutableSequence
typing.ByteString collections.abc.ByteString
typing.MappingView collections.abc.MappingView
typing.KeysView collections.abc.KeysView
typing.ItemsView collections.abc.ItemsView
typing.ValuesView collections.abc.ValuesView
typing.ContextManager contextlib.AbstractContextManager
typing.AsyncContextManager contextlib.AbstractAsyncContextManager
typing.re.Pattern re.Pattern
typing.re.Match re.Match

I'm a visual learner

In a nutshell, this code

from typing import List, Tuple, Dict, Set, FrozenSet

def do_thing(x: List[Tuple[str, ...]], y: Dict[str, Set[str]]) -> FrozenSet:

becomes this

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

or this, if you're running python < 3.9 or enable the futures option

from __future__ import annotations

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

Features:

  • Performs in-line substitution for new types
  • Adds new imports for upgrade types which need them
  • Adds __futures__ imports if the futures flag is enabled
  • Removes no longer needed typing imports

Note: even though we remove and add imports reasonably well, I would recommend running this in tandem with hooks like isort to aggregate and sort your imports, and flake8 to discover any unused imports neither were able to remove. Otherwise you risk needing to do some manual cleanup from time to time (though it should be pretty rare).

Config

To use this with pre-commit, simply add this to your config file:

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints

and while futures imports are added automatically if you're running Python older than 3.9, you can also enable them explicitly, by adding a --futures arg.

This is required, e.g., when maintaining code that needs to support older Python versions.

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints
    args: [ '--futures=true' ]

For more information about available arguments, see the function definitions.

Running this once on my codebase

If you wish to run this once on your codebase, that's not easy to do without pre-commit, as it piggybacks on that process quite a bit.

However, installing pre-commit and configuring the hook to run will take you less than a minute. These are the steps:

  • Run pip install pre-commit
  • Run touch .pre-commit-config.yaml
  • Copy the configuration shown above into the file
  • Run pre-commit run --all-files

Running this once on a single file

To run the upgrade on a single file, you can clone the repo and run python -m upgrade_type_hints from the src folder, or something equivalent.

Known imperfections

  • We have a hard time removing common import typing imports, since we don't have a full inventory of all possible places typing could be used. Seeing something like this, you might think this is easy to handle

    import typing
    
    x: typing.List

    but extending this example to a thousand-line file, the way we've structured the logic, there is no way to know whether there is a valid typing.Optional somewhere in the file.

  • We might remove typing imports in a file where you needed them for more than just type annotations. An example of this is custom type declarations:

    from typing import List
    
    x: List  # this will be upgraded and the import will be removed
    y = List[str]  # this will be left without its required import

    The reason for this is that custom type declarations are not a part of the ast objects we look at.

Both points are resolved by running flake8.

Supporting the project

Please leave a ✭ if this project helped you 👏 and contributions are always welcome!

Comments
  • Don't upgrade generics - part 2

    Don't upgrade generics - part 2

    I observe the same issue as #22, using v1.0.1.

    Starting with the file

    from __future__ import annotations
    
    from typing import Any, Callable, Mapping
    
    MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    def func(x: Mapping[str, Any], y: Any) -> None:
        return None
    
    callback: MessageCallback
    callback = func
    

    This originally runs without errors, but not after running pep585-upgrade has run:

    $ python -V
    Python 3.8.6
    $ python test.py
    $ pre-commit run
    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    - files were modified by this hook
    
    Fixing src/workflows/test.py
    
    isort....................................................................Failed
    - hook id: isort
    - files were modified by this hook
    
    $ git diff
    index 72539fc..56bc282 100644
    --- a/test.py
    +++ b/test.py
    @@ -1,6 +1,7 @@
     from __future__ import annotations
    
    -from typing import Any, Callable, Mapping
    +from collections.abc import Mapping
    +from typing import Any, Callable
    
     MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    $ python test.py
    Traceback (most recent call last):
      File "test.py", line 6, in <module>
        MessageCallback = Callable[[Mapping[str, Any], Any], None]
    TypeError: 'ABCMeta' object is not subscriptable
    
    opened by Anthchirp 11
  • Support for typing.cast call

    Support for typing.cast call

    The code-base I'm modernizing has the following code:

    project_acl_list = cast(Optional[List[str]], project.acl)
    

    Unfortunately, this is not transformed.

    I'm willing to fix that, but I don't know yet how...

    Could we discuss how to implement the transformation for this code?

    opened by rominf 9
  • end_lineno error while running python 3.7

    end_lineno error while running python 3.7

    Adding hook via:

      - repo: https://github.com/sondrelg/pep585-upgrade
        rev: ''
        hooks:
          - id: upgrade-type-hints
            args: [ '--futures=true' ]
    

    When running in a python 3.9 virtualenv everything works, but switching to python 3.7 causes this error:

    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    
    opened by ArcLightSlavik 8
  • pre-commit autoupdate warning

    pre-commit autoupdate warning

    Running pre-commit autoupdate leads to the following result: Updating https://github.com/sondrelg/pep585-upgrade ... updating v1.0.1 -> v1.

    Running pre-commit autoupdate again will result in the following warning:

    [WARNING] The 'rev' field of repo 'https://github.com/sondrelg/pep585-upgrade' appears to be a mutable reference (moving tag / branch).
    Mutable references are never updated after first install and are not supported. 
    See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. 
    Hint: `pre-commit autoupdate` often fixes this.
    

    It would be cool if pre-commit autoupdate would work on this hook as expected. I suspect that the problem is that all 3 tags point to the same commit.

    opened by alkatar21 7
  • Consider module docstring while inserting future import

    Consider module docstring while inserting future import

    Summary

    When the python file consists of module docstring, the formatter inserts the from __future__ import annotation on the top of the docstring.

    Previous Behavior

    Source

    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    from typing import List
    
    a: List[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
        pass
    

    Generated

    from __future__ import annotations
    
    
    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    
    a: list[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
    

    The PR adds a function to find the module docstring. It will insert future import after the module docstring.

    opened by kracekumar 4
  • Shebang is not respected

    Shebang is not respected

    Input:

    #!/usr/bin/env python3
    from typing import AbstractSet
    
    s: AbstractSet
    

    Output:

    from collections.abc import Set
    #!/usr/bin/env python3
    
    s: Set
    

    I didn't prepare the fix yet, but I think that the best solution would be to save the first occurrence of non-comment, non-docstring expressions while scanning AST and put the imports before that. What do you think?

    opened by rominf 4
  • Create/push a tag for the package

    Create/push a tag for the package

    Hi guys, Thanks a lot for this great package, it makes the code much more pythonic! I have one favor: could you create/push a tag for the package? Else I have this annoying warning:

    [WARNING] The 'rev' field of repo 'https://github.com/snok/pep585-upgrade' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
    
    opened by MRigal 3
  • don't add another import line from same package

    don't add another import line from same package

    For an input file like this:

    from collections.abc import Callable
    
    from typing import Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    when applying pep585 hook I get:

    from collections.abc import Collection
    from collections.abc import Callable
    
    def a(a: Callable, b: Collection):
        pass
    

    while I expect:

    from collections.abc import Callable, Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    It could be argued this is the job of e.g. isort, however - pep585-upgrade already tries to be nice in terms of imports - e.g. in a larger import set - this new import line gets added nearby other import from same package and not at the end (which would be simplest I guess)

    opened by kretes 1
  • Don't Upgrade Generics

    Don't Upgrade Generics

    The typing package supports generics, whereas collections.abc does not. Hence, when this tool replaces typing types that are used as generics with their equivalent types in collections.abc, it breaks the build. I found it helpful to run this tool as a one-off and then manually undo these cases though, so thank you!

    opened by Kurt-von-Laven 1
  • Imports removal corrupts imports

    Imports removal corrupts imports

    Minimal working example:

    Input:

    from typing import Any, Type, TypeVar
    
    t: Type
    

    Output:

    from typing import AnyVar
    
    t: type
    

    I have a fix for this already. Please wait for a PR with a fix and tests.

    opened by rominf 1
  • Correct name of .pre-commit-config.yaml

    Correct name of .pre-commit-config.yaml

    I got an error from pre-commit which suggested it was looking for a .pre-commit-config.yaml file, when I changed the filename, as in this commit.

    Possibly OS specific, this was on MacOS Big Sur, Python 3.9.

    opened by james-d-f 1
Releases(v1.0.1)
Owner
snok
Open source collaboration organization made by @jonasks & @sondrelg
snok
personal dotfiles for rolling release linux distros

dotfiles Screenshots: Directions: Deploy my dotfiles with yadm Packages from arch listed in .installed-packages Information on osu! see ~/Games/osu!/.

-pacer- 0 Sep 18, 2022
An Insurance firm providing tour insurance is facing higher claim frequency

An Insurance firm providing tour insurance is facing higher claim frequency. Data is collected from the past few years. Made a model which predicts the claim status using CART, RF & ANN and compare t

1 Jan 27, 2022
This code can help you with auto update for-TV-advertisements in the store.

Auto-update-files-for-TV-advertisements-in-the-store This code can help you with auto update for-TV-advertisements in the store. It was write for Rasp

Max 2 Feb 20, 2022
Earth centric orbit propagation tool. Built from scratch in python.

Orbit-Propogator Earth centric orbit propagation tool. Built from scratch in python. Functionality includes: tracking sattelite location over time plo

Adam Klein 1 Mar 13, 2022
Python communism - A module for initiating the communist revolution in each of our python modules

Python communist revolution A man once said to abolish the classes or something

758 Jan 03, 2023
Interactivity Lab: Household Pulse Explorable

Interactivity Lab: Household Pulse Explorable Goal: Build an interactive application that incorporates fundamental Streamlit components to offer a cur

1 Feb 10, 2022
Structured Exceptions for Python

XC: Structured exceptions for Python XC encourages a structured, disciplined approach to use of exceptions: it reduces the overhead of declaring excep

Bob Gautier 2 May 28, 2021
Ramadhan countdown - Simple daily reminder about upcoming Ramadhan

Ramadhan Countdown Bot Simple bot for displaying daily reminder about Islamic pr

Abdurrahman Shofy Adianto 1 Feb 06, 2022
Listen Surah, prepare for next and Endless life...

Al-Quran In this repository, I have linked up all Surah with Arabic-Bangla Audio From Youtube. So, you just need to choose and listen. and the ( surah

SpiderX 1 Sep 30, 2022
Create a program for generator Truth Table

Python-Truth-Table-Ver-1.0 Create a program for generator Truth Table in here you have to install truth-table-generator module for python modules inst

JehanKandy 10 Jul 13, 2022
Python solutions to Codeforces problems

CodeForces This repository is dedicated to my Python solutions for CodeForces problems. Feel free to copy, contribute and/or comment. If you find any

Shukur Sabzaliev 15 Dec 20, 2022
A Python library for inspecting JVM class files (.class)

lawu Lawu is a human-friendly library for assembling, disassembling, and exploring JVM class files. It's highly suitable for automation tasks. Documen

Tyler Kennedy 45 Oct 23, 2022
List of all D&D 5e monsters: WotC + popular third-party sourcebooks

Xio's Guide to Monsters If you're a DM like me, and you have multiple sources of D&D 5e monsters that include WotC as well as third-party suppliers, y

20 Jan 06, 2023
A simple code for processing images to local binary pattern.

This figure is gotten from this link https://link.springer.com/chapter/10.1007/978-3-030-01449-0_24 LBP-Local-Binary-Pattern A simple code for process

Happy N. Monday 3 Feb 15, 2022
An open source recipe book from the awesome staff of Clinical Genomics

meatballs An open source recipe book from the awesome staff of Clinical Genomics.

Clinical Genomics 2 Dec 07, 2021
PyDateWaiter helps waiting special day & calculating remain days till that day with Python code.

PyDateWaiter (v.Beta) PyDateWaiter helps waiting special day(aniversary) & calculating remain days till that day with Python code. Made by wallga gith

wallga 1 Jan 14, 2022
Pequenos programas variados que estou praticando e implementando, leia o Read.me!

my-small-programs Pequenos programas variados que estou praticando e implementando! Arquivo: automacao Automacao de processos de rotina com código Pyt

Léia Rafaela 43 Nov 22, 2022
TMTC Commander Core

This commander application was first developed by KSat for the SOURCE project to test the on-board software but has evolved into a more generic tool for satellite developers to perform TMTC (Telemetr

robamu 8 Dec 14, 2022
Powerful virtual assistant in python

Virtual assistant in python Powerful virtual assistant in python Set up Step 1: download repo and unzip Step 2: pip install requirements.txt (if py au

Arkal 3 Jan 23, 2022
Library for mocking AsyncIOMotorClient built on top of mongomock.

mongomock-motor Best effort mock for AsyncIOMotorClient (Database, Collection, e.t.c) built on top of mongomock library. Example / Showcase from mongo

Michael Kryukov 43 Jan 04, 2023