The state of mypy

At last year's Python Language Summit, Guido van Rossum gave an introduction to "type hints", which are an optional feature to allow static checkers to spot type errors in Python programs. At this year's summit, he discussed mypy, which is one of several static type checkers for Python. It is being used by Dropbox, Van Rossum's employer, on its large Python codebase—with good results.

Van Rossum began by noting that he has been working on mypy, along with quite a few others. It uses the type hints that were standardized in PEP 484 and the function annotation syntax that came with PEP 3107. Because of that, multiple projects can use the annotations. Other users of the syntax include the PyCharm IDE and Google's pytype tool. It is a "big tent", he said, which is exactly what was intended.

Type hints are optional and will always remain that way. They are "not everyone's cup of tea" and can sometimes just get in the way. There are "lots of reasons you don't want to use type annotation", he said.

But he and Dropbox have found it useful. Annotations are being added to a multi-million line application. Part of the reason is to help new employees come up to speed on that codebase. Rather than have to look through a million lines of code to figure out if a function returns a list or a dict, they can just consult the stub file—the .pyi file where type annotations are often placed—or get an error from mypy.

The larger a codebase is, the more benefit it will get from adding annotations and actually checking them with a tool, Van Rossum said. But everyone starts with a small codebase and it can be hard to recognize when it transitions into a large one. He recommends that projects start looking into annotations "sooner than you would like to".

Mypy was started by Jukka Lehtosalo as a Python variant with type checking. Van Rossum and Lehtosalo met at PyCon 2013, where they discussed the project and eventually agreed that it would be more successful as an add-on to standard Python. That required a change in the syntax for mypy type annotations so that the Python parser did not need to change.

Over the years, there was a lot of discussion about type hints and how they should work. He gave a keynote about type hints at PyCon 2015 where he "overwhelmed most of the audience with too much detail way too fast". PEP 484 was eventually accepted. Since then, the typeshed repository has opened up to collect stub files with annotations for the standard library and other modules. Those stubs can be used both by mypy and by any other tools that are consuming the type hints.

Mypy has been used to find missing stub files. Those get turned into bug reports to add the stubs, so the typeshed repository is growing effectively. Dropbox has a team working on mypy and has adopted it internally. The results so far have validated the idea that type hints are useful, he said.

But, for Dropbox, type hints needed to work for its Python 2.7 codebase. Various things were tried before the Dropbox developers settled on type comments. There are lots of tools out there that think they know how to parse Python or projects that are using a different implementation of Python, which makes it difficult to simply adopt the Python-3-based type annotations in 2.7. He did not want to make changes to the upstream Python 2.7 code to support the annotations.

Someone from Google spoke up to say that the company does use the type annotations in 2.7 code, but has a "stripper" that removes them for tools that don't understand them. But Van Rossum said he wanted "unadulterated 2.7 code". He also looked at using Python docstrings, but there are already some "pseudo type annotations" in the docstrings that have not kept up with the code changes.

PEP 484 has "provisional" status, which has allowed more features to "sneak in" after its release in Python 3.5.0. All of these new features will show up by 3.5.2

Van Rossum then went through some of the changes to PEP 484 since it was accepted. The @overload decorator for overloaded functions was originally only allowed in stub files, but that has been extended to allow it in Python source files as well. Mypy has not yet added support for that, however. There is a new Text type that can be used for code that straddles the 2/3 divide. It is defined as a Unicode string for Python 2 and as the str type for Python 3.

There is also a new Type[C] type for use with classes. The C argument is a class and any arguments that use that annotation must be a subclass of C. That will allow factories to specify their class and return type. There are types to explicitly support the new coroutine syntax using async and await, though mypy does not yet support it.

Another addition that is coming to PEP 484 is the NewType() feature that will allow creating new type aliases. There has been some difficulty in naming the feature, which caused Van Rossum to jokingly refer to it as "BoatyMcBoatType". He is hopeful that a way to declare variables that doesn't require a type comment can come in some version of Python after 3.6. It would be nice, but is not urgent, he said.

Alex Gaynor asked if it was time to consider adding type annotations directly into the standard library. When the feature was first added, Van Rossum said that he did not want annotations in the library, and to put them in stub files, but is it time to readdress that?

Van Rossum said that the standard library is "very crusty code" and he worries that someone adding type annotations for everything in it will either make mistakes in 1% of the code or create merge conflicts all over the place. Either would be painful. For new modules, though, he would accept type annotations as part of the submission. Right now, producing stubs for the standard library is working well, it is parallelizable and creates no merge conflicts. Eventually, he said, his answer on this will change.