Type hints

One of the headline features targeted at Python 3.5 (which is due in September) is type hinting (or type hints). Guido van Rossum gave an introduction to the feature at a Python Language Summit session. Type hints are aimed at static analysis, so several developers of static analysis tools were involved in the discussion as well.

The current proposal for type hints for Python is laid out in PEP 484. It uses "function annotations as I originally envisioned them", Van Rossum said. Those annotations were proposed back in 2000 with the idea that they would provide information to the interpreter to "generate super good code". But the annotations don't really help with code generation, so they are not meant to help the interpreter at this point.

Instead, type hints are designed to be ignored by the interpreter and to "not slow it down in most cases". The feature is targeted at being used by a "lint on steroids", he said.

He put up a slide with example code that he said showed some of the problems with the annotation syntax, but gave a reasonable flavor of how it would work. Here is an excerpt:

    from typing import List, Tuple, Callable

    def zip(xx: List[int], yy: List[int]) -> List[Tuple[int, int]]:
        ...
    def zipmap(f: Callable[[int, int], int], xx: List[int],
               yy: List[int]) -> List[Tuple[int, int, int]]:
	...

In the example, zip() takes two arguments that are lists of integers and returns a list of two-integer tuples. zipmap() takes a function that takes two integer arguments and returns an integer along with two lists of integers; it returns a list of three-integer tuples. There is also support for generic types, so that the annotations can go "beyond concrete types", he said.

Stub files are "boring stuff that is nevertheless important", Van Rossum said. They provide a mechanism to annotate functions in C extension modules without becoming a burden on Argument Clinic, which is a domain-specific language for specifying arguments to Python built-ins. Stubs are also useful for things that you can't or don't want to annotate. The stubs are stored in .pyi files corresponding to the Python extension (e.g. base64.pyi) using the same function annotation syntax. There is one addition, though, of an @overload decorator for overloaded functions.

For 3.5, he is hoping to get typing.py added to the standard library. That is the entirety of the changes needed for this proposal, as there are no changes to CPython or to the Python syntax. The addition is "pure Python", but there are "a lot of metaclasses" and other scary stuff in typing.py. There are no plans for annotations for the standard library in 3.5, though he does anticipate some third-party stubs for standard library modules. The mypy tool that served as inspiration for the PEP already has some stubs for the standard library.

Putting typing.py into the standard library sends a signal that this is what the core Python developers want in terms of type hints. It encourages everyone who thinks that type hints are a good thing to use the same syntax. For example, the PyCharm IDE has its own notion of stubs and Google has a bunch of tools that it has released as open source (or will); both of those could benefit from a single standard type hint syntax.

There are no plans to force this feature on anyone that doesn't want to use it. He would like to get it into 3.5 before the feature freeze that accompanies the first beta release (due in May). That target will help "focus the PEP design". The typing.py module would be added as a provisional package, which means that it can still evolve as needed during the rest of the release cycle.

Some have wondered why there isn't a new syntax being designed for type hints. One reason is that typing.py will still work with earlier versions of Python 3, Van Rossum said. Those who are interested can just install it from the Python Package Index (PyPI). New syntax is also a "tar pit of bikeshedding". For 3.6, core developers "might muster up the courage" to add syntax for variable types (rather than use comments as is proposed with PEP 484).

There is a problem with forward references right now that the PEP solves by using string literals rather than types:

    class C:
        def foo(self) -> 'List[C]':
	    ...

Łukasz Langa is working on a way to get around that problem using a __future__ import:

    from __future__ import annotations

That would turn all annotations into string values as they are parsed, which would neatly avoid the problem that the CPython parser can't handle the forward references, while the static analyzers can.

At that point, Mark Shannon from Semmle, which is a company that makes static analyzers for Python and other languages, stepped up to talk about the proposal. He had a number of questions and concerns about the PEP, though syntax was not among them. Shannon said that he didn't care what the syntax was, his worries were about the semantics of the annotations.

Shannon is concerned about the lack of a distinction between classes and types. Also, the scope of type declarations is not well-defined. There is not much support for duck typing, either. Van Rossum admitted that duck typing is not supported, mostly because it doesn't fit well with static type analysis. The intended scope of type declarations is clear in Van Rossum's mind, but it may not be in the PEP, he said.

Shannon said that it was important to stop thinking about programs as code to run. Instead, for static analysis purposes, they should be looked at as a bit of text to analyze. He also suggested that any tools have two modes: "linting" mode to report when the type being used is not the same as what is required and "strict" mode that reports when the tool is unable to prove that the proper type is being used.

Van Rossum invited Shannon to co-author the PEP with him if he was willing to commit to the 3.5 time frame. Shannon said he was willing to work on it under that constraint.

The hope is that there will be new syntax for variable types in the 3.6 time frame, Van Rossum said. Jython developer Jim Baker was in favor of that. It would allow access to the variable annotations from the standard abstract syntax tree (ast) module.

Larry Hastings wondered why the PEP was trying to avoid using Argument Clinic. It is, he said, the perfect place to put this kind of information. Van Rossum said that there must have been some kind of misunderstanding at one point, so he apologized and agreed that Argument Clinic should be used.

The basic idea behind PEP 484 is to create a common notation, Van Rossum said. He was mostly hoping that the assembled developers would not be too unhappy with that notation, which seemed to be true. Thomas Wouters noted that he had not mentioned exceptions, which Van Rossum acknowledged. He has heard about some bad experiences with Java exception checking, so he avoided dealing with that for now. Langa, who is another co-author of the PEP, agreed that exceptions are "out of scope for now".

A PyCharm developer spoke up to note that the project has been doing type inference on Python programs for four years or so. The type system in the PEP is similar to what PyCharm uses, so "we feel it fits the development needs well". PyCharm can infer types for 50-60% of variables in user code, but can't get further than that without getting type annotations for function parameters.

Steve Dower said that the PEP should work well with Visual Studio, though there were still some issues to think about. It currently works by inferring types from the docstrings but could take advantage of the annotations. Other projects and companies also seemed happy with the changes.

Langa noted that at Facebook, at least, having optional typing available (as the Hack language does) eventually led to a cultural shift at the company. At some point, not having the annotations became a red flag during code review, so the company's code is moving toward type annotations everywhere.

Index entries for this article
Conference	Python Language Summit/2015

---

to post comments