Wrangling the typing PEPs

When last we looked in on the great typing PEP debate for Python, back in August, two PEPs were still being discussed as alternatives for handling annotations in the language. The steering council was considering the issue after deferring on a decision for the Python 3.10 release, but the question has been deferred again for Python 3.11. More study is needed and the council is looking for help from the Python community to guide its decision. In the meantime, though, discussion about the deferral has led to the understanding that annotations are not a general-purpose feature, but are only meant for typing information. In addition, there is a growing realization that typing information is effectively becoming mandatory for Python libraries.

Background

Annotations in Python are meant to allow specifying some attribute that gets associated with, originally, function parameters and their return value; eventually that mechanism was extended to variables as well. From the beginning, annotations were used to specify type information for those elements, but the language itself did not require anything other than syntactically correct Python for the value of an annotation; it would turn the value into a Python object that gets stored. Until Python 3.5 in 2015, there was not even a standard on how to specify type information for annotations; that came from the type hints effort.

The annotation information is available at run time in the __annotations__ dictionary for the annotated object, but type hints were largely meant to be used by static type checkers and other tools that never actually consult that dictionary. Instead, those kinds of tools simply parse the Python themselves. But there are libraries that actually use the annotations at run time. There may even be uses of annotations that are not tied to typing information at all, though examples of that are thin on the ground. The Python ecosystem is enormous, however, and the annotations feature was never strictly tied to typing, so Someone Out There surely could be doing their own thing.

PEP 563 ("Postponed Evaluation of Annotations") was accepted in 2017, added as an opt-in feature in Python 3.7, and was set to replace the existing implementation in Python 3.10. The problem with forward references to types that have not yet been defined was the main impetus for the PEP; it deferred the evaluation of annotations by storing them as strings and requiring anyone who wanted to use them as Python objects to call eval() or typing.get_type_hints() to evaluate them. This new behavior was gated by a __future__ import, but was set to become the only available behavior in 3.10.

PEP 563 had further implications beyond just changing how the annotations were handled by the language. In particular, the evaluation of the strings required for run-time uses of the annotation values was done in a different scope than when it was done at compile time, which led to various problems for that use case. So PEP 649 ("Deferred Evaluation Of Annotations Using Descriptors") was a late-breaking proposal that was meant to fix some shortcomings of the earlier PEP. Instead of storing the annotations as strings that need to be evaluated later when they are used, PEP 649 turned them into descriptor functions that would be run once, the first time elements of __annotations__ are accessed. Those functions would effectively operate in the same scope as the existing interpreter uses when it generates the values, so the net result would be generally the same as the existing behavior.

Back in April, given the questions surrounding the feature, the imminent Python 3.10 feature freeze, and the two competing PEPs, the steering council deferred switching the behavior to that of PEP 563. The background here is meant as a capsule summary; the articles linked to (and the links from those) will provide lots more history and details for interested readers.

More recent events

In October, PEP 563 author Łukasz Langa published a lengthy blog post discussing the two PEPs and their strengths and weaknesses. In it he noted that it would be sensible to simply adopt PEP 649 had PEP 563 never come about. Langa found much to like about PEP 649:

Looking at PEP 649 in isolation, it provides a more flexible and elegant solution to the problem. It avoids unparsing annotations from the AST [abstract syntax tree] form back into strings, which is AFAICT [as far as I can tell] unheard of in the world of programming language implementations. It avoids proliferation of strings in annotations which means that in many happy cases the user might be blissfully unaware of them. With PEP 563 they’re user-visible which is suboptimal. Clearly, as long as the PEP can be similarly performant, it represents better engineering.

But, since there is no option to go back in time to before PEP 563 was adopted, that is not the world we live in. If, for example, PEP 563 were to be deprecated in favor of PEP 649, he asked, where would that leave code that is using the new feature and that needs to be compatible with a wide range of Python versions? Depending on whether PEP 649 was added as an opt-in feature (behind a different __future__ import), as the PEP itself suggests, or if it is adopted as the language default, as has also been discussed, there are different kinds of problems for developers using the feature. Langa suggested that maybe a middle course could be found where the one specific case that led to the deferral of PEP 563 as default could use a PEP 649 technique:

This approach, while less pure than PEP 649, would entirely avoid the necessity for complex deprecations which I believe would provide a better end user experience. It would solve what made people unhappy while keeping what makes PEP 563 effective (its availability from Python 3.7 and runtime efficiency).

On November 17, Barry Warsaw posted a note to the python-dev mailing list on behalf of the steering council. It described the status of the two PEPs from the perspective of the council, while rendering a verdict (for now):

We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs.

Defer decision on PEP 563 and 649 in 3.11

As such, at this time, the only reasonable path forward that the SC [steering council] sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users.

The post included a call for help from the community in order to better understand the requirements for both static and dynamic typing throughout the Python ecosystem. There is also a need to rally "typing enthusiasts to help build consensus with either of the proposed PEPs" or to find an alternative, perhaps along the lines of what Langa suggested. Beyond that, the council is looking for a neutral party who can help shepherd some kind of a solution going forward.

As might be guessed, that set off a longish thread discussing the issues. One thing that was clear, at least from the wording of that announcement, is that the council was only really considering annotations for typing purposes, thus precluding other uses of annotations, by implication anyway. Christopher Barker pointed that out as something that could probably use some clarification:

Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases.

He briefly described his use case, which will no longer work with the PEP 563 changes. His code was written after PEP 563 was approved (with the language he quoted), so it may be his "fault" that he used the annotations feature incorrectly. If that is the case, he will be a bit disappointed, but there is a larger issue at hand, he said:

But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years. But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional. In short, it was made clear that anyone not interested in typing could safely ignore the discussions about it. And thus, a number of PEPs came and went, and those among us that did not choose to involve ourselves in the conversation did not pay attention to the details.

And thus we didn't notice that buried in what seemed like a typing PEP, was, in fact, a [deprecation] of any non-typing uses of an existing Python feature. (also to be fair, the title of the PEP is "Postponed Evaluation of Annotations" -- so should have caught the attention of anyone interested in annotations for any use.

Paul Moore agreed with Barker that more clarity is needed:

It's becoming harder and harder for people not particularly interested in static typing to simply ignore it, and any use of annotations to affect runtime behaviour is in a weird grey area. And as a library author, I'm now finding that I'm getting requests to add typing to my code "for my users" (i.e., using types is no longer just a choice I make for my project, it's an API design issue).

Original intentions

While the non-typing users of annotations have a legitimate complaint, Stephen J. Turnbull said, he believes that the writing has been on the wall for some time. His understanding is that typing information is and always has been "considered the primary use case for annotations" by former benevolent-dictator-for-life (BDFL) Guido van Rossum, and that the council has been following that lead. Greg Ewing remembered things somewhat differently:

[...] the BDFL didn't say that, or at least didn't say it very clearly. It sounded more like type hints were just one of many possible uses, and he encouraged people to experiment. There were even discussions about coming up with a convention to manage conflicting uses of annotations in the same code. That wouldn't have happened if typing were considered the only supported use.

Van Rossum is not sure that his communication was completely clear, but said that at least in his mind annotations were always about typing:

My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually. Some folks wanted to enforce types at runtime. Others wanted to use them to generate faster code. Yet others wanted types to be checked by the compiler. The term "gradual typing" wasn't invented (or hadn't reached our community) yet, and offline static type checking wasn't something we had thought of either (I think). But it was clear that typing would have to be optional.

Warsaw remembers things a little differently as well:

My recollection of the history of annotations falls somewhere between Greg’s and Guido’s. Annotations as a feature were inspired by the typing use case (with no decision at the time whether those were to be static or runtime checks), but at the same time allowing for experimentation for other use cases. Over time, annotations-for-typing clearly won the mindset and became the predominant use case.

But Oscar Benjamin pointed to PEP 3107 ("Function Annotations") from 2006, which lists numerous use cases, many of which were not for typing, at least directly. "As I remember it, a decision about the purpose of annotations was *explicitly* not made when they were introduced." Antoine Pitrou concurred: "Annotations were purposefully use case-agnostic, and there was no stated desire to push for one use case or another."

The intent when annotations were added is not necessarily relevant 15 years later, because the language and its users have largely excluded non-typing use cases. Warsaw said that "while all the signs are there for 'annotations are for typing', this has never been explicitly or sufficiently codified". He suggested that a PEP be written to that effect, in order to remove all doubt.

[...] We can lament the non-typing use of annotations, but I think that horse is out of the barn and I don’t know how you would resolve conflicts of use for typing and non-typing annotations. It’s been a slow boil, with no definitive pronouncement, and that needs to be fixed, but I think it's just acknowledging reality. That’s my personal opinion.

Pressure

As Moore noted, there is an increasing effort to push for typing annotations in libraries throughout the Python world. Steve Dower also sees that type checking is not really optional for many projects, which is leading to some people advocating for "annotations everywhere". The "optional" typing feature is heading toward being mandatory, Moore said:

There's a subtle (maybe not so subtle, actually) and increasing pressure on projects to add typing. Often with little or no justification beyond "you should", as if having typing is a sort of "obvious best practice". Sometimes "because it will make it easier for your users who use typing" is given as a justification, but while that's fair, it's also a disturbing gradual pressure for typing to extend everywhere, manifesting by making it feel like not adding typing is somehow "not caring about your users".

Benjamin said that code editors may be behind some of that push. His students sometimes start adding type annotations even though he has deliberately avoided the topic. "I can only presume that some editor is doing this for them or telling them that they need to do this (students often can't tell the difference between editor warnings and actual errors)." He has also seen a push to add annotations to SymPy, some of which are not particularly useful from a typing perspective but make some editors work better:

Some people apparently want to add type hints that look completely useless to me like

    def f() -> Union[MyClass, Any]

As I understand it this does not give any meaningful information to a type checker but it apparently makes vscode work better

Benjamin is not exactly opposed to adding typing information to SymPy, "but it's a huge amount of work that someone would have to do and I don't want to add useless/inaccurate hints temporarily (as some have suggested to do)". Moore agreed, but said "it is awfully tempting to passive-aggressively annotate everything as Any, just to shut people up :-(". Steven D'Aprano took that joke one step further by suggesting setting up a logical conflict between PEP 8 purism and typing activism:

We could update PEP 8 to ban type annotations, then watch as the people who over-zealously apply PEP 8 to everything AND over-zealously insist on adding type annotations to everything have their heads explode.

Benjamin had suggested that he saw more value for type annotations on the internals of SymPy, but Sebastian Rittau said that users generally want this information for the API, which can be supplied from stub files. Those files, which are only consumed by type checkers, editors, and similar tools, can be created and maintained outside of the projects; they contain type information for the APIs of various libraries. He listed several resources for those who are trying to add typing information (or review pull requests that add it), including a documentation hub, "but there is not much to see at the moment".

Moore was glad to see the pointers, but said that "the most critical missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials". He continued:

TBH [To be honest], I'd quite happily not use typing if I didn't want to and stay quiet. A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it" attitude, whether that's because of community pressure, tool requirements, or whatever.

Rittau said that the documentation hub was meant to eventually have the kinds of information Moore is looking for. Rittau also noted that the typeshed project may help provide a stepping stone:

Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help. While we of course prefer high quality stubs or type annotations shipped with the package in question, typeshed can provide a fairly low barrier of entry for projects that don't have the resources to maintain type annotations themselves. It can also be used as an "incubator", where stubs are created and improved iteratively, until they are deemed ready for inclusion in an upstream package.

The future

Those are valuable resources, obviously, but they do tend to reinforce the message that the future of Python is typed. The pressure that library developers are feeling is real and likely to increase as more and more tools—and developers—come to depend on the availability of typing annotations. While they are optional from a language perspective, they are rapidly becoming mandatory from a community and ecosystem perspective. That is precisely what the "typing-suspicious crowd" (as Turnbull called them) has been worried about and it has come to pass—or soon will.

Meanwhile, though, it seems clear that anyone using annotations for non-typing purposes should be figuring out some other way to accomplish their goals. But the resolution of the two PEPs does not seem any closer at this point. Neither Langa or PEP 649 author Larry Hastings seem inclined to change their PEPs, at least yet, and the hoped-for PEP shepherd has not appeared either (publicly, anyway).

Given that there are 16 months or so before Python 3.12 feature freeze, it might be guessed that situation will have worked itself out by then. The contours of the problem are clearer, and some of the extraneous pieces have been removed from consideration, which should hopefully clear the way for a consensus to emerge. Since "practicality beats purity", according to The Zen of Python, something like what Langa has proposed may well be the "winner". It would seem that 2022 will provide more opportunities to finally put this issue to bed.

from os import fspath, PathLike

def do_something_with_path(path: str | PathLike) -> …:
    …
    f = open(fspath(path))
    …

PathOrStr = TypeVar('PathOrStr', str, PathLike)

def do_something_with_path(path: PathOrStr) -> ...:
    …

T = TypeVar('T', bound=SomeGenericClass)

def foo(a: object, b: type[T]) -> TypeGuard[T]:
    if not isinstance(a, b):
        return False
    # Do additional checks here...