Reviving None-aware operators for Python

The idea of adding None-aware operators to Python has sprung up once again. These would make traversing structures with None values in them easier, by short-circuiting lookups when a None is encountered. Almost exactly a year ago, LWN covered the previous attempt to bring the operators to Python, but there have been periodic discussions stretching back to 2015 and possibly before. This time Noah Kim has taken up the cause. After some debate, he eventually settled on redrafting the existing PEP to have a more limited scope, which might finally see it move past the cycle of debate, resurrection, and abandonment that it has been stuck in for most of the last decade.

Mark Haase and Steve Dower first proposed adding None-aware operators to Python in PEP 505 ("None-aware operators"). At the time, they suggested adding three different operators to Python to handle "None coalescing" (using a default option if something is None), None-aware attribute access (accessing an attribute of a potentially None object without raising an exception), and None-aware indexing (the same but for lists and dictionaries rather than objects). In support of these changes, they noted that many other languages have something similar, and that the operators can make code more concise and readable. The exact details of the translation into Python's existing semantics were the topic of some debate, but the simplest version would be something like this:

    # Current code:
    value = a if a is not None else b
    # Alternate current code that is subtly broken for false-y values:
    value = a or b
    # Proposed code:
    value = a ?? b

    # Current code:
    value = a.b if a is not None else None
    # Proposed code:
    value = a?.b

The original discussion of the PEP didn't end up leading to a decision, and neither did any of the follow up discussions in 2018 , 2022, or 2023. In mid-December 2024, Kim posted a topic in the Python discussion forum seeking to revive the PEP, hoping specifically to "solicit feedback on how I can get the process moving again." Emily Morehouse, a member of Python's Steering Council, was in favor of the idea, offering to sponsor the PEP. Guido van Rossum also approved of the PEP, noting that he had found the equivalent operators in TypeScript to be quite useful. He did raise the question of whether a.b?.c should be None when a.b is None, when b is missing from a, or (his preferred option) both.

Lucas Malor agreed that the ?. operator ought to work when an attribute was missing, but thought that the ?? operator should work differently, and raise an exception if its left argument is undefined. Kim disagreed with Van Rossum and partially agreed with Malor, saying that a.b?.c should raise an exception if a exists and is not None, but does not have an attribute b. Over the rest of the discussion, several people expressed support for all of those alternatives and more.

Python-specific concerns

Van Rossum shared his reasoning for the suggestion by comparing the proposed behavior to JavaScript. In that language, the use of undefined to represent missing attributes makes the behavior of the operators more consistent — and therefore useful for traversing JSON maps that have some ad-hoc structure.

Python works different – if you have a JSON dict a that may or may not have a key k, a["k"] raises, and if we want to have a["k"]?.f to return None, it looks like the ? operator would have to catch the KeyError from a["k"]. If we don't catch that exception, ?. would be much less useful than it is in JavaScript. The same reasoning applies to a.f?.g – we'd have to catch the AttributeError.

But he noted that approach was likely to be contentious, given that "there are a lot of people in the Python community (including core devs) who abhor operations that silence exceptions without an except clause". Kim proved to be one such person, saying that while he appreciated Van Rossum's point, he worries that suppressing exceptions "will introduce as many footguns as we've solved." For example, he pointed out that some developers wouldn't want such an operator to suppress all AttributeError exceptions, or it could potentially silently hide bugs in the code. He later said that he would not support the proposal if it suppressed exceptions — something that Brett Cannon wryly pointed out would put Kim in the same group as the original PEP authors, who don't like the idea of any None-aware operators anymore.

Cannon summarized the disagreement as being about safe traversal of structures versus handling None. Ultimately, he thought that no matter which design was picked, it would remain possible to write bad code using it: "You can't protect everyone from every conceivable issue and at some point have to lean on people being consenting adults." Dower thought that Cannon had identified "the key point".

"Sxderp" pointed out that everyone seemed to more or less agree about the meaning of ??, and that the other operators (?. and ?[]) were the ones causing disagreement. They proposed dropping the other operators from the PEP to focus on ??, an approach that Paul Moore agreed with.

At that point, the discussion changed to whether the ?? operator ought to only check for None, or whether there should be some way to make it work with other sentinel values. Cannon was of the opinion that the language should avoid making None "special", since Python is notable for its pervasive ability to let the user override operators and other language features. Van Rossum disagreed, saying that None was "special, intentionally". Many places in the standard library use None as a special case, and new syntax wouldn't change that.

More magic

Several people went back and forth proposing translations into Python's existing syntax before Van Rossum proposed an even more radical idea: introducing a postfix ? operator that generalizes the effects of the other operators to attribute access, indexing, and bare expressions. With this proposal, a single ? operator would handle short-circuiting an entire expression to be None if any part of it was None, or if there were any missing attributes or keys:

    # Using postfix ? on a lookup:
    value = a.b? # Same as original proposal's "a?.b"
    value = a['key1']['key2']?

Kim supported the idea, but it didn't gain many other supporters, since people were already leery about the existing proposal. Even the less-controversial option had a few detractors, though. Hugo van Kemenade quoted Dower's original explanation for why he initially withdrew PEP 505:

The reason I eventually kind of withdrew that is [I had] a really interesting discussion with Raymond Hettinger at one of our core dev sprints where he basically laid out: the features that go into the language actually change the way people program. And it has far more of a follow-on effect than just how that particular thing gets written. It's going to affect how the rest of the code is written.

At the time, Dower became convinced that adding features like None-coalescing would make Python programmers less likely to validate their data, which would cause verification logic to be spread throughout the code. Kim doesn't see that as a problem, but Dower still holds that position, going so far as to say that if Kim doesn't believe that adding these operators will change the way people use Python, then there's not much point in the proposal. So it's up to the proponents of the PEP to demonstrate that the resulting change would be positive.

Van Rossum shared a detailed example of the kinds of problems he was currently dealing with in TypeScript, and what the Python version would look like with and without the new syntax. Moore thought that kind of code was common, but remained unconvinced that this should motivate a change to the language, rather than a simple addition to the standard library to make traversing data with included None values easier. In fact, several people mentioned existing Python libraries designed to help with this use case, especially Pydantic. Supporters of the proposal were not convinced that external libraries were as useful as an extension to the language, however.

A major point of contention with Van Rossum's proposal was the fact that TypeScript effectively relies on its type checker to prevent accesses to undefined attributes. If Python adopted the TypeScript-flavored approach that Van Rossum suggested, a type checker would be all but mandatory in order to catch things that currently cause AttributeError exceptions in Python, once people start using None-aware operators everywhere. Michael H responded to one of Van Rossum's attempted explanations by saying:

Unless you're suggesting that static typing is required for a first-class experience with new features in Python, this isn't actually any different than getattr is, and I think this is actually an argument against the feature.

He went on to point out that, for all of the possible ideas being tossed around in the thread, several people had given examples that were actually incorrect, showing just how difficult a safe-traversal operator would be to understand and use correctly.

The future

By the end of the discussion, the only real conclusion that had been reached was that Kim intended to rewrite a more focused version of the PEP that dealt only with the ?? operator. Whether that less-controversial proposal will get anywhere remains to be seen, but PEP 505 is still provoking a lot of discussion. For example, Guillaume Desforges posted a separate topic pointing out that the discussion was really going around in circles, and asking whether that had happened before and how the community should address it.

The answer is that, while the community has spent a lot of time discussing the PEP, it has never actually been proposed to the Steering Council. They are the people who must ultimately accept or reject it, so until someone does officially propose it, the idea will never be definitively rejected. For the Steering Council to approve a proposal, however, usually requires the community discussion to have reached some kind of agreement.

Ultimately, with so many competing opinions, the Python community is unlikely to settle this particular discussion anytime soon. But if Kim does rewrite PEP 505, it may come closer than it has in a few years to seeing the matter settled.