Not coalescing around None-aware

The wish for a "None-aware" operator (or operators) is longstanding within the Python community. While there is fairly widespread interest in more easily handling situations where a value needs to be tested for being None before being further processed, there is much less agreement on how to "spell" such an operator (or construct) and on whether the language truly needs it. But the idea never seems to go away, with long discussions erupting every year or two—and no resolution really in sight.

PEP 505

We looked at the idea back in 2018, but PEP 505 ("None-aware operators") goes back to 2015; it seems likely that the idea had been discussed well before that as well. At the time of our article, the Python community was still working on figuring out how it would be governed after Guido van Rossum stepped down from his benevolent dictator role. PEPs were not being accepted or rejected because of that and PEP 505 was deferred.

In Python, it is not uncommon to see code like the following (taken from the more extensive examples in our earlier article):

    if x is None:
        x = 42

The PEP proposes the "??" None-aware (also known as null-coalescing) operator. It returns the left-hand side if it is not None, otherwise the right-hand side is evaluated and returned. So that code above could be replaced with:

    x = x ?? 42

There is also an "augmented assignment" version ("??=") and two operators that apply the None-awareness to indexing ("?[]") and to attribute access ("?.") proposed as well:

    x ??= 42   # same as above
    x?['foo']  # retrieve dict entry 'foo' if x is not None
    x?.foo     # retrieve attribute foo if x is not None

In the last two examples above, if x is None, then the expression evaluates to None as well.

The PEP has come up again a few times since the Python governance upheaval put it on the back burner, including in a Python discussion-forum thread back in April 2021. Kumar Aditya asked about adding a null-coalescing operator; Steven D'Aprano pointed to the PEP and some of the previous discussions. Along the way, PEP co-author Steve Dower said that his plan for the PEP is to drop it "because I now think the additional syntax would not make Python cleaner or a better language".

The PEP came up again in October 2021 when Doug Swarin raised it on the python-dev mailing list. As with Aditya and others, Swarin pointed to the existence of similar operators in other languages (such as JavaScript) and said that it would "result in a significant reduction in boilerplate code" testing for None. He suggested that the indexing ("maybe subscript") and attribute-access ("maybe dot") operators might actually lead to a decrease in code readability, so perhaps they could be removed from the PEP, though he was not personally in favor of doing so. He had also created a pure-Python implementation that he made available for testing.

Van Rossum, who has generally had a favorable inclination toward the PEP all along, thanked Swarin and said that he hoped more discussion would "help convince the Steering Council to accept it". The conversation proceeded along familiar lines, with some strongly advocating for at least the addition of ?? and ??=, while others were expressing concerns about the readability. The specter of "Perl-ification" was present as well; to some, adding more operators of this sort just makes the language look like Perl (or, worse still, line noise).

Revival

On December 11, "yingshaoxo" revived the forum thread that started 18 months earlier and the discussion was off and running again. Yingshaoxo advocated adopting the feature, while noting that multiple languages (JavaScript, Flutter, Kotlin, and C#) already have it. Cameron Simpson noted that there is an existing way to handle the example yingshaoxo posted, but that it does not work in all cases:

    # proposed new syntax
    new_variable = a_variable ?? 'Hello Python'

    # "same" thing with existing syntax
    new_variable = a_variable or 'Hello Python'

That relies on a_variable evaluating to a false value, which is the case for None, but it is also the case for the empty string, 0, and a few other values. D'Aprano pointed out that the same argument was made to resist adding a ternary operator to the language, until PEP 308 ("Conditional Expressions") was abruptly adopted because of the problems that can occur with expressions that unexpectedly evaluate to false:

For many, many years we rejected every request for a ternary if operator with the argument that you can just write

    condition and if_true_value or if_false_value

E.g. len(mylist) > 0 and mylist[0] or "default". Can you see the bug? (Hint: what if the value of mylist[0] is, say, the empty string?).

What followed was a long discussion on "readability". "Fancidev" asserted that the conditional version was more readable:

"y if x is None else x" is more readable than "x ?? y".

Since code is written once and read 100 times, it's probably worth the extra typing.

As might be guessed, others disagreed. Tzu-ping Chung said that the conditional form "requires more mental attention", but did recognize that there is a potential hurdle in needing to learn what ?? means. D'Aprano said that those coming to Python from other languages often complain that the conditional construct in Python is unreadable. "So maybe readability depends on what you are used to."

Fancidev pointed out that ?? can only be applied to a subset of the situations where the more general conditional expression can be used; it requires learning about the operator but can then only be applied to None tests. "Vladimir" agreed, but linked to a section of PEP 505 that argues for ?? because it is more concise. The PEP has an example of initializing default values:

    data = [] if data is None else data

    # or

    data = data if data is not None else []

Either of those will work, but the first puts the operands in an "unintuitive order", while the second is a bit longer and repetitive (i.e. "data if data"). Using the proposed operator, it would simply be:

    data = data ?? []

To Fancidev, those examples just reinforce their belief that the conditional version is more readable, but Chris Angelico argued that "readability" is totally subjective, at least as it is used in discussions like these:

I personally suspect that some people consider something "readable" on the basis that "I can understand what it does based on my pre-existing knowledge of what Python can already do" (meaning that new syntax is ALWAYS less 'readable' than a verbose form that already exists), and other people consider something "readable" on the basis that it is compact and expresses a thought that can be fit into a larger "sentence" or "paragraph" (meaning that a new syntax is almost always more 'readable' than the more-verbose form that already exists).

Coalesce

Marc-André Lemburg took a different tack on the idea. He pointed to the PostgreSQL COALESCE() function as a possible model for "an explicit and elegant way" to solve many of the problems that PEP 505 describes. He suggested adding a Python builtin along the following lines:

coalesce(*args, /, check=None, logic=False)

Return the first value from args which is not the check object (defaults to None). If check is a callable, return the first value from args which check(value) is False. logic may be set to True, to swap the test logic.

The "/" in the argument list separates the positional-only parameters (args in this case) from those that can be specified by position or keyword (check and logic). He goes on to list a few operators as possibilities for the check values, such as a new list.isempty() or the existing math.isnan().

But there is a problem with that approach, as several people pointed out: there is no ability to short-circuit the evaluation of some of the arguments. Using a conditional expression (or the proposed ?? operator) would not evaluate the "else" argument if the value is not None. As Fancidev put it: "it does not lazily evaluate the arguments, which I imagine will be a necessary requirement". Peter Suter added some other ways where the operator approach is superior, including:

Chaining is much clearer and less error prone with operators:

(override ?? fallback).name ?? default
coalesce(coalesce(override, fallback).name, default)

Meanwhile, the "debate" over readability continued, much of it between Angelico and Vladimir, seemingly without changing any minds. In fact, early on in that part of the discussion, Angelico hit the nail on the head. In a message linked above, he decried the idea of "readability" because: "Everyone has their own definition, never backed by any sort of actual studies, and nobody ever changes their mind based on other people's examples." Eventually, after a good bit of back and forth on readability, the thread was locked on December 17.

As should be clear, though, it is a topic that comes up often; until the PEP is either accepted or rejected, it will undoubtedly come up again. Even if the PEP is rejected, it will not be a huge surprise to see the idea pop up again; it has obvious utility and other languages have something similar, which will make it that much more desirable. The arguments against the simplest form (just ?? and ??=) seem to mostly boil down to being a barrier to learning or understanding the language. But, as D'Aprano put it, Python has features for both casual and more serious programmers:

The beauty of Python is that it is accessible to casual programmers. You don't have to use null-coalescing operators any more than you have to write classes, or use closures, or use threads.

But we didn't let those casual programmers stand in the way of Python getting classes, closures, threads, async, regexes etc. Let the casual programmers continue to write using the basic features, and the power users use the power features.

It is hard to predict what will happen from here, especially given that the PEP's authors are no longer pushing it; it seems likely that some other core developer(s) would need to sponsor it for the newly elected steering council to even consider it. If that happens, perhaps an alternate spelling can be found to reduce the unhappiness with the "ugliness" of ??. A reduction in scope (eliminating ?. and ?[]) might also help the prospects of the PEP; down the road those additions could be considered again. One way or another, getting PEP 505 out of limbo would help clarify things quite a bit.

Index entries for this article
Python	None
Python	Python Enhancement Proposals (PEP)/PEP 505

---

to post comments

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> π = 3.1415
>>> print(π)
3.1415

if check_size_limits and (len(message.header ?? "") +
                          len(message.body ?? "")) > limit:
    raise Exception()

if check_size_limits:
    header_length = 0
    if message.header is not None:
        header_length = len(message.header)
    body_length = 0
    if message.body is not None:
        body_length = len(message.body)
    if header_length + body_length > limit:
        raise Exception()

if check_size_limits and ((len(message.header) if message.header is not None else 0) +
                          (len(message.body) if message.body is not None else 0)) > limit:
    raise Exception()

lazy = []
eager = []
for i in range(5):
    lazy.append(lambda: i)
    eager.append(lambda x=i: x)

print(lazy[2]())  # Prints 4
print(eager[2]())  # Prints 2