Corner cases and exception types

Some unanticipated corner cases with Python's new "walrus" operator—described in our Python 3.8 overview—have cropped up recently. The problematic uses of the operator will be turned into errors before the final release, but just what exception should be raised came into question. It seems that the exception specified in the PEP for the operator may not really be the best choice, as a recent discussion hashed out.

PEP 572 ("Assignment Expressions") describes the walrus operator (though not by that name, which came later). It allows making assignments as part of another statement, like an if or while statement—or in a list comprehension. It is this latter use where the corner cases recently arose. The following is how the walrus operator is meant to be used in lists or list comprehensions (from the PEP):

# Reuse a value that's expensive to compute
[y := f(x), y**2, y**3]

# Share a subexpression between a comprehension filter clause and its output
filtered_data = [y for x in data if (y := f(x)) is not None]

But a bug report from Nick Coghlan pointed out some oddities:

>>> [i := 10 for i in range(5)]
[10, 10, 10, 10, 10]
>>> i
10

>>> [False and (i := 10) for i in range(5)]
[False, False, False, False, False]
>>> i
4

He has submitted a pull request that makes some changes to the PEP (including slipping a "walrus operator" reference in) to outlaw these specific instances and others like them. The original PEP had specified that problems found in the parsing of the walrus operator would raise a TargetScopeError exception, which would be a subclass of SyntaxError. But that was questioned by Barry A. Warsaw in the bug:

Guido van Rossum agreed that perhaps it would be better to simply raise SyntaxError, though he was concerned that it would require a full PEP review. Coghlan explained the reasoning:

The discussion soon moved to the python-dev mailing list at the behest of Van Rossum. Warsaw started the thread by posting a summary of the debate. In his view, any PEP change would be relatively minor, so it makes sense to nail that all down before the 3.8 release, which is slated for October. In the bug report, Serhiy Storchaka had mentioned the IndentationError and TabError subclasses of SyntaxError as possible reasons to continue with TargetScopeError, but those "feel different" to Warsaw because those names are self-explanatory.

Tim Peters, who co-authored the PEP with Van Rossum and Chris Angelico, didn't really care what the exception is called, but was not convinced that SyntaxError is any better, really:

Most who posted in the thread were either in favor of switching to SyntaxError or ambivalent about such a change, with Steven D'Aprano being the main exception (so to speak). He was concerned with the idea of emitting a syntax error for something that was not syntactically incorrect. "There's a problem with the *semantics* not the syntax." But Warsaw was not convinced that the distinction is useful: "What you wrote is disallowed, so you have to change your code (i.e. syntax) to avoid the error." He also noted that PEP 572 specifies TargetScopeError as a subclass of SyntaxError, so even under the current definition the distinction is not being made.

Eric V. Smith concurred with Warsaw; in particular, Smith noted that he could not see a reason that a programmer would catch and handle SyntaxError and TargetScopeError separately. He suggested (as did others in the thread) that the text emitted for that error make it clear to the user where the problem lies. For example, Kyle Stanley suggested something like:

SyntaxError: Invalid scope defined by walrus operator: 'i := 10 for i in range(5)'

In a long post, Coghlan agreed with the overall consensus that a simple SyntaxError is the right path forward for 3.8. He does think that there is value in a new exception (perhaps with a better name, such as AssignmentScopeError) that would cover more than just walrus operator errors. But that is something that can wait to be considered for 3.9.

With three steering council members in favor (Coghlan, Warsaw, and Van Rossum) and two of the three PEP authors in agreement (Peters switched to being in favor after Coghlan's message), Van Rossum said that TargetScopeError should be removed. As it turns out, Angelico is also in favor and another council member, Brett Cannon, was on board as well. Coghlan's pull request for the PEP was updated to reflect that; it will presumably be merged along with the code changes.

This episode is another example of the Python development process in action. The long beta cycle for releases helps flush out problems, as with the escape sequences issue we looked at last week. In this case, the corner cases were found by MicroPython developers who were adding the walrus operator to their version of the language, so the diversity of language implementations helps find issues as well. Finding those problems in MicroPython ensured that the PEP would be fixed; the discussion around them allowed the exception issue to be worked out. It certainly shows the advantages of having an active community that is actually using and testing beta releases well in advance of the final release.