"Structural pattern matching" for Python, part 1
We last looked at the idea of a Python
"match" or "switch" statement back in 2016, but it is something that has
been circulating in the Python community both before and since that coverage.
In June it was raised again, with a Python Enhancement Proposal (PEP)
supporting it: PEP 622
("Structural Pattern Matching
"). As that title would imply, the
match statement proposed in the PEP is actually a pattern-matching
construct with many uses.
While it may superficially resemble the C switch statement, a
Python match would do far more than simply choose a chunk of code
to execute based on the value of an expression.
Proposal
Guido van Rossum
introduced
PEP 622 to the python-dev mailing list on June 23; he was one of
five co-authors of the PEP along with Brandt Bucher, Tobias Kohn, Ivan
Levkivskyi, and Talin. Van Rossum's introduction did not include the PEP
itself (original
version), which is somewhat unusual, but instead consisted of the contents of
a README
file, "which is shorter and gives a gentler introduction
than the PEP itself
". It notes that the python-ideas mailing list
had "several extensive discussions
" about some kind of match
statement over the years, which were summarized by Kohn in
a blog
post in 2018. The introduction starts with a fairly simple example:
def http_error(status):
match status:
case 400:
return "Bad request"
case 401:
return "Unauthorized"
case 403:
return "Forbidden"
case 404:
return "Not found"
case 418:
return "I'm a teapot"
case _:
return "Something else"
That is pretty self-explanatory except perhaps for "_", which is a wildcard that will match anything. The idea is that each case will be tried in the order specified, the first to match will be executed, and the match statement terminates; only zero or one of the case entries will be executed and there is no falling through as with C. But this example is also deceptively similar to the C switch statement. Multiple case values can be combined in a single line, however, (e.g. case 401|403:) which is somewhat more compact than the C equivalent.
The next example starts to show where the proposed match statement takes a different path:
# The target is an (x, y) tuple
match point:
case (0, 0):
print("Origin")
case (0, y):
print(f"Y={y}")
case (x, 0):
print(f"X={x}")
case (x, y):
print(f"X={x}, Y={y}")
case _:
raise ValueError("Not a point")
If point is not a tuple of two values, it will match the wildcard case and raise the exception, otherwise it will match one of the four cases given. For the three cases that are not the origin (i.e. (0, 0)), the variables in the case get replaced with their corresponding values from point, thus they can be printed out. These variable "extractions" are new, different behavior than what may be expected in Python; a point that is (0, 1) will not only match the second case, it will bind y to the value 1. Here is some sample output:
# point = (23, 0)
X=23
# point = (0, 'foo')
Y=foo
# point = ('0', '0')
X=0, Y=0
There is lots more to the proposal, as the introductory message and the voluminous PEP describe, including using Python dataclasses, guard clauses, sequence and mapping patterns, extracting sub-patterns with the walrus operator (":=") that came from the contentious PEP 572, and more. Here is an example that (perhaps nonsensically) combines several of those in the introduction:
from dataclasses import dataclass
@dataclass
class Point:
x: int
y: int
def test(item):
match item:
case Point(x, y) if x == y:
print('A Point on the diagonal')
case [ Point(x1, y1), p2 := Point(x2, y2) ]:
print(f'Two Points, x coord of the first is {x1}, the second Point is {p2}')
case { 'bandwidth' : b, 'latency' : l }:
print(f'not a Point but it has {b} for bandwidth with {l} for latency')
The first case uses a guard clause on the Point data
class to restrict it to only match on points with the same value for
x and y. The second uses a sequence pattern to match a
sequence of two points, extracting the second into p2. The last
case is checking for a mapping object that has an entry for the
two fields named bandwidth and latency. These patterns
can be arbitrarily complex and composed in various ways, as might be expected.
One of the areas that drew complaints was the syntax for constants. Instead of using hard-coded values in match statements, developers will likely want to use symbolic constants, as the following example shows:
RED, GREEN, BLUE = 0, 1, 2
match color:
case .RED:
print("I see red!")
case .GREEN:
print("Grass is green")
case .BLUE:
print("I'm feeling the blues :(")
Reaction
Antoine Pitrou had several concerns, but one of those was the "context switch" required when reading the code:
He suggested alternative syntax using with or @ rather than making the case look like the creation of a new object. In a somewhat similar vein, Greg Ewing suggested making it explicit that variables in the case statements were being bound:
E.g.
case Point(?x, 0):This would also eliminate the need for the awkward leading-dot workaround for names to be looked up rather than bound.
The PEP says this was rejected on the grounds that binding is more common than matching constants, but code is read more often than it is written, and readability counts.
Van Rossum noted
that the PEP authors "did plenty of bikeshedding in private
"
about how to specify a case condition; he suggested that if there were a
groundswell of support for some alternative, it could be adopted instead.
Daniel Moisset was strongly in favor of the overall idea, but took some of
the concerns even
further, suggesting that the use of the dot on constants makes it easy
for developers to shoot themselves in the foot:
He listed two possibilities that he found to be less dangerous: using angle brackets around to-be-captured variable names (e.g. "<x>") or effectively putting the capture variables in their own namespace by matching into a "capture object":
match get_node() into c:
case Node(value=c.x, color=RED): print(f"a red node with value {c.x}")
case Node(value=c.x, color=BLACK): print(f"a black node with value {c.x}")
case c.n: print(f"This is a funny colored node with value {c.n.value}")
Rob Cliffe was generally in favor of the PEP, but was concerned
with using "|" for listing alternatives in matches given that it
already has a different meaning (i.e. bitwise OR). In addition, he agreed
with others who were suggesting that "else" be used instead of the
wildcard value as the default case when nothing else
matches. Using "_" "is obscure, and [wouldn't] sit well with
code that already uses that variable for its own purposes
". The
single-underscore "variable" is a convention used in Python to denote a value that is
not needed (i.e. a throwaway value); for example:
x, _, z = foo
That will unpack the three-element foo sequence and throw away the middle value. Its connection to wildcards is somewhat tenuous, though. Chris Angelico wondered if using the ellipsis ("...") made sense instead, which Barry Warsaw also favored. But Van Rossum said that he would find ellipsis confusing:
He also pointed out that other languages use "_" for wildcards as well. The current version of the PEP puts it this way:
One counterargument is that internationalization (i18n) libraries often use the underscore to indicate a string that needs translation (e.g. _('This string should be localized')), which makes the use as a wildcard somewhat confusing. The two uses would not collide, though, because the pattern for a case is treated differently than other language constructs—which may be part of what's causing some of the negative reaction to the PEP.
Marc-André Lemburg, who wrote the rejected PEP 275
("Switching on Multiple Values
") back in 2001, was happy
to see the new PEP, but was surprised, like many others, about the lack
of an else for the catch-all case. Van Rossum, who wrote
the also-rejected
PEP 3103
("A Switch/Case Statement
") in 2006 targeting Python 3,
noted
that, in the 19 years since PEP 275, "there are now some
better ideas to steal from other languages than C's switch. :-)
" He
said that the authors of the current PEP were split on the question of
else to some extent.
A wrinkle with `else` is that some of the authors would prefer to see it aligned with `match` rather than with the list of cases, but for others it feels like a degenerate case and should be aligned with those. (I'm in the latter camp.)
He said that the authors were still discussing it and, once they reach
agreement, would update the proposal. Lemburg was concerned
that "case _:" is "just too easy to
miss when looking at the body of 'match'
". But as Van Rossum and
others noted, a case with just a wildcard (whatever the syntax for
that ends up being) will be valid syntax, so an else is not
strictly needed, though it may still be desirable for other reasons.
The PEP also had a lengthy section on a __match__() protocol that could be used to customize how objects are matched, but it turned out to be a confusing idea with a lot of unclear corner cases. It drew lots of complaints and has been dropped in later versions of the PEP.
Gregory P. Smith raised concerns about confusing readers of the code, especially those who are not well-versed in Python. He posited showing a chunk of code using match to someone who does not know the language:
match get_shape():
case Line(start := Point(x, y), end) if start == end:
print(f"Zero length line at {x}, {y}")
I expect confusion to be the result. If they don't blindly assume the
variables come from somewhere not shown to stop their anguish.
With Python experience, my own reading is:
- I see start actually being assigned.
- I see nothing giving values to end, x, or y.
- Line and Point are things being called, probably class constructions due to being Capitalized.
- But where did the parameter values come from and why and how can end be referred to in a conditional when it doesn't exist yet? They appear to be magic!
Did get_shape() return these? (i think not). Something magic and implicit rather than explicit happens in later lines. The opposite of what Python is known for.
He suggested that making the match conditions look like a call to a class
constructor would simply be too confusing. He presented alternative
syntax, but Ewing called
that "inscrutable [...] in its own way
". Paul Moore disagreed
with the idea that those with Python knowledge would be completely confused
as portrayed by Smith; it may take a bit of effort to get there, but Moore is
confident that developers would come up to speed on match fairly quickly.
Needed?
Mark Shannon questioned
whether the PEP truly outlined a serious problem that needed solving in
the language. The PEP describes some anecdotal evidence about the
frequency of the isinstance()
call in large Python code bases as a justification for the match
feature, but he found that to be a bit odd; "[...] it would be
better to use the standard library, or the top N most popular packages from
GitHub
". He also wondered why the PEP only contained a single
example from the standard library of code that could be improved using
match. "The PEP needs to show that this sort of pattern is widespread.
"
The example cited by the PEP does make it clearer why isinstance() is mentioned. The basic idea is that heterogeneous data is frequently "destructured"—objects of various sorts have their internal data pulled out in different ways—in large Python code bases. The original PEP puts it this way:
We believe this will improve both readability and reliability of relevant code. To illustrate the readability improvement, let us consider an actual example from the Python standard library:
def is_tuple(node):
if isinstance(node, Node) and node.children == [LParen(), RParen()]:
return True
return (isinstance(node, Node)
and len(node.children) == 3
and isinstance(node.children[0], Leaf)
and isinstance(node.children[1], Node)
and isinstance(node.children[2], Leaf)
and node.children[0].value == "("
and node.children[2].value == ")")
With the syntax proposed in this PEP it can be rewritten as below. Note that the proposed code will work without any modifications to the definition of Node and other classes here:
def is_tuple(node: Node) -> bool:
match node:
case Node(children=[LParen(), RParen()]):
return True
case Node(children=[Leaf(value="("), Node(), Leaf(value=")")]):
return True
case _:
return False
The proposed syntax is far more clear—though there are some conceptual hurdles to surmount—but it is not so obvious that the problem is truly widespread. Shannon would seem to be concerned that match may be a feature that is in search of real use cases.
Moisset said that he had put together some extensive notes on the feature. In them, he argued that the feature was not really about pattern matching and was, instead, about introducing algebraic data types, which come from functional languages, into Python. Beyond that, he also described the proposal rather more clearly than the PEP itself does, which is likely part of what got him invited to join the author group of the PEP for round two.
Just over 24 hours after his initial post, Van Rossum called for something of a pause in all of the comments pouring into the mega-thread. He noted four separate items that the PEP authors now knew were contentious (an alternate spelling for "|", else, a different wildcard token instead of "_", and what to do about the dot notation for constants) and asked that folks wait to add additional comments on those choices until the authors could come to some agreement. The final item was also meant to cover the possibility of changing to marking variables (e.g. "?foo") rather than constants as had been suggested. He asked that any other concerns with the PEP be concisely added to his new thread, which several did—though at a much-relaxed pace compared to the original.
That takes us up near the end of June in this tale, but there is more to come. The authors came back with a second version of the PEP, without the __match__() protocol, and dropping the dot notation for constants, replacing it with a requirement that constants in case entries be referenced from some namespace (thus have a dot in their representation: Color.RED), but making few other substantive changes—beyond a gentler introduction courtesy of Moisset. That set off another mega-thread along with several other threads discussing specific aspects of the PEP. We will pick up where we left off soon; stay tuned.
| Index entries for this article | |
|---|---|
| Python | Enhancements |
| Python | match statement |
| Python | Python Enhancement Proposals (PEP)/PEP 622 |