Python exception groups

Exceptions in Python are a mechanism used to report errors (of an exceptional variety); programs can be and are written to expect and handle certain types of exceptions using try and except. But exceptions were originally meant to report a single error event and, these days, things are a tad more complicated than that. A recent Python Enhancement Proposal (PEP) targets adding exception groups, as well as new syntax to catch and handle the groups.

Exceptions

As a quick review, Python exceptions are objects derived from the BaseException class, though user-defined exceptions should be derived from Exception. An exception can be "raised" when an error occurs (using raise), which causes the call stack to be unwound until either the exception is "caught" (in a try ... except block), or it propagates out to the user. Python defines multiple built-in exceptions with generally self-explanatory names, such as IndexError and SyntaxError. The basics of their operation can be seen below:

    >>> 1/0
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ZeroDivisionError: division by zero

    >>> try:
    ...     raise ValueError('value?')
    ... except ValueError as e:
    ...     print(e)
    ... 
    value?

That gives the gist of it, but there are some wrinkles, of course. The except clause can refer to multiple exception types to catch them all and there can be multiple except clauses for separate handling of different exception types. An else clause can be used to do special handling when no exception is caught and a finally clause can be given for code to be executed last, regardless of whether exceptions were caught or not.

Along the way, it was recognized that if an exception was raised while handling an exception, the first exception could get lost, which made for difficult debugging. That led to PEP 3134 ("Exception Chaining and Embedded Tracebacks"), which provided a means to chain exceptions using the __cause__ attribute on exception objects. That attribute would be set in a raise ... from statement; below is a highly contrived example:

    >>> try:
    ...     1/0
    ... except ZeroDivisionError as e:
    ...     try:
    ...         print(x)
    ...     except NameError:
    ...         raise NameError() from e
    ... 
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    ZeroDivisionError: division by zero

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<stdin>", line 7, in <module>
    NameError

Multi-exceptions

More recently, situations have arisen where multiple unrelated exceptions need to be raised and handled together. For example, libraries that provide concurrent operations (e.g. asyncio, Trio) may return aggregated results, thus there could be multiple exceptions of various sorts. Another problem area is with operations that are retried within the library, such as socket.create_connection(), where there could be multiple failures that would be useful for the caller to be able to process.

Currently, though, there is no way for Python programs to process these multi-failures; Python can only handle a single exception, though it can chain exceptions that occurred earlier, as mentioned. PEP 654 ("Exception Groups and except*") has come about to try to tame this problem. It describes some other situations that lead to multiple exceptions needing to be raised "simultaneously" and describes a potential solution. It was introduced in a message to the python-dev mailing list on February 23 by Irit Katriel. She is one of three authors of the PEP, along with Yury Selivanov and Guido van Rossum.

Trio has the trio.MultiError exception type to report a collection of errors, but it has proved difficult to use. That led Trio developer Nathaniel Smith to start working on MultiError2, but that effort "demonstrates how difficult it is to create an effective API for reporting and handling multiple errors without the language changes we are proposing", according to the PEP. The "rationale" section gives some more detail on why the authors have taken this route:

Grouping several exceptions together can be done without changes to the language, simply by creating a container exception type. Trio is an example of a library that has made use of this technique in its MultiError type. However, such an approach requires calling code to catch the container exception type, and then to inspect it to determine the types of errors that had occurred, extract the ones it wants to handle, and reraise the rest.

Changes to the language are required in order to extend support for exception groups in the style of existing exception handling mechanisms. At the very least we would like to be able to catch an exception group only if it contains an exception of a type that we choose to handle. Exceptions of other types in the same group need to be automatically reraised, otherwise it is too easy for user code to inadvertently swallow exceptions that it is not handling.

The proposed solution would add a new ExceptionGroup class (derived from BaseExceptionGroup), which can be used as follows:

    ExceptionGroup('problems', [ ValueError('illegal number'),
                                 NameError('bad name'),
                                 TypeError('no such type') ])

try:
    ...
except *SpamError:
    ...
except *FooError as e:
    ...
except *(BarError, BazError) as e:
    ...

In a traditional try-except statement there is only one exception to handle, so the body of at most one except clause executes; the first one that matches the exception. With the new syntax, an except* clause can match a subgroup of the exception group that was raised, while the remaining part is matched by following except* clauses. In other words, a single exception group can cause several except* clauses to execute, but each such clause executes at most once (for all matching exceptions from the group) and each exception is either handled by exactly one clause (the first one that matches its type) or is reraised at the end.

The exception group is processed recursively, looking for matching exception types at all of the levels of the hierarchy. Those that are found are extracted and processed before the next except* is consulted. As mentioned, any that slip through the cracks are reraised. It should be noted that a given try block can either have except or except* clauses; they cannot be mixed.

Backward compatibility

Adding a new type and syntax is meant to help ensure that there are no backward-compatibility concerns with the proposals. Exception groups can be treated as exceptions are today for handling and display. The except statement does not change either, so nothing that runs today will break with these changes. On the other hand, though, libraries that start raising exception groups are changing their API; the PEP gives its authors' expectations in that regard:

Our premise is that exception groups and except* will be used selectively, only when they are needed. We do not expect them to become the default mechanism for exception handling. The decision to raise exception groups from a library needs to be considered carefully and regarded as an API-breaking change. We expect that this will normally be done by introducing a new API rather than modifying an existing one.

In the python-dev thread, there was a good deal of discussion of how to handle the difference between an exception like ValueError and an exception group that contains a ValueError—possibly somewhere deep inside the hierarchy of exceptions and groups. Users may expect existing code to work in a congruent fashion, even if the exception has now migrated into a group. As Smith put it:

The intuition is that things will be simplest if ExceptionGroup is kept as transparent and meaningless as possible, i.e. ExceptionGroup(ValueError) and ValueError mean exactly the same thing -- "some code inside this block raised ValueError" -- and ideally should be processed in exactly the same way. (Of course we can't quite achieve that due to backcompat issues, but the closer we can get, the better, I think?)

But, as Katriel pointed out, the two are not the same, and effectively cannot be, because the behavior of except is not being changed. That led Marco Sula to ask why except could not simply be "upgraded" to handle exception groups. As Van Rossum pointed out, there is a problem because exceptions and exception groups are distinct kinds of objects:

Good question. Here's an example:

try:
    . . .
except OSError as err:
    if err.errno != ENOENT:
        raise
    . . .

If this would catch ExceptionGroup(OSError), the `err` variable would be an ExceptionGroup instance, which does not have an `errno` attribute.

The answer to Sula's query was duly added to the PEP. But there are others who are worried about adding non-obvious behaviors with regard to the difference between the two types; as Cameron Simpson put it:

I want the code to do what I said, not have some magic which silently/invisibly intercepts ExceptionGroups which contain something buried deep in their subgroup tree.

[...] I certainly do not want ExceptionGroup([AttributeError]) conflated with AttributeError. That fills me with horror.

Paul Moore and others were concerned that the "catch-all" clause except Exception would stop functioning in this new exception-group world, but Van Rossum explained that exception groups are still derived from Exception, so unless one is using new APIs that can raise an ExceptionGroup and have started using except*, nothing will change.

Many in the thread were trying to find ways to avoid needing the new except* syntax. The general idea would be to process the exceptions inside of the groups to try to match them with the existing except clauses in some defined fashion. That may run afoul of the worry about "magic" behavior; it may also be difficult to define the operation in a way that will satisfy all of the various constituencies—quite possibly why the authors avoided the problem by clearly separating the two.

In the end, it is a somewhat esoteric change, but one that solves some real problems that have come up. Conceptually, it makes a fair amount of sense, though it is, perhaps, somewhat annoying that there seems to be no way to smoothly fit it into the existing language (particularly for Python founder Van Rossum one might guess). Trio has already tried to make it fit, without being entirely successful. The final form of the feature may not have gelled yet, but something down this path is likely bound for the language before too long.