Improved error reporting for CPython 3.10—and beyond

In a fast-paced talk at PyCon 2022 in Salt Lake City, Utah, Pablo Galindo Salgado described some changes he and others have made to the error reporting for CPython 3.10. He painted a picture of a rather baffling set of syntax errors reported by earlier interpreter versions and how they have improved. This work is not done by any means, he said, and encouraged attendees to get involved in making error reporting even better in future Python versions.

Galindo Salgado prefaced his talk with something of a warning that he has been told that he speaks rather quickly; with a chuckle, he suggested attendees prepare themselves for the ride. He introduced himself as a CPython core developer and a member of the steering council; beyond that, he is also the release manager for versions 3.10 and 3.11 of the language.

He began with a story of his days as a PhD student in physics, where he was using Python as a tool for his research. One day a friend showed him a Python syntax-error message that they could not figure out. They showed it to another student and all three of them were stumped; three physics students who were studying to try to solve the mysteries of the universe were unable to find a simple syntax error. The error message looked something like:

      File "ex.py", line 13
	def integrate(method, x, y, sol):
	^
    SyntaxError: invalid syntax

As can be seen, there is nothing obviously wrong with the def statement, but looking at the (presumably simplified) code makes it clear where the actual problem lies:

    configuration = {
	'integrator' : 'rk4',
	'substep' : 0.0001,
	'butcher_table' : {
	    1 : 1/6,
	    2 : 1/3,
	    3 : 1/3,
	    4 : 1/6,
    }

    def integrate(method, x, y, sol):
	...

A closing brace was left out in the definition of configuration, but the CPython error message is pointing to the following statement, which is not particularly helpful.

He showed several more examples of where the parser misleads programmers with its messages. Leaving out a comma in a list definition or omitting a closing square bracket can lead to an error on the following statement, which may be far from where the problem actually arose. These "not good" messages can confuse veteran Python coders, but they are really problematic for those trying to learn the language.

Galindo Salgado put up a slide with the "worst one of all", which is the dreaded "SyntaxError: unexpected EOF while parsing". It "helpfully" refers to a line number one past the end of the file and has a caret ("^") pointing to nothing at all. He asked for hands of people who have seen that error and most of the room raised theirs. Beyond the lack of helpfulness of the error message, "how many times do you have to explain to someone what 'EOF' means?" That is not particularly friendly to new programmers.

New parser

The poor error messages are not there "because we are lazy", he said. It was difficult to get the information needed for better messages in the parser—until Python 3.9. A new parser for CPython was introduced in that version of the language. That parser "allowed us to start thinking about how we can improve these things; can we improve the experience of people writing Python who make syntax errors?"

It is not just important for those learning the language, but also for those who use it regularly; developers at all levels have difficulty understanding many of the syntax-error messages generated by the interpreter. He has friends who are working with the language, so he has been "extremely happy that I have fixed many of these error messages" for them and others, Galindo Salgado said.

The new parser is based on a parsing expression grammar (PEG), instead of the old one based on an LL(1) parser. He was part of the group, with Guido van Rossum and Lysandros Nikolaou, who wrote PEP 617 ("New PEG parser for CPython") and implemented the parser. He noted that the commits for the original parser and the PEG parser were made almost exactly 30 years apart, in 1990 and 2020.

There were some shortcomings of the old parser, which was part of why it was replaced, but the new parser also allows new features that the old one could not support. For example, it allows multiple context managers in a with statement without having to resort to backslashes. It also allows the new match statement syntax. "This is only possible with the new parser", he said.

New error messages

The PEG parser also allows for a bunch of improved error messages. Many of the examples he gave in the talk came from a section in the "What's new in Python 3.10". There are "quite a lot of them", Galindo Salgado said, so he would only be talking about a subset. A common mistake for new Python programmers now has a much friendlier message:

    # Python < 3.10
    >>> if x > y
      File "<stdin>", line 1
	if x > y
	       ^
    SyntaxError: invalid syntax

    # Python 3.10+
    >>> if x > y
      File "<stdin>", line 1
	if x > y
		^
    SyntaxError: expected ':'

Forgetting the colon at the end of if, for, and other similar constructs happens frequently, so getting a clearer message that specifies what is missing and where it should go will help. Users who forget to specify a value in a dictionary used to just get a generic syntax error message pointing to the closing brace, but things have improved:

	values={ 'a' : 1, 'b' : }
			      ^
    SyntaxError: expression expected after dictionary key and ':'

Beyond that, using "=" in an if statement instead of "==" will get a suggestion about what the mistake is rather than the generic "invalid syntax" complaint:

	if x = y:
	   ^^^^^
    SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

Similarly, a forgotten comma in a dictionary definition, which might not be obvious in a complicated initialization expression, will get "Perhaps you forgot a comma?" with the caret pointing to where it likely should go. "This has saved me at least ten times already", he said.

The IndentationError messages have been improved as well. Now the message tells the programmer the line number of the if (or other statement) that is causing the need for indentation. In addition, the dreaded "EOF" message, which can happen when a dictionary or other similar construct is missing its closing punctuation, now gives an actually useful error message:

	vals = { 'a' : 3, 'b' : 4
	       ^
    SyntaxError: '{' was never closed

"This is probably one of the ones that people like the most", he said.

Difficulties

"It turns out that adding error messages is quite hard", Galindo Salgado said. For example, when looking to add the missing comma test, the first step was to teach the parser to recognize it. The first attempt at a rule might be that when it sees an expression followed by another expression, without anything in between, it is a missing comma. But that test is way too simplistic. It will trigger on a missing "in" for a for loop or, even, the new match statement; "no good, right?" He showed a bunch of bug reports that resulted from the change, all of which have been fixed at this point.

A raw PEG parser does its work in exponential time, which means that it takes an amount of time proportional to an exponent of the length of the input. In order to avoid that, the CPython parser uses "packrat parsing", which uses a cache to do the parsing in linear time.

But sometimes it can still get into some ugly exponential backtracking. "This is pretty funny", Galindo Salgado said. An obvious syntax error of 21 open braces, followed by a colon (and EOF), took Python 3.10 around two seconds to parse, while making it 42 open braces, ran for more than an hour. He fixed the problem back in February for Python 3.10.3. "Now they [users] don't need to spare one hour to find out that's a syntax error", he said with a chuckle.

Adding error messages is difficult because it is stretching the parser in directions where it is less tested. "We have validated the real grammar of the language" many times, Galindo Salgado said, "we know it's fast, we know it works". A parser likes to know about what is correct in the language, but, when testing these error conditions, it is being used to investigate the "infinitely big world of things that are not Python".

It takes a lot more effort to validate the parser once these incorrect constructs are being recognized as well. More recently, a lot of improvements have been made to the parser and the tools used to validate it, but sometimes things still slip through. Then people make fun of them on Twitter; "Please don't make fun of us in Twitter", he said to laughter.

Suggestions and tracebacks

More syntax error message improvements are planned for 3.11 and beyond. There are, of course, other kinds of errors in Python programs, including errors at run time. Another new feature in 3.10 adds suggestions when a user misspells an identifier in their program. For example:

    >>> import collections
    >>> collections.namedtuplo
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: module 'collections' has no attribute 'namedtuplo'. Did you mean: 'namedtuple'?

    >>> schwarzschild_black_hole = None
    >>> schwarschild_black_hole
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'schwarschild_black_hole' is not defined. Did you mean: 'schwarzschild_black_hole'?

This facility "works with everything": modules, custom classes, things in the standard library, and so on. This is a highly useful feature, he said, which actually helped him as he was developing it; "it's quite cool".

Galindo Salgado went on to explain how the feature is implemented. First, they extended the AttributeError exception to add two pieces of information: the name being looked up and the target object where Python tried (and failed) to find it. Then a "word distance" function is used to try to find the closest match to the name. All of the possibilities in the object are checked and the one with the smallest word distance from the name is suggested.

But there is a problem: those kinds of exceptions (and others, like NameError, where suggestions are made) can happen in the normal functioning of a program and finding the closest match is an expensive operation. If it were computed every time the exception is raised, "it would make Python much slower". So, instead, the match is only computed when the exception is about to be printed; the exception has bubbled all the way up to the top level and the interpreter is about to exit anyway. This is part of what makes adding error messages hard, he said; it is important to ensure that the non-error paths are not penalized when adding extra information to help in the error case.

Something he is excited about that is coming is better tracebacks for Python. The feature comes from PEP 657, which "has a horrible name 'Include Fine Grained Error Locations in Tracebacks'", he said, but it is much better than it sounds. He authored the PEP with Ammar Askar and Batuhan Taskaya; the feature will be added in 3.11. He gave an example similar to the following in the PEP:

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
	x['a']['b']['c']['d'] = 1
    TypeError: 'NoneType' object is not subscriptable

One of those things is None, "but which one it is, you don't know". In Python 3.11, though, the traceback will show exactly where the problem lies:

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
	x['a']['b']['c']['d'] = 1
	~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

In addition, tracebacks from failing programs will show which function call failed, so multiple calls on the same line are no longer mystifying as to which caused the error:

    Traceback (most recent call last):
    ...
      File "query.py", line 24, in add_counts
	return 25 + query_user(user1) + query_user(user2)
	            ^^^^^^^^^^^^^^^^^
      File "query.py", line 32, in query_user
	return 1 + query_count(x['a']['b']['c']['user'])
	                       ~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

Likewise, multiple divisions on a line will indicate which caused the division by zero, multiple uses of the same attribute name on different objects will indicate the guilty one, and so on.

This is done by adding extra information to every bytecode instruction about where the operation is in the program. Each operation stores the starting and ending line numbers along with the starting and ending column positions. The offending line is reparsed and combined with the information from the failing bytecode to produce the far more useful traceback.

Helping out

"We would love for you to help us" in making the error messages for Python even better, Galindo Salgado said. Python bugs have recently migrated to GitHub, so he recommended people go there to suggest error-message improvements. Sometimes the developers will say that it is difficult or impossible to implement them, but other times, they have been able to improve an error message based on an issue of that nature. He encouraged new Python programmers, in particular, to point out error messages that have caused them problems; similarly, Python teachers should point out the errors that are causing their students the most trouble.

For those who want to work on implementing better error messages, he suggested starting with the "Guide to CPython's Parser" in the Python Developers Guide. It is "very technical but I think it reads quite nicely", he said; it will allow readers to understand how the parser works in great detail. There is a section at the end on adding and validating new error messages. That will allow developers to add an error message and a "bunch of test cases", which can hopefully then go into the CPython mainline.

There is a growing group of people working on these improvements, "which is super great". Many of the improvements were proposed and implemented by the community and not by the core developers. It is important to keep an open mind when proposing improvements, however, since some of them may not be possible or may lead to problems elsewhere. Sometimes, even if they fix the target message, the core developers have to turn them down because of other things that break.

The "moral of the story" is that if you are working on your PhD and lose your battle with syntax errors, you can study for a few years about parsers and grammars, then help replace "the parser for one of the most popular languages in the world". After that, you can become a core developer and help improve the situation with those syntax errors. Or you can wait for someone else to have that experience "and then you can use it", he concluded—to laughter and applause.

[I would like to thank LWN subscribers for supporting my trip to Salt Lake City for PyCon.]

Index entries for this article
Conference	PyCon/2022
Python	CPython/Error messages

---

to post comments

$ gcc -c test.c
test.c: In function ‘printhello’:
test.c:3:12: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
    3 | int main() {
      |            ^
test.c:6: error: expected ‘{’ at end of input

$ clang -c test.c
In file included from test.c:1:
./printhello.h:1:18: error: expected ';' after top level declarator
void printhello() 
                 ^
                 ;
1 error generated.