Escape sequences in Python strings

A change for Python 3.8—currently in beta—has produced some user-visible warnings, but the problem is often in code that a user cannot (or should not) change: third-party modules. The problem that the warning is trying to highlight is real, however. The upshot is that the handling of escape sequences (or non escape sequences, in truth) in Python string literals is in a rather messy state at this point.

Python string literals come in many different forms, but the main ones look like the following:

    a = 'foo'
    b = "bar"
    c = '''
    baz
    '''

But what to do about string literals with invalid escape sequences in them? A programmer who has put '\latex' as part of a string literal (to pick a not entirely random example) presumably actually wants '\\latex', which is what Python currently translates it to. Python does emit a DeprecationWarning in that case, but the warning was invisible by default until Python 3.7. However, that same programmer probably does not want '\tan(x)' to turn into a tab plus 'an(x)', but that is exactly what happens.

The change for Python 3.8 is to further elevate the warning to a SyntaxWarning, with plans to turn that into a SyntaxError in Python 3.9. A bug report filed in February 2018 shows the path of the change. But shortly after the Python 3.8 beta releases were made, Raymond Hettinger reported that he was seeing the warnings "pop up from time to time" from various third-party packages. Aaron Meurer concurred with Hettinger and pointed out a number of other problems he had encountered.

Serhiy Storchaka, who authored the patch for 3.8, replied that one of the examples given in the reports was in the PyParsing module, but that it showed a real problem that needed addressing, thus proving the usefulness of the change. A docstring in PyParsing referred to a regular expression, but did not escape the backslashes in the expression. The line in the docstring (delimited by ''') was:

    make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>")

In mid-July, Hettinger reported a problem he encountered with the Bottle web framework, where, once again, the warning was showing up, but in code his students could not change. He said: "I think it is poor form to bombard end-users with warnings about things they can't fix." Storchaka pointed out that the problem in Bottle resulted from a bad backport of a change made five years ago.

But even though the warning became visible in Python 3.7, it was still difficult for projects to see. The problem is that the warning is emitted when the .py file is compiled into a .pyc file and never shown after that point. For many projects, as Meurer pointed out, that means it is quite difficult to actually test for the presence of the warning, even when the projects are diligently looking for it:

That led Nathaniel Smith to argue that the deprecation cycle was not really followed for these strings. He suggested perhaps storing these kinds of warnings in the .pyc file so that they could be emitted later, and not just at compilation time. Around the same time, Hettinger elevated the visibility of the problem with a post to the python-dev mailing list:

He also noted that ASCII art in docstrings was affected and that Python had been living with its current behavior for nearly 30 years without any huge downside. The main problem that warning for illegal escape sequences is trying to solve is for file names on Windows systems, where the backslash is used as a directory separator, he said. Many of those problems cannot even be caught by the warning; his example was '..\training\new_memo.doc', which would produce a corrupted file name, but no warning. One solution for those types of strings is to use Python's raw strings: r'..\training\new_memo.doc'.

But Chris Angelico was concerned that allowing illegal escape sequences to pass silently is problematic. He did a quick survey of some other languages and found that most gave a warning (or, in the case of Lua, an error), but that all of the others just ignored the spurious \ entirely, rather than turning it into \\ as Python does. "IMO Python's solution is better, but Lua's is best. A bad escape is an error."

In general, the consensus in the thread seems to be to slow down the process of turning the warning into something that users see, at least until the developers of modules in the Python Package Index (PyPI) can address the problem. There were lots of suggestions for ways to find these strings automatically in ways that will be seen by the developers, but those will need to wait for the Python 3.9 cycle.

The behavior with escape sequences has other wrinkles, as Steven D'Aprano pointed out. For example, escaped non-delimiting quotes are treated differently than other escape sequences:

    py> "abc \d \' xyz"
    "abc \\d ' xyz"

So D'Aprano believes a change needs to be made, but that the current pace is too fast: "This isn't a critical problem that needs to be fixed soonest." But Angelico is worried about simply kicking the can down the road. He would like to see some active steps be taken so that developers can see and fix these problems. The more-visible warning added for 3.8 serves that purpose but, as D'Aprano noted, there will likely be some undesirable effects of that, beyond just user unhappiness. Users will file bugs, for example, often without looking to see if there is an existing bug, and they may or may not file the bug in the right place. Furthermore: "The benefit of the desired change is relatively low."

While D'Aprano is generally in favor of giving warnings (and, presumably, eventually errors) for these illegal escape sequences, he was not convinced that it has been adequately justified. He mentioned that it would allow new escape sequences, but did not find that to be of huge benefit. Angelico was adamant, however, that the problem is larger than just blocking new escape sequences:

Storchaka has submitted a pull request to revert his change that elevated the DeprecationWarning to a SyntaxWarning. Based on the mood in the thread, one would guess that will be merged, but, if the problem is truly going to be addressed, additional work will be needed. No one has suggested dropping the DeprecationWarning, though some seem to think there is marginal utility to changing the long-established "escape illegal escape sequences" feature. But it is clear that doing so without adequate warning causes problems of a different—and more user-visible—sort.