Python discusses deprecations

Feature deprecations are often controversial, but many projects find it necessary, or desirable, to lose some of the baggage that has accreted over time. A mid-November request to get rid of three Python standard library modules provides a case in point. It was initially greeted as a good idea since the modules had been officially deprecated starting with Python 3.6; there are better ways to accomplish their tasks now. But, of course, removing a module breaks any project that uses it, at least without the project making some, perhaps even trivial, changes. The cost of that is not insignificant, and the value in doing so is not always clear, which led to higher-level conversation about deprecations.

Module removal

Victor Stinner posted a notice to the python-dev mailing list saying that he wanted to remove asyncore, asynchat, and smtpd from the standard library for Python 3.11 (due in October 2022). His justification noted that there are better alternatives (asyncio for the first two and aiosmtpd, available in PyPI, for the last). As can be seen in the links, all of the modules suggested for removal are documented to be deprecated and have been since Python 3.6 was released in 2016. But Stinner also noted that the DeprecationWarning for those modules was only emitted at run time in Python 3.10, which drew an objection from Petr Viktorin:

According to the policy, the warning should be there for *at least* two releases. (That's a minimum, for removing entire modules it might make sense to give people even more time.)

Viktorin is referring to PEP 387 ("Backwards Compatibility Policy"), which says that features cannot be removed "without notice between any two consecutive releases". Stinner interpreted that to mean the documentation notice (for 4 releases) was sufficient, but Viktorin and others did not agree. As can be seen in the bug tracking the removal, which goes back to 2016, Stinner added a note about the removal plan and, after getting several approvals from other core developers, merged that change on November 15.

But Brett Cannon thought that the removal was not in keeping with the PEP, thus premature. He noted that the steering council (which he is a member of) can grant exceptions, so Stinner opened an issue with the council to resolve the question. On December 6, the council issued its ruling that the removals should not be done for Python 3.11, so Stinner duly reverted the change.

Wider context

During that discussion, Viktorin started another thread about the more general question of deprecations for the language. It was motivated, in part, by rebuilds the Fedora project was doing using the alpha versions of Python 3.11. He pointed to a few examples of problems that arise from names that have been removed from standard library modules under the deprecation policy. He wondered:

Are these changes necessary? Does it really cost us that much in maintainer effort to keep a well-tested backwards compatibility alias name, or a function that has a better alternative?

I think that rather than helping our users, changes like these are making Python projects painful to maintain. If we remove them to make Python easier for us to develop, is it now actually that much easier to [maintain]?

He also said that he was complaining that the PEP guideline was meant as a minimum of two releases, but it is often treated as "wait two releases and remove". He was not talking about "code that's buggy, insecure, or genuinely hard to maintain", but that routinely removing names of various sorts makes it harder for Python-based projects to keep up. He asked if there might be a need for an alternative formulation:

If deprecation now means "we've come up with a new way to do things, and you have two years to switch", can we have something else that means "there's now a better way to do things; the old way is a bit worse but continues to work as before"?

Stinner pointed out that all of the examples he cited had been deprecated (and emitted a warning) for far longer than two cycles (the most recent was deprecated in Python 3.5). Viktorin acknowledged that but: "as far as I know, they haven't really caused problems in all that time"

Stinner described some of the problems that arise from leaving deprecated functions in the code base. They cause users and developers to do some extra thinking when they are encountered: "Why is it still there? What is its purpose? Is there a better alternative?". The answers to those questions are sometimes only found in the bug tracker or Git repository. Beyond that, some deprecations are being done to prod developers into using better alternatives; as Stinner mentioned for the removals he wanted to do, deprecated code is effectively unmaintained:

Open issues in asyncore, asynchat and smtpd have been closed as "wont fix" because these modules are deprecated. These modules are basically no longer maintained.

But Viktorin said that he is not arguing against the removal "if something's an attractive-looking trap", but there are other types of changes that basically boil down to the core Python developers enforcing their opinions on names and functionality:

Python users want to write commits that either bring value, or that are fun. Mass-replacing "failUnless" with "assertTrue" just because someone decided it's a better name is neither. Same with a forced move to the latest version of a function, if you don't use the bells and whistles it added.

Stinner had said that those who are using deprecated functions are building up technical debt: "An application using multiple deprecated functions will break with a future Python version." But Viktorin said in some cases that is the fault of the core developers:

But "will break with a future Python version" just means that people's code breaks because *we break it*. If we stopped doing that (in the simple cases of name aliases or functions that are older but not dangerous), then their code wouldn't break.

Christopher Barker said that keeping old names around "forever" was not a good solution, but that name changes like for failUnless should not be made if it is only done because someone liked the new name better. Removing the older versions of the names does have beneficial effects, Serhiy Storchaka said, since typos that land on the older valid names can persist in a code base without being noticed:

It is so easy to make a typo and write assertEquals instead of assertEqual or assertRaisesRegexp instead of assertRaisesRegex. Tests are passed, and warnings are ignored. Then I run tests with -Werror to test new warnings and get a lot of unrelated failures because of PRs merged at last half-year. I am very glad that these long time ago deprecated aliases are finally removed.

Increasing the window

Some kind of tool that helped maintainers make these simple substitution changes would be a nice addition, Stephen J. Turnbull said. Barker agreed but noted that a big problem is trying to support applications across multiple versions of the language, some of which have the new feature and some of which have to use the older name.

But do we need to support running the same code on 3.5 to 3.10? I don't think so. If you can't upgrade Python to a supported version, you probably shouldn't upgrade your code or libraries.

Which is a thought — maybe the policy should be that we remove things when the new way is usable in all supported versions of Python. So as of today (if I'm correct) anything needed in 3.5 can be dropped.

That idea, in a somewhat different guise, was suggested by Eric V. Smith, who was also in favor of tools to assist maintainers:

[...] I think a useful stance is "we won't remove anything that would make it hard to support a single code base across all supported python versions". We'd need to define "hard", maybe "no hasattr calls" would be part of it.

Reliable tools to make the migration between versions would help, too.

Cannon restated Smith and Barker's ideas in a more concrete form:

I think Eric was suggesting more along the lines of PEP 387 saying that deprecations should last as long as there is a supported version of Python that lacks the deprecation. So for something that's deprecated in 3.10, we wouldn't remove it until 3.10 is the oldest Python version we support. That would be October 2025 when Python 3.9 reaches EOL and Python 3.13 comes out as at that point you could safely rely on the non-deprecated solution across all supported Python versions (or if you want a full year of overlap, October 2026 and Python 3.14).

I think the key point with that approach is if you wanted to maximize your support across supported versions, this would mean there wouldn't be transition code except when the SC [steering council] approves of a shorter deprecation. So a project would simply rely on the deprecated approach until they started work towards Python 3.13, at which point they drop support for the deprecated approach and cleanly switch over to the new approach as all versions of Python at that point will support the new approach as well.

That is a direction Viktorin would like to see, but if the PEP were to change along those lines (as Cannon suggested might be a path forward), he would rather see some recognition that there is a cost to these kinds of changes, but also to retain some flexibility:

I'm not looking for a contract, rather a best practice. I think we should see Python's benign warts as nice gestures to the users: signs that we're letting them focus on issues that matter to them, rather than forcing them to join a quest for perfection. If a wart turns out to be a tumor, we should be able to remove it after the 2 years of warnings (or less with an exception). That's fine as a contract. But I don't like "spring cleaning" -- removing everything the contract allows us to remove.

Ensuring more perfect code should be a job for linters, not the interpreter/stdlib.

Terry Reedy noted that the change to yearly releases also shortened the minimum deprecation period from three years to two, so the idea of waiting until a replacement is available in all supported Python versions makes sense. Looking to the future:

Python is nearly 30 years old. I am really glad it is not burdened with 30 years of old names. I expect someone reading this may write some version of Python 50 years from now. I would not want [them] to have to read about names deprecated 60 years before such a time.

Jeremiah Paige mentioned a tool that might be used to help maintainers going forward: pyupgrade. It is a tool to upgrade code to newer versions of Python, but does not cover some of the cases that Viktorin mentioned. That could presumably be fixed with code contributions to the project. Stinner suggested a collaboration between the Python core developers and the pyupgrade developers to ensure that deprecations are added to the tool as things are being removed from the language.

The deprecation problem is something that crops up for Python and other projects with some frequency. There is a desire to remove old cruft (or "dead batteries") from the language, but there is obviously a balance to be struck. Other projects, the Linux kernel being a famous example, generally never deprecate anything unless it can be shown that it is no longer in use—or actively harmful. While there was some agreement with Viktorin in the thread, there were others who are intent on removing things as soon as reasonably possible. Whether that turns out to be longer than the minimum of two releases remains to be seen.