Python and "dead" batteries

Python is, famously, a "batteries included" language; it comes with a rich standard library right out of the box, which makes for a highly useful starting point for everyone. But that does have some downsides as well. The standard library modules are largely maintained by the CPython core developers, which adds to their duties; the modules themselves are subject to the CPython release schedule, which may be suboptimal. For those reasons and others, there have been thoughts about retiring some of the older modules; it is a topic that has come up several times over the last year or so.

It probably had been discussed even earlier, but a session at the 2018 Python Language Summit (PLS) is the starting point this time around. At that time, Christian Heimes listed a few modules that he thought should be considered for removal; he said he was working on a PEP to that end. PEP 594 ("Removing dead batteries from the standard library") surfaced in May with a much longer list of potentially dead batteries. There was also a session at this year's PLS, where Amber Brown advocated moving toward a much smaller standard library, arguing that including modules in the standard library stifles their growth. Some at PLS seemed to be receptive to Brown's ideas, at least to some extent, though Guido van Rossum was apparently not pleased with her presentation and "stormed from the room".

PEP 594

After PLS, Heimes posted the first draft of PEP 594 to the python-dev mailing list. It is a much more ambitious list than the one he made back in 2018; there are 31 modules listed, but four of those are actually modules that were once proposed for retirement that the PEP now recommends keeping. The modules are scattered throughout the standard library, but: "the majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI [the Python Package Index]."

The PEP lists alternatives from PyPI for most of the modules, along with a justification for their removal (or, in those few cases, their retention). In addition, the deprecation schedule being proposed is described; the modules agreed upon will be documented as "deprecated" in the upcoming 3.8 release and may raise a PendingDeprecationWarning exception. In 3.9, the modules will start raising DeprecationWarning exceptions and in 3.10 they will be removed along with their tests and documentation. Given the Python support window, the modules will still be supported by the core team until the end of life for Python 3.9, which is estimated to occur in 2026.

The modules listed to be removed in the first draft were:

Type	Modules
Data encoding	binhex, uu, and xdrlib
Multimedia	aifc, audioop, colorsys, chunk, imghdr, ossaudiodev, sndhdr, and sunau
Networking	asynchat, asyncore, cgi, cgitb, smtpd, and nntplib
OS interface	crypt, macpath, nis, and spwd
Miscellaneous	fileinput, formatter, imp, msilib, and pipes

As might be guessed, the PEP posting set off a bit of a storm of both suggestions for other modules to consider for removal, as well as concerns about some of the targeted modules. In particular, Andrew Svetlov suggested that socketserver might be a good candidate for removal, but it is used by http.server and others, so Heimes said he decided against it. But that led Glenn Linderman to suggest removing that module as well. He pointed out that it lacked a lot of functionality (e.g. HTTPS support) and that the PEP suggests removing the cgi module, further reducing its utility.

Van Rossum and others saw the value in http.server, though. It is used by other tools and modules, for one thing, but it is also useful as a quick and dirty local HTTP server—remembering how to configure and run a full web framework is too heavyweight for that kind of task. Linderman thought that the Bottle web framework might make a reasonable alternative for a simple, local server, but it turns out that Bottle uses http.server as well.

Several argued against removing specific modules. One of the more controversial choices that Heimes made was nntplib, which provides client-side code for accessing Network News Transfer Protocol (NNTP) services. Antoine Pitrou raised an objection to removing nntplib among others he thought were dubious choices (cgitb for generating tracebacks for web pages and crypt for interfacing to the crypt() one-way hash function). "NNTP is still quite used (often through GMane, but probably not only) so I'd question the removal of nntplib."

André Malo agreed with Pitrou about nntplib; he wondered how much of a maintenance burden it would be given how old the protocol is. But Victor Stinner pointed out that the "maintenance burden is real even if it's not visible"; for example, there are a number of sporadic test failures from nntplib in the continuous integration (CI) system and "nobody managed to come with a fix.. in 6 years". Beyond that, the administrator of the server used in some of those tests has asked about support for the NNTP compression extension, so there are still features that nntplib may need.

Giampaolo Rodolà thought that if nntplib was on the chopping block, telnetlib should probably join it, though he was not actually advocating that:

Heimes replied that he had missed telnetlib (it has since been added), but that nntplib does have a high maintenance burden because it has no maintainer, outstanding bugs, and missing features. Rodolà also argued against removing crypt and spwd (which provides access to the shadow password file). He noted that the reasons behind removing those two were security related, which makes handling their removal different than others on the list; since it may be useful to be able to work with passwords on Unix systems, having something available to do so out of the box would be good. But Heimes said that those two modules have some serious security problems that makes them "very dangerous batteries".

The nature of the PEP as an omnibus including many different modules means that objections should be handled differently, Van Rossum said. The PEP will eventually either need to be accepted or rejected as a whole, which means that any particular modules eliciting complaints should probably simply be dropped off the list:

Not dead

Several commenters objected to the name of the PEP, arguing that "dead" was not an accurate description of the state of many of the modules. Stinner said: "A module is never 'dead', there are always users, even if there are less than 5 of them." He was generally in favor of the overall plan, even though he had multiple concerns and questions. Steven D'Aprano was also unhappy with the name; the batteries are working, he said, they are just unloved. But he was worried about users who will not find it easy to add the batteries back into their Python after they get removed.

The current thinking seems to be that many of the modules that get removed will move over to PyPI in some form; it is possible they could even use the existing name because PyPI disallows module-name collisions with the standard library. Or, at least, the "useful" modules will move. How that will work, exactly, and how to make it easy for users affected by the module removal to fix it, are still up in the air, though it has been discussed in a thread on the Python Discourse forum. The only firm position that the PEP takes is that the core developers would stop maintaining the modules once they are removed (and the relevant Python versions reach their end of life). While moving modules to PyPI and providing some path for users to start picking them up from there is attractive, it has some possible downsides too; there are concerns that doing so could lead to another event-stream incident.

Meanwhile, Barry Warsaw was a bit worried that the PEP didn't go far enough toward solving the longstanding tension between various goals for the standard library:

Heimes has posted a second draft of the PEP on python-dev, then moved the discussion to a Discourse thread, perhaps in the interests of involving those who are not inclined toward the mailing list. For the most part, the responses were similar, mostly pleas to keep certain modules, though it is clear that some are not really aware of the maintenance burden that is borne by the core developers in keeping things going for the standard library. It is also clear from the whole discussion that there are multiple places where the core developers simply have not been able to keep up with the maintenance duties.

While moving some set of modules out of the standard library will certainly help alleviate the maintenance headaches for those modules, it will not magically grow new maintainers for them. It is possible that it might cause some interested parties to step up to fix, maintain, and, even, develop new features for some of the modules, especially if the tie to the CPython release schedule and development process has been holding some back from those chores.

In the end, it is a tricky balancing act to provide enough "batteries included" that Python is useful out of the box without overburdening the core maintainers—or constraining potentially better solutions. One of the complaints about the standard library is that it locks users into a particular approach that may be suboptimal. For example, by most accounts the Requests package provides a much more rational approach to using HTTP from Python, but users often opt for the standard library "equivalents" because they come with Python. Moving these standard libraries to PyPI might allow other modules to rise to the occasion.

On the flip side of that, of course, is that environments that are limited in their choices of third-party packages (e.g. due to policy or internet connectivity) will have a starting point with fewer features. For the most part, the modules under consideration here will not likely truly be a barrier for users that need them if they are no longer shipped with Python. It will be the case that some programs suddenly stop working unexpectedly but, with luck, a path toward minimizing even that will be found.