LWN: Comments on "Python and "dead" batteries"

Python and "dead" batteries

ceplm — Mon, 25 Jan 2021 10:32:16 +0000

There is hope that these modules would shift to independent PyPI packages, so you can get it from there.

Python and "dead" batteries

HelloWorld — Tue, 25 Jun 2019 23:17:09 +0000

> For example there is no way of take ownership of modules if the original author disappears.
That sounds like a feature to me, not a bug. How would you protect users from wrongdoers who seize unmaintained modules and then ship malware that way?

Python and "dead" batteries

njs — Thu, 20 Jun 2019 07:13:13 +0000

There's definitely been discussion of this option – it's basically how Python has been shipping 'pip' for a few years now, and one of the outcomes of Amber's talk at the language summit was the idea to start experimenting with doing the same for IDLE. There's also a discussion here that goes into more detail: https://discuss.python.org/t/1738

Python and "dead" batteries

NAR — Fri, 14 Jun 2019 10:53:26 +0000

"Unsuspecting developers could be writing new code today that won't be supported on 3.10 when it's released around 2021."

If those unsuspecting developers will maintain their code in 2021, then they will update their code. If they don't maintain it, their users are screwed anyway.

Python and "dead" batteries

rgmoore — Thu, 13 Jun 2019 19:32:07 +0000

That's not exactly what I had in mind, but I like it. The basic approach of bundling libraries with the distribution that aren't organizationally part of the core is a good compromise. It avoids the complaint that moving things out of the core makes them unavailable to people with strict install policies while still getting the benefit of developing them outside the core.

Python and "dead" batteries

logang — Thu, 13 Jun 2019 19:10:48 +0000

I don't think the 2026 number is valid and it still should be slower. At this rate, users who are affected are forced to update their code or not support versions past 3.9. Just because 3.9 is supported until 2026 doesn't really help the issue. Unsuspecting developers could be writing new code today that won't be supported on 3.10 when it's released around 2021. That's effectively only two years between looking perfectly safe to use and requiring it to be changed; and only one year before their users start complaining about warnings. The python team should mark the documentation as deprecated for a *lot* longer than that so users writing new code have a lot longer to notice the deprecation and stop using those features.

This is one of the things I think Linus gets right with Linux: his uncompromising stance toward breaking users and the general policy of very long deprecation cycles -- typically features aren't removed until there is a reasonable argument that nobody is using them. If more libraries and programming languages took that stance we'd be in a much better position. Too many projects just break things and foist the pain and responsibility on their user base.

Python and "dead" batteries

rgmoore — Thu, 13 Jun 2019 18:50:05 +0000

If they are going to remove modules they should use a multiple-cycle process that starts with deprecation in the documentation a couple cycles later start printing warnings and a couple cycles after that finally removing.

They are proposing a multi-cycle process. They're suggesting only one cycle each for "pending deprecation" and "deprecation" rather than several, but according to the article that would still leave them available until 2026, so they're not disappearing overnight by any means. And presumably they could be rescued from deprecation if a maintainer were willing to step forward and promise to support them moving forward.

Python and "dead" batteries

logang — Thu, 13 Jun 2019 17:12:38 +0000

I know this comment is a bit of a troll but I think it gets at the heart of an important issue. People choose to use standard libraries largely because they have a high chance of being maintained and available for the foreseeable future. People avoid PyPi modules because the maintenance and quality story is often questionable.

Python caused a lot of headaches for their users with the 3.0 transition and I don't think they can afford pushing more pain their users way. If they are going to remove modules they should use a multiple-cycle process that starts with deprecation in the documentation a couple cycles later start printing warnings and a couple cycles after that finally removing. The only code they should remove quickly is code that they can prove no one is using because it's obviously broken -- and I don't think many of the candidates that were proposed fit that bill. The argument that a module could be made available through PyPi so it can be quickly removed from the standard library is a bad one.

Some of the candidate modules don't seem like a large maintenance burden anyway compared to the pain that would be caused for potential users. For example: colorsys, fileinput, imghdr, pipes, and probably others seem like relatively small helper functions that shouldn't take a lot of effort to maintain but would cause anyone who chose to use them big pains if they were to suddenly disappear. Also, based on its documentation, imghdr shows improvements have been made in recent versions so, today, it seems well maintained and appropriate to use.

Python and "dead" batteries

cesarb — Thu, 13 Jun 2019 12:09:29 +0000

Was this before or after the npm event-stream incident? I expect taking over abandoned modules to have become harder after that.

Python and "dead" batteries

roc — Thu, 13 Jun 2019 11:19:24 +0000

Yeah, I'm not saying PyPI is great.

Python and "dead" batteries

dottedmag — Thu, 13 Jun 2019 09:37:16 +0000

I remember taking over an abandoned module by sending a e-mail to PyPI maintainers and answering some questions.

Python and "dead" batteries

LtWorf — Thu, 13 Jun 2019 08:47:51 +0000

I don't think Pypi is that modern. For example there is no way of take ownership of modules if the original author disappears. So on such occasions, forks are required, and of course developers won't know of the fork, and so on…

Python and "dead" batteries

smcv — Thu, 13 Jun 2019 08:23:20 +0000

Perl has had the same thing for a long time ("dual-life" modules that exist both bundled with Perl and on CPAN). It seems like a good approach.

Python and "dead" batteries

fyrchik — Thu, 13 Jun 2019 07:14:43 +0000

It is still possible to work with uuencode format using `binascii` module.

Python and "dead" batteries

NYKevin — Thu, 13 Jun 2019 02:04:43 +0000

I think it depends on your perspective. One possible perspective:

This code is useful.
Useful code should be maintained.
Therefore, somebody should maintain the code, regardless of whether it continues living in the stdlib.

Another possibility:

No core developer wants to maintain this code.
Unmaintained code should not live in the stdlib.
Therefore, we should remove the code from the stdlib, regardless of whether some other developers want to maintain it elsewhere.

These perspectives are not in conflict with one another. In fact, they are entirely orthogonal. You can agree with either or both of them without contradiction. However, I feel obligated to point out that the first perspective does not identify who, precisely, ought to maintain the code. Unless someone volunteers, that perspective is rather academic.

Equally obvious is that you can disagree with either or both of these perspectives, of course. For example, I have no evidence whatsoever that "no developer wants to maintain this code" is in fact true.

Python and "dead" batteries

ms-tg — Thu, 13 Jun 2019 01:45:32 +0000

> If anything, this seems like an argument in favor of replacing the current stdlib version with the better version from PyPI, not removing the feature from stdlib completely.

> It also seems to me that this might be a sign that there needs to be a different relationship between the core developers and PyPI. Instead of having a separate set of modules that are developed in stdlib, all of the stdlib modules could be developed in PyPI. Long term support versions could be branched to include in the stdlib, but users who wanted the newest features could replace it with the latest, greatest version if they chose. It would require changing the PyPI rules for those modules, but it would provide a useful compromise between development speed and stability.

Is this project similar to what you're thinking of?
https://stdgems.org/

Python and "dead" batteries

rgmoore — Thu, 13 Jun 2019 00:35:30 +0000

I disagree with the idea that just because there is something better as a 3rd party dep, the stdlib version should be killed off.

If anything, this seems like an argument in favor of replacing the current stdlib version with the better version from PyPI, not removing the feature from stdlib completely.

It also seems to me that this might be a sign that there needs to be a different relationship between the core developers and PyPI. Instead of having a separate set of modules that are developed in stdlib, all of the stdlib modules could be developed in PyPI. Long term support versions could be branched to include in the stdlib, but users who wanted the newest features could replace it with the latest, greatest version if they chose. It would require changing the PyPI rules for those modules, but it would provide a useful compromise between development speed and stability.

Python and "dead" batteries

JohnVonNeumann — Thu, 13 Jun 2019 00:04:41 +0000

> The modules are scattered throughout the standard library, but: "the majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI [the Python Package Index]."

See I get what's going on and that the community doesn't have enough maintainers, but IMO if I can do something with the stdlib I'll most likely do it with the stdlib, I try to avoid calling in 3rd party deps if I can. Naturally, this only works for smaller pieces of work but I disagree with the idea that just because there is something better as a 3rd party dep, the stdlib version should be killed off. I'm slightly simplifying their argument though.

Python and "dead" batteries

ms-tg — Wed, 12 Jun 2019 23:49:11 +0000

I’d like to remind the members of the Python community that hang out on LWN of the nearly identical problem domain encountered somewhat earlier by the Ruby community, and that the solution path chosen by the Ruby community is probably also ultimately going to be what’s most attractive for the Python community.

Out of a similar set of options, it was decided to strip many libraries over time out of the stdlib, and move them to Ruby gems distributed on rubygems.org (equivalent of PyPI).

As mentioned near the end of this article as a “if only we could” suggestion, this separation of project and code maintenance of each library is decoupled from the pace of their removal from Ruby releases, by including a set of “bundled” and “default” gems with each of the forthcoming Ruby releases. I imagine a Python solution will at some point adopt the same decoupled structure.

For more information on the Ruby process here, please see the official website for this effort:

https://stdgems.org/

Python and "dead" batteries

rgmoore — Wed, 12 Jun 2019 21:02:58 +0000

While moving some set of modules out of the standard library will certainly help alleviate the maintenance headaches for those modules, it will not magically grow new maintainers for them.

This seems like the key point to me. If the packages are unmaintained, that's the problem, not whether they're distributed with the standard library or through PyPI. The only way this improves things is if being moved out of the standard library makes it easier to attract developers. That seems like something that needs to be discussed.

Python and "dead" batteries

roc — Wed, 12 Jun 2019 20:30:39 +0000

More proof, if any were needed, that modern package management with some curation beats "batteries included".

Python and "dead" batteries

MortenSickel — Wed, 12 Jun 2019 20:18:12 +0000

Maybe for you. Which of the listed modules do you need? Are you standing up to maintain them?

Python and "dead" batteries

pgdx — Wed, 12 Jun 2019 18:18:25 +0000

You will also have the option of installing legacylib [1].

[1] https://github.com/tiran/legacylib

Python and "dead" batteries

NYKevin — Wed, 12 Jun 2019 17:18:37 +0000

After Python 3.x drops support for these modules, there is nothing stopping you from pulling their source code out of the Git history.

I mean, it obviously wouldn't have upstream support, but at the end of this year, neither will 2.7.

Python and "dead" batteries

mirabilos — Wed, 12 Jun 2019 17:13:03 +0000

Hehe… nice one.

“chunk” is a PITA… I tried to make https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=u... extend it first, but it’s only for reading, *and* it’s buggy, so I gave up and implemented the whole shit myself, using “chunk” only as a rough guideline.

“uu” is a mandatory piece of Unix history (and still in use) though… so it must not go. I cannot imagine this to be a maintenance burden, either… either it works or it’s trivial to fix and never worked. It’s not like uuencode format changes now.

Python and "dead" batteries

ikm — Wed, 12 Jun 2019 16:47:02 +0000

Yet another reason to stick to 2.7.