Python finally offloads some batteries [LWN.net]

Python finally offloads some batteries

Posted Mar 17, 2022 1:10 UTC (Thu) by pabs (subscriber, #43278)
Parent article: Python finally offloads some batteries

It seems like the Python standard library is where modules go to die.
It might be better to just ditch the entirety of the standard library.
There could be different groups (like the Python Packaging Authority) defining sets of blessed modules that are modern and well maintained using best practices.

to post comments

Python finally offloads some batteries

Posted Mar 17, 2022 1:35 UTC (Thu) by jafd (subscriber, #129642) [Link] (4 responses)

Please no.

It’s bad enough in Perl where modules go away from core and back between releases.

Python finally offloads some batteries

Posted Mar 17, 2022 1:37 UTC (Thu) by pabs (subscriber, #43278) [Link] (3 responses)

Perl definitely shouldn't be doing that; my approach would be to leave them out permanently.

Python finally offloads some batteries

Posted Mar 17, 2022 12:28 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

I don't know. Doing that has the advantage that moving things in and out of core is uncontroversial and done routinely, and that things that leave core are *still easily accessible* (to the majority of users who can install stuff as needed), because they're still on CPAN. This in turn is easy because the majority of the standard library is maintained using the same build system as CPAN, so moving things from core to CPAN or dual-lifing them in both is literally a matter of running a couple of commands: almost no source changes are usually needed. In Python, my impression is that migrating a module is quite a big job, so there's a temptation to just drop it and leave it nowhere and its few remaining users out in the cold.

Python finally offloads some batteries

Posted Mar 17, 2022 13:20 UTC (Thu) by Wol (subscriber, #4433) [Link]

Yup. The Python people need to create some fjords for old batteries to pine for.

Cheers,
Wol

Python finally offloads some batteries

Posted Mar 17, 2022 14:25 UTC (Thu) by anselm (subscriber, #2796) [Link]

In Python, my impression is that migrating a module is quite a big job, so there's a temptation to just drop it and leave it nowhere and its few remaining users out in the cold.

The code is there, and making Python packages for PyPI isn't exactly rocket science. The seven people who actually still use ossaudiodev or nntplib can do their own legwork, or else the modules must not have been all that essential after all.

Python finally offloads some batteries

Posted Mar 17, 2022 5:34 UTC (Thu) by garyvdm (subscriber, #82325) [Link] (8 responses)

Respectfully I disagree.

Yes, you have your medium to big projects that need a setup.py/requirments.txt/pyproject.toml , and for those projects, doing this would not matter.

But there are just as many small projects. Think single script, where installation = drop the script into /usr/local/bin. This simplicity is valuable, and If you remove the entire, or a large portion of the stdlib, you take away the ability for this to be done.

Python finally offloads some batteries

Posted Mar 17, 2022 7:48 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (7 responses)

I would divide the stdlib into four non-equal groups:

1. Modules that belong in the stdlib, because they are basically language extensions or language services. itertools, dataclasses, enum, importlib, ast, sys, etc. all fall into this category. Compare and contrast the java.lang.* package. In particular, sys is never going anywhere because it literally cannot live anywhere other than the stdlib (at least in the CPython implementation).
2. Modules that belong in the stdlib, because the services they provide are so basic and so slow-moving that there are no real advantages to splitting them out. pathlib, collections, heapq, math, etc.
3. asyncio, which is a whole separate ball of wax all by itself. Probably this needs to stay as part of the stdlib because it's tightly coupled to the async/await syntax. In retrospect, I'm somewhat suspicious of that design decision, but it's far too late to change now regardless of whether it was a good idea at the time.
4. Everything else, which could probably go into PyPI and maybe have some sort of "automatically acquire the latest stable version at installation/upgrade time" logic (which all of the distros would promptly yank out and replace with package manager dependencies). Depending on who you ask, this group *may or may not* contain any or all of the following: old protocols and file formats, current protocols and file formats, higher-level services such as logging, GUI/TUI-related stuff like tkinter and curses, and maybe a few other misc. things that need or want to move faster than the stdlib can reasonably accommodate.

The main sticking point, IMHO, is the boundary between (2) and (4), as well as whether or not (4) should be installed-by-default or require separate installation. The advantage of installed-by-default over keep-in-stdlib is, of course, that the libraries can continue to be actively developed and maintained independently of Python's release schedule. The main disadvantage is that, if nobody actually steps up to maintain the libraries, they may get quite old and outdated...

Python finally offloads some batteries

Posted Mar 17, 2022 8:41 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (1 responses)

That would make python kinda useless for several thousands of users.

There's a million scripts running on rpi made by people that have no idea how to track dependencies and figure out how to manage them and lock and install and all that crap.

At some point you have to decide if you want to close shop and just let another better maintained language take over the spot.

Python finally offloads some batteries

Posted Mar 17, 2022 17:23 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

Why? I just described a solution in which all of the same modules would still be installed by default, meaning your individual scripts wouldn't need to track anything.

Python finally offloads some batteries

Posted Mar 17, 2022 18:29 UTC (Thu) by logang (subscriber, #127618) [Link] (4 responses)

I strongly disagree.

This would be like the Linux Kernel developers deciding they don't have time to maintain large swaths of drivers and just dropping them. Then expecting people who need them to get the oot drivers from github using dkms and shift the weight on distributions to package all these drivers and deal with the resulting dependency hell. It would be a _giant_ mess. What would really happen is the drivers would be maintained even worse then they are now and become even more broken, and people who need them would be out of luck.

I think the emphasis should be on growing the project and the number of developers, not splintering off poorly maintained code into situations of even worse maintenance. Dropping modules that are obsolete and which nobody really uses is fine (allowing for the option of a maintainer who cares to step up); but dropping useful modules that people depend upon is not.

IMO, the best solution to the urllib issue would be to absorb the requests module into the standard library and bring all their developers with it. A development model similar to the kernel where a subsystem maintainer collects and sends patches upstream to the core python maintainers. There may be issues with this, but none that couldn't be worked out in the long run. Yes, the requests developers may need to do more work to ensure backwards compatibility, but they'd get help dealing with underlying core changes that affect their module. The python core team may have to make process changes to allow for a higher volume of security changes to stable releases. But after the pain, the benefit is a quality url module in the standard library and urllib could be re-implemented as a thin wrapper over it, and/or dropped after a very long deprecation period.

I've written many simple scripts that run on urllib because I have no interest in dealing with the additional dependency. If requests was in the standard library I'd certainly use it universally, but it is not and in many circumstances that disqualifies it. I suspect this is a common practice. The core team have a responsibility not to break such widespread usage with no real alternative for developers.

Python finally offloads some batteries

Posted Mar 17, 2022 21:17 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

> but dropping useful modules that people depend upon is not.

If nobody is willing to maintain the code, it is de facto unmaintained. You can slap a "maintained" label on it all you like, but that does not cause maintenance work to get done.

> but they'd get help dealing with underlying core changes that affect their module.

I believe we've all learned the hard way that core changes should be backwards-compatible, and so this kind of help should not be required in the vast majority of reasonable cases.

> But after the pain, the benefit is a quality url module in the standard library and urllib could be re-implemented as a thin wrapper over it, and/or dropped after a very long deprecation period.

They certainly can't do that until requests stops depending on urllib.

Python finally offloads some batteries

Posted Mar 18, 2022 8:25 UTC (Fri) by milesrout (subscriber, #126894) [Link] (2 responses)

It's a bit different with Python, because the language itself is - pretty much - stable. They don't go out of their way to break core language interfaces in every version.

Linux developers, on the other hand, have no qualms about changing core interfaces in any old version. They don't exactly *go out of their way to*, especially where it would complicate backporting fixes to older versions. But look at the discussions happening around list iterators. They clearly are willing to change fundamental interfaces quite readily.

This means that out-of-tree modules for the kernel are in a very different level of support (none at all) than third-party modules for Python. The whole *point* of Python is a stable interface against which to write third-party modules! That's what a language *is*!

Python finally offloads some batteries

Posted Mar 18, 2022 15:46 UTC (Fri) by logang (subscriber, #127618) [Link] (1 responses)

That seems like wishful thinking at best. It is not nearly as stable as you think it is. Most python packages only support of subset of python reasons for lots of good reasons. If large swaths of the python library are now in PyPi they also now gain complicated dependencies between them as well. Maybe the kernel's driver API experiences more churn, but the point remains.

As NYKevin pointed out the requests module depends on urllib (yikes) so if that module gets removed from the standard library then you've broken requests for the latest release of python.

If tons of important modules are ejected then the core teams haS to stop removing or deprecating things to avoid the same dependancy hell the kernel would have with out of tree drivers.

Python finally offloads some batteries

Posted Mar 18, 2022 17:40 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

> Most python packages only support of subset of python reasons for lots of good reasons.

In my experience, this "subset" is usually of the form "version 3.x or later" for some value of x (or, for a handful of very old libraries, "version 2.7.x only"). I don't believe I have seen a whole lot of libraries that set a maximum version, other than the ones which were never ported to 3.

Python finally offloads some batteries

Posted Mar 17, 2022 9:47 UTC (Thu) by ddevault (subscriber, #99589) [Link] (2 responses)

Some batteries is a good thing. The alternative is npm, which is a disaster.

Python finally offloads some batteries

Posted Mar 18, 2022 7:48 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (1 responses)

Well js lacks even kinda basic functions, but I agree with you.

The other problem with npm is that somehow they've decided (clearly without ever taking the time to do any measurement) that a library of 1 function is faster than putting a bunch of related functions all into the same library.

I think the original idea was to save time on doing js files downloads… but apparently the person saying this didn't know about all the headers and roundtrips that need to happen before a file can be downloaded.

Python finally offloads some batteries

Posted Mar 18, 2022 17:37 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

> I think the original idea was to save time on doing js files downloads… but apparently the person saying this didn't know about all the headers and roundtrips that need to happen before a file can be downloaded.

I believe this may have been a contributing factor to the invention and adoption of HTTP/2, because now the server can just say upfront "here are all of the libraries you are going to need, when you eventually get around to running the JS."