Python finally offloads some batteries [LWN.net]

Python finally offloads some batteries

Posted Mar 17, 2022 7:48 UTC (Thu) by NYKevin (subscriber, #129325)
In reply to: Python finally offloads some batteries by garyvdm
Parent article: Python finally offloads some batteries

I would divide the stdlib into four non-equal groups:

1. Modules that belong in the stdlib, because they are basically language extensions or language services. itertools, dataclasses, enum, importlib, ast, sys, etc. all fall into this category. Compare and contrast the java.lang.* package. In particular, sys is never going anywhere because it literally cannot live anywhere other than the stdlib (at least in the CPython implementation).
2. Modules that belong in the stdlib, because the services they provide are so basic and so slow-moving that there are no real advantages to splitting them out. pathlib, collections, heapq, math, etc.
3. asyncio, which is a whole separate ball of wax all by itself. Probably this needs to stay as part of the stdlib because it's tightly coupled to the async/await syntax. In retrospect, I'm somewhat suspicious of that design decision, but it's far too late to change now regardless of whether it was a good idea at the time.
4. Everything else, which could probably go into PyPI and maybe have some sort of "automatically acquire the latest stable version at installation/upgrade time" logic (which all of the distros would promptly yank out and replace with package manager dependencies). Depending on who you ask, this group *may or may not* contain any or all of the following: old protocols and file formats, current protocols and file formats, higher-level services such as logging, GUI/TUI-related stuff like tkinter and curses, and maybe a few other misc. things that need or want to move faster than the stdlib can reasonably accommodate.

The main sticking point, IMHO, is the boundary between (2) and (4), as well as whether or not (4) should be installed-by-default or require separate installation. The advantage of installed-by-default over keep-in-stdlib is, of course, that the libraries can continue to be actively developed and maintained independently of Python's release schedule. The main disadvantage is that, if nobody actually steps up to maintain the libraries, they may get quite old and outdated...

to post comments

Python finally offloads some batteries

Posted Mar 17, 2022 8:41 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (1 responses)

That would make python kinda useless for several thousands of users.

There's a million scripts running on rpi made by people that have no idea how to track dependencies and figure out how to manage them and lock and install and all that crap.

At some point you have to decide if you want to close shop and just let another better maintained language take over the spot.

Python finally offloads some batteries

Posted Mar 17, 2022 17:23 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

Why? I just described a solution in which all of the same modules would still be installed by default, meaning your individual scripts wouldn't need to track anything.

Python finally offloads some batteries

Posted Mar 17, 2022 18:29 UTC (Thu) by logang (subscriber, #127618) [Link] (4 responses)

I strongly disagree.

This would be like the Linux Kernel developers deciding they don't have time to maintain large swaths of drivers and just dropping them. Then expecting people who need them to get the oot drivers from github using dkms and shift the weight on distributions to package all these drivers and deal with the resulting dependency hell. It would be a _giant_ mess. What would really happen is the drivers would be maintained even worse then they are now and become even more broken, and people who need them would be out of luck.

I think the emphasis should be on growing the project and the number of developers, not splintering off poorly maintained code into situations of even worse maintenance. Dropping modules that are obsolete and which nobody really uses is fine (allowing for the option of a maintainer who cares to step up); but dropping useful modules that people depend upon is not.

IMO, the best solution to the urllib issue would be to absorb the requests module into the standard library and bring all their developers with it. A development model similar to the kernel where a subsystem maintainer collects and sends patches upstream to the core python maintainers. There may be issues with this, but none that couldn't be worked out in the long run. Yes, the requests developers may need to do more work to ensure backwards compatibility, but they'd get help dealing with underlying core changes that affect their module. The python core team may have to make process changes to allow for a higher volume of security changes to stable releases. But after the pain, the benefit is a quality url module in the standard library and urllib could be re-implemented as a thin wrapper over it, and/or dropped after a very long deprecation period.

I've written many simple scripts that run on urllib because I have no interest in dealing with the additional dependency. If requests was in the standard library I'd certainly use it universally, but it is not and in many circumstances that disqualifies it. I suspect this is a common practice. The core team have a responsibility not to break such widespread usage with no real alternative for developers.

Python finally offloads some batteries

Posted Mar 17, 2022 21:17 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

> but dropping useful modules that people depend upon is not.

If nobody is willing to maintain the code, it is de facto unmaintained. You can slap a "maintained" label on it all you like, but that does not cause maintenance work to get done.

> but they'd get help dealing with underlying core changes that affect their module.

I believe we've all learned the hard way that core changes should be backwards-compatible, and so this kind of help should not be required in the vast majority of reasonable cases.

> But after the pain, the benefit is a quality url module in the standard library and urllib could be re-implemented as a thin wrapper over it, and/or dropped after a very long deprecation period.

They certainly can't do that until requests stops depending on urllib.

Python finally offloads some batteries

Posted Mar 18, 2022 8:25 UTC (Fri) by milesrout (subscriber, #126894) [Link] (2 responses)

It's a bit different with Python, because the language itself is - pretty much - stable. They don't go out of their way to break core language interfaces in every version.

Linux developers, on the other hand, have no qualms about changing core interfaces in any old version. They don't exactly *go out of their way to*, especially where it would complicate backporting fixes to older versions. But look at the discussions happening around list iterators. They clearly are willing to change fundamental interfaces quite readily.

This means that out-of-tree modules for the kernel are in a very different level of support (none at all) than third-party modules for Python. The whole *point* of Python is a stable interface against which to write third-party modules! That's what a language *is*!

Python finally offloads some batteries

Posted Mar 18, 2022 15:46 UTC (Fri) by logang (subscriber, #127618) [Link] (1 responses)

That seems like wishful thinking at best. It is not nearly as stable as you think it is. Most python packages only support of subset of python reasons for lots of good reasons. If large swaths of the python library are now in PyPi they also now gain complicated dependencies between them as well. Maybe the kernel's driver API experiences more churn, but the point remains.

As NYKevin pointed out the requests module depends on urllib (yikes) so if that module gets removed from the standard library then you've broken requests for the latest release of python.

If tons of important modules are ejected then the core teams haS to stop removing or deprecating things to avoid the same dependancy hell the kernel would have with out of tree drivers.

Python finally offloads some batteries

Posted Mar 18, 2022 17:40 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

> Most python packages only support of subset of python reasons for lots of good reasons.

In my experience, this "subset" is usually of the form "version 3.x or later" for some value of x (or, for a handful of very old libraries, "version 2.7.x only"). I don't believe I have seen a whole lot of libraries that set a maximum version, other than the ones which were never ported to 3.