LWN: Comments on "Users and Python packaging"

Users and Python packaging

smammy — Tue, 07 Mar 2023 07:43:27 +0000

Aside from the comparatively “easy” problem of pip defaulting to system-level installation, my sense is that the most widespread issue is with packages that require external tools to build, usually because they need to compile some C code or want to link to a non-Python library.

I'm trying to think of whether there's a language package manager that does significantly better than Python on this and I'm drawing a blank. With most languages I can think of, it still comes down to, “oh yeah, you have to install such a such compiler and this and that library and these header files and make sure pkg-config knows about it,” and so on.

Does anyone know of a language package manager that does a particularly good job with this stuff?

Users and Python packaging

callegar — Mon, 20 Feb 2023 17:51:39 +0000

> Python (and C) cannot deal with diverging versions. There is no way for libjpeg62 to co-exist with libjpeg8 in the same program.

Python and C "cannot deal with diverging versions" in quite different ways.

Two versions of the same library cannot generally coexist in C *in the same program*, but they can certainly coexist *in the same environment*. This is the reason why you can `apt install` libjepeg62 and libjpeg8 at the system level and have programs in the same /usr/bin use either of the two.

Two versions of the same package cannot coexist in Python at all. They cannot coexist at the environment level. If you need two programs using two versions of the same package either you play tricks at packaging time, vendoring the packages inside the programs, or you patch the programs to use the same version of the package, or the typical distribution package manager will never be able to deal with this case, because the typical distribution level package manager does not know (and probably does not want to know) about having separate environments.

Users and Python packaging

mathstuf — Thu, 16 Feb 2023 17:25:09 +0000

> There is no way for libjpeg62 to co-exist with libjpeg8 in the same program

C libraries support mangling everything. You still can't use them in the same TU, but they can co-exist at runtime with a suitable dollop of `#define jpeg_func myjpeg_func` lines in a header, moving the headers in the install tree, and changing the SONAME on the library. I wouldn't call it fun, but it is at least possible.

I've yet to see a way to make it work reliably for Python at all because the lookup names are in the importing source without a preprocessor to get in there and mix all the names around.

Users and Python packaging

smurf — Thu, 16 Feb 2023 15:45:02 +0000

> So, if you need say libjpeg8 for something and libjpeg62 for something else,

Then you need to badger whoever still needs jpeg62 to freakin' update their code and get with the times. Or do it yourself and submit a patch. Still way cheaper (and, most likely, way more secure) than writing your own JPEG decoder.

> IMHO it is simply non realistic to think that packaging tools born with at the os level and that have a view of the system as a "single thing" can deal with it.

It should be. It's why distros exist, and IMHO they do a passable job for the most part.

Python (and C) cannot deal with diverging versions. There is no way for libjpeg62 to co-exist with libjpeg8 in the same program, and the same applies for versioned Python modules. As dependencies become more complex, the chance that you see colliding requirements approaches one, and the only way around that is to update your dependencies. You need to do that anyway, long-term.

Users and Python packaging

callegar — Thu, 16 Feb 2023 12:42:49 +0000

Conda has some very nice ideas and at the same time some aspects that appear simply too rough by today standards. No antialiasing on some application for instance is something that really delivers a bad impression. A bootstrapping problem, I understand: https://github.com/ContinuumIO/anaconda-issues/issues/6833

Users and Python packaging

callegar — Thu, 16 Feb 2023 09:04:45 +0000

IMHO Python is very specific wrt regular packaging that is done in a distro. Imagine having to build a distribution and packaging tools for it with the special rule that you cannot have two versions of the same item unless the package manager is capable of creating a self-contained mini-os for each variant. So, if you need say libjpeg8 for something and libjpeg62 for something else, the package manager should be able to create two separate environments where everything is duplicated but for the different libjpegs. Then you find out that you need libssl3 and libssl1.1 and the package manager would need to split out a new mini-system for libjpeg8+libssl3, one for libjpeg62+libssl3, one for libjpeg8+libssl1.1... and so on. Incidentally, this is the main reason why "pip installing at the system level" is always the wrong thing. There simply cannot be a single system python if not for a carefully, painfully, fragile hand crafted one made together to deal with the distro infrastructure needs and this is going to be so delicate that any attempt at touching it can immediately break it. On any system it is unavoidable to have a combinatorial explosion of "pythons" one for every possible useful/needed combination of package versions and IMHO it is simply non realistic to think that packaging tools born with at the os level and that have a view of the system as a "single thing" can deal with it.

Users and Python packaging

peter-b — Tue, 14 Feb 2023 14:17:56 +0000

Just a note to people following along: "H. Vetinari" is an obvious pseudonym, referencing the character of "Havelock Vetinari" from Terry Pratchett's "Discworld" novels. It's a fancy alternative to "J. Bloggs" or "A.N. Other".

Users and Python packaging

NYKevin — Sat, 11 Feb 2023 22:33:31 +0000

Sure, but the point is, app developers want their app to work. Distros want the whole platform to work. This is a source of friction.

Users and Python packaging

mbunkus — Sat, 11 Feb 2023 21:14:43 +0000

I actually do ship a C++ application that runs on all of those platforms. For certain ones I have to bundle a very small set of libraries that aren't available or are too old. My configuration system takes care of the detection & deciding which copies to use (system ones or the bundled ones). For others I have to install certain libraries from some well-known third-party applications.

The caveat is that by "ship" I actually mean "ship the source code that can be compiled on all of them". Wrt. binaries I also ship those, but those are distro+version-specific binaries compiled on each of them with an additional cross-platform binary in AppImage format (this one really runs on all of the mentioned platforms, at the cost of bundling all libraries safe for libc).

As for packaging Python for a wide range of software: Borg backup provides all-in-one bundles that run on a very wide range of distributions & versions. It is basically an archive containing a Python binary & all the libraries required by the Python interpreter & all the Python modules required by Borg itself. It's unpacked into a temporary directory on the fly on each invocation. To me this seems have all the drawbacks of an AppImage (having to bundle the world) and springkling having to do a bunch of I/O each time it's called, making it… super bad. Or something.

With today's distro us application developers only have a very limited number of ways to distribute stuff:

1. Offer cross-distro/cross-version compatible binaries which bundling the world
2. Offer one binary for each (distro, version) pair we want to support
3. Only distribute source code & let others do 1 or 2

That's the reality. Python isn't different. All languages face the same problems here, and the solutions all boil down to one of the points above for any non-trivial piece of software:

• node.js applications? install the world, this is basically bundling and therefore 1
• letting packages get packaged by distros themselves? this is basically option 3 chosen by the developer, and the the distro packages then use option 2
• commercial software coming als installable shell scripts? bundle the world again (e.g. Graylog coming as a Debian package that contains its own Java interpreter & all the assorted libraries; it's really funny) → option 1
• Python applications? either come along as software & let the user deal with (option 3) or tell you exactly how to run a venv (which is basically bundling the world) or offer you a bundled Python interpreter as Borg does (which really is bundling the world)
• all those programs telling you to just run a Docker container? they're obviously bundling the world _including the C library_

Users and Python packaging

poc — Sat, 11 Feb 2023 18:25:30 +0000

Exactly my point.

Users and Python packaging

pizza — Sat, 11 Feb 2023 16:06:32 +0000

> * Distros want stability. App developers want the new shiny thing.

I disagree with that. App developers want platform stability, except for one new shiny thing (or things) that is different for every app.

Evidence? Once an app is "working" they just keep shipping it with what it was developed against, unless forced to update to something newer by external pressure.

Users and Python packaging

pizza — Sat, 11 Feb 2023 16:02:42 +0000

You missed the point -- users care about *software* being installed -- not language-specific library packages.

You can't install a "python application" like you can a C, Java, Go, Rust, etc application. Unless you're using a distribution (or similar mechanism) that hides all of the (very) messy Python packaging mess from the user.

Users and Python packaging

cyperpunks — Sat, 11 Feb 2023 09:07:23 +0000

> My distro already has a package installer and it installs everything I need from package repositories. Why does Python have to be
> different from C, Java, C++, Go, etc? (rhetorical question)

The core of problem is that shipping Python code is both very easy and extremely complex.

For pure Python code which is coded correctly so it works on Python 2.7, 3.6, 3.7, 3.8, 3.9, 3.10 and 3.11: just download the script and run.

For Python code that uses a large set of C/C++ libs it can be compared to maintaining a large subset of several Linux distros, which is extremely complex.

Try to ship a modern C++ lib that works on rhel7, rhel8, rhel9, ubuntu 18.04->ubuntu 22.10, Fedora 37, opensuse 15.4, alpine linux, gentoo, debian 10 and 11, now add macOS (x86 and arm) and Windows.

I would guess its less than a handful of people in the industry who manages to do this.

Users and Python packaging

Wol — Sat, 11 Feb 2023 08:41:13 +0000

And you've completely ignored the user :-)

The reason I switched to gentoo, was I needed "recent" features in an application (lilypond) which was not available from the distro repository. (Okay, I was mad and also fancied tackling the learning curve - I didn't regret it :-)

Cheers,
Wol

Users and Python packaging

NYKevin — Sat, 11 Feb 2023 03:39:33 +0000

No, because then you'd have to talk about Rust and Go and all the other languages that are perfectly happy with their existing packaging systems, and their immediate reaction will be "Why would we change anything? It works so well!" IMHO it's intractable anyway, because application developers and distro developers have diametrically opposed preferences in most technical matters:

* Distros want stability. App developers want the new shiny thing.
* Distros want every app to use the same version of libfoo. App developers want to use whatever version of libfoo supports their use case.
* Distros want exactly one copy of libfoo on the system. App developers do not care about this at all, as long as it works.

No amount of "discussion" is going to resolve those conflicts, and it is pointless to try and make a packaging solution that satisfies all of them at once.

Users and Python packaging

himi — Sat, 11 Feb 2023 03:08:48 +0000

> That amount of space might be insignificant when you're doing some scientific stuff and your data is measured in petabytes, but to me this is not what a sensible packaging solution should ever do.

Trust me, it's far from trivial in an HPC context. The thing that people do in that context (when they're following the bouncing ball provided by the doco, and that doco points them at conda) is to do the install in their home directory, which is generally small, tightly constrained by quotas (including inode quotas), and low performance. All that random gunk that's taking up 4.5GB of space will likely hit those constraints /very/ quickly, particularly the inode quotas (which no one ever thinks about these days, since they're basically irrelevant on any modern filesystem . . . except high performance distributed filesystems like, for example, Lustre).

Just as importantly, even if they /can/ get it installed, the performance characteristics of the filesystem mean that it's going to behave in ways they're not anticipating, and the fact that it's pulled in /all/ of its own system libraries and dependencies means it's going to be unable to use the underlying hardware efficiently (particularly network/IB/etc, but also potentially GPUs and even the full capabilities of the native CPU).

These problems apply to pip as well, in different ways, and to just about anything else out there packaging python plus native code, but conda makes it particularly hard because it assumes that it's in control of /everything/ related to the python installation. It also likes to do things like adding itself to the user's shell profile, which can then break /other/ things the user wants to run (since other tools end up trying to load libraries from the conda installation rather than the system libraries or optimised versions of things provided by vendors or the local apps team).

HPC is outside the scope of most of these packaging discussions, sure, but it does provide some solid reasons why the approach that conda currently uses is probably not what we want to standardise on . . .

Users and Python packaging

mpr22 — Sat, 11 Feb 2023 00:21:51 +0000

> My distro already has a package installer and it installs everything I need from package repositories.

I'm very happy for you that you do not need any piece of software that your distro doesn't package.

Users and Python packaging

rahulsundaram — Fri, 10 Feb 2023 22:57:19 +0000

> Why does Python have to be different from C, Java, C++, Go, etc? (rhetorical question).

I will answer it anyway. Python is not different. Other than C and C++ language specific package managers (Conan exist but is more of a niche player) are popular even for Linux users because they often want the latest libraries etc that distribution aren't packaging or they want a cross platform workflow and other operating systems don't have it packaged. Go has go install built-in. Maven is very popular for Java etc.

Users and Python packaging

poc — Fri, 10 Feb 2023 22:49:42 +0000

I don't want to install Python packages. I want to install software.

My distro already has a package installer and it installs everything I need from package repositories. Why does Python have to be different from C, Java, C++, Go, etc? (rhetorical question). Unless this is solved, the rest of the discussion is academic as far as I'm concerned.

Users and Python packaging

opsec — Fri, 10 Feb 2023 19:33:19 +0000

> It is my personal opinion that they are massively overthinking this.

Funny, I was thinking they are massivily simplifing by limiting it to python 8-)

Coming from the FreeBSD ports tree, I'm wondering why the packaging topic is being seen as a python specific issue. All distros need to package many projects written in one or more languages and need to cope with several dependency tracking systems (at least one for each language).

So shouldn't this whole discussion be much broader ? It makes it more complex, but we'll save a few iterations by that ?

Users and Python packaging

smurf — Thu, 09 Feb 2023 20:10:16 +0000

NB, in my case it was CadQuery. After miniconda was done my disk was 4.5 GBytes smaller, separate Python3.10 installation included.

That amount of space might be insignificant when you're doing some scientific stuff and your data is measured in petabytes, but to me this is not what a sensible packaging solution should ever do.

Users and Python packaging

smurf — Thu, 09 Feb 2023 17:25:55 +0000

Did you ever try to install any nontrivial software whose installation instructions read "run 'conda XXX'" into a non-conda environment? When the people involved all use conda and thus the non-conda tooling is nonexistent and/or suffers from bit rot?

Good luck.

Users and Python packaging

smitty_one_each — Thu, 09 Feb 2023 16:20:04 +0000

> I have yet to find any language that has better tooling than Python as they are all virtually the same.

I had the impression that ruby gems were a rounder wheel.

Users and Python packaging

southey — Thu, 09 Feb 2023 15:40:57 +0000

All I can say is great that it works for you. I have yet to see that it helps me in any way and it continues to be the worst solution ever every time I have to deal with it. Just last week a student using conda with R couldn't install a package because the R version in the conda environment was old. Took ages to realize that because the Linux distro had installed the latest version. Sure that is rare, but it should have never happened.

One of the reasons for using a distro is that versions are updated with respect to the various libraries provided. That does not happen with conda so you are left with maintaining each Python package across each conda environment used. I just don't know how people can use it while keeping packages up to date especially across multiple computers. This is particularly an issue with new features and other change that come with new versions of Python and packages. Code that works in one environment fails in another resulting in lost time troubleshooting and solving it. The worst aspect is when it was the developer that used it as that has larger concerns (was a major issue with packages that needed to support multiple versions of Python 2 and 3).

I have yet to find any language that has better tooling than Python as they are all virtually the same. Newer languages may appear to be better (whatever that means to you) as the developers have usually learnt from experiences with other languages. However, that quickly disappears once you get a reasonable developer and user base.

(Yes, I am a sample size of one and that is 100% of the user cases.)

Users and Python packaging

auc — Thu, 09 Feb 2023 14:50:54 +0000

> Conda compatibility is a hard problem to solve

What do you mean by this ? Compatibility with what ?

Users and Python packaging

auc — Thu, 09 Feb 2023 14:48:40 +0000

The Elephant in the room, yes ... most Python core devs don't appear to see it. Looks too much like the Postgres avatar ? :)

Users and Python packaging

NYKevin — Thu, 09 Feb 2023 07:51:16 +0000

It is my personal opinion that they are massively overthinking this.

Users don't *care* about majority cases vs. edge cases. In fact, users don't want to think about "use cases" at all, if they can help it. Users want a tool that Just Works with a minimum of fuss.[1] If such a tool would have existed 20 years ago, then everything would be fine by now, but you can't get there from here. The *lack* of such a tool has resulted in the development of an elaborate ecosystem[2] of workarounds and partial solutions. Projects and workflows have grown up around them. An entire generation of Python programmers have gotten used to the idea that Python Packaging Is Hard, and they've learned to make do with what was available. This has resulted in users developing opinions about How Packaging Ought To Work, which is frankly a bad thing. In just about every language that does packaging right, the answer to "How does packaging work?" is, as far as the average user is concerned, "It works very well, thank you for asking." Python is nowhere near that ideal.

The only way I can imagine the situation improving is if PyPA does a complete about-face on their current approach:

0. Stop telling everyone that you're unofficial. Stop saying that packaging.python.org is "just" documenting well-known solutions. If you have to persuade the SC to give you more authority, or if you have to start writing PEPs for 80% of everything you do, that's just the price of admission.
1. Pick a tool. For the sake of argument, I'll assume that tool is Pip, but you could just as easily pick pipenv, or Poetry, or some other thing, but pick *something.*
2. Take the Unicode approach: Pip (or whatever tool you selected) must make a reasonable attempt to support all use cases. If there is a tool that does X, and X is related to Python packaging, then you accept any pull requests that would add X to Pip, unless they're broken or otherwise problematic. This includes copying code from other projects, if the licenses are compatible. (Do not pick a tool unless its maintainer is willing to commit to this.)
2½. To avoid making the UX any worse than it already is, a limited amount of bikeshedding will likely need to happen on those pull requests, but such bikeshedding should not last for longer than a week or two at most. Develop a voting system or steal Debian's, if needs be, just don't get bogged down in nonsense.
3. Once Pip (or whatever) supports all use cases of tool Y, then tool Y stops being recommended on packaging.python.org. Maybe move it to a different page or something if you don't want to hurt the maintainer's feelings.
4. If Pip is selected: Make a serious effort to improve Pip's default behavior so that it better comports with user expectations. Maybe this means that Pip needs to change names to avoid breaking backcompat, or maybe that means you need to introduce a new tool. Point is, the One Tool To Rule Them All must not install into system site-packages by default. Years of experience has shown that basically nobody wants that, except perhaps on Windows, and maybe not even there.
5. If you identify a use case that is really hard for Pip (or whatever) to support, then you write a PEP laying out what users are supposed to do instead of using Pip, and explaining why it is out of scope for Pip. Such a PEP should be well-justified, and should not describe a workflow that is significantly more painful than whatever users are doing today. It should preferably not require the use of other tools, except to the extent that those tools are compatible with Pip. If possible, try to work with the maintainer of the other tool to enable this compatibility, rather than declaring the whole thing out of scope.[3]
6. Workflows are not use cases. Implementation details are not use cases. If a user tells you that (for example) they want their packages installed in a very specialized way so that their non-Python software can reach inside the venv and fiddle with it, or (for example) they want to do some convoluted task "in one command" (they won't accept two or three commands), that is out of scope and they just have to live with it or hack on top of it.[4]

This would be a long, slow, painful, and toilsome road for the PyPA to go down. But that's the hole that Python has dug for itself. The first step is to put down the shovel.

[1]: https://xkcd.com/353/
[2]: https://xkcd.com/1987/
[3]: "Other tool" is a phrase which here means "conda." Conda compatibility is a hard problem to solve, and I'm not expecting this to happen any time soon. It would be nice if (as a starting point) Pip and venv were capable of hosting C libraries and other non-Python software, and conda were capable of installing such things into venvs in addition to its native environments. This would require significant work on both sides, and probably a fair amount of discussion of the specific mechanisms involved.
[4]: https://xkcd.com/1172/

Users and Python packaging

jhoblitt — Thu, 09 Feb 2023 00:52:11 +0000

After switching to conda... It is hard to imagine ever going back. It is amazing how poor the tooling is in the python world compared to other languages with much smaller user based.