Users and Python packaging

Posted Feb 16, 2023 9:04 UTC (Thu) by callegar (guest, #16148)
In reply to: Users and Python packaging by opsec
Parent article: Users and Python packaging

IMHO Python is very specific wrt regular packaging that is done in a distro. Imagine having to build a distribution and packaging tools for it with the special rule that you cannot have two versions of the same item unless the package manager is capable of creating a self-contained mini-os for each variant. So, if you need say libjpeg8 for something and libjpeg62 for something else, the package manager should be able to create two separate environments where everything is duplicated but for the different libjpegs. Then you find out that you need libssl3 and libssl1.1 and the package manager would need to split out a new mini-system for libjpeg8+libssl3, one for libjpeg62+libssl3, one for libjpeg8+libssl1.1... and so on. Incidentally, this is the main reason why "pip installing at the system level" is always the wrong thing. There simply cannot be a single system python if not for a carefully, painfully, fragile hand crafted one made together to deal with the distro infrastructure needs and this is going to be so delicate that any attempt at touching it can immediately break it. On any system it is unavoidable to have a combinatorial explosion of "pythons" one for every possible useful/needed combination of package versions and IMHO it is simply non realistic to think that packaging tools born with at the os level and that have a view of the system as a "single thing" can deal with it.

Users and Python packaging

Posted Feb 16, 2023 15:45 UTC (Thu) by smurf (subscriber, #17840) [Link] (2 responses)

> So, if you need say libjpeg8 for something and libjpeg62 for something else,

Then you need to badger whoever still needs jpeg62 to freakin' update their code and get with the times. Or do it yourself and submit a patch. Still way cheaper (and, most likely, way more secure) than writing your own JPEG decoder.

> IMHO it is simply non realistic to think that packaging tools born with at the os level and that have a view of the system as a "single thing" can deal with it.

It should be. It's why distros exist, and IMHO they do a passable job for the most part.

Python (and C) cannot deal with diverging versions. There is no way for libjpeg62 to co-exist with libjpeg8 in the same program, and the same applies for versioned Python modules. As dependencies become more complex, the chance that you see colliding requirements approaches one, and the only way around that is to update your dependencies. You need to do that anyway, long-term.

Users and Python packaging

Posted Feb 16, 2023 17:25 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> There is no way for libjpeg62 to co-exist with libjpeg8 in the same program

C libraries support mangling everything. You still can't use them in the same TU, but they can co-exist at runtime with a suitable dollop of `#define jpeg_func myjpeg_func` lines in a header, moving the headers in the install tree, and changing the SONAME on the library. I wouldn't call it fun, but it is at least possible.

I've yet to see a way to make it work reliably for Python at all because the lookup names are in the importing source without a preprocessor to get in there and mix all the names around.

Users and Python packaging

Posted Feb 20, 2023 17:51 UTC (Mon) by callegar (guest, #16148) [Link]

> Python (and C) cannot deal with diverging versions. There is no way for libjpeg62 to co-exist with libjpeg8 in the same program.

Python and C "cannot deal with diverging versions" in quite different ways.

Two versions of the same library cannot generally coexist in C *in the same program*, but they can certainly coexist *in the same environment*. This is the reason why you can `apt install` libjepeg62 and libjpeg8 at the system level and have programs in the same /usr/bin use either of the two.

Two versions of the same package cannot coexist in Python at all. They cannot coexist at the environment level. If you need two programs using two versions of the same package either you play tricks at packaging time, vendoring the packages inside the programs, or you patch the programs to use the same version of the package, or the typical distribution package manager will never be able to deal with this case, because the typical distribution level package manager does not know (and probably does not want to know) about having separate environments.