Python packaging targets [LWN.net]

Python packaging targets

Posted Feb 23, 2023 14:30 UTC (Thu) by hrw (subscriber, #44826)
Parent article: Python packaging targets

Binary compatibility is kind of sorted out with manylinux environments. But they are based on old releases of distributions so it can be hard to collect all build dependencies.

Forcing to install from distribution is wrong. What if current distro release has 1-2 years old package only and we need features from newer one? Change distro or distro release?

to post comments

Python packaging targets

Posted Feb 24, 2023 6:08 UTC (Fri) by cyperpunks (subscriber, #39406) [Link] (15 responses)

> What if current distro release has 1-2 years old package only and we need features from newer one?

It's very simple, but it seems to a be a secret for most users:

You create a bug against the package in distro and ask for upgrade.

If the upstream maintainer has done development corrretlt and not broken backward compatibility and all packages that depend on the package by explicit version check has used equal and greater than (>=) and not equal, it takes distro maintainer 1-2 hours work to import new version, build it and send update to repos. After 3-7 days in testing (you can enable testing repo to the package the next day) new version will be availalble for all users in the distro.

If upstream maintainer has broken backward compatibility the distro maintainer must check all packages that depend on the problematic package and ping maintainers of all borken packages due to the borken backward compatibility. The distro maintainer of the now borken package must now check if a fix exist or resolve it by a local fix or stop the upgrade of your package.

If upgrade A of forces upgrade of package B, the whole process must be repeated for package B so on to all deps have been resolved.

As seen the amount of work needed is direct product of how upstream maintainers of packages does development. If they don't break backward compatibility and use sane version requirement expressions an package upgrade is available for all users in a few days.

Python packaging targets

Posted Feb 24, 2023 7:45 UTC (Fri) by hrw (subscriber, #44826) [Link] (5 responses)

Not every distribution is rolling release one.

I have systems with Debian stable. Some stuff can be updated thanks to backports repo but there are Debian developers who do not do backports.

Python packaging targets

Posted Feb 24, 2023 11:32 UTC (Fri) by cyperpunks (subscriber, #39406) [Link] (3 responses)

This is not rolling, it's simple update of a package. All sane distros should allow that?

Python packaging targets

Posted Feb 24, 2023 12:19 UTC (Fri) by TomH (subscriber, #56149) [Link] (2 responses)

Not really - stable distros usually try to limit updates within a release to bug fixes. See for example the Fedora policy:

https://docs.fedoraproject.org/en-US/fesco/Updates_Policy...

Python packaging targets

Posted Feb 24, 2023 20:33 UTC (Fri) by cyperpunks (subscriber, #39406) [Link] (1 responses)

Well, since Fedora 37 GA there have been about 500 updates per month:

https://bodhi.fedoraproject.org/releases/F37

Python packaging targets

Posted Feb 24, 2023 22:09 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> Well, since Fedora 37 GA there have been about 500 updates per month:

Fedora has ~70000 packages. So 500 is still a pretty small number of updates in comparison. Also keep in mind that Fedora's updates tend to far more liberal (semi rolling) in practice compared to say RHEL, Debian or Ubuntu. It is quite common for folks to find that their distribution doesn't ship with the version of something they want if they are looking for closer to the latest.

Python packaging targets

Posted Mar 2, 2023 7:09 UTC (Thu) by oldtomas (guest, #72579) [Link]

If it's Debian we are talking about, and if the upgrade itself is source-compatible (which was assumed upthread) then creating a backport isn't much more difficult than pip-installing something: just download the source .deb, unpack it, install the build depencencies (which you must, perhaps, tweak to reflect what you have on your stable), perhaps tweak some dependencies, then dpkg-buildpackage.

Compared with some boxes I meet regularly at work, which are "lots of little .venv directories, all alike", this might be a nicer option, after all.

Python packaging targets

Posted Feb 24, 2023 23:42 UTC (Fri) by rra (subscriber, #99804) [Link] (8 responses)

I am afraid this is excessively optimistic, even apart from the general principle that the point of stable distributions is to not do this.

Common problems that prevent the distribution from just updating to a new version:

1. The new version broke backward compatibility in some way (possibly intentionally, possibly by accident), and now other packages that depend on it don't work and also need to be updated (or may not have updates available yet).
2. The new version drops support for older versions of Python that are supported in that release of the distribution.
3. The new version has new dependencies on Python packages that aren't already packaged.
4. The new version has new dependencies on C libraries or Rust crates or what have you that aren't already packaged.

There are some extremely well-behaved Python libraries that never break backward compatibility and support a wide range of versions, but this is quite a lot of work and a lot of maintainers don't do this, either due to lack of time or because this isn't how they want to spend their resources and effort.

One of the reasons why this discussion is so difficult is that system integration is a substantial amount of ongoing work. Things break. A lot. And quite a lot of this work is volunteer.

Python packaging targets

Posted Feb 25, 2023 7:45 UTC (Sat) by cyperpunks (subscriber, #39406) [Link] (1 responses)

All this is true.

The quality of software available in pypi varies a lot and some packages break compability in a random minor upgrade.

To "resolve" this owners of other projects using the project with borken compability pin version requirement to a specific version.
This is also careless, as now is any project that needs a new of version of problematic package blocked from being installed.

The reason is that Python has global name space, there can be only version of problematic package installed and this package must
be present at runtime. Python is special here. Rust or Golang can create a suitable build environment, create binaries, install and software will run forever.

Python has the worst mix of all: an easy to learn language which causes lots of low quality package to be published to repos, careless owners which break compability, even more careless owners which pin version requirements combined with globale namespace and
no different between build and runtime environment.

The future of this is not promising.

Python packaging targets

Posted Feb 25, 2023 21:59 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

> The reason is that Python has global name space, there can be only version of problematic package installed and this package must be present at runtime. Python is special here. Rust or Golang can create a suitable build environment, create binaries, install and software will run forever.

It very much is *possible* to make hermetic builds of Python. It's just that there's so much you need to package, that you are effectively bundling an entire Python installation (like Flatpak etc.). Distros don't like this degree of vendoring, but they also have problems with Go and Rust.

Python packaging targets

Posted Mar 2, 2023 8:27 UTC (Thu) by callegar (guest, #16148) [Link] (5 responses)

This would be easier if you could

apt install some-python-pkg-1.0

/and/ also

apt install some-python-pkg-2.0

without having the latter replace the former. It would allow the distro to provide the new package, while temporarily maintaining stuff that might get broken with the old package.

In fact, given the python approach of not allowing this kind of thing, I now tend to think that a few things would be simpler if there was a policy so that at least the "major" version of the package was formally required to become a part of the package name. Even without a formal requirements, for some packages this seems to be already done.

Python packaging targets

Posted Mar 2, 2023 17:58 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (4 responses)

How do you propose these two package co-exist on the disk? If both are installed, what does `import some_python_pkg` do?

Python packaging targets

Posted Mar 2, 2023 21:19 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

Sorts and picks the highest version number (using standard version sorting), unless a suitable piece of metadata in the installed package directory (or import line, since obviously we'd want to augment that and allow version ranges!) specifies otherwise.

This is not rocket science. It's just a tricky compatibility and syntax thing.

Python packaging targets

Posted Mar 3, 2023 0:43 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> This is not rocket science. It's just a tricky compatibility and syntax thing.

Isn't this kind of the use case that this whole article series has been about though? It might not be rocket science, but it certainly doesn't seem easy. We (as a species) actually seem to be better at rocket science if you were to ask me…

Corner cases that come to mind:

- Two libraries provide metadata to describe their dependencies and they disagree about what version to use.
-- What is preferred?
-- Does every library/application need to specify their full transitive requirements?
- How do I prod random script developers to provide this information? Can it be specified in-source or is it sidecar metadata?
- If they are allowed to co-exist, what does `sys.modules` look like now?

Python packaging targets

Posted Mar 3, 2023 13:30 UTC (Fri) by nix (subscriber, #2304) [Link]

> - Two libraries provide metadata to describe their dependencies and they disagree about what version to use.

I don't see how (in the absence of hermetic namespacing) this can be anything but a runtime error. Of course, again, other languages can do this easily...

> - How do I prod random script developers to provide this information? Can it be specified in-source or is it sidecar metadata?

I was assuming both: a file in the package directory versioning lots of things at once, or extra syntax for import. Sidecar seems totally wrong: this is something *the package* is expecting, not something the admin is supposed to be nailing down (how could the admin possibly know the right answer?)

Obviously if a given import doesn't have a version nailed in, the system works exactly as it does now: anything else would be a compatibility disaster. (Needless to say I have written none of this code so I'm totally bikeshedding and waving away all the no-doubt-horrible problems!)

Python packaging targets

Posted Mar 2, 2023 21:19 UTC (Thu) by atnot (guest, #124910) [Link]

Like in many other language ecosystems (and even sort of python with virtualenv), there would be some shim that tells the interpreter/compiler what packages the program should have available in what version. This would be read from some sort of file provided along with the program or source code.