Python packaging targets [LWN.net]

Python packaging targets

Posted Feb 24, 2023 23:42 UTC (Fri) by rra (subscriber, #99804)
In reply to: Python packaging targets by cyperpunks
Parent article: Python packaging targets

I am afraid this is excessively optimistic, even apart from the general principle that the point of stable distributions is to not do this.

Common problems that prevent the distribution from just updating to a new version:

1. The new version broke backward compatibility in some way (possibly intentionally, possibly by accident), and now other packages that depend on it don't work and also need to be updated (or may not have updates available yet).
2. The new version drops support for older versions of Python that are supported in that release of the distribution.
3. The new version has new dependencies on Python packages that aren't already packaged.
4. The new version has new dependencies on C libraries or Rust crates or what have you that aren't already packaged.

There are some extremely well-behaved Python libraries that never break backward compatibility and support a wide range of versions, but this is quite a lot of work and a lot of maintainers don't do this, either due to lack of time or because this isn't how they want to spend their resources and effort.

One of the reasons why this discussion is so difficult is that system integration is a substantial amount of ongoing work. Things break. A lot. And quite a lot of this work is volunteer.

to post comments

Python packaging targets

Posted Feb 25, 2023 7:45 UTC (Sat) by cyperpunks (subscriber, #39406) [Link] (1 responses)

All this is true.

The quality of software available in pypi varies a lot and some packages break compability in a random minor upgrade.

To "resolve" this owners of other projects using the project with borken compability pin version requirement to a specific version.
This is also careless, as now is any project that needs a new of version of problematic package blocked from being installed.

The reason is that Python has global name space, there can be only version of problematic package installed and this package must
be present at runtime. Python is special here. Rust or Golang can create a suitable build environment, create binaries, install and software will run forever.

Python has the worst mix of all: an easy to learn language which causes lots of low quality package to be published to repos, careless owners which break compability, even more careless owners which pin version requirements combined with globale namespace and
no different between build and runtime environment.

The future of this is not promising.

Python packaging targets

Posted Feb 25, 2023 21:59 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

> The reason is that Python has global name space, there can be only version of problematic package installed and this package must be present at runtime. Python is special here. Rust or Golang can create a suitable build environment, create binaries, install and software will run forever.

It very much is *possible* to make hermetic builds of Python. It's just that there's so much you need to package, that you are effectively bundling an entire Python installation (like Flatpak etc.). Distros don't like this degree of vendoring, but they also have problems with Go and Rust.

Python packaging targets

Posted Mar 2, 2023 8:27 UTC (Thu) by callegar (guest, #16148) [Link] (5 responses)

This would be easier if you could

apt install some-python-pkg-1.0

/and/ also

apt install some-python-pkg-2.0

without having the latter replace the former. It would allow the distro to provide the new package, while temporarily maintaining stuff that might get broken with the old package.

In fact, given the python approach of not allowing this kind of thing, I now tend to think that a few things would be simpler if there was a policy so that at least the "major" version of the package was formally required to become a part of the package name. Even without a formal requirements, for some packages this seems to be already done.

Python packaging targets

Posted Mar 2, 2023 17:58 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (4 responses)

How do you propose these two package co-exist on the disk? If both are installed, what does `import some_python_pkg` do?

Python packaging targets

Posted Mar 2, 2023 21:19 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

Sorts and picks the highest version number (using standard version sorting), unless a suitable piece of metadata in the installed package directory (or import line, since obviously we'd want to augment that and allow version ranges!) specifies otherwise.

This is not rocket science. It's just a tricky compatibility and syntax thing.

Python packaging targets

Posted Mar 3, 2023 0:43 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> This is not rocket science. It's just a tricky compatibility and syntax thing.

Isn't this kind of the use case that this whole article series has been about though? It might not be rocket science, but it certainly doesn't seem easy. We (as a species) actually seem to be better at rocket science if you were to ask me…

Corner cases that come to mind:

- Two libraries provide metadata to describe their dependencies and they disagree about what version to use.
-- What is preferred?
-- Does every library/application need to specify their full transitive requirements?
- How do I prod random script developers to provide this information? Can it be specified in-source or is it sidecar metadata?
- If they are allowed to co-exist, what does `sys.modules` look like now?

Python packaging targets

Posted Mar 3, 2023 13:30 UTC (Fri) by nix (subscriber, #2304) [Link]

> - Two libraries provide metadata to describe their dependencies and they disagree about what version to use.

I don't see how (in the absence of hermetic namespacing) this can be anything but a runtime error. Of course, again, other languages can do this easily...

> - How do I prod random script developers to provide this information? Can it be specified in-source or is it sidecar metadata?

I was assuming both: a file in the package directory versioning lots of things at once, or extra syntax for import. Sidecar seems totally wrong: this is something *the package* is expecting, not something the admin is supposed to be nailing down (how could the admin possibly know the right answer?)

Obviously if a given import doesn't have a version nailed in, the system works exactly as it does now: anything else would be a compatibility disaster. (Needless to say I have written none of this code so I'm totally bikeshedding and waving away all the no-doubt-horrible problems!)

Python packaging targets

Posted Mar 2, 2023 21:19 UTC (Thu) by atnot (guest, #124910) [Link]

Like in many other language ecosystems (and even sort of python with virtualenv), there would be some shim that tells the interpreter/compiler what packages the program should have available in what version. This would be read from some sort of file provided along with the program or source code.