Python packaging targets

I've been doing a lot of in-the-background thinking about the *many* difficult packaging situations that get lots of discussion here and elsewhere. I'm mostly coming from a desire tokeep the advantages my distribution-provided package management has while still allowing for random programs to do stupid stuff without breaking the world.

It seems to me that there are two particularly key things that distros and various non-distro package/dependency management systems could agree to cooperate on that would make a huge difference and would likely consolidate a lot of the stuff we're endlessly fighting against into a relatively simpler set of problems with much more shared code/behaviour (though still involving a chunk of new work!)

1. Distros should be much happier having multiple versions of the same package available, with suitable cleverness to provide the "correct' version to any software that runs

2. Tools like pip/cargo/npm/uncle Tom Cobley and all should be able to make use of the OS provided package manager to satisfy dependencies where suitable versions are available.

Imagine a conversation something like this:

UTC: $OS, I'm installing Package_FOO v1.1, it depends on Package_BAR>=1.1<2.2, Package_BAZ>=5.5<7.7, and Package_QUX>=6.3

$OS: I can provide Package_FOO 1.5, 1.6, or 1.7, and Package_BAZ 5.8, but no Package_QUX

UTC: Here's Package_QUX 6.9, and Package_FOO v1.1

$OS: Gotcha, Fam!

Then whenever Package_FOO is used, it gets suitable versions of its dependencies, possibly using some magic with containers and bind/union mounts to de-dup a bunch of stuff.

If we've got this in place, it would also potentially make it easier to support a variety of non-privileged package installation stories, which could be exciting...

There are a lot of hairy details to sort out, of course, but I can't help thinking this is a potential way towards happiness!

Hairy details that occur to me:

1. Version specifications will probably be a lot more complex than in my example, including multiple allowable version ranges, details of how the packages are built (which CPUs are supported, which features are enabled etc.) and more.

2. The dependency graph for some packages might be arbitrarily complex, so we'd probably end up with UTC providing the whole graph to $OS to sort out multiple possible combinations

3. The upgrade story needs some care(!) We'd want to allow $OS to upgrade Package_BAR to 1.9 and remove 1.7 when no packages depend on it, with the user able to control which packages get their dependencies upgraded when. This is basically the same as a normal distro upgrade process except that different packages might have different upgrade schedules, as well as different version requirements.

4. We'd want users (and sysadmins) to have suitable policy control over how UTC-provided package are shared (single package only, per user, per non-distro-provider, system wide...)

5. The interface at which "Packages" meet is *complex*. Version requirements and matching will likely be different depending on whether we're talking about dynamic linking to libraries, calling external binaries, connecting to other processes over pipes/networks...

6. Obligatory https://xkcd.com/927/

7. All the still remaining reasons why Package Management Is Hard that this idea doesn't address.

8. The whole concept is a half-baked cockamamie wild goose chase dreamt up by a chump who would need much smarter people to shoot it down / fix it / implement it :-P