Coghlan: The Python packaging ecosystem [LWN.net]

Here's a lengthy piece from Nick Coghlan on how Python software gets to users. "There have been a few recent articles reflecting on the current status of the Python packaging ecosystem from an end user perspective, so it seems worthwhile for me to write-up my perspective as one of the lead architects for that ecosystem on how I characterise the overall problem space of software publication and distribution, where I think we are at the moment, and where I'd like to see us go in the future."

to post comments

Coghlan: The Python packaging ecosystem

Posted Sep 18, 2016 22:43 UTC (Sun) by walex (guest, #69836) [Link] (25 responses)

The typical thorough misunderstanding of packaging from someone in the Python swamp (not ecosystem): no addressing of issues like proper dependencies, signatures, integrity verification, overriding, sharing among users, multiple concurrent versions, etc., most of which have been addressed for decades by RPM, more recently DPKG and others. Put another way: naive disregard for system maintainability. Same as for most other single-user single-language oriented "ecosystems". What a joke.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 7:25 UTC (Mon) by niner (guest, #26151) [Link] (3 responses)

That's why we're aiming for meta data that allows for 100 % automatic creation of RPM (and therefore DPKG) packages for arbitrary Perl 6 modules. Because those issues are all already solved there and this will integrate much better with existing systems.
That said, Perl 6 makes the job easier, as it already supports having multiple concurrent versions of modules installed. This makes system wider installation much more useful, too.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 22:42 UTC (Mon) by ooyoim6o (guest, #110371) [Link] (2 responses)

I tried to find info about this facility with RPM metadata in Perl 6. Can you point me to some documentation about it? It sounds very interesting to me.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 14:44 UTC (Tue) by niner (guest, #26151) [Link] (1 responses)

Unfortunately, there's not much to see yet. It's still in very early stages. But so is the Perl 6 module ecosystem, so this is the right time to chose the right direction.

There are some thoughts about a declarative build system: https://github.com/perl6/toolchain-bikeshed/blob/master/b...

Part of the reason for me mentioning Perl 6 is that we would welcome input on this :) Even by people who have no interest in Perl 6 but may have some issue with existing systems.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 21:06 UTC (Tue) by ooyoim6o (guest, #110371) [Link]

Very cool. I'll take a look at this. Sadly I have long since abandoned the Redhat flavors and have little to contribute on technical merits. I really like the idea behind this tooling all the same!

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 12:15 UTC (Mon) by edomaur (subscriber, #14520) [Link] (12 responses)

Uh ?

You mean RPM solve the issues Docker & Co solve because RPM is the root of a whole slew of problems by itself ?

I don't agree with you.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 12:49 UTC (Mon) by niner (guest, #26151) [Link] (11 responses)

I've yet to hear of an issue that docker solves that would be impossible to fix with RPM.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 13:57 UTC (Mon) by torquay (guest, #92428) [Link] (2 responses)

It's not so much rpm/deb/whatever, but the processes these tools represent. To name one, it's a major cause of bottlenecks due to the middle-man problem. Someone has to make the package, get it included in the distro, keep it updated in a timely manner, etc. This person is usually different than the original author of the software. On top of that, each distro insists on having its own packaging flavor, leading to much wasted effort and the balkanization of the Linux landscape.

rpm/deb/etc are fine for low-level OS-components that are seldom changing (eg. security updates only), but are completely the wrong tools for distribution of user facing software such as Firefox, LibreOffice, Inkscape, Gimp, etc. This is why cross-distro tools such as flatpak have arisen. (I personally use LibreOffice in flatpak format, as well as other apps).

In a similar vein, I can completely understand why Python has its own packaging ecosystem. Apart from the need to aim for cross-platform functionality, no doubt part of the reason is to get away from all the problems that rpm/deb/etc bring.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 14:31 UTC (Mon) by Felix (subscriber, #36445) [Link] (1 responses)

still a tool I really would like to see is something which works like yum/dnf but with virtualenvs. To me this would entail:
- simple installation of rpms from my distro (for "complicated" packages) and custom repos (ideally with automatic wheel->rpm generation when using the activated-by-default "pypi" repo)
- nice, uncluttered output from dnf
- integrity, security etc. completely solved by using the default mechanism of dnf/rpm
- simple way to update packages in a virtualenv

Of course the CPython developers also have to care about other operating systems like Windows and Mac so most Linux-specific solutions like RPM do not appeal to them...

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 14:34 UTC (Mon) by bandrami (guest, #94229) [Link]

Isn't that the theory, at least, behind Guix? (Though it's not "there yet", I think it's fair to say.) But it's a set of containerized user-installable multi-version-installable packages that at least try to stay out of each others' way. And as a bonus the package maintenance is (or at least can be) declarative.

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 20:01 UTC (Mon) by edomaur (subscriber, #14520) [Link] (7 responses)

Oh, me too in fact.

Except I have _NO_ time to spent to solve manually rpm hell dependencies. It's better to isolate the system (rpm) and the application (pip or docker or any other solution)

Sad, but reality.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 1:43 UTC (Tue) by NightMonkey (subscriber, #23051) [Link] (6 responses)

Reality is what we make it, at least with computers, which do only what we tell them to do. Nothing more, nothing less. :)

This attitude is what seasoned Systems Administrators just laugh at, and cry from. Developers just keep trying to kick the "dependency" issues down the road until at some unknown point they magically don't happen anymore. You would desperately not want to hire a good SysAdmin to keep you out of dependency traps, but you'd rather just smash the problem into tiny pieces, all waiting to undermine your goals. "Why have one package management system, when you could have 5?" ;)

The problem is exactly this: You are using general purpose Linux distributions to support narrowly construed tasks. This is backwards.

This was why Gentoo was invented. It is a distribution *configuration machine*. You build your userspace to deal with the task you wish the 'machine' (virtual or otherwise) to do. Nothing more. I want a: "Database Server", I create a "Database Distro" with Gentoo. I want an "ActiveMQ" server, I create a "ActiveMQ Distro" with Gentoo. Etc.

This is why Gentoo is used to build ChromeOS, CoreOS, etc. It can be used to craft very specific builds of the entire kernel/userspace stack for very narrowly-defined purposes.

The 'problem' is that no one, at least in business, wants to 'pay' the 'cost' of doing the up-front work to design the whole stack to be aligned to the end-purpose. So, we have this horrid mix of general purpose distributions being asked to do very narrow tasks. But the cost is just paid under a different line-item on the budget (hey, we're restarting things but don't know why, but it fixes things. Pass me the blindfold.). But the cost is still paid.

</rant>

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 2:25 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> seasoned Systems Administrators
You mean, guys who never have to write actual applications and who have certificates with names like "Managing RHEL 5.2.1.123.234 released in 2007 on Sep 15, 12:33:12.0123Z"? Yeah, sure.

We developers actually have to produce stuff that runs on multiple OSes, often including Mac OS X and Windows. And even when we need to write apps for a single platform like Debian, we often have to use custom package versions for some dependencies.

Please, tell me how you can get two versions of pycurl installed into the system Python. One of them supports a particular brand of hardware SSL tokens and the other one has new TLS code required by another app.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 2:58 UTC (Tue) by NightMonkey (subscriber, #23051) [Link] (2 responses)

> Please, tell me how you can get two versions of pycurl installed into the system Python. One of them supports a particular brand of hardware SSL tokens and the other one has new TLS code required by another app.

There are other options besides pycurl for a lot of uses pycurl is put towards. This sounds like you-all haven't assessed the field of options for your needs. A good sysadmin would work with you and other developers to find a good solution... be it urllib2, urllib3, and using pip or similar to make local installs of pycurl. But, this is not the level I'm addressing. This should be something either fixed in upstream, or a new library be found. I have heard that Python's library world is getting old and creaky from lack of attention; perhaps this is a symptom?

Also, don't put words in other people's mouths. Whatever straw man you were conjuring up with your first sentence is not the sort of experienced sysadmin I was referring to. You don't think good sysadmins don't have to support multiple OSes? And build and deploy for them? You are sadly misinformed if that's the case.

Cheers.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 4:10 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> There are other options besides pycurl for a lot of uses pycurl is put towards.
Nope. There's already software dependent on a special version of pycurl with smartcard support. So you're basically suggesting that this software shouldn't exist. Sorry, not an option.

> You don't think good sysadmins don't have to support multiple OSes? And build and deploy for them? You are sadly misinformed if that's the case.
If these "good sysadmins" think that DEB or RPM is the universal answer to dependency management then they are definitely clueless.

I would even make a stronger statement that DEB or RPM are almost NEVER the answer for the cases when language-specific package managers are used today.

And no, I'm not putting any words in anybody's mouth. Let me quote:
> no addressing of issues like proper dependencies, signatures, integrity verification, overriding, sharing among users, multiple concurrent versions, etc., most of which have been addressed for decades by RPM, more recently DPKG and others.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 16:33 UTC (Tue) by lsl (subscriber, #86508) [Link]

> Nope. There's already software dependent on a special version of pycurl with smartcard support. So you're basically suggesting that this software shouldn't exist. Sorry, not an option.

Ok, so you have some software that depends on a specific ancient release of a non-backwards-compatible library. Or a custom hacked-up copy of it. If changing that software is not feasible, then deploying it with a private copy of that library is indeed a reasonable solution. You push it onto your existing heap of technical debt that *must* be kept working and move on.

So far I agree. I just don't get why you would make that your default mode of operation for *new* development. Life is just so much more convenient when you can install all your dependencies with a single dnf/yum/apt command. If you absolutely have to use some new API that was just introduced upstream, you may still need to ship a private copy for some time on slower-moving distros but at least it's bounded. The one thing you need to avoid is relying on libraries that break their APIs all the time without any notion of backwards compat. You won't get a common interface across distros if those are involved.

Coghlan: The Python packaging ecosystem

Posted Sep 20, 2016 14:35 UTC (Tue) by raven667 (subscriber, #5198) [Link]

I have no idea what Cyberax is on about but application runtimes you are describing can also be provided by containers built using just about any linux distro and each has their own tooling for building the distro itself that you can use if you need.

Coghlan: The Python packaging ecosystem

Posted Sep 22, 2016 5:39 UTC (Thu) by edomaur (subscriber, #14520) [Link]

>> Reality is what we make it, at least with computers, which do only what we tell them to do. Nothing more, nothing less. :)

Well, sometime you have to work within the reference frame you are given, it's not always a choice. For example, it's not so infrequent to have to battle against corporate paralisys to get things done.

>>This attitude is what seasoned Systems Administrators just laugh at, and cry from.

That's a little bit insulting.

>> This was why Gentoo was invented. It is a distribution *configuration machine*. You build your userspace to deal with the task you wish the 'machine' (virtual or >>otherwise) to do. Nothing more. I want a: "Database Server", I create a "Database Distro" with Gentoo. I want an "ActiveMQ" server, I create a "ActiveMQ Distro" with
>>Gentoo. Etc.

Is there an "Enterprise Gentoo" ? No ? So not an option, I need to work in a context where the only choice I have is the Enterprise one, even if it's demonstrably the worst one. Seasoned sysadmins knows that very well...

Coghlan: The Python packaging ecosystem

Posted Sep 23, 2016 23:50 UTC (Fri) by satbyy (guest, #107278) [Link] (7 responses)

I had the same experience with Rust's package manager "cargo". To install a rust package along with its dependencies, you run "cargo build". It downloaded ~50 deps from GitHub into $HOME/.cargo. I asked on IRC how to undo this download, they basically said to "rm -rf" them one-by-one by manually comparing their version numbers in Cargo.toml. Crazy ! In Debian, it's just "apt-get clean".

Coghlan: The Python packaging ecosystem

Posted Sep 24, 2016 0:04 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Why would you want to delete it? If you want to free up the disk space then just delete ~/.cargo entirely.

Coghlan: The Python packaging ecosystem

Posted Sep 24, 2016 0:16 UTC (Sat) by satbyy (guest, #107278) [Link] (5 responses)

I wish I could :)

It's because there's no equivalent of "apt-get autoremove" in cargo. I mean, if I uninstall a cargo package, it's dependencies are left scattering in ~/.cargo -- even if they are unused by any other package. There's no way to remove it expect manually. Deleting ~/.cargo means that I must re-download 500 packages again just because I wanted to delete 50 of those packages.

Coghlan: The Python packaging ecosystem

Posted Sep 24, 2016 5:35 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

You can.

~/.cargo is not a repository of installed packages, this is merely a cache for downloaded objects. So it can be deleted on a whim, you will only need to re-download the packages in it.

"apt-get autoremove" makes no sense at all, there's no central repository of Cargo package users. A source checkout in /home/joe/stuff-on-intermittently-connected-nfs/in/forgotten/directory/from/10/years/ago might be using a Cargo package and there's no real way to know it.

The most you can sanely get is a simple LRU eviction policy for Cargo packages.

I wish people would stop conflating language-specific package managers like Cargo or Maven with system-level package managers. They are designed to do almost completely different things.

Coghlan: The Python packaging ecosystem

Posted Sep 24, 2016 9:15 UTC (Sat) by satbyy (guest, #107278) [Link] (3 responses)

> ~/.cargo is not a repository of installed packages

Except that it is. "cargo install" will put executables in ~/.cargo/bin which will be in your $PATH. And cargo doesn't track which binaries live in that dir, you must track it yourself in a spreadsheet. IMHO if a package manager has a command to do operation "X", it must also have a command to undo "X" (when it makes sense). In my example, X = download deps. Internet bandwidth is expensive where I live. Asking user to re-download 500 dependencies of 10 different packages because I want to delete 50 deps of this package, is simply an excuse for bad design.

Coghlan: The Python packaging ecosystem

Posted Sep 24, 2016 9:23 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

No, it still is not. The ~/.cargo/bin is merely a place for built tools, kinda like "mvn install" in Java which "installs" the package into the repository. It's not a replacement for system package managers and it doesn't try to be it.

And if bandwidth is expensive at your location, then perhaps it makes sense to never delete packages.

Coghlan: The Python packaging ecosystem

Posted Sep 27, 2016 21:45 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

There is a --root= argument to cargo install. I have support scripts for it, but I do things like "cargo install --root=$HOME/misc/root/$project_name-$version" all the time. I can then select them using a wm-add-root script which amends things like PATH, LD_LIBRARY_PATH, etc. Parallel installation, removing old versions, etc. are simple with this scheme.

Coghlan: The Python packaging ecosystem

Posted Sep 27, 2016 22:21 UTC (Tue) by satbyy (guest, #107278) [Link]

Thanks for pointing out --root. The last time I checked it wasn't there. Seems cargo is improving rapidly. Cheers!

Coghlan: The Python packaging ecosystem

Posted Sep 19, 2016 12:28 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

So I get to deal with Anaconda/miniconda at work and one thing that annoys me is that if things use either one, they're stuck in that universe pretty hard. Is there a way that I can build Anaconda bits against my own versions of dependencies and also make relocatable installs of Anaconda? As far as I can tell, Anaconda doesn't really use external stuff all that well (best would be to use the system SSL support rather than us having to ship an easily out-of-date version…) and relocation for packages (that work on all 3 platforms) requires shipping Anaconda with tarballs and rerunning the installation so the baked-in paths can work (and until 2.0 were limited to extremely short path lengths).

Coghlan: The Python packaging ecosystem

Posted Sep 29, 2016 12:36 UTC (Thu) by callegar (guest, #16148) [Link]

Lack of support for multiple concurrent versions of packages is a big nuisance. Even when you work in virtual environments, since you may need two packages that require the same package, but at different versions. Furthermore, pip has a tendency to automatically upgrade dependencies when you upgrade something. The matter is even worse now that wheels are used as the standard packaging format, because not only you have requirements for compatibility at the source level, you also have some that exist only at the binary level. E.g. scipy 0.18.1 is compatible with numpy 1.11.1, but the /binary wheel/ for scipy 0.18.1 requires an older numpy, because it was compiled against an older numpy and the abi has changed.

Unfortunately, is not just a matter of lack of support for this in pip. It is also the language that misses support for versioned packages, something like

from foo["1.11":"1.17", "1.20"] import bar

would be nice.

The mess of distutils/setuptools/easypackage is also an issue.

Having packages self-contained in versioned zip files a la java would help. The sad thing is that it is possible to do so, but almost no one does.