Hashemi: The Many Layers of Packaging [LWN.net]

On his blog, Mahmoud Hashemi has an in-depth look at Python packaging, but much of it is applicable to packaging software in any language. "Python was designed to be cross-platform and runs in countless environments. But don't take this to mean that Python's built-in tools will carry you anywhere you want to go. I can write a mobile app in Python, does it make sense to install it on my phone with pip? As you'll see, a language's built-in tools only scratch the surface. So, one by one, I'm going to describe some code you want to ship, followed by the simplest acceptable packaging process that provides that repeatable deployment process we crave." (Thanks to Paul Wise.)

to post comments

Hashemi: The Many Layers of Packaging

Posted May 12, 2017 4:35 UTC (Fri) by k8to (guest, #15413) [Link] (6 responses)

The author suggests virtualenv is neither necessary nor instrumental for packaging. And I agree, but I'd go much farther.

Virtualenv is unhelpful and fragile. All it does of any real value is:

1 - adjusts the module search path to use a particular dir
2 - provides command line tools (pip, python, etc) that automatically use that directory

Even for continuous integration or testing goals, it's not worth using unless those tools only run in a controlled environment (a server farm, a containerized runtime).

virtualenv changes version independently from main python, breaks in bizzare unhandled ways (long directories, nested virtualenvs sometimes work and sometimes don't).

Even for scripted automation, I'd recommend doing the necessary work to simply download your dependencies to a stable location and inserting it first on the path. It won't break in weird ways if you run a child process that does the same thing, and any errors will be transparent.

Hashemi: The Many Layers of Packaging

Posted May 12, 2017 5:46 UTC (Fri) by fredrik (subscriber, #232) [Link] (1 responses)

I find virtualenv to be excellent for service deployment. Especially when something like Ansible is used to get repeatable deployments and upgrades. Admittedly, this is a sample of one (me), I'm no python deployment expert, and when I use virtualenv I primarily work on and deploy web services.

I haven't heard of nested virtualenvs before. It sounds like a "don't do that then" kind of design flaw if you have to use nested virtualenvs.

Hashemi: The Many Layers of Packaging

Posted May 12, 2017 19:46 UTC (Fri) by k8to (guest, #15413) [Link]

You decide to deploy a thing using virtualenvs. The people building the thing deploying decide to include their dependencies using virtualenvs. Everyone believes in it, it's safe, why worry?

It isn't. It breaks due to independent decisions in ways that are very hard to reproduce.

Hashemi: The Many Layers of Packaging

Posted May 12, 2017 7:37 UTC (Fri) by callegar (guest, #16148) [Link] (3 responses)

Virtualenv has an inherent fragility in that it breaks when one upgrades the main python interpreter. Basically, if you have many virtualenvs in a linux system, they all break when you upgrade the distro and you need to fix them one by one. Having an automated way to deal with this sort of issue would be a great step forward.

Hashemi: The Many Layers of Packaging

Posted May 13, 2017 16:45 UTC (Sat) by drag (guest, #31333) [Link] (2 responses)

I use pyenv, which is modeled after the rbenv tool. To deal with these issues for development purposes.

There exists *env work-alikes for other languages. I started using plenv for perl just recently and from my limited experience it's pretty effective. It would be lovely to see this standardized.

The root cause of the 'fragility' of things like virtualenv is depending on distro-packaged and OS-level software. You can't depend on OS software environments and distro-packaged software if you want to have something that easy to develop on and want to have both testable and recreate-able. If you want to be able to test and support different versions of multiple Linux distributions then depending on OS packaging is a terrible idea.

Using 'naked' virtualenv/venv is essentially half-assing it. So you end up with worst of both worlds were you have to go through the trouble of setting up a venv, keeping track of dependencies and such things on your own, and then depending on OS provided basic libraries and interpreters that update and break things based on their own convenience and time-lines and not your own.

So pyenv is a simple way to manage your own dev environment and divorce it from the OS-level dependencies as much as is currently easily possible. You can set (depending on the shell your using) directory-specific interpreters, shell-session specific interpreters, global default interpreters. You can integrate with things like Emacs so it's aware of your interpreter paths and libs and whatnot so you don't depend on system-version of python for things like lint.

It's not 100% as you still run into some issues with distro-specific dependencies that are used to build the interpreter, like SSL libs. The work around is to use pyenv on the older distributions to recreate the application dev environment you are using on your desktop. So I can do things like develop on Fedora 25 using my native editor and latest python version and still have very little issues targeting Centos 6... Going back to CentOS 5 is a bit more trouble, but it's actually do-able.

Unfortunately you can't expect end users to go through all the work of setting up pyenv.

The holy grail for easy distribution right now is static binaries. You have your user's wget the binary or very simple tarball and it's done. For security you provide a signature file for the download or signed sha256sum file and upload your gpg key to a public key server. The downside of not depending on OS for security updates is offset by the fact that you can rapidly get new binaries built, signed and uploaded and users don't have to do anything more then just re-download them. If a user runs into some sort of breakage with a update usually the 'roll back' is just using the old binary they have copied to a backup dir.

Lack of automation to check signatures and updates is it's real weakness versus distro-specific packaging. Also python is not designed for building static libraries so it's always going to have niggling issues when you try to do it. I've done it with success and had binaries that ran just fine on everything from CentOS 5 to latest Fedora, but it's not simple and you can't use whatever libs you want.

With the success of golang it really is pointing that this sort of thing is the way to go forward. Maybe containers can be made nice enough to solve the issue of repeatable application environments divorced from distribution-specific dependencies with automated signature checking and updates.

Hashemi: The Many Layers of Packaging

Posted May 13, 2017 20:18 UTC (Sat) by lsl (subscriber, #86508) [Link] (1 responses)

> depending on OS provided basic libraries and interpreters that update and break things based on their own convenience and time-lines and not your own.

You might be able to control everything for purely internal programs but if your software has any external users, that's what it's going to get exposed to in production anyway. Users are going to setup their systems as they feel like and won't feel guilty about not having asked for your permission beforehand. And as long as that setup is not totally crazy...why not support it, at least on a best-effort basis?

For most sane libraries that have existed for a long time and provide well-engineered APIs, relying on the OS version seems like the obvious choice. For the more unstable and experimental stuff, yeah, just link it statically.

> Lack of automation to check signatures and updates is it's real weakness versus distro-specific packaging.

Why not drop those static binaries into an RPM or Debian package then? Those won't go into the distro repos but you still get to benefit from the updating and signature checking infrastructure built into those systems.

Hashemi: The Many Layers of Packaging

Posted May 13, 2017 23:57 UTC (Sat) by drag (guest, #31333) [Link]

> For most sane libraries that have existed for a long time and provide well-engineered APIs, relying on the OS version seems like the obvious choice.

Sometimes. C libraries are tricky. However the main point is dealing with python modules themselves. When they depend on C/C++ libraries you have to make special considerations. The most troublesome tend to be things like OpenSSL. It requires distro-specific testing if you don't want to get burned by surprise incompatibilities. Since Python is designed for building dynamic things then it's difficult to create truly 'static' binaries.

Golang really wins in this regard. You can do stuff like setup a 'docker image' that consists of a single binary with some config data for a lot of really really complex golang programs and it 'just works'. I would really love to get that point with python.

The LAST thing you want to do is ever muck around with distro-packaged modules. You don't want to change them, update them, patch them, or do anything. How distro packaging works it is extremely fragile to any third party changes. This is true in my experience for Perl, Emac packages, and python modules. The quickest way to introduce nasty bugs and issues for end users is to try to install stuff that could show up system-wide.

> Why not drop those static binaries into an RPM or Debian package then?

It's not something you get 'for free' just by creating a bunch of rpms and deb packages.

If you want to take advantage of validating signatures and automated updates then you have to setup your own repository for each distro you want to have it be used by. The end user has to add the repository either by hand and use distro-specific things for importing signing keys or you have to build and provide additional rpms/debs that sets up their repo configs and signing keys. Each distro is different. Then you have to test that stuff for each distro and version to make sure it works. And you'll still have to provide binary-only downloads and signature files for people using Arch/Gentoo/Docker or whatever.

For my purposes things I make I send out to people for automation.. like doing migrations and upgrading for clusters of virtual machines. It's all very environment-specific and I can send people a email or write a ticket with a tarball attachment they can extract and it'll probably work regardless how they have their desktop or jump box setup. If something goes wrong I can fix it and send it back out to them again. If they want to fix it themselves I have instructions and code in git.

If I was distributing for public consumption I would definitely take a look at something like Suse's build services for multi-distro rpms and such things. There is probable a nice way to make it all be very nice with little effort, but I haven't investigated because it's not really that helpful for me my users at this point of time.

Hashemi: The Many Layers of Packaging

Posted May 12, 2017 7:45 UTC (Fri) by callegar (guest, #16148) [Link]

What I still find strange, is that after more than 25 years of python and a non-compatible bump from 2 to 3, still there is no possibility to have different versions of the same package side by side letting each piece of code loosely or precisely indicate which one to import. This makes virtualenvs much more necessary than they could be, creating a lot of duplication on the system and in one's workflow (when you decide it is time to update a dependency because of some issue, you need to go through all the virtualenvs you created to do so). Furthermore, even with virtualenvs, this causes issues, particularly with things with a gui that end up installing PyQt[45] and sip, and often result in segmentation faults due to unexpected changes in binary interfaces.

Hashemi: The Many Layers of Packaging

Posted May 15, 2017 14:18 UTC (Mon) by zoobab (guest, #9945) [Link] (3 responses)

Pip sucks big time when you have C dependencies on the system, it just tries to spawn a gcc compiling stuff without any idea what is on the system, and most of the time it fails.

It is 2017, and I pointed out at this problem at Europython 2004, before Egg and pypi.

Hashemi: The Many Layers of Packaging

Posted May 15, 2017 15:13 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

Warning: this turned into a bit of a rant.

Ha, C is "easy". Try Fortran dependencies/code (NumPy and SciPy). Building these packages on your own is an exercise in frustration especially since they basically fork distutils in order to do their magic.

Personally, most of my frustration with Python compilation comes from Windows where only specific Visual Studio versions are supported rather than using something like CMake (patches exist) to support any Visual Studio that works with the code. For example, Python2 (yes, I know) requires the 2008 toolchain, but ships with various solution files depending on the point release meaning you need at least two Visual Studio installs to build it these days.

Why am I building Python myself you ask? Because making distributable software on Windows is awful (though once you have it, it's usually easy).

Hashemi: The Many Layers of Packaging

Posted May 18, 2017 12:58 UTC (Thu) by danielkza (subscriber, #66161) [Link] (1 responses)

Does any language package manager do anything other than hope dependencies are available when building native libraries? i certainly would not expect it to guess what my system is and how to acquire said dependencies.

Hashemi: The Many Layers of Packaging

Posted May 18, 2017 15:14 UTC (Thu) by flussence (guest, #85566) [Link]

I've seen Perl very recently making an attempt to put in (machine-readable) writing any native library dependencies a module may have, but on the whole you're right. The role of distro package maintainer has many years of job security ahead of it.