Distributions wrestle with pip

One of the headline features for Python 3.4 was the inclusion of the pip Python package installer by default. But now that 3.4 is out, distributions are finding that the interactions between Python's pip and system-installed pip programs were not completely thought out. Complicating things further is the virtualenv program that is used to create a private Python environment for development and other purposes.

Barry Warsaw sounded the alarm in a long message to the debian-python mailing list. There is an ensurepip program that is shipped with Python 3.4, but disabled in Debian, that will bootstrap pip for a Python installation. Pip is released on its own schedule, separate from that of the main language and standard library. ensurepip is used to install (or upgrade) the pip package and, as its name implies, ensure that pip is available. Debian has its own .deb-based pip package, though, so ensurepip is not needed for the distribution as package dependencies will pull in pip.

Except that it is needed. When creating a virtual environment using the pyvenv program—which is shipped with Python to create virtual environments—on Debian, the lack of an installed ensurepip causes a failure. To further complicate an already murky situation, the virtualenv command from the Debian python-virtualenv package works just fine. Virtual environments create a Python installation that is entirely separate from any other Python installation on the system. They can be used to simultaneously run multiple versions of Python (2.x and 3.x, say) or to run programs using different library versions (multiple versions of Django, for example).

Debian is not the only distribution struggling with pip for Python 3.4. Fedora has also encountered it. It has also been discussed before by both distributions (and presumably others).

The idea behind bundling pip comes from Python enhancement proposal (PEP) 453. It is really aimed at users of operating systems that don't have the equivalent of a distribution package manager (e.g. Windows or OS X) to make it easier for those users to access packages outside of the standard library at the Python package index (PyPI). But even a distribution with as many packages as Debian has does not package all of those available at PyPI, so there is still a need for pip, even on Linux distributions.

But it is clear from the threads that pip is not universally admired. One of the problems with it was raised by Scott Kitterman back in September: there is no package validation when using pip. There are plans to add that feature, but that is not sufficient for some. As Kitterman put it: "I think that introducing a package download mechanism that is not cryptographically secured with a promise to later insecurely update the mechanism to have security is crazy talk."

In addition, pip does not play well with Python packages that are installed using the standard distribution package manager. It acts as if it owns the whole Python installation and will just overwrite files in the site-packages directory when run as root. It clearly shows its non-Linux bias, which is irritating to some and can lead to unexpected results. For that reason, distributions generally try to restrict pip to either not run as root or to only affect the user-specific package installation (in ~/.local).

Part of the solution may lie in finding a foolproof way to determine whether Python is running in a virtual environment or not. Warsaw outlined several tests that can be made to try to figure that out (and to figure which type of virtual environment it is). He suggested incorporating that into Debian's version of ensurepip so that it would be available for use in environments created with pyvenv, but not globally (which would allow users to run pip as root and potentially step on files installed by Apt and friends).

Fedora has taken a different approach, as described by Bohuslav Kabrda. By using the custom rewheel program, Python packages in the Wheel format (which is how the bundled pip is delivered) can be unpacked, modified for Fedora, then repacked into a new Wheel file. The ensurepip program can call rewheel to install pip properly for Fedora.

Though rewheel is not distribution-specific, Warsaw was not particularly interested in that approach for Debian (or, especially, Ubuntu, where the pip package lives in Universe rather than the main repository). It would introduce a circular dependency for Debian and cross-repository dependency for Ubuntu, Warsaw noted. But Fedora is also preparing for a switch to Python 3 as its default in the not too distant future, so the rewheel plan is expedient, Kabrda said.

There is clearly something of an impedance mismatch between Linux distributions and languages (or other systems) that have their own idea of how packages or add-ons should be installed. Making pip play more nicely with distribution package managers would go a long way toward solving the problem for Python, at least. There are movements in that direction, but we aren't there yet.

[ Thanks to Olav Vitters for a heads-up about this issue. ]