LWN: Comments on "Cooperative package management for Python"

Cooperative package management for Python

laarmen — Fri, 10 Sep 2021 09:09:13 +0000

You're focusing on the functionality here, but there's an important point made by stefanor : venv pulls in several MiB of dependencies.

My guess is that a choice has been made to spare those MiB for all the Debian installs which simply want to use a Python program, not *develop* one (or deploy an application with dependencies not satisfied by Debian...). This is fairly logical and in keeping with other Debian packaging practices, such as not shipping the library headers with the main lib package, etc...

And yes, there's a lot of other ways some space could be saved on Debian installs.

Cooperative package management for Python

j0057 — Wed, 08 Sep 2021 12:54:47 +0000

Oh right, I see, and Debian uses it to keep system-Python dist-packages separate from self-compiled-Python site-packages, so a slightly different use case than keeping system Python packages separate from user-installed Python packages.

Cooperative package management for Python

dbnichol — Tue, 07 Sep 2021 18:22:33 +0000

I'm not sure exactly how it works now, but https://lists.debian.org/debian-python/2014/05/msg00025.html has some background.

Cooperative package management for Python

nevyn — Tue, 07 Sep 2021 04:47:26 +0000

> So the distro package of python3 may not include all the standard library

Using the word "python3" as though it's a single thing is confusing you, and probably others trying to follow along. For almost any project in a distribution you have at least three things worth talking about: 1) All of it (roughly what you get if you installed from an upstream tarball). 2) The major usable piece of it (/usr/bin/python3 and all of stdlib. so you can run arbitrary python3 programs, in this case). 3) Specific dependencies so that distribution package FOO can run correctly.

All three of those things might install /usr/bin/python3, and could thus. be called "python3" but are likely to be very different otherwise.

Cooperative package management for Python

Conan_Kudo — Mon, 06 Sep 2021 23:35:01 +0000

"dist-packages" was always a Debian-ism. At least I've never seen it in any other distribution family. Ruby has a concept of "vendor directories" and "site directories", and Python would probably benefit from a standardized approach like that.

Cooperative package management for Python

j0057 — Sun, 05 Sep 2021 20:35:38 +0000

It used to be that distros were expected to put Python packages in dist_packages and admins their packages in site_packages. I'm probably missing something, but that always seemed fairly sane to me. Most distros however put their Python packages in site-packages. Debian still does it this way, so it doesn't seem entirely unsupported.

Cooperative package management for Python

NYKevin — Thu, 02 Sep 2021 19:14:46 +0000

% apt show ca-certificates
Package: ca-certificates
Version: 20210119
Priority: optional
Section: misc
Maintainer: Julien Cristau <jcristau@debian.org>
Installed-Size: 391 kB
Depends: openssl (>= 1.1.1), debconf (>= 0.5) | debconf-2.0
Breaks: ca-certificates-java (<< 20121112+nmu1)
Enhances: openssl
Download-Size: 247 kB
APT-Manual-Installed: yes
APT-Sources: [my employer's private mirror]
Description: Common CA certificates
Contains the certificate authorities shipped with Mozilla's browser to allow
SSL-based applications to check for the authenticity of SSL connections.
.
Please note that Debian can neither confirm nor deny whether the
certificate authorities whose certificates are included in this package
have in any way been audited for trustworthiness or RFC 3647 compliance.
Full responsibility to assess them belongs to the local system
administrator.

I am willing to concede that it is technically possible that someone, somewhere, might not want to have this ~400 KiB package, and/or OpenSSL, installed.

The other five, however, are nonsense. Two of the packages you list are just more specific versions of venv, and pip-whl and distutils are both part of the Python stdlib proper, which Debian has artificially split out. That leaves lib2to3, and I would be shocked if that *wasn't* self-inflicted by Debian supporting old versions of Python. Upstream certainly shouldn't be depending on it anymore.

In short: The only real argument I'm seeing here is "some people want Python, but don't want OpenSSL." I'm skeptical that this is a very large or interesting class of users, worth the inconvenience it causes to everybody else.

> The harried sysadmin you describe will get an error message describing exactly what they need to do to resolve their issue.

No. They will click the big shiny "deploy" button, deployment will not actually happen, and they will then spend 20+ minutes digging through logs until they eventually figure out that it's the venv step which is failing.

Cooperative package management for Python

kpfleming — Thu, 02 Sep 2021 16:45:54 +0000

... with the inevitable result that such 'ugly' flags will just be subsumed into a curlbash URL that the tutorial tells you to execute.

Cooperative package management for Python

stefanor — Thu, 02 Sep 2021 00:22:00 +0000

> There is no logical reason for that limitation.

Yes there is, it's minimising the payload of Python to support applications written in Python.

On a clean Debian bullseye install, installing python3-venv requires an additional 7 binary packages using 5MiB of disk space, over just python3.
They are: ca-certificates openssl python-pip-whl python3-distutils python3-lib2to3 python3-venv python3.9-venv

We've simplified things over time. python-pip-whl currently contains 31 wheels, in the past these were all separate binary packages.

The harried sysadmin you describe will get an error message describing exactly what they need to do to resolve their issue. This probably one of tens of such issues they'll see while automating their deployment.

Cooperative package management for Python

NYKevin — Wed, 01 Sep 2021 22:42:59 +0000

There is no logical reason for that limitation. The only thing that venv actually does is make a directory with a very specific set of regular files and directories underneath. It does not interact with the system's package manager or site-packages in any way. The scripts which it creates are specifically designed to *prevent* the user from interacting with the system's package manager or site-packages. venv is not "just for developers" either, since it can used in (for example) an automated deployment script for a cattle-not-pets environment. That's why it has a programmatic interface.

Maybe you disagree with me about where exactly to draw the line between "developer-oriented" and "regular" parts of the stdlib. But for better or for worse, the documentation at docs.python.org strongly suggests that this is a standard part of Python, and that the harried sysadmin who just wants to automate their deployment process (without doing a whole lot of faffing about with Docker and Kubernetes) can reasonably assume it will work, because it's part of the whole Batteries Included thing. And of course, it actually *does* work on the sysadmin's laptop, because at some point they installed venv or something which depends on it, and then forgot about it, because setting up a venv is practically the first thing you do after you graduate from writing Hello World programs in Python.

I can understand limiting ensurepip so that it can only install into venv. But splitting out venv altogether just smacks of the pretentious "for your own good" nonsense I got sick of with the GNOME folks years ago. This is not some huge package like GCC or Make, nor is it a set of bindings to a big thing that might not be installed (like Tkinter). venv is a relatively small set of self-contained tools which upstream has loudly advertised as Included.™ I don't see which users actually benefit from breaking it out into a separate package, unless the only benefit is a slightly smaller installation size. But then you might as well go around breaking everything out into separate packages. Let's have python3-pathlib and python3-itertools and python3-enum and...

Cooperative package management for Python

stefanor — Wed, 01 Sep 2021 22:16:21 +0000

> For quite a while, this was not the case, and you simply got a broken venv. Fortunately, it would print an error message explaining the problem, but that's still incredibly obnoxious behavior.

That is still the case. It doesn't complete creating the venv, prints an error explaining why, and exits non-zero.

Cooperative package management for Python

NYKevin — Wed, 01 Sep 2021 21:59:44 +0000

I'm fairly certain that's the whole point. By using a blunt, "why would I ever want to do that?" flag name, you discourage users from blindly copy/pasting garbage commands into their terminals from some poorly-written tutorial they found on the web somewhere.

Cooperative package management for Python

NYKevin — Wed, 01 Sep 2021 21:12:05 +0000

The problem is that nobody wants this:

1. The vast majority of the time, a regular end user (not a developer) either wants to install via their native package manager, or via something like an Ubuntu PPA or Snap/Flatpak. Using pip is a Bad Idea because tools like unattended-upgrades don't know how to interact with it (so you don't get automated security updates, your sysadmin can't audit your installed packages, etc. unless someone goes around building out all of that functionality to support pip). Also, pip can easily install packages which are Not Ready For Production Use,™ usually with very little or no warning to the end user, because it's a developer-oriented tool.
2. The vast majority of the time, a developer wants to install into a venv (virtualenv, or "virtual environment"), which is basically a directory with a bunch of locally-installed pip packages and some environment-variable-modifying scripts. The net effect is that, when the venv is "active," Python looks inside the venv before or instead of the system-wide packages. venvs are very lightweight; you can remove them with rm -r, and they leave no trace on the system. In practice, most native package managers are not designed to do that sort of thing (unless you use something much heavier than a venv, such as spinning up an entire Docker container).

Cooperative package management for Python

NYKevin — Wed, 01 Sep 2021 20:59:29 +0000

> When used by the venv module, it just does what you expect, and creates a venv seeded with pip.

For quite a while, this was not the case, and you simply got a broken venv. Fortunately, it would print an error message explaining the problem, but that's still incredibly obnoxious behavior.

Cooperative package management for Python

cyperpunks — Wed, 01 Sep 2021 20:42:10 +0000

Could an unified package manager (upip) be an ok idea?

$ upip install spam

will first search distro repos for spam and install found package by native pkg package tool.

If not found in native repos, jump to pip (in the background of course) and install into special pip dirs
(which are outside native pkg manager dirs.)

Keep track of packaged install by both pip and native tooling.

If doing:

$ upip remove eggs

would break something in either systems it will complain and prevent such action.

Cooperative package management for Python

stefanor — Wed, 01 Sep 2021 18:22:36 +0000

The easy way to find out the details is to look at what python3-full depends on.

The written documentation for the split is here:
https://www.debian.org/doc/packaging-manuals/python-polic...
And yes, that often lags behind the reality in the archive, it's easy to forget to update documentation. It is currently accurate, though.

Cooperative package management for Python

mathstuf — Wed, 01 Sep 2021 18:08:19 +0000

Perhaps better would be "--i-read-the-docs-and-allow-pip-to-break-things-it-thinks-it-should-not-touch". Everyone is going to be copy-pasting (or (ha!) tab completing) this anyways, so why make it convenient?

Cooperative package management for Python

MattBBaker — Wed, 01 Sep 2021 17:48:17 +0000

> ‑‑break‑system‑packages

Oh this is going to be a fun flag to explain to users.
"Why would I want an option to hose my system install?"
"Okay, it's not an affirmative action telling it to hose your system. Pip just gained the ability to tell if your about to hose your system, and advise you not to. But give you the option to do it anyways if you really want to."

Cooperative package management for Python

xecycle — Wed, 01 Sep 2021 14:27:45 +0000

Not really related to your main point, but python defaults to include $PWD in its sys.path when you say -m without -I, and $PWD can shadow standard libraries (except sys). So if you consider $PWD as untrusted, or if you have some surprisingly-named .py files on $PWD, then that command won’t work, regardless of distro.

Cooperative package management for Python

martin.langhoff — Wed, 01 Sep 2021 14:22:41 +0000

As a former distro packager _and_ library developer I really like this.

- "EXTERNALLY_MANAGED" indicates perspective - external to whom?. Instead, doing something like `echo rpm > MANAGED_BY` is much better, scalable, future-proof.

- We need this same dialog (and mechanisms) for PHP, Perl, Ruby, JS...

Cooperative package management for Python

beagnach — Wed, 01 Sep 2021 12:31:02 +0000

So the distro package of python3 may not include all the standard library... that came as quite the surprise. But yes, it makes sense from a distro package manager point of view.

> The good news is that the python3-full binary package now exists to meet this need. If you're a Pythonista, you can install this and get the stdlib you expect
> python3: The main CPython interpreter package, including the standard runtime stdlib.

The difference between these two packages isn't quite clear to me.

Are there parts of the standard library absent from the python3 package in latest Debian? If so is there an easy way to find out the details?

My experience has been that this is often under-documented - it can be quite hard to establish exactly how components are divided up between the various debian packages relating to a single large project.

Cooperative package management for Python

MrWim — Wed, 01 Sep 2021 09:40:29 +0000

> ensurepip never really made sense in a typical package-managed Linux distro. Distros have package-managers that are responsible for installing things. They don't tend to get on well with other tools messing in the same trees of the filesystem.

I think this is exactly why it makes sense to include venv wherever you include pip. venv is the mechanism that people use to avoid messing with the same trees of the filesystem. By not including it you have a pip that can mess with the distro provided packages, but you don't have the capability to sandbox off these changes.

Note: you don't need to be a Python developer to want pip. You'll need it whenever you want to run any non-distro-provided Python software - not only when developing it. It's exactly these users who are not familiar with the Python packaging tools that are at most risk from breaking their systems in a way that they don't know how to diagnose or fix.

Cooperative package management for Python

stefanor — Wed, 01 Sep 2021 05:49:26 +0000

> If you tell me that your operating system has "Python" on it, I am going to assume that I can use and call into every single part of the stdlib.

The good news is that the python3-full binary package now exists to meet this need. If you're a Pythonista, you can install this and get the stdlib you expect.

But let's dig deeper and questioning the assumption you made. Why do distros break it?

There are some optional parts of the stdlib (e.g. database drivers), and then there are the mechanics of the operating system to consider.

If an app in your operating system is written in Python, it is reasonable for that application to depend on a subset of the Python standard library, to reduce install footprint by minimizing dependencies. This may sound unreasonable at first, but there are things in the Python standard library that you really don't need for app runtime: e.g. documentation, dev headers, stdlib test suite, Tkinter, IDLE, lib2to3, ensurepip, or distutils. As a Python developer, these may seem like sacrosanct parts of the stdlib, but as a distro maintainer, they are not something that you need to waste install CD space for, and most desktop end-users will never miss them. You can significantly reduce the installed size of the Python stack and its dependencies on most end-user systems by making these components optional.

Generally distributions break complex packages up into multiple pieces, trying to find a balance between a minimal core and all the optional bits that users may need for their particular use case. (Not every optional feature will be supported, of course). In Debian, libreoffice is broken into around 200 binary packages, as an extreme example.

Python in Debian is broken into several major pieces:

python3: The main CPython interpreter package, including the standard runtime stdlib.
python3-minimal: Intended for install environments and size constrained CD images. Just the CPython interpreter + a minimalist subset of the stdlib.
python3-doc: Documentation.
python3-dev: C header files, the -config script, and a static version of libcpython.
python3-distutils: The distutils module (only needed at build time)
python3-examples: Examples, Demos and Tools
python3-dbg: A debug build of the CPython interpreter
python3-gdbm: The GNU dbm driver (and dependency on libgdbm)
python3-tk: tk module (and Tcl/Tk dependencies)
python3-venv: Depends on the wheels that ensurepip requires to bootstrap pip into a venv.
libpython3.X-testsuite: The stdlib test suite
idle-python-3.X: IDLE
Technically, there are a few more packages, but these are the functional break-points.

> The entire purpose of ensurepip was to *ensure* that everyone had pip available with every installation of Python, by incorporating it into the stdlib.

And yet ensurepip never really made sense in a typical package-managed Linux distro. Distros have package-managers that are responsible for installing things. They don't tend to get on well with other tools messing in the same trees of the filesystem. (That's what this article is about.) Debian expects Debian users to install pip by apt installing python3-venv or python3-pip (as appropriate), not by running ensurepip to install things in /usr. For this reason Debian has always hobbled ensurepip to print an error message explaining this, when executed directly. When used by the venv module, it just does what you expect, and creates a venv seeded with pip.

Cooperative package management for Python

NYKevin — Wed, 01 Sep 2021 04:13:22 +0000

As mentioned, both venv and ensurepip have programmatic interfaces and are part of the Python stdlib. If you tell me that your operating system has "Python" on it, I am going to assume that I can use and call into every single part of the stdlib. I'm not going to split out parts of the stdlib as separate dependencies, or instruct people to run distro-specific hacks to fix the half-an-installation they got by default. The entire purpose of ensurepip was to *ensure* that everyone had pip available with every installation of Python, by incorporating it into the stdlib. That's why it's called "ensurepip," and not "maybepip" or "optionalpip." By removing it, you are breaking API compatibility with standard (upstream) Python.

Cooperative package management for Python

JanC_ — Wed, 01 Sep 2021 00:57:46 +0000

Based on the 2-3 sites I looked at, it seems like people were just missing a package, as they didn’t have venv (for the correct Python version) installed?

There’s lots of other parts of “Python” that are not installed by default, e.g. development headers, documentation, tests, examples, IDLE, etc. Basically, a default install includes everything you need to run Python programs, but not everything you might need to develop in Python. I assume this is mostly to reduce its size in environments where all those aren’t needed (which is by far the majority of installations).

Cooperative package management for Python

NYKevin — Tue, 31 Aug 2021 22:42:21 +0000

I have no idea if it's still a problem today, but see for example https://www.google.com/search?q=debian+ensurepip+venv

No really, click through. Look at the sheer *number* of people who were (or possibly still are?) inconvenienced by this behavior. I'm sure they had their reasons, but they broke a lot of workflows when they did that.

Cooperative package management for Python

beagnach — Tue, 31 Aug 2021 22:01:20 +0000

What distros are in question here? How does the breakage manifest?

Cooperative package management for Python

NYKevin — Tue, 31 Aug 2021 21:03:45 +0000

It'd be nice if I could depend on python3 -m venv actually working. It depends on ensurepip, which is or was sabotaged or outright removed by some distros because they didn't want users to have a secondary package manager on their systems. The problem is that both ensurepip and venv have programmatic interfaces and are part of the Python standard library. You can't just tear out random bits and pieces of a language's stdlib. When I write code for "Python," I expect the batteries to be included as advertised.