Python packaging targets

As we have seen in earlier articles, the packaging landscape for Python is fragmented and complex, though users of the language have been clamoring for some kind of unification for a decade or more at this point. The developers behind pip and other packaging tools would like to find a way to satisfy this wish from Python-language users and developers, thus they have been discussing possible solutions with increasing urgency, it seems, of late. In order to do that, though, it is important to understand what specific items—and types of Python users—to target.

Integrator or user?

Steve Dower framed the overarching problems of Python packaging—and the discussion thereof—in terms of the types of users that are being targeted. There are distinct groups of users, but the Python packaging community has not defined its audience "so we keep trying to build solutions that Just Work™ for groups of people who aren't even trying to do the same thing, let alone using the same workflow or environment/platform". There are (at least) two roles, but they get conflated:

One concrete example that some of us have been throwing around for years: the difference between an "system integrator" and an "end user". That is, the person who chooses the set of package versions and assembles an environment, and the person who runs the Python command. They don't have to be the same person, and they're pretty clearly totally distinct jobs, but we always seem to act as if end users must act like system integrators (i.e. know how to install compilers, define constraints, resolve conflicts, execute shell commands) whether they want to or not. And then we rile against system integrators who are trying to offer their users a usable experience (e.g. Red Hat, Debian, Anaconda) for not forcing their users to also be system integrators.

[...] To end with a single, discussion-worthy question, and bearing in mind that we don't just set the technology but also the culture of Python packaging: should we be trying to make each Python user be their own system integrator, supporting the existing integrators, or become the sole integrator ourselves?

Paul Moore replied that the packaging community "should be supporting Python users". That means helping many of them not to have to be integrators, for others it means providing a path "to do a simplified integration job". In the dichotomy that Dowers outlined, integrators (and distributors) build the binary artifacts needed for the packages that are required, while users consume those artifacts typically via some form of package manager.

But it is also important that all of those users are able to get Python and its packages in a form that meets their needs, Moore said. He thinks that the community should be supporting the existing integrators (e.g. Linux distributions, Anaconda) and encouraging new ones for niches that are not well served, like "Windows users for whom conda isn't the right solution". But he does not think it makes sense for the Python community to become the sole integrator, unless the steering council wants to bring packaging inside the core-development tent.

If the packaging community wants to better support integrators/distributors, though, it needs cooperation from those integrators, Moore said. There also needs to be better documentation on the Python home page about how to get Python and how to choose an integrator if one is needed. Though the push from users seems to be for something that would include virtual-environment management and more (a la Rust's cargo), that may be a step too far for Python at this point; a unified package installer might be possible, however:

A common install command - either implemented in core Python with "hooks" for integrators to link to their own implementations, or a defined command like pyinstall which all integrators are required to implement with the same interface, so that the user experience is identical.

Python developers who want to do their own integration are important, Ralf Gommers said, but they are also over-represented in the discussion; most language users "want things to work well without too much trouble". He agreed with Moore that better supporting the existing integrators was the right goal. Moore pointed out that unifying the install command would allow the Python Package Index (PyPI) to switch from recommending "pip install" to something more generic; beyond that, though, "if we don't unify interfaces, how do we address this aspect of what users see as 'too many options'"? But Dower said that pip is the right choice for the unified command "because it is how you get packages from PyPI"; the missing piece is a way for pip to know that it should not be installing a package because it should come from the distributor/integrator:

I can imagine some kind of data file in the base Python install listing packages (or referencing a URL listing packages) that pip should warn before installing, and probably messages to display telling the user what they should prefer. Distributors could bundle this file in with their base Python package so that anyone using it gets redirected to the right source.

That would require more coordination between the developers of pip and those for other environments, such as conda and the Linux distributions, Moore said; the longtime message has been that pip is the wrong way to install packages system-wide on Linux, for example. He is not against the idea, but worries that it is not going to be user-friendly:

So users would go to PyPI, find instructions to do pip install X, try that and get told "nope, you should use apt-get install python-X instead"? That doesn't seem like it's going to reduce user confusion, I'm afraid... (It's OK for experienced users, but those are the sort of people for whom the current "be your own integrator" model is acceptable).

No matter what path is chosen, it is important to ensure that the community knows about it, Russell Keith-Magee said. In fact, the communication about the strategy "is just as important (if not more so) than the strategy itself", especially since "it's going to take months or years" to get to the desired end point. While the Python Packaging Authority (PyPA) is not exactly a standards body, it can help guide the community in the right direction:

However, even an informal statement declaring a vision and desired future state for Python packaging would be invaluable in terms of setting community expectations, generating convergence towards that desired future state, and guiding the contributions of those who might be inclined to help get there.

Building versus installing

Installing a package from PyPI can range from a simple, straightforward task, for pure Python packages, to extremely complex, requiring multiple compilers, build systems, and more. In the easiest cases, pip (or something like it) can simply copy a source distribution (sdist), which is a .tar.gz file, and install its files in the proper location so that import can find them. In more complicated cases, where there are dependencies on other PyPI packages, the pyproject.toml file can be consulted in order to pick up those dependencies (and, naturally, their dependencies) before installation. The sdist and pyproject.toml formats are described in PEP 517 ("A build-system independent format for source trees") and PEP 518 ("Specifying Minimum Build System Requirements for Python Projects").

If one or more of the dependencies requires a compiler in order to build a Python extension written in C, C++, Rust, or some other language, it is often up to the user to figure that out and install the right pieces, unless there is a wheel file available on PyPI for the installer's environment. The binary wheel file will contain a library that has already been built, so the build tools are not required. But the combinatorial explosion of environments makes it difficult for PyPI to have wheels available for all of them; this is especially true for packages like NumPy and SciPy, where speed is critical, so the libraries need to be optimized for the specific CPU instruction set and GPUs available.

One way to arrange for that is to split the job into two parts, as Hatch maintainer Ofek Lev described:

To be super specific here, for building a wheel with compiled bits you need 2 distinct things: a build backend and an extension module builder.

The build backend is the thing that users configure to ship the desired files in a zip archive which we call a wheel. The extension module builder is the thing in charge of compiling like cmake that will then communicate with the build backend to say what extra files should be included.

Lev thinks that his extensionlib toolkit provides the right model, though Gommers disagreed. The problem is that Lev is oversimplifying, Gommers said:

You seem to be under the impression that it's a matter of finding some compiler, pointing it at a few source files, and producing a .so/.dll. In reality, dealing with compiled code is (at least) an order of magnitude more complex than anything that any pure Python build backend does.

Henry Schreiner, who maintains several different packaging tools including the scikit-build build-system generator, noted that the packaging landscape has improved a great deal since the days of the "setuptools/distutils monopoly". In particular, setuptools and the now-deprecated distutils are painful for some kinds of projects to use (it "requires up to thousands of lines (14K in numpy [distutils], IIRC) to work"), which is not really maintainable. Instead of having a Python-based builder for extension modules, it makes sense to use external build tools, he said:

Certainly, in the compiled space, many/most users will want a build system like CMake or Meson - building a compiled extension from scratch is really hard, and not something I think we want to compete on. Reusing the years of work and thousands of pre-existing library integrations is valuable.

Gommers put together a lengthy blog post that described his vision for the path forward. That plan would severely restrict the kinds of packages that would come directly from PyPI ("only pure Python (or -any) packages from PyPI, and everything else from a system package manager"). There was some pushback on that, but Dower suggested softening the restriction somewhat, while looking for some consensus:

I think this thread is the place to agree that when a distributor has prebuilt binaries for their build of Python, those should be preferred over unmatched binaries. If we agree that the default workflows should somehow support that, we can do details in new threads.

So an Anaconda or Linux environment that has a binary version of some package available would use that binary in preference to any PyPI wheel that might—or might not—work in that environment. There are lots of wheels at PyPI that work fine outside of the specific environment they were built for; if they get installed on such a system they may not work, however "either in obvious ways (fails to import) or subtle ways (crashes at runtime, silent data corruption, etc.)", Dower said. The problem with the "get your binaries from a distributor" model, Moore said is that it generally means users need to get their Python from the same source:

And that means "switch to a different stack", which for many users is simply not acceptable (especially if they don't know this is going to happen in advance, which they typically don't).

If we were talking about distributors/integrators shipping binaries which worked with whatever Python installation the user had, there would be a lot less heat in this discussion. If I could install the Windows Store Python, and then find I needed to get (say) scipy from somewhere other than PyPI, and there was somewhere that shipped binaries that worked with the Windows Store Python, I wouldn't have the slightest problem (well, the UI of "pip fails, find the right command" needs to be worked out but that's a detail). But I've seen no sign of anyone offering that.

There's a reason no one is offering it, Dower said; it simply is not possible to do so:

The reason that binaries don't work with "whatever installation of Python the user has" is because you need to know about the installation in order to produce the binaries.

[...] Most of the time on many modern desktop and server machines, the ABI is usually close enough that you can squint and get away with it ("wheels which work for "most" users"). Once you get into the territory where that doesn't work, you do need to know all the details of the ABI in order to build a compatible binary for that interpreter. The easiest way to do this is to get the interpreter from the same builder you get the binary from, because they'll have ensured it.

In general, Dower's proposed agreement was well-received, but Moore thought that is was only handling the easy part: "The difficult question is when a distributor doesn't have prebuilt binaries, what should happen?" The right answer, according to Dower, is to request that the distributor provide the package and, failing that, to build it from source "using the distributor's environment as the target"; that way the resulting binary module is compatible with the distribution.

The discussion roared on, but along the way Moore stepped back to look at the overall goal of the discussion. If someone were to wave a magic wand that resulted in a consensus on the single tool to use: "What would we then actually do to make that consensus effective?" He does not have a good answer to that question, but it feels like the community "expects the PyPA to have some sort of influence (or authority) over this, and our biggest issue is basically that we don't..."

Clearly communicating the recommendation would be what many users are looking for, Oscar Benjamin said. "If PyPA said 'we now recommend tool X' and there was clear documentation for using it and migrating to it then I would be happy to go along with that. I think that's where most users are at." Moore was unsure how the PyPA could actually accomplish that: "I genuinely don't know how the average user distinguishes between a formal recommendation from the PyPA and a bunch of random documentation they found on the internet. Is packaging.python.org genuinely that influential?" But Dower thought there was a straightforward answer to that: "I suspect for most users, 'preinstalled with the python.org installer and on PATH by default' is what it would take." Moore agreed; "Now all we have to do is agree what tool to distribute ;)"

There are, of course, several possibilities for which tool that might be, but obviously pip is in a rather privileged position, since it is already shipped with Python. Whether pip can be adapted to be the grand, unified tool was discussed further as the strategy thread began to wind down—after nearly 300, often lengthy, posts. The rest of the discussion on which tool—and how to make that decision—will come in our next installment.