Users and Python packaging

A lot of digital ink has been expended in recounting the ongoing Python packaging saga, which is now in its fourth installment (earlier articles: landscape survey, visions and unification, and pip-conda convergence). Most of that covered conversations that took place in November and the discussion largely settled down over the holidays, but it picked up again with a packaging-strategy thread that started in early January. That thread was based on the results of a user survey about packaging that was meant to help guide the Python Packaging Authority (PyPA) and other interested developers, but the guidance provided was somewhat ambiguous—leading to lots more discussion.

Survey

The packaging survey was analyzed by Shamika Mohanan, who summarized the results in late November. Tzu-ping Chung noted that there seemed to be conflicting answers to two of the questions asked:

People seem to generally disagree with the statement "Python packaging deals well with edge cases and/or unique project requirements", but also feels that "Supporting a wider range of use cases" is the least impactful thing for the PSF [Python Software Foundation] and PyPA to focus on. I kind of sense there's some interesting thinking going on but can't quite identify what exactly.

Brett Cannon suggested a cause, perhaps somewhat cynically: "People want their problem solved, but don't care about others and their problems." Paul Moore had a different take, though; he thought that respondents wanted the packaging community to "continue focusing on the majority cases" and not spend time on supporting edge cases. "To me, that fits with the desire for an 'official' workflow and associated tool(s) - the edge cases would be outside the 'official' workflow, and hence not a priority." Somewhat in keeping with Cannon's comment, however, Moore said that the pip developers have found that "people really don't take kindly to being told their use case is unusual or unsupported". Dustin Ingram thought that by catering too much to the edge cases, everything suffers: it "leads to both the majority cases and edge cases being poorly supported".

At the end of December, Ralf Gommers announced the pypackaging-native site that is meant to collect "the key problems scientific, data science, ML/AI [machine learning/artificial intelligence] and other native-code-using projects & authors have with PyPI, wheels and Python packaging". But on the site he purposely does not offer "more than hints at solutions", since it is meant to foster discussion, larger projects, and Python Enhancement Proposals (PEPs). The site came up regularly in the strategy discussion as a fairly neutral recounting of the trials and tribulations associated with packaging, especially for scientific-Python projects (like NumPy and SciPy) where there are native dependencies, such as for compilers, build tools, and specialized libraries (e.g. for CUDA).

Strategy

In the strategy thread, Mohanan began by noting that several survey respondents had explicitly called for a single tool that would unify the various disparate efforts and eliminate the fragmentation. "In a nutshell, there are too many tools and users are not sure which ones to use." She asked if there is a path toward some kind of unification, and what it would entail.

It is not simply a matter of tools, Filipe Laíns said: "just unifying tools won't do any good, we need to do a full UX [user experience] analysis and then design something that fills the needs of users more naturally". pip already tries to be the single-tool solution, but it suffers from a need to continue supporting all of its historic use cases, which makes it hard to change. In addition, pip and other packaging tools all suffer from a lack of maintainer time.

Moore readily agreed, but noted that for some unified tool to be successful, "it will be at least as important to decide what workflows we won't support, as what we will". The lesson learned from pip is that trying to support every workflow just ends up in a mess, he said. But one problem is that respondents to the survey have likely been making an "implicit assumption that their preferred workflow will be supported" in whatever unified tool comes about. Ultimately, that is where the tension will lie.

But Moore is not so sure that a single, unified tool is worth the enormous amount of effort it would take to get there; it would be a substantial improvement for users, but it would take a lot of resources that might be better used in other ways. "Even as a funded project, with its own hired resources, it would consume a big chunk of attention from the packaging community [...]". He thinks there may be a better way:

One alternative that I think we should consider is continuing to work on the goal of splitting out the various "components" of packaging into reusable libraries. (In particular, I'd like to see an ecosystem where "pip is the only viable approach to installing packages" is no longer the self-evident truth that it is today.) Projects like installer, build, packaging and resolvelib are good examples of this. Using those libraries, and more like them, it will be a lot easier to build new workflow tools, any one of which has the potential to become the unified solution people want. It's definitely going to result in more complexity on the way to something simpler, and I'm pretty sure it's not what the survey respondents were imagining, but I feel that it might be a better trade off for the long term.

Cannon said that in his work on Python in Visual Studio Code (VS Code), he also encounters the problem of everyone expecting that their workflow will be supported. "People often don't have exposure to other workflows, so they innately think their workflow is 'normal', so if something doesn't work for them then it obviously is broken for everyone, right?" Changing to a more common workflow is not something that some people will be willing to do.

He agreed with Moore's suggested approach of reusable libraries for packaging; "If we can get everything backed by standards and then make using those standards easy by making sure they have backing libraries, then we can make it easier for people to experiment a bit as to what the proper UX should be for any front-end packaging tool." He also noted that the complexity in Python packaging is a double-edged sword: "While people bemoan Python's packaging story as being too complex, [its] flexibility is what helped it become the glue language of the programming world."

Conda

The elephant in the room, though, is the scientific-Python world and the conda ecosystem that goes along with it. H. Vetinari worried that it was not really being considered in the discussion. "As long as the ecosystem currently being served by conda cannot be folded back into 'the One True Way', we have not actually solved the schisms in python packaging (i.e. everyone can just use the same tool)." From Vetinari's perspective, either the monumental task of solving most of the problems described at pypackaging-native needs to happen in the unified tool, or to "define large parts of the data science ecosystem as out of scope". Neither of those is particularly palatable, but the PyPA has generally considered the needs of the scientific-Python world as being out of scope for its tools because of how much work it would take to handle the non-Python dependencies, Vetinari said.

But Moore made a distinction between "use cases" and "workflows"; he is adamantly opposed to labeling any particular use case as "out of scope". "I do expect that we may need to ask users to do things in certain ways in order to address their use cases [...]". Beyond that, though, conda has to install its own Python build, which means that anyone who wants to get, say, NumPy through conda, has to suddenly switch away from the Python they are already using; pip and others are meant to work with any (and every) Python installation:

The PyPA focus is on supporting the "standard" builds of Python (python.org, Linux distro builds, Windows Store, self-built interpreters, Homebrew, ...) Solutions that require the user to switch to a different Python build don't fit that remit. I don't think that "declare all of those Python builds as out of scope" has any more chance of being acceptable to the SC [steering council] than "declare a big chunk of the user base" does.

Vetinari suggested that looking at the problem differently might help: "For brainstorming about those solutions, I'd really like us not to think in terms of 'python.org installers' or 'Windows store builds' or 'conda', but in terms of anything that can satisfy the same relevant set of requirements." Moore agreed with that, but noted there is some amount of tension on the boundary between the PyPA and the core language (in the form of the steering council):

But I think that "how do people get Python" is part of the question about a unified solution. In much the same way that the "rust experience" isn't just cargo, it's also rustup. But just as we can't assume the SC would be OK with ignoring a chunk of the user base, I don't think we can assume the SC will accept dropping those ways of getting Python. We can ask, but we can't assume.

This is where the boundary between packaging (the PyPA) and the SC blurs. I personally think that the SC's "hands off" approach to packaging puts us in a bad place as soon as we get close to areas where the SC does have authority. Distutils was dropped from the stdlib, and the packaging community had to pick up the slack. We have to address binary compatibility, but we don't have control over the distribution channels for the interpreter. We provide tools for managing virtual environments, but we don't control the venv mechanism. We install libraries, but we don't control the way import hooks work. Etc.

Vetinari "fully agreed" that solving all of the problems will eventually require support at the language level, thus steering council involvement.

Unification

The discussion cue mentions "unification", Pradyun Gedam said, but everyone seems to have a different idea of exactly what would be unified. He came up with a list of six different aspects of the problem space that might be amenable to unification:

1. Unification of PyPI/conda models (i.e. the non-Python code dependency problem).
2. Unification of the consumer-facing tooling (i.e. consuming libraries).
3. Unification of the publisher-facing tooling (i.e. publishing libraries).
4. Unification of the workflow setups/tooling (i.e. organising files, running tests, linters, etc).
5. Unification/Consistency in the deployment processes (i.e. going from source code -> working application somewhere).
6. Unification/Consistency in `python` usage experience. (i.e. the rustup/pyenv aspects of this, which is absolutely a thing that affects users' "Python Packaging" experience (think pip != python -m pip, or python -m pip vs py -m pip, or python being on PATH but not pip etc)

That list, which others in the thread added on to, gives a nice, quick overview of where the complexity lies. These problems affect multiple users, including Python application users and developers, Python Package Index (PyPI) module developers, system administrators and operators, and so on. Each of those users has their own set of constraints (e.g. operating system, architecture), habits, and prejudices, but they are trying to share various global resources in a way that makes sense for them. It is perhaps not surprising that there is no unified solution that covers everyone. In fact it is hard to see that all of the things in that list can be fully united—at least in any kind of near-term time frame.

There is more to come on this Python-packaging journey for sure. Participants have been nothing if not verbose, with many good points being raised, as well as ideas for how to get where most everyone seems to want to go: a unified user experience for Python package installation. Something that just works for "everyone" and that every project can point to as its means of installation. Further unification could come later—if it ever comes about at all. Stay tuned ...