Python, packaging, and pip—again

Python packaging discussions seem like they often just go around and around, ending up where they started and recapitulating many of the points that have come up before. A recent discussion revolves around the pip package installer, as they often do. The central role that is occupied by pip has both good points and bad. There is a clear need for something that can install from the Python Package Index (PyPI) immediately after Python itself is installed. Whether there should be additional features, including project management, that come "inside the box", as well, is much less clear—not unlike the question of which project management "style" should be chosen.

Early in 2023, we tried to cover a wide-ranging discussion regarding Python packaging, its tools, the governance of the Python Packaging Authority (PyPA), and more. The "authority" part of the name, which originally was meant as something of a joke, is not entirely accurate; there are efforts underway to update (or replace) PEP 609 ("Python Packaging Authority (PyPA) Governance") with a new "packaging council" and a clearer mandate. Meanwhile, there has been some progress in the packaging world since our article series, but it seems likely that none of the participants are completely happy with its extent. There is still a huge amount to do.

There are some PEPs that are being discussed and worked on in that area, including PEP 735 ("Dependency Groups in pyproject.toml"), which is authored by Stephen Rosen and sponsored by Brett Cannon; it will be accepted or rejected by the PEP delegate, Paul Moore. It specifies a way to store package dependencies in a pyproject.toml file that is analogous to the requirements.txt file that pip uses today. The TOML file is meant to be used by various other tools, such as IDEs and program launchers, in ways that are well beyond what the current format can provide. In addition, the requirements.txt file format is not standardized, but trying to do so for now would have lots of backward-compatibility concerns.

It is against that backdrop, and apparently under the assumption that the PEP will be accepted, that Luca Baggi asked if pip would be changed to add dependencies to pyproject.toml. Moore, who is also a pip maintainer, noted that pip "isn't a project manager, it's an installer"; there are other tools for that job. There was a bit of discussion about what it might look like to add the feature, but "sinoroc" agreed with Moore that it is not in pip's scope: "pip does not deal with modifying this kind of data, pip is only a consumer of such data".

Pip maintainer Pradyun Gedam said that he would like to see pip expand into some "additional parts of the workflow" but that there simply are not enough developers to handle the existing load; "if we can't keep up with existing feature [maintenance], it gets worse when we add more features". That led Damian Shaw to suggest that it may be time to "consider bundling a tool with the CPython installer that does support this kind of package/environment manager workflow", though he recognized there would be a high bar for any project to cross to get bundled that way.

Moore cautioned against extending pip further, noting that many of the existing workflow tools use pip under the hood, so any changes would need to take that into account. Beyond that, for a new workflow tool to get adopted, a new PEP, along the lines of PEP 453 ("Explicit bootstrapping of pip in Python installations"), would need to be written and get approved. Things have changed in the ten years since that PEP, so it makes sense to consider a new path, but:

[...] I doubt there's a realistic possibility of anyone ([packaging] council, PyPA or community) being able to come to a decision on which workflow tool is going to be blessed as the "official answer". We've had way too many unproductive discussions on this in the recent past for me to think there's anything even remotely like a consensus.

Sinoroc suggested that pipx, which automatically installs command-line applications from PyPI into virtual environments and adds them to the user's path, might be a better choice for bootstrapping these days. Moore pointed out that pipx uses pip, so it does not remove the need for pip as part of the initial Python install. Both pip and pipx are experimentally available as standalone zipapp applications, which might mean that Python could stop shipping pip, but even then it is still "a non-trivial process getting from 'install Python' to 'ready to go'", he said.

When pip was added to the install, it was done to provide a means for users to install the workflow tool of their choice (as well as other packages of interest), it was not meant to be the be-all and end-all, Moore said. But Shaw thinks that users see it differently: "With pip bundled with the official installer and pypi.org displaying pip install package-name on every package page, the impression is that pip is the blessed official tool to use for managing 3rd party packages, not merely a simple way to bootstrap to your favorite tool of choice." If there is sufficient interest by both CPython developers and those of a project-management tool, he said, it may be worth looking at bundling such a tool to supplant pip.

But some of the difficulties that pip struggles under, such as needing to adhere to the CPython release cycle and to vendor all of its dependencies, would also affect any other tool that gets shipped. It is hard to see the developers of other tools being willing to do so, Moore said. In addition, even if there were candidates, as a core developer he does not "have the appetite to get CPython sucked into the 'which tool is best' controversies this would involve". Perhaps unfortunately, though, making that choice is exactly what last year's Python user survey showed was most desired, Brendan Barnwell said. Not making a choice is:

[...] ultimately incompatible with the goal of solving the fragmentation problem of Python packaging tooling. As long as the tools that are bundled with Python leave out large chunks of functionality that people want, while the ones that aren't bundled compete and none is clearly endorsed by Python, users will feel confused and irritated. It doesn't exactly have to be choosing the "best" one but I think a choice does need to be communicated about which tool(s) are recommended.

But core developer Steve Dower concurred with Moore; he also does not want to get pulled into trying to make that controversial choice. Furthermore, he pointed out that the core developers are not particularly sympathetic to packaging concerns:

And I've raised packaging-like questions with the broader core developer group before - the responses tend to range between "what's packaging" to "I don't care about packaging" to "they can do whatever they want". I'm afraid the most support from the core team comes from those of us who participate in this category, which is very few of us.

That division is not healthy, Moore noted, especially given that the responsibility for ensuring access to PyPI should be shared between the PyPA and core-development team. Since that is not happening, solutions to some of the problems that users complain about cannot come about:

Other solutions are possible. Making zipapps work better, and shipping a pip (or pipx) zipapp with core Python might be an option, for example. But only if the core devs take a more active interest in the deployment side of the developer experience.

That's pretty much where things stand; there was a bit more discussion, which continues as of this writing, about pip and its central—privileged—role in the ecosystem. That is, of course, much like many of the other, interminable discussions that are ongoing in the packaging category of the Python discussion forum. Incremental progress is seemingly being made, but the main problem identified by the user survey—and huge number of complaints before that—remains. It is not at all clear what, if anything, will break the logjam.