LWN: Comments on "A survey of the Python packaging landscape"

A survey of the Python packaging landscape

callegar — Thu, 26 Jan 2023 13:35:23 +0000

> [...] there can be conflicting dependency needs between different packages or applications. If application A needs version 1 of a dependency, but application B needs version 2, only one can be satisfied because only a single version of a package can be active for a particular Python instance.

> It is not possible to specify that the import statement in A picks up a different version than the one that B picks up.

Regardless of solutions to other more difficult problems, such as the management of "native" dependencies, the point above keeps puzzling me, since it seems addressable and still does not seem to enjoy large consideration. On one hand it gets mentioned only with regards to pip, but it is in fact common to all the mentioned packaging options (pip, setuptools, or conda) and it does not go away (not completely at least) even if you use virtual environments.

The main problem is that it gives a huge power to a single author to make a whole ecosystem lag. If you have a project and in your project there is a single dependency X whose author is not prompt (or even not willing) to update its own dependency requirements then that single X is going to prevent you from using up to date versions of all its dependencies that may be themselves dependency of your project. In turn these older packages will impose limits on how new their own dependencies can be, until all your project ends up having to rely on old stuff, giving up on the opportunity to use new features. Using a virtual environment here does not help.

A survey of the Python packaging landscape

sammythesnake — Thu, 26 Jan 2023 11:46:31 +0000

I also had an unconscious assumption about why it was called that. "Don't reinvent the wheel" - don't recompile what somebody else has already compiled...

A survey of the Python packaging landscape

milesrout — Thu, 26 Jan 2023 09:07:40 +0000

It's far from meaningless. How do you reach that conclusion? The solution is really very simple and is covered in the first few paragraphs of this article.

A survey of the Python packaging landscape

njs — Thu, 26 Jan 2023 06:20:04 +0000

Heh. You'll probably find this thread interesting, then: https://discuss.python.org/t/announce-pybi-and-posy/23021

A survey of the Python packaging landscape

CChittleborough — Sun, 22 Jan 2023 08:25:34 +0000

I always assumed the old hacker jargon term “wheel” had something to do with it. Hmm...

A survey of the Python packaging landscape

NYKevin — Fri, 20 Jan 2023 00:32:56 +0000

Well, there is pyproject.toml, but that only has build requirements, not install requirements. There's also setup.cfg, but as far as I can see, that's only providing *defaults* that can be overridden in setup.py.

I think the "standard" solution to this problem is pip freeze, but while this does solve the "I have a setup that works, now how do I replicate it?" problem, it does *not* solve the "I have a setup that works, but it's outdated, now how do I build a newer setup that also works?" problem, and that's arguably the more important problem to solve.

A survey of the Python packaging landscape

iabervon — Thu, 19 Jan 2023 23:57:09 +0000

It's the common pattern where the first attempt is clunky but works, and the next N attempts are not clunky but don't work, and they eventually reach something that can express everything necessary nicely (but you never know whether they're there yet). They eventually deprecate the first thing when they stop finding problems with each newest thing; meanwhile, people just don't migrate off of the first thing, which has always worked and never been deprecated.

For what it's worth, objects that fit the duck type of pathlib.Path as far as any operations you'd want to attempt are really nice to work with, although I wouldn't be annotating any variables with the current return type of importlib.resources.files().

A survey of the Python packaging landscape

NYKevin — Thu, 19 Jan 2023 23:27:43 +0000

Aha, I found it: https://github.com/python/cpython/issues/89838

> FYI the long-term plan is to deprecate pkgutil, so I would use newer APIs as provided by importlib.

So it's to-be-deprecated but not actually deprecated yet. And the churn continues...

A survey of the Python packaging landscape

acarno — Thu, 19 Jan 2023 18:33:25 +0000

I wonder if there will come a day when Randall decides to go entirely meta and creates a comic that references the fact that there's _always_ a relevant xkcd.

Thanks for sharing! :)

A survey of the Python packaging landscape

ehiggs — Thu, 19 Jan 2023 12:48:18 +0000

The fundamental issue as far as I can see is that Python tooling cannot be written in Python because you end up with a bootstrap problem: python depending on [another] Python at runtime. And often people get confused and end up bootstrapping using a single Python - which gives a whole host of other potential problems when the package management stuff depends on a different version of a library than the application.

To wit: pyenv is written in shell scripts and while it's a bit janky, it works very well because it doesn't suffer from this bootstrap problem.

I would recommend Rust for this but someone will just end up writing plugins in PyO3 and reintroduce the boostrap problem again - so it would need something with no (or unpopular) Python bindings. The obvious conclusion: Python tooling should be written in Go.

A survey of the Python packaging landscape

zorro — Thu, 19 Jan 2023 12:05:37 +0000

You were right, it does not work because of the hardcoded absolute paths.

I ended up following the Windows embeddable package approach described in https://fpim.github.io/posts/setting-up-python-windows-em.... This gave me exactly what I wanted. No system-level installation of Python, all dependencies contained in a single folder (including Python itself), an no hardcoded absolute paths. Better than using venv, AFAIC.

A survey of the Python packaging landscape

kleptog — Thu, 19 Jan 2023 09:20:53 +0000

> 4. A venv is basically the world's leakiest userspace container, which is to say that venvs are really not containers at all, but just a bunch of scripts and environment variables that ask Python nicely to pretend that it's running in an isolated environment. Deploying a venv in production "as is" is questionable at best. You're probably better off using real containers instead, at which point the venv is redundant.

Yeah, venvs are fine for development, but for production we use a two stage docker build process that first takes the list of dependencies and produces a bunch of wheels, and then builds a clean docker image with all the wheels installed globally.

The biggest weakness IMHO of pip is that PyPI has no metadata file, so it has to actually download the package (and possibly executes the setup.py) to determine the dependencies. So the package resolver, if it gets into an unresolvable situation, can keep downloading older and older versions in an attempt to make it work. One time we got strange errors from the buildbot and figured it had downloaded 3 year old versions of various packages which somehow worked (and had taken all night to do it).

A survey of the Python packaging landscape

marcH — Thu, 19 Jan 2023 05:11:57 +0000

> That is typically done using the pip package installer, which will either install the package in a site-wide location or somewhere user-specific depending on whether it was invoked with privileges. pip will also download any needed dependencies, but it only looks for those dependencies at PyPI, since it has no knowledge of the distribution package manager. That can lead to pip installing a dependency that actually is available in the distribution's repository,

My super ugly way to work around this issue has been to go and try "apt-get/dnf install python[3]-X" where X is the pip name. It works about 80% of the time.

pip install finally added some "--dry-run" flag that helps with that.

It would be nice for Linux distros to make that mapping easier. Maybe packages that don't follow that naming convention could have a "python-X" alias when possible. It does not have to work 100% of the time, going from 80% to 95% would already help a lot.

A survey of the Python packaging landscape

ranger207 — Thu, 19 Jan 2023 03:22:41 +0000

There is, in fact, an even more relevant xkcd:
https://xkcd.com/1987/

A survey of the Python packaging landscape

NYKevin — Thu, 19 Jan 2023 01:38:32 +0000

Part of the problem is that there has been significant rearranging of the deck chairs for "how do I avoid tying everything to files and directories that really exist in the filesystem?" In the bad old days, you basically couldn't, because a package might embed some data file inside of itself, and try to consult it at runtime (using __file__ to figure out where it was). There was nothing forbidding this, so such packages were simply broken if you tried to install them as anything other than a bunch of actual files and directories that were really mounted somewhere in the filesystem. Over time, various APIs emerged that would "do that, but also DTRT if the package isn't materialized on disk," and were then deprecated in favor of the next API. I'm aware of at least the following:

* pkgutil.get_data(package, path)
* pkg_resources, generally
* importlib.resources.read_text() etc.
* importlib.resources.files(package) and all of the various methods you can call on its return value

Of these, pkgutil and importlib.resources.files() are not explicitly marked deprecated, and both are in the stdlib, so obviously we're supposed to... flip a coin? Well, pkgutil has a much simpler API, so it's probably the one that people will use in practice, but importlib.resources is far newer (despite already having a significant portion of its API surface marked as deprecated), so it's probably the one that they *want* people to use.

A survey of the Python packaging landscape

pj — Thu, 19 Jan 2023 01:34:49 +0000

My python management:
1. python versioning via nix (as a non-main package manager on my ubuntu system)
2. python dependency version pinning via pip-tools

but then I still use setup.py and bdist_wheel for packaging.

A survey of the Python packaging landscape

iabervon — Wed, 18 Jan 2023 20:45:03 +0000

I think what's missing is something built into Python that's like Gentoo's python-exec (but more extensive): an executable that encodes an environment and runs a script in that environment. I think venv is kind of like having the dynamic linker support library paths and sonames (more than like containers), but there's no data format for representing the linker instructions and being able to stick the platform's executable glue on it. There are a bunch of good components for having a script running with the correct versions of packages ready to load, but there's nothing standard for making an executable that starts a script in exactly the state that's going to work and not some other state.

A survey of the Python packaging landscape

NYKevin — Wed, 18 Jan 2023 18:11:34 +0000

I think we are using the word "isolation" to mean different things. To my mind, a venv provides *some* isolation, because you can have more than one, and install different packages (or different versions of the same package) into each venv. A container provides that benefit, but it also isolates non-Python things such as libc.

I am not familiar with your definition of "isolation."

A survey of the Python packaging landscape

mathstuf — Wed, 18 Jan 2023 16:32:09 +0000

AFAIK, venvs bake in their path (e.g., shebang lines at the top of scripts), so relocatability isn't a property they have.

FWIW, this means that "activating" a venv is never actually necessary. At least I never do it. Instead, I just run the `pip`, `python`, or any other tool directly from the venv directory and everything Just Works™.

A survey of the Python packaging landscape

Wol — Wed, 18 Jan 2023 15:26:18 +0000

You probably are.

Okay, I run gentoo, and I installed Sphinx as per the instructions (use venv, I think it said). I'm not a Python user, I don't understand it, and very quickly all hell broke lose.

At which point, I realised that gentoo has a Sphinx package, I installed that instead, and now everything appears to work fine.

The problem, as it appears to me, is that if you are dealing with a clueless luser (which is me and Python), venv is both hard to automate, and hard to explain. BOOM! Now I leave everything to emerge, it's not a problem. "It Just Works (tm)".

Cheers,
Wol

A survey of the Python packaging landscape

SnoopJ — Wed, 18 Jan 2023 14:27:10 +0000

Uptake of PEP 668 by distributions would probably alleviate some of the headaches, but not really the ones being addressed in Chris Warrick's post.

Mostly it would prevent users from damaging their distribution copy by giving distributions with strong opinions about How Python Should Be the option to tell pip (or other package managers) that the site is someone else's problem.

A survey of the Python packaging landscape

jhoblitt — Wed, 18 Jan 2023 14:18:24 +0000

I believe it is progress in that there seems to finally be some maintainer level exceptance of the severity of the python packaging problems.

The scientific computing world has heavily embraced conda, which solves somes of the issues but it is still messy to try use two tools at the same time with conflicting dependency chains.

A survey of the Python packaging landscape

rahulsundaram — Wed, 18 Jan 2023 14:15:51 +0000

> Thanks to this article I now know what a "wheel" is in the Python context (I think), but I'm still not clear why they are called that. How do you get from "pre-compiled Python extension" to "wheel"?

Interesting question. Apparently it is a bit of an inside joke. Refer to

https://discuss.python.org/t/where-the-name-wheel-comes-f...

More about the wheel format in https://peps.python.org/pep-0427/

A survey of the Python packaging landscape

jkingweb — Wed, 18 Jan 2023 13:56:38 +0000

Thanks to this article I now know what a "wheel" is in the Python context (I think), but I'm still not clear why they are called that. How do you get from "pre-compiled Python extension" to "wheel"?

A survey of the Python packaging landscape

zorro — Wed, 18 Jan 2023 13:45:20 +0000

> You're probably better off using real containers instead, at which point the venv is redundant.

I'm on Windows and what I want to do is ZIP the contents of my venv folder and use the resulting ZIP file as a distribution package for my application. My hope is that I can simply unzip the package on any other Windows computer and run the application there without needing to install or cause interference with anything, including Python itself. Am I too optimistic?

A survey of the Python packaging landscape

cyperpunks — Wed, 18 Jan 2023 09:46:29 +0000

Due to this mess, it's meaningless to recommended to use Python for projects that have multiple platforms as targets.

It's often more work to deploy a Python application than maintaining the actual Python code.

A survey of the Python packaging landscape

LtWorf — Wed, 18 Jan 2023 06:52:27 +0000

> A venv is basically the world's leakiest userspace container

I don't understnad. A venv is not there to provide isolation. It's there to basically provide a path to use to load libraries. That's all. It's a completely different thing than a container.

Also, most containers, by default, do not provide any isolation.

A survey of the Python packaging landscape

k8to — Wed, 18 Jan 2023 06:36:37 +0000

I have wasted so much 'devops' time trying to figure out why certain python encryption modules just didn't build on some environments. For some reason it seems much more fragile than building things from source in C ever used to be. Maybe just because the feedback experience is bad, and with modern 'install every time' approaches, it fails in the middle of automation rather than interactively.

It feels like there probably is some middle path of cooperation between the distribution and python that could make this less brittle for the common culrpits. But maybe not.

A survey of the Python packaging landscape

NYKevin — Wed, 18 Jan 2023 05:03:50 +0000

It depends on what you mean by "Just Do It." For certain values of "Just Do It," pip and venv are already in the box and can already Just Do It.

Having said that, they're far from perfect. Problems with pip and venv:

0. You have to create the venv. It doesn't just magically spring into existence when you start installing packages, or prompt you to create it, or something of that nature. (It would probably still be good to have a way to do manual venv creation, but the fact that it's not automatic means that less experienced users will not realize they need to do it in the first place.)
1. pip still wants to install into the global site-packages dir by default (i.e. if no flags are passed and a venv is not active). This is bad. pip should not do that, at least not without some kind of --i-know-what-im-doing flag. It baffles me that the Python folks are dragging their feet on this one - I could've told you that five years ago. To be fair, this is not a totally unreasonable thing to do on platforms with no package manager (i.e. Windows), but even so it's still not ideal for development purposes, and end users probably should not be using pip anyway.
2. pip's handling of dependency management and versioning is rather minimal, and possibly inadequate for some situations.
3. More generally, pip and venv are slightly less abstract than what some users (i.e. academics and other non-software-engineers) would like them to be. They are not opinionated enough, and where they do have opinions, those opinions are sometimes the "wrong" ones (for the scientific use case in particular).
4. A venv is basically the world's leakiest userspace container, which is to say that venvs are really not containers at all, but just a bunch of scripts and environment variables that ask Python nicely to pretend that it's running in an isolated environment. Deploying a venv in production "as is" is questionable at best. You're probably better off using real containers instead, at which point the venv is redundant.
5. If you don't have a C compiler all set up and ready for pip to invoke, then some packages simply can't be installed (by pip), because they are not distributed as compiled binaries and/or because distributing binaries on Linux is generally awkward.
6. Several Linux distributions, notably including Debian and its derivatives, have intentionally sabotaged pip and (indirectly) venv to prevent problem (1) from breaking people's systems. On those systems, you have to install pip separately, even though upstream distributes it as part of the language. This leads to confusion and the perception of pip and venv as "not ready for general use" (e.g. "the tutorial said to run python -m venv env, but when I do that, it gives this weird error about pip and says it isn't going to work!").
7. There are probably at least a few domain-specific problems that I'm unaware of, but the above is just what I could think of off the top of my head.

A survey of the Python packaging landscape

PengZheng — Wed, 18 Jan 2023 05:00:24 +0000

> But wheels are highly specialized for the operating system, architecture, C library, and other characteristics of the environment, which leads to a huge matrix of possibilities. PyPI relies on all of the individual projects to build "all" of the wheels that users might need, which distributes the burden, but also means that there are gaps for projects that do not have the resources of a large build farm to create wheels. Beyond that, some Python applications and libraries, especially in the scientific-computing world, depend on external libraries of various sorts, which are also needed on target systems.

I think it is plausible to combine C/C++ package manager (yes, I mean Conan, which is also written in python) with pip and virtualenv, and let Conan deal with these external libraries.

A survey of the Python packaging landscape

tialaramex — Wed, 18 Jan 2023 01:47:21 +0000

Surely shipping it in the box with Python would be almost exactly like Cargo ?

If there's a tool that's right in the box and is at least pretty good for beginners, then why would they use something else at first while learning? And once all the beginners use it, you're starting to build up a population who want this tool even if maybe an alternative could be better for some things, they've "grown up" with the bundled tool and that's what they want improved.

If Python shipped with a tool that can "Just Do It" when it comes to using modules like Requests, whatever the popular way is to talk to Postgressql or MariaDB or whatever, maybe a couple of popular options for Machine Learning or AWS S3 or other stuff some people need a lot but others don't, that could be huge even if it can't install Bob's Third Mediocre JSON parser v0.1.2 or whatever.

A survey of the Python packaging landscape

hazmat — Wed, 18 Jan 2023 01:08:39 +0000

fwiw, i'm currently using the same setup, imho its pretty decent given the state of the ecosystem, i still find myself occasionally reaching to pip to install a one off tool (pipdeptree, etc ie in venv tree) or pipx for system tools. i wish some of this discussion had actually discussed supply chain security aspects, the tuf pypi integration is forever long :/

A survey of the Python packaging landscape

acarno — Wed, 18 Jan 2023 00:53:55 +0000

Given the current fragmentation, I suspect any attempt by PyPA to unify the current ecosystem would look akin to https://xkcd.com/927/ . And with no language-defined means to enforce it (such as a cargo-like solution), I think it would take a major rev of the language (e.g., the mythical "Python 4") to do so.

A survey of the Python packaging landscape

ryanduve — Tue, 17 Jan 2023 23:51:33 +0000

Here's my current Python environment management:

1. Pyenv, to manage installation of different Python versions. From what I can tell, it's the equivalent of building Python from source, but it takes care of the common build flags and does free namespace management. I used to use the system package manager for this, but Pyenv supports every minor/patch version of Python I've thrown at it, which is really useful.

2. Poetry, to manage dependencies for each project and generate a lock file, which nails down transitive dependency versions.

System-level dependencies are usually just sort of agreed upon within the working group, which works because they rarely need formal specification for our work. If that's not the case, we replace Pyenv with a Docker image and specify whatever is needed on top from there.

It's honestly not ideal. There's a ton of `poetry env remove ...` voodoo that has to be done from time-to-time, and it's always a headache determining which virtual environment is active on someone else's IDE. I hope the PyPA comes up with a single replacement for both the above, packaged with Python's standard library.