I HATE PYTHON [LWN.net]

I HATE PYTHON

Posted May 23, 2025 2:14 UTC (Fri) by geofft (subscriber, #59789)
In reply to: I HATE PYTHON by aragilar
Parent article: Recent disruptive changes from Setuptools

> it doesn't play well with the numerical ecosystem because of its assumptions (and so conda is used)

I've spent the last week at PyCon in a room with the Conda folks and other people working with numerical/scientific computing in Python, trying to both solve these problems for the PyPI-based ecosystem (pip, now also uv and poetry and others) and trying to do it in a way where we're compatible with the Conda ecosystem / where they can adopt the same designs. Honestly the PyPI ecosystem has been _pretty_ good over the last few years for this task, certainly far better than a decade ago when `pip install numpy` would always try to build from source. NVIDIA has resumed offering first-class support for the PyPI ecosystem in the last few years after very loudly quitting it in 2019 (https://medium.com/rapids-ai/rapids-0-7-release-drops-pip...). There's still a lot left to do, but we should be at the point (and are, in my experience) where most people, certainly most individuals, will have just as good a time with pip/uv/etc., so I am genuinely curious what sorts of things you personally find don't work.

> and it tries to hide the real need for design when managing an install (e.g. look at how it installs python, and all the minor things that break due to trying to make an almost-static install).

I'm one of the maintainers of those Python builds and I am absolutely open to bug reports, either here or on our bugtracker. Going after all these minor things is basically my top priority at the moment. Please let me know, I love having more test cases. :-)

to post comments

I HATE PYTHON

Posted May 23, 2025 10:46 UTC (Fri) by aragilar (subscriber, #122569) [Link] (2 responses)

I want to draw a distinction between the underlying standard/tools, which are mostly fine apart from some key spots I know are mostly being worked on (and which I am happy to use), and the higher level wrappers (of which poetry and uv are the poster children, and which I have an issue being described as working for 90%).

I'd classify things on a spectrum from papercuts to blockers. Things listed on https://gregoryszorc.com/docs/python-build-standalone/mai... are close to papercuts, in that they can be avoided by using a different build (or even building from source, plus they're actually documented), but when tools like uv default to using them implicitly, they start moving towards blockers (as people tend to go with the defaults, and so it becomes a matter of debugging layers of issues). Once certain IDEs start getting involved, now you're having to debug someone poorly configured machine and wasting hours or days fixing.

Then there are the parts of the ecosystem where pre-built binaries cannot be used (e.g. MPI), you end up to start adding flags like `--no-binary` and rebuilding half the ecosystem anyway because you need to link to a single BLAS. uv is really not designed for this (neither is poetry, they assume too much about wheels), and while you can (and I have) created your own private index to control what the installers see, using pip is far easier (and less likely to break).

Then there are the fun bits like the PyPI packages of Jupyter plugins just straight up not working (though the conda version works), due to wheels not being able to install to specific paths (because they need to integrate with the system they are installed on).

This all flows from the assumption that pre-built PyPI-associated binaries (with all their limitations) are a solution (rather than a workaround) to users needing to set up their machine properly. It's worth comparing this to R, where the installer on Windows and MacOS sets up a proper dev environment (with compilers), and so these kind of problems do not happen.

I HATE PYTHON

Posted May 24, 2025 22:47 UTC (Sat) by geofft (subscriber, #59789) [Link] (1 responses)

Thank you, this is very helpful and I appreciate you taking the time to write this up! A few of those quirks in that document are gone now (e.g., the musl build is now a dynamic binary, not a static one, so extensions work fine); I'll update it.

> Once certain IDEs start getting involved, now you're having to debug someone poorly configured machine and wasting hours or days fixing.

Just to clarify, do you mean that people use their IDEs to set up uv and thus python-build-standalone, and that makes things harder to debug because it's harder to figure out what their setup is? Or is there something specific to usage inside some IDE versus at the CLI?

> Then there are the parts of the ecosystem where pre-built binaries cannot be used (e.g. MPI), you end up to start adding flags like `--no-binary` and rebuilding half the ecosystem anyway because you need to link to a single BLAS.

This is definitely one of the things we spent time talking about last weekend (and it's also called out on https://pypackaging-native.github.io/key-issues/abi/ , an excellent website written by folks who work on the scientific Python ecosystem and would like the non-conda ecosystem to work well.) It's certainly a deficiency right now, but it's also a high priority for the scientific Python community as a whole. While I'm sure people have concrete examples, if you have a specific favorite prebuilt binary wheel or package name that doesn't work well or a combination of libraries that need to share a single BLAS, again, I love having more test cases. :)

> [...] uv is really not designed for this [...] using pip is far easier (and less likely to break).

uv and pip both install the same packages from the same places, and when you build from source with --no-binary you can link your exact system libraries when using either uv or pip, so I'm curious about this - is this because you can do venv --system-site-packages (or use pip without a venv in some fashion) and get some compiled Python packages from the OS, and uv doesn't support that? Or something else?

> Then there are the fun bits like the PyPI packages of Jupyter plugins just straight up not working (though the conda version works), due to wheels not being able to install to specific paths (because they need to integrate with the system they are installed on).

This one is new to me - do you have a specific plugin / example command that doesn't work? (I know there are tons of people using Jupyter with extensions in production using pip + venv, so I'm assuming it's not "any extension.")

> This all flows from the assumption that pre-built PyPI-associated binaries (with all their limitations) are a solution (rather than a workaround) to users needing to set up their machine properly. It's worth comparing this to R, where the installer on Windows and MacOS sets up a proper dev environment (with compilers), and so these kind of problems do not happen.

Yes, conditional on actually setting up a proper dev environment with compilers and library dependencies, which has its own host of problems. :) This was, after all, the status quo of pip until about a decade ago, and the fact that it was a very bad experience is both why conda came into existence and why pip started doing wheels. (Unless you mean that Python or uv or some other installer should install _its own_ compiler toolchain, independent of what is on the host, and build stuff from source, including C libraries like BLAS? That's an intriguing idea....) Conda supports R as well as Python, and my impression is that its R support is popular for largely the same reasons, in that building all your R packages from source is a difficult experience.

I HATE PYTHON

Posted May 28, 2025 9:55 UTC (Wed) by aragilar (subscriber, #122569) [Link]

> Just to clarify, do you mean that people use their IDEs to set up uv and thus python-build-standalone, and that makes things harder to debug because it's harder to figure out what their setup is? Or is there something specific to usage inside some IDE versus at the CLI?

Sorry, I meant that IDEs providing Python installs as well, and now you've got some unreproducible conda-uv-IDE frakenstate where python points somewhere, pip somewhere else and who knows what the `sys.path` will be. Too many tools want to be in charge (for legitimate reasons) and only pip is really set up to be a bit player where it plays nicely with others (though naturally it too can mess things up, as noted by PEP 668). Hence why I push people to understand what their tools are doing, and not mix them arbitrarily.

> This is definitely one of the things we spent time talking about last weekend (and it's also called out on https://pypackaging-native.github.io/key-issues/abi/ , an excellent website written by folks who work on the scientific Python ecosystem and would like the non-conda ecosystem to work well.)

Yeah, I've provided PRs to it previously :) To me the solution here was the "local" or "custom" wheel tag, which I suggested a while back, but didn't go anywhere.

> uv and pip both install the same packages from the same places

It's not the source that's the problem, it's that uv wanted a venv (at least when I tried it) whereas it should have used (in that particular case) the (purpose-built) conda environment.

The jupyter plugin (I don't recall it's name, I'd need to dig into my notes) in this case was to do VNC on the web, and my colleague (who is not a fan on conda) said he needed to use both conda and pip to get it working. When I had time I found out there was a separate binary that was being shelled out to (I think) to set up a websocket or something, but it would only be set up right with the conda install, because it needed some post-install stuff (I'm not sure why it couldn't have been configured correctly).

> Conda supports R as well as Python

My impression (based on the astronomers and biologists I know) is R from conda is used when people want to do R<->Python connections using rpy2 or similar, and most actual R users just use CRAN. This may not be accurate across the ecosystem though (I'm very much embedded in the university ecosystem).

> Yes, conditional on actually setting up a proper dev environment with compilers and library dependencies, which has its own host of problems.

While I don't think this is trivial, I do think we (the Python community) can do a lot better (and if we assume pre-built wheels can be used, then issues like the discussion around the future of setuptools get brushed aside). R on Windows (and possibly MacOS) installs it's own copy of GCC plus a bunch of other "standard" tools, and I've not heard complaints there (I have previously suggested Python on Windows should make it easy to install MSVC, but I got pushback against that). Conda definitely made things better on Windows, but at least when I tried using it to install packages without wheels, it apparently hadn't added the configuration that the conda gcc should work with the conda python, which seemed like a missed opportunity to me.

It also encourages groups to not use PyPI to upload their packages, as the assumption is that you should produce wheels (NVIDIA was one example, but I know of others that I'm not going to name otherwise someone will squat on their packages). I think having a index where all packages get registered, and a separate (downstream?) index which is a self-contained wheel archive would go a long way to solve this, and then you tell people that the high level tools only work with the wheel archive would be a better experience as then PyPI isn't forced to become a worse conda.