PEP517 [LWN.net]

PEP517

Posted May 22, 2025 5:14 UTC (Thu) by geofft (subscriber, #59789)
In reply to: PEP517 by sam_c
Parent article: Recent disruptive changes from Setuptools

Can you explain what you mean by "Pythonic Makefiles"? I've been in this space (perhaps too deep in this space) for a while and I don't really follow - what is the thing you are trying to do that cannot be done?

My first reading of the comment linked from the comment thread you linked (https://github.com/pypa/packaging-problems/issues/576#iss...) is that while `setup.py install` is happy to put files in an OS-owned directory, pip is not, and for reasons like this, "distro maintainers hate pip because it does NOT do what distro maintainer want." If I'm reading this right, this is exactly backwards: pip's refusal to put files in OS-owned directories started out as a patch from distro maintainers overriding the behavior of upstream pip, because they specifically wanted pip to stop overwriting OS-owned files. That change slowly got upstreamed in a form that hopefully let the OS maintainers drop their patches.

A little later in that thread it's mentioned that "python setup.py install" is used to install e.g. systemd units in /lib. The reason setup.py was able to do this is that it was just an ordinary Python script - but that gives you a way forward; the ability to run ordinary Python scripts has not been deprecated, no matter how much setuptools being invoked through an ordinary script has been deprecated. Is there a path forward where your setup.py script does not use setuptools (which is a Python-packaging-specific tool) but uses generic Python functionality like open("/lib/...", "w")?

(There may be a need for a library that handles some of the details of doing this, like destdir; I think one could write such a library and have it be far simpler and far more useful for this purpose than setuptools.)

Or am I misunderstanding the problem?

to post comments

PEP517

Posted May 22, 2025 12:56 UTC (Thu) by zahlman (guest, #175387) [Link] (1 responses)

> My first reading of the comment linked from the comment thread you linked (https://github.com/pypa/packaging-problems/issues/576#iss...) is that while `setup.py install` is happy to put files in an OS-owned directory, pip is not, and for reasons like this, "distro maintainers hate pip because it does NOT do what distro maintainer want." If I'm reading this right, this is exactly backwards: pip's refusal to put files in OS-owned directories started out as a patch from distro maintainers overriding the behavior of upstream pip, because they specifically wanted pip to stop overwriting OS-owned files. That change slowly got upstreamed in a form that hopefully let the OS maintainers drop their patches.

It's... a lot more complicated than this, unfortunately. First, to clarify a few things about your description:

Pip originally was happy to "put files" in *certain* OS-owned directories. Specifically, it will put them within any *Python environment*. If you tell it to install for the system environment, and give it sudo rights, it would obediently write files to (typically) `/usr/lib/pythonX.Y/site-packages`. Without sudo, it could still write to a user-specific directory; if you run the system Python as a regular user, Python can still `import` from there.

Eventually, distros realized that not only do those system-level installs cause obvious problems for the system package manager, but even a user-level install can cause problems for scripts that come with the system. This is especially problematic when you have a package manager (or interface thereto) *written in Python* that still does useful things when run without sudo. Hence https://peps.python.org/pep-0668/ (I think that's the "upstreamed patch" you have in mind).

Notably, though, this change was *entirely orthogonal* to the transition to the new PEP 517 system.

Separately, the wheel format can *only* put files *into the current Python environment* - as defined by the `sysconfig` standard library module. (By "current" I mean the one that pip is running under.) It can't ordinarily put them in /etc - or ~/.config , even though that isn't system-owned.

However, pip *can* effectively put files in those places (assuming filesystem permissions) if it installs from an sdist and builds from source - because that will request the build backend to do whatever is necessary to make a wheel, including running arbitrary Python code from the package. In the case of Setuptools, it's effectively equivalent to `setup.py install`. (At least the last I checked, pip run with sudo *will not drop permissions* when it invokes the build backend!) But this is not playing nice with either the Python ecosystem *or* with the Linux distro's package management.

----

But the comment is talking about something completely different. It's not about end users causing problems for the distro maintainer by using pip to install a third-party package. It's about the distro maintainer *trying to use* pip to *create a system* package.

They might try to do this by installing the code into a staging folder with pip, then using another tool to pack that tree into .deb or .rpm or whatever. As described above, theoretically the pip/Setuptools combination could be wrangled to write files to /path/to/staging/etc or whatever is needed, even while the Python environment is nominally rooted at /path/to/staging/usr. But it appears that this doesn't work with explicitly provided Setuptools features - you'd have to leverage the fact that you can write arbitrary code in `setup.py`, and do your own file I/O etc. And that code might not play nice with *other* packaging flows....

(Also, pip doesn't really know how to install cross-environment. It figures out installation paths based on the Python executable that's being used to run it. Newer versions have a `--python` argument to try to fix this - but it works by tracking down the other Python executable and *spawning it to re-run the pip code*.)

PEP517

Posted May 23, 2025 1:37 UTC (Fri) by sam_c (subscriber, #139836) [Link]

I'm not talking about using pip here (just invoking `setup.py install` or whatever will have the same issue) but rather any software where upstream expect data files to be installed to standard locations using their own setup.py, and all of their documentation refers to it and so on, which will stop working. The Python community hasn't done a great job of communicating this change.

But I'll reply more to the other comment rather than duplicating it.

PEP517

Posted May 23, 2025 1:48 UTC (Fri) by sam_c (subscriber, #139836) [Link] (1 responses)

> Can you explain what you mean by "Pythonic Makefiles"?

There's a lot of software that used `setup.py` as an entrypoint and relied on it being able to install to arbitrary locations on disk, often using it only because the software was written in Python and they felt it was natural that the build system also be in Python, despite it not needing to be.

It would also often (as the article covers) handle the testsuite but other administrative tasks, as well as "more elaborate" installation.

> The reason setup.py was able to do this is that it was just an ordinary Python script ...

Precisely.

> Is there a path forward where your setup.py script does not use setuptools (which is a Python-packaging-specific tool) but uses generic Python functionality like open("/lib/...", "w")?

Sure, that can be done - but if the software is installed via a wheel, that's not going to work, because there's no way to map extra data file locations in the wheel to real locations on the system. And none of the authors of these pieces of software relying on this functionality working ever seem to be aware of the problem.

i.e. All of it is solvable, it's just that it's a lot of work across a lot of different projects, and I don't think many of them are aware of the need to even do it. The software relying on this generally doesn't want to be installed as a wheel (it's usually "written in Python, not a library that should be imported"), they've just been piggybacking on setuptools and the setup.py entrypoint for years.

PEP517

Posted May 29, 2025 18:33 UTC (Thu) by zahlman (guest, #175387) [Link]

> And none of the authors of these pieces of software relying on this functionality working ever seem to be aware of the problem.

> i.e. All of it is solvable, it's just that it's a lot of work across a lot of different projects, and I don't think many of them are aware of the need to even do it.

My experience of discussion in Python packaging circles has been: people are very aware that Python code gets used and distributed in a wide variety of ways beyond the standard packaging system. But the people who try to "solve packaging" have very little idea of how to track down major projects that work this way. It often seems like the best they can do is to trawl through PyPI looking for sdists that will break in the future and send emails to whatever maintainer email addresses are listed in the metadata. If they don't publish an sdist then there's no clear way to figure out what their needs are to service them better.

A bit of speculation as well as *personal* investigation from Paul Moore (of the pip team) led to his submission of PEP 722, subsequently rejected in favour of PEP 723 by Ofek Lev (author of Hatch). That makes life easier for people who write single-file scripts with dependencies. But there are still huge unanswered questions about several other kinds of projects, e.g. large monorepos.

I had a bunch of post IDs saved on the Python Discourse forums about this general topic, but I lost those when I got kicked out (https://zahlman.github.io/posts/2024/07/31/an-open-letter... ; you may also remember me from https://lwn.net/Articles/988894/) and my local bookmark collection is quite large and hard to organize.

If you personally know people (or projects) that use deprecated workflows around setup.py, I'd heavily encourage you to warn them - and link to https://blog.ganssle.io/articles/2021/10/setup-py-depreca... . If you know projects that use only Python but publish only a "legacy" sdist instead of wheels, https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pu... is a solid reference.

If they otherwise make inappropriate use of setup.py (e.g. to convey metadata that could be in a pyproject.toml), probably the best way to warn them is to point out what happened to Requests with Setuptools version 78 (and 72): twice in less than a year, one of the most popular projects out there, written in pure Python, had its packaging break twice in easily avoidable and completely unnecessary ways. Packaging is part of maintenance, too.

But for those who don't publish to PyPI and whose projects aren't a dependencies for anything else, there might not actually *be* a need to do anything - except to stop upgrading local installations of Setuptools.