(There's no need for all-caps vitriol.) [LWN.net]

(There's no need for all-caps vitriol.)

Posted May 22, 2025 13:22 UTC (Thu) by zahlman (guest, #175387)
In reply to: I HATE PYTHON by dskoll
Parent article: Recent disruptive changes from Setuptools

> Perl happily lets you install both OS packages and packages from CPAN and manages to keep them in separate directories. Why can't Python do this?

Python doesn't do any of the installation. The entire packaging system, and package ecosystem, is kept at arm's length from core language development.

Installers like pip can and do install packages from PyPI separately from Python packages provided by a Linux distro. Not only that, but it has separate system-level and user-level installations. The system package manager will use (typically) /usr/lib/pythonX.Y/dist-packages . When installing for the system Python, pip will use /usr/lib/pythonX.Y/site-packages when run with sudo, and ~/.local/lib/pythonX.Y/site-packages otherwise.

The reason your distro now insists on using virtual environments is because *this folder arrangement is still problematic*. What happens is that a user-installed module shadows a system-provided one that a system script is trying to use. The ~/.local packages are only visible when running that system script as the user, but *this still causes enough problems* that distros decided "just don't give sudo rights to pip" isn't enough protection.

I genuinely don't understand why "easily installing *system-wide* packages" is such a priority. But you *can* do this with venvs (with sudo of course) - for example: make a venv somewhere within /opt and install there - and if it's an "application" with a driver script, maybe even symlink it from /usr/local/bin. In fact, newer versions of pipx (https://pipx.pypa.io/stable/) wrap this exact process for applications (using a shared vendored copy of pip). I have more about this in a personal blog article here: https://zahlman.github.io/posts/2025/01/07/python-packagi...

It may help to think of the virtual environment as effectively a separate installation of Python - because it's close to that, just with symlinks to the "base" Python executable and special access to the standard library, instead of copying it all. If you get annoyed with remembering to "activate" a virtual environment, keep in mind that this actually does very little. Generally you can get by with just giving the path to the virtual environment's `python` symlink.

to post comments

(There's no need for all-caps vitriol.)

Posted May 22, 2025 17:07 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (1 responses)

Fun fact: If a distro wants to support --user, it's not outrageously hard to do so. You just have to make sure that any system packages that require Python set the -s flag (on /usr/bin/python) or the PYTHONNOUSERSITE environment variable (to any value), and then whatever is in the user's home directory is silently ignored. I imagine there are varying levels of difficulty here:

* For stuff managed by whatever init you are using (let's not start a systemd holy war here), this should be trivial.
* For stuff managed by systemd --user (or any equivalent, as well as session-related stuff like GDM/KDM/login/getty/etc.), this should also be trivial.
* That just leaves binaries that the user runs. If it's run through the desktop shell, this can be configured in the .desktop file, and if it's run from the command line, it can be configured in the shebang line... unless you are using the #!/usr/bin/env python trick, in which case you already used up your one argument and cannot pass flags through to Python (or, for that matter, ask env to set the environment variable for you). But if you're the distro, you should know where Python is installed and do not need to use /usr/bin/env in the first place.

I imagine the main reason they don't do this in practice is the shebang. All of the other things can plausibly use the environment variable, which is preferable because it Just Works and does not require pervasively editing /usr/bin/python command line flags everywhere. The shebang, on the other hand, does require pervasive editing and also has the downside of not being automatically inherited by child processes. So it's a question of whether distros are willing to spend the time to support that.

(There's no need for all-caps vitriol.)

Posted May 29, 2025 21:16 UTC (Thu) by zahlman (guest, #175387) [Link]

> unless you are using the #!/usr/bin/env python trick, in which case you already used up your one argument and cannot pass flags through to Python (or, for that matter, ask env to set the environment variable for you). But if you're the distro, you should know where Python is installed and do not need to use /usr/bin/env in the first place.

More to the point, the distro *shouldn't* use /usr/bin/env this way, because that allows the user to override which Python is used - in particular, after "activating" a virtual environment, that would generally be used instead. Which will generally be Bad for system-provided scripts.

(But for the user's use, some versions of env support the -S flag, which *does* allow for passing command-line options like -s along to Python. And, of course, you can always put a shell wrapper around the startup process too.)

> I imagine the main reason they don't do this in practice is the shebang. All of the other things can plausibly use the environment variable, which is preferable because it Just Works and does not require pervasively editing /usr/bin/python command line flags everywhere. The shebang, on the other hand, does require pervasive editing and also has the downside of not being automatically inherited by child processes.

If the child is also Python code, there are other ways to run that code in a new process, but they aren't trivial changes.