|
|
Subscribe / Log in / New account

Fedora ponders the Python 2 end game

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 16:34 UTC (Tue) by fratti (guest, #105722)
Parent article: Fedora ponders the Python 2 end game

For what it's worth, Arch Linux has made the switch of pointing "python" to "python3" years ago, and while that did break many things and annoyed many users, the rest of the distributions benefited from people already being aware that "python" does not always mean "python2", and having fixed scripts for that case.


to post comments

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 16:44 UTC (Tue) by josh (subscriber, #17465) [Link] (25 responses)

The problem is that that makes it much harder to write a portable Python script. Arch points /usr/bin/python at Python 3, so you can't use "#!/usr/bin/python"; some other distributions don't provide "python2" and "python3" names, so you can't always use "#!/usr/bin/python2" (or for that matter "#!/usr/bin/python3"), either.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 16:45 UTC (Tue) by josh (subscriber, #17465) [Link]

(This also applies when trying to invoke Python, such as from a Makefile or other build script.)

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 16:55 UTC (Tue) by fratti (guest, #105722) [Link] (8 responses)

As far as I know, all distributions aside from CentOS 5 (which is end of life) provide a python2 symlink. This may have been an argument years ago, but nowadays as far as I know, #!/usr/bin/env python2 is a safe bet.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:10 UTC (Tue) by josh (subscriber, #17465) [Link] (2 responses)

Windows has the same problem: the current Python 2 installer for Windows doesn't include a "python2.exe", and as far as I know the Python 3 installer for Windows doesn't include "python3.exe" either.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:12 UTC (Tue) by fratti (guest, #105722) [Link] (1 responses)

Windows also doesn't have shebang lines from what I can tell, so this is a moot point.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:19 UTC (Tue) by josh (subscriber, #17465) [Link]

It still runs makefiles, build scripts, and other things that may need to invoke Python.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 12:50 UTC (Wed) by ballombe (subscriber, #9523) [Link] (4 responses)

Alas, it is not a safe bet. There are more and more systems where only /bin/env exists.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 14:55 UTC (Wed) by josh (subscriber, #17465) [Link] (3 responses)

Wait, what? Which systems install env in /bin?

Fedora ponders the Python 2 end game

Posted Aug 3, 2017 20:17 UTC (Thu) by hmh (subscriber, #3838) [Link] (2 responses)

The ones I know of have either /usr -> / or /usr/bin -> /bin symlinked, so /usr/bin/env will just work...

Fedora ponders the Python 2 end game

Posted Aug 3, 2017 20:38 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (1 responses)

Arch does it the other way (/bin is a link to /usr/bin)

$ find / -maxdepth 1 -type l -exec ls -l {} \;
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /lib64 -> usr/lib
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /lib -> usr/lib
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /sbin -> usr/bin
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /bin -> usr/bin

Fedora ponders the Python 2 end game

Posted Aug 4, 2017 5:33 UTC (Fri) by josh (subscriber, #17465) [Link]

That's how many current distributions do it these days, including Fedora. And Debian has an option to do that, if you install the "usrmerge" package.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:00 UTC (Tue) by karkhaz (subscriber, #99844) [Link] (7 responses)

So don't write #!/usr/bin/python, either with a digit or without one. Use /usr/bin/env instead. As far as I know, this should always work (even on the distributions that don't symlink python to python2---are there really modern, not EOLed distros that don't do this?)

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:12 UTC (Tue) by josh (subscriber, #17465) [Link] (6 responses)

env doesn't help with this problem; it just does a PATH search, but you still have to tell it which binary name to invoke. That helps if python isn't in /usr/bin; it doesn't help if the name "python" invokes the wrong version of python.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:18 UTC (Tue) by liw (subscriber, #6379) [Link] (4 responses)

An env shebang also doesn't help in the case your system has some system software installed (such as a backup application, ahem), written in Python, and a user writes their own Perl interprer, name it "python"m and put that before the system Python in their $PATH. Unless the author of the backup application has foreseent this and written a Python2/Python3/Perl polyglot script, the user has shot off their entire leg. We try to make it safe for users to play with siege cannons.

System-installed software shouldn't break when a user installs things early on in their $PATH.

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 17:22 UTC (Tue) by josh (subscriber, #17465) [Link]

System-installed software shouldn't use env, sure; it also knows where Python is, so it shouldn't waste the time searching for it.

But there's a limit to how much you can protect the user from their own attempts to break things.

(That said, I've certainly seen more than a few reports of brokenness that ultimately got tracked down to a user's local installation of python in /usr/local or $HOME, so I mostly agree.)

Fedora ponders the Python 2 end game

Posted Aug 1, 2017 23:39 UTC (Tue) by flussence (guest, #85566) [Link] (2 responses)

Gentoo occasionally takes this problem to new levels of insanity:

The package manager is /usr/bin/emerge. That's a wrapper script (written in python, invoked by a C binary that chases symlinks to figure out which version of python to call it with) that chases symlinks to figure out where the real program is (which is also a python script, but can be installed for one or more of python2/3.x/pypy simultaneously).

Now and again there's support requests from users who mess up their system by manually installing python packages as root outside the OS's control, and have ended up with something in the dependency chain installing a /usr/local/bin/emerge (also a python script) that does something completely different, but is higher priority in $PATH, and gives vague errors that don't make it immediately obvious that the wrong thing's being run.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 3:38 UTC (Wed) by wahern (subscriber, #37304) [Link] (1 responses)

POSIX created the getconf(1) utility just so you could do PATH="$(getconf PATH)" to get the default system provided PATH guaranteed to provide the POSIX utilities. It's one of the rare instances where they didn't just codify existing practice. Perhaps for that reason it's an underused and virtually unknown command. Theoretically Linux distributions could extend it so that something like getconf PYTHON2 prints the default python2 interpreter (if available). That doesn't by itself solve the problem of a simple shebang line, but I think it's the least distributions could do to meet developers half-way.[1]

For Lua scripts I often abuse a coincidence of shell and Lua syntax so I can do

 #!/bin/sh
_=[[
  # shell script code to locate a Lua interpreter
]]
-- Lua code

where _=[[ begins a multiline string in Lua but is a harmless assignment in shell. Because the semantics of shell syntax effectively require parsing the file line-by-line, the shell code never reaches the terminating ]] or subsequent Lua code as long as you invoke exec or exit before reaching ]]. I'm not familiar with Python syntax but perhaps something similar can be done.

[1] Distributions could standardize on the "python2" command name, but you still run into the problem of a bad PATH variable. Ideally you'd be able to do something like #!/usr/bin/env command -p python2, where the -p switch to the command utility only searches the default PATH from getconf PATH. And where env(1) locates command(1), as command(1) is usually a built-in and even when it's available as an independent utility I don't know if you can expect it to be in /bin or /usr/bin--not in the same way that env(1) is reliably at the absolute path /usr/bin/env. But Linux is one of the few Unix systems that concatenate all the shebang command argument words before invoking the interpreter, so the above will work on macOS and some other kernels, but not on Linux.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 8:26 UTC (Wed) by itvirta (guest, #49997) [Link]

> Ideally you'd be able to do something like #!/usr/bin/env command -p python2

Except that you can't do that in Linux, the kernel only passes one argument from the hashbang line. I.e. the line above would run "/usr/bin/env" with "command -p python2" as one argument (plus the name of the script as the second argument). Or well, you could, if this "env" would split the argument to pieces itself.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 15:21 UTC (Wed) by drag (guest, #31333) [Link]

> That helps if python isn't in /usr/bin; it doesn't help if the name "python" invokes the wrong version of python.

It's only part of the solution. If you hardcode the paths then it pretty much forces users to edit your program to suite their environment.

$ which python2
/home/user/.pyenv/shims/python2

$ which python3
/home/user/.pyenv/shims/python3

The situation here is that distributions really don't solve these issues for end users. If users want to be able to take full advantage of the python ecosystem for writing and running their software they are going to have to use a alternative solution.

If you only really to write complex python software for a particular Linux distribution (for a enterprise environment, for example) then trying to only use dependencies in the distribution is fine and is very likely to work. But if your goal is to have something that works on a wide variety of Linux distributions, then you end up stuck with a more 'DIY' approach. Making your software available via pip is really what is going to make it easiest for people to use your software at this point.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 15:10 UTC (Wed) by drag (guest, #31333) [Link] (4 responses)

> The problem is that that makes it much harder to write a portable Python script.

This one of the reasons why people who care about portability between Linux distributions and want to be productive end up using things like pip, pvenv, and pyenv instead of distribution-provided dependencies for anything complex.

> so you can't always use "#!/usr/bin/python2" (or for that matter "#!/usr/bin/python3"), either.

Unless you are writing scripts meant to be shipped part of the operating system then you really really need to avoid using any sort of hard-coded paths to python interpreters. There is absolutely meaningful differences you need to take into consideration with this sort of 'part of the OS' versus 'on the OS'. If you are writing as part of the OS distribution.. use hard coded paths. If you are writing programs as a user of the OS then use '#!/usr/bin/env python'.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 15:39 UTC (Wed) by karkhaz (subscriber, #99844) [Link] (3 responses)

I'm now wondering whether the collective wisdom in this thread is codified anywhere? Somebody pointed out that my comment further up was wrong, and I subsequently looked for guidance. There seems to be no shortage of style guides for python code, but _distribution_ information seems to be sparse. I haven't found a document advising on the shebang line anywhere.

The Debian python packaging guidelines doesn't mention it, and neither even does the Arch Linux one---remarkable because /usr/bin/python on Arch is a symlink to /usr/bin/python3 (which itself is a symlink to /usr/bin/python3.6). There was a bit of breakage back when Arch did that, I suppose similar issues to when Debian switched their /bin/sh to a proper POSIX shell.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 16:42 UTC (Wed) by liw (subscriber, #6379) [Link] (1 responses)

I think that this is covered in the Debian Python policy: https://www.debian.org/doc/packaging-manuals/python-polic... last paragraph in the section: " Maintainers should not override the Debian Python interpreter using /usr/bin/env name. This is not advisable as it bypasses Debian's dependency checking and makes the package vulnerable to incomplete local installations of Python. "

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 18:44 UTC (Wed) by drag (guest, #31333) [Link]

Yep. This makes a lot of sense for python programs you are shipping as part of the distribution. You definitely want hard coded paths.

You want to avoid having these programs be broken by user's shell environments as much as possible.

Fedora ponders the Python 2 end game

Posted Aug 3, 2017 12:45 UTC (Thu) by ewan (guest, #5533) [Link]

I'm now wondering whether the collective wisdom in this thread is codified anywhere?
It's somewhat codified in PEP-0394, which advises on how python should be installed (in short, don't make 'python' point to Python 3) and how to cope with the fact that it might (the less than completely helpful suggestion to make anything that just asks for 'python' be compatible with both Python 2 and Python 3).

Fedora ponders the Python 2 end game

Posted Aug 4, 2017 4:12 UTC (Fri) by anatolik (guest, #73797) [Link] (1 responses)

> some other distributions don't provide "python2" and "python3" names

Which one does not do it and why? There is a PEP that says all python distributions should provide "versioned python" binary. Even macports has pythonX nowdays.

Fedora ponders the Python 2 end game

Posted Aug 22, 2017 8:21 UTC (Tue) by mgedmin (subscriber, #34497) [Link]

Older versions of Debian and Ubuntu did not have a python2, just a python and a python3.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 6:33 UTC (Wed) by joib (subscriber, #8541) [Link] (1 responses)

The python3 version of anaconda (Anaconda3) also installs a python3 binary under the name "python". Which breaks all code that assumes "/usr/bin/env python" gets them a python2 interpreter. Very annoying.

Fedora ponders the Python 2 end game

Posted Aug 13, 2017 14:59 UTC (Sun) by Wol (subscriber, #4433) [Link]

I think the default on gentoo is now python3. I know I've been using one of the MD scripts, and have had to edit the shebang specifically to call python2.

It would certainly make life easier if all distro-supplied stuff used a shebang explicitly to call the version it wanted. But that's a lot of work catching all instances ...

Cheers,
Wol


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds