Linux distributions and Python 2 [LWN.net]

By Jake Edge
June 11, 2018

Python 2.7 will reach its end of life in less than two years—at least for the core development team. Linux distributions need to figure out how to handle the transition given that many of their users are still using that version of the language—and may still be well beyond the end-of-life date. Petr Viktorin and Matthias Klose led a session at the 2018 Python Language Summit to discuss distributions' approaches to deprecating Python 2.

Viktorin works for Red Hat and focused on the Fedora distribution. He wants to help figure out how to help the Python downstreams so that Python 2 can be fully discontinued. There are two different ways to do that; either make sure that everyone switches to Python 3 or simply deprecate Python 2 and "wash our hands" of the problem. He would prefer the first alternative. He will be working on this transition for Red Hat as part of his day job and would like to do it in the community as well; that will minimize the need to maintain Python 2 going forward.

Fedora now has about two-thirds of its packages that are Python 3 compatible. What is left is mostly stuff that no one cares about, he said, but there are still some important packages that need Python 2. There are also packages where the Python 2 version could be dropped in favor of one that works with Python 3, but the distribution does not know if that will upset users or not. In theory, the package maintainer should have some idea, but in many cases, they do not.

The list of things that have not been ported to Python 3 includes the Mercurial and Bazaar distributed version-control systems, plugins for tools like GIMP and Inkscape, and bindings for Samba and various database-management systems. In addition, OpenStack is not yet Python 3 ready.

Fedora is a community distribution and it can at times suffer from a lack of interested maintainers. Some think the model of the interested distribution package maintainer is outdated and that package repositories like the Python Package Index (PyPI) should be used instead. But distribution packagers still have their place, Viktorin said. For example, the licensing information on PyPI is sometimes wrong, which is often caught by the distribution maintainers. One other barrier for community distributions is that there is no real authority to direct the volunteer efforts; so the distribution can't really deprecate Python 2 directly.

In companies it is difficult to get funds allocated for refactoring working Python 2 code to Python 3, so that change is usually pushed by developers from below. On the other hand, advertising the use of Python 3 is useful for recruiting developers, or so he has heard from several sources.

One of the goals for Fedora in this process is to make sure that it doesn't make things harder for those users who have already switched to Python 3. The first step in the transition is to move leaf packages (i.e. those that have no other packages dependent on them) to Python 3. Those changes have to be tested to ensure there are no lingering Python 2 dependencies. If there are problematic packages, where it is hard to switch them over to Python 3, he wants to help get them working.

Fedora still needs to support Python 2.7 as long as it is needed, he said. It also supports Python 3.3, 3.4, and 3.5, but only for testing that packages and applications work with them. Łukasz Langa asked what "as long as needed" means for Python 2.7; Viktorin said that it is as long as someone is depending on it. That didn't sit entirely well; Langa said that people will be reporting bugs in 2.7 to the Python community even though it is past its end of life. But Klose pointed out that the situation is no different than today; "end of life" Python versions are still supported, especially by the enterprise distributions.

Nick Coghlan noted that RHEL 7 shipped with Python 2.7 and it will be supported until 2024. Viktorin said the problem is going to keep happening, no matter who is maintaining old Python versions; what he wants is for whoever does the maintenance to care about it.

There are some "little annoyances" with gradually shifting over to Python 3, however. PEP 394 says that executing "python" should currently invoke Python 2. It also says that "python" in the shebang ("#!") line should mean that the script is compatible with both 2 and 3. However, maintainers of Python scripts may either be unaware of the PEP or just have not gotten around to switching, so the two "requirements" are at odds a bit. As detailed in a pull request to change the PEP, it would make sense to have no python symbolic link at all in some environments, especially in places where removing dependence on Python 2 is desired.

Langa suggested that perhaps switching the python symbolic link to Python 3 could be aligned with a long-term support (LTS) version of Python as was discussed in an earlier session. If, say, the 3.9 release were made in mid-2020 as an LTS, that might make a good point to change the recommendation in the PEP.

Klose took over at that point to give his perspective as a Debian and Ubuntu Python maintainer. For Ubuntu 16.04 LTS and Debian 9 ("stretch"), the python symbolic link points to Python 2. For Ubuntu 18.04 LTS, Python 2 is not part of the installation image. Invoking python will either remind users how to install Python 2 or will note the existence of Python 3 if it is installed.

There is ongoing work to get the distributions Python 3 ready. There are bug reports for things that do not work with Python 3, but they may be ignored if the bugs do not show up for Python 2, he said. What will not be so easily ignored is Lintian, which is a tool that finds problems in Debian packages. Adding Python 3 checks to Lintian should help reduce the number of problems.

Debian has made the decision to ship Python 2 with Debian 10 ("buster"), which will be released in mid-2019; that means Python 2 will be supported in Debian well past its 2020 end of life. For Ubuntu, the plan for the 20.04 LTS release has Python 2 removed from the main repository. That means it will not be supported by Canonical, so the community will need to pick it up if it is to continue at that point.

Index entries for this article
Conference	Python Language Summit/2018
Python	Linux distributions

to post comments

Linux distributions and Python 2

Posted Jun 11, 2018 18:06 UTC (Mon) by hkario (subscriber, #94864) [Link] (34 responses)

On one hand it does feel like the deprecation of Python 2 is happening too soon, when many people are still using it.
On the other hand, Python 2.7 was released on 3rd of July 2010, so over 8 years ago. And just 1 years before Python 3.2, the first backwards compatible Python 3. About the same time the PEP-404 - The Python 2.8 Un-Release - was published.

So people knew they will have to migrate off of Python 2 for a better half of a decade now...

It does feel like without a bit of strong-arming, they will never migrate off of it, as bad as it sounds.

Linux distributions and Python 2

Posted Jun 11, 2018 19:09 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (33 responses)

> they will never migrate off of it, as bad as it sounds.
Bad?

Having a completely stable language is actually a blessing.

Linux distributions and Python 2

Posted Jun 11, 2018 19:18 UTC (Mon) by Otus (subscriber, #67685) [Link] (19 responses)

Yeah, people still use last millennium's C versions.

Unfortunately, (c)python has tied the language and compiler/interpreter version quite strongly.

Linux distributions and Python 2

Posted Jun 11, 2018 19:23 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (16 responses)

C upgrades are basically backwards compatible. Upgrading from C98 to C11 usually requires only minor adjustments.

Py27 to Py3 migration is very painful and gains you pretty much nothing.

Linux distributions and Python 2

Posted Jun 11, 2018 19:55 UTC (Mon) by Otus (subscriber, #67685) [Link] (8 responses)

And yet this year's gcc releases see fit to support -std=c90 et al.

I will be completely unsurprised if 2.7 continues to see use ten years from now. I'd prefer to use it if I knew it would be supported. As is, I'm grudgingly making sure anything new also works under some 3.x.

Linux distributions and Python 2

Posted Jun 13, 2018 7:54 UTC (Wed) by codewiz (subscriber, #63050) [Link] (6 responses)

I'd be surprised if anything shipped as part of current Linux distros still required to be built in C90 mode. It's there just in case, and it's very easy to support for gcc because, as other have noted, the C and C++ languages have never broken compatibility to the level that Python did.

If C98 suddenly switched the meaning of 'char' from being an 8-bit integer to a Unicode codepoint, you can bet that we'd still be using C90 today, officially supported or not.

Linux distributions and Python 2

Posted Jun 13, 2018 8:53 UTC (Wed) by gevaerts (subscriber, #21521) [Link] (5 responses)

As far as I know, a C90 compiler can can have char be a Unicode codepoint just as well as a C98 one.

Linux distributions and Python 2

Posted Jun 13, 2018 10:13 UTC (Wed) by codewiz (subscriber, #63050) [Link] (4 responses)

There's wchar_t, but it's a distinct type from char: http://en.cppreference.com/w/cpp/language/types#Character...

While the standard doesn't even bother saying whether char is signed or unsiged, no sane C compiler would dare switching overnight the representation of C strings. Incrementally switching a large codebase from 8bit charater strings to Unicode would be nearly impossible. Pure insanity! Which is exactly why the transition to Python3 has been going on and on for the past 10 years in spite of the considerable effort the community put into it :-)

Linux distributions and Python 2

Posted Jun 13, 2018 10:55 UTC (Wed) by excors (subscriber, #95769) [Link] (3 responses)

wchar_t is a bit awkward since it can't actually hold Unicode codepoints on Windows.

Win32 did go through an ANSI->Unicode(ish) switch (with Windows NT, I presume), and tried to make it easy for applications to migrate incrementally. If you start with char* and "strings" and strcmp() and MessageBoxA(), you can gradually replace it with TCHAR* and _T("strings") and _tcscmp and MessageBox, and it will compile exactly the same as before. Then you #define UNICODE and suddenly it all gets macroed into WCHAR*/L"strings"/wcscmp/MessageBoxW etc, and you probably get a load of build errors, and you fix some before getting bored and going back to ANSI mode. After enough iterations you might get something that compiles properly with the Unicode API. Then you probably keep the TCHAR/_T/etc macros - it seems standard practice is to use those instead of the Unicode-only versions, even if you know you're never going back to ANSI.

I guess the difference with Python is that Microsoft saw the transition as an unending process, and they knew they would have to support the ANSI APIs forever, so they accepted the ugly macros as the cost of supporting both. Python expected the transition to finish quickly so they could kill off Python 2, so it seems they focused more on designing a nice end state (with arguable success) and less on easing the transition between the two states.

Linux distributions and Python 2

Posted Jun 13, 2018 11:56 UTC (Wed) by codewiz (subscriber, #63050) [Link] (2 responses)

And let's not forget _MBCS, hehe :-)

I guess Microsoft can be excused for the messy Unicode transition. They were among the first to attempt it, and at the time they were still saddled with support for a myriad of encodings from the DOS era. Oouch! :-)

But in 2008 there was no excuse whatsoever for not using the awesome UTF-8 encoding as the internal representation of strings. It was already obvious which way the wind was blowing, and pretty much every dynamic language transitioned to UTF-8, except Python. Actually, PHP 6 also tried moving to UTF-16, but after struggling with performance regressions and compatibility issues for several years, they finally realized it was a very poor decision and ditched the entire release. Then PHP 7 came out with UTF-8 strings.

Later programming languages like Go and Rust could have picked either UTF-8 or UTF-32 without compatibility concerns, and they still went with UTF-8 for simplicity and performance. You lose the O(1) random-access to Unicode codepoints, sure, but how often do you really do that? Whereas having to re-encode all text on the I/O boundary is a major inconvenience and causes all sort of round-trip issues.

Linux distributions and Python 2

Posted Jun 16, 2018 18:39 UTC (Sat) by epa (subscriber, #39769) [Link] (1 responses)

Does it really matter how the Python 3 implementation represents Unicode strings internally? It could use UTF-8, or UTF-16, or UCS-4, or even some wacky encoding of its own, and in principle you wouldn't notice any difference. I think what you are saying is that there should be an easier implicit conversion between byte strings and Unicode strings, with UTF-8-ish encoding and decoding rules used by default.

Linux distributions and Python 2

Posted Jun 24, 2018 14:40 UTC (Sun) by codewiz (subscriber, #63050) [Link]

Well, it does matter for both performance and memory usage, which are observable properties of a program as much as its output.

Earlier versions of the C++ standard intentionally left some things unspecified to give implementations a certain degree of freedom in when choosing the internal representations of data structures, etc. Turned out it was a terrible idea in practice, because some implementations, including GCC, chose to make std::string copy-on-write with reference counting, some had a small-string optimization which would save an allocation, and others would append the \0 on the fly only when someone invoked c_str(), occasionally causing the string to be reallocated.

This caused portability issues where a valid and idiomatic C++ program which would normally execute in 1 second on Linux could take hours and run out of memory on a different standard-compliant run-time which would copy all the strings. And there was no reasonable way to fix the performance bug without causing unpredictable performance regressions in other valid programs.

What I'm getting to here is that the performance characteristics of strings, dictionaries and lists is part of the contract. Even if left unspecified, a lot of code in the wild will start depending on it. And once you specify the exact behavior and complexity of all the basic operations on a container, this leaves very little room for changing the internal representation in a significant way.

Over time, Python 3 grew some clever string optimizations to mitigate the overhead of converting strings on the I/O boundary. But these optimizations are inherently data dependent. Let's say I want to read a 100MB html file into a string and then send it over a socket. This will take roughly 100MB of RAM in Python 2, while a Python 3 program will jump from 100MB to 400MB after someone inserted a single emoji in the middle of the file.

While dynamic, garbage-collected languages are expected to have less predictable runtime behavior, the idea that user-controlled input can undo an important optimization that doubles or even quadruples memory usage is terrifying.This could even be used as a DoS vector against Python servers running in containers with hard memory limits. For similar reasons, dicts typically employ a cryptographic hash function to prevent DoS attacks based on causing too many collisions, which would trigger worst-case O(n) lookups for every key.

And this is just one of the several issues that a clever internal representation will bring over a simple one. Another tricky one that comes to mind is round-trip of utf-8 and other encodings, including input containing invalid codepoints. I've seen Python backup software which failed to restore an archive due to a filename containing an invalid utf-8 character (that file came from AmigaOS which didn't have utf-8, and I never noticed because all cli tools simply handled the filename without trying to read too much into it :-)

These are the kind of concerns you wish a good language runtime would hide from you, so you don't need to audit your codebase to make sure nobody accidentally used the built-in string class to read arbitrary user input. Go and Rust took the simple and elegant approach of declaring that strings are internally represented the same way they're represented externally. Python 2 was essentially already the same way, so isn't it weird that it was perceived as a defect so serious to justify a 10-year long migration to get it fixed?

Linux distributions and Python 2

Posted Jun 14, 2018 8:44 UTC (Thu) by kooky (subscriber, #92468) [Link]

I wouldn't say you get nothing.

I've moved several large Flask web apps from python2 to python3. It has been very satisfying.

We were getting quite a few `unicode` trouble tickets. Fix usually just to run as python3. Changes to code usually very minor and done in a few hours.

Linux distributions and Python 2

Posted Jun 11, 2018 20:19 UTC (Mon) by malefic (subscriber, #37306) [Link] (6 responses)

> Py27 to Py3 migration is very painful

Having migrated numerous codebases, I disagree. For high quality codebases with decent test coverage the migration is straightforward. Instagram, which isn't small by any metric, had made their transition with several months worth of work by a few engineers (https://www.youtube.com/watch?v=66XoCk79kjM)

> gains you pretty much nothing.

A decade of language development is not "nothing". Even if you don't care about numerous language improvements, the real-world performance gains alone are a very good reason to upgrade (https://twitter.com/victorstinner/status/865973964421476352, https://bit.ly/2JEIzkh)

Linux distributions and Python 2

Posted Jun 11, 2018 20:37 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Py3 is often slower than Py2: https://stackoverflow.com/questions/37052139/string-perfo...

You can't say beforehand if your codebase is going to do any better with Py3. I've seen it go both ways.

Linux distributions and Python 2

Posted Jun 12, 2018 5:54 UTC (Tue) by k8to (guest, #15413) [Link]

I think it strongly depends how the str/bytes split affects your particular code.

I've brought very significant codebases to 3 without any significant pain, just cleanups that should have been done anyway, and I've also had code that required huge pain because it wanted to deal mostly with byte strings, but so much of the python3 library doesn't want byte strings.

Linux distributions and Python 2

Posted Jun 14, 2018 3:48 UTC (Thu) by lambda (subscriber, #40735) [Link] (3 responses)

For high quality codebases with decent test coverage the migration is straightforward

Good thing that most code out there is a high-quality code base with decent test coverage then, right?

I just ported a 750 line script from Python 2 to Python 3; it's only fairly recently that I've found there are enough reasons to do so (in this case, availability of Python 3 and not Python 2 in the base install of our distro, plus the typing module and mypy to add static types), and enough barriers have been removed.

Here are the steps I went through to make sure the port was working:

Run 2to3 over the script
Run autopep8 to fix a few issues introduced by 2to3
Run pylint and flake8, fix up issues found
Add type annotations, check types with mypy, fix up type issues that had crept in
Write unit tests (this had originally been a quick script, so not covered by unit tests, but had grown enough that they'd be useful for this port), fix issues discovered by unit tests that still remained even after mypy and pylint fixups
Realize that I'd ported to Python 3.6, and target system was running 3.5, so set up tox and fixed up more things that differed between 3.5 and 3.6 (tests that worked in 3.6 due to deterministic dict ordering started failing non-deterministically in 3.6; thanks to tox for printing the random number seed on each test run for making that easier to reproduce and fix)
Actually try building my package and tests on the target system, have to fix more test issues due to an older version of pytest on the target system.
Finally actually run the result on the target system, and even after all of those steps above (pylint, flake8, mypy, unit tests with pretty good coverage) still have issues to fix

And this is for a fairly simple little script, dealing with some files which just contain ASCII text and ASCII filenames.

Our main codebase uses Twisted, using PB and Jelly to pass objects over the network. There were no automated tests when I started at the company, and while we've been trying to add tests for new features or code that we're touching, we still have pretty poor coverage. Additionally, it's is a networked application that also has to interact with networked filesystems on Windows, macOS, and GNU/Linux, which makes the encoding issues even more fun to deal with. Oh, and most of what is jellied and shared between systems consists of dicts, with different value types for different keys.

At some point, we are going to have to port this to Python 3, but I imagine it's going to take a long time, cause a lot of grief for the developers in the process, and still probably cause regressions for customers even if we're as thorough as we can be by adding unit tests, integration tests, and type annotations.

Linux distributions and Python 2

Posted Jun 15, 2018 15:13 UTC (Fri) by togga (subscriber, #53103) [Link] (2 responses)

Your comment actually makes a case for migrating from python altogether. I've done many attempts on py3 over the years and in most cases py3 is a regression, especially when using python as a glue- pr scripting language. Syntax candy and numerous ways to workaround a broken threading model doesn't make up for the downsides.

Start try to reuse existing python modules from another more "down to earth" language like go and then migrate away is my tip. Python language is obviously not long term stable.

Linux distributions and Python 2

Posted Jun 28, 2018 6:24 UTC (Thu) by larslehtonen (guest, #125318) [Link] (1 responses)

Go is the real Python 3. Is that still at all controversial?

Linux distributions and Python 2

Posted Jun 28, 2018 13:15 UTC (Thu) by anselm (subscriber, #2796) [Link]

Yes.

Linux distributions and Python 2

Posted Jun 13, 2018 9:08 UTC (Wed) by gdt (subscriber, #6284) [Link] (1 responses)

Yeah, people still use last millennium's C versions.

C had a moment not dissimilar to Python's, with the move from K&R to ANSI C.

It was one of the defining moments in the development of free software. Some popular Unixen took the opportunity to charge for their ANSI C compilers, which lead people to discover GCC and then the GNU Project's other tools.

Linux distributions and Python 2

Posted Jun 14, 2018 3:55 UTC (Thu) by lsl (subscriber, #86508) [Link]

> C had a moment not dissimilar to Python's, with the move from K&R to ANSI C.

The difference being that I can still use code written in the K&R dialect and link it together with new sources into a single program. Heck, I can even use K&R style as well as C11 features in the same translation unit. Sure, you get a -Wimplicit-int warning and it's a GCC thing but it works.

Linux distributions and Python 2

Posted Jun 11, 2018 20:59 UTC (Mon) by remicardona (guest, #99141) [Link] (12 responses)

> Having a completely stable language is actually a blessing.

And that's fine, if you're willing to maintain CPython 2.7 when upstream _finally_ cuts the cord.

As far as I'm concerned, there's more than enough py3 work around that I can actively decline py27 offers. Maybe I'm exaggerating a bit, but to me, py27 is now as foreign a language as node or php. I've worked with it and taught it for a long time, but I don't want to deal with its idiosyncrasies. As a professional trainer/teacher, py3 is way easier to teach than py27.

And most companies are either on board with py3 or they're finally realizing that py27 is a dead end and they're going to have a hard time attracting developers (juniors or seasoned) with py27 code bases.

Again, nothing wrong with py27 per se, but good luck finding actively maintained packages that consider py27 as a blessing. Good luck finding devs willing to work on py27 projects. Good luck finding a supported/maintained py27 distribution. And good luck migrating to py3 the day py27 is no longer usable within your organization.

And I mean that sincerely: good luck.

Linux distributions and Python 2

Posted Jun 11, 2018 21:15 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> And that's fine, if you're willing to maintain CPython 2.7 when upstream _finally_ cuts the cord.
It'll be supported by RHEL well into 2027. I don't have to care much about it.

Py3 is not really a different language, it's the same old Py2 with added inconvenience. I've done migrations Py2->3 and haven't found anything worth the effort so far. And I really detest the efforts to force the migration by actively making Py2 more and more inconvenient to use.

Personally, a major motivation for me to finally switch to PyX would be GIL removal.

Linux distributions and Python 2

Posted Jun 12, 2018 12:01 UTC (Tue) by hkario (subscriber, #94864) [Link] (1 responses)

> it's the same old Py2 with added inconvenience

that's, like, your opinion, man

seriously, I'd love to drop python 2 to be able to use generators to their full advantage (`yield from` would make my code so much cleaner)

so, sorry, but if you really think that Py3 is just py2 with few "inconvenient" items added then I don't think you can claim to know Py3

> Personally, a major motivation for me to finally switch to PyX would be GIL removal.

well, it won't happen in py2.7, that's certain

Linux distributions and Python 2

Posted Jun 14, 2018 20:12 UTC (Thu) by togga (subscriber, #53103) [Link]

> well, it won't happen in py2.7, that's certain

http://pypy.org

Linux distributions and Python 2

Posted Jun 14, 2018 9:13 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (3 responses)

> it's the same old Py2 with added inconvenience

Something tells me that you never ventured outside of writing English, because in any other language, having strings in unicode rather than ASCII is a huge improvement.

Then, Python3 has type annotations, which lets you use mypy to do some static checks on the code.

I wrote typedload (https://github.com/ltworf/typedload) to load json-like data into typed data structures, so that once loaded, you know your data has the correct types and can be safely be passed around.

My grandmother did not know how to use a computer, that doesn't mean that computers are useless.

Linux distributions and Python 2

Posted Jun 14, 2018 17:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

JFYI, my native language is Russian so I’m acutely aware of encoding problems since so many of them were used for Cyrillic scripts. I can speak Ukrainian and understand Polish and Czech. I’d used to be able to speak German but haven’t practiced it in decades and lost most of it. Now I’m also studying Mandarin Chinese for a change.

Static annotations are nice, but are kinda beside the point.

Linux distributions and Python 2

Posted Jun 16, 2018 16:33 UTC (Sat) by lsl (subscriber, #86508) [Link] (1 responses)

Except that Python 2 strings aren't ASCII and support Unicode perfectly well: you just put UTF-8 inside them.

Linux distributions and Python 2

Posted Sep 19, 2019 16:11 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

They work but are very error prone, because characters might take any amount of elements, so you have all sort of issues when doing cycles or manipulation.

With unicode objects, you know that they are not array of bytes and a conversion is needed, and you get an error if you didn't do the conversion that you were supposed to do.

Linux distributions and Python 2

Posted Jun 15, 2018 15:28 UTC (Fri) by togga (subscriber, #53103) [Link] (4 responses)

> Good luck finding devs willing to work on py27 projects.

My experience is the opposite. With a py3 project, you are almost guaranteed to end up in an encoding hell. For me, py3 projects raises the "over-engineered" or "system architecture by syntax convenience" flags and you may find yourself in a world of software pain.

Linux distributions and Python 2

Posted Mar 5, 2021 11:22 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (3 responses)

One could go and read a 10 minutes tutorial…

Linux distributions and Python 2

Posted Mar 5, 2021 12:02 UTC (Fri) by pizza (subscriber, #46) [Link] (2 responses)

"go read a tutorial" does not change the data your python3 code encounters out in the real world.

Linux distributions and Python 2

Posted Mar 5, 2021 15:59 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (1 responses)

It helps in knowing how to not write buggy code that handles encodings.

Anyway for "random data in the world" python3 is way better than python2. Which is why the change was made.

In python2 you couldn't really rely on ranges or indexes on strings, because you never knew if you would be splitting a unicode sequence and creating something broken.

But, it worked most of the times… it would just fail when unexpected unicode sequences appeared.

Linux distributions and Python 2

Posted Mar 5, 2021 16:15 UTC (Fri) by foom (subscriber, #14868) [Link]

In python3 you can't really rely on ranges or indexes on strings either, because you never know if you will be splitting a unicode grapheme cluster and creating something broken.

But, it works most of the time… it will just fail when unexpected unicode sequences appear (combining accents, or emoji skin tone modifiers, or flags, or ...)

Contrast this with perl6, which has built in support for correctly preserving grapheme clusters in it's string methods.

Linux distributions and Python 2

Posted Jun 11, 2018 20:11 UTC (Mon) by josh (subscriber, #17465) [Link] (3 responses)

> Langa suggested that perhaps switching the python symbolic link to Python 3 could be aligned with a long-term support (LTS) version of Python as was discussed in an earlier session.

I certainly hope this never happens.

I don't mind if, one day, systems ship without a /usr/bin/python at all. But if it exists, it should continue to point to Python 2, because scripts with #!/usr/bin/python at the top still expect that, and those scripts might never change.

Linux distributions and Python 2

Posted Jun 12, 2018 14:56 UTC (Tue) by rbanffy (guest, #103898) [Link] (2 responses)

One easy way to check is to remove /usr/bin/python and fix whatever breaks by calling /usr/bin/python2.

Anyone publishing software that depends on Python 2 to be at /usr/bin/python is just inviting a lot of future pain.

Linux distributions and Python 2

Posted Jun 14, 2018 0:18 UTC (Thu) by NAR (subscriber, #1313) [Link]

I guess the big problem will be site-specific scripts created years ago that run only once in a month, but if they fail, the sysadmin will get a phone call at 2AM.

Linux distributions and Python 2

Posted Jun 14, 2018 8:53 UTC (Thu) by danpb (subscriber, #4831) [Link]

This doesn't even remotely work for non-trivial programs, as there's no way you can have confidence that you've exercised all code paths simply by running it and seeing what breaks. You need to use an automated analysis and/or conversion tool first to check the entire codease, and after that try to execute it and if it works. Even then for non-trivial programs there's still a good chance it is broken. If the app has good unit test coverage this can help, but that's nowhere near foolproof.

Ideally everyone would none the less do the conversion to py3 and accept the risks/pain, but in reality I'd expect many companies will just keep around an ancient python2 version and not touch their apps/scripts :-(

Linux distributions and Python 2

Posted Jun 11, 2018 23:10 UTC (Mon) by kenmoffat (guest, #4807) [Link] (4 responses)

I thought firefox (and probably thunderbird and seamonkey?) was one of the main reasons for installing Python2 in a modern distro. Sure, some gimp plugins (and the gimp help files) and inkscape also need it.

Linux distributions and Python 2

Posted Jun 12, 2018 16:02 UTC (Tue) by imphil (subscriber, #62487) [Link] (1 responses)

Where does Firefox need Python 2? Building Firefox from source might require Python 2, but I'm not aware of anything in Firefox that requires it at runtime. Ubuntu seems to agree: https://packages.ubuntu.com/bionic/firefox

Linux distributions and Python 2

Posted Jun 12, 2018 16:46 UTC (Tue) by kenmoffat (guest, #4807) [Link]

Yes, I was talking about building it. Last time I checked (probably about 6 weeks ago), python2 was required. If you want to link it to (some) system libs - particularly nss and system certificates - then you need to build it from source.

Linux distributions and Python 2

Posted Jun 13, 2018 12:19 UTC (Wed) by mgedmin (guest, #34497) [Link] (1 responses)

Where does Firefox use Python?

Linux distributions and Python 2

Posted Jun 13, 2018 14:40 UTC (Wed) by kenmoffat (guest, #4807) [Link]

In the build system, it now invokes e.g. configure.py.

0:00.60 Creating Python environment
0:01.54 New python executable in /scratch/working/firefox-60.0.2/firefox-build-dir/_virtualenv/bin/python2.7
0:01.54 Also creating executable in /scratch/working/firefox-60.0.2/firefox-build-dir/_virtualenv/bin/python
0:01.54 Installing setuptools, pip, wheel...done.
0:02.27 running build_ext
0:02.27 building 'psutil._psutil_linux' extension

Linux distributions and Python 2

Posted Jun 12, 2018 1:10 UTC (Tue) by Kamilion (subscriber, #42576) [Link] (7 responses)

I am struck that nobody seems to notice that the python2/3 problem was already solved on windows by providing py.exe as a frontend to any installed python version.

Shouldn't /usr/bin/python simply be treated the same way, and converted into a tool that will select python2 or python3 as appropriate? Seems a lot simpler to get people to *just fix that* in a well-distributed python script *now* rather than going as far as forcing them to convert to python3 right away.

Linux distributions and Python 2

Posted Jun 12, 2018 5:58 UTC (Tue) by k8to (guest, #15413) [Link] (4 responses)

How does it determine what is appropriate?

Linux distributions and Python 2

Posted Jun 12, 2018 14:53 UTC (Tue) by rbanffy (guest, #103898) [Link] (1 responses)

If the script is already compiled to a .pyc, you can probably find out which version of Python compiled it (I never tried that because, frankly, I'm very happy with having python and python3 on /usr/bin).

Linux distributions and Python 2

Posted Jun 12, 2018 15:24 UTC (Tue) by k8to (guest, #15413) [Link]

That's determining which python already ran it, not which python it needs.
I don't believe it's very decidable.

Linux distributions and Python 2

Posted Jun 14, 2018 5:19 UTC (Thu) by kenshoen (guest, #121595) [Link] (1 responses)

They look at shebang line. #!python2 to 2, #!python3 to 3, #!python defaults to 2 (configurable).
https://docs.python.org/3/using/windows.html#customizing-...

Linux distributions and Python 2

Posted Jun 15, 2018 6:47 UTC (Fri) by nirbheek (guest, #54111) [Link]

So the same as Linux then?

py.exe on Windows solves a completely different issue: where is Python installed and how do I invoke it? All pythons (2, 3.4, 3.5, etc) are called python.exe, so putting it in PATH is not an option.

Linux distributions and Python 2

Posted Jun 14, 2018 6:18 UTC (Thu) by lambda (subscriber, #40735) [Link] (1 responses)

Which interpreter is the right one to use for the following code in an executable file named polyglot:

#!/usr/bin/python

print("2/3 = {}".format(2/3))

Will the output remain the same if one system decides to run it with a Python 2 interpreter and another runs it with a Python 3 interpreter?

Linux distributions and Python 2

Posted Oct 2, 2018 9:09 UTC (Tue) by cclauss (guest, #127609) [Link]

Adding from __future__ import division will print a fractional result in both Python 2 and Python 3.

Changing / --> // will print zero in both Python 2 and Python 3.

Linux distribution without Python 2

Posted Jun 17, 2018 10:56 UTC (Sun) by awilfox (guest, #124923) [Link] (1 responses)

As lead of Adélie, which was formed in 2015, I've had the true privilege of not packaging Python 2 ever.

We have migrated literally every Py2-requiring package we've dealt with to Py3, and most of it is done in under a day. We always send those patches upstream, as well.

The only sticking point has been gnome-doc-utils, which has a Py3 port in progress, but there's still some issues with it. I'm hopeful we'll be able to ship docs with our Gnome packages some time in 2019 at their current rate.

We have a tracker bug for software that incorrectly uses /usr/bin/python instead of python3, as well, and that's going to need to be handled eventually: https://bts.adelielinux.org/show_bug.cgi?id=41 Just haven't had the time yet.

My rationale for this has always been that we should not be shipping known-deprecated software, nor should we be "propping up" bad practices such as continuing to rely on known-deprecated software.

Linux distribution without Python 2

Posted Jul 16, 2018 14:52 UTC (Mon) by cstratak (guest, #109730) [Link]

AFAIK gnome-doc-utils has been deprecated in favour of yelp-tools [0], so it might be easier to just use that. Is there a publicly available place where we can track the porting of gnome-doc-utils?

[0] https://wiki.gnome.org/Projects/GnomeDocUtils/MigrationHowTo

Linux distributions and Python 2

Posted Oct 2, 2018 8:59 UTC (Tue) by cclauss (guest, #127609) [Link]

With 456 days until Python 2 end of life, today's stats are:

    408 distros with Python 3.4+
    ============================
    105 distros with Python 3.4
    160 distros with Python 3.5
    122 distros with Python 3.6
     21 distros with Python 3.7

https://github.com/Mariatta/python_versions_and_distros