LWN: Comments on "Linux distributions and Python 2"

Linux distributions and Python 2

foom — Fri, 05 Mar 2021 16:15:48 +0000

In python3 you can't really rely on ranges or indexes on strings either, because you never know if you will be splitting a unicode grapheme cluster and creating something broken.

But, it works most of the time… it will just fail when unexpected unicode sequences appear (combining accents, or emoji skin tone modifiers, or flags, or ...)

Contrast this with perl6, which has built in support for correctly preserving grapheme clusters in it's string methods.

Linux distributions and Python 2

LtWorf — Fri, 05 Mar 2021 15:59:37 +0000

It helps in knowing how to not write buggy code that handles encodings.

Anyway for "random data in the world" python3 is way better than python2. Which is why the change was made.

In python2 you couldn't really rely on ranges or indexes on strings, because you never knew if you would be splitting a unicode sequence and creating something broken.

But, it worked most of the times… it would just fail when unexpected unicode sequences appeared.

Linux distributions and Python 2

pizza — Fri, 05 Mar 2021 12:02:12 +0000

"go read a tutorial" does not change the data your python3 code encounters out in the real world.

Linux distributions and Python 2

LtWorf — Fri, 05 Mar 2021 11:22:00 +0000

One could go and read a 10 minutes tutorial…

Linux distributions and Python 2

LtWorf — Thu, 19 Sep 2019 16:11:53 +0000

They work but are very error prone, because characters might take any amount of elements, so you have all sort of issues when doing cycles or manipulation.

With unicode objects, you know that they are not array of bytes and a conversion is needed, and you get an error if you didn't do the conversion that you were supposed to do.

Linux distributions and Python 2

cclauss — Tue, 02 Oct 2018 09:09:06 +0000

Adding from __future__ import division will print a fractional result in both Python 2 and Python 3.

Changing / --> // will print zero in both Python 2 and Python 3.

Linux distributions and Python 2

cclauss — Tue, 02 Oct 2018 08:59:40 +0000

With 456 days until Python 2 end of life, today's stats are:

    408 distros with Python 3.4+
    ============================
    105 distros with Python 3.4
    160 distros with Python 3.5
    122 distros with Python 3.6
     21 distros with Python 3.7

https://github.com/Mariatta/python_versions_and_distros

Linux distribution without Python 2

cstratak — Mon, 16 Jul 2018 14:52:33 +0000

AFAIK gnome-doc-utils has been deprecated in favour of yelp-tools [0], so it might be easier to just use that. Is there a publicly available place where we can track the porting of gnome-doc-utils?

[0] https://wiki.gnome.org/Projects/GnomeDocUtils/MigrationHowTo

Linux distributions and Python 2

anselm — Thu, 28 Jun 2018 13:15:13 +0000

Yes.

Linux distributions and Python 2

larslehtonen — Thu, 28 Jun 2018 06:24:00 +0000

Go is the real Python 3. Is that still at all controversial?

Linux distributions and Python 2

codewiz — Sun, 24 Jun 2018 14:40:09 +0000

Well, it does matter for both performance and memory usage, which are observable properties of a program as much as its output.

Earlier versions of the C++ standard intentionally left some things unspecified to give implementations a certain degree of freedom in when choosing the internal representations of data structures, etc. Turned out it was a terrible idea in practice, because some implementations, including GCC, chose to make std::string copy-on-write with reference counting, some had a small-string optimization which would save an allocation, and others would append the \0 on the fly only when someone invoked c_str(), occasionally causing the string to be reallocated.

This caused portability issues where a valid and idiomatic C++ program which would normally execute in 1 second on Linux could take hours and run out of memory on a different standard-compliant run-time which would copy all the strings. And there was no reasonable way to fix the performance bug without causing unpredictable performance regressions in other valid programs.

What I'm getting to here is that the performance characteristics of strings, dictionaries and lists is part of the contract. Even if left unspecified, a lot of code in the wild will start depending on it. And once you specify the exact behavior and complexity of all the basic operations on a container, this leaves very little room for changing the internal representation in a significant way.

Over time, Python 3 grew some clever string optimizations to mitigate the overhead of converting strings on the I/O boundary. But these optimizations are inherently data dependent. Let's say I want to read a 100MB html file into a string and then send it over a socket. This will take roughly 100MB of RAM in Python 2, while a Python 3 program will jump from 100MB to 400MB after someone inserted a single emoji in the middle of the file.

While dynamic, garbage-collected languages are expected to have less predictable runtime behavior, the idea that user-controlled input can undo an important optimization that doubles or even quadruples memory usage is terrifying.This could even be used as a DoS vector against Python servers running in containers with hard memory limits. For similar reasons, dicts typically employ a cryptographic hash function to prevent DoS attacks based on causing too many collisions, which would trigger worst-case O(n) lookups for every key.

And this is just one of the several issues that a clever internal representation will bring over a simple one. Another tricky one that comes to mind is round-trip of utf-8 and other encodings, including input containing invalid codepoints. I've seen Python backup software which failed to restore an archive due to a filename containing an invalid utf-8 character (that file came from AmigaOS which didn't have utf-8, and I never noticed because all cli tools simply handled the filename without trying to read too much into it :-)

These are the kind of concerns you wish a good language runtime would hide from you, so you don't need to audit your codebase to make sure nobody accidentally used the built-in string class to read arbitrary user input. Go and Rust took the simple and elegant approach of declaring that strings are internally represented the same way they're represented externally. Python 2 was essentially already the same way, so isn't it weird that it was perceived as a defect so serious to justify a 10-year long migration to get it fixed?

Linux distribution without Python 2

awilfox — Sun, 17 Jun 2018 10:56:37 +0000

As lead of Adélie, which was formed in 2015, I've had the true privilege of not packaging Python 2 ever.

We have migrated literally every Py2-requiring package we've dealt with to Py3, and most of it is done in under a day. We always send those patches upstream, as well.

The only sticking point has been gnome-doc-utils, which has a Py3 port in progress, but there's still some issues with it. I'm hopeful we'll be able to ship docs with our Gnome packages some time in 2019 at their current rate.

We have a tracker bug for software that incorrectly uses /usr/bin/python instead of python3, as well, and that's going to need to be handled eventually: https://bts.adelielinux.org/show_bug.cgi?id=41 Just haven't had the time yet.

My rationale for this has always been that we should not be shipping known-deprecated software, nor should we be "propping up" bad practices such as continuing to rely on known-deprecated software.

Linux distributions and Python 2

epa — Sat, 16 Jun 2018 18:39:50 +0000

Does it really matter how the Python 3 implementation represents Unicode strings internally? It could use UTF-8, or UTF-16, or UCS-4, or even some wacky encoding of its own, and in principle you wouldn't notice any difference. I think what you are saying is that there should be an easier implicit conversion between byte strings and Unicode strings, with UTF-8-ish encoding and decoding rules used by default.

Linux distributions and Python 2

lsl — Sat, 16 Jun 2018 16:33:22 +0000

Except that Python 2 strings aren't ASCII and support Unicode perfectly well: you just put UTF-8 inside them.

Linux distributions and Python 2

togga — Fri, 15 Jun 2018 15:28:24 +0000

> Good luck finding devs willing to work on py27 projects.

My experience is the opposite. With a py3 project, you are almost guaranteed to end up in an encoding hell. For me, py3 projects raises the "over-engineered" or "system architecture by syntax convenience" flags and you may find yourself in a world of software pain.

Linux distributions and Python 2

togga — Fri, 15 Jun 2018 15:13:11 +0000

Your comment actually makes a case for migrating from python altogether. I've done many attempts on py3 over the years and in most cases py3 is a regression, especially when using python as a glue- pr scripting language. Syntax candy and numerous ways to workaround a broken threading model doesn't make up for the downsides.

Start try to reuse existing python modules from another more "down to earth" language like go and then migrate away is my tip. Python language is obviously not long term stable.

Linux distributions and Python 2

nirbheek — Fri, 15 Jun 2018 06:47:39 +0000

So the same as Linux then?

py.exe on Windows solves a completely different issue: where is Python installed and how do I invoke it? All pythons (2, 3.4, 3.5, etc) are called python.exe, so putting it in PATH is not an option.

Linux distributions and Python 2

togga — Thu, 14 Jun 2018 20:12:01 +0000

> well, it won't happen in py2.7, that's certain

http://pypy.org

Linux distributions and Python 2

Cyberax — Thu, 14 Jun 2018 17:21:42 +0000

JFYI, my native language is Russian so I’m acutely aware of encoding problems since so many of them were used for Cyrillic scripts. I can speak Ukrainian and understand Polish and Czech. I’d used to be able to speak German but haven’t practiced it in decades and lost most of it. Now I’m also studying Mandarin Chinese for a change.

Static annotations are nice, but are kinda beside the point.

Linux distributions and Python 2

LtWorf — Thu, 14 Jun 2018 09:13:00 +0000

> it's the same old Py2 with added inconvenience

Something tells me that you never ventured outside of writing English, because in any other language, having strings in unicode rather than ASCII is a huge improvement.

Then, Python3 has type annotations, which lets you use mypy to do some static checks on the code.

I wrote typedload (https://github.com/ltworf/typedload) to load json-like data into typed data structures, so that once loaded, you know your data has the correct types and can be safely be passed around.

My grandmother did not know how to use a computer, that doesn't mean that computers are useless.

Linux distributions and Python 2

danpb — Thu, 14 Jun 2018 08:53:22 +0000

This doesn't even remotely work for non-trivial programs, as there's no way you can have confidence that you've exercised all code paths simply by running it and seeing what breaks. You need to use an automated analysis and/or conversion tool first to check the entire codease, and after that try to execute it and if it works. Even then for non-trivial programs there's still a good chance it is broken. If the app has good unit test coverage this can help, but that's nowhere near foolproof.

Ideally everyone would none the less do the conversion to py3 and accept the risks/pain, but in reality I'd expect many companies will just keep around an ancient python2 version and not touch their apps/scripts :-(

Linux distributions and Python 2

kooky — Thu, 14 Jun 2018 08:44:34 +0000

I wouldn't say you get nothing.

I've moved several large Flask web apps from python2 to python3. It has been very satisfying.

We were getting quite a few `unicode` trouble tickets. Fix usually just to run as python3. Changes to code usually very minor and done in a few hours.

Linux distributions and Python 2

lambda — Thu, 14 Jun 2018 06:18:18 +0000

Which interpreter is the right one to use for the following code in an executable file named polyglot:

#!/usr/bin/python

print("2/3 = {}".format(2/3))

Will the output remain the same if one system decides to run it with a Python 2 interpreter and another runs it with a Python 3 interpreter?

Linux distributions and Python 2

kenshoen — Thu, 14 Jun 2018 05:19:04 +0000

They look at shebang line. #!python2 to 2, #!python3 to 3, #!python defaults to 2 (configurable).
https://docs.python.org/3/using/windows.html#customizing-...

Linux distributions and Python 2

lsl — Thu, 14 Jun 2018 03:55:19 +0000

> C had a moment not dissimilar to Python's, with the move from K&R to ANSI C.

The difference being that I can still use code written in the K&R dialect and link it together with new sources into a single program. Heck, I can even use K&R style as well as C11 features in the same translation unit. Sure, you get a -Wimplicit-int warning and it's a GCC thing but it works.

Linux distributions and Python 2

lambda — Thu, 14 Jun 2018 03:48:55 +0000

For high quality codebases with decent test coverage the migration is straightforward

Good thing that most code out there is a high-quality code base with decent test coverage then, right?

I just ported a 750 line script from Python 2 to Python 3; it's only fairly recently that I've found there are enough reasons to do so (in this case, availability of Python 3 and not Python 2 in the base install of our distro, plus the typing module and mypy to add static types), and enough barriers have been removed.

Here are the steps I went through to make sure the port was working:

Run 2to3 over the script
Run autopep8 to fix a few issues introduced by 2to3
Run pylint and flake8, fix up issues found
Add type annotations, check types with mypy, fix up type issues that had crept in
Write unit tests (this had originally been a quick script, so not covered by unit tests, but had grown enough that they'd be useful for this port), fix issues discovered by unit tests that still remained even after mypy and pylint fixups
Realize that I'd ported to Python 3.6, and target system was running 3.5, so set up tox and fixed up more things that differed between 3.5 and 3.6 (tests that worked in 3.6 due to deterministic dict ordering started failing non-deterministically in 3.6; thanks to tox for printing the random number seed on each test run for making that easier to reproduce and fix)
Actually try building my package and tests on the target system, have to fix more test issues due to an older version of pytest on the target system.
Finally actually run the result on the target system, and even after all of those steps above (pylint, flake8, mypy, unit tests with pretty good coverage) still have issues to fix

And this is for a fairly simple little script, dealing with some files which just contain ASCII text and ASCII filenames.

Our main codebase uses Twisted, using PB and Jelly to pass objects over the network. There were no automated tests when I started at the company, and while we've been trying to add tests for new features or code that we're touching, we still have pretty poor coverage. Additionally, it's is a networked application that also has to interact with networked filesystems on Windows, macOS, and GNU/Linux, which makes the encoding issues even more fun to deal with. Oh, and most of what is jellied and shared between systems consists of dicts, with different value types for different keys.

At some point, we are going to have to port this to Python 3, but I imagine it's going to take a long time, cause a lot of grief for the developers in the process, and still probably cause regressions for customers even if we're as thorough as we can be by adding unit tests, integration tests, and type annotations.

Linux distributions and Python 2

NAR — Thu, 14 Jun 2018 00:18:50 +0000

I guess the big problem will be site-specific scripts created years ago that run only once in a month, but if they fail, the sysadmin will get a phone call at 2AM.

Linux distributions and Python 2

kenmoffat — Wed, 13 Jun 2018 14:40:57 +0000

In the build system, it now invokes e.g. configure.py.

0:00.60 Creating Python environment
0:01.54 New python executable in /scratch/working/firefox-60.0.2/firefox-build-dir/_virtualenv/bin/python2.7
0:01.54 Also creating executable in /scratch/working/firefox-60.0.2/firefox-build-dir/_virtualenv/bin/python
0:01.54 Installing setuptools, pip, wheel...done.
0:02.27 running build_ext
0:02.27 building 'psutil._psutil_linux' extension

Linux distributions and Python 2

mgedmin — Wed, 13 Jun 2018 12:19:32 +0000

Where does Firefox use Python?

Linux distributions and Python 2

codewiz — Wed, 13 Jun 2018 11:56:25 +0000

And let's not forget _MBCS, hehe :-)

I guess Microsoft can be excused for the messy Unicode transition. They were among the first to attempt it, and at the time they were still saddled with support for a myriad of encodings from the DOS era. Oouch! :-)

But in 2008 there was no excuse whatsoever for not using the awesome UTF-8 encoding as the internal representation of strings. It was already obvious which way the wind was blowing, and pretty much every dynamic language transitioned to UTF-8, except Python. Actually, PHP 6 also tried moving to UTF-16, but after struggling with performance regressions and compatibility issues for several years, they finally realized it was a very poor decision and ditched the entire release. Then PHP 7 came out with UTF-8 strings.

Later programming languages like Go and Rust could have picked either UTF-8 or UTF-32 without compatibility concerns, and they still went with UTF-8 for simplicity and performance. You lose the O(1) random-access to Unicode codepoints, sure, but how often do you really do that? Whereas having to re-encode all text on the I/O boundary is a major inconvenience and causes all sort of round-trip issues.

Linux distributions and Python 2

excors — Wed, 13 Jun 2018 10:55:47 +0000

wchar_t is a bit awkward since it can't actually hold Unicode codepoints on Windows.

Win32 did go through an ANSI->Unicode(ish) switch (with Windows NT, I presume), and tried to make it easy for applications to migrate incrementally. If you start with char* and "strings" and strcmp() and MessageBoxA(), you can gradually replace it with TCHAR* and _T("strings") and _tcscmp and MessageBox, and it will compile exactly the same as before. Then you #define UNICODE and suddenly it all gets macroed into WCHAR*/L"strings"/wcscmp/MessageBoxW etc, and you probably get a load of build errors, and you fix some before getting bored and going back to ANSI mode. After enough iterations you might get something that compiles properly with the Unicode API. Then you probably keep the TCHAR/_T/etc macros - it seems standard practice is to use those instead of the Unicode-only versions, even if you know you're never going back to ANSI.

I guess the difference with Python is that Microsoft saw the transition as an unending process, and they knew they would have to support the ANSI APIs forever, so they accepted the ugly macros as the cost of supporting both. Python expected the transition to finish quickly so they could kill off Python 2, so it seems they focused more on designing a nice end state (with arguable success) and less on easing the transition between the two states.

Linux distributions and Python 2

codewiz — Wed, 13 Jun 2018 10:13:29 +0000

There's wchar_t, but it's a distinct type from char: http://en.cppreference.com/w/cpp/language/types#Character...

While the standard doesn't even bother saying whether char is signed or unsiged, no sane C compiler would dare switching overnight the representation of C strings. Incrementally switching a large codebase from 8bit charater strings to Unicode would be nearly impossible. Pure insanity! Which is exactly why the transition to Python3 has been going on and on for the past 10 years in spite of the considerable effort the community put into it :-)

Linux distributions and Python 2

gdt — Wed, 13 Jun 2018 09:08:03 +0000

Yeah, people still use last millennium's C versions.

C had a moment not dissimilar to Python's, with the move from K&R to ANSI C.

It was one of the defining moments in the development of free software. Some popular Unixen took the opportunity to charge for their ANSI C compilers, which lead people to discover GCC and then the GNU Project's other tools.

Linux distributions and Python 2

gevaerts — Wed, 13 Jun 2018 08:53:58 +0000

As far as I know, a C90 compiler can can have char be a Unicode codepoint just as well as a C98 one.

Linux distributions and Python 2

codewiz — Wed, 13 Jun 2018 07:54:12 +0000

I'd be surprised if anything shipped as part of current Linux distros still required to be built in C90 mode. It's there just in case, and it's very easy to support for gcc because, as other have noted, the C and C++ languages have never broken compatibility to the level that Python did.

If C98 suddenly switched the meaning of 'char' from being an 8-bit integer to a Unicode codepoint, you can bet that we'd still be using C90 today, officially supported or not.

Linux distributions and Python 2

kenmoffat — Tue, 12 Jun 2018 16:46:45 +0000

Yes, I was talking about building it. Last time I checked (probably about 6 weeks ago), python2 was required. If you want to link it to (some) system libs - particularly nss and system certificates - then you need to build it from source.

Linux distributions and Python 2

imphil — Tue, 12 Jun 2018 16:02:50 +0000

Where does Firefox need Python 2? Building Firefox from source might require Python 2, but I'm not aware of anything in Firefox that requires it at runtime. Ubuntu seems to agree: https://packages.ubuntu.com/bionic/firefox

Linux distributions and Python 2

k8to — Tue, 12 Jun 2018 15:24:17 +0000

That's determining which python already ran it, not which python it needs.
I don't believe it's very decidable.

Linux distributions and Python 2

rbanffy — Tue, 12 Jun 2018 14:56:49 +0000

One easy way to check is to remove /usr/bin/python and fix whatever breaks by calling /usr/bin/python2.

Anyone publishing software that depends on Python 2 to be at /usr/bin/python is just inviting a lot of future pain.

Linux distributions and Python 2

rbanffy — Tue, 12 Jun 2018 14:53:27 +0000

If the script is already compiled to a .pyc, you can probably find out which version of Python compiled it (I never tried that because, frankly, I'm very happy with having python and python3 on /usr/bin).