LWN: Comments on "Fedora ponders the Python 2 end game"

Fedora ponders the Python 2 end game

togga — Fri, 22 Dec 2017 20:30:00 +0000

No. Python3 is the core problem here (along with end of life for Python2). At least for me, developing in python3 is cumbersome and full of practical issues. Python3 is just not a productive language for me, and therefore I'm seeking alternatives.

Fedora ponders the Python 2 end game

togga — Fri, 08 Dec 2017 22:41:03 +0000

With numpy announcing dropped python support etc this topic is hotter than ever.

"A strongly typed language usually makes for a poor glue language, IME, in terms of productivity."

My thinking is that this might be compensated by the quick compile time of go autogenerating the wrapping layer. This wrapping layer will be order of magnitude faster than python ctypes. With available debug data (introspection of c interfaces) it should be feasible?

Fedora ponders the Python 2 end game

nix — Fri, 01 Sep 2017 15:24:29 +0000

I said 'IIRC' because this too was from vague recollections. I'll ask around. :)

Fedora ponders the Python 2 end game

mgedmin — Tue, 22 Aug 2017 08:21:56 +0000

Older versions of Debian and Ubuntu did not have a python2, just a python and a python3.

Fedora ponders the Python 2 end game

pboddie — Mon, 21 Aug 2017 21:46:10 +0000

Do you have any more information about this Elite bug? Some enquiries have produced an account of another bug, but also some scepticism about one which needed three unavailable bytes to be fixed. Someone even asked Ian Bell for his recollections!

Fedora ponders the Python 2 end game

Wol — Sun, 13 Aug 2017 14:59:41 +0000

I think the default on gentoo is now python3. I know I've been using one of the MD scripts, and have had to edit the shebang specifically to call python2.

It would certainly make life easier if all distro-supplied stuff used a shebang explicitly to call the version it wanted. But that's a lot of work catching all instances ...

Cheers,
Wol

Fedora ponders the Python 2 end game

nix — Fri, 11 Aug 2017 22:08:19 +0000

Old enough to remember when software was completed? When was this?

I can think of two pieces of software that were completed. TeX, because Knuth decreed it (and bugfixes didn't stop even then), and BBC B Elite, which must be considered to eventually have been completed because there was literally no more RAM in which to fix the single remaining known bug: IIRC, it needed three bytes, and after many sweeps through the code, optimizing each time, there was no more fat to be wrung out of it.

Fedora ponders the Python 2 end game

Wol — Fri, 11 Aug 2017 16:27:34 +0000

And silly little changes to the spec cause major difficulties with the code. For example, try the following fortran code ...

FOR I = 1 TO 0
do something
NEXT I

How many times will that "do something"? If it's standards-compliant FORTRAN, it'll do it once. If it's standards-compliant Fortran, it won't do it at all.

Upgrading code between language specifications can be a nightmare of booby traps ...

Cheers,
Wol

Fedora ponders the Python 2 end game

dvdeug — Fri, 11 Aug 2017 06:12:17 +0000

As an admin, you're thrilled to provide support for an endless variety of systems? You don't mind installing ratfor and dealing with the problems everytime you upgrade the Fortran compiler? How about a Modula-3 compiler? Support for Java 1.0 class files that won't run on modern JVMs?

Old enough to remember when software was completed? When was this? When code was written for the Commodore 64 or some other long dead platform? Of what use is code that can't handle Unicode (or even 8-bit characters), or can't handle JPEGs and PNGs or can't handle IPv6? At no point in the history that I'm familiar with has stable software been the norm. The vast majority of software lived and died on short-lived platforms, and even well-written programs that hit the language/OS/platform lottery, if they didn't adapt to new file formats and new demands, died.

Fedora ponders the Python 2 end game

flussence — Wed, 09 Aug 2017 20:58:31 +0000

Those are the 8-bit variants, of course they work. People notice when something as fundamental as UTF-8 breaks (usually!)

I'm talking about things like this:
> perl6 -e 'use experimental :pack; buf32.new(0x10203040, 1, 2, 3, 4).unpack("N").say'
4538991231697411

Or, here's a basic “real world” example I just made up: Read two equal-size image files in farbfeld format, alpha composite them, and write out the result to a new file… what would idiomatic perl6 code for that look like? Probably shorter than this comment if these bits of the language worked, but they don't.

Sorry for what looks like nitpicking some obscure corner of the language, but I've seen a few too many people get burned out exploring these dark corners; they receive the silent treatment when they point out the language is getting in their way, and subsequently ragequit. There's a lot of this broken window syndrome outside of the cool-oneliner-demo APIs, and it's been like this since forever.

Wrong file name encoding made easy

smckay — Tue, 08 Aug 2017 02:40:32 +0000

Ah, so when we run into a non-compliant zip file it is time to barf and die because we are running Python 3. I think I understand now.

Wrong file name encoding made easy

cesarb — Mon, 07 Aug 2017 20:15:04 +0000

> There are no specs for file name encoding in ZIPs. There's no file name encoding indicator either.

Actually, there is. See appendix D of the ZIP spec at https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which says:

- If bit 11 is set, the filename is in UTF-8
- If bit 11 is unset, the filename is in CP437
- The UTF-8 filename can also be in extra record 0x7075

Fedora ponders the Python 2 end game

niner — Mon, 07 Aug 2017 11:12:17 +0000

"and they *only* work with I/O functions. You can't pack, unpack, recast to array or any other kind of high level operation on them."

That's not exactly true:
> perl6 -e '"Ödögödöckö".encode.say'
utf8:0x<c3 96 64 c3 b6 67 c3 b6 64 c3 b6 63 6b c3 b6>

> perl6 -e '"Ödögödöckö".encode.List.say'
(195 150 100 195 182 103 195 182 100 195 182 99 107 195 182)

> perl6 -e 'use experimental :pack; pack("NNN", 1, 2, 3).say'
Buf:0x<00 00 00 01 00 00 00 02 00 00 00 03>

> perl6 -e 'use experimental :pack; pack("NNN", 1, 2, 3).unpack("NNN").say'
(1 2 3)

Fedora ponders the Python 2 end game

bluebirch — Sun, 06 Aug 2017 16:50:28 +0000

Maybe python2 (and optionally /usr/bin/python) should point to pypy's python2.7. For most scripts especially the ones used be administrators should work fine with it. And pypy won't have a problem with security patches.

Fedora ponders the Python 2 end game

flussence — Sat, 05 Aug 2017 21:14:47 +0000

A lot of modern languages try to force the Unicode issue by trying to hide the data from the programmer with a stone wall of abstraction. I've learned from trying to use several of them that there's just no sane default behaviour for all strings. Latin-1 is bad because it forces you to jump through hoops to handle human-readable text correctly, Unicode is *worse* because it forces you to jump through hoops to handle machine-readable text correctly (humans are generally more forgiving parsers).

The most programmer-abusive thing I've tried to use lately is actually perl6's binary types. There's no concept of endianness so the 16/32 bit wide variants are completely unusable for I/O… and they *only* work with I/O functions. You can't pack, unpack, recast to array or any other kind of high level operation on them. The rest of the language is (mostly) sane, but this forgotten corner has all the readability and portability of disassembled SIMD code with the performance of said asm being emulated in a high level language.

Fedora ponders the Python 2 end game

Cyberax — Fri, 04 Aug 2017 08:36:35 +0000

The right idea is to avoid Unicode decoding altogether. Just treat input as a stream of bytes for as long as you can.

For example, I have recently struggled with a Py3-based proxy server that failed with one broken client that sends non-ASCII header names.

Fedora ponders the Python 2 end game

josh — Fri, 04 Aug 2017 05:33:39 +0000

That's how many current distributions do it these days, including Fedora. And Debian has an option to do that, if you install the "usrmerge" package.

Wrong file name encoding made easy

mbunkus — Fri, 04 Aug 2017 05:17:54 +0000

> I mainly use Java and write backend code so the complaint about non-Unicode filenames is hard to understand. Does that happen a lot?

It's really trivial to recreate. Take a non-ASCII file name on Windows, e.g. "möp.txt". Compress it into a ZIP, copy it to Linux and unzip it there. You'll end up with:

$ unzip the-zip.zip
Archive: the-zip.zip
inflating: mp.txt
$ ls -l
total 28
-rw-rw-r-- 1 mosu vj 18617 Aug 3 20:27 'm'$'\366''p.txt'
-rwxr--r-- 1 mosu vj 4272 Aug 4 07:07 the-zip.zip
$ rm m$'\366'p.txt
$ 7z x the-zip.zip
…snipped output…
$ ls -l
total 28
-rw-rw-r-- 1 mosu vj 18617 Aug 3 20:27 'm'$'\302\224''p.txt'
-rwxr--r-- 1 mosu vj 4272 Aug 4 07:07 the-zip.zip

The reason is simple: stupid file formats. There are no specs for file name encoding in ZIPs. There's no file name encoding indicator either. One could always use 7z, of course, where the specs state that file names be encoded in one of the Unicode encodings, but Windows doesn't come with support for 7z out of the box, but for ZIP. Aaaaand try getting people to use some new(ish) format and see how much success you have :(

Another thing that happens more often than I'd like is some mail program or other getting MIME wrong somehow so that an attachment containing non-ASCII characters in its file name being saved with the wrong encoding resulting in similar brokenness.

Fedora ponders the Python 2 end game

smckay — Fri, 04 Aug 2017 04:23:15 +0000

Does Python 3 make it hard to handle malformed text? I mainly use Java and write backend code so the complaint about non-Unicode filenames is hard to understand. Does that happen a lot? Is it a client-side issue? For me the solution would be to tell ops to stop being cute and use normal filenames. :)

It does sound like the Unicode codecs have significant problems if exceptions are part of the default behavior. It's not like you can tell the socket/pipe/file to stop pulling your leg and cough up the <i>good</i> data. Rule #1 of text handling: do the best you can with what you're given and hope no one notices the ÃƒÆ.

Fedora ponders the Python 2 end game

anatolik — Fri, 04 Aug 2017 04:12:51 +0000

> some other distributions don't provide "python2" and "python3" names

Which one does not do it and why? There is a PEP that says all python distributions should provide "versioned python" binary. Even macports has pythonX nowdays.

Fedora ponders the Python 2 end game

lsl — Fri, 04 Aug 2017 01:49:01 +0000

> The main mistake the Python developers made was to underestimate the time it would take to adapt various popular third-party libraries, but by now this is pretty much finished.

I don't think so. I still see Python programs (allegedly supporting Python 3) heavily shitting themselves upon encountering data that cannot be decoded to Unicode. We're talking external data here, like stuff coming in over the network or user-created file names.

Fedora ponders the Python 2 end game

lsl — Fri, 04 Aug 2017 01:23:12 +0000

> Go already had it's Heartbleed: see the Ticketbleed bug at https://blog.filippo.io/finding-ticketbleed/

Uhm, that is not a Go issue at all but a bug in the TLS implementation of some F5 load balancer appliances. You could just trigger the bug using the Go TLS library client-side as it uses TLS session identifers of different size than OpenSSL (which was probably the only thing the vendor tested with).

When you sent a 1-byte session ID to these F5 appliances they still assumed these were 32 bytes long (as that's what OpenSSL and NSS happen to use) and echoed back your session ID plus the 31 bytes immediately following it in memory.

Fedora ponders the Python 2 end game

karkhaz — Thu, 03 Aug 2017 20:38:12 +0000

Arch does it the other way (/bin is a link to /usr/bin)

$ find / -maxdepth 1 -type l -exec ls -l {} \;
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /lib64 -> usr/lib
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /lib -> usr/lib
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /sbin -> usr/bin
lrwxrwxrwx 1 root root 7 Mar 26 22:57 /bin -> usr/bin

Fedora ponders the Python 2 end game

hmh — Thu, 03 Aug 2017 20:17:08 +0000

The ones I know of have either /usr -> / or /usr/bin -> /bin symlinked, so /usr/bin/env will just work...

Fedora ponders the Python 2 end game

smoogen — Thu, 03 Aug 2017 16:18:57 +0000

I expect the point dropped is that every compiler toolkit will compile both for N years. The C compilers would work with both K&R C, ANSI C and C99 while emitting warnings. They would then drop K&R C or make it require an explicit flag. It then would only emit warning for mixing ANSI and C99. Then ANSI needed an explicite flag and then C99 only. All in all it pretty much took 15 years from C99 for various compilers to transition.

The same with Fortran. You could mix and match IV and 77 code. Then you needed a flag, then you needed a flag for F90 etc etc. And the same for C++ code. The transitions for each were slow and Azathoth knows how many compiler writers went mad having to work that kind of code. Which I expect that Guido was trying to avoid.

Fedora ponders the Python 2 end game

intgr — Thu, 03 Aug 2017 15:24:15 +0000

People seem to have the wrong assumption about paths in Python 3. Python does actually properly handle filenames that aren't valid UTF-8; they are escaped with certain Unicode codepoints: https://www.python.org/dev/peps/pep-0383/ (I guess that's like WTF-8 in Rust). I think that's a pretty good compromise: it does the right thing with properly encoded paths (nearly all paths are) but still remains functional with paths that aren't.

> On the very last screen it tries to show you "everything is done" message which is in KOI8-R instead of UTF-8 - with exception being thrown and whole installation rolled back. Just PERFECT handling of strings.

Yes, that's exactly the behavior I want. There was a bug in the program (or translation) and a good programming environment should immediately throw an error, rather than proceed with some unexpected behavior. Even environments that used to play very fast and loose with types and ignore errors, like MySQL and PHP, have recently became significantly stricter. Otherwise, in complex programs, you will end up with latent errors that are much harder to debug and often data loss.

> If you want robustness then python is not for you.

Erm, in one breath you complain about Python being too strict and now you complain that it's not robust?

Fedora ponders the Python 2 end game

ballombe — Thu, 03 Aug 2017 14:09:18 +0000

> It is anticipated that once the Python 2.7 branch is no longer receiving even security updates, we will actively recommend against platforms providing a Python 2.7 stack at all, let alone as the default target of the unqualified "python" command.

The only problem with python3 is the attitude of the python3 developers toward python2. That they do not want to maintain it beyond 2020 is quite understandable. But trying to prevent other people to use it and keep maintaining it is not in the spirit of free software.

Remove that, and everybody would applaud python3 as the new interesting language to look at instead of the thing which is forced down their throat.

Fedora ponders the Python 2 end game

rossburton — Thu, 03 Aug 2017 14:04:13 +0000

For anyone wondering what other people are up to, I've just written up my plans for the Yocto Project at https://wiki.yoctoproject.org/wiki/NoMorePython2.

Because Python 2 is on the way out, and there's really no point in building multiple versions of Python if we can avoid it, we're trying to remove all use of Python 2 from OpenEmbedded Core sooner rather than later.

Thanks to Arch forcing the issue and installing Python 3 as /usr/bin/python most packages are happy to use either 2 or 3 these days. There's just a few tools we use that need Python 2: Qemu, APR, and bmaptools come to mind.

Fedora ponders the Python 2 end game

niner — Thu, 03 Aug 2017 13:45:19 +0000

Funny that you bring this up in a news post that's dealing with "python2/3" vs. "python" command name. What do you hope to gain by shelling out to openssl instead of dynamically linking? The binary still has an interface that may or may not change in incompatible ways. At least for dynamic libraries there's support for versioning. As the article demonstrates, there is no such thing for system commands.

Fedora ponders the Python 2 end game

niner — Thu, 03 Aug 2017 13:25:24 +0000

How can Perl 6 be worse when it's quite possible to do a piecemeal upgrade of a codebase from Perl 5 to Perl 6? If such a change is even desired instead of just combining the best parts of both languages. No Perl 5 developer was left in the rain and those who want, can use Perl 6. What about this was in any way worse than leaving people who cannot upgrade behind and forcing countless pointless man years of effort for the rest including distributions?

Fedora ponders the Python 2 end game

ewan — Thu, 03 Aug 2017 12:45:38 +0000

I'm now wondering whether the collective wisdom in this thread is codified anywhere?

It's somewhat codified in PEP-0394, which advises on how python should be installed (in short, don't make 'python' point to Python 3) and how to cope with the fact that it might (the less than completely helpful suggestion to make anything that just asks for 'python' be compatible with both Python 2 and Python 3).

Fedora ponders the Python 2 end game

Otus — Thu, 03 Aug 2017 10:03:09 +0000

> How on earth can Python 3 be considered "just works" when Python 3 was the thing that broke everything?

Python 3 just works if you are starting from scratch.

Had Python 2 not had as much adoption as it did, the breakage would have been a good idea. As is, they should have deprecated one thing at a time in a manner that would have been backwards compatible for a couple of releases. (And left the meaningless changes, some of which have been since rolled back.)

> How should one ever trust the Python core developers again after this massive screw up?

That is the clincher. I hope they've learned the lesson.

Fedora ponders the Python 2 end game

mpr22 — Thu, 03 Aug 2017 09:31:30 +0000

"Normal" C code is emphatically not C++ code, because normal C code often uses malloc() and seldom explicitly casts the return value to the desired pointer type because C's implicit conversion rules say that void * is implicitly castable to any other pointer type and programmers are often lazy. C++ discarded that implicit conversion.

Fedora ponders the Python 2 end game

roc — Thu, 03 Aug 2017 07:51:10 +0000

WTF-8 was not really "added" to Rust. There's a crate for it, that's all.

OTOH OsString has been part of Rust for a long time exactly because sometimes you need to deal with weird non-Unicode platform strings.

Fedora ponders the Python 2 end game

mjthayer — Thu, 03 Aug 2017 07:46:26 +0000

Having posted that and tried to anticipate the responses, I have to admit that I don't know whether shelling out to openssl(1) is really such a big gain over dynamically linking in most Linux use cases. The main one I see where it is is when you are shipping a binary outside of the distribution and don't want to support multiple openssl shared library ABI versions.

Fedora ponders the Python 2 end game

mjthayer — Thu, 03 Aug 2017 07:10:41 +0000

The thought that always goes through my head when I hear this discussion is "why not link statically but shell out to openssl(1) for encrypted connections". I am sure there is a good reason, which wiser people than me (I am no openssl expert, neither the library nor the command line tool) will tell me. This is not specific to openssl of course. It applies just as much to image format translation.

Generally, there must be a security risk threshold below which static linking can make sense. If you are at risk from a vulnerability in a dependency, you are presumably at risk from vulnerabilities in your own code too, so you have to be vigilant and to do updates from time to time anyway. The point where most updates are due to security issues in components is probably the point where the threshold has been passed.

Fedora ponders the Python 2 end game

khim — Thu, 03 Aug 2017 02:20:22 +0000

Of course I may not be clever enough to appreciate how “awful” Python's string handling really is.

My favorite example was Anaconda few years back. Pick text installer (because you are dealing with small VM), pick Russian language and do everything. On the very last screen it tries to show you "everything is done" message which is in KOI8-R instead of UTF-8 - with exception being thrown and whole installation rolled back. Just PERFECT handling of strings.

OTOH, I don't really care about WTF-8 in rust nor what Go considers a string because (so far) I'm not using either of those languages, and have no plans to do so in the foreseeable future.

That's Ok. If your goal is scripts which kinda-sorta-work-if-you-are-lucky then python or, heck, even bash work. If you want robustness then python is not for you.

Fedora ponders the Python 2 end game

anselm — Thu, 03 Aug 2017 00:37:28 +0000

I can only speak for myself, but I'm way happier with strings (and byte sequences) in Python 3 than I used to be with strings (and Unicode strings) in Python 2. They pretty much seem to do what I expect them to do, and given a little care it is reasonably easy to write programs that work. Of course I may not be clever enough to appreciate how “awful” Python's string handling really is.

OTOH, I don't really care about WTF-8 in rust nor what Go considers a string because (so far) I'm not using either of those languages, and have no plans to do so in the foreseeable future.

Fedora ponders the Python 2 end game

khim — Thu, 03 Aug 2017 00:10:48 +0000

Personally I prefer Python 3 because, among other things, strings work a lot better than they used to in Python 2.

Actually situation with strings is awful in python2 is awful and python3 made it even worse. Why do you think WTF-8 was added to rust? Why to you think Go still considers strings a sequence of bytes with no string attached? World where only nice unicode strings exist is an utopia! That's why they were forced to throw away the notion that file names are strings and introduced path-like objects! And I'm sure it's not the end.

Fedora ponders the Python 2 end game

Cyberax — Thu, 03 Aug 2017 00:06:42 +0000

C vs. C++ breakage happened gradually, early C++ versions were pretty much "C with classes". I've seen million line C applications being translated into C++ by simply renaming the files and doing a few easy changes. And C is a compiled language, so that helps a lot.

And if everything else fails, you can always #include C-based API with minimum fuss even in modern C++ using 'extern "C"{}' blocks.

There's nothing comparable in Python world. The transition was abrupt and it required quite a lot of changes, and being an interpreted language you actually have to test everything.