Fedora ponders the Python 2 end game

Posted Aug 2, 2017 11:00 UTC (Wed) by niner (subscriber, #26151)
In reply to: Fedora ponders the Python 2 end game by mikapfl
Parent article: Fedora ponders the Python 2 end game

How on earth can Python 3 be considered "just works" when Python 3 was the thing that broke everything? How should one ever trust the Python core developers again after this massive screw up? Especially since they have not shown any sign of even acknowledging their mistake.

No, the answer is "it depends". Some names have already be mentioned. For "quick and dirty", "just works" and "takes backwards compatibility really serious" I'd personally just use Perl.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 11:55 UTC (Wed) by Kamilion (guest, #42576) [Link] (17 responses)

Huh.
*Holds up a copy of Perl 6 and Parrot Essentials (2003)*

*connects to VM*
~$ perl --version

This is perl 5, version 22, subversion 1 (v5.22.1) built for x86-linux-gnu-thread-multi (with 58 registered patches, see perl -V for more detail)
Copyright 1987-2015, Larry Wall

*scratches head*

Sooooooo, where's the perl6 I tried to learn 14 years ago, before resorting to learning python?

I mean, yeah, python3's taken a few years, but when you say you'd personally just use Perl, and after saying "Especially since they have not shown any sign of even acknowledging their mistake.", I find myself feeling somewhat amused and confused at the same time.

On the flip side, I try to take a lot of care that python code I touch runs in both 2.7 and 3.4+.
https://github.com/kamilion/customizer/commit/f0b5521ef30...

Fixing the compatibility issues between the versions using u'{}'.format(data) to ensure all data was unicode, even under python 2, wasn't really terribly difficult.
I've had more trouble trying to bring some old python 2.3 code back to life.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 12:05 UTC (Wed) by lkundrak (subscriber, #43452) [Link]

Perl 5 and (perhaps unfortunately named) Perl 6 are different languages. Unlike Python 2, Perl 5 is actively maintained and doing well.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 12:08 UTC (Wed) by niner (subscriber, #26151) [Link] (15 responses)

Perl 6 is a different language than Perl 5. It's a clean break but other than the Python 2 -> 3 transition, it at least brings real improvements like being able to use multiple CPU cores in Perl 6 code (no GIL), state of the art Unicode support rivaled only by Swift, reactive programming, real grammars instead of just regular expressions and lots more.

And again in contrast to Python 2 -> 3, Perl 5 code can still be used via https://github.com/niner/Inline-Perl5/
There's no need to port whole code bases to Perl 6. Perl 5 and Perl 6 code can live side by side in the same program, allowing for a piecemeal transition if so desired.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 12:23 UTC (Wed) by anselm (subscriber, #2796) [Link] (14 responses)

So the Perl 6 developers did everything right except when they tried to figure out a name for their new language.

Frankly, I think that the issues with the Python-2-to-3 transition have been wildly overhyped. It is quite possible to write code that runs with both Python 2.7 and Python 3.x, and there are automated tools to help with that. The main mistake the Python developers made was to underestimate the time it would take to adapt various popular third-party libraries, but by now this is pretty much finished. I personally went over to Python-3-only for new code a while ago and haven't looked back.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 12:30 UTC (Wed) by niner (subscriber, #26151) [Link] (4 responses)

At work we sit on roughly 1.5 million Python 2 expressions baked into this lovely templating language called DTML [1]. There's no way anyone would pay for porting those and no existing tool will help us with that. So come 2020, we can either just continue to use an unsupported Python 2.7, probably compiled from source, or try to get rid of Python entirely. Our system already compiles most of those to Perl code before executing. For getting rid of Python 2 completely however, we'll have to be able to compile whole Python scripts including function definitions (but luckily no class definitions). Lots of work, but at least then we will be able to stay backwards compatible forever. Both with old code and with existing know how.

[1] http://docs.zope.org/zope2/zope2book/DTML.html#using-pyth...

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 14:10 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

Oh. DTML. That sucks. I feel your pain :^(

As far as I'm concerned, Zope did look like a good idea in the early 2000s or so but quickly became a liability. Fortunately I managed to move off it soon afterward. I'm using Django today, which by now works great with Python 3.

Incidentally, there seem to be enough people in situations similar to yours that Python 2.7 support might not go away completely in 2020. It's just that the head Python guys said it won't be them providing that support. In effect, starting today you people have more than two years to pool your money and get Python 2.7 LTS organised. It's not as if the code base required loads of TLC to keep running, so this may be cheaper in the end than moving millions of lines of code to something else.

DTML

Posted Aug 2, 2017 14:21 UTC (Wed) by corbet (editor, #1) [Link]

Way back around then, when I was working to replace the initial PHP-based LWN site, I did a prototype in Zope and DTML. Then I got distracted for a few months. When I came back to it I couldn't understand anything I'd done and had to start over from the beginning trying to figure out what all the weird wrapper layers did. I concluded that it would always be that way and started over with something else... I've never regretted that decision.

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 16:35 UTC (Wed) by smoogen (subscriber, #97) [Link]

Or someone will offer a commercial branch of python-2.7 which has security updates applied to it for another decade. I mean that was the major reason for getting a paid version of Sun Fortran or C in the late 90's.. to keep that K&R and Fortran IV going :).

Fedora ponders the Python 2 end game

Posted Aug 2, 2017 18:54 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Python 2 will be maintained past 2020, there are too many projects dependent on it. RHEL 7 will be supported until 2027 at least and it contains Py2.

Fedora ponders the Python 2 end game

Posted Aug 4, 2017 1:49 UTC (Fri) by lsl (subscriber, #86508) [Link] (8 responses)

> The main mistake the Python developers made was to underestimate the time it would take to adapt various popular third-party libraries, but by now this is pretty much finished.

I don't think so. I still see Python programs (allegedly supporting Python 3) heavily shitting themselves upon encountering data that cannot be decoded to Unicode. We're talking external data here, like stuff coming in over the network or user-created file names.

Fedora ponders the Python 2 end game

Posted Aug 4, 2017 4:23 UTC (Fri) by smckay (guest, #103253) [Link] (7 responses)

Does Python 3 make it hard to handle malformed text? I mainly use Java and write backend code so the complaint about non-Unicode filenames is hard to understand. Does that happen a lot? Is it a client-side issue? For me the solution would be to tell ops to stop being cute and use normal filenames. :)

It does sound like the Unicode codecs have significant problems if exceptions are part of the default behavior. It's not like you can tell the socket/pipe/file to stop pulling your leg and cough up the <i>good</i> data. Rule #1 of text handling: do the best you can with what you're given and hope no one notices the ÃƒÆ.

Wrong file name encoding made easy

Posted Aug 4, 2017 5:17 UTC (Fri) by mbunkus (subscriber, #87248) [Link] (2 responses)

> I mainly use Java and write backend code so the complaint about non-Unicode filenames is hard to understand. Does that happen a lot?

It's really trivial to recreate. Take a non-ASCII file name on Windows, e.g. "möp.txt". Compress it into a ZIP, copy it to Linux and unzip it there. You'll end up with:

$ unzip the-zip.zip
Archive: the-zip.zip
inflating: mp.txt
$ ls -l
total 28
-rw-rw-r-- 1 mosu vj 18617 Aug 3 20:27 'm'$'\366''p.txt'
-rwxr--r-- 1 mosu vj 4272 Aug 4 07:07 the-zip.zip
$ rm m$'\366'p.txt
$ 7z x the-zip.zip
…snipped output…
$ ls -l
total 28
-rw-rw-r-- 1 mosu vj 18617 Aug 3 20:27 'm'$'\302\224''p.txt'
-rwxr--r-- 1 mosu vj 4272 Aug 4 07:07 the-zip.zip

The reason is simple: stupid file formats. There are no specs for file name encoding in ZIPs. There's no file name encoding indicator either. One could always use 7z, of course, where the specs state that file names be encoded in one of the Unicode encodings, but Windows doesn't come with support for 7z out of the box, but for ZIP. Aaaaand try getting people to use some new(ish) format and see how much success you have :(

Another thing that happens more often than I'd like is some mail program or other getting MIME wrong somehow so that an attachment containing non-ASCII characters in its file name being saved with the wrong encoding resulting in similar brokenness.

Wrong file name encoding made easy

Posted Aug 7, 2017 20:15 UTC (Mon) by cesarb (subscriber, #6266) [Link] (1 responses)

> There are no specs for file name encoding in ZIPs. There's no file name encoding indicator either.

Actually, there is. See appendix D of the ZIP spec at https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which says:

- If bit 11 is set, the filename is in UTF-8
- If bit 11 is unset, the filename is in CP437
- The UTF-8 filename can also be in extra record 0x7075

Wrong file name encoding made easy

Posted Aug 8, 2017 2:40 UTC (Tue) by smckay (guest, #103253) [Link]

Ah, so when we run into a non-compliant zip file it is time to barf and die because we are running Python 3. I think I understand now.

Fedora ponders the Python 2 end game

Posted Aug 4, 2017 8:36 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

The right idea is to avoid Unicode decoding altogether. Just treat input as a stream of bytes for as long as you can.

For example, I have recently struggled with a Py3-based proxy server that failed with one broken client that sends non-ASCII header names.

Fedora ponders the Python 2 end game

Posted Aug 5, 2017 21:14 UTC (Sat) by flussence (guest, #85566) [Link] (2 responses)

A lot of modern languages try to force the Unicode issue by trying to hide the data from the programmer with a stone wall of abstraction. I've learned from trying to use several of them that there's just no sane default behaviour for all strings. Latin-1 is bad because it forces you to jump through hoops to handle human-readable text correctly, Unicode is *worse* because it forces you to jump through hoops to handle machine-readable text correctly (humans are generally more forgiving parsers).

The most programmer-abusive thing I've tried to use lately is actually perl6's binary types. There's no concept of endianness so the 16/32 bit wide variants are completely unusable for I/O… and they *only* work with I/O functions. You can't pack, unpack, recast to array or any other kind of high level operation on them. The rest of the language is (mostly) sane, but this forgotten corner has all the readability and portability of disassembled SIMD code with the performance of said asm being emulated in a high level language.

Fedora ponders the Python 2 end game

Posted Aug 7, 2017 11:12 UTC (Mon) by niner (subscriber, #26151) [Link] (1 responses)

"and they *only* work with I/O functions. You can't pack, unpack, recast to array or any other kind of high level operation on them."

That's not exactly true:
> perl6 -e '"Ödögödöckö".encode.say'
utf8:0x<c3 96 64 c3 b6 67 c3 b6 64 c3 b6 63 6b c3 b6>

> perl6 -e '"Ödögödöckö".encode.List.say'
(195 150 100 195 182 103 195 182 100 195 182 99 107 195 182)

> perl6 -e 'use experimental :pack; pack("NNN", 1, 2, 3).say'
Buf:0x<00 00 00 01 00 00 00 02 00 00 00 03>

> perl6 -e 'use experimental :pack; pack("NNN", 1, 2, 3).unpack("NNN").say'
(1 2 3)

Fedora ponders the Python 2 end game

Posted Aug 9, 2017 20:58 UTC (Wed) by flussence (guest, #85566) [Link]

Those are the 8-bit variants, of course they work. People notice when something as fundamental as UTF-8 breaks (usually!)

I'm talking about things like this:
> perl6 -e 'use experimental :pack; buf32.new(0x10203040, 1, 2, 3, 4).unpack("N").say'
4538991231697411

Or, here's a basic “real world” example I just made up: Read two equal-size image files in farbfeld format, alpha composite them, and write out the result to a new file… what would idiomatic perl6 code for that look like? Probably shorter than this comment if these bits of the language worked, but they don't.

Sorry for what looks like nitpicking some obscure corner of the language, but I've seen a few too many people get burned out exploring these dark corners; they receive the silent treatment when they point out the language is getting in their way, and subsequently ragequit. There's a lot of this broken window syndrome outside of the cool-oneliner-demo APIs, and it's been like this since forever.

Fedora ponders the Python 2 end game

Posted Aug 3, 2017 10:03 UTC (Thu) by Otus (subscriber, #67685) [Link] (1 responses)

> How on earth can Python 3 be considered "just works" when Python 3 was the thing that broke everything?

Python 3 just works if you are starting from scratch.

Had Python 2 not had as much adoption as it did, the breakage would have been a good idea. As is, they should have deprecated one thing at a time in a manner that would have been backwards compatible for a couple of releases. (And left the meaningless changes, some of which have been since rolled back.)

> How should one ever trust the Python core developers again after this massive screw up?

That is the clincher. I hope they've learned the lesson.

Fedora ponders the Python 2 end game

Posted Dec 22, 2017 20:30 UTC (Fri) by togga (subscriber, #53103) [Link]

No. Python3 is the core problem here (along with end of life for Python2). At least for me, developing in python3 is cumbersome and full of practical issues. Python3 is just not a productive language for me, and therefore I'm seeking alternatives.