Fedora and Python 2

They got luckier. The user-visible part of UTF-8 handling in Perl vs. Python2 is roughly comparable in terms of interoperability of non-UTF8 vs. UTF8 strings, except that they were a bit more clever with their default encodings so that "simple" things you may want to do "just work".

However, I have personally converted a largish console-based Perl5 codebase from running on Latin-1 (or Latin-8 to be exact, as soon as the € happened) to UTF-8. Let me tell you that this is an exercise in hunting down annoying hard-to-reproduce bugs that you wouldn't wish on your worst enemy. We had the whole gamut – from mojibake in the database through strings which crashed the interpreter when printed to tearing our hair out trying to write code that works correctly in both locales. The only thing that saved us is the fact that you can unscramble real-world mixed UTF-8/Latin8 content safely, thanks to the way UTF-8 is encoded.

Python3's way of strict separation between bytes and strings may be more annoying when you start off, esp. on Windows, but IMHO it's a whole lot easier to make sure that the end result is actually correct.

There certainly are examples where Python maintainers tried to be far cleverer than they needed to be, and then (because humans aren't clever, we're dumb, that's lesson #1) this blew up in their faces.

Python's TLS implementation wants to check whether the TLS server has presented a certificate that's valid for the name of the TLS server you're trying to connect to. This makes sense, certainly as a default, if I said I wanted a TLS connection to foo.example.com, I definitely shouldn't need to roll my own certificate validator, or even explicitly say "Wait, this connection I have, is it _really_ to foo.example.com?" because duh, of course I want those things, let the handful of people who don't want checks ask *not* to check.

And all the people involved in designing this stuff at the IETF stage were conscious that this problem could be hard, and we don't want hard because this is a security system. So the certificates have DNS A-labels inside them. All you need to do is match the DNS name in the certificate (written with A-labels, ie ASCII) against the DNS name you looked up in your DNS system, which is also written with A-labels, ie ASCII. This is really boring and easy. Users don't necessarily understand A-labels, they might be gibberish, but the presentation layer is quite separate, securing that is a UX problem and not relevant to TLS or other low level components. All the tricky human stuff is pushed into the layer that was already dealing with humans, everything that's machine-to-machine needn't care about human cultural complexities like language.

Too simple for Python though, they decided to handle everything with U-labels so they can mark all the types "str". So now suddenly this low-level bit banging code that's supposed to securely move packets ends up with the entire i18n system baked into it, and inherits mysterious presentation layer related bugs. Problem with the matching? Oh sorry, you need to go fix this whole other Python sub-system that has nothing whatsoever to do with TLS ...

Eventually, literally in February this year, sanity finally prevailed and the latest Python 3 actually just does what the RFCs said it ought to do in the first place, massively simplifying the code _and_ making it more correct. Most users will never notice, because this was after all the Right Thing anyway.

python 3 defaults the text APIs to use unicode. That works for me.

If you know that its bytes that you care about use the APIs that give you the bytes.

You can get the env vars as bytes, see file system names as bytes and read files in bytes.

I do not understand the criticism.

Barry

The really egregious one is running commands for their output on Windows.

It's very common on that platform that the encoding of command output is special to the command. Some things follow the system selected locale, some don't. Some even mix fixed strings in ascii with filename based output in utf-16 and crap like that.

Python3 would be okay for this if bytes were as full-featured as Python2 strs used to be, but they're really not. A lot of the standard library really insists on strings.

(Mainly I'm thinking about network protocols here. http, ssl, etc)

>Can you think of any other situation in free software where a team that has abandoned a codebase has not just discouraged someone willing from taking over maintenance

That's not what happened here. Maintenance of the codebase is perfectly fine. The name however is not free to use for forks. That is a situation that is quite common in Free software projects.

What do you mean, "abandoned"? It's not. The Python 2 branch contains a whole lot more changes (looking at the last half year) than Tauthon.

>Can you think of any other situation in free software where a team that has abandoned a codebase has not just discouraged someone willing from taking over maintenance, but in fact used legal pressure to prevent it?
A few years back the libav gang attempted to sue the legitimate FFmpeg project out of existence over its use of the logo. They failed because they didn't actually own any rights to the image to begin with; a third party contributed it. Hasn't stopped them using it, mind you.

This Python naming dispute isn't an act of malice - it's a simple trademark defence. Mozilla does exactly the same thing, which is why we have IceCat, IceWeasel, PaleMoon, Seamonkey etc.

PHP has enough problems that personally I wouldn't use it as a positive example for anything whatsoever, but that's just me. IMHO the PHP6-full-of-Unicode idea was doomed to fail the second they decided to base their work on UTF-16. It's now umpteen years later (well, not quite, but you get the idea) and PHP still doesn't have any better support for Unicode than 5.whatever. That's not "fixing" in my book.

Python2's indiscriminate support for mixing test and binary data caused a ton of hard to track bugs. "the day Django stops supporting Python 2 they will be able to rip out a ton of code that exists purely because it was so easy to mix text and binary data and get it wrong" (cited from the page you linked to). Yes it's annoying to suddenly be required to think about whether your data ist binary or text, and if the latter what encoding, but frankly you should have done that in the first place – the real problem is Python2's inability to support that distinction.

And yes there were some mistakes in early Python 3.x versions, but "acknowledge and fix mistakes" happened, e.g. by ditching 2to3 in favor of six and by extending support for Py2 – not by throwing in the towel because the whole thing was doomed to fail from the beginning. It obviously isn't; Python3 uptake may be slower than expected initially, but it's still way faster than that of Perl6. :-P

If Go is a better fit than Python2/3 for whatever it is you want to do, well, go for it. I'm not going to infer any perceived inferiority of Python2 vs. Python3 from that. It's probably more like "if we need to spend some effort to switch over anyway, let's examine what else might work even better".

There's certainly an argument for normalization, but every person annoyed by Python 3 would likely be more pissed off if, by default, it silently changed text when reading it in. Imagine a text editor where you opened your new novel, "Nous étions à l’étude, quand le Proviseur entra, ..." and changed it to "Nous étudiions, quand le Directeur entra, ..." and fed it back to git to discover a diff that changed every single line in the file.

It's interesting how many languages get this wrong. For instance, Java doesn't even give you code points, much less grapheme clusters. Instead, it gives you 16-bit “char” values (“code units” in Unicode-speak) and then you have some methods like codePointAt that give you the code point at some (char-based) position in the string. And when you want to iterate over the code points in a string, I don't know how you're supposed to get from one index to the next, i.e. whether you need to increase the index by one or by two. It might be that you need to compare against Character.MAX_LOW_SURROGATE, but I'm not sure… Needless to say it doesn't help you dealing with grapheme clusters at all, apparently you're supposed to use third-party libraries like icu4j. All in all, it's a clusterfuck (ba-dum tss!)

> the language’s naive handling of these meat-space characters adds hoops to jump through when dealing with those too.

You need to normalize Unicode before doing meaningful things with it. That's a given in any programming language.

You might find fault with the people who invented Unicode. Blaming your (non-)choice of programming language isn't going to help, except that I can think of lots of ways to make it worse. Just look at Java.

>>> unicodedata.normalize('NFD', u'\u0387') == u'\xB7'
True

My point with the Joel's article was that in my experience large part of people complaining about Python 3 encoding/decoding hell are those who still believe that "string is bunch of bytes" is enough, because they live in the bubble of languages where it is enough (i.e., English and Western European languages). I have converted recently M2Crypto to be py2k/py3k-straddling and I had no problems with Unicode encoding/decoding. What I had problems with, and plenty plenty of them, was that completely messy str/unicode/bytes py2k mess completely confused the real situation. Instead of blaming py3k, I keep blaming py2k and those programmers that their cuckoo-land of "character is one byte" delusion.

And yes, I agree that the implementation in py3k is not perfect, conversion between on-wire eight-bit-per-character to proper str is sometimes problematic, but a lot of work has been spent on it already and the situation is not that bleak, I would call it whatever hell.

Certainly, comparing to the disaster py2k was, py3k is huge improvement.