Fedora and Python 2

Correct me if I’m wrong, but Joel’s point in this article is that:

It does not make sense to have a string without knowing what encoding it uses. […] If you have a string, in memory, in a file, or in an email message, you have to know what encoding it is in or you cannot interpret it or display it to users correctly.

To paraphrase, if you have to display any kind of text to a human user, you (or your programming environment) must explicitly know what encoding to use to translate the byte streams you carry around into intelligible characters.

Now AFAIU, when people complain about the “encode/decode string-hell” they are not really disputing this. From what I gather, these people deplore that by default, various parts of Python 3’s standard library expect their inputs to be Unicode characters, in contexts where there is no reason for them to be.

Personally, while I enjoy Python 3 overall, I agree that the decision to have streams default to meat-world characters rather than bytes is debatable. Not every program has to deal with human-readable strings.

Let’s say though that we all collectively agreed that Python having a bias toward human text is a good thing: let’s assume that dealing with byte-streams that do not map to Unicode characters is so rare that having to sprinkle a few bs and .buffers here and there is not a deal-breaker.

Even then, Python’s approach to human text feels somewhat naive: lengths, indexing, iteration and comparison are all based on code points, which AFAIU do not really represent anything meaningful in meat-space.

For example, Python 3 thinks that 'é' != 'é' because one is 'e'+'\N{COMBINING ACUTE ACCENT}' and the other is '\N{LATIN SMALL LETTER E WITH ACUTE}'. My French AZERTY keyboard makes typing the latter straightforward; I understand that GTK applications make it easy to type the former with “e Control-Shift-U 301”.

I can’t think of a program geared toward human interaction that should consider these two strings different. Python does offer unicodedata.normalize() to solve this specific problem; must we rely on every text-handling Python program out there to make its input go through this function? Arguably, shouldn’t the language abstract this minutiae away from us?

tl;dr: While Joel’s article is a classic and a must-read, I’m not sure it addresses the problems raised by Python 3’s critics:

• the language’s preference toward meat-space characters adds hoops to jump through when dealing with genuine byte-streams;

• the language’s naive handling of these meat-space characters adds hoops to jump through when dealing with those too.

>>> unicodedata.normalize('NFD', u'\u0387') == u'\xB7'
True