Report from the Python Language Summit [LWN.net]

Report from the Python Language Summit

Posted Apr 15, 2015 14:22 UTC (Wed) by jezuch (subscriber, #52988)
In reply to: Report from the Python Language Summit by nix
Parent article: Report from the Python Language Summit

> am finding the bytes / char distinction *really useful*.

Frankly, this usefulness is rather obvious to anyone living in non-ASCII world. When you are using ą, ę, ź, ć, etc. daily (and it's still just a variant of the Latin script), you learn pretty quickly about character encodings. The hard way, usually ;)

to post comments

Report from the Python Language Summit

Posted Apr 15, 2015 15:01 UTC (Wed) by cesarb (subscriber, #6266) [Link]

> Frankly, this usefulness is rather obvious to anyone living in non-ASCII world. When you are using ą, ę, ź, ć, etc. daily (and it's still just a variant of the Latin script), you learn pretty quickly about character encodings.

It's even more useful when diacritics are rare. If you use diacritics all the time, encode/decode bugs are found quickly. If you don't, encode/decode bugs will remain hidden, until one day they blow up as one of the dreaded Unicode encode/decode exceptions (or worse, show up as mojibake).

Report from the Python Language Summit

Posted Apr 15, 2015 20:25 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

I live in a non-ASCII world. My keyboard has four layouts and I routinely us languages with non-Latin (or Latin-with-diacritics) scripts.

Yet I find the whole char/byte distinction to be extremely moronic.

Report from the Python Language Summit

Posted Apr 16, 2015 11:34 UTC (Thu) by HelloWorld (guest, #56129) [Link] (5 responses)

> Yet I find the whole char/byte distinction to be extremely moronic.
Yes, you have stated that many times, and the response is still the same: You're confused. Bytes aren't Characters and Characters aren't bytes, period. It's as simple as that.

Report from the Python Language Summit

Posted Apr 16, 2015 18:11 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

So how many characters are in composite symbols?

The only sane and modern way to do Unicode is UTF-8.

Report from the Python Language Summit

Posted Apr 16, 2015 19:27 UTC (Thu) by HelloWorld (guest, #56129) [Link] (1 responses)

> So how many characters are in composite symbols?
What does that have to do with the fact that bytes are not text/strings/characters?

> The only sane and modern way to do Unicode is UTF-8.
Regardless of whether this is true or not, there is a lot of data in all kinds of encodings, and developers had better think about which one they are going to use when reading that data.

Report from the Python Language Summit

Posted Apr 16, 2015 22:40 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> What does that have to do with the fact that bytes are not text/strings/characters?
The fact that sequences of UCS-4 codepoints are also not text/string/characters, just as sequences of raw bytes.

> Regardless of whether this is true or not, there is a lot of data in all kinds of encodings, and developers had better think about which one they are going to use when reading that data.
Python3 practically forces one to transcode data from one format to another all the time for no specific reason.

Report from the Python Language Summit

Posted Apr 20, 2015 10:38 UTC (Mon) by niner (guest, #26151) [Link]

"So how many characters are in composite symbols?"

Characters? I'd say one. I can definitely say (as far as I understand this anyway) that it's one grapheme and one or more code points.

Perl 6 will deal with strings as sequences of Normalized Form Graphemes (NFG). There's a very interested blog post about what this means:

https://6guts.wordpress.com/2015/04/12/this-week-unicode-...

I guess the only two sane ways of handling Unicode are:
* be completely agnostic and treat stings as opaque sequences of bytes, or
* go all in and work with graphemes whenever possible.

Report from the Python Language Summit

Posted Apr 21, 2015 22:26 UTC (Tue) by nix (subscriber, #2304) [Link]

I'll just tell all the existing systems out there to use UTF-8, even if they don't. I'm sure I can find a way to jam all of Unicode onto the Adafruit-based display board my Python code is talking to: it has a whole 64K of flash and 2K of RAM! I'm sure I can fit glyphs for all of Unicode in there and still have space for everything else it has to do!

No, not everything can use UTF-8, even in an ideal world. And such systems will *always* use different encodings, so to talk to such systems Python's enforced conversion is extremely valuable. And even when you're not, and the system you are talking to uses UTF-8 or some other Unicode variant, the enforced conversion is *still* valuable because it forces you to think about what encoding is in use, and amazingly often it's not straight UTF-8, or it's UTF-8 with extra requirements such as needing to be canonicalized or decanonicalized in a particular way or "oops we didn't say but experimentation makes it clear that $strange_canonicalization is the only way to go". (I have seen all of these on real systems, along with people claiming UTF-8 but meaning UCS-16 because they didn't know there was a difference, and vice versa -- and, in the latter cases, cursed them.)

Report from the Python Language Summit

Posted Apr 16, 2015 21:28 UTC (Thu) by flussence (guest, #85566) [Link] (3 responses)

>Yet I find the whole char/byte distinction to be extremely moronic.

It is, but I'd say because nobody has a clue how to define "char". It can mean all sorts of things depending on where it's used:

* 1 byte in a legacy encoding (or C)
* If you're using a half-baked library, it's 2 or 4 bytes UTF-16. (Qt4 falls into the "completely baked" category as it'll let you backspace over half an emoji character.)
* If you're lucky, someone actually implemented Unicode correctly and a "character" is a variable length sequence of bytes encoding a full ISO-10646 codepoint. Such as U+00C7, which is the character "Ç". Or U+0327, which is a squiggly line and definitely not a character.
* The only sensible and correct definition: a character is the thing you would write by hand — "Ḉ".length == "Ḉ".length == "Ḉ".length — almost no software uses this definition.

Report from the Python Language Summit

Posted Apr 16, 2015 22:38 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> It is, but I'd say because nobody has a clue how to define "char". It can mean all sorts of things depending on where it's used
Correct. And complex scripts or complex characters make it even more complicated.

That's why I violently oppose the definition: "UCS-4 codepoint is a character or GTFO", which Python3 tries to enforce.

Report from the Python Language Summit

Posted Apr 21, 2015 22:31 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

Python says that its internal encoding is Unicode, but the nice thing about the enforced mapping to bytes to get it out anywhere else is that good code need not rely on this at all. As long as you always transcode to/from bytes when leaving the Python world, you can *completely ignore* what internal encoding the thing is using (or, rather, use the Unicode stuff like properties etc as needed inside your code, in the happy knowledge that this will not affect transfer to external systems at all, no matter what encoding they use, as long as a codec exists. And writing codecs is *really* not hard, at least not if you don't care much about performance, e.g. for one-off mappings where no codec exists in Python yet :) )

Report from the Python Language Summit

Posted Apr 21, 2015 23:06 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

A couple of times I had to resort to hacks like putting raw bytes into the LSB of UCS-4 codepoints.

I still think that treating strings as sequences of UTF-8 characters and/or bytes is the best possible way. Enforced UCS-4 rarely helps.