bytes vs. characters [LWN.net]

bytes vs. characters

Posted Apr 17, 2015 14:35 UTC (Fri) by intgr (subscriber, #39733)
In reply to: bytes vs. characters by Cyberax
Parent article: Report from the Python Language Summit

How is it a problem in Python that it insists on data to conform to a specification?

How do you think it should behave? Always return bytes? No, JSON specifies Unicode. Try Unicode first but fall back to bytes? No, that seems like it would be very surprising behavior and cause more bugs than it solves. AFAICT the alternatives are far worse.

The situation you're in is *caused* by implementations being too lenient and accepting junk as input, which is why the vendor has not noticed this issue before.

to post comments

bytes vs. characters

Posted Apr 18, 2015 11:03 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (5 responses)

So how do you put raw bytes into JSON then? How do you reasonably deal with people putting Latin-1 into comments on a website accessible via an API using JSON? Do you use an array of codepoints? Bytes? Personally, I like Python 2's json module which gives you a unicode object for UTF-8 and str for anything else (or pure ASCII). The problems there, however, come from other places such as subprocess.Popen.communicate choking on non-ASCII (so unicode object need encoded, but Latin-1 needed casted to bytes…which is the same as str, but still required for some reason), unicode strings can't be formatted with other unicode objects, but str can be (WTF logic is that?), and other pitfalls (of course, hidden until runtime, lucky me). It would be nice to move to Python 3 which seems like it fixes these problems, but this makes it sound like they just moved the ball around in some kind of shell game.

bytes vs. characters

Posted Apr 20, 2015 7:59 UTC (Mon) by zyga (subscriber, #81533) [Link] (4 responses)

You don't. There's no specification for putting random bytes in json.

At this time you just generate a stream of bytes (not python bytes, just bytes) that has some meaning that is only sensible to you and whoever consumes your byte stream. It's not json.

bytes vs. characters

Posted Apr 20, 2015 11:43 UTC (Mon) by Jonno (guest, #49613) [Link] (3 responses)

> You don't. There's no specification for putting random bytes in json.
Actually there is, it is called an "array of numbers", ie [ 97, 114, 114, 97, 121, 32, 111, 102, 32, 110, 117, 109, 98, 101, 114, 115, 0 ]

bytes vs. characters

Posted Apr 20, 2015 12:21 UTC (Mon) by bcopeland (subscriber, #51750) [Link]

For those, like me, who have a defect where they cannot resist converting ascii-valued hex or decimal numbers into their chr() equivalent, it all checks out.

bytes vs. characters

Posted Apr 20, 2015 13:19 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

The problem is that "no" implementations do this :( . Ruby's libraries (or at least whatever GitLab uses) will happily make non-utf-8 strings in the JSON it exports. And Python2 will happily import it as a str object. There is *lots* of code that will need changing for stricter parsers/generators.

bytes vs. characters

Posted Apr 21, 2015 4:12 UTC (Tue) by lsl (subscriber, #86508) [Link]

Go's JSON package does use those number arrays when appropriate.