bytes vs. characters [LWN.net]

bytes vs. characters

Posted Apr 16, 2015 22:36 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
In reply to: bytes vs. characters by zyga
Parent article: Report from the Python Language Summit

> If you have mixed encoding just frelling use BYTES as that's what you are reading anyway. Bytes. Use bytes and be happy.
Except that you can't do it.

For example, JSON decoder in Python3 _insists_ on decoding strings as strings. Even if they have invalid UTF-8 data. It's bad, but such services do exist out there and sometimes you have to work with them.

Ditto for HTTP headers.

to post comments

bytes vs. characters

Posted Apr 16, 2015 23:15 UTC (Thu) by dbaker (guest, #89236) [Link] (1 responses)

> For example, JSON decoder in Python3 _insists_ on decoding strings as strings. Even if they have invalid UTF-8 data. It's bad, but such services do exist out there and sometimes you have to work with them.

Because the JSON spec requires that strings must be unicode?

Can't you just write a custom object_hook function to pass to the decoder to solve your problem?

bytes vs. characters

Posted Apr 16, 2015 23:19 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> Because the JSON spec requires that strings must be unicode?
Yes, but the reality outside is a little bit different.

> Can't you just write a custom object_hook function to pass to the decoder to solve your problem?
No, 'encoder' parameter is ignored in json.loads and everything else already gets decoded strings.

We simply switched to a third-party library instead.

bytes vs. characters

Posted Apr 17, 2015 7:15 UTC (Fri) by zyga (subscriber, #81533) [Link] (1 responses)

You can decode("UTF-8", "ignore") or something else to "coerce" it to some form of text though I really do value the sanity of that. Just fix your data sources. Even if you use some 3rd party library it's not going to make any of that "json"-like thing work with other libraries (I assume that other customers/APIs need to read it).

HTTP headers are a perfect example of binary data. Handling them as unicode text is broken IMHO. You can just use byte processing for everything there and Python 3.4, AFAIR, fixed some last gripes about lack of formatting support for edge cases like that.

bytes vs. characters

Posted Apr 17, 2015 7:22 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> You can decode("UTF-8", "ignore") or something else to "coerce" it to some form of text though I really do value the sanity of that.
I think it will still be broken. There's a workaround that simply stores binary bytes in the lower byte of UCS-4 codepoints and it sorta works.

I'd love to fix these data sources, but they're out of my control. The vendor knows about it and they plan to base64 binary data in the future, but for now I have to work with what I have.

> You can just use byte processing for everything there and Python 3.4
Not exactly. Most of the built-in library can be used with byte sequences, but third-party libraries are often too careless.

I've fixed tons of code like this:

>def blah(p):
> if fail_to_do_something(p):
> raise SomeException(u"Failed to frobnicate %s!" % p)

It mostly works as is, but occasionally it doesn't.

bytes vs. characters

Posted Apr 17, 2015 14:35 UTC (Fri) by intgr (subscriber, #39733) [Link] (6 responses)

How is it a problem in Python that it insists on data to conform to a specification?

How do you think it should behave? Always return bytes? No, JSON specifies Unicode. Try Unicode first but fall back to bytes? No, that seems like it would be very surprising behavior and cause more bugs than it solves. AFAICT the alternatives are far worse.

The situation you're in is *caused* by implementations being too lenient and accepting junk as input, which is why the vendor has not noticed this issue before.

bytes vs. characters

Posted Apr 18, 2015 11:03 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (5 responses)

So how do you put raw bytes into JSON then? How do you reasonably deal with people putting Latin-1 into comments on a website accessible via an API using JSON? Do you use an array of codepoints? Bytes? Personally, I like Python 2's json module which gives you a unicode object for UTF-8 and str for anything else (or pure ASCII). The problems there, however, come from other places such as subprocess.Popen.communicate choking on non-ASCII (so unicode object need encoded, but Latin-1 needed casted to bytes…which is the same as str, but still required for some reason), unicode strings can't be formatted with other unicode objects, but str can be (WTF logic is that?), and other pitfalls (of course, hidden until runtime, lucky me). It would be nice to move to Python 3 which seems like it fixes these problems, but this makes it sound like they just moved the ball around in some kind of shell game.

bytes vs. characters

Posted Apr 20, 2015 7:59 UTC (Mon) by zyga (subscriber, #81533) [Link] (4 responses)

You don't. There's no specification for putting random bytes in json.

At this time you just generate a stream of bytes (not python bytes, just bytes) that has some meaning that is only sensible to you and whoever consumes your byte stream. It's not json.

bytes vs. characters

Posted Apr 20, 2015 11:43 UTC (Mon) by Jonno (guest, #49613) [Link] (3 responses)

> You don't. There's no specification for putting random bytes in json.
Actually there is, it is called an "array of numbers", ie [ 97, 114, 114, 97, 121, 32, 111, 102, 32, 110, 117, 109, 98, 101, 114, 115, 0 ]

bytes vs. characters

Posted Apr 20, 2015 12:21 UTC (Mon) by bcopeland (subscriber, #51750) [Link]

For those, like me, who have a defect where they cannot resist converting ascii-valued hex or decimal numbers into their chr() equivalent, it all checks out.

bytes vs. characters

Posted Apr 20, 2015 13:19 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

The problem is that "no" implementations do this :( . Ruby's libraries (or at least whatever GitLab uses) will happily make non-utf-8 strings in the JSON it exports. And Python2 will happily import it as a str object. There is *lots* of code that will need changing for stricter parsers/generators.

bytes vs. characters

Posted Apr 21, 2015 4:12 UTC (Tue) by lsl (subscriber, #86508) [Link]

Go's JSON package does use those number arrays when appropriate.