2to3? [LWN.net]

2to3?

Posted May 30, 2025 21:34 UTC (Fri) by Cyberax (✭ supporter ✭, #52523)
In reply to: 2to3? by mb
Parent article: Recent disruptive changes from Setuptools

> I guess you probably only process texts with ASCII compatible encoding, right?

If you want, I can tell you the tales of DOS vs. DOS Alternative encodings for matrix printers. Or about KOI vs. Win1251 settings for the Linux console fonts.

The problem is that you can't really do anything meaningful with true strings, unless you:

1. Just work within ASCII.
2. Know EXACTLY what you're doing and have enough context information.

The 1. is trivially handled by UTF-8 encoding (ASCII stays ASCII), and it actually covers the majority of needs. And 2. requires you to be aware of the current locale (i->I or i->İ), permissible string splits, invisible characters, LTR switches, etc.

So Python ends up with the disadvantages of both approaches. Python strings are not just bags of bytes, they are some magic black boxes that mere mortals are not supposed to open. Yet they don't provide enough context information to do "interesting" stuff, like being able to correctly split strings at word breaks, or handle case conversion.

to post comments

2to3?

Posted May 30, 2025 21:43 UTC (Fri) by mb (subscriber, #50428) [Link] (3 responses)

Nope. The actual problem was that many developers only ever tested ascii and delivered their software if it was "good". And that's trivially true, just as you said.
Today that breaks.
And that's a good thing.
If you handle strings you *need* to be aware of (2). Always.

2to3?

Posted May 30, 2025 21:46 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Nope. The actual problem was that many developers only ever tested ascii and delivered their software if it was "good".

And I would take that over Py3 string mess. It could have been easily fixed by just setting the string encoding to utf-8.

> If you handle strings you *need* to be aware of (2). Always.

No, you don't. If you're only using string operations for protocol-level stuff, like splitting the HTTP headers on a ":" character, then you can just treat strings as bags of bytes. Same for filesystem operations.

2to3?

Posted May 30, 2025 21:49 UTC (Fri) by mb (subscriber, #50428) [Link] (1 responses)

yeah, right. If you don't use strings ("just treat strings as bags of bytes"), then you don't need to use strings.
That's trivially true and I don't see the problem. Just use bytes.

2to3?

Posted May 30, 2025 21:58 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> yeah, right. If you don't use strings ("just treat strings as bags of bytes"), then you don't need to use strings.

That is correct. The problem is that the type for string literals in Python is, well, a string. It pops up in a myriad of places in the standard library.

For example, the exception message is (you guessed it) a string.

> That's trivially true and I don't see the problem. Just use bytes.

Well, I recommend you to try your own advice.