|
|
Log in / Subscribe / Register

2to3?

2to3?

Posted May 29, 2025 21:49 UTC (Thu) by zahlman (guest, #175387)
In reply to: 2to3? by Cyberax
Parent article: Recent disruptive changes from Setuptools

Python 3 before 3.5 supported concatenation of bytes to bytes (and of course str to str); Python 3 today still does not support concatenation of bytes to str or vice-versa, and won't going forward. The bytes type also still doesn't have a .format method, although minds might more conceivably be changed about that.

PEP 461 is about the % operator for formatting, which still doesn't allow mixed types - it just allows you to have, for example, a byte with value 37 followed by a byte with value 115 in a bytes (or bytearray) and then replace those with a different sequence of bytes.

It's a little hard to imagine how an error handler failing to convert between str and bytes could cause such a headache - because an error handler would presumably be formatting *an error message*, which means the left-hand side should clearly be a string *literal*. And in Python 3 before 3.5, just as today, formatting a bytes into that with % would simply simply give the wrong result (with a 'b' prefix and quotes and escapes). And once you see badly formatted error messages in the logs, you just track them down to the handler according to the template they're formatted into. (Supposing, for the sake of argument, that the formatting isn't actually beneficial in determining what went wrong in the code that had the original error.)

If you mean that code decoded byte data into strings in order to be formatted, and then wasn't properly converted back to bytes, I sympathize; but the original 2.x code in this case *was then already broken* and could have easily failed silently instead (by implicitly using the wrong encoding on either side, or both sides). I understand that many data protocols use "tags" in a binary data stream that are intended to be readable as ASCII (e.g. fourCCs, or HTTP headers interpreted before an encoding has been negotiated); but insertion into such binary data is fundamentally an array-splicing operation, not a text formatting one. (Or if you create HTTP headers from scratch, you must sanitize the data anyway - see https://stackoverflow.com/questions/5251824 - and catching an exception from .encode('ASCII') is as good a way as any.)


to post comments

2to3?

Posted May 30, 2025 1:33 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

> PEP 461 is about the % operator for formatting, which still doesn't allow mixed types - it just allows you to have, for example, a byte with value 37 followed by a byte with value 115 in a bytes (or bytearray) and then replace those with a different sequence of bytes.

Yes, thanks for the correction.

> It's a little hard to imagine how an error handler failing to convert between str and bytes could cause such a headache - because an error handler would presumably be formatting *an error message*, which means the left-hand side should clearly be a string *literal*.

You lose the original error message and get a useless type mismatch exception as a result. Which is NOT helpful when you're trying to debug the code that led to the error using logs.

> If you mean that code decoded byte data into strings in order to be formatted, and then wasn't properly converted back to bytes, I sympathize; but the original 2.x code in this case *was then already broken* and could have easily failed silently instead (by implicitly using the wrong encoding on either side, or both sides).

Py2 was almost perfect for data processing. The strings were just bytes and encodings could be ignored, for the most part. That's how Go treats strings, btw.

And I have _never_ _ever_ had a case where Py3 strings helped me. Not once was I in a situation: "Wow! That string/bytes split saved the day for me, good job Python developers!"

2to3?

Posted May 30, 2025 7:46 UTC (Fri) by mb (subscriber, #50428) [Link] (5 responses)

>And I have _never_ _ever_ had a case where Py3 strings helped me

I guess you probably only process texts with ASCII compatible encoding, right?

For people processing texts with non-ASCII encoding py3 is a *huge* step forward into the right direction.
It basically put an end to incorrect decoding as it forces developers to do the correct thing.

Ich möchte nicht mehr zurück.

2to3?

Posted May 30, 2025 21:34 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> I guess you probably only process texts with ASCII compatible encoding, right?

If you want, I can tell you the tales of DOS vs. DOS Alternative encodings for matrix printers. Or about KOI vs. Win1251 settings for the Linux console fonts.

The problem is that you can't really do anything meaningful with true strings, unless you:

1. Just work within ASCII.
2. Know EXACTLY what you're doing and have enough context information.

The 1. is trivially handled by UTF-8 encoding (ASCII stays ASCII), and it actually covers the majority of needs. And 2. requires you to be aware of the current locale (i->I or i->İ), permissible string splits, invisible characters, LTR switches, etc.

So Python ends up with the disadvantages of both approaches. Python strings are not just bags of bytes, they are some magic black boxes that mere mortals are not supposed to open. Yet they don't provide enough context information to do "interesting" stuff, like being able to correctly split strings at word breaks, or handle case conversion.

2to3?

Posted May 30, 2025 21:43 UTC (Fri) by mb (subscriber, #50428) [Link] (3 responses)

Nope. The actual problem was that many developers only ever tested ascii and delivered their software if it was "good". And that's trivially true, just as you said.
Today that breaks.
And that's a good thing.
If you handle strings you *need* to be aware of (2). Always.

2to3?

Posted May 30, 2025 21:46 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Nope. The actual problem was that many developers only ever tested ascii and delivered their software if it was "good".

And I would take that over Py3 string mess. It could have been easily fixed by just setting the string encoding to utf-8.

> If you handle strings you *need* to be aware of (2). Always.

No, you don't. If you're only using string operations for protocol-level stuff, like splitting the HTTP headers on a ":" character, then you can just treat strings as bags of bytes. Same for filesystem operations.

2to3?

Posted May 30, 2025 21:49 UTC (Fri) by mb (subscriber, #50428) [Link] (1 responses)

yeah, right. If you don't use strings ("just treat strings as bags of bytes"), then you don't need to use strings.
That's trivially true and I don't see the problem. Just use bytes.

2to3?

Posted May 30, 2025 21:58 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> yeah, right. If you don't use strings ("just treat strings as bags of bytes"), then you don't need to use strings.

That is correct. The problem is that the type for string literals in Python is, well, a string. It pops up in a myriad of places in the standard library.

For example, the exception message is (you guessed it) a string.

> That's trivially true and I don't see the problem. Just use bytes.

Well, I recommend you to try your own advice.

2to3?

Posted Jun 2, 2025 1:40 UTC (Mon) by zahlman (guest, #175387) [Link] (3 responses)

> And I have _never_ _ever_ had a case where Py3 strings helped me. Not once was I in a situation: "Wow! That string/bytes split saved the day for me, good job Python developers!"

Every time that you open a file in text mode in Python 3, you are being helped by the new system - which enables Python to read those files in "universal text mode" while applying a text encoding at the same time. So now you just supply an 'encoding' keyword argument, and you don't have to separately decode each line (and don't need to worry about how newlines are encoded!). Nor do you ever wrestle with the 'codecs' standard library module.

Every time you specify the wrong encoding when opening a file, Python 3 helps you by forcing you to attempt the decoding at the correct point in your program - thus reporting the error at the time you read from the file, instead of arbitrarily far into the rest of your code.

Every time you *don't* see a UnicodeDecodeError from a line of code that calls .encode, or a UnicodeEncodeError from a line of code that calls .decode, you are being helped by Python 3's refusal to convert implicitly from one type to the other (with a poorly documented encoding choice) before using your own explicit conversion back - see e.g. https://stackoverflow.com/questions/5096776 and https://stackoverflow.com/questions/9644099 (and a whole pile of duplicates, whether or not they've been recognized as such yet).

Since Python 3.3 (see https://peps.python.org/pep-0393/), every time you use len() on a string that includes characters outside the BMP, Python 3 helps you by always correctly counting the characters you put into the string, and not *sometimes, implicitly* decomposing them into surrogate pairs because you have a "narrow Unicode build". It also helps you by saving space versus older "wide" builds in cases where you *don't* have such characters in the string. (For that matter, it can store an ASCII string as a proper Unicode string with only one byte per character.)

Python 3 lets you use a bytes object as a sequence of byte values, i.e., unsigned 8-bit integer values. Every time you index into it and get an integer result, rather than a length-1 bytes object, you benefit. Similarly, since Python 3.2, you benefit from int.from_bytes and int.to_bytes existing - an enhancement that would make little sense to add as long as there's still a risk of people trying to use bytes to represent text directly. Similarly, Python 3 gave you bytes.fromhex, and 3.5 added bytes.hex - which would again be nonsense when the type is used as a defective representation of text. (Needing to reach for strangely named standard library methods could be viewed as having been a kind of safety measure here.)

Every time you had to deal with legacy binary data that has embedded text data, or text in a legacy encoding - say, script data from an old reverse-engineered game, with custom control codes to indicate who's speaking or to control the text scroll etc., or uses some bytes to represent custom glyphs with two ASCII characters on the same tile - Python 3 helped you by keeping the type calculus clear. It does this by reserving a type that only ever represents the raw data, and not any extracted, custom-decoded strings, which must be in the other type. I don't personally use type annotations for much of anything, but type checkers like Mypy certainly appreciate the extra clarity, too.

2to3?

Posted Jun 2, 2025 2:32 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Every time that you open a file in text mode in Python 3, you are being helped by the new system - which enables Python to read those files in "universal text mode" while applying a text encoding at the same time.

Thanks for reminding me. Yes, I sometimes forget to add the binary mode and this often bites me later when Python inanely converts files.

Bonus point if the computer where it's deployed has an ISO encoding, and not UTF-8.

> Every time you specify the wrong encoding when opening a file, Python 3 helps you by forcing you to attempt the decoding at the correct point in your program - thus reporting the error at the time you read from the file, instead of arbitrarily far into the rest of your code.

Or not reporting, and just silently corrupting the data.

> Since Python 3.3 (see https://peps.python.org/pep-0393/), every time you use len() on a string that includes characters outside the BMP, Python 3 helps you by always correctly counting the characters you put into the string

Sigh. Now it's clear that people like you are the result of this string disaster. You never once stopped to ask: "WHAT is the length of string?"

Is it the number of characters? The number of code points? The number of graphemes? What exactly do you mean by it? Why do you even _want_ to know the length of a string?

Py3 string length does not even uphold this equality: "len(string) = len(split1) + len(split2)" for all arbitrary strings split on the "character" boundary!

2to3?

Posted Jun 2, 2025 10:23 UTC (Mon) by intelfx (subscriber, #130118) [Link] (1 responses)

> Sigh. Now it's clear that people like you are the result of this string disaster.

Whatever point you were trying to make, I'm sure it would have been better off *without* this personal attack.

2to3?

Posted Jun 2, 2025 12:51 UTC (Mon) by Wol (subscriber, #4433) [Link]

I can understand Cyberax' frustration ...

To clarify his point (and yes it took me a while to get it)

"why do other people think it's perfectly okay to trash my data when I haven't told them to!".

File opens should NOT DEFAULT to "assume the contents are mutable and can be altered by the computer without warning". It's a disaster in waiting ...

Still, it goes back as far as ftp, so that's a long way - I can still remember trying to recover the contents of a binary data file that had been transferred in text mode ... and even if it is text, I can remember trying to get printers to work because some stupid system thought it would treat it according to the system convention not the printer definition ...

Cheers,
Wol


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds