|
|
Log in / Subscribe / Register

2to3?

2to3?

Posted Jun 2, 2025 1:40 UTC (Mon) by zahlman (guest, #175387)
In reply to: 2to3? by Cyberax
Parent article: Recent disruptive changes from Setuptools

> And I have _never_ _ever_ had a case where Py3 strings helped me. Not once was I in a situation: "Wow! That string/bytes split saved the day for me, good job Python developers!"

Every time that you open a file in text mode in Python 3, you are being helped by the new system - which enables Python to read those files in "universal text mode" while applying a text encoding at the same time. So now you just supply an 'encoding' keyword argument, and you don't have to separately decode each line (and don't need to worry about how newlines are encoded!). Nor do you ever wrestle with the 'codecs' standard library module.

Every time you specify the wrong encoding when opening a file, Python 3 helps you by forcing you to attempt the decoding at the correct point in your program - thus reporting the error at the time you read from the file, instead of arbitrarily far into the rest of your code.

Every time you *don't* see a UnicodeDecodeError from a line of code that calls .encode, or a UnicodeEncodeError from a line of code that calls .decode, you are being helped by Python 3's refusal to convert implicitly from one type to the other (with a poorly documented encoding choice) before using your own explicit conversion back - see e.g. https://stackoverflow.com/questions/5096776 and https://stackoverflow.com/questions/9644099 (and a whole pile of duplicates, whether or not they've been recognized as such yet).

Since Python 3.3 (see https://peps.python.org/pep-0393/), every time you use len() on a string that includes characters outside the BMP, Python 3 helps you by always correctly counting the characters you put into the string, and not *sometimes, implicitly* decomposing them into surrogate pairs because you have a "narrow Unicode build". It also helps you by saving space versus older "wide" builds in cases where you *don't* have such characters in the string. (For that matter, it can store an ASCII string as a proper Unicode string with only one byte per character.)

Python 3 lets you use a bytes object as a sequence of byte values, i.e., unsigned 8-bit integer values. Every time you index into it and get an integer result, rather than a length-1 bytes object, you benefit. Similarly, since Python 3.2, you benefit from int.from_bytes and int.to_bytes existing - an enhancement that would make little sense to add as long as there's still a risk of people trying to use bytes to represent text directly. Similarly, Python 3 gave you bytes.fromhex, and 3.5 added bytes.hex - which would again be nonsense when the type is used as a defective representation of text. (Needing to reach for strangely named standard library methods could be viewed as having been a kind of safety measure here.)

Every time you had to deal with legacy binary data that has embedded text data, or text in a legacy encoding - say, script data from an old reverse-engineered game, with custom control codes to indicate who's speaking or to control the text scroll etc., or uses some bytes to represent custom glyphs with two ASCII characters on the same tile - Python 3 helped you by keeping the type calculus clear. It does this by reserving a type that only ever represents the raw data, and not any extracted, custom-decoded strings, which must be in the other type. I don't personally use type annotations for much of anything, but type checkers like Mypy certainly appreciate the extra clarity, too.


to post comments

2to3?

Posted Jun 2, 2025 2:32 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Every time that you open a file in text mode in Python 3, you are being helped by the new system - which enables Python to read those files in "universal text mode" while applying a text encoding at the same time.

Thanks for reminding me. Yes, I sometimes forget to add the binary mode and this often bites me later when Python inanely converts files.

Bonus point if the computer where it's deployed has an ISO encoding, and not UTF-8.

> Every time you specify the wrong encoding when opening a file, Python 3 helps you by forcing you to attempt the decoding at the correct point in your program - thus reporting the error at the time you read from the file, instead of arbitrarily far into the rest of your code.

Or not reporting, and just silently corrupting the data.

> Since Python 3.3 (see https://peps.python.org/pep-0393/), every time you use len() on a string that includes characters outside the BMP, Python 3 helps you by always correctly counting the characters you put into the string

Sigh. Now it's clear that people like you are the result of this string disaster. You never once stopped to ask: "WHAT is the length of string?"

Is it the number of characters? The number of code points? The number of graphemes? What exactly do you mean by it? Why do you even _want_ to know the length of a string?

Py3 string length does not even uphold this equality: "len(string) = len(split1) + len(split2)" for all arbitrary strings split on the "character" boundary!

2to3?

Posted Jun 2, 2025 10:23 UTC (Mon) by intelfx (subscriber, #130118) [Link] (1 responses)

> Sigh. Now it's clear that people like you are the result of this string disaster.

Whatever point you were trying to make, I'm sure it would have been better off *without* this personal attack.

2to3?

Posted Jun 2, 2025 12:51 UTC (Mon) by Wol (subscriber, #4433) [Link]

I can understand Cyberax' frustration ...

To clarify his point (and yes it took me a while to get it)

"why do other people think it's perfectly okay to trash my data when I haven't told them to!".

File opens should NOT DEFAULT to "assume the contents are mutable and can be altered by the computer without warning". It's a disaster in waiting ...

Still, it goes back as far as ftp, so that's a long way - I can still remember trying to recover the contents of a binary data file that had been transferred in text mode ... and even if it is text, I can remember trying to get printers to work because some stupid system thought it would treat it according to the system convention not the printer definition ...

Cheers,
Wol


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds