2to3?
2to3?
Posted Jun 2, 2025 1:40 UTC (Mon) by zahlman (guest, #175387)In reply to: 2to3? by Cyberax
Parent article: Recent disruptive changes from Setuptools
Every time that you open a file in text mode in Python 3, you are being helped by the new system - which enables Python to read those files in "universal text mode" while applying a text encoding at the same time. So now you just supply an 'encoding' keyword argument, and you don't have to separately decode each line (and don't need to worry about how newlines are encoded!). Nor do you ever wrestle with the 'codecs' standard library module.
Every time you specify the wrong encoding when opening a file, Python 3 helps you by forcing you to attempt the decoding at the correct point in your program - thus reporting the error at the time you read from the file, instead of arbitrarily far into the rest of your code.
Every time you *don't* see a UnicodeDecodeError from a line of code that calls .encode, or a UnicodeEncodeError from a line of code that calls .decode, you are being helped by Python 3's refusal to convert implicitly from one type to the other (with a poorly documented encoding choice) before using your own explicit conversion back - see e.g. https://stackoverflow.com/questions/5096776 and https://stackoverflow.com/questions/9644099 (and a whole pile of duplicates, whether or not they've been recognized as such yet).
Since Python 3.3 (see https://peps.python.org/pep-0393/), every time you use len() on a string that includes characters outside the BMP, Python 3 helps you by always correctly counting the characters you put into the string, and not *sometimes, implicitly* decomposing them into surrogate pairs because you have a "narrow Unicode build". It also helps you by saving space versus older "wide" builds in cases where you *don't* have such characters in the string. (For that matter, it can store an ASCII string as a proper Unicode string with only one byte per character.)
Python 3 lets you use a bytes object as a sequence of byte values, i.e., unsigned 8-bit integer values. Every time you index into it and get an integer result, rather than a length-1 bytes object, you benefit. Similarly, since Python 3.2, you benefit from int.from_bytes and int.to_bytes existing - an enhancement that would make little sense to add as long as there's still a risk of people trying to use bytes to represent text directly. Similarly, Python 3 gave you bytes.fromhex, and 3.5 added bytes.hex - which would again be nonsense when the type is used as a defective representation of text. (Needing to reach for strangely named standard library methods could be viewed as having been a kind of safety measure here.)
Every time you had to deal with legacy binary data that has embedded text data, or text in a legacy encoding - say, script data from an old reverse-engineered game, with custom control codes to indicate who's speaking or to control the text scroll etc., or uses some bytes to represent custom glyphs with two ASCII characters on the same tile - Python 3 helped you by keeping the type calculus clear. It does this by reserving a type that only ever represents the raw data, and not any extracted, custom-decoded strings, which must be in the other type. I don't personally use type annotations for much of anything, but type checkers like Mypy certainly appreciate the extra clarity, too.
