Python 2.8?

_Strings_ are NOT bytes. They are characters. Python 3 strings are arrays of characters, and bytes are just that - bytes. Bytes are not strings. Bytes are bytes. A "string of bytes" isn't a phrase that makes sense. Bytes in Python 3 is a byte array. That's all that it is. Period. End of discussion. It could be a byte array whose contents is an _encoded_ string, which you might decode into an array of characters (aka a string). But it's NOT a string.

>Python 3 made working with them extremely cumbersome.

No, it made it wonderfully simple. You're just using it wrong.

Your example is also broken.

>open('test-\ud800.txt', 'w').close()

You're already off to a bad start. Did you even try your example in Python 3?

>>> open('test-\ud800.txt', 'w').close()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 5: surrogates not allowed

Let me show you what you meant to do:

>>> open(b'test-\ud800.txt', 'w').close()
>>> [open(f, 'r').close() for f in os.listdir(b'.')]
[None]

Note how it doesn't crap out. You need to get your head out of Python 2 and learn Python 3 properly.

You're grossly wrong. There are many rationales for the migration, none of them being wrong Unicode design in Python. Some of them are:

1) Poor Python performance, caused by runtime type reconciliation because of
2) Dynamic typing, which then requires
3) Lots of unit tests required to keep that dynamic types in shape.

Go is fast, because it's staticly typed language. There's no risk that a method will be called with unplanned data type, because compiler does the hard work with its type system. Thanks to that less unit testing is required and that, in turn, allows for faster feature delivery.

> >>> open('test-\ud800.txt', 'w').close()
> UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 5: surrogates not allowed

Well you were right it doesn't work in 3.5, but it does so in an entirely reasonable way. This was not true in earlier versions.

> The real world out there is not unicode, it is composed of strings of bytes.

Actually, the real world has both strings and bytes. Strings are things you can display to a human. You can not reliably display bytes. A type system the distinguishes between the two reduces the odds of displaying unintelligible crap to users.

The distinction was unnecessary when ASCII was the sole encoding. Even after that it wasn't a pressing issue when the only people likely to see your strings were in the same office. But then the internet came along, and the solution we had adopted was to represent printable strings as the tuple (encoding, bytes) - and there were 100's of encodings. Agreeing on an encoding when you are all in the same office is one thing, but negotiating a common encoding between two remote systems so they can exchange strings was unworkable.

Unicode was meant to fix it, but initially it didn't. Amazingly, it appears they failed to understand the problem because they gave us multiple encodings for Unicode. Fortunately a C programmer came along, gave us UTF-8 which the world has settled on. Somewhat less fortunately early systems (Java, Windows) have adopted the failed earlier attempts at encoding Unicode. Python 3 has avoided that mess - it's internal representation looks pretty good to me.

Python 3's biggest mistake, which as you point out they still haven't recovered from, is they didn't abandon making "open('name', 'rt')" the default. The core problem is what 't' means isn't well defined on Unix (whereas "open(b'name', 'rb')" is always well defined). In fact it was never well defined - line endings have always varied between systems. But extending 't' to define the encoding broke it completely because it no longer just effected line endings - it was the whole document, and the file names that pointed to it. Worse, assuming 't' on file handles that transport bytes (like many pipes and sockets) meant they failed to work at all whereas before they did.

But the solution is easy enough. In python3, get into to habit of treating casting all file names to bytes, opening all files in binary, and doing the conversations yourself - which is how it should be done. You will find this hugely annoying, as there will be many times you won't know what encoding to use. But in that case you will be able to put the blame where it belongs - which on whoever wrote the data, not python3.

For files that should be printable ASCII (eg, most config files in /etc) I find "open('...', encoding='ascii', errors='ignore')" a reasonable way to pretend the world hasn't changed in 30 years.

>>> b'abc/def/ghi'.split(b'/')
[b'abc', b'def', b'ghi']
>>> b'abc def ghi'.split()
[b'abc', b'def', b'ghi']

I think this didn't work in 3.0; I don't know when that changed. IIRC at the same time other string manipulations were implemented fro bytestrings.

I've been meaning to write something similar for some old stuff where things like ½ in recipes got mangled.

Regarding [open(f, 'r').close() for f in os.listdir(b'.')]: That sounds like a bug, at least to me.

Python 3.6 now uses UTF-8/surrogatepass on Windows in os.fsdecode() / os.fsencode(). Hopefully, these functions are almost never used on Windows, since Windows has a native support for Unicode. For example, command line arguments, list filenames in a directory, get the hostname, ... : Windows return data directly as Unicode.

The surrogatepass error handler is required to support the same character set than Windows. Windows does allow surrogate characters in filenames. It's really weird and does not conform to Unicode standards which deny these characters in UTF-* encodings.

Python 3 respects Unicode standards: surrogate characters are not allowed in the UTF-8 codec for example. It allowed to implement new nice error handlers for UTF-8: surrogateescape, surrogatepass, etc. By the way, Python 3.6 has a new interesting "namereplace" error handler.