bytes vs. characters [LWN.net]

bytes vs. characters

Posted Apr 23, 2015 15:27 UTC (Thu) by lopgok (guest, #43164)
In reply to: bytes vs. characters by marcH
Parent article: Report from the Python Language Summit

I would like a simple way in python 3 to be able to read the names of all the files in a directory. In python 3, it skips over some files which I suspect are not in the current codespace. In python 2, it just reads the names of all of the files.

I understand it is problematic to do string processing on oddly constructed strings, but it is mission critical for me to be able to see all the files in a directory. If an exception was raised it would really suck, but it would suck less than silently skipping file names that it didn't understand.

That is the reason I have not migrated all of my development to python 3.

to post comments

bytes vs. characters

Posted Apr 23, 2015 17:00 UTC (Thu) by cesarb (subscriber, #6266) [Link] (4 responses)

> I would like a simple way in python 3 to be able to read the names of all the files in a directory. In python 3, it skips over some files which I suspect are not in the current codespace. In python 2, it just reads the names of all of the files.

I just tested here, and the python3 in this machine returns all filenames in os.listdir('.'), even the one I created with an invalid UTF-8 encoding.

Skipping over some files was true in Python 3.0 (https://docs.python.org/3/whatsnew/3.0.html#text-vs-data-...):

"Note that when os.listdir() returns a list of strings, filenames that cannot be decoded properly are omitted rather than raising UnicodeError."

(The same paragraph mentions that you could still use os.listdir(b'.') to get all filenames as bytes, so even with Python 3.0 you already had a way to read the name of all the files.)

But that was probably changed in Python 3.1, when PEP 383 (https://www.python.org/dev/peps/pep-0383/) was implemented, since with it there are no "filenames that cannot be decoded properly".

bytes vs. characters

Posted Apr 23, 2015 22:09 UTC (Thu) by lopgok (guest, #43164) [Link] (3 responses)

It is still broken with python 3 when I tested it about 2 or 3 months ago. It was either python 3.3 or python 3.4

I have a directory which is read just fine with python 2.7, but skips files with python 3.

bytes vs. characters

Posted Apr 24, 2015 11:44 UTC (Fri) by cesarb (subscriber, #6266) [Link] (2 responses)

Does it still skip files if you use the "bytes" interface (os.listdir(b'.'))?

I just took a quick look at the current Python source code for os.listdir (https://hg.python.org/cpython/file/151cab576cab/Modules/p...), and it only has code to skip the "." and ".." entries, as it's documented to do. In both the "str" and the "bytes" case, it adds every entry other than these two. For it to skip anything else on os.listdir, readdir() from glibc has to be skipping it, and it should affect more than just Python.

Or is the problem with something other than os.listdir?

bytes vs. characters

Posted Apr 24, 2015 14:18 UTC (Fri) by lopgok (guest, #43164) [Link] (1 responses)

It is os.listdir. I have not tried accessing it in binary yet.

I do find it odd that the OS can list the file and I can manipulate the file name on the command line, but because it has some odd characters in it, python silently skips over it.

bytes vs. characters

Posted May 9, 2015 21:34 UTC (Sat) by nix (subscriber, #2304) [Link]

Well, if you want to read something no matter what its encoding, you use bytes mode. That's what bytes mode is *for*. Python 3 is really very consistent here (unlike Python 2, for which you had to guess and hope.)

bytes vs. characters

Posted Apr 23, 2015 17:07 UTC (Thu) by marcH (subscriber, #57642) [Link] (12 responses)

> In python 3, it skips over some files which I suspect are not in the current codespace [...] but it is mission critical for me to be able to see all the files in a directory.

"Mission-critical" relying on filename garbage, mmm.... really? What kind of mission?

bytes vs. characters

Posted Apr 23, 2015 18:03 UTC (Thu) by zlynx (guest, #2285) [Link]

Silent failure is always a STUPID idea.

If Python3 really is silently ignoring invalid filenames then that should be marked as a critical flaw.

The real world is not a perfect place with perfectly encoded strings.

bytes vs. characters

Posted Apr 23, 2015 20:33 UTC (Thu) by lsl (subscriber, #86508) [Link] (6 responses)

Why would they be garbage?

bytes vs. characters

Posted Apr 23, 2015 20:47 UTC (Thu) by marcH (subscriber, #57642) [Link] (5 responses)

> Why would they be garbage?

How else is any invalid encoding displayed?

bytes vs. characters

Posted Apr 23, 2015 22:07 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

they are only invalid if you decide ahead of time that they are supposed to be UTF8 strings.

the spec allows them to be a string of bytes (excluding null and /), no encoding is required.

bytes vs. characters

Posted Apr 23, 2015 22:30 UTC (Thu) by marcH (subscriber, #57642) [Link]

> they are only invalid if you decide ahead of time that they are supposed to be UTF8 strings.

They are invalid if you decide that they are supposed to be in some encoding and some filename uses any *other* encoding. Then garbage gets displayed: for real.

https://en.wikipedia.org/wiki/Mojibake (search page for "garbage")

It's less rare with removable media or ID3
https://en.wikipedia.org/wiki/ID3 (search page for "mojibake")

> the spec allows them to be a string of bytes (excluding null and /), no encoding is required.

As far as filenames are concerned, you meant: *the lack of* a spec.

http://www.dwheeler.com/essays/fixing-unix-linux-filename... (search page for... "garbage")

> no encoding is required.

Which command do you typically use instead of "ls"? hexdump?

bytes vs. characters

Posted Apr 24, 2015 6:57 UTC (Fri) by mbunkus (subscriber, #87248) [Link] (2 responses)

It's not about the display of broken information. That's the easy part.

But a reliable tool, especially one running on filesystems where nearly everything goes (including newlines in them and having no discernible encoding at all), should be able to handle such files, too. This goes double for tools where the developer doesn't control the input. Backup software is the prime example.

How often such files happen? You'd be surprised… Email clients are still broken and annotate file names with the wrong character set resulting in broken file names when saving. ZIP files don't have any encoding information at all, so unpacking one with a file name containing non-ASCII characters often results in ISO encoded file names on my UTF-8 system. And so on.

Therefore treating file names as anything else than a sequence of bytes is, in general, a really bad idea. Only force encodings in places where you need that encoding; displaying the file name being the prime example. If you store it in a database use binary column formats (or if you must use hex then use some kind of escaping mechanism like URL encoding an UTF-8 representation). UTF-8 representations have their own problems regarding file names, think of normalization forms and the fun you're having with Macs and non-Macs.

Treating file names correctly is hard enough. Forcing them into any kind of encoding only makes it worse.

bytes vs. characters

Posted Apr 24, 2015 17:22 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

> Only force encodings in places where you need that encoding; displaying the file name being the prime example.

Thanks a lot, this clarifies.

So the core issue seems to be: the filename being the only file handle. Lose the name and you lose the file. I agree it shouldn't be like this. For instance you can have an iterator that returns some opaque FileObject that does not really care about the name. Does Python have this?

bytes vs. characters

Posted Apr 25, 2015 8:19 UTC (Sat) by peter-b (guest, #66996) [Link]

> So the core issue seems to be: the filename being the only file handle. Lose the name and you lose the file. I agree it shouldn't be like this. For instance you can have an iterator that returns some opaque FileObject that does not really care about the name. Does Python have this?

Yes. listdir(x) where x is bytes returns the raw filenames as bytes.

https://docs.python.org/3.4/library/os.html?highlight=lis...

bytes vs. characters

Posted Apr 23, 2015 22:11 UTC (Thu) by lopgok (guest, #43164) [Link] (3 responses)

Mission critical is not hard real-time. Mission critical means the mission fails when the program fails to read files in a directory.

Some failures are just annoying, but this one is not for me.

bytes vs. characters

Posted Apr 23, 2015 22:40 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

> Mission critical means the mission fails when the program fails to read files in a directory.

And the question was: what kind of mission relies on filename garbage?

It's not 100% clear if you actually care about the filenames, or not and just their content.

BTW I totally agree that silently skipping broken filenames is a massive bug.

bytes vs. characters

Posted Apr 24, 2015 0:44 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> And the question was: what kind of mission relies on filename garbage?
For example, a cloud storage client that is used to backup users' files.

bytes vs. characters

Posted May 9, 2015 21:36 UTC (Sat) by nix (subscriber, #2304) [Link]

That would be a backup program written by someone who doesn't understand the difference between Python's byte- and string-based interfaces. I wouldn't trust any backup program written by someone who didn't understand that! God only knows what it's doing to the file content...