LWN: Comments on "Filesystems and case-insensitivity"

Filesystems and case-insensitivity

Serentty — Thu, 06 Jun 2019 05:09:19 +0000

Encoding Chinese text based on primitives is the Holy Grail of Chinese text encoding, but no one has actually been able to come up with a realistic solution for it, and it's probably just not realistic. Korean text on the other hand is really easy to encode based on primitives, as it's just 22 letters combined in predictable ways.

Filesystems and case-insensitivity

Cyberax — Fri, 31 May 2019 18:37:02 +0000

Technically, most of CJK characters can be decomposed into simpler characters. About 70% of Mandarin characters follow the "radical-phonetic" model and can theoretically be composed.

Filesystems and case-insensitivity

smurf — Fri, 31 May 2019 15:06:18 +0000

True, that.

Seems that quite a few of Chinese people with interesting names (i.e. using archaic characters) suddenly couldn't get an official document any more because, surprise, their name wasn't in the "official" charset …

Filesystems and case-insensitivity

excors — Thu, 30 May 2019 14:54:43 +0000

https://en.wikipedia.org/wiki/Template:CJK_ideographs_in_... has a helpful list with the numbers of CJK codepoints, and I assume only the earliest ones were needed for legacy compatibility - the rest were presumably added because they couldn't already be represented. Recently Unicode 10.0 added "CJK Extension F" (7473 codepoints) so it seems they're still not finished. Then there's all the other scripts being added, like Tangut ("a major historic script of China") with another ~7000 codepoints. And about 1700 emojis (https://unicode.org/emoji/charts/emoji-list.html).

Maybe the 64K limit could have lasted for many more years if they had made some different design choices early on, but given the goal of being a universal standard for all text, it seems inevitable the limit would be broken eventually. It's better to have broken it earlier than later.

Filesystems and case-insensitivity

smurf — Thu, 30 May 2019 14:18:16 +0000

Hmm. If that rule had actually been followed, we'd still have room on the base plane (i.e. codepoints below 65536).

(How many primitives would you need for Chinese?)

On the other hand, in that case we wouldn't all use UTF-8 by now – simply because that would require twice the storage space for Chnese text, more or less. Nowadays that doesn't really matter, but at the time it was a problem.

Filesystems and case-insensitivity

Serentty — Wed, 29 May 2019 23:00:11 +0000

This is not a deficiency with Unicode. Precomposed characters such as É have only ever been encoded in Unicode as a matter of compatibility with legacy encodings, and wouldn't have been included if not for this. They continue to be used because they save you a few bytes, which you might as well go for even if compression makes it moot in the end. Combining diacritics have always been the preferred method as they are much more flexible, and allow users to compose arbitrary characters without needing to constantly update their software or risk mojibake. Many scripts in Unicode work entirely though combining diacritics and it works just fine; the Indic scripts are good examples. It should be noted that the legacy encodings for these scripts usually worked that way as well. Conformant implementations will treat composed and decomposed characters identically, so the advantage of going down the rabbit hole of trying to provide every precomposed character anyone might ever want isn't really worth it when composition works just as well. If you notice that combining diacritics aren't giving you the nice hand-tweaked glyphs that precomposed characters are, and you end up with the diacritic looking all wrong, take it up with the developer of the text renderer or the font, because that's not how Unicode is supposed to work.

Filesystems and case-insensitivity

james — Thu, 13 Dec 2018 10:33:42 +0000

Would it calm your anger to point out that LibreOffice can search using regexps?

Filesystems and case-insensitivity

pr1268 — Wed, 12 Dec 2018 23:45:36 +0000

there is no reasonable rationale for those other space characters (including U+00a0) in file names.

Agreed, but try telling that to those fools who auto-generated the spreadsheet with \u00a0 spaces. </angry rant>

Filesystems and case-insensitivity

corbet — Mon, 10 Dec 2018 16:07:59 +0000

For the curious, Linus has also sounded off on the idea; he is not impressed either.

Filesystems and case-insensitivity

davez — Mon, 10 Dec 2018 15:04:50 +0000

Anybody wanting to add case-insensitivity to a filesystem should read this blog post:

http://drewthaler.blogspot.com/2007/12/case-against-insen...

Also, consider the fact that Apple's iOS filesystem is case sensitive, *not* case insensitive.

Filesystems and case-insensitivity

zlynx — Sat, 08 Dec 2018 23:37:00 +0000

Fedora's default Exim configuration did not make it insensitive. I had to add that myself because my server was rejecting a lot of mail, mostly from Gmail users randomly capitalizing things.

> lowercase_local:
> driver = redirect
> data = ${lc:${local_part}}

Filesystems and case-insensitivity

smurf — Fri, 07 Dec 2018 16:31:20 +0000

Its cast must be preserved in transit but terminal MTAs are free to treat them as case insensitive. Most do, these days.

Filesystems and case-insensitivity

gioele — Fri, 07 Dec 2018 11:45:32 +0000

> I *still* have problems because users insist on knowing whether email addresses have capital letters or not (they are case-insensitive, for historical reasons, because a lot of the early systems mangled case).

These users are right: the local-part of an email address _is_ case sensitive. Only the domain is case insensitive.

RFC 5321, section 2.4 [1]:

> The local-part of a mailbox MUST BE treated as case sensitive. Therefore, SMTP implementations MUST take care to preserve the case of mailbox local-parts. In particular, for some hosts, the user "smith" is different from the user "Smith". However, exploiting the case sensitivity of mailbox local-parts impedes interoperability and is discouraged. Mailbox domains follow normal DNS rules and are hence not case sensitive.

[1] https://tools.ietf.org/html/rfc5321#section-2.4

Filesystems and case-insensitivity

Wol — Fri, 07 Dec 2018 11:09:12 +0000

> But leaving that aside for the moment, what is the gain? I've been working on case sensitive Linux file systems for many years, and never felt the need to have "level1.MAP" really load "level1.map".

Because you're a programmer used to thinking in byte strings.

I *still* have problems because users insist on knowing whether email addresses have capital letters or not (they are case-insensitive, for historical reasons, because a lot of the early systems mangled case).

So the short answer is, YOU may not feel the gain, but a lot of other people WILL.

(Along the same lines, I remember being sent a second copy of some newsletter because "some people said they couldn't read the attachment". Ie pretty much all Windows systems, because the sender had somehow lost the extension and those systems didn't recognise the file "newsletter" as a pdf. Of course, my gentoo system didn't give a monkeys :-)

Cheers,
Wol

Filesystems and case-insensitivity

rleigh — Thu, 06 Dec 2018 12:34:11 +0000

Example for ZFS:

    % zfs get all rpool/ROOT/default
    NAME                PROPERTY              VALUE                  SOURCE
    rpool/ROOT/default  type                  filesystem             -
    rpool/ROOT/default  creation              Sun Jun 12 10:46 2016  -
    …
    rpool/ROOT/default  utf8only              on                     -
    rpool/ROOT/default  normalization         formD                  -
    rpool/ROOT/default  casesensitivity       sensitive              -
    …

Case sensitivity is selectable. You can force it to only store valid UTF-8, and you can choose whether the UTF-8 is normalised or not. Or you can allow it to store anything. This gives you full backward compatibility with historical UNIX behaviour should you want it, or you can restrict it case sensitive normalised UTF-8. Having this selectable on a per-filesystem basis gives you a great deal of flexibility, and it defaults to something sensible (the above are the defaults).

Filesystems and case-insensitivity

Wol — Thu, 06 Dec 2018 10:16:51 +0000

I believe there are two control characters RS1 and RS2? Basically standing for "Repeat String"? Which were used on a system I worked on, and actually were a damn good fix for "how many characters does a tab stand for?". So most lines in my FORTRAN source code would have been physically stored on disk as "<RS1><6>code..."

And if you had a lot of spaces it saved a fair few bytes over tab-encoding, plus being completely unambiguous.

Cheers,
Wol

Filesystems and case-insensitivity

Wol — Thu, 06 Dec 2018 10:02:26 +0000

I worked with a system that had filenames of the form <space><backspace>NNNN

Bearing in mind users weren't supposed to go anywhere near them, it was a pretty good way of stopping people scanning the filesystem and messing about with them. I agree for user visible files, it's a good idea, but not all files are meant to be user visible and some of them can't be hidden.

Cheers,
Wol

Filesystems and case-insensitivity

hummassa — Tue, 04 Dec 2018 13:27:50 +0000

The initial argument still holds: there is no reasonable rationale for those other space characters (including U+00a0) in file names.

Filesystems and case-insensitivity

smurf — Tue, 04 Dec 2018 10:24:18 +0000

Don't worry – Unicode has a bunch more spaces, including zero-width ones.

On second thought: do worry.

Filesystems and case-insensitivity

pr1268 — Tue, 04 Dec 2018 08:13:23 +0000

one true space space character (U+0020)

Um, there's more than one space: ' ' and ' '. One is \u0020 (good ol' ASCII 0x20) and the other is \u00a0.

I was personally burned by the second "space" above appearing in an Excel spreadsheet (to the exclusion of the "one true space character" you mentioned). >:-(

Filesystems and case-insensitivity

flussence — Mon, 03 Dec 2018 20:24:58 +0000

ls took care of that a few years ago…

~/test $ ls
'aaabc'$'\b''d'  'aaabc'$'\n''d'   aaabd
~/test $ ls --version
ls (GNU coreutils) 8.30
Packaged by Gentoo (8.30 (p01))

Filesystems and case-insensitivity

ale2018 — Mon, 03 Dec 2018 12:30:32 +0000

Ah, poorly written shell scripts, eh? Because you obviously think that being slave of over-complicated command lines is fine? A good percentage of my command lines start with find . -name whatever | xargs... Yes, I know I can write -print0 and -0, I do that when I write shell scripts.

When I find a filename with spaces I just move it away.

For the record, the normalization step and control characters were never taken care of. For example:

    ~$ touch aaabd $(printf 'aaabc\bd')  "$(printf 'aaabc\nd')"
    ~$ ls -lt | head -5
    total 3686968
    -rw-r--r--  1 ale ale                 0 Dec  3 13:21 aaabd
    -rw-r--r--  1 ale ale                 0 Dec  3 13:21 aaabc
    d
    -rw-r--r--  1 ale ale                 0 Dec  3 13:21 aaabd

Control characters where never forbidden. Consider that human beings are sometimes uncertain about the name they're typing and type a backspace (\b) in it. So, why isn't that beautiful too? Perhaps, users should have a clue. In the words of the Ancient Philosophy, rubbish in, rubbish out.

Filesystems and case-insensitivity

gioele — Sun, 02 Dec 2018 17:19:15 +0000

> Is there any reason not to treat i, İ, I, and ı the same for case-folding purposes on the file system?

Sure they could. But doing it is hard (and computationally expensive).

This is what I meant with

> What the developers could do is a kind of case-insensitive look-up that also clusters together "similar" letters. Defining which characters are similar opens, however, another can of worms (see `confusables.txt` from Unicode or all the discussions around IDNA and its Nameprep algorithm).

Filesystems and case-insensitivity

epa — Sun, 02 Dec 2018 16:42:27 +0000

Is there any reason not to treat i, İ, I, and ı the same for case-folding purposes on the file system?

I am not asking whether they are the same in all uses. I know that in Turkish i and ı are different letters. What I'm suggesting is that for making a case-insensitive filesystem lookup -- where you have already waved goodbye to a strict 1-1 mapping between byte sequences and directory entries -- it surely doesn't matter that much to gloss over the distinction and treat all these four characters the same. Similarly I would consider it a feature, not a bug, if accented characters could be preserved in filenames, but ignored when matching. There are pairs of words in German that differ only in accent, but it's very unlikely an accent would be the only difference between two human-written document names.

Now, you may with some justice argue that loose matching like this belongs in user space, not the kernel. But in the end it's not my preferences or anyone else's that matter. What matters is to efficiently implement the existing (de facto or de jure) standards. What behaviour is Samba required to support with the Turkish uppercase and lowercase letters? The kernel should provide the semantics that Samba needs so it doesn't have to laboriously scan the whole directory to match a filename.

Filesystems^H directories and case-insensitivity

marcH — Sun, 02 Dec 2018 08:00:41 +0000

If you thought case-insensitivity was ill-conceived...

As a coincidence I recently got hurt by this NTFS "feature": *per-directory* case-sensitivity
https://github.com/vector-of-bool/vscode-cmake-tools/issu...

In other words: filenames are case sensitive in some directories but not in other directories next or below them on the very same filesystem.

Filesystems and case-insensitivity

jezuch — Sat, 01 Dec 2018 11:24:33 +0000

I mean all the bytes below 0x20. This is not text, they have no place in a *character* encoding. Apart from that I'm totally fine with ASCII compatibility, even though it's typically American culturally insensitive invention ;)

Filesystems and case-insensitivity

mpr22 — Fri, 30 Nov 2018 18:36:21 +0000

Windows has to be case-insensitive because older versions of Windows were case-insensitive because they were compatible with MS-DOS and MS-DOS was case-insensitive.

And MS-DOS was written with the (probably unconscious) assumption that natural-language text was in ASCII, where case-insensitivity is cheap to implement.

Filesystems and case-insensitivity

k8to — Fri, 30 Nov 2018 17:55:48 +0000

Indeed, the existence of the file accessed by a given byte sequence will vary depending on locale. This is true for many situations, not just the rather clear turkish one.

This leads to a problem where a user or process in one locale should get different results from the kernel than another. This traditionally was viewed as a rathole and I've seen many situations where osx behaves in bizarre ways due to this sort of thing.

The proposal here seems to be to push the rules into the filesystem or directory, which effectively means having locale behavior independent of the user / process, which means we will get a fun matrix of file name locale vs user locale. I'm not a fan.

Filesystems and case-insensitivity

ScottMinster — Fri, 30 Nov 2018 17:38:07 +0000

What is the real use case of a case insensitive file system? I understand the interoperability use case of working with case insensitive file systems or Samba clients. And if the answer is really just "other systems do it and they can't change because it would break too much", that makes sense.

But leaving that aside for the moment, what is the gain? I've been working on case sensitive Linux file systems for many years, and never felt the need to have "level1.MAP" really load "level1.map". I've occasionally had to deal with writing software that expects extensions in lower case and been given files with those extensions in upper case, and that is annoying, but it's more due to laziness on the file creator's part of not using the standard extension case.

The only two justifications I can come up with for wanting case insensitivity is to avoid problems with unexpectedly cased files and to avoid user confusion (i.e., "Document1.doc" and "document1.doc" being different files). Those are worthy goals, but it seems like there are so many tricky problems with case insensitivity that it is hardly worth the trouble. It's relatively easy in English, but as many people point out, there are problems in many other languages.

But even in English, it could cause problems. Can you "mv makefile Makefile" in a case insensitive filesystem, or would you get the error "'makefile' and 'Makefile' are the same file"? Also, as another poster pointed out, globbing in various programs would likely have to change to be consistent.

And once you enable it, you can never really disable it without inevitably breaking programs.

So why do systems like Windows and MacOS do it? Did they underestimate the difficulty and are stuck with that decision?

I could see how it could be useful for things like Samba servers, but given all the complicated edge cases, it doesn't seem like it's a good idea for general use. Though once it's working for Samba, some distribution will likely turn it on everywhere, to try to be more user friendly.

Filesystems and case-insensitivity

perennialmind — Fri, 30 Nov 2018 16:25:39 +0000

You mean end-of-string delimiters, end-of-line delimiters, tabs, and the codes needed for controlling a terminal such as escape and erase? Setting aside hurdles to adoption, one can imagine hoisting those into markup. Perhaps there's even a spec for plainer-than-plain-text for when such markup exists (i.e. HTML). If so, it might be perfect for filenames.

ASCII compatibility was the selling point for UTF-8. Beyond the above, even the oddballs are still in use. Take for example "group separator" which stands in for FNC1 in barcodes.

Somebody else will have to defend the C1 block though.

Filesystems and case-insensitivity

jezuch — Fri, 30 Nov 2018 09:09:33 +0000

I guess the concept of control characters should have been retired long time ago. I also think that it was a huge mistake to bring them to UTF-8 along with the rest of ASCII. But I'm pretty sure someone will explain to me that they are in fact critical and there are further control characters in the Unicode spec anyway :)

Filesystems and case-insensitivity

jezuch — Fri, 30 Nov 2018 09:04:33 +0000

Spaces in file names are wonderful. You can name your files like a human being, not a slave of poorly written shell scripts with broken quoting :) (I have a pet peeve too)

Filesystems and case-insensitivity

perennialmind — Thu, 29 Nov 2018 21:04:26 +0000

That would be a potential problem for a mount option, but not when setting a per-directory attribute, since the directory must be empty. The same problem applies to the name collision problem though. Imposing new semantics on a filesystem would be problematic. Maybe with a superblock change? Meh. The per-directory attribute just seems cleaner overall.

Filesystems and case-insensitivity

perennialmind — Thu, 29 Nov 2018 20:47:34 +0000

That's why I thought it was a brilliant choice to require that directories be empty in order to switch on the "text mode dentries" attribute. You sidestep the "reinterpret" problem in trade for a save/copy errors that are easy to surface to end users. I'm not sure how overlayfs union mounts would work though.

Filesystems and case-insensitivity

smurf — Thu, 29 Nov 2018 18:34:07 +0000

Well, that's easy enough to fix – verify that the xattr matches the filename before using it.

A non-malicious source of the same problem is somebody renaming the file with an old non-extended-filename-compatible tool (or libc).

Filesystems and case-insensitivity

quotemstr — Thu, 29 Nov 2018 18:28:22 +0000

UTF-8 won. We should start giving it its well-earned victory parade. Don't you that, after we're finished banning non-UTF-8 encodings, and after we ban illegal or bizarre code sequences, and after we start normalizing filenames into consistent form, we'll end up in a better place? If in the process of doing that we can't access files called ♀ from old volumes without special compatibility mount options, so be it.

Filesystems and case-insensitivity

bfields — Thu, 29 Nov 2018 17:29:10 +0000

What's an example of an application that does that?

Filesystems and case-insensitivity

cesarb — Thu, 29 Nov 2018 17:28:39 +0000

> There may be some user-space breakage due to normalization or case folding of file names that will need to be handled as well.

I wonder how many of these will be security vulnerabilities. We already had one in git not too long ago (CVE-2014-9390 git: arbitrary command execution vulnerability on case-insensitive file systems).

Filesystems and case-insensitivity

cesarb — Thu, 29 Nov 2018 17:05:32 +0000

Not if they're base64: abcdef and ABCDEF are different values in base64. Git can get away with base16 because it uses a short 160-bit hash, but other hash algorithms have much longer outputs (256 or even 512 bits).

Filesystems and case-insensitivity

zdzichu — Thu, 29 Nov 2018 15:57:01 +0000

I guess name-as-provided is for displaying in user interface?

% getfattr --name=user.name-as-provided malware.sh 
# file: malware.sh
user.name-as-provided="Safe.pdf"