Working with UTF-8 in the kernel [LWN.net]

Working with UTF-8 in the kernel

Posted Mar 29, 2019 10:05 UTC (Fri) by smurf (subscriber, #17840)
In reply to: Working with UTF-8 in the kernel by ikm
Parent article: Working with UTF-8 in the kernel

There's also the problem of composites. Unicode, in its infinite wisdom(*), has multiple ways to store the same character (an 'ä' is either a single latin1 character, or an 'a' followed by a combining diaeresis – any sane designer would have stored the modifiers first, but I digress). You need to agree on one form with which to represent file names because the user typically can't easily generate the other, and even copy+paste tends to get mangled.

There's another problem here. Correct case folding is locale dependent. One example: Turkish has an i and an ı (i without the dot). Unicode helpfully has an İ (capital I with a dot) right next to it. Guess what happens when you case-fold these in Turkey vs. everywhere else.

to post comments

Working with UTF-8 in the kernel

Posted Mar 29, 2019 11:21 UTC (Fri) by Sesse (subscriber, #53779) [Link] (40 responses)

Normalization is already dealt with in the patch.

Also, I don't think you can blame Unicode for the fact that Turkish and English has different alphabets.

Working with UTF-8 in the kernel

Posted Mar 29, 2019 13:32 UTC (Fri) by drag (guest, #31333) [Link] (37 responses)

> Also, I don't think you can blame Unicode for the fact that Turkish and English has different alphabets.

What it does mean is that your case insensitive lookups for the file system are actually case sensitive with insensitive elements. How sensitive it ends up being depends on what language a user uses.

Unless, of course, the kernel is aware of the user's locale and changes the responses accordingly.

Working with UTF-8 in the kernel

Posted Mar 29, 2019 22:53 UTC (Fri) by mirabilos (subscriber, #84359) [Link] (36 responses)

The kernel is never aware of the user’s locale, so this belongs into userspace.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 13:17 UTC (Sat) by SLi (subscriber, #53131) [Link] (6 responses)

But the point is, there is no "case folding" independent of language; the rules for doing it are inherently language dependent.

Working with UTF-8 in the kernel

Posted Apr 4, 2019 8:33 UTC (Thu) by dvdeug (subscriber, #10998) [Link] (5 responses)

The rules for doing it are virtually language independent. Turkish and a small set of related languages do have a problematic difference with the dotted i, but the rest of the Latin-script languages all agree, and there seems to be no disagreements among the other languages that use casing scripts. It's unfortunate that 1% of the world's population won't get proper casing, but at this point practical compatibility with other operating systems seems more important.

Working with UTF-8 in the kernel

Posted Apr 5, 2019 8:11 UTC (Fri) by dgm (subscriber, #49227) [Link] (4 responses)

> practical compatibility with other operating systems seems more important.

So Linux cannot exchange data with MacOS and Windows?! PANIC!

Or put another way: if I show you that less than 1% of the population really wants or needs a case-insensitive filesystem, can I disregard your claims?

Working with UTF-8 in the kernel

Posted Apr 8, 2019 2:02 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (3 responses)

If you want to support FAT or NTFS, you need to support case-insensitive filesystems. You can half-ass it and write out potentially corrupt filesystems, but I think most of the users of these filesystems with Windows don't want that. Fortunately, there are rules for locale-insensitive case-folding, and they aren't random or arbitrary.

Working with UTF-8 in the kernel

Posted Apr 8, 2019 21:18 UTC (Mon) by foom (subscriber, #14868) [Link] (2 responses)

> Fortunately, there are rules for locale-insensitive case-folding, and they aren't random or arbitrary.

That may be, but FAT, exFAT, and NTFS don't use the unicode case folding rules. If the justification is to make something compatible with those systems, do we actually need the (rather complex) unicode rules?

Working with UTF-8 in the kernel

Posted Apr 8, 2019 23:30 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (1 responses)

What rules do they use?

In what way are the Unicode case-folding rules rather complex? They are for the most part fairly simple, one to one matchings of characters, with a few exceptions that you just have to deal with. The German ß and the various titlecase characters in Unicode are there and are going to have to be dealt with.

Working with UTF-8 in the kernel

Posted Apr 9, 2019 15:35 UTC (Tue) by foom (subscriber, #14868) [Link]

NTFS and exFAT only maps a single utf16 code unit to another single utf16 code unit, via a lookup table written to disk during filesystem creation. No unicode normalization, no multicharacter equivalencies, and no folding for any characters above FFFF.

You say that other cases "have to be dealt with"...but we have widely used examples showing that to not actually be the case.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 13:59 UTC (Sat) by SLi (subscriber, #53131) [Link] (28 responses)

Can you explain how the user space could help here? For a Turkish user, any system that considers "I" and "i" the same filename/string modulo case is broken. Both are letters and both have upper and lower forms, but they are different letters.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 16:45 UTC (Sat) by foom (subscriber, #14868) [Link] (27 responses)

The case folding for a filesystem must be consistent, so it can't have such locale-specific rules.

Neither Mac nor windows filesystems' case folding is locale sensitive, either. (NTFS does write a file during filesystem creation containing the case folding rules for that drive, so you _could_ make them be whatever you like, at the risk of breaking everything...)

Everyone likes to bring up this example, but I rather expect the likelihood of normal Turkish users noticing and caring that they can't create two such files in the same directory is rather a theoretical problem, not an actual one.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 18:19 UTC (Sat) by nybble41 (subscriber, #55106) [Link] (4 responses)

The fact that case folding is broken everywhere else it's been implemented offers a good argument against implementing it in Linux.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 21:41 UTC (Sat) by mirabilos (subscriber, #84359) [Link] (3 responses)

case-insensitive filename comparison belongs into userspace, because there, if you setlocale(3) correctly, things like toupper(3), tolower(3) and strcoll(3) work properly even for Turkish.

So, the kernel should have nothing to do with this *at all*.

Working with UTF-8 in the kernel

Posted Apr 1, 2019 9:45 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (2 responses)

The kernel needs to provide the encoding and normalisation userspace can work from. You can’t push those to userspace, because userspace then ends up guessing (badly) the filesystem encoding.

Working with UTF-8 in the kernel

Posted Apr 5, 2019 1:24 UTC (Fri) by xtifr (guest, #143) [Link] (1 responses)

If the case-insensitive FS is a user-space overlay on an existing filesystem, then you don't *need* to guess anything. It will do case-folding based on the locale of the user who mounted the overlay.

This means *all* the overheads will truly need to be present only for those who actively *use* the system.

I honestly don't know how this is all going forward without *at least* a user-space proof-of-concept system.

Working with UTF-8 in the kernel

Posted Apr 6, 2019 21:56 UTC (Sat) by foom (subscriber, #14868) [Link]

> It will do case-folding based on the locale of the user who mounted the overlay.

But that behavior would be pretty awful -- which files you can access depending upon your current locale? There's a reason filesystems (including this ext4 proposal) store the mapping used when creating the filesystem...

> without *at least* a user-space proof-of-concept system.

Two have been mentioned already. Android has an overlay filesystem for local access, and samba implements it when exporting the filesystem over the network.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 21:44 UTC (Sat) by mirabilos (subscriber, #84359) [Link] (6 responses)

Yes, it must be consistent. What if a new release of Unicode comes out? Boom.

Another reason why this belongs into userspace.

And no, the turkish case is not theoretical. They have words which only differ in the dot above the i, and in one case, one of the two words is normal and one a rather crass insult, which led to (IIRC) a knife attack (well, some kind of real-life attack at the person) because they had no dotless i on their keyboard when texting.

I’ll quote someone else: just because your latin alphabet has 26 letters, not everyone else’s does. Imagine if we’d *always* (independent on what word it’s in) make “oo” compare the same as “u”, for example.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 21:51 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> And no, the turkish case is not theoretical. They have words which only differ in the dot above the i, and in one case, one of the two words is normal and one a rather crass insult, which led to (IIRC) a knife attack (well, some kind of real-life attack at the person) because they had no dotless i on their keyboard when texting.
This is the story: https://gizmodo.com/a-cellphones-missing-dot-kills-two-pe...

Although I personally wouldn't blame the cellphone here.

Working with UTF-8 in the kernel

Posted Mar 30, 2019 22:43 UTC (Sat) by mpr22 (subscriber, #60784) [Link]

The situation was bad.

Bad technology made it worse.

The cellphone doesn't get off scot-free here.

Working with UTF-8 in the kernel

Posted Mar 31, 2019 1:46 UTC (Sun) by foom (subscriber, #14868) [Link] (3 responses)

Hopefully the filesystem records what mapping it was created with, like NTFS does. Otherwise, some of your files may become inaccessible when a new mapping is switched to (which, iirc, did happen on HFS+ before. That's not good...)

Re: Turkish swears -- you can name your files either word just fine -- the filesystem does not be change your chosen filename to the other name! Only if you try to make files named both, in the same directory, will you run into an issue. I still claim that is *highly* unlikely.

If we treated oo and u as the same for filename comparison purposes, because that was a very common language's policy, I rather suspect that also wouldn't be a huge problem. (It'd be weird to have such behavior, as that isn't a common policy, however.)

Working with UTF-8 in the kernel

Posted Mar 31, 2019 19:17 UTC (Sun) by naptastic (guest, #60139) [Link]

> because that was a very common language's policy

Which one‽ I've never heard of this and I am dying to know! MY BRAIN IS HUNGRY

Working with UTF-8 in the kernel

Posted Apr 4, 2019 5:37 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

Hopefully the filesystem records what mapping it was created with, like NTFS does. Otherwise, some of your files may become inaccessible when a new mapping is switched to (which, iirc, did happen on HFS+ before. That's not good...)

This seems like the key to me. If the case folding rules can change, there's no way to guarantee that the same file will always be accessible the same way, and that's true whether the case folding happens in the kernel or in userspace.

Working with UTF-8 in the kernel

Posted Apr 4, 2019 12:28 UTC (Thu) by bosyber (guest, #84963) [Link]

> If we treated oo and u as the same for filename comparison purposes, because that was a very common language's policy
Is it? I know that it might be that way effectively in German, but in Dutch it is absolutely not, they are completely different sounds (the german u sound is closer to Dutch oe double sound, but not oo which is a loong vowel in Dutch.).

Working with UTF-8 in the kernel

Posted Apr 1, 2019 6:46 UTC (Mon) by marcH (subscriber, #57642) [Link] (14 responses)

> Everyone likes to bring up this example, but I rather expect the likelihood of normal Turkish users noticing and caring that they can't create two such files in the same directory is rather a theoretical problem, not an actual one.

Check the numerous, real-world examples and references given in the comment to the previous LWN article: https://lwn.net/Articles/784041/ It's not just Turkish: like another natural language topic case-sensitivity is very complex and (among others) locale-specific - not just in theory but in practice.

foom wrote:
> Neither Mac nor windows filesystems' case folding is locale sensitive, either. (NTFS does write a file during filesystem creation containing the case folding rules for that drive, so you _could_ make them be whatever you like, at the risk of breaking everything...)

Interesting, references?

nybble41 wrote:
> The fact that case folding is broken everywhere else it's been implemented offers a good argument against implementing it in Linux.

Wait... should Linux be "bug for bug" compatible or linguistically correct?

Working with UTF-8 in the kernel

Posted Apr 3, 2019 2:51 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (8 responses)

Linux should implement the standard rules for case folding, and not worry about the linguistic details.

Working with UTF-8 in the kernel

Posted Apr 3, 2019 5:07 UTC (Wed) by marcH (subscriber, #57642) [Link] (7 responses)

What are these "standard rules" and which locale are they based on? US English?

Working with UTF-8 in the kernel

Posted Apr 3, 2019 6:18 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (6 responses)

http://www.unicode.org/versions/Unicode12.0.0/ch05.pdf offers basically the standard rules, though some details you might have to refer to the technical reports. It's based on all locales; Turkish and some Lithuanian dictionary usage are about the only locales that have a case exception from standard case folding.

Working with UTF-8 in the kernel

Posted Apr 8, 2019 17:53 UTC (Mon) by hkario (subscriber, #94864) [Link] (5 responses)

and German... SS being upper case of ß, but ss being the lower case of SS...

Working with UTF-8 in the kernel

Posted Apr 8, 2019 20:30 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (4 responses)

That's not an exception. In all locales, ß uppercases to SS.

Working with UTF-8 in the kernel

Posted Apr 9, 2019 18:30 UTC (Tue) by mirabilos (subscriber, #84359) [Link] (3 responses)

Except it doesn’t, any more, in German.

Working with UTF-8 in the kernel

Posted Apr 10, 2019 0:50 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (2 responses)

Citation needed. If you're talking about the Capital Eszett, it has been explicitly excluded from case-folding because it is non-standard. If the German speakers really wanted a change, given they are the only modern language group using it, it would change in all locales.

Working with UTF-8 in the kernel

Posted Apr 17, 2019 22:15 UTC (Wed) by chithanh (guest, #52801) [Link]

I think it is more complex than that.

ß (U+00DF) indeed has no uppercase mapping in Unicode.
But ẞ (U+1E9E) has a lowercase mapping of ß.

So if you start with ẞ and then convert to lowercase and then to uppercase again you might end up with SS.

Also, if you perform a case-insensitive filename match for ẞ it will return a file named ß.
But a case-insensitive filename match for ß will not return a file named ẞ.

Working with UTF-8 in the kernel

Posted Apr 17, 2019 22:40 UTC (Wed) by marcH (subscriber, #57642) [Link]

> Also, if you perform a case-insensitive filename match for ẞ it will return
a file named ß.
> But a case-insensitive filename match for ß will not return a file named ẞ.

Just for fun, some more "real-world" case insensitivity (from comments in the previous LWN thread)
https://www.google.com/search?q=FRANCAIS
https://www.google.com/search?q=FRANÇAIS

Good luck supporting this in your filesystem.

> If the German speakers really wanted a change,...

Thanks, you just confirmed case sensitivity is not "hard science" no matter how hard Unicode tries to pretend it is. What a surprise considering it's part of natural languages. That's why it definitely has a place in high level interface user interfaces like file explorers, choosers and maybe interactive command lines even (with some autocorrection) but certainly not "hardwired" at a very low level in filesystems where it has already been seen causing damage.

Working with UTF-8 in the kernel

Posted Apr 4, 2019 13:00 UTC (Thu) by foom (subscriber, #14868) [Link] (4 responses)

> Interesting, references?

Search for $upcase -- the name of the 128KB pseudo file stored on in every NTFS filesystem. You can also look at the NTFS filesystem driver for Linux.

This file contains the corresponding uppercased character (2 bytes) for each one of the 65536 unicode characters. When windows wants to compare filenames, it simply indexes each character in each string through this table, to make an uppercase string, before doing the comparison.

When you reformat a drive it writes the newest mapping to the file, and that partition will use the same mapping as long as you keep it.

And, yes, I am quite aware that everyone who knows anything about unicode is crying out in distress at the utter WRONGness of what I said above...

But of course, the secret is that users aren't really the ones who care about case insensitive comparisons... They are using gui file pickers and such higher level tools where the filesystem case behavior doesn't matter.

Note the primary use cases given for Linux (samba exports, Android emulating a FAT filesystem on top of ext4) are all about *software* expectations, not humans. Software that was written with hardcoded filenames of the wrong case. That's why ntfs's braindead case folding is not really a problem.

Which does rather bring into question whether implementing "correct" normalization and case folding in Linux even has a point... It won't make it more compatible with the legacy software to do that...

Working with UTF-8 in the kernel

Posted Apr 8, 2019 6:21 UTC (Mon) by cpitrat (subscriber, #116459) [Link] (3 responses)

If the primary use case is to be compatible with NTFS, why not implement it the same way ? As I understand it, NTFS support will require a fake unicode version ?

Working with UTF-8 in the kernel

Posted Apr 8, 2019 21:49 UTC (Mon) by foom (subscriber, #14868) [Link]

I don't know.

It does seem rather incongruous to me to justify the feature via by pointing to samba's emulation of NTFS case folding, and Android's emulation of FAT file name lookup rules, but then implementing unicode normalization and correct unicode case folding...which those don't do.

Working with UTF-8 in the kernel

Posted Apr 11, 2019 20:49 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

Because, as I understand it, utf-16 is now seen to have been a mistake.

Forcing all filenames to be valid utf-16 will break quite a lot elsewhere ... I think that if you want to implement the utf universe properly in utf-16, you end up back with the 8-bit codeset mess, only bigger ...

Cheers,
Wol

Working with UTF-8 in the kernel

Posted Apr 11, 2019 23:15 UTC (Thu) by foom (subscriber, #14868) [Link]

Er what? I don't really understand your comment, but NTFS doesn't implement utf-16.

It stores filenames as arbitrary sequences of 16-bit values. There are a few tens of values you cannot use (ascii control characters 0-31, and some ascii punctuation), but everything else is fair game. In particular, invalid utf16 containing broken surrogate pairs is perfectly fine.

Working with UTF-8 in the kernel

Posted Apr 7, 2019 22:40 UTC (Sun) by jschrod (subscriber, #1646) [Link]

Ahem...

I think a case could be made that one could blame Unicode for *not* representing these different alphabets. After all, the same code point is used for "different" characters in the alphabets - if one agrees to your statement that these are really different alphabets.

But, in real life, too much water has flown under this bridge to discuss it outside an evening in a wine bar with some friends who are encoding freaks. I have to admit I would be part of such a discussion... ;-)

Cheers, Joachim

Working with UTF-8 in the kernel

Posted Apr 11, 2019 13:12 UTC (Thu) by robbe (guest, #16131) [Link]

I blame Unicode for the fact that they did not spend a codepoint for LATIN SMALL LETTER I WITH DOT ABOVE.

Working with UTF-8 in the kernel

Posted Mar 29, 2019 14:54 UTC (Fri) by mina86 (guest, #68442) [Link] (4 responses)

There's also the curious case of lc(ẞ) = ß but uc(ß) = SS. Or sigma having two lower case forms. I anticipate a world of pain and kernel devs will have only themselves to blame. ;)

Working with UTF-8 in the kernel

Posted Mar 30, 2019 4:27 UTC (Sat) by gps (subscriber, #45638) [Link] (1 responses)

Nah, that world of pain already exists. Today it is distributed among umpteen different application and library implementations that differ.

What's one more? Some of the above could even go away after this in some system designs.

Working with UTF-8 in the kernel

Posted Mar 31, 2019 14:41 UTC (Sun) by mina86 (guest, #68442) [Link]

> Nah, that world of pain already exists. Today it is distributed among umpteen different application and library implementations that differ.

That may be so but I'd rather my Samba server crashed than my kernel oopsed or executed malicious code because Unicode was handled incorrectly. Just like putting HTTP server inside the kernel wasn't a good idea, I'm not yet convinced that putting Unicode handling is.

Working with UTF-8 in the kernel

Posted Apr 8, 2019 6:24 UTC (Mon) by cpitrat (subscriber, #116459) [Link] (1 responses)

If only more people were as reasonable as Irish who changed the Gaelic alphabet to be compatible with typewriters (changing there accentuated laters with combinations of naked letters, like gh) and therefore now don't have any problem like this ...

Working with UTF-8 in the kernel

Posted Apr 11, 2019 20:57 UTC (Thu) by Wol (subscriber, #4433) [Link]

Well if you do this, aren't you going eventually to end up with just one letter in your alphabet? Okay, I'm being facetious, but this happened to the Roman alphabet in the ?1400s, when printing arrived. Pre-printing, the written English alphabet had a fair few more letters than 26. I'm not sure of the details, but at least one example is the replacement of thorn (looks like the Yen symbol) with Y, hence all the signs "Ye Olde Coffee Shop".

Cheers,
Wol