Filesystems and case-insensitivity

Posted Nov 29, 2018 12:41 UTC (Thu) by chithanh (guest, #52801)
In reply to: Filesystems and case-insensitivity by eru
Parent article: Filesystems and case-insensitivity

> I would hope that is never. UTF-8 can represent all characters now in practical use.

That is not correct. In particular, Unicode (and by extension UTF-8) is deficient regarding some characters in African languages, due to the Unicode consortium's policy regarding precomposed characters vs. combining diacritics. They don't want to introduce new equivalences.

to post comments

Filesystems and case-insensitivity

Posted May 29, 2019 23:00 UTC (Wed) by Serentty (guest, #132335) [Link] (5 responses)

This is not a deficiency with Unicode. Precomposed characters such as É have only ever been encoded in Unicode as a matter of compatibility with legacy encodings, and wouldn't have been included if not for this. They continue to be used because they save you a few bytes, which you might as well go for even if compression makes it moot in the end. Combining diacritics have always been the preferred method as they are much more flexible, and allow users to compose arbitrary characters without needing to constantly update their software or risk mojibake. Many scripts in Unicode work entirely though combining diacritics and it works just fine; the Indic scripts are good examples. It should be noted that the legacy encodings for these scripts usually worked that way as well. Conformant implementations will treat composed and decomposed characters identically, so the advantage of going down the rabbit hole of trying to provide every precomposed character anyone might ever want isn't really worth it when composition works just as well. If you notice that combining diacritics aren't giving you the nice hand-tweaked glyphs that precomposed characters are, and you end up with the diacritic looking all wrong, take it up with the developer of the text renderer or the font, because that's not how Unicode is supposed to work.

Filesystems and case-insensitivity

Posted May 30, 2019 14:18 UTC (Thu) by smurf (subscriber, #17840) [Link] (4 responses)

Hmm. If that rule had actually been followed, we'd still have room on the base plane (i.e. codepoints below 65536).

(How many primitives would you need for Chinese?)

On the other hand, in that case we wouldn't all use UTF-8 by now – simply because that would require twice the storage space for Chnese text, more or less. Nowadays that doesn't really matter, but at the time it was a problem.

Filesystems and case-insensitivity

Posted May 30, 2019 14:54 UTC (Thu) by excors (subscriber, #95769) [Link] (2 responses)

https://en.wikipedia.org/wiki/Template:CJK_ideographs_in_... has a helpful list with the numbers of CJK codepoints, and I assume only the earliest ones were needed for legacy compatibility - the rest were presumably added because they couldn't already be represented. Recently Unicode 10.0 added "CJK Extension F" (7473 codepoints) so it seems they're still not finished. Then there's all the other scripts being added, like Tangut ("a major historic script of China") with another ~7000 codepoints. And about 1700 emojis (https://unicode.org/emoji/charts/emoji-list.html).

Maybe the 64K limit could have lasted for many more years if they had made some different design choices early on, but given the goal of being a universal standard for all text, it seems inevitable the limit would be broken eventually. It's better to have broken it earlier than later.

Filesystems and case-insensitivity

Posted May 31, 2019 15:06 UTC (Fri) by smurf (subscriber, #17840) [Link]

True, that.

Seems that quite a few of Chinese people with interesting names (i.e. using archaic characters) suddenly couldn't get an official document any more because, surprise, their name wasn't in the "official" charset …

Filesystems and case-insensitivity

Posted May 31, 2019 18:37 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Technically, most of CJK characters can be decomposed into simpler characters. About 70% of Mandarin characters follow the "radical-phonetic" model and can theoretically be composed.

Filesystems and case-insensitivity

Posted Jun 6, 2019 5:09 UTC (Thu) by Serentty (guest, #132335) [Link]

Encoding Chinese text based on primitives is the Holy Grail of Chinese text encoding, but no one has actually been able to come up with a realistic solution for it, and it's probably just not realistic. Korean text on the other hand is really easy to encode based on primitives, as it's just 22 letters combined in predictable ways.