LWN: Comments on "Case-insensitive ext4"

Case-insensitive ext4

calumapplepie — Wed, 04 Oct 2023 18:51:37 +0000

You get an event when the queue overflows. If you clear the cache on receiving such an event, you can provide consistency guarantees, at the cost of bad performance while the cache is rebuilt. Since the queue is pretty big, overflows shouldn't happen too much.

Case-insensitive ext4

nix — Mon, 08 Apr 2019 19:38:42 +0000

inodes have no name, and we already have a mechanism for one file having as many names as you like: hard links. I'm sure this can be repurposed somehow...

Case-insensitive ext4

lkundrak — Mon, 08 Apr 2019 06:57:09 +0000

> I'd be inclined to copy the Windows trick, actually, where they have two file names.

Wasn't their patent on a file having two names what the TomTom lawsuit was about?

Case-insensitive ext4

Wol — Fri, 05 Apr 2019 20:01:12 +0000

I'd be inclined to copy the Windows trick, actually, where they have two file names.

Seeing as it seems it can only be activated on an empty directory, the i-nodes would have to have space for two entries - the canonical form and the user form. Any file-system access converts to canonical and searches on both, but a collision on the canonical name will take the appropriate behaviour.

That might also provide a way for switching a used directory to case-insensitive - the canonical field would start out empty, but writing to the directory would do a canonical search and return an error if there was a collision. Be a bit messy though possibly.

Cheers,
Wol

Case-insensitive ext4

Wol — Thu, 04 Apr 2019 17:09:06 +0000

> Where would calls to open() be getting these correctly named but incorrectly capitalised/normalised filenames from?

The user, maybe?

What about the use case where I type in a name in a picker, and it displays a bunch of matches?

Or what about the case where I typed in the name on the command line? Some of us still use a command line, you know ...

Cheers,
Wol

Case-insensitive ext4

bosyber — Thu, 04 Apr 2019 13:13:38 +0000

That might be working for some cases, but I wonder how I'd like that.

I am Dutch, currently living in Germany, and often conversing in English too (like here; plus sometimes receiving Czech documents). Now, none of those are really difficult cases, I think (SS/ss and ß notwithstanding), but they do have differences in how characters need to be interpreted, but while my usage is for a large part context/file dependant, there are overlaps in what I use when, sometimes in a single context, like in chats, online, and sometimes stored in a single directory, due to where, by whom, and for what a document was created.

Not sure how what you propose can work correctly when generalized. Imagine I for some reason add Turkish in the mix too, or when I use a more complex, different set of languages as daily use.

Case-insensitive ext4

jthill — Sat, 30 Mar 2019 22:28:20 +0000

Seriously, I think the only even halfway-respectable choice here is to encode the specific locale and case sensitivity with each path component. If that means having a filesystem (probably also system) catalog of encountered locales and a locale index on each directory entry, so be it. Case folding differs by locale, and locale can be either an intrinsic text attribute or a matter of user interpretation. Punting it to system administration is just begging for botchery, it leaves you with third-party implementations of arbitrary choices affecting the results of fundamental operations. If I think my `README` is case-insensitive and my `Makefile` is case-sensitive but `*.exe` should be case-insensitive, well, sadly enough that's a reasonably widely understood situation these days

Case-insensitive ext4

jra — Fri, 29 Mar 2019 22:55:00 +0000

We already have an incrementally maintained, userspace translation cache in Samba. It catches the simple cases where we've seen a filename before - we cache it.

Unfortunately it isn't enough. Cache misses are the problem. If the SMB client sends a filename "foo" and it isn't in the directory, we don't know if it doesn't exist, or exists under another case (e.g. as "Foo"). In that case we need to scan the directory. This gets really expensive, really quickly.

We don't negatively cache as we're often used to export filesystems that local processes are also modifying.

I've been wanting a case-insensitive filesystem lookup option in Linux for a long time (I think ZFS and XFS already have it, however flawed).

Case-insensitive ext4

mirabilos — Fri, 29 Mar 2019 22:41:00 +0000

The whole idea is crazy.

Additionally, did someone ask the Turkish people? (PHP fucked them up, because the word “function” contains a dotted i, and PHP insists on being case-insensitive…)

There, I ≠ i because they have I ↔ ı and i ↔ İ so you need locales.

This is the ultimate proof that case-insensitivity cannot (and therefore MUST NOT) be done on the filesystem level.

Case-insensitive ext4

nim-nim — Fri, 29 Mar 2019 10:52:42 +0000

Assuming the lowest layers of the stack could handle conversions transparently (which I'm doubtful of, that would require low-level knowledge about every possible encoding variation on earth), you still need to know the encoding(s) you start with. Meaning, you have to put at least one pivot encoding definition inside your filesystem.

That's the part people object to, because they are used to the simplicity of pushing encoding problems somewhere else, with "filenames are streams of bytes". Which was not true even for original UNIX. Actual original Unix filename bytes were 7bit ASCII bytes and nothing else.

But 7bit ASCII is useless in a modern i18n world. So you need to record other pivot encoding(s) in filesystems¹.

¹ Record, not reproduce the mistake of original UNIX, that assumed there was a single encoding that would never evolve so there was no need to make it explicit; easy mistake to made in the simpler computer age they lived in; inexcusable mistake to make today.

Case-insensitive ext4

Cyberax — Fri, 29 Mar 2019 09:20:18 +0000

> How is a call to open() getting the filename to open? Either it's going to from an existing directory scan, in which case the capitalisation/normal form should already be correct, or it's going to be because a user has selected a file - in which case the shell/picker/whatever should be able to do that work already?
You have an SMB request to open a file, with a file name. There's nothing else.

You can try a happy case and just attempt an open() with the provided name. If it fails, you need to scan the directory to find a matching file with a different case.

And you can't really cache the negative result, patterns like "if !exists(fname) {creat(fname);}" are exceedingly common.

Case-insensitive ext4

Karellen — Fri, 29 Mar 2019 09:16:09 +0000

Why?

How is a call to open() getting the filename to open? Either it's going to from an existing directory scan, in which case the capitalisation/normal form should already be correct, or it's going to be because a user has selected a file - in which case the shell/picker/whatever should be able to do that work already?

Where would calls to open() be getting these correctly named but incorrectly capitalised/normalised filenames from?

Case-insensitive ext4

Cyberax — Fri, 29 Mar 2019 04:21:38 +0000

inotify is not recursive. It's also best-effort and its notifications are asynchronous.

fanotify is better, but it also can drop events from time to time under high load.

Case-insensitive ext4

pabs — Fri, 29 Mar 2019 03:03:37 +0000

What are they missing now that recent Linux versions offer rename notifications and other directory change notifications?

Case-insensitive ext4

jccleaver — Thu, 28 Mar 2019 21:01:40 +0000

> Do you just assume that *all* paths can be normalized regardless of location or host filesystem?

I think by System 7.5 (or 7.1 Pro) you did, because if I recall correctly that's how File Exchange/PC Exchange did its work.

Remember, in classic Mac OS the colon ':' was the directory separator in paths, and you could use '/'s to your heart's content. Actually, you could use pretty much anything to your heart's content, including spaces, punctuation (since no one in the Mac side cared about extensions) and even weird graphs like the f-hook or florin https://en.wikipedia.org/wiki/%C6%91#Appearance_in_comput... , which I still find myself occasionally doing on OS X 20 years later.

Anyway, with /. \. and : being used in different locations, there was definitely path-mangling going on below the interface. But general users didn't have to care, and most Mac programs didn't deal with constructed path names, and *never* had to worry about shell-quoting for spaces and whatnot.

Between this freeform text attitude, the resource and data fork dichotomy, and the use of Type and Creator codes, I definitely feel like we've lost some good capabilities on the Mac side in the quest for broader interoperability.

Case-insensitive ext4

k8to — Thu, 28 Mar 2019 20:40:23 +0000

I agree with most of that.

However, I'm not sure that allowing people to create both Foo and foo and then having applications use an interface that "asks for foo" in an insensitive fashion is going to produce a lot of happiness.

Case-insensitive ext4

k8to — Thu, 28 Mar 2019 20:35:24 +0000

Yeah, mounting case sensitive filesystem on classic MacOS would have been messy. I'm sure I did this at some point with e.g. Basilisk II mounting the Linux filesystem underneath it, but that had a hefty translation layer to support the other oddities of classic MacOS like forks etc.

I think that was the approach taken by other people too, probably one of Apple Single or Apple Double representations which probably had some solution for NFS which was still in vogue in the 90s.

It wasn't that nice an experience for the Mac users or the non-mac users. I never programmed against it to experience the extra sharp edges, though.

Case-insensitive ext4

mathstuf — Thu, 28 Mar 2019 20:14:09 +0000

I still wonder how this would have worked even on Classic Mac OS. Do you just assume that *all* paths can be normalized regardless of location or host filesystem? If so, do you just not support filesystems with alternate paths? Though I suppose the Windows solution of mangling unsupported names on the render side works too[1]. However, this means that any path manipulation has to do a syscall to get some canonical representation of a path or each program has to have a "pathcmp" function to determine that "foo" and "FOO" are really the same thing.

Would you have expected the shown Makefile snippet to work on Classic Mac OS or would an error that "no rule to make FOO" be acceptable?

[1]Making a path appear in Explorer via a network share with the name "CON1" renders as some mangled name. Creating a file with that mangled name then shows two files with the same name appear. Deleting either one via the UI deletes the one with the real mangled name first (I assume given a HANDLE, they can be differentiated).

Case-insensitive ext4

jccleaver — Thu, 28 Mar 2019 19:11:00 +0000

Ultimately, this really brings to mind how important initial design is for things.

Classic Mac OS was designed with case-insensitivity in mind, had no manual tools that needed to be imported with minimal effort rather than a complete rewrite, and had no shell mechanics to emulate.

Case Insensitivity #JustWorks when people expect it and are going through translation layers (and aren't in the business of writing drivers), and doesn't when people assume low level access.

Case-insensitive ext4

Cyberax — Thu, 28 Mar 2019 18:36:36 +0000

> As I pointed out, this isn't true, at least not on Linux: It would be possible to use an incrementally maintained
Nope. There is no way to maintain this cache with any sort of consistency guarantees. Linux filesystem change notifications are not up to it.

Case-insensitive ext4

rweikusat2 — Thu, 28 Mar 2019 16:10:22 +0000

Someone claimed that Samba opening a file residing on a case-sensitive filesystem would require a pre-open, linear directory traversal. As I pointed out, this isn't true, at least not on Linux: It would be possible to use an incrementally maintained, userspace translation cache instead, however, unless I again have to use Samba for something in a resource-constrained environment, I'm not going to implement that and in the unlikely case that this would happen, I'd certainly not go through the rather pointless hassle of trying to contribute a non-trivial change to an open sausage project, as I have neither the time to do this nor the social skills and pedigree to do so successfully.

Case-insensitive ext4

nybble41 — Thu, 28 Mar 2019 15:58:10 +0000

> Casing is something else but once you get past the encoding point casing becomes a less harder to tackle.

Not much less. Casing rules depend not just on encoding but also locale, and while it may be practical to enforce a single universal encoding and normalization scheme you're definitely not going to get away with enforcing a single universal locale.

The logical way to handle normalization is to simply disallow non-normalized filenames. The kernel doesn't change the encoding or compare different normal forms, it just verifies that the names of new files are in a particular normal form and returns an error if they aren't. Since all names are already in the same normal form comparisons reduce to exact binary matches. The equivalent for case would be to disallow either lowercase or uppercase characters in filenames (assuming you could even clearly define what is "uppercase" or "lowercase"—it depends on the locale). People put up with that in the DOS era but I don't think it would be considered acceptable today.

The odds that encoding or normalization would be permitted to vary per-filesystem or per-subtree are negligible. Applications aren't prepared to deal with that, nor should they be expected to do so. Any conversions needed for shared filesystems should be handled at the lowest layers of the filesystem, between the storage or network and the kernel.

Case-insensitive ext4

smurf — Thu, 28 Mar 2019 14:28:21 +0000

Yes. Besides case insensitivity there's also the issue of differently-normalized file names. I would like to have one well-defined on-disk normal form. Otherwise I save "hëllo.txt" (e, combining diaraesis) and then fail to open "hëllo.txt" (e-with-diaraesis). This problem affects my desktop interface as well as web servers with "interesting" URLs.

Case-insensitive ext4

mjthayer — Thu, 28 Mar 2019 11:20:27 +0000

To my mind part of the problem is the expectation that there is a clean solution to the problem. Human writing has been in existence and evolution for thousands of years, and not as a single consistent entity at that. Even Latin script is split into many many sub-scripts (the Turkish "i" is not an electronic invention). Computer encodings, locales and filesystems have not been around that long, but have also picked up more than enough legacy which we can't just wish away. And we want file names to solve a range of problems, some of which are more relevant to users and some to developers.

Having said that, having file systems encode file names as byte strings and having mechanisms to query those uninterpreted or case-insensitive (or whatever) as processes require seems to me a reasonable square of the circle.

Case-insensitive ext4

hkario — Thu, 28 Mar 2019 11:05:14 +0000

Good luck writing about the German ß, that can be upper-cased to either SS (the old convention) or ẞ (the new convention).
But when you down-case SS, you do it to ss.

Case-insensitive ext4

mpr22 — Thu, 28 Mar 2019 10:44:01 +0000

Case-insensitivity on ext4 volumes has to be done in userspace, because doing it in the kernel breaks userspace.

I agree that the Gtk file chooser having case-sensitive autocomplete is daft, but... I don't actually care, because I hate the Gtk file chooser anyway for other, more fundamental design decisions.

Case-insensitive ext4

hkario — Thu, 28 Mar 2019 10:42:56 +0000

you can treat them as the same to the same degree as you can consider "y" to be equivalent to "i", or "oo" to "u"

Just because you grew up with an alphabet that has 26 letters doesn't mean that it's the only alphabet in use.

Case-insensitive ext4

nilsmeyer — Thu, 28 Mar 2019 09:01:30 +0000

Maybe that's just a matter of convenience? I don't know about keyboards for France, when I want to write É I have to hit three keys in a specific order.

Case-insensitive ext4

nim-nim — Thu, 28 Mar 2019 08:47:47 +0000

> For some strange reason many (most?) French people think the upper case are without accents

People are just used to broken systems (broken keyboards on typewriters and computers, broken apps, in legacy typesetting "someone broke the small bit of lead corresponding to the diacritic on the capitalized letter"). Humans habits are a huge source of inertia. When Microsoft finally got around to fix Office for French some Microsoft clients actually complained it was now correcting words to the correct spelling.

Proper typesetting shops take care to type correct french (typesetting apps correct the windows user breakage), and Linux diverged long ago from the "official" AZERTY layout to make uppercase with diacritics easy to type (French Canadians were smarter: they fixed their official layout to put caps with diacritics proeminently on them. Pity you can't buy Canadian French keyboards easily in France).

Case-insensitive ext4

nim-nim — Thu, 28 Mar 2019 08:39:42 +0000

Any shared filesystem (network filesystem, removable media) will soon become completely unusable if different systems write on it with different default encodings. Treating filenames as opaque bunches of bytes does not work because you need to convey filenames to humans at some point. Humans do not understand raw bytes they understand decoded bytes, and that requires knowledge of the encoding used in filenames.

So any shared filesystem will need to export to userspace the encoding used for each part of its tree (either a single encoding for everything, or separate encodings per subtree).

Casing is something else but once you get past the encoding point casing becomes a less harder to tackle.

Case-insensitive ext4

daniels — Thu, 28 Mar 2019 08:11:02 +0000

> I've yet to see a legitimate use case for putting this brain damage in the kernel. Does anyone actually have one?

No, everyone involved is just doing this for absolutely no reason at all. Weird.

Case-insensitive ext4

Cyberax — Thu, 28 Mar 2019 07:35:31 +0000

This is doable and is fairly easy, given that Samba has a well-defined pluggable VFS layer.

But this will break a ton of other software that wants to directly modify the disk files. It will also mean that Linux's VFS is inadequate for a fairly common use-case.

Case-insensitive ext4

andrewsh — Thu, 28 Mar 2019 07:32:46 +0000

That’s why upper-case in documents should be a text style, not ACTUAL capitals in the text.

Case-insensitive ext4

patrakov — Thu, 28 Mar 2019 07:28:28 +0000

Wouldn't it simplify things if SAMBA stopped any attempts to export an existing directory tree? I.e. mandate that the only way to make a new file exported is to copy it in via the SMB protocol, quite possibly from localhost. Keep filenames opaque, keep files in a clearly-private area, teach users not to mess with them (like they don't directly mess with MySQL files). Keep whatever attributes Windows needs in some sort of a database.

Case-insensitive ext4

mathstuf — Thu, 28 Mar 2019 03:56:31 +0000

What do you expect of tools that have to deal with non existent files in a case insensitive world? Mainly, I'm thinking of build tools here, but there are other categories too where this crops up. Should make expect that there is a dependency here?

foo:
touch foo
bar: Foo
cp Foo bar

Because if so, this means that tools now need to make a syscall just to do path manipulation to be accurate (something like canonpath() that would give a path which is the same for all equivalent input paths maybe by doing tolower() and normalization). And it has to work for paths that don't exist yet. And I don't think that can even be correct because that path might end up having a bind mount in there at some point which changes behavior (yeah, low chance, but kernels don't always have that luxury).

Yeah, case insensitivity might be useful at the UI level, but even there you still have to deal with paths using binary data or invalid utf8 because a file that the GUI can't delete is a wonderful thing to diagnose and resolve. Personally, I don't find it that useful (but I encourage you to file an issue against GTK for the completion thing).

Case-insensitive ext4

dw — Thu, 28 Mar 2019 02:10:00 +0000

As a recent MacOS refugee (after a long previous history on desktop Linux), frankly I find dicking around with case on a desktop extremely annoying, after discovering things don't have to be that way. Sure it's supposed to be in userspace -- good luck retrofitting all that, and really it's just passing the blame. Darwin has an in-kernel approach that works for users, and I'd be very happy to enable this flag the instant it becomes available.

I was disgusted just last night to discover a Gtk chooser dialog's autocomplete was case sensitive. In a GUI. In 2019. Total disconnect between Linux and what the real world has been doing successfully for decades, and what actual users expect. No doubt someone will pop up to say 'but I prefer it that way', well, you're free patch whatever brainwrong you like into your desktop, but most people cannot and do not want that -- it's why contemporary developers are walking around with MacBooks rather than Linux boxes

Case-insensitive ext4

dw — Thu, 28 Mar 2019 02:05:26 +0000

as a recent MacOS refugee, frankly I find dicking around with case intensely annoying on a desktop after discovering things don't have to be that way. Sure it's supposed to be in userspace -- good luck with that. Darwin has an approach that works for users, and I'd be very happy to enable this flag the moment it becomes available.

Was disgusted just last night to discover a Gtk chooser dialog's autocomplete was case sensitive. In a GUI. Total disconnect between Linux and what the real world has been doing successfully for decades now..

Case-insensitive ext4

rahulsundaram — Thu, 28 Mar 2019 00:50:57 +0000

> there are obviously "file server use cases" where the existing mechanisms perform well enough.

Have you talked to Samba developers and asked them if they are happy with the current performance or would like to see better support from the kernel? If you haven't I would encourage you to do that or talk to enterprises supporting Samba or even large customers. I think you will find that perspectives useful to add to your opinions.

Case-insensitive ext4

Cyberax — Wed, 27 Mar 2019 21:26:25 +0000

Samba exists, but it does case insensitivity in a very expensive way. There are other use-cases as well, like mounting FAT filesystems.

I've seen this firsthand - I'm using a Linux server for TimeMachine backups for Mac OS X. TimeMachine is braindead - it creates hundreds of thousands files in the same directory. With the default settings Samba slowed down to a crawl.

Fortunately, TimeMachine doesn't care about file name cases. So by following steps from here: https://wiki.samba.org/index.php/Performance_Tuning I was able to speed up backups by something like 10x. This is not insignificant and it would be nice for Linux to handle similar use-cases natively.

Case-insensitive ext4

Cyberax — Wed, 27 Mar 2019 21:19:13 +0000

As I understand, the kernel will keep track of canonicalized names in file cache, so it won't have to do a search.

There's also the problem of making sure that no duplicate files exist.