LWN: Comments on "The ABI status of filesystem formats"

Why ext4?

mina86 — Mon, 26 Oct 2020 20:57:18 +0000

FYI, there’s Rock Ridge extension which adds POSIX semantics to an ISO file-system.

Why ext4?

eru — Tue, 20 Oct 2020 05:38:48 +0000

Squashfs, unlike cramfs, does not have problematic size limitations: "Files up to 2^64 bytes are supported; file systems can be up to 2^64 bytes" (https://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html).

Why ext4?

WolfWings — Sun, 18 Oct 2020 16:19:22 +0000

Oh, yup, I'd missed that one going through the main filesystems, that one looks to nail it and he "ROMFS but a full Linux filesystem" so it's just about ideal.

The ABI status of filesystem formats

rwmj — Thu, 15 Oct 2020 11:11:36 +0000

I'm wondering if what Josh really wants is some kind of deduplicating block layer. I have a (non-upstream) nbdkit allocator for this. The devil may be in the details however since it may be possible that the bitmaps being combined are not actually identical at the block size level.

Why ext4?

NYKevin — Tue, 13 Oct 2020 15:30:19 +0000

What about EROFS? It doesn't look very complicated, but it does appear to support most things you might reasonably want to do.

https://www.kernel.org/doc/html/latest/filesystems/erofs....

The ABI status of filesystem formats

marcH — Tue, 13 Oct 2020 03:29:02 +0000

> The way i see it, the only thing a formal spec does is to force upon (someone) to take ALL discussions about the subject of the article up front.

No, you can write a spec that formalizes an implementation. This is how most RFCs are written.

The ABI status of filesystem formats

ncm — Mon, 12 Oct 2020 16:42:52 +0000

I understand the desire to use ext4 for a read-only fs as a need for features maybe not-supported in certain other filesystems.

I don't understand a reluctance to certify e2fsck as the arbiter of what may be mounted as an ext2/3/4 filesystem. If it is discovered that some bit pattern is a security hole, surely e2fsck will be updated to patch up an image that tickles that security hole?

It seems not necessary to remain bitwise compatible with filesystems that *current* e2fsck would alter, but that older e2fsck would allow. e2fsck should continue to accept non-broken images, and to fix known-broken ones. So, we could accept a future kernel requiring that a file system image that used to mount as-is be fixed by a new e2fsck before it may be mounted. We could even have e2fsck refuse to fix certain old images it finds too confusing, as it might for any damaged image.

make_ext4fs to generate filesystem

mebrown — Mon, 12 Oct 2020 16:36:06 +0000

The yocto build system has a utility called 'pseudo'.

It uses an LD_PRELOAD library such that anything doing a mknod() libc call is saved to a database instead of invoking the underlying mknod() call, and instead just creates a regular placeholder file.

Then when you run mke2fs, similarly the LD_PRELOAD then hooks stat() such that mke2fs reading the placeholder file actually reads the details from the database instead of the filesystem.

Seems a little hacky at first, but seems to work well enough in practice.

The ABI status of filesystem formats

error27 — Mon, 12 Oct 2020 13:46:29 +0000

I didn't mean writing a spec is a waste of time. Specs are good because they encourage rigorous thinking. I meant it would be very difficult and a waste of time to try detect filesystems like Josh's in advance and prevent them from mounting.

But although specs have their uses, in the kernel, specs cannot work in the way you have said. The rule is never "You can't break the spec." The rule is "You can't break userspace."

Here is an example. Ten years ago glibc changed memcpy(). https://lwn.net/Articles/414467/ The spec said that the new implementation of memcpy() was valid but kernel developers find this attitude rage inducing. We would never do that in the kernel. You could theoretically still change memcpy(), but first you would have to fix all the applications that rely on the old behavior.

In the glibc example, it's not the flash developers who suffered, it's the regular Linux users. For the users, they had a program which worked and now it doesn't work. They don't care about specs. As kernel developers we care about users first and the spec second.

Why ext4?

WolfWings — Sun, 11 Oct 2020 21:27:59 +0000

SquashFS _requires_ compression, so there'd be a compression -> decompression layer involved. CRAMFS has a ~256MB total size limit. ROMFS is 32-bit so it'd run into obstacles around the 4GB mark (or maybe 2GB, I don't think it's been tested for >2GB total size to my knowledge) but it would be the lightest weight option IMHO.

But this does boil down to they were abusing an invalid filesystem data structure configuration that was not previously checked. That they were declaring that a particular flag marked the filesystem as such a monstrosity doesn't change that they were laying multiple copies of a data structure on top of each other instead of pushing upstream for a proper "read only" flag that did away with those structures entirely is the core invalidity.

The ABI status of filesystem formats

WolfWings — Sun, 11 Oct 2020 21:23:29 +0000

...this seems like they're simply using the wrong filesystem entirely. ROMFS has been around since somewhere in the 2.6's and compiles down to literally a couple kilobytes of total binary size in most cases.

The ABI status of filesystem formats

NYKevin — Sun, 11 Oct 2020 19:55:32 +0000

> So probably that approach is a waste of time.

I find this remark deeply confusing, so much so that I think I must have misunderstood you.

I proposed creating a formal spec, and declaring that to be the defining line of "what the kernel is prepared to support." In response to this, you described what happens if you don't have a spec and/or don't care about said spec, and then ambiguously said "that approach is a waste of time."

It sounds as if you were trying to refute my suggestion, but your argument before this line appears to *agree* with me, because it describes problems that arise when you don't have a spec. So, if you could clarify your position, that would help me to understand your argument a lot better.

The ABI status of filesystem formats

error27 — Sat, 10 Oct 2020 19:43:38 +0000

The problem is that we don't care about a spec, we care about "User space used to work and now it doesn't." So the fix is to program a lot of validation into the kernel so that Josh's filesystem never works from square one. If it never worked to begin with then it can never be broken. It's a lot of work to code this and it doesn't necessarily make Josh any happier.

So probably that approach is a waste of time.

make_ext4fs to generate filesystem

tytso — Fri, 09 Oct 2020 17:42:46 +0000

Sorry: s/libsparse file/libsparse library/

make_ext4fs to generate filesystem

tytso — Fri, 09 Oct 2020 17:40:07 +0000

Interesting. I didn't know about this feature in the 6th edition Unix's mkfs. It's not something which the BSD Fast File System supported, and my involvement with Unix started with BSD 4.3.

Using a proto file so that the system can more easily create device files does make sense, and of course these days we need to have something which supports SELinux attributes as well. That's one of the reasons why contrib/e2fsdroid exists. (The other is that e2fsdroid uses the Android libsparse file to create an image which omits the all zero blocks and which is compatible with the existing Android image creation tools, and I didn't want to drag that into mke2fs as a shared library dependency, since that would make distro installer release engineers.... cranky.)

Patches to support some kind of proto file which supports SELinux attributes, and which uses the qemu image file creation functions in libext2fs for mke2fs would certainly be gratefully accepted. If we had that, plus the per-block dedup functionality to create shared blocks file system added, then yes, we could upstream all of that functionality into mke2fs, and allow other embedded systems developers access to e2fsdroid functionality without having to build AOSP....

make_ext4fs to generate filesystem

jhhaller — Fri, 09 Oct 2020 15:05:26 +0000

The -d option would still require root access to allow creating device file entries which may not match the host's devices.
Setting the correct file ownership is another aspect requiring root access. The protofile addresses both of these issues,
as it overwrites the filesystem owner, permissions, and allows creating device files. I doubt mkfs was originally
concerned with CI systems, but creating filesystem images on a non-native host was likely an early concern.

Why ext4?

eru — Fri, 09 Oct 2020 12:02:33 +0000

>There are filesystems which are designed to be read-only, such as ISO9660. Why abuse ext4?

Maybe he needed a file system that supports all features expected of a native Linux file system. ISO9660 does not qualify. (But squashfs would? It would also automatically save space).

The ABI status of filesystem formats

roc — Fri, 09 Oct 2020 10:35:17 +0000

What I meant to add was: if you write formal specifications the right way, you can use these tools to get more value out of the spec, which tilts the cost/benefit tradeoff towards formal specifications.

The ABI status of filesystem formats

roc — Fri, 09 Oct 2020 10:33:49 +0000

One nice thing about really formal specs (as opposed to careful English specs, which aren't really formal) is that for many specification languages you can find (or write) tools that leverage the specs to do useful work. For example, for a language like Alloy, fuzzing and model-checking tools can automatically generate testcases that are valid according to the spec and exercise all or most of the interesting edge cases (including many that the spec authors didn't think of).

You may also be able to prove useful consistency properties from the spec, e.g. that a "valid" filesystem doesn't have unaccounted-for blocks, etc.

Why ext4?

epa — Fri, 09 Oct 2020 08:06:54 +0000

Yes, I wasn't replying to your comment, but to sfeam's one.

The ABI status of filesystem formats

bangert — Fri, 09 Oct 2020 07:11:22 +0000

A formal spec is not the panacea that one may be inclined to assume.

The way i see it, the only thing a formal spec does is to force upon (someone) to take ALL discussions about the subject of the article up front. That's practically impossible to get right and specifically when done well takes a huge amount of effort. If the developers were forced to do it, it would consume their time for a very long time - if someone else were to do it it would/could have a higher degree of defects.

You are highly likely to just debate the formal specification instead of the implementation.

While these objections apply to many situations the extra effort a formal spec introduces is worth it in (some|other) cases.

And although the conclusion of the article, that this subject will likely be discussed again, may be true, that is not in itself a bad thing (and the article does not say so).

Why ext4?

Yenya — Fri, 09 Oct 2020 06:38:22 +0000

Well, my (rhetorical) question was, why Josh Triplett even decided to use this "optimized" ext4 image instead of going for some filesystem which is read-only by design. Not whether there can arise the same kind of problems with ISO 9660.

Why ext4?

epa — Fri, 09 Oct 2020 03:45:17 +0000

ISO 9660 has a standard formally specifying it. That means you can decide whether a given file system image is valid, and if the kernel doesn’t work with a valid image, that is clearly a kernel bug. So no, it doesn’t quite have the same general problem as ext4.

In principle the point still stands: there could be a bug in the kernel code when handling a malformed file system image, and that bug could cause security problems, yet somebody might be relying on the exiting behaviour because they use one of these malformed images. But that seems much less likely with ISO 9660, partly because it’s much simpler than ext4, and partly because it’s understood by many different OSes, so you’d soon find out if your image depended on a Linux-specific quirk.

I have a CD-ROM that’s meant to be in ISO 9660 format but many of the filenames contain the / character. Linux does not handle this well. But it’s clearly incorrect by the standard, so you can say this is no bug in Linux.

make_ext4fs to generate filesystem

djwong — Fri, 09 Oct 2020 02:58:03 +0000

Yeah... mke2fs -d takes care of slurping a directory tree into the new filesystem now.

Also, that's a very interesting man page link-- now I've learned where the mkfs.xfs "protofile" format comes from! The manpage for mkfs.xfs even features a very similar example file.

Er, thanks!

Why ext4?

NYKevin — Fri, 09 Oct 2020 01:14:25 +0000

But that argument even applies to e2fsprogs itself. How do we know there isn't some weird combination of flags you can pass to tune2fs or mke2fs which somehow makes an evil filesystem? At that point, you would have to break e2fsprogs backcompat anyway. This is probably a lot less *likely* than the "some third-party tool generates a weird filesystem image" case, but some of those third-party tools are actually rather popular (e.g. the article mentions FreeBSD), and breaking them would also be unfortunate. So I'm not sure how much sense it makes to draw a bright line around e2fsprogs as the single point of compatibility.

make_ext4fs to generate filesystem

jhhaller — Thu, 08 Oct 2020 22:39:03 +0000

I wrote an alternate version of this tool at Alcatel-Lucent (twice because the first one got lost). Historically, Unix mkfs could make a filesystem from a list of files and contents, and write that filesystem to a file or device. Most any team generating filesystem images needs such a tool if they want to generate that ISO without needing root access. And, who really wants to give root access to one's CI system? We typically used this as a TFTP boot image for embedded Unix systems, and the filesystem was in RAM. That was another story, as the TFTP large block option was required, and no one had implemented a TFTP server with large block support at that time, and the filesystems were typically more than the 64K 512 byte block TFTP transfer size limit.

It was not hard to slightly modify mke2fs to take a manifest of files (and file of contents for regular files), and use the functions available to mke2fs to write the special files, permissions, directories, and file content to the ISO. I based the manifest loosely on the syntax of the contents of the proto argument of Unix 6th edition mkfs command - http://man.cat-v.org/unix-6th/8/mkfs - and added options for other file types including both symbolic and hard links. But, by using mke2fs, it at least used the functions built into the filesystem to populate the parts of the filesystem, making corruption significantly less likely. It would make two passes through the internal data model of the manifest, the first time writing inodes, and the second time writing the file contents. I don't remember if all-zero blocks were suppressed to support holes, I suspect not, as few files we installed had large all-zero blocks.

It was just more work than I wanted to do to get approval for releasing the changes externally, and get it accepted into ext4 source code, so it stayed internal. Since we never redistributed the tool externally, there was no need to share it, even though the change was under GPL. Sorry, Google and Ted Ts'o.

The ABI status of filesystem formats

roc — Thu, 08 Oct 2020 22:10:42 +0000

C++ is an extreme example. Filesystem formats are vastly simpler than C++. We know how to write formal specifications for quite complicated things.

The ABI status of filesystem formats

khim — Thu, 08 Oct 2020 20:49:54 +0000

Format specification only works for simple data structures. Look on C++: it has formal spec, ISO-approved one… yet discussions about if something is a valid C++ or not is not uncommon.

Why ext4?

Yenya — Thu, 08 Oct 2020 20:23:20 +0000

Well, a read-only by-design filesystem would not need a bitmap of free blocks and free inodes. And being able to make the r/only filesystem smaller for the same amount of data makes sense, unlike that abuse of ext4.

Why ext4?

sfeam — Thu, 08 Oct 2020 20:19:22 +0000

"There are filesystems which are designed to be read-only, such as ISO9660. Why abuse ext4?"

Choosing a different filesystem might lessen Ts'o's personal stake in the push-back but the same general arguments and principles would apply to any filesystem, right? The most solid argument Ts'o raises is that a read-only file system may still be a malware vector if the kernel code handling it fails to consider some edge case or unforeseen content in a data structure. Any patch to fix that vulnerability could hypothetically break an out-of-tree application that had been relying on the previous behavior. Is that to be considered a regression or not? In fact the general argument applies even if the filesystem is not read-only.

The ABI status of filesystem formats

khim — Thu, 08 Oct 2020 19:51:23 +0000

Agree 100% there. I'm not sure I would want to push e2fsprogs to support it, but it sounds precisely as something much more useful then author who made this tool thinks.

I only hope that's not something which differentiates the product which he is developing from other such tools and it could be open-sourced and shared.

Why ext4?

Yenya — Thu, 08 Oct 2020 19:45:57 +0000

There are filesystems which are designed to be read-only, such as ISO9660. Why abuse ext4?

The ABI status of filesystem formats

NYKevin — Thu, 08 Oct 2020 18:04:07 +0000

My 2¢ (which should not count for much, seeing as I am neither a Linux developer nor someone who regularly works with filesystem images):

This problem would be a lot easier if ext4 had a formal specification saying "This is what a valid filesystem should look like." I'm not really happy with the idea that "valid" just means "something you got from mke2fs(8)." Obviously, people are going to want to do experimental things that are not within the scope of a general-purpose tool like mke2fs. A formal spec would help to delimit the lines between "weird, but supported" and "don't do that, or else you're on your own."

I get the sense that some developers* don't like formal specs. A formal spec does not need to be huge or complicated like the C or POSIX standards; it could just be a short description of what e2fsprogs currently expects and supports, in plain and simple language. In fact, I imagine they could take https://www.kernel.org/doc/html/latest/filesystems/ext4/i..., sweep it for bugs, ambiguities, or outdated information, and then slap a "this is the formal spec" notice on it. This would not necessarily be easy (ambiguities in particular are hard to find and hard to fix), but it wouldn't require convening a huge standards committee or something ridiculous like that.

* This is not a reference to any individual, and it is especially not a reference to Mr. Ts'o. I do not know what Mr. Ts'o thinks of formal specs.

The ABI status of filesystem formats

sbaugh — Thu, 08 Oct 2020 17:46:29 +0000

Triplett's library sounds interesting and somewhat-generally-useful, actually. Repeating the description here:

>The short version is that I needed a library to rapidly turn
>dynamically-obtained data into a set of disk blocks to be served
>on-the-fly as a software-defined disk, and then mounted on the other
>side of that interface by the Linux kernel. Turns out that's *many
>orders of magnitude* faster than any kind of network filesystem like
>NFS. It's slightly similar to a vvfat for ext4. The less blocks it can
>generate and account for and cache, the faster it can run, and
>microseconds matter.

It's an unusual use case, but it doesn't seem like one which is totally unacceptable. I could imagine using it myself. It would be nice to see it upstreamed to e2fsprogs.