On the need for a new removable device filesystem

Removable storage devices, such as the USB "thumb drive," can be a pain. They are slow and often prone to errors, but, perhaps worst of all, they all seem to be designed for the VFAT filesystem. VFAT gets the job done much of the time, but it is showing its age; this filesystem was never meant for the size of contemporary devices or files. There is also the little nagging issue of the patents on the filesystem format and the associated Linux-hostile company that is actively asserting those patents. Despite all of this, removable devices are often the easiest way to ship files between machines. Given that, do we need to come up with a new filesystem to ease the pain of using these devices?

Dan Luedtke's answer is "yes"; he has implemented a new filesystem called "Lanyard" (or "LanyFS") intended for use on removable devices. He claims better performance and scalability than VFAT along with a native Linux implementation. The code shows its early-stage nature — there are a lot of things that would need to be fixed before it could be considered for inclusion into the mainline kernel — but the mainline is clearly where Dan would like it to go. The rest of the development community is not entirely convinced that we need a new filesystem for this use case, though.

The first question is: why not stick with VFAT? For all of its troubles, it has worked well enough for a long time. The biggest motivator for a change, arguably, is the 4GB limit on file size. One can deal with poor performance, especially when the real bottleneck is likely to be the device itself. But if one wants to store a sufficiently large file on the device, VFAT will simply fail. Such files are increasingly common, so users are running into this problem. The exFAT filesystem format is held out as an alternative, but it is far more proprietary than VFAT. Given that VFAT has already been the subject of lawsuits, vendors will think carefully before switching to exFAT; Sharp has licensed the filesystem for Android devices, but there do not appear to be a whole lot of other takers at this time.

Given increasing networking speeds, one could certainly consider just using the network to move a file that is too large for VFAT. On a local network this approach might well be faster than using a removable drive. Setting up network transfers is not always easy, though; most computers are, by default, configured in ways that do not allow random strangers to dump large files on their drives. Getting around that obstacle is likely to be too much even for moderately skillful users. Use of a third-party site to transfer files is workable when the files are small; even if it is possible for very large files, it's not something that will be tolerably fast on most networks.

Removable drives, instead, are easy, so the "sneakernet" approach to file transfer is likely to stay with us for some time. Does that mean that we need a new filesystem format to better support this use? Filesystem developer Ted Ts'o thinks not:

That is an interesting thought: Linux is now strong and prevalent enough that we can simply expect the industry to pick up our way of doing things. That approach has not always worked out in the past, but things might truly be different this time around. Increasingly, devices like music players, handsets, and digital cameras run Linux internally; these gadgets already are, to a first approximation, removable storage devices with a bit of extra hardware. Other devices, such as televisions, also tend to run Linux internally. Supporting a native Linux filesystem on these devices should be a relatively easy thing to do. It would be faster (assuming the underlying storage isn't severely optimized for VFAT only), more feature-rich, and lacking in patent aggressors. There is very little, in other words, not to like.

Well, there would be a few small problems. There are still some pesky users out there with non-Linux systems that might want to access the filesystems on their devices. In many cases, the increasing use of the MTP protocol could sidestep that question altogether; indeed, recent MTP-using Android devices are likely using it to export an ext4 filesystem. There would still be cases where users on these other platforms would want to mount filesystems directly, though, especially on pure storage devices; bringing proper implementations of Linux filesystems to those platforms is, evidently, not as easy as one might think.

Filesystems like ext4 also were not designed with removable devices in mind. They tend not to be all that robust against unexpected removal of the device unless fairly aggressive flushing of data is used (in fairness, VFAT filesystems are also easily corrupted that way). The file ownership model used by Linux filesystems tends not to translate well to removable devices, since one system's user IDs typically have no meaning elsewhere. So something like the user and group mount options patch may be required to make things work well. Most Linux filesystems have not been designed around the very large pages and erase blocks used on flash devices and, thus, do not perform as well as they could; see this article for lots of details. These are issues that can be worked out, certainly, but they remain in need of working out at this time.

There is one other complication: according to Arnd Bergmann there is another filesystem waiting on the wings:

Needless to say, such an entry has the potential to stir things up a bit. A filesystem designed with input from both "a major flash vendor" and a developer like Arnd should work well indeed on small removable devices and should be well integrated into Linux. This manufacturer could also employ the "include a windows driver in a small partition on the device" trick, making interoperability with most Windows systems Just Work. Putting the filesystem code into the Linux kernel would make support readily available on mobile devices. This scheme might just succeed.

So what we may see is not Linux pushing one of its native filesystem formats onto the world. Instead, the world might just adopt a new format that just happened to be well supported in Linux first. That could be the best of all worlds: we would have a way to interoperate on removable drives that is free, scalable, and widely supported. Getting there may well be worth the trouble of adding yet another filesystem type.

---

(Log in to post comments)