Removable storage devices, such as the USB "thumb drive," can be a pain.
They are slow and often prone to errors, but, perhaps worst of all, they
all seem to be designed for the VFAT filesystem. VFAT gets the job done much
of the time, but it is showing its age; this filesystem was never meant for
the size of contemporary devices or files. There is also the little
nagging issue of the patents on the filesystem format and the associated
Linux-hostile company that is actively asserting those patents. Despite
all of this, removable devices are often the easiest way to ship files
between machines. Given that, do we need to come up with a new filesystem
to ease the pain of using these devices?
Dan Luedtke's answer is "yes"; he has
implemented a new filesystem called "Lanyard" (or "LanyFS") intended for
use on removable devices. He claims better performance and scalability
than VFAT along with a native Linux implementation. The code shows its early-stage
nature — there are a lot of things that would need to be fixed before it
could be considered for inclusion into the mainline kernel — but the
mainline is clearly where Dan would like it to go. The rest of the
development community is not entirely convinced that we need a new
filesystem for this use case, though.
The first question is: why not stick with VFAT? For all of its troubles, it
has worked well enough for a long time. The biggest motivator for a
change, arguably, is the 4GB limit on file size. One can deal with poor
performance, especially when the real bottleneck is likely to be the device
itself. But if one wants to store a sufficiently large file on the device,
VFAT will simply fail. Such files are increasingly common, so users are
running into this problem. The exFAT filesystem format is
held out as an alternative, but it is far more proprietary than VFAT. Given
that VFAT has already been the subject of lawsuits, vendors will think
carefully before switching to exFAT; Sharp has licensed the filesystem for
Android devices, but there do not appear to be a whole lot of other takers
at this time.
Given increasing networking speeds, one could certainly consider just using
the network to move a file that is too large for VFAT. On a local network
this approach might well be faster than using a removable drive. Setting
up network transfers is not always easy, though; most computers are, by
default, configured in ways that do not allow random strangers to dump
large files on their drives. Getting around that obstacle is likely to be
too much even for moderately skillful users. Use of a third-party site to
transfer files is workable when the files are small; even if it is possible
for very large files, it's not something that will be tolerably fast on
Removable drives, instead, are easy, so the "sneakernet" approach to file
transfer is likely to stay with us for some time. Does that mean that we
need a new filesystem format to better support this use? Filesystem
developer Ted Ts'o thinks not:
I used to think that we would need an IP unencumbered file system,
given issues around TomTom and Microsoft, but these days, given how
quickly Linux has taken over the embedded and mobile landscape
for all but the most tiniest of devices, I don't think that's as
important of an issue, since we can just simply use a native linux
file system. In the time that it would take to get some other new
file system adopted across the industry, it's likely Linux will
have enough market share to perhaps compel the other OS vendors to
provide interoperability solutions.
That is an interesting thought: Linux is now strong and prevalent enough
that we can simply expect the industry to pick up our way of doing things.
That approach has not always worked out in the past, but things might truly
be different this time around. Increasingly, devices like music players,
handsets, and digital cameras run Linux internally; these gadgets already
are, to a first approximation, removable storage devices with a bit of
extra hardware. Other devices, such as televisions, also tend to run Linux
internally. Supporting a native Linux filesystem on these devices should be a relatively
easy thing to do. It would be faster (assuming the underlying storage
isn't severely optimized for VFAT only), more feature-rich, and lacking in
patent aggressors. There is very little, in other words, not to like.
Well, there would be a few small problems. There are still some pesky
users out there with non-Linux systems that might want to access the
filesystems on their devices. In many cases, the increasing use of the MTP
protocol could sidestep that question altogether; indeed, recent MTP-using
Android devices are likely using it to export an ext4 filesystem. There
would still be cases where users on these other platforms would want to
mount filesystems directly, though, especially on pure storage devices;
bringing proper implementations of Linux filesystems to those platforms is,
evidently, not as easy as one might think.
Filesystems like ext4 also were not designed with removable devices in
mind. They tend not to be all that robust against unexpected removal of
the device unless fairly aggressive flushing of data is used (in fairness,
VFAT filesystems are also easily corrupted that way). The file ownership
model used by Linux filesystems tends not to translate well to removable
devices, since one system's user IDs typically have no meaning elsewhere.
So something like the user and group mount
options patch may be required to make things work well. Most Linux
filesystems have not been designed around the very large pages and erase
blocks used on flash devices and, thus, do not perform as well as they
could; see this
article for lots of details. These are
issues that can be worked out, certainly, but they remain in need of
working out at this time.
There is one other complication: according to
Arnd Bergmann there is another filesystem waiting on the wings:
There will be patches very soon for a new file system from a major
flash vendor that I'm cooperating with. I haven't seen the patches
myself yet, but the design is similar to a prototype that was done
thesis I supervised
. I hope that the new implementation is
similarly simple to this design, and also able to provide optimum
performance on most flash media.
Needless to say, such an entry has the potential to stir things up a bit.
A filesystem designed with input from both "a major flash vendor" and a
developer like Arnd should work well indeed on small
removable devices and should be well integrated into Linux. This
manufacturer could also employ the "include a
windows driver in a small partition on the device" trick, making
interoperability with most Windows systems Just Work. Putting the
filesystem code into the Linux kernel would make support readily available
on mobile devices. This scheme might just succeed.
So what we may see is not Linux pushing one of its native filesystem
formats onto the world. Instead, the world might just adopt a new format
that just happened to be well supported in Linux first. That could be the
best of all worlds: we would have a way to interoperate on removable drives
that is free, scalable, and widely supported. Getting there may well be
worth the trouble of adding yet another filesystem type.
to post comments)