Examining exFAT
For years, the Linux community mostly ignored exFAT; it was a proprietary format overshadowed by an unpleasant patent cloud. A Linux driver existed, though, and was shipped as a proprietary module on various Android devices. In 2013, the code for this driver escaped into the wild and was posted to a GitHub repository. But that code was never actually released under a free license and the patent issues remained, so no serious effort to upstream it into the mainline kernel was ever made.
The situation stayed this way for some years. Even Microsoft's decision to join the Open Invention Network (OIN) in 2018 did not change the situation; exFAT, being outside the OIN Linux System Definition, was not covered by any new patent grants. Some people pointed this out at the time, but it didn't raise a lot of concern. Most people, it seemed, had simply forgotten about exFAT, which has a relatively limited deployment overall.
In July of this year, though, Valdis Klētnieks posted that he
had "beaten into shape
" the exFAT code and wondered how it
might be upstreamed. The ensuing discussion made it clear that the patent
issues were still a show-stopper for inclusion; that discussion also
included a couple
of pointed suggestions to the Microsoft employees on the list that perhaps
they could help to change that situation. By all appearances, that prod started
an internal discussion that ended with Microsoft agreeing with the
addition of exFAT to the mainline kernel.
One never really knows what is going on in large companies. The exclusion of exFAT from Microsoft's commitment to OIN looked like a deliberate, old-time Microsoftian act, but it now looks likely that opening up exFAT is just one of those things that nobody thought about until it was brought to their attention.
Greg Kroah-Hartman wasted no time in taking Klētnieks's code and proposing
it for addition to the staging tree for further work, of which it is
said to need a fair amount. That drew an equally quick objection
from Christoph Hellwig, who said it would be better to "just review
the damn thing and get it into the proper tree
". He is unhappy
about how filesystems have been handled in the staging tree in the past,
and mentioned the handling of the EROFS
filesystem as a particular sore point. That sparked a whole subthread
on the remaining concerns about EROFS that has little to do with exFAT.
The code quality of the exFAT implementation is of concern generally; that is the kind of thing that can be improved over time in the staging tree. But there are a couple of deeper issues that could yet prove to be a sticking point for exFAT. One is a complaint from Pali Rohár that the posted specification is incomplete. In particular, he said, the "TexFAT" extension is not documented. As Klētnieks pointed out, this extension seems to be used only by Windows CE, so it may prove to be a feature that the rest of the world can do without.
The bigger concern, perhaps, is that this filesystem module should not exist at all, so tweaking it will not help the situation. As Hellwig put it:
The right course, he said, is to just add the necessary support to the
kernel's existing VFAT filesystem. Kroah-Hartman replied that
he had tried to do that once "a few years ago
" and concluded
that it wouldn't work. But, he said, it may well be easier now that the
specification has been posted. If exFAT support were to be reimplemented
entirely, perhaps as part of the existing VFAT code, the staging version
could simply be deleted once it outlived its usefulness. It would not be
the first time such a thing had happened.
Hellwig is clearly not
convinced that things will play out that way, but his concerns may not be
enough to keep the exFAT code out of the staging tree. Whether
that step is taken or not, though, there is clearly some work to be done
before exFAT truly becomes a part of the mainline kernel. But, then, after
thirteen years out in the cold, there is probably no point in being in a
hurry to get full support upstream now.
| Index entries for this article | |
|---|---|
| Kernel | Filesystems/exFAT |
Posted Aug 30, 2019 19:05 UTC (Fri)
by shasta (subscriber, #79915)
[Link] (11 responses)
That "relatively limited deployment" includes all photographers/videographers working with SD cards bigger than 32GB...
Posted Aug 30, 2019 19:33 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
Posted Aug 30, 2019 20:06 UTC (Fri)
by Deleted user 129183 (guest, #129183)
[Link] (3 responses)
(the old Matthew Garrett’s post about four enterprise clustering filesystems is probably also relevant here)
Posted Sep 2, 2019 9:46 UTC (Mon)
by hcmh (subscriber, #103342)
[Link] (2 responses)
Posted Sep 2, 2019 11:36 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (1 responses)
https://lists.linuxfoundation.org/pipermail/ksummit-discu...
Posted Sep 4, 2019 0:10 UTC (Wed)
by Deleted user 129183 (guest, #129183)
[Link]
Exactly this one.
Posted Aug 30, 2019 20:22 UTC (Fri)
by pizza (subscriber, #46)
[Link] (3 responses)
Posted Sep 5, 2019 14:51 UTC (Thu)
by anselm (subscriber, #2796)
[Link]
The main reason why exFAT is interesting for digital cameras is not the fact that it allows larger file systems but that it allows larger files. This doesn't really matter if all you want to store are teensy 48-MB pictures, but if you're shooting video then the fact that the maximum supported file size on FAT32 is 4 GiB becomes a nuisance, especially now that 4K video (or even higher) is mainstream on consumer devices. ExFAT bumps the maximum file size up into the pebibyte range, which is likely to remain theoretical for a while.
Posted Sep 5, 2019 17:38 UTC (Thu)
by mattdm (subscriber, #18)
[Link] (1 responses)
Posted Sep 6, 2019 7:13 UTC (Fri)
by jem (subscriber, #24231)
[Link]
Posted Nov 28, 2019 2:05 UTC (Thu)
by WolfWings (subscriber, #56790)
[Link] (1 responses)
VFAT/FAT32 works fine up to 2TB in the spec (larger with larger-sector devices) and Windows will read/write those filesystems just fine, but will refuse to format anything >32GB in FAT32/VFAT.
Linux and Mac don't care, and neither do any of the camera vendors that pre-date or chose to skip exFAT support.
The only limitation was a single file couldn't be >=2GB, which is why a lot of software and hardware both segment video recording to "first keyframe after 1GB" to get roughly 1GB sized chunks.
Posted Nov 28, 2019 7:52 UTC (Thu)
by jem (subscriber, #24231)
[Link]
Posted Aug 30, 2019 19:06 UTC (Fri)
by clugstj (subscriber, #4020)
[Link] (2 responses)
Posted Aug 30, 2019 19:32 UTC (Fri)
by nedrichards (subscriber, #23295)
[Link]
Posted Aug 30, 2019 19:35 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
> Today we’re pleased to announce that Microsoft is supporting the addition of Microsoft’s exFAT technology to the Linux kernel.
is plain enough to me. They *are* OIN members.
Posted Aug 30, 2019 19:11 UTC (Fri)
by flussence (guest, #85566)
[Link] (26 responses)
They can contribute to Samsung's existing implementation if they're serious about Linux.
Posted Aug 30, 2019 20:21 UTC (Fri)
by Deleted user 129183 (guest, #129183)
[Link] (25 responses)
What they really _really_ mean: Microsoft ♥ that they can use Linux for their cloud computing service and WSL completely _for free_.
But they would 😡 if Linux-based systems became a real competition to their operating system on end-user devices. Something about OEM licences and such.
> They can contribute to Samsung's existing implementation
As it is pointed in the article, Samsung’s implementation violates copyright. Therefore it isn’t a real solution.
Posted Aug 30, 2019 20:58 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (23 responses)
As others said in this thread, it won't be surprising if Windows 11 moves to Linux kernel instead of the old creaky NT kernel.
Posted Aug 30, 2019 21:29 UTC (Fri)
by Deleted user 129183 (guest, #129183)
[Link] (16 responses)
I find it completely unlikely. They would have nothing to gain by it, and they would need to rewrite probably over half of their operating system, and we all know it is a recipe for a disaster.
Posted Aug 30, 2019 22:09 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (15 responses)
Switching to Linux would allow them to stop investing in improvements in these areas.
> and they would need to rewrite probably over half of their operating system, and we all know it is a recipe for a disaster.
Windows NT kernel design even makes it much easier to conversion in userspace shims. For example, IOCTLs are usually self-contained packets with a length field, not custom structures passed as "void *" into the kernel space.
Posted Aug 30, 2019 22:50 UTC (Fri)
by scientes (guest, #83068)
[Link] (2 responses)
It doesn't help that they try to segment the market by limiting the number of half-open TCP connections:
Posted Aug 31, 2019 18:11 UTC (Sat)
by Sesse (subscriber, #53779)
[Link]
Posted Nov 7, 2019 17:58 UTC (Thu)
by Spudd86 (subscriber, #51683)
[Link]
Posted Aug 31, 2019 6:09 UTC (Sat)
by epa (subscriber, #39769)
[Link] (10 responses)
It is much easier to get Linux working on every desktop (with no particular requirement to run Windows binaries or even binaries from old Linux systems) than to get Linux+Win32/64 layer on every desktop with full backward compatibilty. Yet even the former goal is proverbially unreachable.
Posted Aug 31, 2019 10:21 UTC (Sat)
by smurf (subscriber, #17840)
[Link]
Doing this should easily be within the capability of Microsoft. They have enough people they could throw at a problem like this, and long term they'd probably need fewer coders than they do now.
Posted Aug 31, 2019 16:45 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (7 responses)
> It is much easier to get Linux working on every desktop
Posted Sep 1, 2019 6:15 UTC (Sun)
by epa (subscriber, #39769)
[Link] (2 responses)
Posted Sep 1, 2019 6:56 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
And for installers Windows already uses quite a few hacks.
Some parts of Windows will still be tricky - the graphics stack there is top-notch and is well-integrated with such subsystems as font rendering.
Posted Sep 2, 2019 7:12 UTC (Mon)
by cpitrat (subscriber, #116459)
[Link]
Exactly, just look on GOG the number of games that do not run on modern windows (typically 8+). Interestingly, many of them run fine on wine!
Posted Sep 2, 2019 8:48 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link] (2 responses)
And let's not even mention that it is easier to partition a disk from linux than from the windows installer.
What you normally experience with windows is a windows distribution created by the computer maker for that machine. Not at all stock windows.
Posted Sep 2, 2019 18:42 UTC (Mon)
by flussence (guest, #85566)
[Link]
That was 17 years ago, sure, but every time I've had to babysit an installer for someone else in the past decade it's been a similarly miserable experience. Windows 8/10 manages to make it even worse because they can't multitask while doing OS updates. Microsoft really doesn't want people to use their OS.
Posted Sep 2, 2019 18:57 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Windows NT had a purely text-based installer and it needed a floppy disk with drivers if you had a RAID controller, but otherwise it just used BIOS-based IO to install the initial system. And the initial system had support for IDE/SCSI/ATA drivers.
Consumer versions of Windows had a GUI installer, working in 640x480 16 color VGA mode. I don't remember ever needing drivers for the initial installation on consumer hardware. Windows also happily used VESA BIOS drivers for the GUI.
Both consumer and server versions had network drivers (and back then most network cards were NE2000-compatible). Sound indeed was bad, with only SoundBlaster working out of box on NT.
Posted Sep 20, 2019 14:03 UTC (Fri)
by epa (subscriber, #39769)
[Link]
Posted Mar 28, 2020 4:21 UTC (Sat)
by yuhong (guest, #57183)
[Link]
Posted Sep 2, 2019 7:08 UTC (Mon)
by cpitrat (subscriber, #116459)
[Link]
If they were investing significantly in it, it wouldn't suck that much.
Posted Aug 31, 2019 18:09 UTC (Sat)
by alison (subscriber, #63752)
[Link] (5 responses)
I'm more concerned that Android will move to the Zircon kernel and pull a lot of corporate developers along with it. That could, in the long run, really be a blow to Linux.
Posted Aug 31, 2019 18:18 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Posted Aug 31, 2019 18:34 UTC (Sat)
by alison (subscriber, #63752)
[Link] (2 responses)
Posted Aug 31, 2019 18:40 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Aug 31, 2019 18:47 UTC (Sat)
by alison (subscriber, #63752)
[Link]
It feels to me as a mere user like the advent of LLVM/Clang has made GCC much better.
Posted Sep 1, 2019 8:40 UTC (Sun)
by smurf (subscriber, #17840)
[Link]
Posted Aug 31, 2019 9:45 UTC (Sat)
by ceplm (subscriber, #41334)
[Link]
Posted Aug 31, 2019 7:25 UTC (Sat)
by zdzichu (subscriber, #17118)
[Link]
Posted Aug 31, 2019 12:34 UTC (Sat)
by grawity (subscriber, #80596)
[Link] (1 responses)
Posted Sep 2, 2019 15:11 UTC (Mon)
by Conan_Kudo (subscriber, #103240)
[Link]
Posted Oct 1, 2019 23:17 UTC (Tue)
by frispete (subscriber, #89956)
[Link]
I remember a couple of occasions, where I needed to fire up some Virtualbox Windows VM just to get some vfat/exfat issues fixed on damaged SD cards. Interestingly, even Android on Samsung devices failed in this regard.
The gap, that needs to be closed, is bigger, than it appears at first.
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
How large are your cards? It's my understanding that this is the default up to 32GB. Even if you don't need (or want) more storage than that on a single card, it's getting increasingly difficult to buy smaller sizes in the top-quality "pro" SD card lines. So, really, this isn't a "eh, it's been 13 years, what's a few more" situation. It's quite timely!
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
What they really mean: “Microsoft has temporarily stopped beating their spouse in public. Smile for the cameras, or else.”
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Well, the NT kernel sucks in many ways. Its filesystem layer is slow, the scheduler is way behind Linux and the network stack is so ancient that it's not even funny.
If Microsoft decides to drop the kernel-level compatibility then it's actually surprisingly realistic. Especially if Microsoft decides to port their shell to Linux, then they'd just need to implement parts of the kernel API that is exposed for the Win32 API DLLs (user32.dll, gdi32.dll, ...).
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Merging Windows and Linux
Examining exFAT
Porting the Windows userspace at a lower level than Wine would fix a lot of issues, actually. It's just not feasible for Wine.
Well, "classic" Linux has been trying this for the last 20 years or so. With pretty much zero success to show for it.
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Examining exFAT
Mostly ignored?
Fortunately, ten years ago fuse-exfat landed. Everyone installed it and moved on.
I only skimmed through the spec, but I got the impression that exFAT had more differences from FAT32 than it had similarities... Does it actually make sense for both to be part of the same driver?
Examining exFAT
It doesn't seem like it should. At least one other person tried unifying the drivers (I don't remember who, though) before and it was incredibly ugly...
Examining exFAT
Examining exFAT
