|
|
Subscribe / Log in / New account

Examining exFAT

By Jonathan Corbet
August 30, 2019
Linux kernel developers like to get support for new features — such as filesystem types — merged quickly. In the case of the exFAT filesystem, that didn't happen; exFAT was created by Microsoft in 2006 for use in larger flash-storage cards, but there has never been support in the kernel for this filesystem. Microsoft's recent announcement that it wanted to get exFAT support into the mainline kernel would appear to have removed the largest obstacle to Linux exFAT support. But, as is so often the case, it seems that some challenges remain.

For years, the Linux community mostly ignored exFAT; it was a proprietary format overshadowed by an unpleasant patent cloud. A Linux driver existed, though, and was shipped as a proprietary module on various Android devices. In 2013, the code for this driver escaped into the wild and was posted to a GitHub repository. But that code was never actually released under a free license and the patent issues remained, so no serious effort to upstream it into the mainline kernel was ever made.

The situation stayed this way for some years. Even Microsoft's decision to join the Open Invention Network (OIN) in 2018 did not change the situation; exFAT, being outside the OIN Linux System Definition, was not covered by any new patent grants. Some people pointed this out at the time, but it didn't raise a lot of concern. Most people, it seemed, had simply forgotten about exFAT, which has a relatively limited deployment overall.

In July of this year, though, Valdis Klētnieks posted that he had "beaten into shape" the exFAT code and wondered how it might be upstreamed. The ensuing discussion made it clear that the patent issues were still a show-stopper for inclusion; that discussion also included a couple of pointed suggestions to the Microsoft employees on the list that perhaps they could help to change that situation. By all appearances, that prod started an internal discussion that ended with Microsoft agreeing with the addition of exFAT to the mainline kernel.

One never really knows what is going on in large companies. The exclusion of exFAT from Microsoft's commitment to OIN looked like a deliberate, old-time Microsoftian act, but it now looks likely that opening up exFAT is just one of those things that nobody thought about until it was brought to their attention.

Greg Kroah-Hartman wasted no time in taking Klētnieks's code and proposing it for addition to the staging tree for further work, of which it is said to need a fair amount. That drew an equally quick objection from Christoph Hellwig, who said it would be better to "just review the damn thing and get it into the proper tree". He is unhappy about how filesystems have been handled in the staging tree in the past, and mentioned the handling of the EROFS filesystem as a particular sore point. That sparked a whole subthread on the remaining concerns about EROFS that has little to do with exFAT.

The code quality of the exFAT implementation is of concern generally; that is the kind of thing that can be improved over time in the staging tree. But there are a couple of deeper issues that could yet prove to be a sticking point for exFAT. One is a complaint from Pali Rohár that the posted specification is incomplete. In particular, he said, the "TexFAT" extension is not documented. As Klētnieks pointed out, this extension seems to be used only by Windows CE, so it may prove to be a feature that the rest of the world can do without.

The bigger concern, perhaps, is that this filesystem module should not exist at all, so tweaking it will not help the situation. As Hellwig put it:

It basically is a reimplementation of fs/fat/ not up to kernel standards with a few indirections thrown in to also support exfat. So no amount of work on this codebase is really going to bring us forward.

The right course, he said, is to just add the necessary support to the kernel's existing VFAT filesystem. Kroah-Hartman replied that he had tried to do that once "a few years ago" and concluded that it wouldn't work. But, he said, it may well be easier now that the specification has been posted. If exFAT support were to be reimplemented entirely, perhaps as part of the existing VFAT code, the staging version could simply be deleted once it outlived its usefulness. It would not be the first time such a thing had happened.

Hellwig is clearly not convinced that things will play out that way, but his concerns may not be enough to keep the exFAT code out of the staging tree. Whether that step is taken or not, though, there is clearly some work to be done before exFAT truly becomes a part of the mainline kernel. But, then, after thirteen years out in the cold, there is probably no point in being in a hurry to get full support upstream now.

Index entries for this article
KernelFilesystems/exFAT


to post comments

Examining exFAT

Posted Aug 30, 2019 19:05 UTC (Fri) by shasta (subscriber, #79915) [Link] (11 responses)

> Most people, it seemed, had simply forgotten about exFAT, which has a relatively limited deployment overall.

That "relatively limited deployment" includes all photographers/videographers working with SD cards bigger than 32GB...

Examining exFAT

Posted Aug 30, 2019 19:33 UTC (Fri) by smurf (subscriber, #17840) [Link]

Well, not "all". Many [can be convinced to] use plain VFAT on larger cards.

Examining exFAT

Posted Aug 30, 2019 20:06 UTC (Fri) by Deleted user 129183 (guest, #129183) [Link] (3 responses)

Yes, but who does _that_? (per <https://xkcd.com/619/>)

(the old Matthew Garrett’s post about four enterprise clustering filesystems is probably also relevant here)

Examining exFAT

Posted Sep 2, 2019 9:46 UTC (Mon) by hcmh (subscriber, #103342) [Link] (2 responses)

I don't know that Matthew Garrett Post, could you (or someone) provide a link or something to search for to find it?

Examining exFAT

Posted Sep 2, 2019 11:36 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

Examining exFAT

Posted Sep 4, 2019 0:10 UTC (Wed) by Deleted user 129183 (guest, #129183) [Link]

> Probably this one.

Exactly this one.

Examining exFAT

Posted Aug 30, 2019 20:22 UTC (Fri) by pizza (subscriber, #46) [Link] (3 responses)

My DSLR (with its 48MB-odd raw files) always formats its CF and SD cards with VFAT. I doubt it's unusual.

Examining exFAT

Posted Sep 5, 2019 14:51 UTC (Thu) by anselm (subscriber, #2796) [Link]

The main reason why exFAT is interesting for digital cameras is not the fact that it allows larger file systems but that it allows larger files. This doesn't really matter if all you want to store are teensy 48-MB pictures, but if you're shooting video then the fact that the maximum supported file size on FAT32 is 4 GiB becomes a nuisance, especially now that 4K video (or even higher) is mainstream on consumer devices. ExFAT bumps the maximum file size up into the pebibyte range, which is likely to remain theoretical for a while.

Examining exFAT

Posted Sep 5, 2019 17:38 UTC (Thu) by mattdm (subscriber, #18) [Link] (1 responses)

How large are your cards? It's my understanding that this is the default up to 32GB. Even if you don't need (or want) more storage than that on a single card, it's getting increasingly difficult to buy smaller sizes in the top-quality "pro" SD card lines. So, really, this isn't a "eh, it's been 13 years, what's a few more" situation. It's quite timely!

Examining exFAT

Posted Sep 6, 2019 7:13 UTC (Fri) by jem (subscriber, #24231) [Link]

VFAT may be the default on SD cards up to 32 GB, but it is possible to reformat bigger cards to use VFAT instead of exFAT. "But who does that?" Your camera does. If you put an exFAT card into a camera that does not support exFAT, it tells you the card has to be formatted before it can be used, and offers to do so.

Examining exFAT

Posted Nov 28, 2019 2:05 UTC (Thu) by WolfWings (subscriber, #56790) [Link] (1 responses)

Only if you were trying to format them under Windows.

VFAT/FAT32 works fine up to 2TB in the spec (larger with larger-sector devices) and Windows will read/write those filesystems just fine, but will refuse to format anything >32GB in FAT32/VFAT.

Linux and Mac don't care, and neither do any of the camera vendors that pre-date or chose to skip exFAT support.

The only limitation was a single file couldn't be >=2GB, which is why a lot of software and hardware both segment video recording to "first keyframe after 1GB" to get roughly 1GB sized chunks.

Examining exFAT

Posted Nov 28, 2019 7:52 UTC (Thu) by jem (subscriber, #24231) [Link]

The thing is, large memory cards come pre-formatted to exFAT. People expect them to just work, and don't want to be bothered with the hassle of formatting them. I guess a large proportion of users don't even know what "formatting a memory card" means.

Examining exFAT

Posted Aug 30, 2019 19:06 UTC (Fri) by clugstj (subscriber, #4020) [Link] (2 responses)

Where does Microsoft actually say that it won't use the patent system to attack Linux exFAT users?

Examining exFAT

Posted Aug 30, 2019 19:32 UTC (Fri) by nedrichards (subscriber, #23295) [Link]

If it's in the OIN Linux System definition they can't (they're OIN licencees).

Examining exFAT

Posted Aug 30, 2019 19:35 UTC (Fri) by smurf (subscriber, #17840) [Link]

This quote

> Today we’re pleased to announce that Microsoft is supporting the addition of Microsoft’s exFAT technology to the Linux kernel.

is plain enough to me. They *are* OIN members.

Examining exFAT

Posted Aug 30, 2019 19:11 UTC (Fri) by flussence (guest, #85566) [Link] (26 responses)

What their ad campaign says: “Microsoft ♥ Linux”
What they really mean: “Microsoft has temporarily stopped beating their spouse in public. Smile for the cameras, or else.”

They can contribute to Samsung's existing implementation if they're serious about Linux.

Examining exFAT

Posted Aug 30, 2019 20:21 UTC (Fri) by Deleted user 129183 (guest, #129183) [Link] (25 responses)

> What they really mean

What they really _really_ mean: Microsoft ♥ that they can use Linux for their cloud computing service and WSL completely _for free_.

But they would 😡 if Linux-based systems became a real competition to their operating system on end-user devices. Something about OEM licences and such.

> They can contribute to Samsung's existing implementation

As it is pointed in the article, Samsung’s implementation violates copyright. Therefore it isn’t a real solution.

Examining exFAT

Posted Aug 30, 2019 20:58 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (23 responses)

Given the sad state of Linux on classic desktop, Microsoft doesn't care at all. And they are still selling online Office subscription for Androids just fine.

As others said in this thread, it won't be surprising if Windows 11 moves to Linux kernel instead of the old creaky NT kernel.

Examining exFAT

Posted Aug 30, 2019 21:29 UTC (Fri) by Deleted user 129183 (guest, #129183) [Link] (16 responses)

> it won't be surprising if Windows 11 moves to Linux kernel instead of the old creaky NT kernel

I find it completely unlikely. They would have nothing to gain by it, and they would need to rewrite probably over half of their operating system, and we all know it is a recipe for a disaster.

Examining exFAT

Posted Aug 30, 2019 22:09 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (15 responses)

> They would have nothing to gain by it
Well, the NT kernel sucks in many ways. Its filesystem layer is slow, the scheduler is way behind Linux and the network stack is so ancient that it's not even funny.

Switching to Linux would allow them to stop investing in improvements in these areas.

> and they would need to rewrite probably over half of their operating system, and we all know it is a recipe for a disaster.
If Microsoft decides to drop the kernel-level compatibility then it's actually surprisingly realistic. Especially if Microsoft decides to port their shell to Linux, then they'd just need to implement parts of the kernel API that is exposed for the Win32 API DLLs (user32.dll, gdi32.dll, ...).

Windows NT kernel design even makes it much easier to conversion in userspace shims. For example, IOCTLs are usually self-contained packets with a length field, not custom structures passed as "void *" into the kernel space.

Examining exFAT

Posted Aug 30, 2019 22:50 UTC (Fri) by scientes (guest, #83068) [Link] (2 responses)

> the network stack is so ancient that it's not even funny.

It doesn't help that they try to segment the market by limiting the number of half-open TCP connections:

https://www.fsf.org/bulletin/2007/fall/antifeatures/

Examining exFAT

Posted Aug 31, 2019 18:11 UTC (Sat) by Sesse (subscriber, #53779) [Link]

That was 1996…

Examining exFAT

Posted Nov 7, 2019 17:58 UTC (Thu) by Spudd86 (subscriber, #51683) [Link]

They've stopped that, YEARS ago. IIRC XP was the last time this was a thing.

Examining exFAT

Posted Aug 31, 2019 6:09 UTC (Sat) by epa (subscriber, #39769) [Link] (10 responses)

The Wine project has been doing this for a quarter of a century and, while there has been impressive progress, it’s a long way off the level of compatibility needed to be branded as ‘Windows’. Microsoft has more resources but adding more people does not magically fix the issues Wine faces. Not to mention device drivers and 101 annoying fiddly things that will depend on the NT kernel or the quirks of its interfaces to user space.

It is much easier to get Linux working on every desktop (with no particular requirement to run Windows binaries or even binaries from old Linux systems) than to get Linux+Win32/64 layer on every desktop with full backward compatibilty. Yet even the former goal is proverbially unreachable.

Merging Windows and Linux

Posted Aug 31, 2019 10:21 UTC (Sat) by smurf (subscriber, #17840) [Link]

Well, the easiest way forward is probably a Windows runtime that uses Wayland as its graphics back-end. Run the rest of the Windows kernel in a KVM sandbox, attach the Linux file system to a random drive letter. Add appropriate features to a bunch of management programs. Done (step one, that is; step two is to phase out a heap of legacy code in favor of native Linux implementations).

Doing this should easily be within the capability of Microsoft. They have enough people they could throw at a problem like this, and long term they'd probably need fewer coders than they do now.

Examining exFAT

Posted Aug 31, 2019 16:45 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

> but adding more people does not magically fix the issues Wine faces
Porting the Windows userspace at a lower level than Wine would fix a lot of issues, actually. It's just not feasible for Wine.

> It is much easier to get Linux working on every desktop
Well, "classic" Linux has been trying this for the last 20 years or so. With pretty much zero success to show for it.

Examining exFAT

Posted Sep 1, 2019 6:15 UTC (Sun) by epa (subscriber, #39769) [Link] (2 responses)

That’s my point — in 20 years even Linux on the desktop has been hard to achieve; how much harder if you require 100% Windows compatibility (including device drivers and every crappy installer) as well. It’s just not feasible in the real world.

Examining exFAT

Posted Sep 1, 2019 6:56 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Windows is not 100% compatible with itself anymore. The "backwards compatibility" faction in MS basically lost the good fight. Reasonable compatibility is doable, though. Even for many drivers that use the UMDF.

And for installers Windows already uses quite a few hacks.

Some parts of Windows will still be tricky - the graphics stack there is top-notch and is well-integrated with such subsystems as font rendering.

Examining exFAT

Posted Sep 2, 2019 7:12 UTC (Mon) by cpitrat (subscriber, #116459) [Link]

> Windows is not 100% compatible with itself anymore.

Exactly, just look on GOG the number of games that do not run on modern windows (typically 8+). Interestingly, many of them run fine on wine!

Examining exFAT

Posted Sep 2, 2019 8:48 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (2 responses)

Have you even ever tried to install windows? Up until windows 7 it might not have the drivers to use the hard disk, and it never had network drivers and the screen would be 600x400.

And let's not even mention that it is easier to partition a disk from linux than from the windows installer.

What you normally experience with windows is a windows distribution created by the computer maker for that machine. Not at all stock windows.

Examining exFAT

Posted Sep 2, 2019 18:42 UTC (Mon) by flussence (guest, #85566) [Link]

I remember the last time I (ever) installed windows myself, sitting and waiting all day for the updater to pull down a 35MB sound driver and 90MB video driver before the machine was usable.

That was 17 years ago, sure, but every time I've had to babysit an installer for someone else in the past decade it's been a similarly miserable experience. Windows 8/10 manages to make it even worse because they can't multitask while doing OS updates. Microsoft really doesn't want people to use their OS.

Examining exFAT

Posted Sep 2, 2019 18:57 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Yes, many times and since the early Windows NT 4.0 era.

Windows NT had a purely text-based installer and it needed a floppy disk with drivers if you had a RAID controller, but otherwise it just used BIOS-based IO to install the initial system. And the initial system had support for IDE/SCSI/ATA drivers.

Consumer versions of Windows had a GUI installer, working in 640x480 16 color VGA mode. I don't remember ever needing drivers for the initial installation on consumer hardware. Windows also happily used VESA BIOS drivers for the GUI.

Both consumer and server versions had network drivers (and back then most network cards were NE2000-compatible). Sound indeed was bad, with only SoundBlaster working out of box on NT.

Examining exFAT

Posted Sep 20, 2019 14:03 UTC (Fri) by epa (subscriber, #39769) [Link]

Ah I understand, you didn't mean provide a Win32 API on top of Linux, you meant provide a Windows NT kernel API and run the existing Windows subsystems on top of that.

Examining exFAT

Posted Mar 28, 2020 4:21 UTC (Sat) by yuhong (guest, #57183) [Link]

A lot of the problems can be solved by using a lot of the Windows source code directly, like MSHTML, .NET Framework, Jet 4.0, MSVCRT, etc...

Examining exFAT

Posted Sep 2, 2019 7:08 UTC (Mon) by cpitrat (subscriber, #116459) [Link]

> Switching to Linux would allow them to stop investing in improvements in these areas.

If they were investing significantly in it, it wouldn't suck that much.

Examining exFAT

Posted Aug 31, 2019 18:09 UTC (Sat) by alison (subscriber, #63752) [Link] (5 responses)

> it won't be surprising if Windows 11 moves to Linux kernel instead of the old creaky NT kernel.

I'm more concerned that Android will move to the Zircon kernel and pull a lot of corporate developers along with it. That could, in the long run, really be a blow to Linux.

Examining exFAT

Posted Aug 31, 2019 18:18 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Zircon won't be able to match Linux's performance for a long time. On the other hand, it's likely to be way more secure. Hm... It's actually a tough choice.

Examining exFAT

Posted Aug 31, 2019 18:34 UTC (Sat) by alison (subscriber, #63752) [Link] (2 responses)

I suspect that the choice is not tough if you prefer permissive licenses. The widespread adoption of Zircon by other will, over time, truly weaken Linux, I fear.

Examining exFAT

Posted Aug 31, 2019 18:40 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Well, Linux does need a kick in the butt in many areas. So I don't see anything wrong with a real Open Source competitor. And it's not like Linux's GPL enforcement has any teeth either.

Examining exFAT

Posted Aug 31, 2019 18:47 UTC (Sat) by alison (subscriber, #63752) [Link]

>I don't see anything wrong with a real Open Source competitor.

It feels to me as a mere user like the advent of LLVM/Clang has made GCC much better.

Examining exFAT

Posted Sep 1, 2019 8:40 UTC (Sun) by smurf (subscriber, #17840) [Link]

Well, that's one side. The other is that Zircon is supposed to have a stable ABI. Thus we can (in principle, assuming somebody is motivated to do the work, which I assume will happen once there are otherwise-unsupportable Zircon phones out there) support the kernel side of the thing in Linux and bingo every Zircon driver out there just works.

Examining exFAT

Posted Aug 31, 2019 9:45 UTC (Sat) by ceplm (subscriber, #41334) [Link]

What that _really_ means is that Microsoft gave up on Windows staying as a recognized *server* operating system and they trying not to stay behind the curve. Almost (or over?) half of the deployments on their own cloud platform is on Linux. I believe, they just gave up and trying to follow the crowd.

Mostly ignored?

Posted Aug 31, 2019 7:25 UTC (Sat) by zdzichu (subscriber, #17118) [Link]

I don't agree with "For years, the Linux community mostly ignored exFAT". People with digital cameras needed support badly.
Fortunately, ten years ago fuse-exfat landed. Everyone installed it and moved on.

Examining exFAT

Posted Aug 31, 2019 12:34 UTC (Sat) by grawity (subscriber, #80596) [Link] (1 responses)

I only skimmed through the spec, but I got the impression that exFAT had more differences from FAT32 than it had similarities... Does it actually make sense for both to be part of the same driver?

Examining exFAT

Posted Sep 2, 2019 15:11 UTC (Mon) by Conan_Kudo (subscriber, #103240) [Link]

It doesn't seem like it should. At least one other person tried unifying the drivers (I don't remember who, though) before and it was incredibly ugly...

Examining exFAT

Posted Oct 1, 2019 23:17 UTC (Tue) by frispete (subscriber, #89956) [Link]

While it is good to finally see some movement in this area, it's still a long way to get serious support for low end windows filesystems on a Linux desktop, because fsck and friends are lacking as well (to put it politely).

I remember a couple of occasions, where I needed to fire up some Virtualbox Windows VM just to get some vfat/exfat issues fixed on damaged SD cards. Interestingly, even Android on Samsung devices failed in this regard.

The gap, that needs to be closed, is bigger, than it appears at first.


Copyright © 2019, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds