LWN: Comments on "XFS: There and back ... and there again?"

XFS: There and back ... and there again?

nye — Wed, 06 May 2015 11:03:27 +0000

I think you've confused this for a thread in which somebody is looking for platitudes on writing crash-resistant software.

XFS: There and back ... and there again?

dlang — Tue, 05 May 2015 20:41:05 +0000

data can sit in the OS cache for quite a long time, depending on how the file is written to. Ext3 tried to flush it out fairly quickly, but even it could run into serious problems doing so

The bottom line is that if you want to be sure that data is on disk, you need to do a fsync.

Yes, if the system is idle, it will try to proactively push data out so that it can more easily throw the pages away if needed, but if you rely on that for critical stuff, you are going to run into problems someday. It doesn't matter what filesystem you use.

XFS: There and back ... and there again?

nye — Tue, 05 May 2015 12:53:26 +0000

That's a different bug, unless the delayed cache flush problem can cover delays of many hours: I was specifically wondering about files *not being written to at the time*. Is it possible that it could still be down to that?

Basically, once upon a time I found a fair number of files wiped out after a power failure, and the only reason I noticed straight away was because one of them was /etc/passwd. It's *possible* that I'd [un]installed something earlier in the day that might have had a need to add/remove/alter a user account, but there was definitely nothing like that going on at the time.

What I learned when attempting to understand this at the time was that XFS was specifically intended to work with battery-backed RAID arrays, and not as a general-purpose filesystem, so I chalked it up to a bad choice for a desktop machine and moved on. Nevertheless, this is one of the two biggest data loss events I've ever experienced (the other being a botched reiserfs resize) so it's stuck with me even though it was long ago now.

XFS: There and back ... and there again?

dgc — Mon, 04 May 2015 22:55:40 +0000

http://sandeen.net/wordpress/computers/xfs-does-not-null-...

XFS reflink status?

dgc — Mon, 04 May 2015 22:52:50 +0000

Being actively worked on. Won't be ready for a while - maybe 4.3 or 4.4?

-dave.

XFS: There and back ... and there again?

nye — Tue, 21 Apr 2015 13:16:50 +0000

>..., and the zeroing of all open files when there is an unclean shutdown. None of those were true, but they have become part of the XFS lore.

Does anyone know what the real reason was for files that were not being written to at the time to be zero'd on unclean shutdown, and/or what circumstances would trigger it?

I guess it's water under the bridge now, but it would be nice to know nevertheless.

XFS reflink status?

gmatht — Sat, 18 Apr 2015 05:31:43 +0000

What is the status of XFS reflink? I have seen work on reflink early 2013, e.g.:
http://oss.sgi.com/archives/xfs/2013-01/msg00135.html
but have not found any patches on LKML. Has it been merged yet?

XFS: Why

Wol — Sun, 12 Apr 2015 13:47:14 +0000

What's the quote? "Never underestimate the bandwidth of a van full of tapes on the motorway"?

Cheers,
Wol

XFS: Why

marcH — Fri, 10 Apr 2015 22:38:29 +0000

Isn't USB2 supposed to be as fast as all but the fastest (and crazy expensive) UHS cards? Not even mentioning USB3.

From that respect you should be able to ignore the removable and cross-compatibility aspects of the SDXC card in your camera?

Windows users better do this anyway for other reasons: https://en.wikipedia.org/wiki/Windows_Photo_Viewer#Issues

XFS: Why

nerdshark — Fri, 10 Apr 2015 17:35:27 +0000

Actually, the two biggest ext* drivers on Windows are relatively up-to-date. The ext2fsd driver doesn't come signed, but I'm pretty sure the Paragon ExtFS driver *is*.

https://www.paragon-software.com/home/extfs-windows/
http://www.ext2fsd.com/

XFS: Why

dlang — Fri, 10 Apr 2015 12:26:51 +0000

I have a 128G card in my camera, and at a recent work outing I set it up on automatic and it shot over 700 pictures in a few hours. When I shoot manually, it's not uncommon for me to shoot 300+ pictures at an event. The camera's "shots left" counter only goes to 2000 so I'm not sure exactly how many shots the card should hold, but it's in the 3-4K range

it takes a long time to download this data from the SD card when directly read from a computer, and a long time to send this much data over a wired Gig-E network. I sure wouldn't want to depend on having a cell or wifi network available that could handle me (and others around me) shooting at these sorts of volumes.

XFS: Why

pizza — Fri, 10 Apr 2015 12:15:56 +0000

> fwiw, my current camera currently has 48 *G*B of SD-card capacity, and that is enough to store 600 photos

Yeah, 48MB per (losslessly compressed 14-bit RAW) image really adds up, doesn't it?

XFS: Why

Wol — Fri, 10 Apr 2015 08:13:16 +0000

> With the rise of The Cloud, NAS appliances and all other networked things SDXC will very slowly die like every other removable media,

And then you take a device (like a camera) somewhere where there is no network capability and you're stuffed. This blind faith in the ever-present external support network gets boring. That's how you get disasters - a failure somewhere else knocks your system out and you end up running around like a headless chicken because you have NO BACKUP.

fwiw, my current camera currently has 48 *G*B of SD-card capacity, and that is enough to store 600 photos - yes, six *hundred*. wtf am I supposed to do if I go on the holiday of a lifetime, with minimal access to civilisation (yes I'm that sort of a person), and go mad with the camera?

Cheers,
Wol

XFS: Why

yuhong — Fri, 10 Apr 2015 05:24:25 +0000

To be honest, the 32GB formatting limit for FAT32 dates back to Win2000. But yes exFAT is on my poorly written wishlist on my blog.

XFS: Why

marcH — Tue, 07 Apr 2015 18:08:11 +0000

If the SD Card Association or JEDEC or anything similar started actually caring about cross-compatibility and selling products, they could settle on some variant of UDF. At which point Microsoft would not be able to pretend there are some technical issues; their frivolous patent game would then be clear to even the least technical consumers.

SDXC right now looks a little bit like a repeat of *data* Minidisc in the 90s. Minidiscs were technically good, cheap and had a really good window of opportunity for data before flash memory became cheap and ubiquitous. For data they never got off the ground because of artificial crippling and became completely irrelevant once flash was there.

With the rise of The Cloud, NAS appliances and all other networked things SDXC will very slowly die like every other removable media, leaving the minority of users who really need removable media forever in this painful cross-compatibility situation.

> Unfortunately, to me this doesn't seem realistic at the moment. I'd be happy if someone could prove me wrong.

Different analysis and reasons - same pessimistic feelings and conclusion :-)

XFS: There and back ... and there again?

Paf — Tue, 07 Apr 2015 16:17:46 +0000

Is there perhaps video of Dave's talk? (I'm guessing not since no link was provided, but I thought I'd ask anyway. :) )

Thank you, Jake, for a fantastic article (and Dave for a great talk).

XFS: Why

jem — Tue, 07 Apr 2015 12:39:56 +0000

No doubt do all systems read DVD discs. But we're talking about portable r/w storage on removable media like USB hard drives, USB flash drives and SD cards. Things like block sizes, partitions or raw disk, etc. come into play. As you see on the pages I linked to earlier, there *are* problems, and if UDF formatted media does not work as well as a flash disk out of the box, people are not going to use it.

If there are problems with Windows interoperability, I doubt asking for help from Microsoft is going to help. I bet the answer you'll get is: "use exFAT".

Don't get me wrong, I would love to use UDF as a universal file system that is interoperable between different operating systems. Unfortunately, to me this doesn't seem realistic at the moment. I'd be happy if someone could prove me wrong.

XFS: Why

marcH — Tue, 07 Apr 2015 07:38:03 +0000

> There is not much the FOSS community can do if Microsoft or Apple aren't interested. Microsoft's "solution" to the interoperability problem is exFAT. [...] They know the licensing requirements are not compatible with the Linux world, and that's just how they like things to be.

UDF has a major difference: it's not going to be possible to make all these DVDs and other disks suddenly unreadable or patented.

XFS: Why

jem — Tue, 07 Apr 2015 07:29:34 +0000

Of course the issues are solvable. If there's a will there is a way. But this requires all parties to work towards this goal. There is not much the FOSS community can do if Microsoft or Apple aren't interested. Microsoft's "solution" to the interoperability problem is exFAT. The true innovation is not the file system itself, but the way Microsoft has managed to make its patented system an industry standard. They know the licensing requirements are not compatible with the Linux world, and that's just how they like things to be.

Microsoft has even managed to make the exFAT file system "mandatory" on SDXC cards. There is no technical reason an SDXC card can't be formatted using FAT32. In fact, a FAT32 volume can easily span 2 TiB, which is the upper limit of SDXC cards. Of course, although perfectly valid, there's no guarantee devices support e.g. a 64 GB FAT formatted SD card, because they are not "supposed" to.

XFS: Why

marcH — Mon, 06 Apr 2015 17:07:46 +0000

There are some practical and technical issues but none seems unsolvable - unlike exFAT patents (I've used UDF successfully).

I think the main reasons these issues haven't been solved yet:
- lack of interest in true cross-platform compatibility
- lack of interest in removable media. Everything more conveniently stored in the NSA-monitored cloud.

So we'll probably never have a very good solution here.

XFS: Why

jem — Mon, 06 Apr 2015 16:57:34 +0000

> or UDF.

UDF is nice, but has its own problems. Depending on the type of removable media, either the whole disk should be used for the file system, or the disk should be partitioned. This blog post from 2010 explains a hack making the disk appear both partitioned and unpartitioned at the same time: http://sipa.ulyssis.org/2010/02/filesystems-for-portable-...

But there are more dark clouds on the horizon; see this page: http://askubuntu.com/questions/27936/can-and-should-udf-b...

XFS: Why

flussence — Sun, 05 Apr 2015 21:19:25 +0000

+1 to this, UDF is a very nice filesystem for anything not physically screwed down. More or less equivalent to FAT32 in functionality, but without the legal muddy water.

The hardest part for me is remembering which of four different mkudffs switches set the string that shows up in /dev/disk/by-label/...

XFS: Why

marcH — Sun, 05 Apr 2015 02:05:56 +0000

> If you need cross OS compatibility, you need some fat variation.

or UDF.

XFS: Why

cesarb — Sat, 04 Apr 2015 12:32:39 +0000

Since it's a driver, I believe it has to be signed on all non-obsolete versions of Windows. Perhaps that's why the development of the ext driver for Windows stalled? Or is it just a lack of interest?

XFS: Why

deepfire — Sat, 04 Apr 2015 10:53:27 +0000

An ext driver for Windows, that can mount a modern ext4 FS image?

XFS: Why

sdalley — Fri, 03 Apr 2015 10:47:52 +0000

XFS has better (dynamic) allocation of inodes.

Creating large numbers of tiny files on ext4 ran me into an ENOSPC when the disk blocks were still less than half used, simply because the (allocated-at-mkfs-time) inodes had run out.

XFS handles all that automatically; you don't have to guess ahead of time that you're going to need vast numbers of inodes.

XFS: Why

dlang — Fri, 03 Apr 2015 00:34:20 +0000

what linux distro can't handle an XFS filesystem? there are several that can't boot from them, but they've all included the tools to be able to mount and access them for a very long time

If you need cross OS compatibility, you need some fat variation.

XFS: Why

rahvin — Fri, 03 Apr 2015 00:23:19 +0000

There is one very good reason to use ext2/3/4. Near universal compatibility with distributions and even other OS's. Heck you can even download an ext driver for windows.

Memristors schmemristors

rahvin — Fri, 03 Apr 2015 00:06:33 +0000

Not commenting on HP or the fact that the company is driven by MBA's rather than Engineers these days. But Memsistors were a theoretical idea for a long time and there is real science and real research behind them being proven as real. They are a recognized reality, not a fanciful projection at this point, at least as I understand the science behind them and the consensus of the scientific community at this point.

The challenge has always been (just like most things), if the now proven item can actually be constructed at a reasonable cost and are conducive towards mass production. In other words the actual engineering. Even if a researcher can build a single memsistor in a lab for a few million bucks it doesn't mean anyone is going to be able to build them in massive quantity at sizes that will make their use practical.

It's entirely possible memsistors could be impossible to construct in volume or smaller than a fridge which would make them irrelevant. HP seems to think they are not only constructable but economical but only time will tell. If they are successful though memsistors could change the entire computer industry.

XFS: There and back ... and there again?

dgc — Thu, 02 Apr 2015 21:24:43 +0000

Yup, XFS uses feature masks for compatibility, not name changes, and original Irix filesystems were never supported on Linux. e.g. linux never had dirv1 support, so no Irix filesystem prior to dirv2 (1998) would ever mount on Linux. And, yes, we have dropped support for old misfeatures over time (e.g. unwritten extents have to be enabled) and required features that existed when dirv2 came about. The result is that really old filesystems won't mount on modern kernels as a trade off for keeping the test matrix for current development manageable...

-Dave.

XFS: There and back ... and there again?

dgc — Thu, 02 Apr 2015 21:17:40 +0000

reflink is being worked on, in the early prototype stage. It's not upstream yet.

-Dave.

XFS: Why

dlang — Thu, 02 Apr 2015 17:22:41 +0000

The short version is that either will work find on a single disk laptop/disktop situation.

XFS is going to start showing it's advantages when you get to more advanced/larger disk systems (large RAID arrays)

ext* is a filesystem designed for small systems that's growing towards supporting large systems. XFS is a filesystem designed for large systems that's been getting slimmed down and optimized so that it also works well on small systems.

One example of this is that there have been bugs reported on ext* that the developers could not duplicate because none of them have access to similar hardware (where similar hardware is a fairly modest SCSI RAID array)

I've been using XFS for quite a few years on systems (since well before ext4) and so for me the question is the opposite, what's so compelling about ext4 to cause me to move away from xfs? :-)

I made the shift during the ext3 days when ext3 was crippling for anything that did fsyncs for data reliability. That particular bug has been fixed in ext4, but that's not enough reason to switch.

XFS: There and back ... and there again?

deater — Thu, 02 Apr 2015 16:50:25 +0000

> Impressive though that it's still called XFS...never a need for XFS2,
> XFS3, or XFS4. Seems like they got it right the first time. Legacy or
> not, it's one of the best.

I wouldn't go that far. I definitely have some XFS images lying around that were created on old IRIX systems that mounted just fine under Linux 2.4 but will not mount with newer kernels due to support for older features being dropped. So some sort of versioning difference has happened even if it's not reflected in the filesystem name.

XFS: Why

bfields — Thu, 02 Apr 2015 16:07:08 +0000

A lot of the really important differences are hard to summarize on a spec sheet: how they perform on a wide variety of workloads, how reliable they are, how easy it is for developers to support them. So, not being an expert on either filesystem, but working with people who are--I almost always rely on their judgement, which means going with whatever they chose for the distro's default. Boring answer, I apologize....

XFS: Why

Felix.Braun — Thu, 02 Apr 2015 14:39:00 +0000

I have considered giving XFS a try on one of my computers in the past but wasn't able to find any reason to prefer it over ext4. I'm all for using the right tool for the job. But I couldn't find any information about the advantages XFS has over ext4 apart from the fact that XFS allows filesizes up to 8 EB.

So when would you choose XFS over ext4?

XFS: There and back ... and there again?

walters — Thu, 02 Apr 2015 13:21:08 +0000

reflink support for per-file snapshots has been added

Really? I only see reflink bits in mainline for BTRFS and OCFS2.

XFS: There and back ... and there again?

jdub — Thu, 02 Apr 2015 09:17:48 +0000

I had to check, given the context. LWN has good Google Juice!

XFS: There and back ... and there again?

lkundrak — Thu, 02 Apr 2015 09:10:42 +0000

Haha!

How did you notice?

XFS: There and back ... and there again?

eru — Thu, 02 Apr 2015 09:09:27 +0000

One could say that in the Gartner Group's "hype cycle", XFS is now at the "plateau of productivity". One evidence for this is that RHEL7 at last suggests it as the default filesystem (or at least CentOS7 did, when I installed it recently, and I guess it is following RHEL there).

XFS: There and back ... and there again?

epa — Thu, 02 Apr 2015 07:16:15 +0000

A common definition says that a legacy system is one in current use.