LWN: Comments on "No Btrfs by default in Fedora 16"

No Btrfs by default in Fedora 16

rodgerd — Fri, 12 Aug 2011 03:24:51 +0000

An md-cache driver.

No Btrfs by default in Fedora 16

cmccabe — Thu, 11 Aug 2011 18:22:50 +0000

As other posters have pointed out, when snapshots are done at the filesystem level, they are done atomically. So you get a single consistent view of what the filesystem was like at a point in time. You can't get that with your script, unless you remount the filesystem as read-only while it's running-- which I don't think you want to do.

Another thing you can't do is copy-on-write within files. Let's say your filesystem contains a bunch of 20 gigabyte virtual machine images. Between snapshot A and snapshot B, I modify 1 megabyte of data inside one of these giant images. Your hardlink script needs to copy the whole 20 gigabyte virtual machine image, since it has changed. But btrfs' copy-on-write only needs to copy the part that has changed.

btrfs isn't the right filesystem for every application. But you should at least give it a chance.

No Btrfs by default in Fedora 16

anton — Thu, 11 Aug 2011 15:27:14 +0000

isn't a copy on write approach the opposite of what one would normally choose to get high random write performance?

No, if you read the position paper on log--structured file systems, of which COW file systems are a generalization, write performance was the main argument for this style of file system. In particular, random writes are very slow on an in-place-updating file system, because they require many seeks. In contrast a copy-on-write file system can just write them (and their meta-data) sequentially, i.e., quickly, requiring just one seek (and maybe another one to update the root block). Of course, if you want to read the resulting files sequentially, this strategy loses; you cannot have it all.

No Btrfs by default in Fedora 16

raven667 — Thu, 11 Aug 2011 15:13:22 +0000

Snapshots don't replace backups but they do compliment very well in two ways. One is that you can run your backup against the point-in-time snapshot and eliminate any inconsistancy between the backup of individual files that are being modified while the backup is running and two, you can potentially take snapshots more frequently than you can take backups. Depending on whether your backup system allows self-serivice restores, snapshots can also have the benefit of allowing a local user or admin to restore files, from accidental deletion for example, without making a round trip through your large backup system, without mounting tapes, etc.

No Btrfs by default in Fedora 16

nix — Thu, 11 Aug 2011 14:30:24 +0000

Writing to the middle of a file always *could* lead to running out of disk space (think sparse files), and any program which can't handle it on files that others might touch is broken. (This means databases are safe: anyone who makes a DB file sparse without the RDBMS knowing about it is playing with fire.)

No Btrfs by default in Fedora 16

nye — Thu, 11 Aug 2011 12:35:33 +0000

>Reiserfs's main problem, for me, was that some minor FS corruption can and most probably will result in a completely unusable file system if you happen to have a FS image with a reiserfs file system on the disk

Only if you use the --rebuild-tree option to reiserfsck which is explicitly designed to trawl over a partition looking for anything which might be reiserfs structures as a last resort for recovering a badly wrecked filesystem which would otherwise be a write-off.

'Doctor, it hurts when I do this'.

No Btrfs by default in Fedora 16

smurf — Thu, 11 Aug 2011 12:03:03 +0000

Snapshots have the intriguing advantage that they an give you "this instant in time" file system state to back up, instead of "some random jumble of inconsistent state during whatever time the backup took to run". Combine that with FS-level transactions and you don't need to dump your database contents any more. Win.

They also have the less-than-intriguing disadvantage that writing to the middle of a file can result in running out of disk space. Databases typically don't cope with that at all well. (Running out while extending a file, OTOH, is a well-understood problem.)

No Btrfs by default in Fedora 16

SLi — Thu, 11 Aug 2011 12:00:31 +0000

Except that if you *do* have a feature-rich backup system, capable of backing up multiple different systems, you aren't just going to toss it out because one filesystem suddenly got some dedup support. At least I'm not going to do that, and I'm the OP :) Dedup is nice, but I really don't think it should be thought of as something that replaces higher level deduplication done by backup systems.

No Btrfs by default in Fedora 16

smurf — Thu, 11 Aug 2011 11:56:52 +0000

Reiserfs's main problem, for me, was that some minor FS corruption can and most probably will result in a completely unusable file system if you happen to have a FS image with a reiserfs file system on the disk. Instant bad karma.

I do hope that btrfsck will do better.

No Btrfs by default in Fedora 16

cmccabe — Thu, 11 Aug 2011 00:33:20 +0000

I think filesystem snapshots are a good replacement for what the original poster was trying to do with rsync and hard links.

Basically, you don't need a clever and feature-rich backup system in userspace if you have a filesystem that provides snapshots that work well.

Of course, neither one of these things can replace a real backup to a separate physical computer or to tapes that are stored in a different place.

No Btrfs by default in Fedora 16

SLi — Wed, 10 Aug 2011 12:07:47 +0000

That sounds like a nice idea at first, but then having a backup system inherently tied to a single filesystem is perhaps not such a great idea, especially as a clever and feature-rich backup system would still have to keep track of all the state in userspace. And still supporting hard links sanely would be quite critical for compatibility with existing systems...

No Btrfs by default in Fedora 16

SLi — Wed, 10 Aug 2011 12:03:46 +0000

Yes, they always keep saying that in all threads that come up about this, and I've never quite understood what they mean by it. Still, I've hit this multiple times.

What does it mean for "the file" to be in one directory and a hard link in another? For example in my backuppc setup there's a pool of files indexed and named by (basically) their SHA1 sums, and multiple hardlinked copies of the backed up filesystem state. Now as far as I can tell, if you just happen to have lots of files with the same content in one directory, you are going to hit this bug, right? As the hard link is only a reference count, it doesn't make sense to say that the file "is" in the pool directory, does it?

On disk format vs. mount options.

gmatht — Tue, 09 Aug 2011 22:20:46 +0000

I am not sure what you mean. For example mount options will clearly also have an effect, and it seems likely that the performance in this particular benchmark relates to Btrfs defaulting to datacow. Btrfs also supports O(1) file copies so in many ways it has better performance than ext4.

Fixing on disk format is clearly a priority, but that is orthogonal to performance vs. features. For example, checksumming, and the "feature" of allowing hundreds of hardlinks per directory, requires format changes. On the other hand adding a heuristic to toggle datacow probably wouldn't.

No Btrfs by default in Fedora 16

mrons — Tue, 09 Aug 2011 22:13:18 +0000

> Are you doing dedup using rsync on that filesystem?

Yes.

After following the threads in the above bugzilla, I see that the problem is not as severe as I thought. To quote Chis Mason:

> Please keep in mind this is only a limit on the number of links to a single file where the links and the file are all in the same directory

So with that limit I would likely be able to re-create my filesystem on btfs, but only just.

No Btrfs by default in Fedora 16

raven667 — Tue, 09 Aug 2011 22:01:59 +0000

Are you doing dedup using rsync on that filesystem? I don't know about btrfs but if this is the use case then is there some other way to accomplish deduped backups on btrfs?

No Btrfs by default in Fedora 16

mrons — Tue, 09 Aug 2011 21:43:53 +0000

Wow, that limitation on hard links is hard to believe.

I have a filesystem (ext4) that has has over 8 million hard links.

What are they thinking here? Surely this is a mistake.

No Btrfs by default in Fedora 16

Lennie — Tue, 09 Aug 2011 20:47:36 +0000

It will be a long time before SSD has a somewhat equal price/space ratio like HDD. So I really hope Linux gets something like bcache or flashcache or ZFS-L2ARC or whatever fits Linux best. Then we can just use HDD with SSD as a large fast cache.

No Btrfs by default in Fedora 16

etrusco — Tue, 09 Aug 2011 20:38:53 +0000

HTree was itself the feature added. It made things better, but not spectacular...

No Btrfs by default in Fedora 16

cmccabe — Tue, 09 Aug 2011 20:12:02 +0000

There was a feature added to ext2/ext3 called directory indexing which I think was designed to address the slowness with large directories.

I think the ext4 multiblock allocator probably also helps the situation with directories with many small files.

A lot of these optimizations will be less important once the world moves to SSD. On hard drives, though, keeping related stuff close together is the name of the game.

No Btrfs by default in Fedora 16

bcopeland — Tue, 09 Aug 2011 18:41:26 +0000

Every FS has its sweet spots and corner cases. Ext3 used to be quite bad at reading large directories with many small files (the reason being that scanning the HTree returns files in approximately random disk order). Nightly backups exemplify that particular workload.

I don't know if any changes in ext4 specifically addressed that, so perhaps it is still the case. But then all the tools starting doing sort-by-inode to sidestep the problem so it's less of a problem today.

No Btrfs by default in Fedora 16

iabervon — Tue, 09 Aug 2011 17:36:59 +0000

Would that feature be described as "Per-file compression and nocow control"? Because that's in the 2.6.39 features list. I assume that the performance tests (and, probably, actual applications) aren't using that yet.

No Btrfs by default in Fedora 16

SLi — Tue, 09 Aug 2011 08:29:06 +0000

And I obviously mean hardlinks, not symlinks.

No Btrfs by default in Fedora 16

SLi — Tue, 09 Aug 2011 07:49:16 +0000

I must add that performance is one of the other things that keeps me using ext4, but that's not a btrfs-only gripe.

Seriously I have tried many other filesystems with several real-world benchmarks. In nearly all benchmarks ext4 is among the best performers, not always stellar but really good anyway. All other filesystems I have tested (xfs, jfs, btrfs) seem to perform *really* poorly in some rather pathological but realistic and real-world cases.

For some filesystems the pathological case could be deleting huge directory structures, for others creating a lot of small files, and for many of them mass symlinking.

The one filesystem that generally performed even better than ext* (I think ext3 at the time, which did have some performance problems) was reiser4. However on three separate occasions of mere testing use I experienced massive corruption with reiser4 when the developers had already declared it stable (Some bug was always fixed after I reported that...), so while it seemed promising at the time, I would never use it in production use without a lot of further testing.

But then I think such performance tuning with btrfs would be premature at this point where such basic features as symlinks are not supported well enough.

No Btrfs by default in Fedora 16

SLi — Tue, 09 Aug 2011 07:36:34 +0000

The one showstopper I have run into that definitely prevents me from using btrfs is that it limits the number of symlinks in a single directory to something ridiculously low (it depends apparently on everything including the length of the filenames, but you can hit the limit after 31 symlinks and will hit it after a few hundred).

https://bugzilla.kernel.org/show_bug.cgi?id=15762

No Btrfs by default in Fedora 16

butlerm — Tue, 09 Aug 2011 06:02:16 +0000

If you don't have performance in mind before you start designing a filesystem, nothing else will matter when you are done. The on disk format will dictate everything about how well your filesystem will perform in the real world. And of course the on disk format is the one thing that you won't be able to change.

No Btrfs by default in Fedora 16

dlang — Mon, 08 Aug 2011 23:58:21 +0000

for firefox, the change to have a backup file and only sync the new file every few minutes would be a good change on every filesystem.

No Btrfs by default in Fedora 16

butlerm — Mon, 08 Aug 2011 23:06:36 +0000

Random reads are not nearly as much of a problem, as long as the index pages are well cached, which they would be for relatively small files. The problem with most copy on write filesystems is that every random access involves an index lookup to locate the right data page, and the indexes for something that has experienced a lot of random writes have essentially one entry per data page. If you have very large files, that may mean two or more physical I/Os to do a random read, one to look up an uncached index page, and one to read the data page.

It seems to me that what BTRFS needs for something like Firefox is the ability to disable copy-on-write for individual files, except when a user takes a volume snapshot. Alternatively, applications like Firefox could keep a backup database file, and only fsync the primary database file every ten minutes or so, and check on startup to see if the primary database file needs to be recovered from the backup.

No Btrfs by default in Fedora 16

rodgerd — Mon, 08 Aug 2011 20:11:31 +0000

1/ Optimising performance before it's feature complete would be a bit of a poor idea.

2/ Some of the performance is just inherent to COW filesystems. If you look at benchmarketing exercises between ZFS/OpenSolaris and btrfs/Linux with something like a Postgresql load you'll see quite comparable performance.

3/ I would expect btrfs, long term, to shine in the same places ZFS does, which is with many spindles, and using all the features (data integrity, snapshots, and so on).

It feels kind of odd sticking up for btrfs.

No Btrfs by default in Fedora 16

brunock — Mon, 08 Aug 2011 19:15:19 +0000

Yes, but BTRFS is supposed to be a general purpose file system.
And busy relational databases aren't a server only feature.
Think of your spam filter or about what Firefox does in the
background all the time.
So I guess normal users will get hit by this problem, too.

No Btrfs by default in Fedora 16

butlerm — Mon, 08 Aug 2011 18:46:59 +0000

For all of its advantages, isn't a copy on write approach the opposite of what one would normally choose to get high random write performance? A filesystem like BTRFS is doing several times as much work for each random write. For general use, BTRFS seems like the way to go, but I can't see anyone choosing to host a busy relational database on it anytime soon. The same goes for ZFS.

What about subvolume support in Anaconda?

rahulsundaram — Mon, 08 Aug 2011 16:39:19 +0000

Unlikely for Fedora 16 really consider we are nearing the Alpha release but do file a feature request.

What about subvolume support in Anaconda?

dcg — Mon, 08 Aug 2011 16:35:44 +0000

That's not a problem for me, as long as it's optionally available. My problem with Btrfs and Fedora right now is that Anaconda can't handle subvolumes at all and configuring different subvolumes for different paths is only possible doing it manually after the installation.

It would be nice if in F16 could at least add subvolume support in Anaconda...

No Btrfs by default in Fedora 16

masoncl — Mon, 08 Aug 2011 16:08:35 +0000

It's really just juggling the features I want btrfsck to have with keeping up with QA and patch review on the kernel bits.

We have a lot of very active development work going on, and while my all of my R&D time on is fsck, I still need to patch and test the kernel parts too. I did let a few distractions creep in, such as redoing the metadata locking in 3.1-rc1, but that was a major performance bottleneck.

I think f17 is a better choice for Fedora. We're working in parallel on the fsck and closing off the few remaining performance issues.

No Btrfs by default in Fedora 16

trey — Mon, 08 Aug 2011 15:58:38 +0000

FYI:

http://marc.info/?l=linux-btrfs&m=131235467525138&...
http://marc.info/?l=linux-btrfs&m=131240481916170&...

No Btrfs by default in Fedora 16

brunock — Mon, 08 Aug 2011 15:48:41 +0000

Don't forget that performance also still is a major problem:

http://free.linux.hp.com/~enw/ext4/3.0/large_file_creates...
http://free.linux.hp.com/~enw/ext4/3.0/random_writes.html

No Btrfs by default in Fedora 16

rahulsundaram — Mon, 08 Aug 2011 15:25:51 +0000

$rpm -qf /sbin/btrfsck
btrfs-progs-0.19-13.fc15.x86_64

It can report errors but not correct them yet. Should be there soon, I hope

https://lists.fedoraproject.org/pipermail/devel/2011-Augu...

No Btrfs by default in Fedora 16

and — Mon, 08 Aug 2011 15:21:36 +0000

> Among other things, those requirements include a working filesystem
> checker, and that is not yet on offer.

Wasn't fsck.btrfs anounced to be released RSN back in february? I wonder what keeps it back for so long...