No Btrfs by default in Fedora 16

[Posted August 8, 2011 by corbet]

From:		Josef Bacik <josef-AT-toxicpanda.com>
To:		Development discussions related to Fedora <devel-AT-lists.fedoraproject.org>
Subject:		Btrfs status for F16
Date:		Mon, 8 Aug 2011 08:44:44 -0400
Message-ID:		<CAEzrpqesaT7PTvCy4=NP+RP4qMef8W0AY=3wWZ_24rNP+Vj_2w@mail.gmail.com>
Archive‑link:		Article

Hello,

In order to hopefully (and I understand this is a unrealistically big
hope) stem the amount of hostile comments and random remarks about
Btrfs not being ready for F16 that I get with _every_ bz that get's
filed against it, let me announce this as clearly as possible

BTRFS WILL NOT BE THE DEFAULT FILE SYSTEM FOR F16

Fesco outlined basic requirements that needed to be met by Alpha for
the switch to be allowed to happen and we have not met those
requirements so it won't be happening for F16.

I appreciate those who will continue to use it and report bugs, we are
working very hard on trying to get everything more stable and it is a
slow going process.  With your help we will be in a better situation
for F17.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 15:21 UTC (Mon) by and (guest, #2883) [Link] (13 responses)

> Among other things, those requirements include a working filesystem
> checker, and that is not yet on offer.

Wasn't fsck.btrfs anounced to be released RSN back in february? I wonder what keeps it back for so long...

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 15:25 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

$rpm -qf /sbin/btrfsck
btrfs-progs-0.19-13.fc15.x86_64

It can report errors but not correct them yet. Should be there soon, I hope

https://lists.fedoraproject.org/pipermail/devel/2011-Augu...

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 15:58 UTC (Mon) by trey (guest, #37500) [Link]

FYI:

http://marc.info/?l=linux-btrfs&m=131235467525138&...
http://marc.info/?l=linux-btrfs&m=131240481916170&...

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 15:48 UTC (Mon) by brunock (guest, #76114) [Link] (9 responses)

Don't forget that performance also still is a major problem:

http://free.linux.hp.com/~enw/ext4/3.0/large_file_creates...
http://free.linux.hp.com/~enw/ext4/3.0/random_writes.html

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 18:46 UTC (Mon) by butlerm (subscriber, #13312) [Link] (5 responses)

For all of its advantages, isn't a copy on write approach the opposite of what one would normally choose to get high random write performance? A filesystem like BTRFS is doing several times as much work for each random write. For general use, BTRFS seems like the way to go, but I can't see anyone choosing to host a busy relational database on it anytime soon. The same goes for ZFS.

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 19:15 UTC (Mon) by brunock (guest, #76114) [Link] (3 responses)

Yes, but BTRFS is supposed to be a general purpose file system.
And busy relational databases aren't a server only feature.
Think of your spam filter or about what Firefox does in the
background all the time.
So I guess normal users will get hit by this problem, too.

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 23:06 UTC (Mon) by butlerm (subscriber, #13312) [Link] (2 responses)

Random reads are not nearly as much of a problem, as long as the index pages are well cached, which they would be for relatively small files. The problem with most copy on write filesystems is that every random access involves an index lookup to locate the right data page, and the indexes for something that has experienced a lot of random writes have essentially one entry per data page. If you have very large files, that may mean two or more physical I/Os to do a random read, one to look up an uncached index page, and one to read the data page.

It seems to me that what BTRFS needs for something like Firefox is the ability to disable copy-on-write for individual files, except when a user takes a volume snapshot. Alternatively, applications like Firefox could keep a backup database file, and only fsync the primary database file every ten minutes or so, and check on startup to see if the primary database file needs to be recovered from the backup.

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 23:58 UTC (Mon) by dlang (guest, #313) [Link]

for firefox, the change to have a backup file and only sync the new file every few minutes would be a good change on every filesystem.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 17:36 UTC (Tue) by iabervon (subscriber, #722) [Link]

Would that feature be described as "Per-file compression and nocow control"? Because that's in the 2.6.39 features list. I assume that the performance tests (and, probably, actual applications) aren't using that yet.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 15:27 UTC (Thu) by anton (subscriber, #25547) [Link]

isn't a copy on write approach the opposite of what one would normally choose to get high random write performance?

No, if you read the position paper on log--structured file systems, of which COW file systems are a generalization, write performance was the main argument for this style of file system. In particular, random writes are very slow on an in-place-updating file system, because they require many seeks. In contrast a copy-on-write file system can just write them (and their meta-data) sequentially, i.e., quickly, requiring just one seek (and maybe another one to update the root block). Of course, if you want to read the resulting files sequentially, this strategy loses; you cannot have it all.

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 20:11 UTC (Mon) by rodgerd (guest, #58896) [Link] (2 responses)

1/ Optimising performance before it's feature complete would be a bit of a poor idea.

2/ Some of the performance is just inherent to COW filesystems. If you look at benchmarketing exercises between ZFS/OpenSolaris and btrfs/Linux with something like a Postgresql load you'll see quite comparable performance.

3/ I would expect btrfs, long term, to shine in the same places ZFS does, which is with many spindles, and using all the features (data integrity, snapshots, and so on).

It feels kind of odd sticking up for btrfs.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 6:02 UTC (Tue) by butlerm (subscriber, #13312) [Link] (1 responses)

If you don't have performance in mind before you start designing a filesystem, nothing else will matter when you are done. The on disk format will dictate everything about how well your filesystem will perform in the real world. And of course the on disk format is the one thing that you won't be able to change.

On disk format vs. mount options.

Posted Aug 9, 2011 22:20 UTC (Tue) by gmatht (guest, #58961) [Link]

I am not sure what you mean. For example mount options will clearly also have an effect, and it seems likely that the performance in this particular benchmark relates to Btrfs defaulting to datacow. Btrfs also supports O(1) file copies so in many ways it has better performance than ext4.

Fixing on disk format is clearly a priority, but that is orthogonal to performance vs. features. For example, checksumming, and the "feature" of allowing hundreds of hardlinks per directory, requires format changes. On the other hand adding a heuristic to toggle datacow probably wouldn't.

No Btrfs by default in Fedora 16

Posted Aug 8, 2011 16:08 UTC (Mon) by masoncl (subscriber, #47138) [Link]

It's really just juggling the features I want btrfsck to have with keeping up with QA and patch review on the kernel bits.

We have a lot of very active development work going on, and while my all of my R&D time on is fsck, I still need to patch and test the kernel parts too. I did let a few distractions creep in, such as redoing the metadata locking in 3.1-rc1, but that was a major performance bottleneck.

I think f17 is a better choice for Fedora. We're working in parallel on the fsck and closing off the few remaining performance issues.

What about subvolume support in Anaconda?

Posted Aug 8, 2011 16:35 UTC (Mon) by dcg (subscriber, #9198) [Link] (1 responses)

That's not a problem for me, as long as it's optionally available. My problem with Btrfs and Fedora right now is that Anaconda can't handle subvolumes at all and configuring different subvolumes for different paths is only possible doing it manually after the installation.

It would be nice if in F16 could at least add subvolume support in Anaconda...

What about subvolume support in Anaconda?

Posted Aug 8, 2011 16:39 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]

Unlikely for Fedora 16 really consider we are nearing the Alpha release but do file a feature request.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 7:36 UTC (Tue) by SLi (subscriber, #53131) [Link] (20 responses)

The one showstopper I have run into that definitely prevents me from using btrfs is that it limits the number of symlinks in a single directory to something ridiculously low (it depends apparently on everything including the length of the filenames, but you can hit the limit after 31 symlinks and will hit it after a few hundred).

https://bugzilla.kernel.org/show_bug.cgi?id=15762

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 7:49 UTC (Tue) by SLi (subscriber, #53131) [Link] (7 responses)

I must add that performance is one of the other things that keeps me using ext4, but that's not a btrfs-only gripe.

Seriously I have tried many other filesystems with several real-world benchmarks. In nearly all benchmarks ext4 is among the best performers, not always stellar but really good anyway. All other filesystems I have tested (xfs, jfs, btrfs) seem to perform *really* poorly in some rather pathological but realistic and real-world cases.

For some filesystems the pathological case could be deleting huge directory structures, for others creating a lot of small files, and for many of them mass symlinking.

The one filesystem that generally performed even better than ext* (I think ext3 at the time, which did have some performance problems) was reiser4. However on three separate occasions of mere testing use I experienced massive corruption with reiser4 when the developers had already declared it stable (Some bug was always fixed after I reported that...), so while it seemed promising at the time, I would never use it in production use without a lot of further testing.

But then I think such performance tuning with btrfs would be premature at this point where such basic features as symlinks are not supported well enough.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 18:41 UTC (Tue) by bcopeland (subscriber, #51750) [Link] (4 responses)

Every FS has its sweet spots and corner cases. Ext3 used to be quite bad at reading large directories with many small files (the reason being that scanning the HTree returns files in approximately random disk order). Nightly backups exemplify that particular workload.

I don't know if any changes in ext4 specifically addressed that, so perhaps it is still the case. But then all the tools starting doing sort-by-inode to sidestep the problem so it's less of a problem today.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 20:12 UTC (Tue) by cmccabe (guest, #60281) [Link] (3 responses)

There was a feature added to ext2/ext3 called directory indexing which I think was designed to address the slowness with large directories.

I think the ext4 multiblock allocator probably also helps the situation with directories with many small files.

A lot of these optimizations will be less important once the world moves to SSD. On hard drives, though, keeping related stuff close together is the name of the game.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 20:38 UTC (Tue) by etrusco (guest, #4227) [Link]

HTree was itself the feature added. It made things better, but not spectacular...

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 20:47 UTC (Tue) by Lennie (subscriber, #49641) [Link] (1 responses)

It will be a long time before SSD has a somewhat equal price/space ratio like HDD. So I really hope Linux gets something like bcache or flashcache or ZFS-L2ARC or whatever fits Linux best. Then we can just use HDD with SSD as a large fast cache.

No Btrfs by default in Fedora 16

Posted Aug 12, 2011 3:24 UTC (Fri) by rodgerd (guest, #58896) [Link]

An md-cache driver.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 11:56 UTC (Thu) by smurf (subscriber, #17840) [Link] (1 responses)

Reiserfs's main problem, for me, was that some minor FS corruption can and most probably will result in a completely unusable file system if you happen to have a FS image with a reiserfs file system on the disk. Instant bad karma.

I do hope that btrfsck will do better.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 12:35 UTC (Thu) by nye (subscriber, #51576) [Link]

>Reiserfs's main problem, for me, was that some minor FS corruption can and most probably will result in a completely unusable file system if you happen to have a FS image with a reiserfs file system on the disk

Only if you use the --rebuild-tree option to reiserfsck which is explicitly designed to trawl over a partition looking for anything which might be reiserfs structures as a last resort for recovering a badly wrecked filesystem which would otherwise be a write-off.

'Doctor, it hurts when I do this'.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 8:29 UTC (Tue) by SLi (subscriber, #53131) [Link] (11 responses)

And I obviously mean hardlinks, not symlinks.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 21:43 UTC (Tue) by mrons (subscriber, #1751) [Link] (10 responses)

Wow, that limitation on hard links is hard to believe.

I have a filesystem (ext4) that has has over 8 million hard links.

What are they thinking here? Surely this is a mistake.

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 22:01 UTC (Tue) by raven667 (subscriber, #5198) [Link] (9 responses)

Are you doing dedup using rsync on that filesystem? I don't know about btrfs but if this is the use case then is there some other way to accomplish deduped backups on btrfs?

No Btrfs by default in Fedora 16

Posted Aug 9, 2011 22:13 UTC (Tue) by mrons (subscriber, #1751) [Link] (1 responses)

> Are you doing dedup using rsync on that filesystem?

Yes.

After following the threads in the above bugzilla, I see that the problem is not as severe as I thought. To quote Chis Mason:

> Please keep in mind this is only a limit on the number of links to a single file where the links and the file are all in the same directory

So with that limit I would likely be able to re-create my filesystem on btfs, but only just.

No Btrfs by default in Fedora 16

Posted Aug 10, 2011 12:03 UTC (Wed) by SLi (subscriber, #53131) [Link]

Yes, they always keep saying that in all threads that come up about this, and I've never quite understood what they mean by it. Still, I've hit this multiple times.

What does it mean for "the file" to be in one directory and a hard link in another? For example in my backuppc setup there's a pool of files indexed and named by (basically) their SHA1 sums, and multiple hardlinked copies of the backed up filesystem state. Now as far as I can tell, if you just happen to have lots of files with the same content in one directory, you are going to hit this bug, right? As the hard link is only a reference count, it doesn't make sense to say that the file "is" in the pool directory, does it?

No Btrfs by default in Fedora 16

Posted Aug 10, 2011 12:07 UTC (Wed) by SLi (subscriber, #53131) [Link] (6 responses)

That sounds like a nice idea at first, but then having a backup system inherently tied to a single filesystem is perhaps not such a great idea, especially as a clever and feature-rich backup system would still have to keep track of all the state in userspace. And still supporting hard links sanely would be quite critical for compatibility with existing systems...

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 0:33 UTC (Thu) by cmccabe (guest, #60281) [Link] (5 responses)

I think filesystem snapshots are a good replacement for what the original poster was trying to do with rsync and hard links.

Basically, you don't need a clever and feature-rich backup system in userspace if you have a filesystem that provides snapshots that work well.

Of course, neither one of these things can replace a real backup to a separate physical computer or to tapes that are stored in a different place.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 12:00 UTC (Thu) by SLi (subscriber, #53131) [Link] (2 responses)

Except that if you *do* have a feature-rich backup system, capable of backing up multiple different systems, you aren't just going to toss it out because one filesystem suddenly got some dedup support. At least I'm not going to do that, and I'm the OP :) Dedup is nice, but I really don't think it should be thought of as something that replaces higher level deduplication done by backup systems.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 15:13 UTC (Thu) by raven667 (subscriber, #5198) [Link]

Snapshots don't replace backups but they do compliment very well in two ways. One is that you can run your backup against the point-in-time snapshot and eliminate any inconsistancy between the backup of individual files that are being modified while the backup is running and two, you can potentially take snapshots more frequently than you can take backups. Depending on whether your backup system allows self-serivice restores, snapshots can also have the benefit of allowing a local user or admin to restore files, from accidental deletion for example, without making a round trip through your large backup system, without mounting tapes, etc.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 18:22 UTC (Thu) by cmccabe (guest, #60281) [Link]

As other posters have pointed out, when snapshots are done at the filesystem level, they are done atomically. So you get a single consistent view of what the filesystem was like at a point in time. You can't get that with your script, unless you remount the filesystem as read-only while it's running-- which I don't think you want to do.

Another thing you can't do is copy-on-write within files. Let's say your filesystem contains a bunch of 20 gigabyte virtual machine images. Between snapshot A and snapshot B, I modify 1 megabyte of data inside one of these giant images. Your hardlink script needs to copy the whole 20 gigabyte virtual machine image, since it has changed. But btrfs' copy-on-write only needs to copy the part that has changed.

btrfs isn't the right filesystem for every application. But you should at least give it a chance.

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 12:03 UTC (Thu) by smurf (subscriber, #17840) [Link] (1 responses)

Snapshots have the intriguing advantage that they an give you "this instant in time" file system state to back up, instead of "some random jumble of inconsistent state during whatever time the backup took to run". Combine that with FS-level transactions and you don't need to dump your database contents any more. Win.

They also have the less-than-intriguing disadvantage that writing to the middle of a file can result in running out of disk space. Databases typically don't cope with that at all well. (Running out while extending a file, OTOH, is a well-understood problem.)

No Btrfs by default in Fedora 16

Posted Aug 11, 2011 14:30 UTC (Thu) by nix (subscriber, #2304) [Link]

Writing to the middle of a file always *could* lead to running out of disk space (think sparse files), and any program which can't handle it on files that others might touch is broken. (This means databases are safe: anyone who makes a DB file sparse without the RDBMS knowing about it is playing with fire.)