An update on bcachefs [LWN.net]

An update on bcachefs

Posted May 23, 2018 23:51 UTC (Wed) by rahvin (guest, #16953)
In reply to: An update on bcachefs by Cyberax
Parent article: An update on bcachefs

There are a lot of people using it day to day including myself.

btrfs gets a bad rap for reasons I don't quite understand. All these new filesystems end up with tons of corner cases once they get broad deployment, btrfs was moving pretty fast until it got pushed out of beta then it slowed down as they spent and continue to spend tons of time fixing all these corner/edge cases. Although I'd argue significant prior available resources were pulled away when Oracle bought Sun as they were a prime backer.

Personally I'll be surprised if bcachefs doesn't run into the same long tail of corner cases once it's mainlined. I remember reading something saying it takes at least 10 years after a filesystem is mainlined before they deal with all these issues, after all it was pretty much 10 years after XFS was brought in before it really got stable.

to post comments

An update on bcachefs

Posted May 24, 2018 1:21 UTC (Thu) by simcop2387 (subscriber, #101710) [Link] (3 responses)

I think a lot of the bad rap came not from the btrfs developers themselves, but a few distros that started supporting it in production a bit too early. This caused a number of people to start using it and then experience those corner cases. It got a lot of testing but left a bad taste in people's mouths.

An update on bcachefs

Posted May 24, 2018 10:30 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

"Corner cases" my right foot. The reason for the bad taste is that btrfs would completely eat itself even when using supposedly stable/fully-supported configurations on stable hardware.

I've had two [might-as-well-be-]total filesystem losses with btrfs, both after clean shutdown/reboot cycles on lightly-loaded, battery-backed hardware that hasn't so much as hiccupped before or since.

An update on bcachefs

Posted May 25, 2018 15:54 UTC (Fri) by drag (guest, #31333) [Link]

There are two major times my personal Linux boxes ate my data...

1st time was when my cat yanked out a PCI card that wasn't screwed in. Corrupted xfs metadata, had to sort through tens of thousands of files in the recovery bin.

2nd time was while using btrfs.

There was also a couple times while using btrfs on external storage device I expanded it wedged itself into a corner. Ran out of inodes or something like that. It was a long time ago. Didn't lose data, but the file system was effectively rendered worthless and needed to be rebuilt.

Now these things are all partially my fault, but I have used Linux on garbage-level (sometimes literally) hardware for years and it's been proven to be tough.

I am looking forward to bcachefs getting support from various distro installers.

An update on bcachefs

Posted May 24, 2018 13:11 UTC (Thu) by dcg (subscriber, #9198) [Link]

I don't think being unstable was really that much of a problem at first; people expected btrfs to be unstable anyway. The problem is that it kept being unstable because there weren't developers that would focus on fixing that. There were developers that worked on adding features (often not cleanly, eg qgroups were very unstable and still today they impact performance so much that people recommends not using them unless it's necessary), but the core was left unattended. Which is the reason why, still today and after so many years, btrfs does not have clean, ZFS-class RAID5/6 support, or integrated encryption. The lack of improvement in some key fields seem to be making people feel that btrfs is stagnating, and who would blame them? It is in fact one of the motivations behind bcachefs.

As someone who has been using btrfs for many years without problems I also feel that btrfs is not appreciated enough (and that bcachefs is over-hyped - Overstreet has criticized btrfs for its stability, but I have seen several cases of people having corruption issues in the bcachefs IRC channel). But I wonder how much can that last if btrfs keeps not progressing in basic features like RAID5/6?

An update on bcachefs

Posted May 24, 2018 6:14 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

A show-stopping bug in BTRFS RAIDs was discovered only _last_ _year_, after BTRFS had been marked as production-ready for at least a couple of years.

From my experience, I'm using BTRFS on my personal external RAID tower. It got corrupted every single time I tried its multi-device support. Mostly because of botched RAID rebuilds after power failures or disk replacements.

I've got tired of this and switched to a good old MD-based RAID. I'm still using BTFS for snapshots, but there's no way in hell I'm going to touch its multi-device snapshot within the next 5 years or so.

An update on bcachefs

Posted May 24, 2018 7:48 UTC (Thu) by khim (subscriber, #9252) [Link] (5 responses)

Have BTRFS developers ever claimed that it's RAID code is production-ready? AFAIK it was always maked red (== experimental, don't use yet) in their table.

An update on bcachefs

Posted May 24, 2018 7:53 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

This page cheerfully says that it's OK: https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_... Back then this page also looked nice and cheerful: https://btrfs.wiki.kernel.org/index.php?title=RAID56&...

An update on bcachefs

Posted May 25, 2018 19:26 UTC (Fri) by khim (subscriber, #9252) [Link] (3 responses)

Ok, I'll bite. You say that "back then" page looked nice and cheerful. Here's the full history.

When exactly RAID56 wasn't marked with red and word "Unstable"?

When you are ignoring warnings written in red... bad things happen... that's just how life is it...

An update on bcachefs

Posted May 25, 2018 20:54 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

"Write hole still exists" is a far cry from "will corrupt your data irrevocably on rebuild".

Also, I wasn't aware of this page's existence. RAID56 search on Google gives another page that looked OK.

An update on bcachefs

Posted May 31, 2018 21:45 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

To be fair, fixing the write hole is tricky - in fact it seems to me that it is inherent and unavoidable in a simple raid 5 implementation.

Even md raid has only just fixed it, with the addition of a journal (a feature that might still be experimental). The journal is intended primarily as an optimisation, flushing updates to SSD before saving them to the full raid on spinning rust. It fixes the write hole almost as a side-effect.

(Note, understanding the journal is on my to-do list, I need to learn more about it before I can document it ... :-)

Cheers,
Wol

An update on bcachefs

Posted Jun 2, 2018 17:52 UTC (Sat) by koverstreet (subscriber, #4296) [Link]

Except btrfs is copy on write - the raid5 hole exists because of doing updates in place on existing stripes, so why btrfs is doing it that way I have no idea.

An update on bcachefs

Posted May 24, 2018 21:50 UTC (Thu) by rahvin (guest, #16953) [Link] (1 responses)

I know it's had issues but it's this bad rap history that follows it around that I think does more reputation damage than the current code deserves. This is one of the reasons it bothers me so much that Redhat decided to devote resources to expanding feature to XFS instead of spending those resources stabilizing btrfs. Btrfs could be IMO the best filesystem Linux has ever had if we could get experienced filesystem developers working on it instead of everyone developing their own filesystem. Which is probably why Redhat's decision bothers me.

It just seems like everyone wants to work on something new rather than try to fix what we've got. I get that, fixing bugs is hard and boring rather than developing something new and shiny. But that new and shiny is still going to take 10 years to stabilize just like every filesystem Linux has had.

This is actually a question I'd like to see answered at one of these filesystem conventions, Why is it so hard to get experienced developers to work stabilizing the fs's we have rather than building new ones?

An update on bcachefs

Posted May 24, 2018 22:51 UTC (Thu) by nivedita76 (subscriber, #121790) [Link]

XFS is around 25 years old, and has been in Linux since the early 00's, not new and shiny. BTRFS is the "new and shiny" fs by comparison. XFS history

An update on bcachefs

Posted May 24, 2018 7:18 UTC (Thu) by vadim (subscriber, #35271) [Link] (1 responses)

Probably because BTRFS got merged too early and had serious gotchas for the cool features.

Eg, the "raid1 volumes only mountable once RW if degraded" issue was there until very recently and is not the kind of thing one wants to deal with on a RAID setup.

Compression until recently came with warnings.

Compression is still quite half-assed on the UI side. Why can't I easily check how well a file is being compressed? Why do I have to 'defrag' to compress, on a SSD? I don't care if it's fragmented, I just want it compressed. There's no need to move already compressed data around.

Snapshots are in theory very cool, but in practice easily cause horrible performance issues. Eg, I unwisely had installed snapper and allowed DNF to make a couple dozen snapshots of my filesystem. Cleaning that up took an entire day, on an SSD, with the first snapshots taking hours to remove and completely locking up the machine. I don't want to imagine how that would work on an HDD.

Despite my best efforts so far I haven't figured out a way to do a btrfs scrub in such a way that it doesn't make the entire desktop lag, on a computer with a SSD.

An update on bcachefs

Posted May 25, 2018 2:03 UTC (Fri) by zlynx (guest, #2285) [Link]

One thing to point out about slow btrfs scrubs is that it depends on your IO scheduler. Use the standard CFQ or BFQ. If you use deadline or noop then of course the scrub is going to use up all the IO because nothing will stop it.

It still causes some extra latency. On my email server I use multi-queue BFQ and I can feel a bit of extra delay when loading each IMAP message from it during a scrub, but I don't think it's that serious.

And on my laptop with NVMe SSD the btrfs scrub is done in about 30 seconds, so it isn't a big deal there either. It reads and verifies at hundreds of megabytes per second.