LWN: Comments on "A bcache update"

A bcache update

Lennie — Tue, 17 Jul 2012 16:30:34 +0000

"This would mean that I can't add ssd bcache to existing device/filesystem, right? Would be nice to be able to add and remove bcache when needed for a existing dev/fs."

I believe if you've formatted your device ones for bcache you can still always add/remove a cache device. In other words, it should run just fine without any cache device.

A bcache update

dev — Sun, 27 May 2012 18:22:36 +0000

I guess it's dangerous until it supports barriers and can't be used for databases. Even standard FS can get corrupted on power failure...

A bcache update

ViralMehta — Thu, 24 May 2012 11:58:06 +0000

It looks close to the idea what Windows is doing when we attach USB drive, isn't it ? It uses USB drive's memory as extended RAM....

ZFS on Linux

jospoortvliet — Sun, 20 May 2012 00:25:42 +0000

no s* sherlock... ZFS is what, 5 years older? Yeah, it is probably still ahead in many scenarios...

A bcache update

Spudd86 — Fri, 18 May 2012 13:54:40 +0000

Another solution if you're using btrfs is to just set up a bcache for each btrfs device you want to use (Which is the only way it would work if you're using btrfs internal RAID, I suppose). Then your data is redundant always...

A bcache update

butlerm — Thu, 17 May 2012 06:30:33 +0000

This is why the block layer really ought to support write "threads" with thread specific write barriers. It is trivial to convert a barrier to a cache flush where it is impractical to do something more intelligent, but thread specific barriers are ideal for fast commits of journal entries and the like.

In a typical journalling filesystem, it is usually only the journal entries that need to be synchronously flushed at all. Most writes can be applied independently. If a mounted filesystem had two or more I/O "threads" (or flows), one for journal and synchronous data writes, and one for ordinary data writes, an intelligent lower layer could handle a barrier on one thread by flushing a small amount of journal data, while the other one takes its own sweet time - with a persistent cache, even across a system restart if necessary.

Otherwise, the larger the write cache, the larger the delay when a journal commit comes along. Call it a block layer version of buffer bloat. As with networking, other than making the buffer smaller, the typical solution is to use multiple class or flow based queues. If you don't have flow based queuing, you really don't want much of a buffer at all, because it causes latency to skyrocket.

As a consequence, I don't see how write back caching can help very much here, unless all writes (or at least meta data for all out of order writes) are queued in the cache, so that write ordering is not broken. Am I wrong?

A bcache update

koverstreet — Thu, 17 May 2012 01:29:40 +0000

The problem is that if you haven't been writeback caching _everything_ - i.e. your sequential writes have been bypassing the cache - you still need that cache flush to the backing device.

If a mode where all writes were writeback cached and cache flushes were never sent to the backing device would be useful, it'd be trivial to add.

A bcache update

koverstreet — Thu, 17 May 2012 01:27:53 +0000

I would like to see something to back that up.

A bcache update

butlerm — Wed, 16 May 2012 18:14:09 +0000

Next question. Considering that there is a write back cache on persistent media, does that mean when the block layer issues a write cache flush (force unit access) command, that bcache synchronously forces the blocks to SSD, but _not_ synchronously to the backing store device?

Assuming the cache is reliable and persistent, that is exactly what you want in most situations. If a higher level cache flush command actually forces a cache flush of a persistent write back cache, write back mode will be approximately useless. You might as well just leave it in RAM.

Similar/related project

aorth — Wed, 16 May 2012 10:42:07 +0000

Ok, it's a bit over my head, but I'd imagine there must be some benefit to RapidCache. Surely the developer had a legitimate reason to warrant developing it?

ZFS on Linux

abacus — Wed, 16 May 2012 09:02:58 +0000

Has someone already published a comparison of the in-kernel ZFS on Linux and BTRFS ? Some people claim that ZFS is better.

A bcache update

intgr — Wed, 16 May 2012 07:35:38 +0000

> Well, you want the sequential writes to bypass the cache, which is what bcache does

Oh, I didn't realize that. That should take care of most of it. There's probably still some benefit to sorting writeback by size, but I'm not sure whether it's worth the complexity.

Similar/related project

Rudd-O — Wed, 16 May 2012 06:01:00 +0000

RapidCache does exactly what the LRU block cache already does. There is no gain in fixing and setting aside an amount of RAM to do RapidCache.

A bcache update

Rudd-O — Wed, 16 May 2012 05:59:28 +0000

Seriously,

Compared to the ZIL and the L2ARC (both in ZFS and available now in-kernel for Linux), bcache is not even *remotely* close.

A bcache update

koverstreet — Wed, 16 May 2012 01:52:22 +0000

Well, you want the sequential writes to bypass the cache, which is what bcache does.

If you can show me a workload that'd benefit though - I'm not opposed to the idea, it's just a question of priorities.

A bcache update

intgr — Tue, 15 May 2012 22:34:43 +0000

> But for that to be useful we'd have to have writeback specifically pick bigger sequential chunks of dirty data and skip smaller ones, and I'm not sure how useful that actually is

That's certainly an important optimization for mixed random/sequential write workloads whose working set is larger than the SSD. To make best use of both kinds of disks, random writes should persist on the SSD as long as possible, whereas longer sequential writes should be pushed out quickly to make more room for random writes.

A bcache update

dwmw2 — Tue, 15 May 2012 19:13:25 +0000

"SSDs are persistent storage and the data is still in the SSD, so why can't this be persistent? The only way this could be a problem is if the mapping is in memory and not written out to the SSD ever."

Or if your SSD is like almost all SSDs ever seen, where the internal translation is a "black box" which you can't trust, which is known for its unreliability especially in the face of unexpected power failure, and which you can't debug or diagnose when it goes wrong.

Similar/related project

aorth — Tue, 15 May 2012 16:35:12 +0000

Cool! I like the idea of using faster, less-reliable storage as a layer above slow, more-reliable storage. On that note, another interesting read/write through cache is RapidDisk's RapidCache. Check it out:

http://rapiddisk.org/index.php?title=RapidCache

RapidCache uses RAM, so you can't cache AS MUCH, but it's probably faster... probably some scenarios where you'd want to use one or the other depending on your data set, applications, etc.

A bcache update

arekm — Tue, 15 May 2012 12:29:58 +0000

"That device is formatted with a special marker, and its leading blocks are hidden when accessing the device by way of bcache."

This would mean that I can't add ssd bcache to existing device/filesystem, right? Would be nice to be able to add and remove bcache when needed for a existing dev/fs.

A bcache update

Lennie — Tue, 15 May 2012 09:05:56 +0000

Did you see my other comment ?

About how bcache will support more than one SSD in the future and how it will save 2 copies of your precious data on different SSDs instead of one:

http://lwn.net/Articles/497126/

A bcache update

Tobu — Tue, 15 May 2012 07:56:32 +0000

I think the SSD will be a single point of failure in writeback mode, because the underlying filesystem would have megabytes of metadata or journal blocks not written in the right order, which is bad corruption. I don't know how SSDs tend to fail; if they fail into a read-only the writes would still be recoverable in this case, as long as bcache can replay from a read-only SSD. Maybe a filesystem that handles SSD caching itself could avoid that risk.

A bcache update

koverstreet — Tue, 15 May 2012 07:13:04 +0000

Yep. I took it out of the wiki, no need anymore.

A bcache update

alonz — Tue, 15 May 2012 04:16:30 +0000

Do you mean the advice to mount filesystems with “-o nobarrier” is outdated?

A bcache update

jzbiciak — Tue, 15 May 2012 04:15:50 +0000

That reads like poorly translated Japanese or Chinese documentation...

A bcache update

koverstreet — Tue, 15 May 2012 03:43:36 +0000

For that we wouldn't really need tight integration - you could conceivably just tell bcache the stripe size of the backing device, and it wouldn't consider partial stripes sequential with full stripes.

But for that to be useful we'd have to have writeback specifically pick bigger sequential chunks of dirty data and skip smaller ones, and I'm not sure how useful that actually is. Right now it just flushes dirty data in sorted order, which makes things really easy and works quite well in practice - in particular for regular hard drives, even if your dirty data isn't purely sequential you're still minimizing seek distance. And if you let gigabytes of dirty data buffer up (via writeback_percent) - the dirty data's going to be about as sequential as it's gonna get.

But yeah, there's all kinds of interesting tricks we could do.

A bcache update

omar — Tue, 15 May 2012 02:42:28 +0000

Are there any plans to turn this into a module which can be used by something like a RHEL 2.6.32 kernel?

A bcache update

dlang — Tue, 15 May 2012 00:55:20 +0000

>> It would also seem that the best way to run such a cache would be with a relatively tight integration with the raid layer, such that backing store devices are marked with metadata indicating the association with the cache device, so that the group of them can properly be reassembled after a hardware failure, possibly on a different system. The raid layer could then potentially use the cache as a write intent log as well, which is a big deal for raid 5/6 setups.

>I think if you're using bcache for writeback caching on top of your raid5/6,
things will work out pretty well without needing any tight integration;
bcache's writeback tries hard to gather up big sequential IOs, and raid5/6 will handle those just fine.

Where tight integration would be nice would be if bcache can align the writeback to the stripe size and alignment the same way that it tries to align the SSD writes to the eraseblock size and alignment.

A bcache update

koverstreet — Mon, 14 May 2012 23:56:29 +0000

> Another interesting problem that comes with writeback caching is the
> implementation of barrier operations. Filesystems use barriers (implemented as
> synchronous "force to media" operations in contemporary kernels) to ensure that
> the on-disk filesystem structure is consistent at all times. If bcache does not
> recognize and implement those barriers, it runs the risk of wrecking the
> filesystem's careful ordering of operations and corrupting things on the
> backing device. Unfortunately, bcache does indeed lack such support at the
> moment, leading to a strong recommendation to mount filesystems with barriers
> disabled for now.

I need to delete that from the wiki, it's out of date - it's only old style
(pre 2.6.38) barriers that were a problem, because they implied much stricter
ordering. Cache flushes/FUA are much easier, and are handled just fine.

A bcache update

koverstreet — Mon, 14 May 2012 23:54:21 +0000

> My number one question is does bcache preserve the (safely written) contents
> of the cache across restarts? That would be a major advantage in avoiding cache
> warm up time in many applications, particularly large databases.

Yes. That's been priority #1 from the start - it's particularly critical if you
want to be able to use writeback... at all.

> In addition, clearing out the first block of the backing device doesn't seem
> like such a great idea on simple configurations. If the SSD fails, the
> remaining filesystem would practically be guaranteed to be in a seriously
> inconsistent state upon recovery. I would be concerned about using this in a
> sufficiently critical setup without the ability to mirror the cache devices.

Yeah. For now, you can just use a raid1 of SSDs. Eventually, I'll finish
multiple cache device support - you'll be able to have multiple SSDs in a cache
set, and only dirty data and metadata will be mirrored.

> It would also seem that the best way to run such a cache would be with a
> relatively tight integration with the raid layer, such that backing store
> devices are marked with metadata indicating the association with the cache
> device, so that the group of them can properly be reassembled after a hardware
> failure, possibly on a different system. The raid layer could then potentially
> use the cache as a write intent log as well, which is a big deal for raid 5/6
> setups.

I think if you're using bcache for writeback caching on top of your raid5/6,
things will work out pretty well without needing any tight integration;
bcache's writeback tries hard to gather up big sequential IOs, and raid5/6 will
handle those just fine.

> If you want write performance improvements, the ideal thing to do is not to
> synchronously flush dirty data in the cache to the backing store on receipt of
> a write barrier, but rather to journalize all the writes themselves on the
> cache device, so that they can be applied in the correct order on system
> recovery. Otherwise you get a milder version of problem where synchronous
> writes are held up by asynchronous ones, which can dramatically affect the time
> required for an fsync operation, a database commit, and other similar
> operations, if there are other asynchronous writes pending.

Yes. But all that's really required for that is writeback caching (things get
complicated if you only want to use writeback caching for random writes, but
I'll handwave that away for now).

> On the other hand, a device like this could make it possible for the raid /
> lvm layers to handle write barriers properly, i.e. with context specific
> barriers, that only apply to one logical volume, or subset of the writes from a
> specific filesystem, without running the risk of serious inconsistency problems
> due to the normal inability to write synchronously to a group of backing store
> devices. If you have a reliable, low overhead write intent log, you can avoid
> that problem, and do it right, degrading to flush the world mode when a
> persistent intent log is not available.

Well, old style write barriers have gone away - now there's just cache flushes
(and FUA), which are much saner to handle.

A bcache update

russell — Mon, 14 May 2012 23:28:29 +0000

could it be that the SSD is a single point of failure in front of a redundant set of disks. So writing to the SSD is probably no better than keeping it in RAM. Power supply failure vs SSD failure.

A bcache update

butlerm — Mon, 14 May 2012 23:25:41 +0000

My number one question is does bcache preserve the (safely written) contents of the cache across restarts? That would be a major advantage in avoiding cache warm up time in many applications, particularly large databases.

In addition, clearing out the first block of the backing device doesn't seem like such a great idea on simple configurations. If the SSD fails, the remaining filesystem would practically be guaranteed to be in a seriously inconsistent state upon recovery. I would be concerned about using this in a sufficiently critical setup without the ability to mirror the cache devices.

It would also seem that the best way to run such a cache would be with a relatively tight integration with the raid layer, such that backing store devices are marked with metadata indicating the association with the cache device, so that the group of them can properly be reassembled after a hardware failure, possibly on a different system. The raid layer could then potentially use the cache as a write intent log as well, which is a big deal for raid 5/6 setups.

If you want write performance improvements, the ideal thing to do is not to synchronously flush dirty data in the cache to the backing store on receipt of a write barrier, but rather to journalize all the writes themselves on the cache device, so that they can be applied in the correct order on system recovery. Otherwise you get a milder version of problem where synchronous writes are held up by asynchronous ones, which can dramatically affect the time required for an fsync operation, a database commit, and other similar operations, if there are other asynchronous writes pending.

On the other hand, a device like this could make it possible for the raid / lvm layers to handle write barriers properly, i.e. with context specific barriers, that only apply to one logical volume, or subset of the writes from a specific filesystem, without running the risk of serious inconsistency problems due to the normal inability to write synchronously to a group of backing store devices. If you have a reliable, low overhead write intent log, you can avoid that problem, and do it right, degrading to flush the world mode when a persistent intent log is not available.

A bcache update

ballombe — Mon, 14 May 2012 22:40:22 +0000

why such a negative style, Jon ? You could just as well have written

> Such a device could simultaneously increase the performance ecstasy that comes with solid-state storage and the overwhelming financial joy associated with rotating storage.

(just kidding.)

A bcache update

alankila — Mon, 14 May 2012 21:37:48 +0000

Hmh. Looks like garbage comparison. They use XFS, which was changed a lot between 2.6.32 (which is their flashcache kernel) and 3.1 (for bcache).

A bcache update

Beolach — Mon, 14 May 2012 21:18:43 +0000

The bcache wiki has a Performance section, which includes a link to just such a comparison.

A bcache update

alankila — Mon, 14 May 2012 20:51:05 +0000

There's also the facebook flashcache which does something similar. I've been running it experimentally as a writeback cache on a test server and so far it seems to do what it promises. It can be experimented with today, and contains only about 6400 lines of code by a quick count, and can be insmod'd into kernel rather than patched in.

Maybe it's somehow critically far worse, but then again, perhaps it's true that perfect is the enemy of good. We'll probably know when we can benchmark these technologies against each other.

bcache cache-sets

Lennie — Mon, 14 May 2012 20:49:06 +0000

Also as briefly mentioned in the older article: bcache has cache-sets, you can assign several SSDs to a backing store.

Eventually bcache is supposed to get support for mirroring the dirty data, so your dirty data will be stored on two SSDs before it is written to your already redundant backing store (like a RAID1 of HDDs).

Any data that is only a cached copy of data already written to the backing store will only be stored ones on one of the SSDs.

When that has been added that should take away most concerns people might have about their data.

A bcache update

corbet — Mon, 14 May 2012 20:10:01 +0000

Yes, exactly...as I tried to explain in that same paragraph. The data does exist on SSD, but it's only useful if the post-meteorite kernel knows what data is there. So the index and such have to be saved to the SSD along with the data.

A bcache update

blitzkrieg3 — Mon, 14 May 2012 20:07:30 +0000

> Obviously, writeback caching also carries the risk of losing data if the system is struck by a meteorite before the writeback operation is complete.

It isn't clear to me why this is true. SSDs are persistent storage and the data is still in the SSD, so why can't this be persistent? The only way this could be a problem is if the mapping is in memory and not written out to the SSD ever.