LWN: Comments on "Bcache: Caching beyond just RAM"

Generation number

wiza — Fri, 17 May 2013 08:55:25 +0000

I don't quite understand the effect of maintaining the generation number. Since FTL has used some algorithm to avoid wearing out, what good is the generation number?

Ramdisks and hard drives as cache devices

nix — Thu, 22 Jul 2010 20:59:00 +0000

Unfortunately if you want to spot cache invalidations, the lack of leases on NFSv3 is a killer, because you still have to roundtrip to the server to see if your cache is stale, and roundtrips are the slow part :( fs-cache slows NFS *down* quite considerably in my experience, for exactly this reason.

Ramdisks and hard drives as cache devices

neilbrown — Tue, 20 Jul 2010 11:38:56 +0000

Yes, NFSv3 helps with writing large files. But metadata operations (create, unlink, chmod, mv) still need to be synchronous and some workloads can be very metadata-heavy (untar is a good example, 'make' on a big project tends to delete and create lots of relatively small files too).
So a low-latency cache can definitely improve the performance of an NFS server.

Bcache: Caching beyond just RAM

intgr — Sat, 17 Jul 2010 16:10:36 +0000

How are you planning to protect file systems from being mounted if it has a write-back cache that is not activated or not available? Sounds like there is no infrastructure for this currently, but could badly corrupt data if not recognized.

Bcache: Caching beyond just RAM

bill_mcgonigle — Sat, 17 Jul 2010 03:55:10 +0000

I was wondering the same thing. As I understand it, the main difference is that with Flashcache you tell it, "here, you're in front of this device". And then you mount the flashcache device.

bcache uses a kernel feature that lets it hook in 'from the side'. So, you still mount the regular device. In theory you can insert/pull/insert bcache as much as you want and your apps would be fine to keep on running. I think that makes it ultimately more powerful.

You could even imagine a workload where you have a bazillion disks that receive sporadic heavy access (maybe remote disks where old data isn't useful), and you could have some sort of monitor that would keep a few SSD's busy by inserting them as bcaches on-demand.

Bcache: Caching beyond just RAM

bill_mcgonigle — Thu, 15 Jul 2010 22:59:54 +0000

There's quite a bit of similarity. L2ARC is read-only, for wide random accesses. bcache aims to be read/write. ZFS also separates out its ZIL for writes. If I'm building a ZFS box I'd use a big MLC drive for the L2ARC and a smaller SLC drive for the ZIL as the workloads differ. You could probably set bcache to be read-only and put a filesystem's journal on a different drive if you wanted a more-like-ZFS segregation. bcache has the nice attribute of just being able to pull it and keep running - ZFS isn't usually set up that way. The L2ARC and ZIL have the checksumming today whereas bcache will get to that. Of course, bcache is much more general and useful in situations where ZFS has no relevance. It's good to have chisels and screwdrivers.

Bcache: Caching beyond just RAM

rilder — Mon, 12 Jul 2010 20:30:01 +0000

I recently came across L2ARC caching used in Solaris with ZFS + SSD. Is there any improvement or difference between that and bcache ?

Bcache: Caching beyond just RAM

rilder — Mon, 12 Jul 2010 19:50:04 +0000

Interesting project. However, I wonder how this will work with something like cleancache(http://lwn.net/Articles/389873/). Cleancache works in the realm of VM. So is bcache supposed to complement cleancache or is a replacement for it.

The transparency of bcache is quite nice. Is bcache available as module(so that it can be built out of tree and used) ?

Bcache: Caching beyond just RAM

etienne — Mon, 12 Jul 2010 09:16:41 +0000

> there is a feature in the firmware of the drives which auto-parks the heads after 8 seconds of inactivity

Probably maximum power saving more, you can usually see/adjust (and save the setup) using hdparm.

Bcache: Caching beyond just RAM

MisterIO — Mon, 12 Jul 2010 00:14:08 +0000

Well, with kernel bugs I meant kernel bugs not affecting this code and which by no means start writing random songs of some unknown guy(yeah, sorry, I don't know him) over your data, but still freeze the system. At that point basically the only real problem you could have is losing data on your hd(either just the data that didn't make it to the hd or the whole file you were writing at the time of the failure. The second case happened to me frequently with ext4 and I lost many normal data files because of it, while I was able instead to save some sources I was coding simply because I used git and I committed quite often).

Bcache: Caching beyond just RAM

alankila — Sun, 11 Jul 2010 14:39:50 +0000

Excellent news. I look forwards to using this technology.

bcache and cluster filesystems?

koverstreet — Sun, 11 Jul 2010 03:22:46 +0000

Yeah, nothing more complicated than that. Make a new bio flag to invalidate a region of cache, and make sure everything that isn't a cache ignores it. Making bcache handle it would be trivial, it'd just be a matter of making sure ocfs2 always invalidates what it needs to.

Bcache: Caching beyond just RAM

koverstreet — Sun, 11 Jul 2010 03:20:03 +0000

Yeah, that's all correct :)

Kernel bugs are a different matter entirely though - if a program doesn't do what you think it does, it could be doing anything. If it's buggy it could be overwriting all your important data with Rick Astley songs, or opening the door the the velociraptor cages. There's just no way to tell at that point.

You test everything as best you can, but software's complicated, there's always something lurking and no complete guarantees.

O_DIRECT read of sector 0

wstearns — Sat, 10 Jul 2010 21:13:53 +0000

You're correct; bcache does cache block devices. I phrased it poorly; the idea I failed to get across was that the filesystems we'd normally submit to bcache to cache are on individual partitions (which linux treats as block devices) as opposed to the entire drive, allowing you to bypass bcache on sector 0 and still cache the partitions.

Bcache: Caching beyond just RAM

dlang — Sat, 10 Jul 2010 20:32:45 +0000

if the data isn't written to the bcache HD and you do not have battery backed cache on the system, the data will be lost. This is the same as with any storage/filesystem.

however (once the write behind features are in bcache) once the data is written to the bcache storage it should push those changes back to the real device after the next boot. If it doesn't the write-behind feature can't be considered done and useable ;-)

Bcache: Caching beyond just RAM

MisterIO — Sat, 10 Jul 2010 16:18:16 +0000

Ooouuuh, it's so cold being alone in the shadow on winter!

Was my previous question very retarded? I noticed it's the only comment in the thread which was completely ignored. So I thought that knowing to have said something really stupid is still more useful than nothing. Thus, this post was born.

bcache and cluster filesystems?

skissane — Sat, 10 Jul 2010 11:20:46 +0000

it would be interesting to work out, what extra features would bcache need to provide so that a clustered filesystem could use it without issue... i assume just some API to tell bcache to invalidate some block...

O_DIRECT read of sector 0

eludias — Sat, 10 Jul 2010 10:45:52 +0000

I was under the impression that bcache cached block devices (hence the name ), and not filesystems. But you have a good point: if I would cache the partition and not the whole device, I might be able to circumvent it by reading outside the partition.

Bcache: Caching beyond just RAM

koverstreet — Sat, 10 Jul 2010 08:59:59 +0000

It is. Currently if you shut down uncleanly it'll invalidate the cache as it'll be inconsistent, but handling that is part of safe write behind caching on, which I'm working on now.

Bcache: Caching beyond just RAM

giraffedata — Sat, 10 Jul 2010 03:20:12 +0000

Is it persistent across reboots? I.e. will my frequently accessed data still be available from the SSD after a reboot?

O_DIRECT read of sector 0

wstearns — Fri, 09 Jul 2010 15:37:35 +0000

Since bcache can only cache individual filesystems and I assume your actual filesystems are on partitions (as opposed to a single filesystem covering the entire drive), bcache won't be caching sector 0 as that's not part of a filesystem.

Bcache: Caching beyond just RAM

eludias — Fri, 09 Jul 2010 12:27:39 +0000

How will O_DIRECT be handled? I need this to circumvent any caching layer to keep my hardware healthy.

Background: I've also got one of those nasty WD Caviar Green (1Tb) drives, and there is a feature in the firmware of the drives which auto-parks the heads after 8 seconds of inactivity. This interacts quite badly with an OS which saves its data once per 30 seconds, resulting in a drive worn down in about 6 months.

Now the way to circumvent this behaviour is to read something from the drive, say once per 5 seconds. And the most efficient way to do so is to read from the cache of the drive, so we re-read sector 0 over and over again. With O_DIRECT we can bypass the disk cache of Linux traditionally, but will this also be the case when using bcache?

Bcache: Caching beyond just RAM

MisterIO — Fri, 09 Jul 2010 08:03:22 +0000

Is it safe in the presence of kernel bugs(or similar problems) for writes? I mean, we've seen things link ext4 delaying writes for 30 secs, but that had obvious relayability problems. Let's say that bcache successfully caches a certain amount of writes(that is they are complete on the cache), but then the system freezes, so the cache isn't written back to the actual hd. At the next boot, does bcache remember to perform those writes?

Bcache: Caching beyond just RAM

koverstreet — Fri, 09 Jul 2010 03:52:31 +0000

Completely different design and completely unrelated. I started bcache a couple months before flashcache was announced. Flashcache is simpler and being used in production now. Bcache has, in my opinion, a lot more potential than flashcache.

Bcache: Caching beyond just RAM

mwalls — Fri, 09 Jul 2010 03:36:02 +0000

Isn't this substantially duplicating Facebook's flashcache, so are they related?

bcache and cluster filesystems?

koverstreet — Fri, 09 Jul 2010 03:28:34 +0000

Yeah, it'd quickly explode. Probably be fun to watch, though.

bcache and cluster filesystems?

skissane — Fri, 09 Jul 2010 02:56:25 +0000

How would this interact with a cluster file system such as OCFS?
I assume it wouldn't work, unless the cluster file system was aware of the
bcache, and had some way to do cache invalidation of the blocks which had
been changed by other nodes in the cluster....

Bcache and filesystem recovery

koverstreet — Fri, 09 Jul 2010 02:15:24 +0000

It doesn't handle barriers just yet. That's coming soon, it's needed for write behind caching.

This isn't a correctness issue, if you don't have barriers you just have to wait for writes to complete before you start another to guarantee ordering. Barriers have historically taken awhile to add to things like raid, lvm and drbd, bcache is nothing special.

There is a bug in that if you're running bcache on something that supports barriers, bcache won't indicated that they're no longer supported... I haven't worried much about that since you can disable them in the filesystem and it's only till I get barriers implemented in bcache.

Bcache: Caching beyond just RAM

koverstreet — Fri, 09 Jul 2010 01:21:26 +0000

You could stuff everything into a filesystem, sure, but what would be the point? Caching really is doing something else, it wouldn't really be able to share any filesystem code, and it'd be tied to that one filesystem.

The main thing is the allocation strategy you want for caching is _completely_ different than for filesystems. Fragmentation isn't a real problem in the cache, since we can free fixed sized chunks regardless of what's in them. This means we're free to write data to the cache however we want to get the best performance. A filesystem has to retain data for an arbitrary amount of time, and thus needs to pay a lot of attention to making sure free space doesn't fragment too much.

Putting an external journal on an SSD gets you a bit of what bcache is after, but it'll only help with writes, and not to the extent bcache can. How would you effectively use an 80 gb journal? With bcache, you'll be able to fill up your caches with almost all dirty writes, and then write them out to your RAID6 with no restrictions on ordering - potentially turning a huge portion of random writes into mostly sequential ones, and even more will get overwritten in the cache before the raid ever sees them.

Bcache: Caching beyond just RAM

akumria — Fri, 09 Jul 2010 00:08:54 +0000

Perhaps I'm not understanding everything.

But it seems to me, that you could move the bcache functionality into the filesystem.

i.e. Why can't btrfs store all writes on that faster medium and them replicate them back to the slower one?

Also, isn't this akin to storing the journal of a filesystem externally (I've never done it). i.e. Why not point the ext3/ext4 journal at the SSD?

Would you get the same (or better) benefit?

It seem bcache is useful is the underlying filesystem doesn't do it, but in the above two cases, is there much benefit?

Insight appreciated.

Thanks,
Anand

Ramdisks and hard drives as cache devices

koverstreet — Thu, 08 Jul 2010 23:32:25 +0000

> (Reliable, crash resistant)

That's what I've been working towards :)

Safe write behind caching really is a large step up in terms of the guarantees the cache has to be able to make - with writethrough caching, you just have to make sure you return good data, or nothing. And btree code is hard enough, so I've been working on getting the easy cases rock solid first - but safe write behind caching is absolutely happening. I've got some of the preliminary stuff out of the way and I'm hoping to have a rough initial version before too long, depending on how debugging the new stuff goes.

Ramdisks and hard drives as cache devices

jlayton — Thu, 08 Jul 2010 23:26:42 +0000

> NFS servers are supposed to do synchronous writes, right? (Reliable, crash resistant) write behind caching would be outstanding.

Not since NFSv3 was implemented. That gave the ability to do safe asynchronous writes. The client sends a bunch of WRITEs to the server and then issues a COMMIT. If the server crashes, the client will reissue uncommitted writes.

With NFSv2 however, you're correct that the server is supposed to do sync writes (and most do these days, at least by default).

Ramdisks and hard drives as cache devices

koverstreet — Thu, 08 Jul 2010 23:19:43 +0000

Bootloaders are a special situation - the most practical thing to do would be to just not enable write behind caching for /boot. For everything else, sync happens when you switch off write behind caching, and sync without switching it off doesn't really make any sense, the cached device is going to be inconsistent as long as write behind caching is on.

Bcache: Caching beyond just RAM

koverstreet — Thu, 08 Jul 2010 23:16:01 +0000

Checksumming won't save you here - what's it supposed to compared the checksum against? If you fill up your cache, turn of caching, write some data, and then turn caching back on without telling bcache about it... well, computers can't always save you from rm -rf either.

I am looking into invalidating the cache contents if a filesystem was already opened read write, or perhaps checking if the first page or so matches what it previously had - this would catch it provided the filesystem superblock changed. But the real performance gains are to be had with write behind caching, and none of this really matters there, since the cached device is now inconsistent if you use it without the cache. By the time bcache saw the cache was out of date it'd be too late to do anything.

Bcache: Caching beyond just RAM

sflintham — Thu, 08 Jul 2010 17:34:52 +0000

This approach of intercepting bio requests in the background allows us to start and stop caching on the fly, to add and remove cache devices, and to boot with or without bcache

Am I right in thinking this will only be possible once the checksumming feature mentioned at the bottom of the article is in place? Otherwise, what would stop the cache returning an out-of-date copy of a block modified while it was offline?

Bcache and filesystem recovery

abacus — Thu, 08 Jul 2010 16:54:50 +0000

An important issue has not been discussed in the above article, and that is how file system recovery works when using Bcache. Does Bcache honor the write ordering requirements communicated by filesystems such that e.g. journal-based filesystem recovery still works ?

Unique UUIDs

wstearns — Thu, 08 Jul 2010 16:19:10 +0000

You're correct; at least in the case of RAID 1 mdadm, the md device and the underlying partitions all share the exact same UUID. When I use bcache on a raid array, I don't cache the underlying partitions, I cache the raid device itself. That lets the raid code handle a missing component partition, raid resync, and any other raid issues internally.
If you copy a filesystem with dd or raid split as you mention, you still have to tell bcache both what UUID and what block device to cache, letting you specify the original partition or the new copy, whichever you want cached. So bcache shouldn't introduce new problems; a non-bcache linux system will have problems already if you have 2 different filesystems with the same UUID as others have mentioned.

Bcache: Caching beyond just RAM

martinfick — Thu, 08 Jul 2010 15:40:35 +0000

Why would you want it in front of a DRBD device? If you really want to sacrifice reliability for performance with a DRBD device, simply use protocol B or C, it will likely be more reliable.

Bcache: Caching beyond just RAM

faramir — Thu, 08 Jul 2010 14:57:49 +0000

Given that many distributions now use UUIDs rather then device names in /etc/fstab, not having unique UUIDs is already a recipe for disaster.

Ramdisks and hard drives as cache devices

etienne — Thu, 08 Jul 2010 14:57:08 +0000

> I imagine that Bcache caches O_DIRECT reads and writes, which could give a caching advantage even where an application has elected to bypass the ordinary buffer cache.

An application like a boot-loader installation/upgrade which tries to tell which disk sectors to load from the MBR?
I hope there is a sync() to write to (the real destination) disk.