Bcache: Caching beyond just RAM [LWN.net]

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 3:09 UTC (Thu) by NightMonkey (subscriber, #23051)
Parent article: Bcache: Caching beyond just RAM

Could this be used to make caches out of magnetic platter drives, or tmpfs, or any block device?

to post comments

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 3:23 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link] (5 responses)

Aside from tmpfs not being a block device, yes. I don't know that using hard drives as cache would ever be worthwhile - it's written with the assumption that random reads are fast, if you were to try that you'd want it to try to have it cache decent sized chunks of contiguous data (on cache miss, always read 64 kb or so in at a time). But it would certainly work.

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 3:35 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

a local magnetic drive may be faster than a network drive

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 11:13 UTC (Thu) by ewan (guest, #5533) [Link] (1 responses)

And a small fast disk system (e.g. striped 15k SAS disks) may be a useful cache over a big slow one (e.g. a huge RAID of 5400rpm SATA disks).

I wonder if bcache can be stacked? So you could have an SSD over some SAS disks over a SATA RAID?

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 11:24 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

There's no way to do that now - all the cache devices in the system are pooled together and used symmetrically - but it is something I've been considering down the road. It might not be all that much more work, I'm just curious if the performance would justify it. Most people's working set just isn't all that huge, and also the big problem with your big slow raid6 is random writes, which bcache should be able to mostly eliminate. But we'll see.

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 3:42 UTC (Thu) by NightMonkey (subscriber, #23051) [Link] (1 responses)

I see. It might be interesting to use with a hard drive (or RAID 0 set) as a write-behind cache, once that is implemented. Or perhaps in front of a high-latency DRBD device. Some interesting possibilities! :)

Bcache: Caching beyond just RAM

Posted Jul 8, 2010 15:40 UTC (Thu) by martinfick (subscriber, #4455) [Link]

Why would you want it in front of a DRBD device? If you really want to sacrifice reliability for performance with a DRBD device, simply use protocol B or C, it will likely be more reliable.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 3:43 UTC (Thu) by wstearns (guest, #4102) [Link] (7 responses)

To add to Kent's answer, you could use a traditional (non-tmpfs) ramdisk as it's a full block device. In a sense you're overriding the cache strategy by devoting the ramdisk's memory to caching one or more performance critical filesystems. It might be useful in some situations even though the system as a whole has less efficient use of memory. Using a hard drive as cache might actually be useful if the block device/filesystem you're caching is over a slow link or significantly slower than the cache device. If your local drive hard drive is at least as large as the remote drive, you also have the option of using raid 1 and setting the remote drive to "write-mostly" (so that all reads come from the local drive except in case of drive failure; see "man mdadm").

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 14:20 UTC (Thu) by butlerm (subscriber, #13312) [Link] (6 responses)

"In a sense you're overriding the cache strategy by devoting the ramdisk's memory to caching one or more performance critical filesystems."

I imagine that Bcache caches O_DIRECT reads and writes, which could give a caching advantage even where an application has elected to bypass the ordinary buffer cache.

Bcache seems ideal for use in NFS servers, and iSCSI and FCoE targets. NFS servers are supposed to do synchronous writes, right? (Reliable, crash resistant) write behind caching would be outstanding.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 14:57 UTC (Thu) by etienne (guest, #25256) [Link] (1 responses)

> I imagine that Bcache caches O_DIRECT reads and writes, which could give a caching advantage even where an application has elected to bypass the ordinary buffer cache.

An application like a boot-loader installation/upgrade which tries to tell which disk sectors to load from the MBR?
I hope there is a sync() to write to (the real destination) disk.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:19 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

Bootloaders are a special situation - the most practical thing to do would be to just not enable write behind caching for /boot. For everything else, sync happens when you switch off write behind caching, and sync without switching it off doesn't really make any sense, the cached device is going to be inconsistent as long as write behind caching is on.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:26 UTC (Thu) by jlayton (subscriber, #31672) [Link] (2 responses)

> NFS servers are supposed to do synchronous writes, right? (Reliable, crash resistant) write behind caching would be outstanding.

Not since NFSv3 was implemented. That gave the ability to do safe asynchronous writes. The client sends a bunch of WRITEs to the server and then issues a COMMIT. If the server crashes, the client will reissue uncommitted writes.

With NFSv2 however, you're correct that the server is supposed to do sync writes (and most do these days, at least by default).

Ramdisks and hard drives as cache devices

Posted Jul 20, 2010 11:38 UTC (Tue) by neilbrown (subscriber, #359) [Link] (1 responses)

Yes, NFSv3 helps with writing large files. But metadata operations (create, unlink, chmod, mv) still need to be synchronous and some workloads can be very metadata-heavy (untar is a good example, 'make' on a big project tends to delete and create lots of relatively small files too).
So a low-latency cache can definitely improve the performance of an NFS server.

Ramdisks and hard drives as cache devices

Posted Jul 22, 2010 20:59 UTC (Thu) by nix (subscriber, #2304) [Link]

Unfortunately if you want to spot cache invalidations, the lack of leases on NFSv3 is a killer, because you still have to roundtrip to the server to see if your cache is stale, and roundtrips are the slow part :( fs-cache slows NFS *down* quite considerably in my experience, for exactly this reason.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:32 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

> (Reliable, crash resistant)

That's what I've been working towards :)

Safe write behind caching really is a large step up in terms of the guarantees the cache has to be able to make - with writethrough caching, you just have to make sure you return good data, or nothing. And btree code is hard enough, so I've been working on getting the easy cases rock solid first - but safe write behind caching is absolutely happening. I've got some of the preliminary stuff out of the way and I'm hoping to have a rough initial version before too long, depending on how debugging the new stuff goes.