Ramdisks and hard drives as cache devices [LWN.net]

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 3:43 UTC (Thu) by wstearns (guest, #4102)
In reply to: Bcache: Caching beyond just RAM by NightMonkey
Parent article: Bcache: Caching beyond just RAM

To add to Kent's answer, you could use a traditional (non-tmpfs) ramdisk as it's a full block device. In a sense you're overriding the cache strategy by devoting the ramdisk's memory to caching one or more performance critical filesystems. It might be useful in some situations even though the system as a whole has less efficient use of memory. Using a hard drive as cache might actually be useful if the block device/filesystem you're caching is over a slow link or significantly slower than the cache device. If your local drive hard drive is at least as large as the remote drive, you also have the option of using raid 1 and setting the remote drive to "write-mostly" (so that all reads come from the local drive except in case of drive failure; see "man mdadm").

to post comments

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 14:20 UTC (Thu) by butlerm (subscriber, #13312) [Link] (6 responses)

"In a sense you're overriding the cache strategy by devoting the ramdisk's memory to caching one or more performance critical filesystems."

I imagine that Bcache caches O_DIRECT reads and writes, which could give a caching advantage even where an application has elected to bypass the ordinary buffer cache.

Bcache seems ideal for use in NFS servers, and iSCSI and FCoE targets. NFS servers are supposed to do synchronous writes, right? (Reliable, crash resistant) write behind caching would be outstanding.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 14:57 UTC (Thu) by etienne (guest, #25256) [Link] (1 responses)

> I imagine that Bcache caches O_DIRECT reads and writes, which could give a caching advantage even where an application has elected to bypass the ordinary buffer cache.

An application like a boot-loader installation/upgrade which tries to tell which disk sectors to load from the MBR?
I hope there is a sync() to write to (the real destination) disk.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:19 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

Bootloaders are a special situation - the most practical thing to do would be to just not enable write behind caching for /boot. For everything else, sync happens when you switch off write behind caching, and sync without switching it off doesn't really make any sense, the cached device is going to be inconsistent as long as write behind caching is on.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:26 UTC (Thu) by jlayton (subscriber, #31672) [Link] (2 responses)

> NFS servers are supposed to do synchronous writes, right? (Reliable, crash resistant) write behind caching would be outstanding.

Not since NFSv3 was implemented. That gave the ability to do safe asynchronous writes. The client sends a bunch of WRITEs to the server and then issues a COMMIT. If the server crashes, the client will reissue uncommitted writes.

With NFSv2 however, you're correct that the server is supposed to do sync writes (and most do these days, at least by default).

Ramdisks and hard drives as cache devices

Posted Jul 20, 2010 11:38 UTC (Tue) by neilbrown (subscriber, #359) [Link] (1 responses)

Yes, NFSv3 helps with writing large files. But metadata operations (create, unlink, chmod, mv) still need to be synchronous and some workloads can be very metadata-heavy (untar is a good example, 'make' on a big project tends to delete and create lots of relatively small files too).
So a low-latency cache can definitely improve the performance of an NFS server.

Ramdisks and hard drives as cache devices

Posted Jul 22, 2010 20:59 UTC (Thu) by nix (subscriber, #2304) [Link]

Unfortunately if you want to spot cache invalidations, the lack of leases on NFSv3 is a killer, because you still have to roundtrip to the server to see if your cache is stale, and roundtrips are the slow part :( fs-cache slows NFS *down* quite considerably in my experience, for exactly this reason.

Ramdisks and hard drives as cache devices

Posted Jul 8, 2010 23:32 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

> (Reliable, crash resistant)

That's what I've been working towards :)

Safe write behind caching really is a large step up in terms of the guarantees the cache has to be able to make - with writethrough caching, you just have to make sure you return good data, or nothing. And btree code is hard enough, so I've been working on getting the easy cases rock solid first - but safe write behind caching is absolutely happening. I've got some of the preliminary stuff out of the way and I'm hoping to have a rough initial version before too long, depending on how debugging the new stuff goes.