An update on bcachefs

If you use LVM, you can have an SSD-based PV and an HDD-based PV and then form a cached LV based on a small chunk of the SSD and a big chunk of the HDD. The terminology in LVM is to create a cached LV using a backing LV and a cache pool.

You can enjoy the usual benefits of LVM to allocate portions of the SSD or HDD for different purposes, migrating LV content between backing devices, etc. You might make an SSD-only filesystem as well as an SSD-cached filesystem with a larger HDD. You can also drop and add cache on an existing LV without having to reformat the backing filesystem, in case you change your mind about how much SSD to allocate for caching.

I think the cache management commands can be done while a filesystem in the LV is online, but I actually haven't done that yet. I've set up a write-back cached volume using 60GB of SSD for 2TB of nearline disk. My system RAM provides enough buffer cache for reads, so this is mostly to give me a durable write buffer so my programs don't wait for the HDD to complete writes as often.

I'm a little old-fashioned, so I do my RAID mirroring via mdraid and treat those as PVs for LVM. So, I have an SSD mirror backing some small filesystems and cache pool, and an HDD mirror backing bulk filesystems with some cached and some uncached.