COW and volume management? [LWN.net]

COW and volume management?

Posted Jan 27, 2026 11:12 UTC (Tue) by koverstreet (subscriber, #4296)
In reply to: COW and volume management? by taladar
Parent article: Filesystem medley: EROFS, NTFS, and XFS

> Plus, on a purely technical level, I am sure ZFS, being the first of this type of file system has many design flaws that could be improved starting from scratch a generation later, much like any technology that tries something new and then gets stuck supporting existing behavior and design decisions as it gains users. Filesystems are probably affected by that more than most given that they deal with data at rest and not just some ephemeral data like e.g. network protocols.

"Flaws" isn't a fair characterization; the approach the ZFS devs took was a pragmatic one, and the only one that had any chance of success given the scope of what they were trying to do.

When you're trying to do something ambitious, you don't shoot for the moon - you can't reimagine everything at once, you can't take on too many big risks, that's how you get stuck in second system syndrome; you'll never complete your task. You have to work incrementally, and pick and choose what you're going to reimagine and what risks you're going to take.

But - that does mean that ZFS is essentially the original Unix filesystem underneath. Blocks, not extents, and data structures that are clearly in that lineage.

bcachefs is that "shoot for the moon and reimagine everything" project. It only happened because I got profoundly lucky with some of the original key design decisions (the b-tree implementation in particular), and for the first five years I wasn't attempting a full filesystem - I had much smaller targets (a block cache, and then something more intermediate) where I was able to successfully complete and ship building blocks for the full filesystem; it wasn't until those were successful that I realized attempting a filesystem was even possible.

So in a way you're right, but don't sell the ZFS folks short - they deserve massive amounts of credit and respect for showing everyone what a modern filesystem should be capable of.

And there's a whole list of things that took years to crack before I had something better:

- Snapshots: snapshots + extents are a difficult combination, Dave Chinner and others were poking me about this for years before I had a working design (I've got notebooks filled with scribbles of different ways of solving the "extents can overlap in arbitrary ways" problem).

- fsck/self healing: bcachefs benefited massively from the hindsight of seeing what other filesystems were doing, and just having a lot more time to get it done. Going straight for full self healing and skipping a conventional fsck does not get you what you want, nor does just starting with a conventional offline fsck. I had experience with proto-runtime-fsck from the early days immediately after bcache, and time to explore a lot of different options; bcachefs takes a mixed approach that's turned out to be very comprehensive, but that took a lot of time to get right.

- The erasure coding design is entirely original; it delivers much better performance than RAIDZ but had many difficult problems to solve over years of development (it all looks simple in hindsight, but it wasn't simple to figure out when it hadn't been done before!).

- The whole "filesystem that really is a database underneath" approach has always been the dream - having that makes my life so much easier now. (Was working on the last bits of hooking up erasure coding to reconcile last night for automatic resilvering when a drive goes away, when I realized the on disk data structures were wrong; when using the reconcile_*_phys btrees for doing pending work in on-disk LBA order on rotating disks, we can't mix that in with the pending work for user data extents; we need to repair stripes first; _then_ user data extents to get anything that wasn't erasure coded, or we'll do a ton of unnecessary work; oh damn, that was a whole 10 lines of code to fix). But no existing b-tree implementation was remotely fast enough, even today - that was the big one that I cracked 15 years ago, with design decisions that look prescient in hindsight, and then benchmarking bcache on early Google SSDs when we realized "oh damn, binary search is actually really slow" and pulling math out of the ether to do magic with eytzinger search trees...

to post comments

COW and volume management?

Posted Jan 27, 2026 12:45 UTC (Tue) by Wol (subscriber, #4433) [Link] (4 responses)

> - The whole "filesystem that really is a database underneath" approach has always been the dream - having that makes my life so much easier now.

Wowser! And that's moving into the mainstream now? Again, massive kudos to you if you've managed to pull it off, but that's been around for ages.

I hesitate to say it was true of Pick - it was sort of - but that was more "the storage system is permanent virtual memory". The AS400 had a database as a filesystem, no? And there were loads of rumours that NTFS was originally going to be a derivative of the Pick database - that was back in the days of NT3.1 whenever that was.

"If I can see so far, it's because I'm standing on the shoulders of giants" - but you're a giant yourself for pulling it off!

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 13:15 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (3 responses)

Yup.

We're not using it to do anything particularly novel yet, just doing normal filesystem stuff but done better and more cleanly, so far. And it's still not quite as general as I want; the main remaining limitation is that keys are fixed size at 160 bits (64 bit inum, 64 bit offset, 32 bit snapshot); key types really need to be per-btree definable, bpos has been increasingly overloaded in different places and it's annoying.

It's also definitely nowhere near SQL, where you define tables and indexes and do joins; it's more of a NoSQL database. But it's got a solid transaction model, locking model (with a full cycle detector, as real databases do), triggers, nice iterators, lots of support for extending the database schema - it's generally quite pleasant to work with.

The challenge with making this stuff mainstream has been performance. Historically, Unix filesystems were completely unlike databases because they needed to be simple and fast - inodes as flat arrays on disk, blocks indexed by radix trees hanging off the inodes. Lots of simple specialized data structures.

Bu conventional filesystems have started to look more and more like databases over the years as they've gotten more complex. Extents mean radix trees don't work anymore, you need btrees, journaling means you need a transaction model, and every additional feature filesystems want to do pushes you more in the direction of "I really just want a real general purpose database for this stuff".

The game changer was when, in the early days of bcache, we pushed the performance of the "btree with big log structured nodes" design to the point where it could support a filesystem with no per-inode sharding - that was when I knew bcachefs was possible.

COW and volume management?

Posted Jan 27, 2026 13:45 UTC (Tue) by Wol (subscriber, #4433) [Link] (2 responses)

> We're not using it to do anything particularly novel yet, just doing normal filesystem stuff but done better and more cleanly, so far. And it's still not quite as general as I want; the main remaining limitation is that keys are fixed size at 160 bits (64 bit inum, 64 bit offset, 32 bit snapshot); key types really need to be per-btree definable, bpos has been increasingly overloaded in different places and it's annoying.

> It's also definitely nowhere near SQL, where you define tables and indexes and do joins; it's more of a NoSQL database. But it's got a solid transaction model, locking model (with a full cycle detector, as real databases do), triggers, nice iterators, lots of support for extending the database schema - it's generally quite pleasant to work with.

Don't confuse SQL and Relational, they're two completely separate things in a marriage :-)

And you say "NoSQL" - do you mean No SQL, or Not Only SQL? :-) (I'm sure you don't mean the database actually called "NoSQL" :-)

But it might be worth taking a look at Pick's Not Only SQL model. I don't understand what you're going about key types and btree especially, but Pick is hash-based not btree based. Investigate Linear/Dynamic hashing. That swaps btree's "all accesses take the same time" for hashing's "most accesses are extremely fast" and the linear/dynamic algorithm makes resizing order 1, not order N. I've been thinking about making ScarletDMEs hash buckets into btrees so in the worst case access degrades to btrees ?order root N? rather than a naive hash's order N. (And speeds up splitting an overflowed bucket, too.)

But if you're looking at "the file system is a database", don't restrict yourself to SQL/Relational. Just because it's taken the universities by storm so that nobody knows anything else, doesn't mean it's the best, or even has any ideas worth nicking :-)

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 15:07 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (1 responses)

SQL may not be the prettiest language, and the relational model may not be what you want in every context - it's too abstracted when you really need to optimize for performance - but there's a reason it's taught, it's a good model, and it's something every programmer should probably know.

Hash tables are no-go in on disk data structures, they destroy locality and you can't do range lookups. Extents require range lookups, along with a lot of other things filesystems need to do, and we very frequently need to do efficient scans - btrees are much better suited to that.

Locality is a prime consideration when designing on disk data structures, and frequently you'll be doing short linear scans - linear searches are not slow as long as you keep them short.

E.g., the way bcachefs does full writeable snapshots, with versioning (so snapshots are at key granularity, not btree node granularity like btrfs) uses the low bits of the key as the snapshot ID - so a lookup finds either a key in the current snapshot, or an ancestor snapshot - that requires range lookups. But then for writeable snapshots, the snapshot ID is not a linear history, it's a tree; so we also have to skip over keys in unrelated branches of the snapshots tree. This search will be short for a variety of reasons (the main one being that creates always get a new inode number that won't be shared by any other snapshots), so snapshot lookups are fast; you can't do these kinds of tricks with hash tables.

COW and volume management?

Posted Jan 27, 2026 21:02 UTC (Tue) by Wol (subscriber, #4433) [Link]

> SQL may not be the prettiest language, and the relational model may not be what you want in every context - it's too abstracted when you really need to optimize for performance - but there's a reason it's taught, it's a good model, and it's something every programmer should probably know.

The relational *model* is great, yes. The engineering is flawed by design :-(

Pick allows you to stick a 4NF view directly into the database as a table row ... that's why an optimiser is a waste of time ... (now that most RDBMSs have arrays and structures, they've finally caught up, 40 years late, but SQL can't query them cleanly)

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 13:59 UTC (Tue) by malmedal (subscriber, #56172) [Link] (2 responses)

> But - that does mean that ZFS is essentially the original Unix filesystem underneath.

what? ZFS is a log-structured filesystem, like e.g. Sprite LFS and NetApp WAFL. Very different tradition from the old modify-in-place unix filesystem.

COW and volume management?

Posted Feb 5, 2026 13:26 UTC (Thu) by intelfx (subscriber, #130118) [Link] (1 responses)

> what? ZFS is a log-structured filesystem, like e.g. Sprite LFS and NetApp WAFL.

It's not a log-structured filesystem, it is a write-anywhere-*ish* (but not really) kind of filesystem with a log (journal) hanging off its side for performance.

Very different things.

The core concept underpinning DMU is still just a (potentially recursive) map of blocks. That is the "original Unix filesystem". It's just ext2 with (admittedly, a lot of) extra steps.

COW and volume management?

Posted Feb 5, 2026 16:05 UTC (Thu) by malmedal (subscriber, #56172) [Link]

> it is a write-anywhere-*ish* (but not really)

ish? Fully I believe. The only exception to write anywhere I am aware of is the uberblock. IIRC you basically have two logs, one for the uberblock only and one for everything else.

I believe NetApp does the same thing, the root inode, just like ZFS's uberblock, is constrained to a few known locations to avoid having to read every byte on every disk on a cold start.

> (but not really) kind of filesystem with a log (journal) hanging off its side for performance.

No, both metadata and data are always written to a new location when updated. There is no update in place neither in ZFS nor in NetApp WAFL.