COW and volume management? [LWN.net]

COW and volume management?

Posted Jan 26, 2026 11:22 UTC (Mon) by koverstreet (subscriber, #4296)
In reply to: COW and volume management? by jmalcolm
Parent article: Filesystem medley: EROFS, NTFS, and XFS

The split with the kernel needed to happen, unfortunately. Since then things have been massively smoother and my life has been much less stressful. Project's been growing, people are migrating and the focus is back on the technical.

The reasons for that split being a topic for another conversation, because I'm just drinking my coffee and enjoying a nice drama free day, and hoping to keep it that way, thank you very much :)

I should have erasure coding ready for testers inside of a week, I deferred the announcement lifting the experimental label to get that done due to popular demand (and post reconcile, there really wasn't much left). Gonna have some very cool stuff coming out this year - our ability to manage large numbers of drives is turning into something next level.

And we've got all the self healing stuff, complete and thorough fsck - and image creation tools that produce images on par with what EROFS can do.

But the thing that excites me the most is how debuggable the system is.

to post comments

COW and volume management?

Posted Jan 27, 2026 8:15 UTC (Tue) by drago01 (subscriber, #50715) [Link] (18 responses)

Being out of tree means it has to compete with ZFS. If someone goes through that pain, why not just use ZFS?

COW and volume management?

Posted Jan 27, 2026 9:00 UTC (Tue) by taladar (subscriber, #68407) [Link] (15 responses)

I can't speak for others but ZFS very much feels like a project that has peaked and has wasted a lot of its potential on that whole licensing fight. Any desire to even try it has been gone for many years at this point and the ZFS hype was another very off-putting thing about it.

Plus, on a purely technical level, I am sure ZFS, being the first of this type of file system has many design flaws that could be improved starting from scratch a generation later, much like any technology that tries something new and then gets stuck supporting existing behavior and design decisions as it gains users. Filesystems are probably affected by that more than most given that they deal with data at rest and not just some ephemeral data like e.g. network protocols.

COW and volume management?

Posted Jan 27, 2026 9:55 UTC (Tue) by marcH (subscriber, #57642) [Link] (5 responses)

I tried it only briefly but the concepts and user interface looked better than everything else I knew at the time. Very impressive.

It has real, "big iron" users too and they seem generally pretty happy too.

I have not researched it extensively but I don't remember hearing about any serious issue with it.

> and the ZFS hype was another very off-putting thing about it.

How do you know it was hype? I don't think any company tried to promote it, or at least not recently...

> I am sure ZFS, ...

"sure" without ever trying it? What is this based on?

> ... being the first of this type of file system has many design flaws

Maybe you could find and list at least one flaw?

> ... that could be improved starting from scratch a generation later, much like any technology that tries something new and then gets stuck supporting existing behavior and design decisions as it gains users.

Again, not an expert but I try to stay informed and so far it seems like other filesystems are still playing catch up...

> wasted a lot of its potential on that whole licensing fight.

Agreed, a real pity. That could almost make you wish it's also bad technically :-)

COW and volume management?

Posted Jan 28, 2026 8:33 UTC (Wed) by taladar (subscriber, #68407) [Link] (1 responses)

I did actually try it once for a little while in the early days when the hype was at its peak. From what i remember the ZFS design had some issues around requiring huge amounts of kernel memory which presented a big problem for alternate implementations even on license compatible systems like the BSDs and it quickly ran out of memory on systems with the 2GB of RAM that were still common at the time. Overall I was not very impressed, especially since 90% of its supposed advantages are meaningless on the small scale of just 2-4 disks or so that you might have in a typical home system.

COW and volume management?

Posted Feb 5, 2026 17:27 UTC (Thu) by georgh-cat (guest, #158897) [Link]

Is that memory issue something you experimented yourself?

I had a 750GB mirror running on OpenSolaris at home on 2008 (still using ZFS today and that pool was migrated once to 4TB disks), and I was afraid of the memory thing.

It was never an issue. Even when approaching full disk capacity, never had the memory usage be anywhere near high or even a potential problem. Then had the same server migrated to FreeBSD (OpenSolaris not being maintained anymore) and same experience.

If there ever were memory issues, I was somehow not affected or it was a myth or misunderstanding of how it worked.

COW and volume management?

Posted Feb 5, 2026 12:38 UTC (Thu) by anton (subscriber, #25547) [Link]

How do you know it was hype?

The kind of fearmongering used for selling ZFS was really off-putting. One supposedly-relevant scenario that ZFS defends against is when a HDD writes a block to the wrong disk sectors. This scenario has never happened in my experience, and the way HDDs work, it cannot happen in the way that the scenario suggests (by the read-write-head arm moving to the wrong track without any additional hardware failure).

That being said, we have used ZFS on many machines since about 2019 (mainly because it was not clear at the time whether RAID1 works well on btrfs), it works nicely. The only downside I see in it is that it is its own universe that ignores things like /etc/fstab and you need to learn the ZFS terminology even for terms that have a correspondence outside the ZFS universe. Another downside I read about is that ZFS supposedly does not utilize RAM as well as native Linux file systems which make better use of the buffer cache or somesuch. We certainly see little free RAM on our machines with ZFS. That's why I have decided to go for btrfs again for one machine recently.

COW and volume management?

Posted Feb 7, 2026 7:53 UTC (Sat) by cypherpunks2 (guest, #152408) [Link]

> Maybe you could find and list at least one flaw?

ARC is not well-integrated with the rest of the memory management system, so the system can OOM before ARC detects memory pressure and shrinks itself (so it's more than just an "ARC presents itself as used memory which confuses people" UI issue).

The cache for mapped files is unnecessarily duplicated in both ARC and the page cache.

There's still no support for defragmentation, which is an issue on high-speed IOPS-limited NVMe drives as CoW filesystems tend to experience heavy fragmentation.

COW and volume management?

Posted Apr 13, 2026 22:47 UTC (Mon) by dankamongmen (subscriber, #35141) [Link]

> Maybe you could find and list at least one flaw?

i've used ZoL on several machines for over a decade now. i've also written a large disk-management tool that has had to support all variety of filesystems. i've encountered zero problems with it from a user, despite generally running tip-of-git checkouts with most features enabled. can't say enough good about it from that perspective.

as an app writer and administrator, it has some annoying issues. it lives outside the standard filesystem management (fstab / systemd mount objects) with its own systemd services. i'm still unsure what exactly zed does, or what a ZEDLET is, or why it's written ZEDLET. you won't find the zpools in /sys/block, nor in /proc/diskstats, meaning you won't find them in say iostat, nor in blkid. compiling against libzfs is a shifting and rewardless PITA due to libspl clashing with all manner of userspace entities (random.h last i checked), and the OpenZFS developers have explicitly declared that one ought not compile against libzfs, essentially limiting sane userspace to the OpenZFS toolset.

whether or not one's bootup toolchain supports zfs this week is another question, one which i sidestep by never having zfs in my boot chain. as a result, i have an annoying mix of mdadm and zfs on these machines.

so it's great tech, but it's not without its annoyances. these annoyances seem furthermore not to be issues of bugs nor design flaws, but operational and philosophical decisions.

COW and volume management?

Posted Jan 27, 2026 11:12 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (8 responses)

> Plus, on a purely technical level, I am sure ZFS, being the first of this type of file system has many design flaws that could be improved starting from scratch a generation later, much like any technology that tries something new and then gets stuck supporting existing behavior and design decisions as it gains users. Filesystems are probably affected by that more than most given that they deal with data at rest and not just some ephemeral data like e.g. network protocols.

"Flaws" isn't a fair characterization; the approach the ZFS devs took was a pragmatic one, and the only one that had any chance of success given the scope of what they were trying to do.

When you're trying to do something ambitious, you don't shoot for the moon - you can't reimagine everything at once, you can't take on too many big risks, that's how you get stuck in second system syndrome; you'll never complete your task. You have to work incrementally, and pick and choose what you're going to reimagine and what risks you're going to take.

But - that does mean that ZFS is essentially the original Unix filesystem underneath. Blocks, not extents, and data structures that are clearly in that lineage.

bcachefs is that "shoot for the moon and reimagine everything" project. It only happened because I got profoundly lucky with some of the original key design decisions (the b-tree implementation in particular), and for the first five years I wasn't attempting a full filesystem - I had much smaller targets (a block cache, and then something more intermediate) where I was able to successfully complete and ship building blocks for the full filesystem; it wasn't until those were successful that I realized attempting a filesystem was even possible.

So in a way you're right, but don't sell the ZFS folks short - they deserve massive amounts of credit and respect for showing everyone what a modern filesystem should be capable of.

And there's a whole list of things that took years to crack before I had something better:

- Snapshots: snapshots + extents are a difficult combination, Dave Chinner and others were poking me about this for years before I had a working design (I've got notebooks filled with scribbles of different ways of solving the "extents can overlap in arbitrary ways" problem).

- fsck/self healing: bcachefs benefited massively from the hindsight of seeing what other filesystems were doing, and just having a lot more time to get it done. Going straight for full self healing and skipping a conventional fsck does not get you what you want, nor does just starting with a conventional offline fsck. I had experience with proto-runtime-fsck from the early days immediately after bcache, and time to explore a lot of different options; bcachefs takes a mixed approach that's turned out to be very comprehensive, but that took a lot of time to get right.

- The erasure coding design is entirely original; it delivers much better performance than RAIDZ but had many difficult problems to solve over years of development (it all looks simple in hindsight, but it wasn't simple to figure out when it hadn't been done before!).

- The whole "filesystem that really is a database underneath" approach has always been the dream - having that makes my life so much easier now. (Was working on the last bits of hooking up erasure coding to reconcile last night for automatic resilvering when a drive goes away, when I realized the on disk data structures were wrong; when using the reconcile_*_phys btrees for doing pending work in on-disk LBA order on rotating disks, we can't mix that in with the pending work for user data extents; we need to repair stripes first; _then_ user data extents to get anything that wasn't erasure coded, or we'll do a ton of unnecessary work; oh damn, that was a whole 10 lines of code to fix). But no existing b-tree implementation was remotely fast enough, even today - that was the big one that I cracked 15 years ago, with design decisions that look prescient in hindsight, and then benchmarking bcache on early Google SSDs when we realized "oh damn, binary search is actually really slow" and pulling math out of the ether to do magic with eytzinger search trees...

COW and volume management?

Posted Jan 27, 2026 12:45 UTC (Tue) by Wol (subscriber, #4433) [Link] (4 responses)

> - The whole "filesystem that really is a database underneath" approach has always been the dream - having that makes my life so much easier now.

Wowser! And that's moving into the mainstream now? Again, massive kudos to you if you've managed to pull it off, but that's been around for ages.

I hesitate to say it was true of Pick - it was sort of - but that was more "the storage system is permanent virtual memory". The AS400 had a database as a filesystem, no? And there were loads of rumours that NTFS was originally going to be a derivative of the Pick database - that was back in the days of NT3.1 whenever that was.

"If I can see so far, it's because I'm standing on the shoulders of giants" - but you're a giant yourself for pulling it off!

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 13:15 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (3 responses)

Yup.

We're not using it to do anything particularly novel yet, just doing normal filesystem stuff but done better and more cleanly, so far. And it's still not quite as general as I want; the main remaining limitation is that keys are fixed size at 160 bits (64 bit inum, 64 bit offset, 32 bit snapshot); key types really need to be per-btree definable, bpos has been increasingly overloaded in different places and it's annoying.

It's also definitely nowhere near SQL, where you define tables and indexes and do joins; it's more of a NoSQL database. But it's got a solid transaction model, locking model (with a full cycle detector, as real databases do), triggers, nice iterators, lots of support for extending the database schema - it's generally quite pleasant to work with.

The challenge with making this stuff mainstream has been performance. Historically, Unix filesystems were completely unlike databases because they needed to be simple and fast - inodes as flat arrays on disk, blocks indexed by radix trees hanging off the inodes. Lots of simple specialized data structures.

Bu conventional filesystems have started to look more and more like databases over the years as they've gotten more complex. Extents mean radix trees don't work anymore, you need btrees, journaling means you need a transaction model, and every additional feature filesystems want to do pushes you more in the direction of "I really just want a real general purpose database for this stuff".

The game changer was when, in the early days of bcache, we pushed the performance of the "btree with big log structured nodes" design to the point where it could support a filesystem with no per-inode sharding - that was when I knew bcachefs was possible.

COW and volume management?

Posted Jan 27, 2026 13:45 UTC (Tue) by Wol (subscriber, #4433) [Link] (2 responses)

> We're not using it to do anything particularly novel yet, just doing normal filesystem stuff but done better and more cleanly, so far. And it's still not quite as general as I want; the main remaining limitation is that keys are fixed size at 160 bits (64 bit inum, 64 bit offset, 32 bit snapshot); key types really need to be per-btree definable, bpos has been increasingly overloaded in different places and it's annoying.

> It's also definitely nowhere near SQL, where you define tables and indexes and do joins; it's more of a NoSQL database. But it's got a solid transaction model, locking model (with a full cycle detector, as real databases do), triggers, nice iterators, lots of support for extending the database schema - it's generally quite pleasant to work with.

Don't confuse SQL and Relational, they're two completely separate things in a marriage :-)

And you say "NoSQL" - do you mean No SQL, or Not Only SQL? :-) (I'm sure you don't mean the database actually called "NoSQL" :-)

But it might be worth taking a look at Pick's Not Only SQL model. I don't understand what you're going about key types and btree especially, but Pick is hash-based not btree based. Investigate Linear/Dynamic hashing. That swaps btree's "all accesses take the same time" for hashing's "most accesses are extremely fast" and the linear/dynamic algorithm makes resizing order 1, not order N. I've been thinking about making ScarletDMEs hash buckets into btrees so in the worst case access degrades to btrees ?order root N? rather than a naive hash's order N. (And speeds up splitting an overflowed bucket, too.)

But if you're looking at "the file system is a database", don't restrict yourself to SQL/Relational. Just because it's taken the universities by storm so that nobody knows anything else, doesn't mean it's the best, or even has any ideas worth nicking :-)

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 15:07 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (1 responses)

SQL may not be the prettiest language, and the relational model may not be what you want in every context - it's too abstracted when you really need to optimize for performance - but there's a reason it's taught, it's a good model, and it's something every programmer should probably know.

Hash tables are no-go in on disk data structures, they destroy locality and you can't do range lookups. Extents require range lookups, along with a lot of other things filesystems need to do, and we very frequently need to do efficient scans - btrees are much better suited to that.

Locality is a prime consideration when designing on disk data structures, and frequently you'll be doing short linear scans - linear searches are not slow as long as you keep them short.

E.g., the way bcachefs does full writeable snapshots, with versioning (so snapshots are at key granularity, not btree node granularity like btrfs) uses the low bits of the key as the snapshot ID - so a lookup finds either a key in the current snapshot, or an ancestor snapshot - that requires range lookups. But then for writeable snapshots, the snapshot ID is not a linear history, it's a tree; so we also have to skip over keys in unrelated branches of the snapshots tree. This search will be short for a variety of reasons (the main one being that creates always get a new inode number that won't be shared by any other snapshots), so snapshot lookups are fast; you can't do these kinds of tricks with hash tables.

COW and volume management?

Posted Jan 27, 2026 21:02 UTC (Tue) by Wol (subscriber, #4433) [Link]

> SQL may not be the prettiest language, and the relational model may not be what you want in every context - it's too abstracted when you really need to optimize for performance - but there's a reason it's taught, it's a good model, and it's something every programmer should probably know.

The relational *model* is great, yes. The engineering is flawed by design :-(

Pick allows you to stick a 4NF view directly into the database as a table row ... that's why an optimiser is a waste of time ... (now that most RDBMSs have arrays and structures, they've finally caught up, 40 years late, but SQL can't query them cleanly)

Cheers,
Wol

COW and volume management?

Posted Jan 27, 2026 13:59 UTC (Tue) by malmedal (subscriber, #56172) [Link] (2 responses)

> But - that does mean that ZFS is essentially the original Unix filesystem underneath.

what? ZFS is a log-structured filesystem, like e.g. Sprite LFS and NetApp WAFL. Very different tradition from the old modify-in-place unix filesystem.

COW and volume management?

Posted Feb 5, 2026 13:26 UTC (Thu) by intelfx (subscriber, #130118) [Link] (1 responses)

> what? ZFS is a log-structured filesystem, like e.g. Sprite LFS and NetApp WAFL.

It's not a log-structured filesystem, it is a write-anywhere-*ish* (but not really) kind of filesystem with a log (journal) hanging off its side for performance.

Very different things.

The core concept underpinning DMU is still just a (potentially recursive) map of blocks. That is the "original Unix filesystem". It's just ext2 with (admittedly, a lot of) extra steps.

COW and volume management?

Posted Feb 5, 2026 16:05 UTC (Thu) by malmedal (subscriber, #56172) [Link]

> it is a write-anywhere-*ish* (but not really)

ish? Fully I believe. The only exception to write anywhere I am aware of is the uberblock. IIRC you basically have two logs, one for the uberblock only and one for everything else.

I believe NetApp does the same thing, the root inode, just like ZFS's uberblock, is constrained to a few known locations to avoid having to read every byte on every disk on a cold start.

> (but not really) kind of filesystem with a log (journal) hanging off its side for performance.

No, both metadata and data are always written to a new location when updated. There is no update in place neither in ZFS nor in NetApp WAFL.

COW and volume management?

Posted Jan 27, 2026 10:46 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

bcachefs doesn't have the licensing issue and, if and when it's mature and trusted, it'll (hopefully) be a no-brainer to have it merged back in to upstream.

COW and volume management?

Posted Jan 27, 2026 12:12 UTC (Tue) by koverstreet (subscriber, #4296) [Link]

No, it won't be a no brainer -

I've always believed in "just write the best code you can - the rest will sort itself out later". Not being upstream hasn't hurt the project at all - on the contrary, if anything adoption has accelerated over the past six months. We're no longer tied to a project with engineering practices that haven't changed much over the past 20 years, and right now I'm wondering how the kernel is going to manage the Rust transition over the next 10-20 years.

At some point, the broader engineering community is going to realize that all the code we depend on - essentially, all of it - is going to have to be rewritten in Rust. The level of complexity we can manage in C has been hitting limits; we cannot keep doing the same old forever. Systems engineering is going to get so much easier if we can manage the Rust transition, and there are so many techniques in debugability, introspection, static analysis and formal verification that that opens up. There is a mountain of work that needs to happen to get there - but it needs to happen, or we'll be stuck with a mountain of legacy code in a language new people aren't learning with the people that actually understand it dying off.

To understand how this impacts myself personally and the bcachefs project: my rule of thumb is that when a bug in upstream, non-bcachefs kernel code comes across my desk (and they do), it's going to take at least 10 times as long to debug as any bcachefs bug. And I'm going to be stuck with it personally; there are people who have been tearing through the bcachefs code and can debug anything (literally; I'm half-following the IRC channel right now) and will tell me exactly what's broken and what needs to be fixed - and I'd be more than comfortable just giving commit access to the repo but they seem to like using me as the coding bot - but when it's an mm-involved bug or anything else I'll hear back a week later - "uhh, maybe you better take this one on".

There's ways to write complex code and make it testable and debugable, but it has to be an object of serious study, you have to prioritize it. Fortunately, Rust is going to make this a hell of a lot easier, with things like the Display trait, and a type system that's comprehensive enough that you don't have to stare at code for a couple hours in order to safely change one little thing - but "how to write complex systems that we can maintain and debug in the wild" is something that needs more thought and study in the kernel community.

So I'll be taking a wait-and-see approach. I have seen some positive developments over the past six months - people who weren't finally seem to be talking somewhat more seriously about automated testing, and that's been a huge gap that was really painful for bcachefs (my test dashboard would blow up every rc1 rebase with bugs in other subsystems, and I can't do work or merge code without good test data, so all work would stop - often for weeks - while dealing with that). But I think things have a ways to go before I could seriously consider it - and that's if they even wanted bcachefs in the kernel, and I think everyone knows the rough outline of that part of the story :)