LWN: Comments on "Bcachefs may be headed out of the kernel"

What next?

fpemud — Mon, 11 Aug 2025 03:46:22 +0000

Will bcachefs officially release as a kernel patch? If it is kicked out, I plan to include it when building my kernel --since I'm already applying several other patches.

Linus had had lots of patience with him

queenkjuul — Mon, 07 Jul 2025 12:38:48 +0000

Building a kernel is not difficult. Certainly not harder than learning how to use advanced recovery tools for an experimental filesystem. It's genuinely baffling to me that anyone *using an experimental Linux filesystem* couldn't figure out how to install a new kernel.

Though if bcachefs were a DKMS module instead, they wouldn't even have to. It seems that's probably the better option, then, until the filesystem is no longer experimental.

Overstreet brings this on himself

taladar — Mon, 07 Jul 2025 07:24:30 +0000

But "does not change" is a useless property on software that does not work yet and never has.

Overstreet brings this on himself

NYKevin — Sun, 06 Jul 2025 20:13:26 +0000

> Except for maybe ext4; I've seen a ton of user feedback and I think I've only seen one person - ever - mention lost data with ext4, and that in passing with no details

ext4 had the whole O_PONIES/fsync/data=ordered debacle. And no, I don't care to relitigate that, but from the user's perspective, data loss is data loss regardless of whose fault it is. "It worked on ext3 and then I lost data on ext4" = data loss.

Business decisions

koverstreet — Sat, 05 Jul 2025 14:04:48 +0000

Ouch! I'm sorry that happened to you, that sounds rough.

Would you be able to get me any info on what happened? Was it the casefolding+overlayfs incompatibility?

This isn't completely surprising: in the stabilizing process, I have been focusing on issues that would take the filesystem offline or result in data loss - bugs that affect only specific applications have been talking a back seat.

That's changing now, critical bugs have been dropping fast, so over the next six months I'll be able to prioritize bugs like this better.

Just use bundled dependencies

sunshowers — Sat, 05 Jul 2025 01:04:24 +0000

Yes, that's definitely the right approach. The primary goal of most software ought to be to deliver value to users, not to worry about being packaged in Linux distros. Whether distros can package some software or not is pretty far down the list of what I think most teams should care about.

Linus had had lots of patience with him

Kluge — Fri, 04 Jul 2025 21:43:43 +0000

I don't see the point in lecturing users about how to meet their needs.

Overstreet brings this on himself

swiley — Fri, 04 Jul 2025 18:12:37 +0000

Linus being heavy handed is a large part of the reason the kernel is so nice. Don't forget the GNU Hurd kernel is open source under a similar license and isn't nearly as pleasant to use or work on.

Business decisions

koverstreet — Fri, 04 Jul 2025 15:09:04 +0000

No, I think bcachefs will do fine if we get kicked out of the kernel.

a) there's every reason to think it would only be temporary: this happened before with Lustre, and they're getting ready to go back in. Hopefully the next time around there would be a real back and forth conversation about process. This isn't ZFS - that one is outside the kernel permanently because of licensing.

b) Being outside the kernel opens up interesting possibilities. Maybe we finish the fuse interface, and fuse performance gets fixed - there is no inherent reason running in userspace via fuse has to be a noticeable performance hit for buffered IO. There'd be a lot of work to get to that point, but multiple people are working on fuse performance right now, it's been a very long standing problem with a lot of interest.

That would be very cool.

With more work, a filesystem outside the kernel might even be faster. Applications that need O_DIRECT are a bit of a problem, but solvable with L4 style IPC (and maybe a LD_PRELOAD so applications don't have to be converted). But the block layer is heavier than it needs to be, even with blk-mq - it retains a lot of legacy baggage, and we could easily just start talking to NVME devices directly.

So who knows.

But this is all pie in the sky stuff, and for the next two years I'm going to be staying focused on bcachefs itself - there's still a lot of work to be done. More online fsck, more self healing, finishing erasure coding, performance work, better management and monitoring (especially with respect to degraded data), more rebalance improvements (need to fix a bug where we still spin if you try to stuff more data into your background_target than it can hold) - better allocator policy for large numbers of devices, failure domains, > 64 device support (and maybe try for > 256) - there's a lot of cool stuff lined up for after stabilization finishes, and judging from the current rate of bug reports we're mostly there.

I try not to worry too much about the big picture process stuff. As long as I can stay focused on the code, the rest will sort itself out eventually.

Overstreet brings this on himself

SLi — Fri, 04 Jul 2025 12:48:25 +0000

I think there's a level of misunderstanding about what "stable" in e.g. Debian's context means. It does not mean "works". It means "does not change".

That is, Debian will often package software that is very experimental and buggy, and that may get into stable. That is not a problem in itself. The stable guarantee is that it will not change in incompatible ways, usually including Hyrum's law. A crash may not be important enough to fix in stable. The promise is that it keeps working as it has worked so far.

The biggest reason why Debian typically would have a problem with packaging rapidly changing software is the thought that *if* they have to change it—usually to fix a security issue, sometimes another important enough bug—then backporting the fix to the version in Debian stable may be too difficult, making it unsupportable within Debian's stability promise.

Overstreet brings this on himself

taladar — Fri, 04 Jul 2025 11:41:06 +0000

In experimental software with a decent development pace that still has to reach the first stable version versions that are a few weeks old is ancient. Are you sure it is not you who don't understand how stable releases of software work? They do not happen until you reach a first version you can consider stable because they are meant to preserve stability. If stability has not yet been reached in the first place there is no point in supporting older versions for any length of time at all.

Overstreet brings this on himself

highvoltage — Fri, 04 Jul 2025 10:53:07 +0000

Nope, I'm sorry Kent, you fundamentally and clearly simply don't understand how stable releases of software works. Even versions of bcachefs-tools that were a few weeks old in Debian were "ancient". On that bar, there's no way you could ever include it in a stable release. Also, your demands go far out of Debian policies, bcachefs-tools has zero chance of a future if it can't release any code that's suitable for long-term use.

Overstreet brings this on himself

taladar — Fri, 04 Jul 2025 10:42:44 +0000

You obviously have strong opinions on this topic but for someone on the side of this debate that claims they need professional structure processes and Kent is behaving in an unprofessional manner your post reads incredibly toxic and honestly would put me off even trying to work with the Linux developer community if this is the tone to be expected in debates there.

Business decisions

taladar — Fri, 04 Jul 2025 10:37:11 +0000

I wouldn't say the problem is always on the side of the one claiming to be special, it can often also indicate a deeply broken culture in the rest of the system, e.g. most social change in history started out as a rebellion against the established norms and most revolutionary technical discoveries and scientific theories had to fight uphill battles against the majority in their field too.

All that can really be said is that someone feeling the need to claim that they are special means that a problem exists on one side or the other.

Just use bundled dependencies

taladar — Fri, 04 Jul 2025 09:59:33 +0000

> Or in general how Rust code would live in a world where not everything is Rust and follows Rust ways of thinking.

The non "Rust way of thinking" is really just the C way of thinking based on workaround for the flaws in the C development model. The only way to become completely compatible with that would be to copy all those flaws, in particularly the hand-wavy attitude towards ABI compatibility that allows the C world to pretend that ABIs are compatible when there is hope that they are reasonably similar if you squint a little.

Business decisions

zdzichu — Fri, 04 Jul 2025 05:23:54 +0000

To be clear, anything claiming to be special is counter-productive, not only filesystems.

Overstreet brings this on himself

koverstreet — Fri, 04 Jul 2025 00:19:44 +0000

Every filesystem has stories of lost filesystems. (Except for maybe ext4; I've seen a ton of user feedback and I think I've only seen one person - ever - mention lost data with ext4, and that in passing with no details).

Consider that.

You don't get to the type of reliability ext4 has by writing perfect code, modern filesystems are just too big - no one can write that much perfect code. And ext4 achieves its reliability through simple on disk data structures, with the important stuff in fixed locations - it definitely does not have perfect code.

You get it by iterating on every single bug report you can find, by making the most of every bug report, looking for everything that went wrong or could have gone better, especially all the debug tooling and information so you can see what went wrong. You do it by being proactive.

Rinse and repeat, over and over, until the bug reports stop coming in and you're confident you can handle anything Murphy's law throws at you.

Overstreet brings this on himself

MichaelRose — Thu, 03 Jul 2025 22:27:29 +0000

It seems obvious that any filesystem that needs constant changes at a moments notice to avert disaster isn't nor has it ever been stable.

Being used by the kind of folks who you suggest would find it burdensome to build a kernel or install a third party repo indicates that you have successfully sold your project to at least some users who have no business using an experimental filesystem.

It suggests that inclusion in the kernel was a mistake that shouldn't be compounded by subordinating process designed around comparitively stable software to support your unstable software.

The best remediation would be to clearly communication to those folks who shouldn't be using your software in the first place by warning folks then removing it. Those so capable can migrate to building it or using a third party repo. Those not so can migrate their data.

In the future it could always be added back when it's actually stable.

Business decisions

skx — Thu, 03 Jul 2025 20:58:37 +0000

The last time I remember "This filesystem is special" was reiserfs.

The reworking there was also highly controversial and there were strong personalities and much talking-past each other on both sides of the debate.

Overstreet brings this on himself

koverstreet — Thu, 03 Jul 2025 19:00:17 +0000

It's also not just about the bugfix itself - it's also about risk mitigation.

I often tell people: when you're looking at e.g. a syzbot bug, don't do the minimum to make the report go away.

Reproduce it locally, read through the log output with an eye towards the behavior of the whole system. Make sure that the behavior in response to the error makes sense (which is a lot more than just not crashing!), and see if you can find other things to improve. That could be logging (we can't debug if we can't see what's going on), repair, or even - like in this case - looking for ways to limit the impact of similar bugs in the future.

We've now had two separate bugs in two weeks where this new repair mode has proved useful.

The second one, for which I just determined the root cause this morning, involved a filesystem with a 400GB postgres database (and a whole bunch of other data) where the directory structure got trashed.

(Two different bugs in two weeks? I'd say getting the code out there quickly was justified; I've learned to trust my intuition).

Proximate cause was a flaky USB controller and a crazy iscsi setup - which is exactly the sort of thing I love to see: I want people doing the craziest oddball crap they can imagine to break things _now_, before the experimental label gets lifted.

It turns out 6.16 broke btree node read retries for validate errors - not IO errors, we had tests for that, and there's a story as to why the tests were missing; the error injection we were used to rely on from one subsystem was dropped with a supposedly equivalent error injection mechanism from a different subsystem - except the new one wasn't tested for anything except IO error injection, the other functionality was completely broken.

Ow. Testing is important.

But we had a lot of logging available to sift through to find out what went wrong, and one thing we're getting in 6.16 (which incidentally also was the patchset that introduced the regression) is much improved logging for data read and btree read errors - which made the missing retry from rest of the replicas blindingly obvious.

And now I'm about to commit and push new tests for the relevant error path, and the user who hit the second bug is getting most of his stuff back thanks a combination of journal rewind (that didn't repair everything, the journal didn't go back far enough - we didn't catch it early enough) and writing code to find files by hash/filetype (almost nothing was completely lost, just ended up in lost+found).

And I'm writing new tests today.

TL;DR: defense in depth, risk mitigation wherever possible, and always have eyes on as much of the system's behavior as possible when things go wonky.

Overstreet brings this on himself

Kluge — Thu, 03 Jul 2025 18:20:27 +0000

Two points: 1) Bugs come in all shapes and sizes. The best, even only way to fix a bug may be to add a feature, although that's rare. 2) The distinction between bugfix and feature is a lot clearer in stable code than unstable/experimental code.

It may make sense for Kent to be solely responsible for any breakage that new post-merge window Bcachefs code causes, but I don't see a good rationale for saying that the same rules should rigidly apply to stable and experimental (and optional) parts of the kernel.

Recovery in userspace?

nstiurca — Thu, 03 Jul 2025 14:54:38 +0000

> I would be very surprised if [performance] is a serious concern in recovery scenarios.

It's a very serious concern IMO. The need for recovery never comes at a convenient time; need for recovery almost always happens when you're working on something else which probably has time pressure. If recovery takes 5-10 minutes, then ok you go get a cup of coffee and it's not too big a deal. If it takes 3 hours, it can at best ruin your day's productivity, or at worst miss a presentation or a deadline or have an extended outage in some critical process. Put differently, if something is moderately slow all the time, that's probably manageable because you can plan and work around it; but if something is very slow at an unexpected time, it's much harder to deal with and carries more unmanaged risk.

Just use bundled dependencies

Sesse — Thu, 03 Jul 2025 09:25:23 +0000

> I think the creation of npm, Cargo etc had very little to do with Windows.

My personal experience is that these things mainly come out of the needs of macOS, not Windows.

But Cargo specifically struck me as short-sighted from the get-go; there was a long design document/manifesto that talked about the evolution of any given piece of code and how it would be installed and how Cargo would think about this, and _not once_ did it mention that a distribution may want to package the resulting program. Or in general how Rust code would live in a world where not everything is Rust and follows Rust ways of thinking.

Second request

ejr — Thu, 03 Jul 2025 03:44:27 +0000

Thank you.

The bcachefs development process

koverstreet — Wed, 02 Jul 2025 22:10:06 +0000

No, we're not talking about a "cool new feature".

I understand that you're speaking from personal experience, but this is filesystem development we're talking about, where we've got a massive investment in QA processes - and we rely on those heavily, especially because we need to make calls like this on a regular basis. It's not uncommon to have patches that justify going out quickly, and this patch was not outside the norm in complexity or likelyhood of breakage; but it was outside the norm in terms of potential for risk mitigation.

We do a lot of risk mitigation in the filesystem world :)

I'm afraid your experience isn't likely to be comparable - I don't see you weighing the cost/benefit analysis or talking about those QA processes, and those are critical to decisions like this one.

Just use bundled dependencies

raven667 — Wed, 02 Jul 2025 19:11:17 +0000

This is maybe a bit of a tangent but this depends a lot on how much time you have to work on upgrade issues and testing, and how much your development needs to track HEAD for its dependencies. I have some software we (Netbox, a python/Django app) which generally depends on a recent supported python and the latest libraries from pip and isn't tested with anything else, that can become a bit of a pain to maintain on an RHEL which is getting long in the tooth (separate SCL python when we used EL7 and tweaks to make sure the right versions/paths were used for everything) but most of our in-house software is perl and I was happy with the features and performance of RHEL5 with perl 5.8 (aside from missing the process supervision and log collection from systemd/journald in EL7, which was a fantastic upgrade). We upgrade OS when the hardware is no longer supportable, not because we are chomping at the bit for new features or libraries or are resource constrained, the server we bought in 2010 would probably still run our workload just fine today, and aside from a security update in freeradius which was a mandatory behavior change, I can't recall ever having an RHEL patch update actually break anything, its very minor, rare and unmemorable if that happened.

There is a time savings in not having to perform a major upgrade of the OS every 6mo-1y and re-test everything to find the unexpected behavior changes, and only doing that work once every 5-10y. Not every team can absorb the churn while still making forward progress on their other priorities. I'm sure that the config automation we've built can get 95% of our app configured and running on the latest Fedora, but finding and fixing that other 5% to make it _perfect_ is substantially more work for diminishing returns and competes for time with bug fixes and changes that other people are waiting for.

Like a lot of these things, which approach is "best" depends more on the makeup of your team than on the technical details being "objectively better" one way or another.

Time pressure

pbonzini — Wed, 02 Jul 2025 17:19:23 +0000

No problem :))

Business decisions

pbonzini — Wed, 02 Jul 2025 17:18:10 +0000

It is, but it makes a point. Ultimately the future of bcachefs is a lot more hazy if it's not upstream.

So if you agree that it's dramatic, maybe you also agree that this isn't "a hill to die for", to quote my earlier comment, and it's worth following the rules that the upstream community has set for itself.

Social problems don't always have technical solutions, but technical solutions can help. The dual tier tree could be one, as it helps you probe how many adopters are okay with the content from Linus's tree and how many want the bleeding edge.

The bcachefs development process

wtarreau — Wed, 02 Jul 2025 16:21:36 +0000

I understand your feeling, I'm also always torn between merging/not merging my stuff in my projects, but admittedly here it's a new feature, and the fact that it saved two people's data in two weeks doesn't exclude the risk of causing trouble (even if less important) to others (or even just developers/testers), and that's precisely the point of the development process.

In haproxy for example we're much less strict than the kernel regarding the merge window (much smaller team, so the -next branch is super small), so we'll code roughly for 4 months out of 6 and try hard to mostly do bug fixes in the last two months. Guess what ? The vast majority of post-release bugs are caused by apparently harmless changes from the last month, that used to work perfectly when and where tested.

The likelihood that your cool new recovery feature that was missing before it was developed, would be missing to numerous users until next merge window remains small, and alone would be sufficient to justify merging that cool stuff only in the next release, possibly with even more testing, positive feedback and bug fixes that would have had time to accumulate by then. And it would also avoid the risk to create precedents where everyone starts to include more important changes after the merge window.

Overstreet brings this on himself

rsidd — Wed, 02 Jul 2025 14:59:15 +0000

Yes, it's about priorities.

It's about prioritising your needs and your perceived needs of your users over the project management needs of a project that has maybe 3-4 billion users (counting Android).

It is clear you don't see that despite any argument people throw at you. Good luck.

The bcachefs development process

koverstreet — Wed, 02 Jul 2025 14:53:27 +0000

The patch in question - journal rewind - is already being used by myself and another user for a completely different bug - the user had a (replicated) filesystem with a 400GB postgres database on a flakey USB controller. (What will they think of next?).

Unfortunately the issue wasn't detected early enough, so it's making it hard to tell what the root cause was (but flaky hardware makes it sound like a cascade failure - cool). Even so, about a terabyte of data is being recovered that wouldn't have been otherwise.

So if it's being used successfully for two different bugs in two weeks, getting it in was the right call. I've learned to trust my intuition.

Overstreet brings this on himself

koverstreet — Wed, 02 Jul 2025 14:49:07 +0000

It's a succinct way of saying "we will make every attempt possible to harden the filesystem against everything imaginable, and when something bad happens we will make every attempt to recover, and learn from it so it doesn't happen again".

It's about priorities.

Second request

corbet — Wed, 02 Jul 2025 14:29:49 +0000

I'll repeat — speaking to the thread as a whole — I think we have had plenty of posts telling people what they need to do. I really doubt anything useful will come from more of them; please let's bring this to a close.

Well...

kragil — Wed, 02 Jul 2025 14:16:49 +0000

Mr Overstreet, you need to play by the rules. It is as simple as that. Mr Torvalds is calling the shots, you need to be on his good side, that is just the way things are and your users can wait PERIOD Nobody will sponsor you or invest in your FS if it gets this kind of bad press all the time.

Just imagine an alternative reality where the above would have always been true ...

Overstreet brings this on himself

dwon — Wed, 02 Jul 2025 09:33:26 +0000

> a) We do not lose user data

You *do*, though. If bcachefs didn't lose user data, there would be no urgency for getting these patches merged after the merge window has closed.

You seem to think "we don't lose user data" is a unique strength of the way you do development, but it's really just a combination of you overstating your filesystem's robustness and you not playing well with others.

I read through this thread as a bystander, and honestly I'm coming to the same conclusion as Linus: you seem to have a really rigid way of thinking about how to do development, and you don't seem to think it's important to adhere to Linus's process, or Debian's process, both of which exist in part to make it possible for generalists like him and Ted to make enough sense of what a bunch of a specialists like you are doing, and turn it into something people actually want to use. They're also designed to take care of a whole bunch of other concerns that users care about.

Your dismissive attitude to the work they're doing, just because they're prioritizing concerns that you don't personally understand the value of, is really insufferable.

Overstreet brings this on himself

Wol — Wed, 02 Jul 2025 08:59:21 +0000

Why can't it be both?

Let's take knives for example - your eating knives are usually pretty blunt and safe, they're safe in *any* environment.

Kitchen knives, on the other hand, tend to be sharp and dangerous - fine when used as designed. The "experimental" tag marks bcachefs as a kitchen knife - not safe when given to script kiddies ...

Cheers,
Wol

Just use bundled dependencies

Wol — Wed, 02 Jul 2025 08:53:21 +0000

The gentoo package management system has tools which invoke cpan (perl-cleaner).

Which is fine once you realise that - you need to invoke it separately, and I've known several systems where "emerge" gets wedged until you run it.

That's one of my little niggles with gentoo - updating is a dance of several commands which is fine once you've discovered them all, but there's always the odd corner case to bite you ...

Cheers,
Wol

Overstreet brings this on himself

jengelh — Wed, 02 Jul 2025 08:21:31 +0000

>No, I do not say that it is a stable fs - it's marked experimental for a reason!

bcachefs.org has this to say, though:

>Already working and stable

So which one is it?

Just use bundled dependencies

niner — Wed, 02 Jul 2025 08:18:02 +0000

Indeed. I'm glad we have chosen to go into the other direction and deploy openSUSE Tumbleweed with very frequent updates. Sounds insane on a first glance, but the systems this applies to are redundant, very well covered by automated tests and of course intensely monitored. Combine that with a staggered rollout and you get a very stable system that avoids the hoops you have to jump through to support stone age versions and also makes developers happy as they get to use the shiny toys. And of course you get the (usually) faster, more secure and more powerful software.

Overstreet brings this on himself

taladar — Wed, 02 Jul 2025 08:10:11 +0000

Turn that around. Architectures that don't have the means to provide test/CI systems for all developers writing code that should compile there are probably not suitable for the upstream kernel.