Couple points

- Testing needs to be not just a filesystem thing, but a kernel thing. When bcachefs was in the kernel, it was really quite shocking how frequently (every release cycle!) my test dashboard would blow up due to regressions from other subsystems. Frequently when I went to report these the subsystem maintainers didn't even have the resources to reproduce them. It's far more time consuming for someone else to debug your code, who doesn't know it, after it's hit mainline and is mixed in with thousands of other patches; lack of testing is a giant tax on developer time.

But - the xfstests model of writing tests is extremely dated, the "golden master" approach to deciding if a test passes or fails is pretty painful to work with. So I don't see how a solution designed only for running xfstests is going to be of use to the rest of the kernel.

> Ts'o said it might be easier to get some one-time funding to simply develop a tool, rather than creating a project like KernelCI, but for filesystem testing, which requires ongoing fundraising to maintain

Machine resources do require ongoing funding, though - and there's a real disconnect here, the machine resources required to do real filesystem or kernel testing are significant. With Hetzner's recent increase in prices, the monthly bill for bcachefs testing is now $2800/month, that I pay for out of pocket, and that's doing it the most economical way possible - and I keep those machines loaded. When I was talking to Darrick Wong about XFS testing, he mentioned that his OCI bill (that Oracle covers) is $30k/month - a cloud vendor like Google or Oracle is going to charge you ~10x more for the same compute as getting bare metal machines from Hetzner.

When I went to the KernelCI board about making ktest available to the kernel community, the response I got from a board member was "well, we don't see the point of spending that much, we'd maybe do $500/month, and we'd need to see PowerPoint slides first".

Google and Intel pay for pretty significant machine resources for Syzbot and 0day - it's not like there aren't deep pocketed vendors who depend on Linux and have demonstrated a willingness to fund testing, and it would be to everyone's benefit to have a rack or a couple racks of machine for general kernel testing available to the maintainer community, but someone's got to make that argument (especially that it'll get used!).

> Christian Brauner raised the problem of "flaky" tests, those that only pass sometimes. Ts'o said he has an internal version of his harness that allows tests to be marked flaky; if they fail, they are run three more times and are only reported if all of those fail. He has meant to add that feature into the public version, since it is useful, but has not found the time.

This hides bugs. There's always a "long tail" of bugs that rarely reproduce until you throw a ton of resources at them. My approach is - run the full test suite on every commit, and my dashboard can show me the history of any test; that means if a test is flaky I can check - did it start being flaky recently, has it always been flaky - a lot of subtle behavioral regressions (especially subtle performance regressions) show up this way.

Glad to see testing being talked about more, I hope it translates into real progress.