Kernel quality control, or the lack thereof

Posted Dec 9, 2018 14:20 UTC (Sun) by saffroy (guest, #43999)
In reply to: Kernel quality control, or the lack thereof by iabervon
Parent article: Kernel quality control, or the lack thereof

Fuzzing is extremely useful, but it still needs a *thinking* developer to help it generate interesting cases in reasonable time.

Besides tests themselves, it helps a LOT to have some kind of test coverage report, just to remind you of which parts of the code are never touched by any of your current tests.

Do people publish such coverage reports for the kernel?

Kernel quality control, or the lack thereof

Posted Dec 10, 2018 9:49 UTC (Mon) by metan (subscriber, #74107) [Link] (4 responses)

The tool to generate coverage for kernel is maintained at https://github.com/linux-test-project/lcov it should work, but I haven't tried it.

However I can pretty much say that the main problems I see are various corner cases that are rarely hit (i.e. mostly failures and error propagation) and drivers. My take on this is that there is no point in doing coverage analysis when the gaps we have are enormous and easy to spot. Just have a look at our backlog of missing coverage in LTP at the moment https://github.com/linux-test-project/ltp/labels/missing%..., and these are just scratching the surface with most obviously missing syscalls. We may try to proceed with the coverage analysis once we are out of work there, which will hopefully happen at some point.

The problems with corner cases can be likely caught by combination of unit testing and fuzzing. Drivers testing is more problematic though, there is only so much you can do with qemu and emulated hardware. Proper driver testing needs a reasonably sized lab stacked with hardware and it's much more problematic to set up and maintain which is not going to happen unless somebody invests reasonable amount of resources into it. But there is light at the end of the tunnel as well, as far as I know Linaro has a big automated lab stacked with embedded hardware to run tests on, we are trying to tackle automated server grade hardware lab here in SUSE, and I'm pretty sure there is a lot more outside there just not that visible to the general public.

Kernel quality control, or the lack thereof

Posted Dec 10, 2018 12:57 UTC (Mon) by nix (subscriber, #2304) [Link] (3 responses)

Yeah -- and lcov won't help with the sorts of things this LWN post is talking about anyway. 100%-coverage filesystems could easily still have all these bugs, because they relate to specific states of the filesystem, and *no* coverage system could *possibly* track whether we got complete coverage of all possible corrupted filesystems! (Or, indeed, all possible states of the program: for all but the most trivial programs there are far too many.)

There is no alternative to thinking about these problems, I'm afraid. There is no magic automatable road to well-tested software of this complexity.

Kernel quality control, or the lack thereof

Posted Dec 10, 2018 13:14 UTC (Mon) by metan (subscriber, #74107) [Link] (2 responses)

Exactly there is nothing that would replace well thought tests written by senior developers, we only need to throw more manpower on the problem, which seems to be happening albeit slowly.

Kernel quality control, or the lack thereof

Posted Dec 11, 2018 17:37 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

Indeed if there is any part of the kernel this has really happened for, filesystems and in particular XFS must be it, and probably have the best test coverage of all. I mean, xfstests is called that for a *reason*. :) (I tell a lie: RCU has gone to the next step beyond this, formal model verification. Coming up with a formal model of XFS would be... a big job!)

Kernel quality control, or the lack thereof

Posted Dec 11, 2018 20:59 UTC (Tue) by marcH (subscriber, #57642) [Link]

Interesting, now the question is: how much did/do xfstests offer for the two specific features reported above?