LWN: Comments on "Kernel quality control, or the lack thereof"

Kernel quality control, or the lack thereof

PaulMcKenney — Sat, 05 Jan 2019 22:29:35 +0000

Especially in my part of the Linux kernel, there is great value in preventing problems from reaching the -tip tree, let alone Linus's tree, let alone distributions, let alone customers. This great value stems from the fact that RCU bugs tend to be a bit difficult to reproduce and track down. It is therefore quite important to test the common cases.

Nevertheless, your last sentence is spot on. It is precisely because rcutorture forces rare code paths and rare race conditions to execute more frequently that the number of RCU bugs reaching customers is kept down to a dull roar.

Kernel quality control, or the lack thereof

joseph.h.garvin — Sat, 05 Jan 2019 18:06:03 +0000

I think you have things backwards. If there is a bug in commonly executed code it's going to be exposed even if there isn't a test. It's the infrequently executed code that tends to contain bugs (e.g. handling error conditions). Testing the frequently executed code still has value in that it can prevent problems from reaching customers, but bugs in frequently executed code will tend to be discovered very quickly. In a sense what the entire point of tests is is to make it so some code paths are more frequently executed.

Kernel quality control, or the lack thereof

PaulMcKenney — Sun, 30 Dec 2018 15:44:34 +0000

Assuming that timing considerations permitted it, one approach would be to run the code in a simulator that provided fault-injection for memory. That said, to your point, you would have to inject the fault rather carefully to trigger that particular set of assertions.

But if the point was in fact to warn about unreliable memory, mightn't this sort of fault injection nevertheless be quite useful?

Kernel quality control, or the lack thereof

GoodMirek — Sun, 30 Dec 2018 11:22:33 +0000

There are safety nets triggered only in cases impossible to reach via software means, but possible to happen in case of HW failure.
I saw that multiple times while working on embedded systems.

E.g.:
explosiveness=255;
if explosiveness !=255 then assert;

In theory, it should never assert. In reality, it is desirable to minimize a risk that 'explosiveness' variable is stored in a failed memory cell, prior that cell is used for indication of explosiveness of any kind.

Or this case:
if <green> then
explosiveness=0;
else
explosiveness=255;
if (explosiveness!=0 and explosiveness!=255) then assert;

It is very rare to trigger and almost impossible to test such assertions, but when I saw them triggered in reality, even once in a lifetime, I appreciated their merit.

Kernel quality control, or the lack thereof

PaulMcKenney — Tue, 25 Dec 2018 00:05:47 +0000

The safety-critical software that I know of (admittedly an obsolete and vanishingly small fraction of the total) limited "if" statements for exactly this reason.

But yes, Murphy will always be with us. So even in safety critical code, at the end of the day it is about reducing risk rather than completely eliminating it.

And to your point about Ariane 5's failed proof of correctness... Same issue as the classic failed proof of correctness for the binary search algorithm! Sadly, a proof of correctness cannot prove the assumptions on which it is based. So Murphy will always find a way, but it is nevertheless our job to thwart him. :-)

Kernel quality control, or the lack thereof

anton — Mon, 24 Dec 2018 20:42:02 +0000

I would expect that defensive coding practices that lead to unreachable code (and thus <100% coverage) are particularly widespread in safety-critical software. I.e., you cannot trigger this particular safety-net code, and you are pretty sure that it cannot be triggered, but not absolutely sure; or even if you are absolutely sure, you foresee that the safety net might become triggerable after maintenance. Will you remove the safety net to increase your coverage metric?

OTOH, how do you test your safety net? Remember that Ariane 5 was exploded by a safety net that was supposed (and proven) to never trigger.

Kernel quality control, or the lack thereof

Wol — Thu, 20 Dec 2018 12:26:58 +0000

That reminds me of a bug I found in Fortran 77 for Univac ... which led to data loss.

The guys at the company I was contracted to were trying to abstract out the OS-specific features, like in the OPEN statement. You could declare a file as temporary, which led to it disappearing when it got closed - quite a nice feature IF IMPLEMENTED CORRECTLY!

So, I opened a temporary file on a FUNIT, then later on re-used the same FUNIT for a permanent file. THE OS DIDN'T CLEAR THE TEMPORARY STATUS. So when I closed the permanent file, it disappeared ... :-)

Cheers,
Wol

Kernel quality control, or the lack thereof

PaulMcKenney — Thu, 20 Dec 2018 04:37:01 +0000

Dr. Who has to do that every few years as the actors change?

Kernel quality control, or the lack thereof

neilbrown — Thu, 20 Dec 2018 03:45:16 +0000

Isn't it a simple matter of writing a virus (or a worm .. or a worm with a virus) which hunts out all bugs on the Internet and removes them. I remember Doctor Who did that once to get rid of all the photos of himself, so it can't be too hard.

Kernel quality control, or the lack thereof

PaulMcKenney — Wed, 19 Dec 2018 16:04:38 +0000

In the old days, I would have agreed. But these days, "rm" can often be undone using "git checkout", or, in more extreme cases, "git clone". ;-)

Kernel quality control, or the lack thereof

mathstuf — Wed, 19 Dec 2018 13:43:22 +0000

I prefer `rm` for that job. Works really well. ;)

Kernel quality control, or the lack thereof

PaulMcKenney — Tue, 18 Dec 2018 14:05:57 +0000

If you didn't know better, you might think that there is no magic bullet to take out all bugs. :-)

Kernel quality control, or the lack thereof

error27 — Tue, 18 Dec 2018 11:46:02 +0000

Single return coding style introduces "forgot to set the error code" bugs. A "goto out;" might do a ton of things or it might not do anything so it is a mystery, but a "return -EINVAL;" is unambiguous.

Kernel quality control, or the lack thereof

marcH — Tue, 11 Dec 2018 20:59:21 +0000

Interesting, now the question is: how much did/do xfstests offer for the two specific features reported above?

Kernel quality control, or the lack thereof

marcH — Tue, 11 Dec 2018 20:53:55 +0000

> Hey, you asked! :-)

Sincere thanks!

Kernel quality control, or the lack thereof

nix — Tue, 11 Dec 2018 17:37:54 +0000

Indeed if there is any part of the kernel this has really happened for, filesystems and in particular XFS must be it, and probably have the best test coverage of all. I mean, xfstests is called that for a *reason*. :) (I tell a lie: RCU has gone to the next step beyond this, formal model verification. Coming up with a formal model of XFS would be... a big job!)

Kernel quality control, or the lack thereof

PaulMcKenney — Tue, 11 Dec 2018 14:11:49 +0000

I have seen 80% used with some decent results. But it really depends on the code and its user base. 100% of your commonly executed code really does need to be covered. But of course the more users your code has, the larger the fraction of your code is commonly executed.

If only (say) 30% of your code is tested, you very likely need to substantially increase your coverage. If (say) 90% of your code is tested, there is a good chance that there is some better use of your time than getting to 91%. But for any rule of thumb like these, there will be a great many exceptions, for example, the safety-critical code mentioned earlier.

Hey, you asked! :-)

Kernel quality control, or the lack thereof

marcH — Tue, 11 Dec 2018 07:37:27 +0000

> Managers look at the % numbers and it causes developers to rearrange code to have a single return

Interesting, could you share a simplified example?

> My experience with code coverage metrics has been mostly negative.

While error-handling code, corner cases and... backup configurations are notoriously untested, I agree there are diminishing returns and better trade-offs past some point. Curious what is experts' guestimation of where that percentage typically is.

Kernel quality control, or the lack thereof

JdGordy — Tue, 11 Dec 2018 05:51:38 +0000

Meanwhile, MISRA coding standards require single returns... :/

Kernel quality control, or the lack thereof

PaulMcKenney — Mon, 10 Dec 2018 21:50:04 +0000

And anything less than 100% race coverage similarly guarantees a hole in your testing. As does anything less than 100% configuration-combination coverage. As does anything less than 100% input coverage. As does anything less than 100% hardware-configuration testing. As does ...

For most types of software, at some point it becomes more important to test more races, more configurations, more input sequences, and more hardware configurations than to provide epsilon increase in coverage by triggering that next assertion. After all, testing and coverage is about reducing risk given the time and resources at hand. Therefore, over-emphasizing one form of testing (such as coverage) will actually increase overall risk due to the consequent neglect of some other form of testing.

Of course, there are some types of software where 100% coverage is reasonable, for example, certain types of safety-critical software. But in this case, you will be living under extremely strict coding standards so as to (among a great many other things) make 100% coverage affordable.

Kernel quality control, or the lack thereof

NAR — Mon, 10 Dec 2018 21:18:20 +0000

But anything less than 100% coverage guarantees that some part of the code is not tested...

Kernel quality control, or the lack thereof

sandeen — Mon, 10 Dec 2018 20:35:49 +0000

"One hopes that these test will be added to a well maintained test project"

You may wish to subscribe to fstests@vger.kernel.org or peruse git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git if this sort of thing is of interest to you.

Kernel quality control, or the lack thereof

PaulMcKenney — Mon, 10 Dec 2018 20:21:38 +0000

I certainly have seen this. And it can be even worse, due to penalizing good assertions and discouraging debugging code. So oddly enough, one reason why coverage is not a panacea is exactly because some people believe that it is in fact a panacea! :-)

Kernel quality control, or the lack thereof

shemminger — Mon, 10 Dec 2018 19:59:55 +0000

My experience with code coverage metrics has been mostly negative.
Managers look at the % numbers and it causes developers to rearrange code to have a single return to maximize the numbers.

Kernel quality control, or the lack thereof

PaulMcKenney — Mon, 10 Dec 2018 18:42:16 +0000

Code coverage is table stakes in this game, if that. Even 100% code coverage won't guarantee that you exercised the relevant race conditions, patterns of data on mass storage, combinations of kernel-boot parameters and Kconfig options, and so forth. For but one example, 100% code coverage in a CONFIG_PREEMPT=n kernel would not find a bug due to inopportune preemption that could happen in a CONFIG_PREEMPT=y kernel.

Don't get me wrong, code coverage is fine as far as it goes, and it seems likely that the Linux kernel community would do well to do more of it, but it is not a panacea. In particular, beyond a certain point, it is probably not the best place to put your testing effort.

Kernel quality control, or the lack thereof

xorbe — Mon, 10 Dec 2018 14:43:08 +0000

They need code coverage metrics, not just "billions of operations over a period of days."

Kernel quality control, or the lack thereof

metan — Mon, 10 Dec 2018 13:14:54 +0000

Exactly there is nothing that would replace well thought tests written by senior developers, we only need to throw more manpower on the problem, which seems to be happening albeit slowly.

Kernel quality control, or the lack thereof

nix — Mon, 10 Dec 2018 12:57:27 +0000

Yeah -- and lcov won't help with the sorts of things this LWN post is talking about anyway. 100%-coverage filesystems could easily still have all these bugs, because they relate to specific states of the filesystem, and *no* coverage system could *possibly* track whether we got complete coverage of all possible corrupted filesystems! (Or, indeed, all possible states of the program: for all but the most trivial programs there are far too many.)

There is no alternative to thinking about these problems, I'm afraid. There is no magic automatable road to well-tested software of this complexity.

Kernel quality control, or the lack thereof

metan — Mon, 10 Dec 2018 09:49:40 +0000

The tool to generate coverage for kernel is maintained at https://github.com/linux-test-project/lcov it should work, but I haven't tried it.

However I can pretty much say that the main problems I see are various corner cases that are rarely hit (i.e. mostly failures and error propagation) and drivers. My take on this is that there is no point in doing coverage analysis when the gaps we have are enormous and easy to spot. Just have a look at our backlog of missing coverage in LTP at the moment https://github.com/linux-test-project/ltp/labels/missing%..., and these are just scratching the surface with most obviously missing syscalls. We may try to proceed with the coverage analysis once we are out of work there, which will hopefully happen at some point.

The problems with corner cases can be likely caught by combination of unit testing and fuzzing. Drivers testing is more problematic though, there is only so much you can do with qemu and emulated hardware. Proper driver testing needs a reasonably sized lab stacked with hardware and it's much more problematic to set up and maintain which is not going to happen unless somebody invests reasonable amount of resources into it. But there is light at the end of the tunnel as well, as far as I know Linaro has a big automated lab stacked with embedded hardware to run tests on, we are trying to tackle automated server grade hardware lab here in SUSE, and I'm pretty sure there is a lot more outside there just not that visible to the general public.

Kernel quality control, or the lack thereof

marcH — Sun, 09 Dec 2018 17:28:55 +0000

Features or security? Sad but the priority has to be the former to do business. Have fewer more secure features and you lose in the market place almost every time.

Thinking of it computer security is a bit like... healthcare: extremely opaque and nearly impossible for customers to make educated choices about it. From a legal perspective I suspect it's even worse, breach after breach and absolutely zero liability. To top it up class actions are no longer, killed by arbitration clauses in all Terms and Conditions. Brands might be more useful in security though.

https://www.google.com/search?q=site%3Aschneier.com+liabi...

Kernel quality control, or the lack thereof

saffroy — Sun, 09 Dec 2018 14:20:29 +0000

Fuzzing is extremely useful, but it still needs a *thinking* developer to help it generate interesting cases in reasonable time.

Besides tests themselves, it helps a LOT to have some kind of test coverage report, just to remind you of which parts of the code are never touched by any of your current tests.

Do people publish such coverage reports for the kernel?

Kernel quality control, or the lack thereof

mupuf — Sun, 09 Dec 2018 13:32:05 +0000

> Validation and automation have a lesser reputation than development and tend to attract less talent. One possible and extremely simple way to address this is to treat the *development* of tests and automation to the same open-source and code review standards.

This is what we do in the i915 community. No feature lands in DRM without a test in IGT, and CI developers are part of the same team.

My view on this is that good quality comes from:
1) Well written driver code, peer reviewed to catch architectural issues
2) Good tests exercising the main use case, and corner cases. Tests are considered at the same level as driver code.
3) Good understand of the CI system that will execute these tests
4) Good following of the bugs filed when these tests fail

Point 1) is pretty much well done in the Linux community.

Point 2) is hard to justify when tests are not executed, but comes more naturally when we have a good CI system

Point 3) is probably the biggest issue for the Linux CI systems: The driver usually covers a wide variety of HW and configuration which cannot all be tested in CI at all time. This leads to complexity in the CI system that needs to be understood by developers in order to prevent regressions. This is why our CI is maintained and developed in the same team developing the driver.

Point 4) is coming pretty naturally when introducing a filtering system for CI failures. Some failures are known and pending fixing, and we do not want these to be considered as blocking for a patch series. We have been using bugs to create a forum of discussion for developers to discuss how to fix these issues. These bugs are associated to CI failures by a tool doing pattern matching (https://intel-gfx-ci.01.org/cibuglog/). The problem is that these bugs are now every developer's responsibility to fix, and that requires a change in the development culture to hold up some new features until some more important bugs are fixed.

I guess we are getting quite good at CI, and I am really looking forward to us in the CI team to have more time to share our knowledge and tools for others to replicate! We have already started working on an open source toolbox for CI (https://gitlab.freedesktop.org/gfx-ci), as discussed at XDC 2018 (https://xdc2018.x.org/slides/GFX_Testing_Workshop.pdf).

Kernel quality control, or the lack thereof

iabervon — Sun, 09 Dec 2018 11:17:28 +0000

I think, for this case, fuzzing is probably more useful that developer-written tests. If a developer misses the code for some checks necessary to maintain security constraints, what are the chances they'll write tests that verify that using the API in a way they didn't intend doesn't violate security constraints they didn't think about? I'd be more convinced if they taught a fuzzing framework how to call their API and set it loose on a filesystem with a lot of interesting cases. I care somewhat less that it does what it's supposed to do than that whatever it actually does is something the caller is allowed to do.

Kernel quality control, or the lack thereof

luto — Sun, 09 Dec 2018 01:46:34 +0000

This reminds me of a bug I found once. In a -rc4 kernel, I decided to play with the new O_TMPFILE feature. My first basic experiment resulted in fs corruption:

https://lwn.net/Articles/562296/

Kernel quality control, or the lack thereof

marcH — Sat, 08 Dec 2018 16:45:31 +0000

> And, tests that aren't run don't really exists, for all practical purposes.

Agreed 200%, this is the core issue:

> > We ended up here because we *trusted* that ...

Either tests already exist and it's just the matter of the extra mile to automate them and share their results.

Or there's no decent, repeatable and re-usable test coverage and new features should simply not be added until there is. "Thanks your patches looks great, now where are your tests results please?". Not exactly ground-breaking software engineering.

Exceptions could be tolerated for hardware-specific or pre-silicon drivers which require very specific test environments and for which vendors can only hurt themselves anyway. That clearly doesn't seem the case of XFS or the VFS.

Validation and automation have a lesser reputation than development and tend to attract less talent. One possible and extremely simple way to address this is to treat the *development* of tests and automation to the same open-source and code review standards.

Kernel quality control, or the lack thereof

vomlehn — Sat, 08 Dec 2018 01:20:00 +0000

One hopes that these test will be added to a well maintained test project, though such projects are even less common than a well maintained development project. LTP comes to mind but there may be other possibilities. And, tests that aren't run don't really exists, for all practical purposes.

Kernel quality control, or the lack thereof

mgross — Fri, 07 Dec 2018 20:38:57 +0000

This is a hard problem. I'm glad to see it getting attention.

Kernel quality control, or the lack thereof

johannbg — Fri, 07 Dec 2018 20:27:21 +0000

Ah the inevitable cutting the ( legacy ) fat off and calling a flag day draws near with everyone must upgrading to that kernel release when that day comes.

Dare I predict 2020 will be the year when that day comes in the spirit of Jon predictions will soon be upon us ;)

Kernel quality control, or the lack thereof

zblaxell — Fri, 07 Dec 2018 19:21:52 +0000

> users could overwrite a setuid file without resetting the setuid bits, time stamps would not be updated,

I always have to read these patch sets to make sure that when the above are features, they aren't getting removed.

There's a "if (!is_dedupe)" in the code, so it's all good.