LWN: Comments on "Coverity's kernel code quality study"

*BSD, so what?

freethinker — 2004-12-16T00:46:51+00:00

Don't feed the trolls :)

*BSD, so what?

nix — 2004-12-15T13:17:37+00:00

Niche software doesn't deserve broad attention.

What a parochial point of view. You realise the same `reasoning' could be applied by Windows people to Linux and the MacOS?

Everything deserves attention, if just to find good ideas in it and reuse them elsewhere.

Coverity's kernel code quality study

alexs — 2004-12-15T08:26:18+00:00

aside to the promotion
this press release shows
how a code project can improve
over some period of time
when a well tuned toolset is applied.

i suppose that a lot of kernel improvements
of the near and far future will be the result
of improved gnu compilers and possible assisted
by the occasional usage of the intel compilers.
stanford checker might decrease in his contribution
to the fixing rate as things tend to get tougher
in a phase where the overall amount of bugs in
the software is magnitudes below the normal rate.

lets give you some "troll bait": Linux is mature now. ;-)

Coverity's kernel code quality study

emkey — 2004-12-15T00:48:23+00:00

Shouldn't the holy grail be knowing exactly where all those bugs are? :-)

Coverity's kernel code quality study

MathFox — 2004-12-15T00:42:45+00:00

You should realise that any (automated or manual) bug-checking process only finds a subset of the bugs that exist in a program. There ain't no silver bullet! The Stanford/Coverity bug checker will only find some of the bugs and be blind for the others.
Running an automated checker on a program will find a lot of bugs in the first run... but it will not find some bugs that hide in the bling spot of the checker. You can fix the bugs that the scanner finds and rerun it; but it can not help you with the bugs that it is blind for. So there will allways be an unknown(!) number of bugs left after a perfect scan.

It makes more sense to compare the numbers of bugs that were found on the first run of the scanner between different projects, or the total number of bugs spotted by the scanner instead of the number of <residual> bugs that the scanner finds after several iterations of bug fixing.

The holy grail in software testing is knowing the exact number of bugs in the product. :)

*BSD, so what?

gvy — 2004-12-15T00:15:12+00:00

May I reiterate that it's BSDs' problem an not anyone else's, deterring people off development and not providing incentive to cooperate?

I bet someone of that crowd whined in Ladislav's inbox to make his banner on distrowatch.com include "BSD" which is totally irrelevant there IMHO.

And I've personally taken enough whining instead of people doing something real to promote their favours to have this approach to that crowd, sorry.

Niche software doesn't deserve broad attention. Especially when providing nothing new to this world.

Coverity's kernel code quality study

brouhaha — 2004-12-14T23:58:41+00:00

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.

Certainly it CAN be compared. If you were going to choose an OS today, would you decide to ignore the higher bug count of <some proprietary os> just because the owner hasn't had those Coverity fixes but might get them (or the equivalent) in the future?

Bugs per kLOC

hppnq — 2004-12-14T23:02:22+00:00

I assume the Coverity program has a proper parser that allows for at least a proper comparison of the actual number of lines of code it has inspected. It must be much harder, for instance, to compare the complexity (that is inevitably related to the number of bugs found, I would say) of two programs.

But I fully agree with you that the press release reads more like a promotional flyer, which is a bit strange considering these people must know the tricks of the scientific trade.

OpenBSD

JoeBuck — 2004-12-14T22:07:16+00:00

There's a problem with comparisons, though. Coverity started off as the "Stanford Checker", and those guys have been working with the Linux kernel for years, feeding back bugs as they find them. Not as much work has been done on the BSDs.

OpenBSD

error27 — 2004-12-14T20:34:08+00:00

Someone did a comparison and BSD was far worse.

But most of the bugs were in one small area of the code that wasn't used much... Compatability for something.

Bugs per kLOC

man_ls — 2004-12-14T19:47:00+00:00

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.

Well said. Bugs per thousand lines of code (kLOC) can only be evaluated as a relative number, since we cannot know:

if blank lines of code are counted,
if comments are counted either,
if coding style matters (lone '{'s or '}'s)...

Otherwise, we can only suppose it is non-blank, non-comment lines of code what we are counting (the usual industry standard); and play with broad estimates, which I will presently do for the fun of it.

The figure given by Carnegie Mellon University, 20 or 30 bugs per kLOC, is definitely not for released software, but probably for written software before any testing happens. After release, the number would rather be 1 to 5 bugs per kLOC in commercial software. For mission-critical code, the count can be as low as 0.1 bugs per kLOC (as in Shuttle software), depending on cricicity and budget. Project size is also a factor.

Of course the rate in Linux is lower than in "commercial enterprise software"; an operating system kernel arguably is mission-critical software. 0.17 bugs per kLOC looks like a lot, even if those bugs are in device drivers, or especially then since they can take down the whole system, corrupt data, etc. (I remember estimates for w2k were 2 bugs per kLOC after release, but that includes the whole operating system, not just the kernel.)

But there is more. Nobody would expect that, after fixing the 985 bugs, Linux would magically become error-free. So 0.17 bugs per kLOC must be a lowest-bound estimate; the real figure will be higher.

All in all, a poor press release with not much real value, but great promotion for the Stanford Code Checker.

OpenBSD

freethinker — 2004-12-14T18:36:44+00:00

I wonder what a similar study would find in OpenBSD. I found some references to 2000-odd bugs found in Linux and OpenBSD at some time in the past, but no breakdown.

Coverity's kernel code quality study

MathFox — 2004-12-14T18:21:53+00:00

My take on the story is:

After four years of pestering the Linux coders with "Stanford code checker" reports, we hardly find any bugs with our code checker (which is the rebadged Stanford checker).

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.

Coverity's kernel code quality study

southey — 2004-12-14T18:14:38+00:00

Note that 54% are in device drivers and 41% are due to NULL pointer dereferences. So Linux is probably has an even better quality than it first appears.