Bugs per kLOC

Posted Dec 14, 2004 19:47 UTC (Tue) by man_ls (guest, #15091)
In reply to: Coverity's kernel code quality study by MathFox
Parent article: Coverity's kernel code quality study

The number Coverty presents can not be compared with the number of bugs found in other projects before Coverty fixes.

Well said. Bugs per thousand lines of code (kLOC) can only be evaluated as a relative number, since we cannot know:

if blank lines of code are counted,
if comments are counted either,
if coding style matters (lone '{'s or '}'s)...

Otherwise, we can only suppose it is non-blank, non-comment lines of code what we are counting (the usual industry standard); and play with broad estimates, which I will presently do for the fun of it.

The figure given by Carnegie Mellon University, 20 or 30 bugs per kLOC, is definitely not for released software, but probably for written software before any testing happens. After release, the number would rather be 1 to 5 bugs per kLOC in commercial software. For mission-critical code, the count can be as low as 0.1 bugs per kLOC (as in Shuttle software), depending on cricicity and budget. Project size is also a factor.

Of course the rate in Linux is lower than in "commercial enterprise software"; an operating system kernel arguably is mission-critical software. 0.17 bugs per kLOC looks like a lot, even if those bugs are in device drivers, or especially then since they can take down the whole system, corrupt data, etc. (I remember estimates for w2k were 2 bugs per kLOC after release, but that includes the whole operating system, not just the kernel.)

But there is more. Nobody would expect that, after fixing the 985 bugs, Linux would magically become error-free. So 0.17 bugs per kLOC must be a lowest-bound estimate; the real figure will be higher.

All in all, a poor press release with not much real value, but great promotion for the Stanford Code Checker.

to post comments

Bugs per kLOC

Posted Dec 14, 2004 23:02 UTC (Tue) by hppnq (guest, #14462) [Link]

I assume the Coverity program has a proper parser that allows for at least a proper comparison of the actual number of lines of code it has inspected. It must be much harder, for instance, to compare the complexity (that is inevitably related to the number of bugs found, I would say) of two programs.

But I fully agree with you that the press release reads more like a promotional flyer, which is a bit strange considering these people must know the tricks of the scientific trade.