Compared to commercial? [LWN.net]

Compared to commercial?

Posted Mar 6, 2006 21:40 UTC (Mon) by dwheeler (subscriber, #1216)
In reply to: Compared to commercial? by scripter
Parent article: Coverity releases first defect survey results

Good luck getting comparitive data. There is some, but it's not easy to get.

As noted in this article about Coverity, CMU data suggests that a 5.7MSLOC system should have over 5,000 defects, that is, 0.87 defects/KSLOC. Nearly all the projects had a smaller rate than that. If the CMU data is comparible to the Coverity results -- and that is a MIGHTY big if -- then nearly all the OSS projects are doing much better than average.

Although comparitive information from Coverity is hard to find, there is some comparitive data from Reasoning. See the reliability section of "Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers", which references the studies from Reasoning and Coverity.

If you know of more comparitive data, I'd love to hear about it.

(Log in to post comments)

Compared to commercial?

Posted Mar 7, 2006 7:24 UTC (Tue) by peterh (subscriber, #4225) [Link]

You'll probably find that it's fairly meaningless comparing bug data
generated by humans and that produced by software analysis tools like
the Coverity checker. As I understand it the analysis done by their checker
is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect). In some sense the checker is good at finding "obvious" bugs, but "deep" errors won't always be found. Low bug counts here don't necessarily compare meaningfully to the number of bugs in the code derived through some other means.

That said, there may well be a correlation between the sorts of bugs that the checker can detect and all bugs (ie. good programmers make fewer of both sorts of bug, bad programmers make more of both). Hence a comparison between software bug counts produced with the same tool is interesting.

Compared to commercial?

Posted Mar 7, 2006 13:50 UTC (Tue) by daniels (subscriber, #16193) [Link]

Actually, Coverity's checker is pretty thorough: it turns up stuff you wouldn't necessarily expect an automated checker to.

(But, of course, there are some false positives, and undoubtedly false negatives. But it's turning up stuff even a skilled human wouldn't necessarily get, and good luck finding someone sufficiently skilled, willing to review the entire X codebase.)

Compared to commercial?

Posted Mar 7, 2006 19:44 UTC (Tue) by kleptog (subscriber, #1183) [Link]

Obviously the checker itself needs to be taught how to deal with various aspects of the programs it checks. For example, for PostgreSQL it doesn't appear to recognise that elog(ERROR, ...) never returns, leading to many spurious warnings about using variables inappropriately.

It's still a really nice technology and could help track-down some of the more obscure bugs.

Compared to commercial?

Posted Mar 7, 2006 20:12 UTC (Tue) by daniels (subscriber, #16193) [Link]

That's just a matter of training, though: it at least appeared to recognise all the cases like that for X. The only false-negative I saw like that was some really hideous error-handling code involving longjmp.

Compared to commercial?

Posted Mar 8, 2006 5:32 UTC (Wed) by peterh (subscriber, #4225) [Link]

Yes, but you're missing the point. Automated checkers are supposed to be thorough --- they tend to find the sorts of bugs that you wouldn't expect to find yourself.

There's been a reasonable amount of research recently on using reasonably classical compiler-type program analyses, such as abstract interpretation and dataflow analyses to detect bugs (Metal/Coverity checker and Saturn out of Stanford, and I think Cousot and others were doing work on verification of aerospace systems in France, and there are no doubt more that I can't think of right now). The real innovation of Metal is that it has a "find bugs at all cost" mentality, irrespective of the theoretical soundness of what it does. The result is probably quite good as a tool for finding certain classes of bugs. But I doubt it's sensible to make conclusions about the total bug count of a program based on what the checker detects.

Compared to commercial?

Posted Mar 8, 2006 16:02 UTC (Wed) by samth (subscriber, #1290) [Link]

As I understand it the analysis done by their checker is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect).

I expect you are correct in your conclusions (no interesting checkers are complete, and most commercial ones are unsound), but you have sound and complete backwards. Sound means no false negatives, and complete means no false positives.