Sponsored link
Vyatta –
Linux & Open Source
Alternative to Cisco –
Advanced Routing,
Firewall, VPN, QoS..
Free Download ->
| |||||||||||
Compared to commercial?Compared to commercial?Posted Mar 7, 2006 7:24 UTC (Tue) by peterh (subscriber, #4225)In reply to: Compared to commercial? by dwheeler Parent article: Coverity releases first defect survey results
You'll probably find that it's fairly meaningless comparing bug data
That said, there may well be a correlation between the sorts of bugs that the checker can detect and all bugs (ie. good programmers make fewer of both sorts of bug, bad programmers make more of both). Hence a comparison between software bug counts produced with the same tool is interesting.
(Log in to post comments)
Compared to commercial? Posted Mar 7, 2006 13:50 UTC (Tue) by daniels (subscriber, #16193) [Link] Actually, Coverity's checker is pretty thorough: it turns up stuff you wouldn't necessarily expect an automated checker to.
(But, of course, there are some false positives, and undoubtedly false negatives. But it's turning up stuff even a skilled human wouldn't necessarily get, and good luck finding someone sufficiently skilled, willing to review the entire X codebase.)
Compared to commercial? Posted Mar 7, 2006 19:44 UTC (Tue) by kleptog (subscriber, #1183) [Link] Obviously the checker itself needs to be taught how to deal with various aspects of the programs it checks. For example, for PostgreSQL it doesn't appear to recognise that elog(ERROR, ...) never returns, leading to many spurious warnings about using variables inappropriately.
It's still a really nice technology and could help track-down some of the more obscure bugs.
Compared to commercial? Posted Mar 7, 2006 20:12 UTC (Tue) by daniels (subscriber, #16193) [Link] That's just a matter of training, though: it at least appeared to recognise all the cases like that for X. The only false-negative I saw like that was some really hideous error-handling code involving longjmp.
Compared to commercial? Posted Mar 8, 2006 5:32 UTC (Wed) by peterh (subscriber, #4225) [Link] Yes, but you're missing the point. Automated checkers are supposed to be thorough --- they tend to find the sorts of bugs that you wouldn't expect to find yourself.
There's been a reasonable amount of research recently on using reasonably classical compiler-type program analyses, such as abstract interpretation and dataflow analyses to detect bugs (Metal/Coverity checker and Saturn out of Stanford, and I think Cousot and others were doing work on verification of aerospace systems in France, and there are no doubt more that I can't think of right now). The real innovation of Metal is that it has a "find bugs at all cost" mentality, irrespective of the theoretical soundness of what it does. The result is probably quite good as a tool for finding certain classes of bugs. But I doubt it's sensible to make conclusions about the total bug count of a program based on what the checker detects.
Compared to commercial? Posted Mar 8, 2006 16:02 UTC (Wed) by samth (subscriber, #1290) [Link] As I understand it the analysis done by their checker is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect).I expect you are correct in your conclusions (no interesting checkers are complete, and most commercial ones are unsound), but you have sound and complete backwards. Sound means no false negatives, and complete means no false positives.
|
|||||||||||