| |||||||||||||
Coverity releases first defect survey results
Coverity, which has a contract with the U.S. Department of Homeland Security to investigate the defect rates in a number of free software projects, has announced the availability of its first set of results. "The LAMP stack -- Linux, Apache, MySQL, and Perl/PHP/Python -- showed
significantly better software quality above the baseline with an average of
0.290 defects per thousand lines of code compared to an average of 0.434 for
the 32 open source software projects analyzed." A table of results is available, with more details for those who register with the site. A quick glance shows a few projects with high bug rates (Amanda, Firebird, net-snmp, X) and others with quite low rates (gcc, openvpn, python, perl). On the other hand, ethereal shows a very low defect rate, which can be hard to square with the long list of security advisories from that project.
(Log in to post comments)
Coverity releases first defect survey results Posted Mar 6, 2006 15:23 UTC (Mon) by kirkengaard (subscriber, #15022) [Link] It'd be better of we knew their operational criteria for "defect".
Defect criteria Posted Mar 6, 2006 15:29 UTC (Mon) by corbet (editor, #1) [Link] I'd say a "defect" is a bug which can be detected by Coverity's tools.As an added note, the first set of reports for the kernel has made it to some of the developers, and patches are already circulating.
Defect criteria Posted Mar 6, 2006 15:45 UTC (Mon) by kirkengaard (subscriber, #15022) [Link] I had guessed that much. :) I'm just a greedy little monkey when it comes to details.
I find it interesting to note that the 2.6 kernel "blows clean" on the defects table, since there are obviously some to be reported.
Found on linux-kernel Posted Mar 6, 2006 16:39 UTC (Mon) by kirkengaard (subscriber, #15022) [Link] Date Mon, 6 Mar 2006 00:49:59 -0500From Dave Jones <> Subject Re: Coverity Open Source Defect Scan of Linux
On Sun, Mar 05, 2006 at 09:35:11PM -0800, Ben Chelf wrote:
> Right now, we're guarding access to the actual defects that we report
The last time I asked about access to your bug list, I was asked to
Dave
--
Defect criteria Posted Mar 6, 2006 23:01 UTC (Mon) by petegn (guest, #847) [Link] Thas all very well apart from the fact that evry single piece of code can be written in several ways and still achieve the same result .
So that said is it a fault or just another way they had not dreamt up of doing things ..
PS i amnot a programmer but have been around computers long enough to know that there is a lot more than one way of skining a billy goat (Gates)
Pete .
Defect criteria Posted Mar 7, 2006 7:31 UTC (Tue) by man_ls (subscriber, #15091) [Link] Yeah, just being "around computers" helps a lot to understand programming. Well, now that I think about it, my boss has been around computers for all of his working career and he knows nil about programming (for him "refactoring" is a four letter word). And my boss's boss, too. And my family. And some of my friends too!
Well, now that I think about it: just being "around computers" does not help when it comes to programming. Please download some Sorry for feeding the troll, folks, but it's the first message I read from petegn that does not contain obscenities so I felt compelled to answer.
Defect criteria Posted Mar 8, 2006 0:50 UTC (Wed) by nix (subscriber, #2304) [Link] But it *does* have more stupid gratuitous Microsoft-bashing. (This example makes even less sense than usual.)
Compared to commercial? Posted Mar 6, 2006 15:29 UTC (Mon) by scripter (subscriber, #2654) [Link] How does the OSS defect rate compare to "average" (if there is such a thing) proprietary software?
Coverity has, in my opinion, one of the better static code analysis tools available. People tend to value Coverity Prevent because of the low rate of false positives produced -- the defects it reports usually need to be fixed. However, Coverity Prevent can't catch many types of architectural vulnerabilities. For that, human analysis is required.
Average proprietary defect rate? Posted Mar 6, 2006 15:51 UTC (Mon) by kirkengaard (subscriber, #15022) [Link] For that, you'd have to get them to air their dirty laundry. To some trustworthy third-party willing to actually disclose the figures they come up with, rater than the ones they were paid for. Coverity does seem like a good party for that, but just try and get the closed-source, all-for-profit PR departments to buy into not-guaranteed-to-be-good advertising that compares them to their competitors directly, from which users might draw negative conclusions about their software quality.
Compared to commercial? Posted Mar 6, 2006 21:40 UTC (Mon) by dwheeler (guest, #1216) [Link] Good luck getting comparitive data. There is some, but it's not easy to get.As noted in this article about Coverity, CMU data suggests that a 5.7MSLOC system should have over 5,000 defects, that is, 0.87 defects/KSLOC. Nearly all the projects had a smaller rate than that. If the CMU data is comparible to the Coverity results -- and that is a MIGHTY big if -- then nearly all the OSS projects are doing much better than average. Although comparitive information from Coverity is hard to find, there is some comparitive data from Reasoning. See the reliability section of "Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers", which references the studies from Reasoning and Coverity. If you know of more comparitive data, I'd love to hear about it.
Compared to commercial? Posted Mar 7, 2006 7:24 UTC (Tue) by peterh (subscriber, #4225) [Link] You'll probably find that it's fairly meaningless comparing bug datagenerated by humans and that produced by software analysis tools like the Coverity checker. As I understand it the analysis done by their checker is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect). In some sense the checker is good at finding "obvious" bugs, but "deep" errors won't always be found. Low bug counts here don't necessarily compare meaningfully to the number of bugs in the code derived through some other means.
That said, there may well be a correlation between the sorts of bugs that the checker can detect and all bugs (ie. good programmers make fewer of both sorts of bug, bad programmers make more of both). Hence a comparison between software bug counts produced with the same tool is interesting.
Compared to commercial? Posted Mar 7, 2006 13:50 UTC (Tue) by daniels (subscriber, #16193) [Link] Actually, Coverity's checker is pretty thorough: it turns up stuff you wouldn't necessarily expect an automated checker to.
(But, of course, there are some false positives, and undoubtedly false negatives. But it's turning up stuff even a skilled human wouldn't necessarily get, and good luck finding someone sufficiently skilled, willing to review the entire X codebase.)
Compared to commercial? Posted Mar 7, 2006 19:44 UTC (Tue) by kleptog (subscriber, #1183) [Link] Obviously the checker itself needs to be taught how to deal with various aspects of the programs it checks. For example, for PostgreSQL it doesn't appear to recognise that elog(ERROR, ...) never returns, leading to many spurious warnings about using variables inappropriately.
It's still a really nice technology and could help track-down some of the more obscure bugs.
Compared to commercial? Posted Mar 7, 2006 20:12 UTC (Tue) by daniels (subscriber, #16193) [Link] That's just a matter of training, though: it at least appeared to recognise all the cases like that for X. The only false-negative I saw like that was some really hideous error-handling code involving longjmp.
Compared to commercial? Posted Mar 8, 2006 5:32 UTC (Wed) by peterh (subscriber, #4225) [Link] Yes, but you're missing the point. Automated checkers are supposed to be thorough --- they tend to find the sorts of bugs that you wouldn't expect to find yourself.
There's been a reasonable amount of research recently on using reasonably classical compiler-type program analyses, such as abstract interpretation and dataflow analyses to detect bugs (Metal/Coverity checker and Saturn out of Stanford, and I think Cousot and others were doing work on verification of aerospace systems in France, and there are no doubt more that I can't think of right now). The real innovation of Metal is that it has a "find bugs at all cost" mentality, irrespective of the theoretical soundness of what it does. The result is probably quite good as a tool for finding certain classes of bugs. But I doubt it's sensible to make conclusions about the total bug count of a program based on what the checker detects.
Compared to commercial? Posted Mar 8, 2006 16:02 UTC (Wed) by samth (subscriber, #1290) [Link] As I understand it the analysis done by their checker is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect).I expect you are correct in your conclusions (no interesting checkers are complete, and most commercial ones are unsound), but you have sound and complete backwards. Sound means no false negatives, and complete means no false positives.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 6, 2006 15:33 UTC (Mon) by nix (subscriber, #2304) [Link] Part of this is a question of the severity of any given bug. Ethereal runs with high privileges and listens to the network, so that virtually any crash or hang can be parlayed into a remote attack: X doesn't listen so promiscuously (on most distros), so its high rate is comparatively safe; and GCC's low rate doesn't help much, as it's so widely used and so large that even a low rate of bugs-per-line translates into a large number of actual bugs (although few are security holes because not many things other than the cursed libtool --mode=install ever run GCC as root).
(In any case, the more complex the code, the less likely it is that a static checker like Coverity's will isolate real faults without special understanding of that project. It has special understanding of the Linux kernel's idioms, IIRC, but it certainly doesn't have special understanding of the idioms used in GCC , say, so it can't find most of the classes of flaws that are really found in GCC. Of course, if Coverity was free software, it could be extended by the maintainers of specific projects so that it could find flaws specific to those projects. But, oh, look, it isn't, despite early promises that it would be. So it'll never be as good as it could be in those areas.)
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 6, 2006 16:11 UTC (Mon) by shahms (subscriber, #8877) [Link] From reading the articles about Coverity's static checker (at least, the articles on the early versions of the Stanford checker), there is no reason to suspect they haven't written domain-specific extensions for at least some of the idioms and API rules of the projects they checked. One of the main points of their tool is making "system-specifc programmer written compiler extensions" relatively easy. The architecture described by those papers is powerful and extensible, able to detect and act on almost-arbitrary pattern rules. It's really a pretty nice architecture as described and I imagine the Coverity tool addresses the most urgent shortcomings of the early implementation.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 6, 2006 19:49 UTC (Mon) by Ross (subscriber, #4065) [Link] I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number. Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal".
I think the difference may be that X is much bigger than the server. The bug may simply have been in other places than the part exposed to the network. Also, the X server normally uses an authentication mechanism like MIT Magic Cookies which will limit the exposure of bugs to a very small portion of the code if the shared secret is not known to the attacker. (Assuming the cookies aren't weak and that the server throttles brute force attacks.)
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 6, 2006 20:17 UTC (Mon) by nijhof (subscriber, #4034) [Link] Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal".It is certainly "normal" for Debian -- you have to take out the --nolisten tcp manually if you want it to listen to tcp traffic
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 6, 2006 21:49 UTC (Mon) by Ross (subscriber, #4065) [Link] That's strange because I remember doing specifically on my Debian system. But obviously my memory is not infallible; I didn't even remember the option correctly. It is in the form you posted, though I just checked the man page and it is only prefixed with one hyphen.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 5:24 UTC (Tue) by mattdm (subscriber, #18) [Link] I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number. Check out Fedora Core, then. Still runs as root (although possibly limited by SELinux), but doesn't listen on TCP by default. Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal". -nolisten tcp with X.org. Actually, I think this *is* the default in the upstream Gnome GDM, so....
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 13:51 UTC (Tue) by daniels (subscriber, #16193) [Link] > I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number.
I can't think of one off the top of my head that does listen for TCP connections by default.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 15:22 UTC (Tue) by grouch (guest, #27289) [Link] Do you mean '-nolisten tcp'?
From my default /etc/X11/xinit/xserverrc file on Debian:
exec /usr/bin/X11/X -dpi 100 -nolisten tcp
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 18:33 UTC (Tue) by oak (subscriber, #2786) [Link] SUSE X servers don't listen on TCP sockets by default either.(I don't think this has been the case always, I'd guess it to have changed within last couple of years.)
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 8, 2006 5:52 UTC (Wed) by djm (subscriber, #11651) [Link] OpenBSD isn't a "distribution", but it has privilege separated Xorg so the vast majority no longer runs as root. Disabling the TCP listener isn't so much of a win as it breaks ssh forwarding of X sessions, and because X11's authentication is pretty good (barring chumps who run "xhost +").
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 8:34 UTC (Tue) by rwild (guest, #16004) [Link] > not many things other than the cursed libtool --mode=install ever run GCC as root
Bugs notwithstanding, you should be able to prevent that mostly by doing a DESTDIR install as user, and copying over and libtool --mode=finish as root.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 8, 2006 0:49 UTC (Wed) by nix (subscriber, #2304) [Link] Indeed, although I'm not sure what effect this has on things that want to install with unusual userids (and there are some).
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 9, 2006 10:25 UTC (Thu) by rwild (guest, #16004) [Link] Yes (therefore the "mostly"). I usually "make install -k DESTDIR=.." then and fix the setuid manually. Not so great. Better to get fast-install mode fixed in Libtool on more systems (which avoids the relinking in the first place).
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 7, 2006 19:10 UTC (Tue) by Junior_Samples (guest, #26737) [Link] "In any case, the more complex the code, the less likely it is that a static checker like Coverity's will isolate real faults without special understanding of that project."Au contraire. Complexity in and of itself is a defect. The measurement of complexity is considered the one of best metrics of software quality there is. It has been shown time and again that defects grow exponentially as complexity is increased. The fact that an automated parser should struggle with code which is "more complex" and full of obscure idioms should be proof enough that the code is defective. Sadly, use of complexity analysis seems to be completely absent in the OSS domain despite the fact that basic measurement of complexity using McCabe's or Halstead's methods are very straightforward, and freely available automated parsers exist. The Linux Journal has an article about one such tool for Python.
Rate of bugs and rate of security holes are mostly uncorrelated Posted Mar 8, 2006 0:54 UTC (Wed) by nix (subscriber, #2304) [Link] Code which is complex is only defective if it's doing a simple job despite that complexity.
Things like GCC, say, are doing very complex jobs: now perhaps GCC is more complex than it needs to be if all were ideal, but there's no way you'll ever get it simple enough to make static analysis easy. It does a very complex job.
(And so do the knottier parts of the kernel.)
Coverity releases first defect survey results Posted Mar 6, 2006 19:11 UTC (Mon) by jwb (guest, #15467) [Link] Ethereal's vulnerabilities are almost always in plug-in code modules. Some random person will contribute a dissector for, say, the protocol used by AIM. The bugs nearly all take the form of unprotected string copies into arrays of type char, either using a length that should be considered untrusted, or by null-termination.
Maybe the Coverity checker didn't look at the plug-ins.
Coverity releases first defect survey results Posted Mar 6, 2006 20:54 UTC (Mon) by jmayer (subscriber, #595) [Link] Ethereal has a very small number of plugins, and most bugs are *NOT* inthe plugins. Your information is wrong here.
Coverity releases first defect survey results Posted Mar 6, 2006 21:15 UTC (Mon) by jwb (guest, #15467) [Link] Hrmm, just looking at the Ethereal notices posted by this site:
A stack-based buffer overflow was discovered in the OSPF dissector... ethereal denial of service in IRC dissector...
- the ISAKMP dissector could exhaust system memory
The iSNS dissector... The SMB SID snooping capability in ethereal ... The SNMP dissector in ethereal ... certain SIP messages ... The AIM dissector ... The SPNEGO dissector ... Buffer overflow in the MMSE dissector ... a problem in DICOM dissection ... An invalid RTP timestamp ... The HTTP dissector ... improperly formatted SMB packet ...
As far as I can tell, every vulnerability in Ethereal has been in the dissectors. Perhaps I should not be using the terms "plug-in" and "dissector" interchangably, but clearly the dissectors are the problem. And you can fairly easily find all the future Ethereal vulnerabilities by grepping the dissector source directory for "char buf[256];".
I've always felt that the only good way to run Ethereal would be to operate the dissectors in a separate, non-privileged process. So far Ethereal has resisted privilege separation, even though the topic has been broached on the mailing list numerous times. Perhaps in the future they will implement this.
The only safe way to run Ethereal today is to recompile it with almost all the protocols disabled, capture the packet stream using tcpdump or another utility, chown the dump file to a non-user, then run Ethereal against the dump file in the non-users account. Any other use of Ethereal, especially against live data off the wire, is extremely hazardous.
Coverity releases first defect survey results Posted Mar 7, 2006 21:10 UTC (Tue) by azhrei_fje (guest, #26148) [Link] The only safe way to run Ethereal today is to recompile it with almost all the protocols disabled, capture the packet stream using tcpdump or another utility, chown the dump file to a non-user, then run Ethereal against the dump file in the non-users account. Any other use of Ethereal, especially against live data off the wire, is extremely hazardous. Unless I'm missing something, that seems a bit excessive. Why not create a named pipe and tell ethereal to read the pipe as the dump, then run tcpdump and tell it to write its data to the same pipe? I must be missing some aspect of this (perhaps issues with reading/writing data from/to pipes?), but it seems pretty straight-forward. I suppose it could be that ethereal might want to seek on the input (?), but I can't see why it would...
Coverity releases first defect survey results Posted Mar 9, 2006 10:06 UTC (Thu) by jmayer (subscriber, #595) [Link] We are in the process of doing something like that: The current cvs HEADhas a utility called dumpcap, which does the capturing and ethereal uses it instead of capturing directly. We still need to change tethereal to do the same. So today ethereal (and tethereal hopefully soon) can be run without root privileges. This of course is not an issue on Windows or BSD, where this can be achieved by other means today. Still, it will noticably speed up the capture start on those systems too. Note that this only addresses part of the problem, as faulty dissectors will still allow malicious traffic coded for faulty dissectors to take over the user running ethereal. But at least it will not present immedeate root access any more.
Bugs can be different Posted Mar 6, 2006 20:36 UTC (Mon) by proski (subscriber, #104) [Link] Here's another way to look at that. If we check a detailed changelog for some software, e.g. Linux kernel, and select pure bugfixes done between previous and current releases, how many of them would Coverity find if run on the previous release? My guess is 10% at most, unless cleaning up and stabilizing the code was the main objective of the release. What users consider bugs is always not what software considers bugs. Therefore, judging code by computer-detectable bugs seems misleading to me.
Bugs can be different Posted Mar 7, 2006 13:43 UTC (Tue) by brother_rat (subscriber, #1895) [Link] and conversely how many of the (genuine) bugs discovered by coverity would a human discover? The fact is humans and computers are good at different things, but compliment each other well.
"defect" rate inversely proportional to prior scrutiny? Posted Mar 6, 2006 21:54 UTC (Mon) by jabby (guest, #2648) [Link] Perhaps I'm being naive or falling victim to the urge to generalize, but I can't resist pointing out what I perceive to be a correlation with these projects and observations...
It seems to me that projects which have received relatively high levels of scrutiny in the recent past and/or have received a high level of bug reports (presumably as a result of said scrutiny) would tend to have a lower level of "defects", as detected by code verification tools. They have in a sense been baptized by fire.
Then again, I have no special knowledge of the relative level of scrutiny directed at these projects. I'm just drawing this conclusion based on the LWN blurb. If anyone else with more specific knowledge could comment on this possible trend, I'd appreciate it.
"defect" rate inversely proportional to prior scrutiny? Posted Mar 8, 2006 12:32 UTC (Wed) by ramdyne (subscriber, #536) [Link] For ethereal the high level of security problems is directly related to the project focussing on code quality.
See http://www.ethereal.com/lists/ethereal-dev/200603/msg0013... and http://www.ethereal.com/lists/ethereal-dev/200603/msg0018... for the developers reaction to this article.
Andreas Sikkema
"defect" rate inversely proportional to prior scrutiny? Posted Mar 8, 2006 18:53 UTC (Wed) by corbet (editor, #1) [Link] It seems they weren't entirely impressed, and perhaps rightly so. I should have left that last sentence out. Pre-coffee excuses and all that; I apologize.
Coverity releases first defect survey results Posted Mar 7, 2006 9:01 UTC (Tue) by kleptog (subscriber, #1183) [Link] It would've been nice of them to indicate what *versions* they tested. CVS or released for example. Also, presumably they generate false positives. I hope at some point we get some statistics about how many of the found defects were real and how serious they were, but I wouldn't bet on it.
Coverity releases first defect survey results Posted Mar 7, 2006 23:42 UTC (Tue) by tialaramex (subscriber, #21167) [Link] Figuring out whether any strange code identified as a potential defect by Coverity is actually a bug (causes the software to behave in a way that doesn't match the design/ intention of the programmer) can be fairly difficult. Figuring out whether that bug manifests itself with real-world scenarios is also fairly difficult. Figuring out whether there's a real security problem (often the most serious type of bug) can be so hard that it's safest to assume that all bugs of certain types are security holes even if, in fact, few of them are ever actually exploited. However, the only broadly accepted "proof" that a security hole exists is the "proof of concept" exploit, actual code that breaks the security but has a harmless payload.
Yet, taking the same defect, and turning it into a piece of code that's obviously correct and properly annotated is often fairly trivial. So it may be that you can spend a day or two fixing all the Apache problems found by Coverity, while a colleague takes a week just to examine a single reported defect, then decides that while it is technically a bug, it can only cause any problems in some unlikely corner case.
So all of that means you're right, it's unlikely that anyone will collect such statistics, because doing so is much more work than fixing the defects and has no direct pay-off except to qualitatively validate the usefulness of Coverity.
|
|||||||||||||