LWN: Comments on "Coverity releases first defect survey results"

Rate of bugs and rate of security holes are mostly uncorrelated

rwild — Thu, 09 Mar 2006 10:25:09 +0000

Yes (therefore the "mostly"). I usually "make install -k DESTDIR=.." then and fix the setuid manually. Not so great. Better to get fast-install mode fixed in Libtool on more systems (which avoids the relinking in the first place).

Coverity releases first defect survey results

jmayer — Thu, 09 Mar 2006 10:06:23 +0000

We are in the process of doing something like that: The current cvs HEAD
has a utility called dumpcap, which does the capturing and ethereal uses
it instead of capturing directly. We still need to change tethereal to do
the same. So today ethereal (and tethereal hopefully soon) can be run
without root privileges. This of course is not an issue on Windows or
BSD, where this can be achieved by other means today. Still, it will
noticably speed up the capture start on those systems too.
Note that this only addresses part of the problem, as faulty dissectors
will still allow malicious traffic coded for faulty dissectors to take
over the user running ethereal. But at least it will not present
immedeate root access any more.

"defect" rate inversely proportional to prior scrutiny?

corbet — Wed, 08 Mar 2006 18:53:53 +0000

It seems they weren't entirely impressed, and perhaps rightly so. I should have left that last sentence out. Pre-coffee excuses and all that; I apologize.

Compared to commercial?

samth — Wed, 08 Mar 2006 16:02:19 +0000

As I understand it the analysis done by their checker is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect).

I expect you are correct in your conclusions (no interesting checkers are complete, and most commercial ones are unsound), but you have sound and complete backwards. Sound means no false negatives, and complete means no false positives.

"defect" rate inversely proportional to prior scrutiny?

ramdyne — Wed, 08 Mar 2006 12:32:52 +0000

For ethereal the high level of security problems is directly related to the project focussing on code quality.

See http://www.ethereal.com/lists/ethereal-dev/200603/msg0013... and http://www.ethereal.com/lists/ethereal-dev/200603/msg0018... for the developers reaction to this article.

Andreas Sikkema

Rate of bugs and rate of security holes are mostly uncorrelated

djm — Wed, 08 Mar 2006 05:52:48 +0000

OpenBSD isn't a "distribution", but it has privilege separated Xorg so the vast majority no longer runs as root. Disabling the TCP listener isn't so much of a win as it breaks ssh forwarding of X sessions, and because X11's authentication is pretty good (barring chumps who run "xhost +").

Compared to commercial?

peterh — Wed, 08 Mar 2006 05:32:38 +0000

Yes, but you're missing the point. Automated checkers are supposed to be thorough --- they tend to find the sorts of bugs that you wouldn't expect to find yourself.

There's been a reasonable amount of research recently on using reasonably classical compiler-type program analyses, such as abstract interpretation and dataflow analyses to detect bugs (Metal/Coverity checker and Saturn out of Stanford, and I think Cousot and others were doing work on verification of aerospace systems in France, and there are no doubt more that I can't think of right now). The real innovation of Metal is that it has a "find bugs at all cost" mentality, irrespective of the theoretical soundness of what it does. The result is probably quite good as a tool for finding certain classes of bugs. But I doubt it's sensible to make conclusions about the total bug count of a program based on what the checker detects.

Rate of bugs and rate of security holes are mostly uncorrelated

nix — Wed, 08 Mar 2006 00:54:28 +0000

Code which is complex is only defective if it's doing a simple job despite that complexity.

Things like GCC, say, are doing very complex jobs: now perhaps GCC is more complex than it needs to be if all were ideal, but there's no way you'll ever get it simple enough to make static analysis easy. It does a very complex job.

(And so do the knottier parts of the kernel.)

Defect criteria

nix — Wed, 08 Mar 2006 00:50:49 +0000

But it *does* have more stupid gratuitous Microsoft-bashing. (This example makes even less sense than usual.)

Rate of bugs and rate of security holes are mostly uncorrelated

nix — Wed, 08 Mar 2006 00:49:25 +0000

Indeed, although I'm not sure what effect this has on things that want to install with unusual userids (and there are some).

Coverity releases first defect survey results

tialaramex — Tue, 07 Mar 2006 23:42:33 +0000

Figuring out whether any strange code identified as a potential defect by Coverity is actually a bug (causes the software to behave in a way that doesn't match the design/ intention of the programmer) can be fairly difficult. Figuring out whether that bug manifests itself with real-world scenarios is also fairly difficult. Figuring out whether there's a real security problem (often the most serious type of bug) can be so hard that it's safest to assume that all bugs of certain types are security holes even if, in fact, few of them are ever actually exploited. However, the only broadly accepted "proof" that a security hole exists is the "proof of concept" exploit, actual code that breaks the security but has a harmless payload.

Yet, taking the same defect, and turning it into a piece of code that's obviously correct and properly annotated is often fairly trivial. So it may be that you can spend a day or two fixing all the Apache problems found by Coverity, while a colleague takes a week just to examine a single reported defect, then decides that while it is technically a bug, it can only cause any problems in some unlikely corner case.

So all of that means you're right, it's unlikely that anyone will collect such statistics, because doing so is much more work than fixing the defects and has no direct pay-off except to qualitatively validate the usefulness of Coverity.

Coverity releases first defect survey results

azhrei_fje — Tue, 07 Mar 2006 21:10:09 +0000

The only safe way to run Ethereal today is to recompile it with almost all the protocols disabled, capture the packet stream using tcpdump or another utility, chown the dump file to a non-user, then run Ethereal against the dump file in the non-users account. Any other use of Ethereal, especially against live data off the wire, is extremely hazardous.

Unless I'm missing something, that seems a bit excessive. Why not create a named pipe and tell ethereal to read the pipe as the dump, then run tcpdump and tell it to write its data to the same pipe?

I must be missing some aspect of this (perhaps issues with reading/writing data from/to pipes?), but it seems pretty straight-forward. I suppose it could be that ethereal might want to seek on the input (?), but I can't see why it would...

Compared to commercial?

daniels — Tue, 07 Mar 2006 20:12:09 +0000

That's just a matter of training, though: it at least appeared to recognise all the cases like that for X. The only false-negative I saw like that was some really hideous error-handling code involving longjmp.

Compared to commercial?

kleptog — Tue, 07 Mar 2006 19:44:26 +0000

Obviously the checker itself needs to be taught how to deal with various aspects of the programs it checks. For example, for PostgreSQL it doesn't appear to recognise that elog(ERROR, ...) never returns, leading to many spurious warnings about using variables inappropriately.

It's still a really nice technology and could help track-down some of the more obscure bugs.

Rate of bugs and rate of security holes are mostly uncorrelated

Junior_Samples — Tue, 07 Mar 2006 19:10:15 +0000

"In any case, the more complex the code, the less likely it is that a static checker like Coverity's will isolate real faults without special understanding of that project."

Au contraire. Complexity in and of itself is a defect. The measurement of complexity is considered the one of best metrics of software quality there is. It has been shown time and again that defects grow exponentially as complexity is increased. The fact that an automated parser should struggle with code which is "more complex" and full of obscure idioms should be proof enough that the code is defective.

Sadly, use of complexity analysis seems to be completely absent in the OSS domain despite the fact that basic measurement of complexity using McCabe's or Halstead's methods are very straightforward, and freely available automated parsers exist. The Linux Journal has an article about one such tool for Python.

Rate of bugs and rate of security holes are mostly uncorrelated

oak — Tue, 07 Mar 2006 18:33:00 +0000

SUSE X servers don't listen on TCP sockets by default either.

(I don't think this has been the case always, I'd guess it to have
changed within last couple of years.)

Rate of bugs and rate of security holes are mostly uncorrelated

grouch — Tue, 07 Mar 2006 15:22:09 +0000

Do you mean '-nolisten tcp'?

From my default /etc/X11/xinit/xserverrc file on Debian:

exec /usr/bin/X11/X -dpi 100 -nolisten tcp

Rate of bugs and rate of security holes are mostly uncorrelated

daniels — Tue, 07 Mar 2006 13:51:24 +0000

> I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number.

I can't think of one off the top of my head that does listen for TCP connections by default.

Compared to commercial?

daniels — Tue, 07 Mar 2006 13:50:11 +0000

Actually, Coverity's checker is pretty thorough: it turns up stuff you wouldn't necessarily expect an automated checker to.

(But, of course, there are some false positives, and undoubtedly false negatives. But it's turning up stuff even a skilled human wouldn't necessarily get, and good luck finding someone sufficiently skilled, willing to review the entire X codebase.)

Bugs can be different

brother_rat — Tue, 07 Mar 2006 13:43:46 +0000

and conversely how many of the (genuine) bugs discovered by coverity would a human discover? The fact is humans and computers are good at different things, but compliment each other well.

Coverity releases first defect survey results

kleptog — Tue, 07 Mar 2006 09:01:23 +0000

It would've been nice of them to indicate what *versions* they tested. CVS or released for example. Also, presumably they generate false positives. I hope at some point we get some statistics about how many of the found defects were real and how serious they were, but I wouldn't bet on it.

Rate of bugs and rate of security holes are mostly uncorrelated

rwild — Tue, 07 Mar 2006 08:34:20 +0000

> not many things other than the cursed libtool --mode=install ever run GCC as root

Bugs notwithstanding, you should be able to prevent that mostly by doing a DESTDIR install as user, and copying over and libtool --mode=finish as root.

Defect criteria

man_ls — Tue, 07 Mar 2006 07:31:55 +0000

Yeah, just being "around computers" helps a lot to understand programming. Well, now that I think about it, my boss has been around computers for all of his working career and he knows nil about programming (for him "refactoring" is a four letter word). And my boss's boss, too. And my family. And some of my friends too!

Well, now that I think about it: just being "around computers" does not help when it comes to programming. Please download some -dev packages and start hacking -- it's never too late -- before engaging in meaningful discussions about code. Even some Bash scripting will help a lot to understand how to program.

Sorry for feeding the troll, folks, but it's the first message I read from petegn that does not contain obscenities so I felt compelled to answer.

Compared to commercial?

peterh — Tue, 07 Mar 2006 07:24:20 +0000

You'll probably find that it's fairly meaningless comparing bug data
generated by humans and that produced by software analysis tools like
the Coverity checker. As I understand it the analysis done by their checker
is neither sound (what it reports may or may not be a bug) nor complete (it isn't guaranteed to find all bugs, even of the classes of bug that it can in principle detect). In some sense the checker is good at finding "obvious" bugs, but "deep" errors won't always be found. Low bug counts here don't necessarily compare meaningfully to the number of bugs in the code derived through some other means.

That said, there may well be a correlation between the sorts of bugs that the checker can detect and all bugs (ie. good programmers make fewer of both sorts of bug, bad programmers make more of both). Hence a comparison between software bug counts produced with the same tool is interesting.

Rate of bugs and rate of security holes are mostly uncorrelated

mattdm — Tue, 07 Mar 2006 05:24:00 +0000

I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number.

Check out Fedora Core, then. Still runs as root (although possibly limited by SELinux), but doesn't listen on TCP by default.

Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal".

-nolisten tcp with X.org. Actually, I think this *is* the default in the upstream Gnome GDM, so....

Defect criteria

petegn — Mon, 06 Mar 2006 23:01:28 +0000

Thas all very well apart from the fact that evry single piece of code can be written in several ways and still achieve the same result .

So that said is it a fault or just another way they had not dreamt up of doing things ..

PS i amnot a programmer but have been around computers long enough to know that there is a lot more than one way of skining a billy goat (Gates)

Pete .

Found on linux-kernel

davej — Mon, 06 Mar 2006 22:39:46 +0000

Indeed, it did.

"defect" rate inversely proportional to prior scrutiny?

jabby — Mon, 06 Mar 2006 21:54:40 +0000

Perhaps I'm being naive or falling victim to the urge to generalize, but I can't resist pointing out what I perceive to be a correlation with these projects and observations...

It seems to me that projects which have received relatively high levels of scrutiny in the recent past and/or have received a high level of bug reports (presumably as a result of said scrutiny) would tend to have a lower level of "defects", as detected by code verification tools. They have in a sense been baptized by fire.

Then again, I have no special knowledge of the relative level of scrutiny directed at these projects. I'm just drawing this conclusion based on the LWN blurb. If anyone else with more specific knowledge could comment on this possible trend, I'd appreciate it.

Rate of bugs and rate of security holes are mostly uncorrelated

Ross — Mon, 06 Mar 2006 21:49:30 +0000

That's strange because I remember doing specifically on my Debian system. But obviously my memory is not infallible; I didn't even remember the option correctly. It is in the form you posted, though I just checked the man page and it is only prefixed with one hyphen.

Compared to commercial?

dwheeler — Mon, 06 Mar 2006 21:40:05 +0000

Good luck getting comparitive data. There is some, but it's not easy to get.

As noted in this article about Coverity, CMU data suggests that a 5.7MSLOC system should have over 5,000 defects, that is, 0.87 defects/KSLOC. Nearly all the projects had a smaller rate than that. If the CMU data is comparible to the Coverity results -- and that is a MIGHTY big if -- then nearly all the OSS projects are doing much better than average.

Although comparitive information from Coverity is hard to find, there is some comparitive data from Reasoning. See the reliability section of "Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers", which references the studies from Reasoning and Coverity.

If you know of more comparitive data, I'd love to hear about it.

Coverity releases first defect survey results

jwb — Mon, 06 Mar 2006 21:15:06 +0000

Hrmm, just looking at the Ethereal notices posted by this site:

A stack-based buffer overflow was discovered in the OSPF dissector... ethereal denial of service in IRC dissector...

- the ISAKMP dissector could exhaust system memory
- the FC-FCS dissector could exhaust system memory
- the RSVP dissector could exhaust system memory
- the ISIS LSP dissector could exhaust system memory
- the IrDA dissector could crash
- the SLIMP3 dissector could overflow a buffer
- the BER dissector was susceptible to an infinite loop
- the SCSI dissector could dereference a null pointer and crash
- the sFlow dissector could dereference a null pointer and crash
- the RTnet dissector could dereference a null pointer and crash
- the SigComp UDVM could go into an infinite loop or crash
- the X11 dissector could attempt to divide by zero
- if SMB transaction payload reassembly is enabled the SMB dissector
could crash (by default this is disabled)
- if the "Dissect unknown RPC program numbers" option was enabled, the
ONC RPC dissector might be able to exhaust system memory (by default
this is disabled)
- the AgentX dissector could overflow a buffer
- the WSP dissector could free an invalid pointer
- iDEFENSE discovered a buffer overflow in the SRVLOC dissector

The iSNS dissector... The SMB SID snooping capability in ethereal ... The SNMP dissector in ethereal ... certain SIP messages ... The AIM dissector ... The SPNEGO dissector ... Buffer overflow in the MMSE dissector ... a problem in DICOM dissection ... An invalid RTP timestamp ... The HTTP dissector ... improperly formatted SMB packet ...

As far as I can tell, every vulnerability in Ethereal has been in the dissectors. Perhaps I should not be using the terms "plug-in" and "dissector" interchangably, but clearly the dissectors are the problem. And you can fairly easily find all the future Ethereal vulnerabilities by grepping the dissector source directory for "char buf[256];".

I've always felt that the only good way to run Ethereal would be to operate the dissectors in a separate, non-privileged process. So far Ethereal has resisted privilege separation, even though the topic has been broached on the mailing list numerous times. Perhaps in the future they will implement this.

The only safe way to run Ethereal today is to recompile it with almost all the protocols disabled, capture the packet stream using tcpdump or another utility, chown the dump file to a non-user, then run Ethereal against the dump file in the non-users account. Any other use of Ethereal, especially against live data off the wire, is extremely hazardous.

Coverity releases first defect survey results

jmayer — Mon, 06 Mar 2006 20:54:20 +0000

Ethereal has a very small number of plugins, and most bugs are *NOT* in
the plugins. Your information is wrong here.

Bugs can be different

proski — Mon, 06 Mar 2006 20:36:20 +0000

Here's another way to look at that. If we check a detailed changelog for some software, e.g. Linux kernel, and select pure bugfixes done between previous and current releases, how many of them would Coverity find if run on the previous release? My guess is 10% at most, unless cleaning up and stabilizing the code was the main objective of the release. What users consider bugs is always not what software considers bugs. Therefore, judging code by computer-detectable bugs seems misleading to me.

Rate of bugs and rate of security holes are mostly uncorrelated

nijhof — Mon, 06 Mar 2006 20:17:37 +0000

Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal".

It is certainly "normal" for Debian -- you have to take out the --nolisten tcp manually if you want it to listen to tcp traffic

Rate of bugs and rate of security holes are mostly uncorrelated

Ross — Mon, 06 Mar 2006 19:49:59 +0000

I haven't seen a distribution yet which doesn't run the X server as root and have it listening on TCP port 6000 + display number. Now it is certainly possible to disable the TCP X transport (using the -notcp option when starting the server), but I don't think it is "normal".

I think the difference may be that X is much bigger than the server. The bug may simply have been in other places than the part exposed to the network. Also, the X server normally uses an authentication mechanism like MIT Magic Cookies which will limit the exposure of bugs to a very small portion of the code if the shared secret is not known to the attacker. (Assuming the cookies aren't weak and that the server throttles brute force attacks.)

Coverity releases first defect survey results

jwb — Mon, 06 Mar 2006 19:11:38 +0000

Ethereal's vulnerabilities are almost always in plug-in code modules. Some random person will contribute a dissector for, say, the protocol used by AIM. The bugs nearly all take the form of unprotected string copies into arrays of type char, either using a length that should be considered untrusted, or by null-termination.

Maybe the Coverity checker didn't look at the plug-ins.

Found on linux-kernel

corbet — Mon, 06 Mar 2006 16:45:37 +0000

Shortly after that, Dave started posting fixes, so I assume this one got worked out.

Found on linux-kernel

kirkengaard — Mon, 06 Mar 2006 16:39:42 +0000

Date Mon, 6 Mar 2006 00:49:59 -0500
From Dave Jones <>
Subject Re: Coverity Open Source Defect Scan of Linux

On Sun, Mar 05, 2006 at 09:35:11PM -0800, Ben Chelf wrote:

> Right now, we're guarding access to the actual defects that we report
> for a couple of reasons: (1) We think that you, as developers of Linux,
> should have the chance to look at the defects we find to patch them
> before random other folks get to see what we found and (2) From a
> support perspective, we want to make sure that we have the appropriate
> time to engage with those who want to use the results to fix the code.
> Because of this second point, I'd ask that if you are interested in
> really digging into the results a bit further for your project, please
> have a couple of core maintainers (or group nominated individuals) reach
> out to me to request access. As this is a new process for us and still
> involves a small number of packages, I want to make sure that I
> personally can be involved with the activity that is generated from this
> effort.
>
> So I'm basically asking for people who want to play around with some
> cool new technology to help make source code better. If this interests
> you, please feel free to reach out to me directly. And of course, if
> there are other packages you care about that aren't currently on the
> list, I want to know about those too.

The last time I asked about access to your bug list, I was asked to
sign the equivalent of a non-compete agreement. Is this still in place?

Dave

--
http://www.codemonkey.org.uk
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Rate of bugs and rate of security holes are mostly uncorrelated

shahms — Mon, 06 Mar 2006 16:11:55 +0000

From reading the articles about Coverity's static checker (at least, the articles on the early versions of the Stanford checker), there is no reason to suspect they haven't written domain-specific extensions for at least some of the idioms and API rules of the projects they checked. One of the main points of their tool is making "system-specifc programmer written compiler extensions" relatively easy. The architecture described by those papers is powerful and extensible, able to detect and act on almost-arbitrary pattern rules. It's really a pretty nice architecture as described and I imagine the Coverity tool addresses the most urgent shortcomings of the early implementation.

Average proprietary defect rate?

kirkengaard — Mon, 06 Mar 2006 15:51:42 +0000

For that, you'd have to get them to air their dirty laundry. To some trustworthy third-party willing to actually disclose the figures they come up with, rater than the ones they were paid for. Coverity does seem like a good party for that, but just try and get the closed-source, all-for-profit PR departments to buy into not-guaranteed-to-be-good advertising that compares them to their competitors directly, from which users might draw negative conclusions about their software quality.