LWN: Comments on "Comparing free and proprietary software defect rates"

Race conditions

ncm — 2003-02-23T08:36:12+00:00

Linux also has many more people looking for race conditions in it than any of the proprietary systems that they compared it to. Therefore, we can reasonably expect it to compare favorably in that area vs. the other systems.

This is not to say Linux would show well, on an absolute scale, in its number of remaining race conditions. It probably stinks. The proprietary stacks, though, probably stink far, far worse, for exactly the same reasons that they leak worse.

Stanford CHECKER?

wh — 2003-02-20T19:44:23+00:00

True, but the story's author interprets the results of Reasoning's findings differently. Just because Linux has less memory leaks (found by the Stanford Checker) than other operating systems doesn't mean that Linux has less race conditions (not found by the Stanford Checker) than other operating systems.

Stanford CHECKER?

ncm — 2003-02-20T18:45:19+00:00

Exactly: Linux came out better because it attracts public and academic scrutiny, and improves as a direct result of that scrutiny. It's not because the coders are more competent, or motivated. Sometimes they are, sometimes they aren't. The difference is in how it improves.

Stanford CHECKER?

haraldt — 2003-02-20T14:33:37+00:00

Still, if the Linux kernel is one of the few to have been tested this way, what does this tell us?

Stanford CHECKER?

wh — 2003-02-20T13:48:45+00:00

I believe that the only reason why Linux has so few of these
computer-findable bugs is that the people from the Standford "Checker"
project already ran their program on the Linux source and all the bugs
found by it got fixed afterwards.

Smatch

error27 — 2003-02-15T22:00:47+00:00

Thanks for the Smatch plug. :) The website is smatch.sf.net and since the article mentions null pointer dereferences the results for the dereference testing of 2.5.60 are here.

Smatch is too new to have had much influence yet. The Stanford Checker has made a measurable difference on the number of kernel errors. Unfortunately, their bug database only goes up to 2.4.1. They have released newer lists of bugs to lkml but when I released bug lists it was not as effective as a database (especially for long lists).

I am excited about the possibilities for Smatch to improve the kernel. The code for the Smatch derefence check about 130 lines long. A lot of that was cut and paste. Any of the other four checks should be possible with Smatch as well.

Comparing free and proprietary software defect rates

giraffedata — 2003-02-13T19:53:14+00:00

I think the results will be similar everywhere.

Around 1992, there was a research paper on the same topic, with the same results. In that case, a team of programmers fed random input to a whole bunch of Unix programs and noted when the programs crashed. Where they had source code, they debugged the crash and sent the information to the maintainer of the code.

Actually, the paper's main point was the source of the bugs (buffer overrun, etc.) but there was an unmistakeable difference in bug rates between free software and commercial software. At the time, this surprised many of us because the common wisdom in the industry said commercial software was bound to have fewer bugs because of all the investment in testing and because commercial publishers had more to lose from bugs.

But there was another result in that paper which I found much more interesting, which I think explained the phenomenon. The study was a followup on a study done the same way years earlier, which had found the same difference. In the followup, the programmers looked for the same bugs that had been reported in the original study. In commercial software, nearly all of the bugs were still present. In free software, nearly all of the bugs had been removed.

As a software developer for a major software publisher at the time, this didn't surprise me one bit. The software development machine of IBM is not capable of fixing a product just because it learns it's broken. But an individual free software developer not only is capable of releasing a fix, but insists on it as a matter of pride.

--
Bryan Henderson bryanh@giraffe-data.com
San Jose, California

Comparing free and proprietary software defect rates

rakoch — 2003-02-13T16:15:21+00:00

In a commercial setting all kind of "under the hood" work is not very
rewarding. It's much more interesting for a programmer or a team to
implement features that are visible. On the other hand the Linux TCP/IP
stack is probably one of the most scrutinized pieces of software in the
free software world.

There are plenty of worthy free software projects on sf.net. Most of them
simply cannot compare to their commercial competitors. Why? Because they
have a problem the Linux kernel has not: They lack developers.

A TCP stack simply isn't the focus of most commercial OSes. An exception
might be routers which was probably the "embedded device" in the test. You
bet for Cisco the TCP/IP stack is important. But for Sun/IBM/HP it's much
more important to have their Unix scale to dozens of CPUs. And for MS the
IE was probably the most important part of the OS until DRM and Palladium
came along.

So concluding from the quality of the TCP/IP stack to the quality of the
rest is pretty misleading. I'd be curious about a comparison of compilers,
though.