Comparing free and proprietary software defect rates [LWN.net]

Comparing free and proprietary software defect rates

Posted Feb 13, 2003 16:15 UTC (Thu) by rakoch (guest, #4666)
Parent article: Comparing free and proprietary software defect rates

In a commercial setting all kind of "under the hood" work is not very
rewarding. It's much more interesting for a programmer or a team to
implement features that are visible. On the other hand the Linux TCP/IP
stack is probably one of the most scrutinized pieces of software in the
free software world.

There are plenty of worthy free software projects on sf.net. Most of them
simply cannot compare to their commercial competitors. Why? Because they
have a problem the Linux kernel has not: They lack developers.

A TCP stack simply isn't the focus of most commercial OSes. An exception
might be routers which was probably the "embedded device" in the test. You
bet for Cisco the TCP/IP stack is important. But for Sun/IBM/HP it's much
more important to have their Unix scale to dozens of CPUs. And for MS the
IE was probably the most important part of the OS until DRM and Palladium
came along.

So concluding from the quality of the TCP/IP stack to the quality of the
rest is pretty misleading. I'd be curious about a comparison of compilers,
though.

(Log in to post comments)

Comparing free and proprietary software defect rates

Posted Feb 13, 2003 19:53 UTC (Thu) by giraffedata (subscriber, #1954) [Link]

I think the results will be similar everywhere.

Around 1992, there was a research paper on the same topic, with the same results. In that case, a team of programmers fed random input to a whole bunch of Unix programs and noted when the programs crashed. Where they had source code, they debugged the crash and sent the information to the maintainer of the code.

Actually, the paper's main point was the source of the bugs (buffer overrun, etc.) but there was an unmistakeable difference in bug rates between free software and commercial software. At the time, this surprised many of us because the common wisdom in the industry said commercial software was bound to have fewer bugs because of all the investment in testing and because commercial publishers had more to lose from bugs.

But there was another result in that paper which I found much more interesting, which I think explained the phenomenon. The study was a followup on a study done the same way years earlier, which had found the same difference. In the followup, the programmers looked for the same bugs that had been reported in the original study. In commercial software, nearly all of the bugs were still present. In free software, nearly all of the bugs had been removed.

As a software developer for a major software publisher at the time, this didn't surprise me one bit. The software development machine of IBM is not capable of fixing a product just because it learns it's broken. But an individual free software developer not only is capable of releasing a fix, but insists on it as a matter of pride.

--
Bryan Henderson bryanh@giraffe-data.com
San Jose, California