[This article was contributed by Joe 'Zonker' Brockmeier]

Back in February, Reasoning, Inc. released a study that surprised few in the Linux and open source community. Specifically, Reasoning found that the Linux kernel's TCP/IP stack had fewer defects than implementations from vendors with proprietary versions.

This time around, Reasoning has focused on Apache. Reasoning looked at Apache 2.1-dev, released at the end of January this year, and found that release to be about the same quality as commercial software. Reasoning's study was not sponsored by a vendor, nor does the company have any real motivation to find that open source is better or worse than proprietary software. Instead, the company is using studies of open source projects to help promote its testing services. Open source makes an ideal promotional device because Reasoning can actually release the full results to the study, including source code where errors are found.

The company uses a method of automated testing that tests for memory leaks, NULL pointer dereferences, bad deallocations, out of bounds array access and/or uninitialized variables. These are classified as defects. In 58,944 lines of code (LOC) spread out in 360 files, there were a total of 31 defects, or a defect density of 0.53 per thousand lines of code. According to Reasoning, the average defect density for commercial applications is 0.51 per thousand lines of code.

Of 31 defects, 29 of the defects were NULL pointer dereferences and 2 were uninitialized variables -- no memory leaks, bad deallocations or out of bounds defects were found in Apache 2.1-dev. The detailed version of Reasoning's report lists each of the 31 defects, giving the location of the defect, a description and the actual defect in a code fragment taken from the file with the defect.

One might wonder why Reasoning chose to look at a development version of Apache rather than a more mature version that had been out for a while. Certainly, very few people are likely to be deploying a development version of Apache on production sites -- making it less comparable to a release of a proprietary product. Apparently, they decided to review a less mature version of an open source project to point out how the open source development model benefits a project in the long run.

It would be interesting for Reasoning to track Apache's development and compare its quality against proprietary code after it has been released and in use for some time. One suspects that the Apache release would fare better than proprietary projects after it had been formally released and in use for some time.

Obviously, the study doesn't provide the full picture. It only measures certain types of defects, and doesn't take into account the software's features, performance or other qualities. But, at least in the area of software defects, Reasoning's study reflects well on the open source model by demonstrating what many users of open source already know -- that open source produces code of a quality that is at least comparable to proprietary software.