Reasoning's Apache study
[This article was contributed by Joe 'Zonker' Brockmeier]
Back in February, Reasoning,
a study that surprised few in the Linux and open source community.
Specifically, Reasoning found that the Linux kernel's TCP/IP stack had
fewer defects than implementations from vendors with proprietary
This time around, Reasoning has focused on Apache. Reasoning looked at
Apache 2.1-dev, released at the end of January this year, and found that
release to be about the same quality as commercial software. Reasoning's
study was not sponsored by a vendor, nor does the company have any real
motivation to find that open source is better or worse than proprietary
software. Instead, the company is using studies of open source projects
to help promote its testing services. Open source makes an ideal
promotional device because Reasoning can actually release the full
results to the study, including source code where errors are found.
The company uses a method of automated testing that tests for memory
leaks, NULL pointer dereferences, bad deallocations, out of bounds array
access and/or uninitialized variables. These are classified as defects.
In 58,944 lines of code (LOC) spread out in 360 files, there were a
total of 31 defects, or a defect density of 0.53 per thousand lines of
code. According to Reasoning, the average defect density for commercial
applications is 0.51 per thousand lines of code.
Of 31 defects, 29 of the defects were NULL pointer dereferences and 2
were uninitialized variables -- no memory leaks, bad deallocations or
out of bounds defects were found in Apache 2.1-dev. The detailed version
of Reasoning's report lists each of the 31 defects, giving the location
of the defect, a description and the actual defect in a code fragment
taken from the file with the defect.
One might wonder why Reasoning chose to look at a development version of
Apache rather than a more mature version that had been out for a while.
Certainly, very few people are likely to be deploying a development
version of Apache on production sites -- making it less comparable to a
release of a proprietary product. Apparently, they decided to review a
less mature version of an open source project to point out how the open
source development model benefits a project in the long run.
It would be interesting for Reasoning to track Apache's development and
compare its quality against proprietary code after it has been released
and in use for some time. One suspects that the Apache release would
fare better than proprietary projects after it had been formally
released and in use for some time.
Obviously, the study doesn't provide the full picture. It only measures
certain types of defects, and doesn't take into account the software's
features, performance or other qualities. But, at least in the area of
software defects, Reasoning's study reflects well on the open source
model by demonstrating what many users of open source already know --
that open source produces code of a quality that is at least comparable
to proprietary software.
to post comments)