Reasoning Releases Results of a Software Code Audit of the Apache Web Server

[Posted July 1, 2003 by ris]

Reasoning has announced the results of a study in which the company inspected the code of the Apache Open Source Web Server V2.1. Reasoning found that the Apache Open Source server had a similar defect density compared to the average defect density of several proprietary equivalents.

Apache 2.1 -- Strange

Posted Jul 1, 2003 22:09 UTC (Tue) by pate (guest, #10) [Link] (2 responses)

Any idea why they tested a development snapshot of apache for comparison against 'several proprietary equivalents'? Unless their comparison was made against pre-release versions of proprietary code, this seems like FUD rather than real information.

On the other hand, the reports are interesting, and could be of value to the apche developemnt community in finding and stomping on bugs. Hmm, maybe Reasoning should look at GCC.

-pate

Apache 2.1 -- Strange

Posted Jul 1, 2003 23:45 UTC (Tue) by iabervon (subscriber, #722) [Link] (1 responses)

Because they were specifically trying to determine if OSS gets written better, or if it gets debugged more effectively. The conclusion seems to be that there is not a significant difference in development, but that OSS becomes more stable as it matures. Of course, this makes sense under the "many eyes" theory, since a development version will contain a lot of code seen only by the author so far. They'd already done a study of mature software and found that OSS was substantially better, and they wanted to determine if OSS programmers were just more careful or something, or if the process caused it to improve over time.

Apache 2.1 -- Strange

Posted Jul 3, 2003 17:26 UTC (Thu) by mmealman (guest, #9223) [Link]

That would be a true assessment if say they compared Apache 2.1 with the beta of ISS or the beta of some other commercial software. The summary of the study on their website didn't mention what they compared Apache to however.

Reasoning Releases Results of a Software Code Audit of the Apache Web Server

Posted Jul 2, 2003 4:51 UTC (Wed) by piman (guest, #8957) [Link]

This also depends on how you define a "defect." Certain things, like a buffer overflow or a race condition, definitely are. But if Microsoft was auditing Outlook, the ability to run scripts automatically, or load images or links from remote servers, isn't a defect. If (say) OpenBSD was to do the audit, it definitely would be.

I don't know if the "proprietary equivalents" for webservers contain similar problems, but proprietary software seems to differentiate between "design flaws" and "defects" (the former never being fixed), where I find free software usually treats serious design flaws as bugs like any other.

Since Reason's methods seem to be automated, it's likely that they don't pick up these sort of problems. This isn't a jab at Reason, who seem to be doing interesting stuff (in research and in practice), but just a reflection on their results.