LWN: Comments on "Reasoning Releases Results of a Software Code Audit of the Apache Web Server"

Apache 2.1 -- Strange

mmealman — Thu, 03 Jul 2003 17:26:23 +0000

That would be a true assessment if say they compared Apache 2.1 with the beta of ISS or the beta of some other commercial software. The summary of the study on their website didn't mention what they compared Apache to however.

Reasoning Releases Results of a Software Code Audit of the Apache Web Server

piman — Wed, 02 Jul 2003 04:51:45 +0000

This also depends on how you define a "defect." Certain things, like a buffer overflow or a race condition, definitely are. But if Microsoft was auditing Outlook, the ability to run scripts automatically, or load images or links from remote servers, isn't a defect. If (say) OpenBSD was to do the audit, it definitely would be.

I don't know if the "proprietary equivalents" for webservers contain similar problems, but proprietary software seems to differentiate between "design flaws" and "defects" (the former never being fixed), where I find free software usually treats serious design flaws as bugs like any other.

Since Reason's methods seem to be automated, it's likely that they don't pick up these sort of problems. This isn't a jab at Reason, who seem to be doing interesting stuff (in research and in practice), but just a reflection on their results.

Apache 2.1 -- Strange

iabervon — Tue, 01 Jul 2003 23:45:12 +0000

Because they were specifically trying to determine if OSS gets written better, or if it gets debugged more effectively. The conclusion seems to be that there is not a significant difference in development, but that OSS becomes more stable as it matures. Of course, this makes sense under the "many eyes" theory, since a development version will contain a lot of code seen only by the author so far. They'd already done a study of mature software and found that OSS was substantially better, and they wanted to determine if OSS programmers were just more careful or something, or if the process caused it to improve over time.

Apache 2.1 -- Strange

pate — Tue, 01 Jul 2003 22:09:33 +0000

Any idea why they tested a development snapshot of apache for comparison against 'several proprietary equivalents'? Unless their comparison was made against pre-release versions of proprietary code, this seems like FUD rather than real information.

On the other hand, the reports are interesting, and could be of value to the apche developemnt community in finding and stomping on bugs. Hmm, maybe Reasoning should look at GCC.

-pate