experiment based comparisons - also futile

experiment based comparisons - also futile

> How do you feel about more experiment-based methods of
> comparing the security of different systems?

Pessimistic.

If you configure the systems well and monitor them closely, the rate of compromise is nearly zero on all of them, and you get no data. If you configure them badly, you don't get a measure of the inherent security of the systems, you get a profile of the attacking community. The Honeynet project posted a paper on that a year ago. They had the opposite problem from the CERT list - people were misinterpreting their estimated time to compromise figures, and they had to add a disclaimer that their data didn't say that Windows was less secure than Linux just because their honeypots had shorter lifetimes. See:

http://www.honeynet.org/papers/trends/life-linux.pdf

If you abandon honeypots and try penetration tests, you fare no better in terms of figuring out relative security. For example, in the SANS "Hacker Techniques, Exploits & Incident Handling" class (SEC-504) on day 5 the student teams try to break into a mixed network of slightly vulnerable Windows and Linux hosts the instructor sets up. The good teams always end up owning 100% of both kinds, and then slink home quaking in their boots for fear of what could happen to their own networks - which they suddenly realize are even less secure than their instructor's testbed.

The best large scale study comparing relative security of organizations that I've heard of was conducted by Microsoft in the wake of their Nachi/Welchi problems in the summer of 2003. About half of their large corporate customers got nailed badly, and about half didn't. They desperately wanted to know the what the differentiating factor was. It turned out to be the quality of the systems administration, not any specific technology. The survivors had layered defenses, well executed patch processes, real time network monitoring, incident response teams, etc. The victims were sloppier, and paid for it. I don't have references for that, sorry. But Microsoft's emphasis in Windows XP SP2 on firewalls, virus scanning, and automatic updates all on by default was a direct reflection of that learning. It has had an effect too - the miscreants have now turned to phishing, in part because worms are so much less effective. However, once again, these data say nothing comparative about Windows versus Linux or any other platform inherently.

Security is hard, and comparing security is even harder. It's not that you can't get helpful answers to interesting questions - we're learning a lot, and improving. But so are the bad guys.