The CERT vulnerability list [LWN.net]

The CERT vulnerability list

Posted Jan 8, 2006 5:09 UTC (Sun) by joey (subscriber, #328)
In reply to: The CERT vulnerability list by jeleinweber
Parent article: The CERT vulnerability list

Thanks, that was the best analysis I've ever read of why every security comparison anyone is likely to see is bunk. You should write it up into a paper. Seriously. (I included some similar stuff in a paper once but you'd do a much better job based on this post.)

How do you feel about more experiment-based methods of comparing the security of different systems? Things like putting up some honeypots of each system and seeing how long it takes before they are compromised in different ways, or analysing a large existing group of systems.

It's obviously flawed when the systems being compared are quite different, like Windows and Linux, because it will depend not only on what holes exist but what exploits are in the wild and which systems attackers are interested in targeting. However, for more interesting comparisons, such as between different linux distributions, or different versions of the same distribution, it seems like a useful approach.

(Log in to post comments)

experiment based comparisons - also futile

Posted Jan 12, 2006 3:15 UTC (Thu) by jeleinweber (subscriber, #8326) [Link]

> How do you feel about more experiment-based methods of
> comparing the security of different systems?

Pessimistic.

If you configure the systems well and monitor them closely, the rate of compromise is nearly zero on all of them, and you get no data. If you configure them badly, you don't get a measure of the inherent security of the systems, you get a profile of the attacking community. The Honeynet project posted a paper on that a year ago. They had the opposite problem from the CERT list - people were misinterpreting their estimated time to compromise figures, and they had to add a disclaimer that their data didn't say that Windows was less secure than Linux just because their honeypots had shorter lifetimes. See:

http://www.honeynet.org/papers/trends/life-linux.pdf

If you abandon honeypots and try penetration tests, you fare no better in terms of figuring out relative security. For example, in the SANS "Hacker Techniques, Exploits & Incident Handling" class (SEC-504) on day 5 the student teams try to break into a mixed network of slightly vulnerable Windows and Linux hosts the instructor sets up. The good teams always end up owning 100% of both kinds, and then slink home quaking in their boots for fear of what could happen to their own networks - which they suddenly realize are even less secure than their instructor's testbed.

The best large scale study comparing relative security of organizations that I've heard of was conducted by Microsoft in the wake of their Nachi/Welchi problems in the summer of 2003. About half of their large corporate customers got nailed badly, and about half didn't. They desperately wanted to know the what the differentiating factor was. It turned out to be the quality of the systems administration, not any specific technology. The survivors had layered defenses, well executed patch processes, real time network monitoring, incident response teams, etc. The victims were sloppier, and paid for it. I don't have references for that, sorry. But Microsoft's emphasis in Windows XP SP2 on firewalls, virus scanning, and automatic updates all on by default was a direct reflection of that learning. It has had an effect too - the miscreants have now turned to phishing, in part because worms are so much less effective. However, once again, these data say nothing comparative about Windows versus Linux or any other platform inherently.

Security is hard, and comparing security is even harder. It's not that you can't get helpful answers to interesting questions - we're learning a lot, and improving. But so are the bad guys.