Weekly Edition
Return to the Security page
| |||||||||||||
The CERT vulnerability list
It's all over the mainstream media: the CERT 2005
vulnerabilities list shows that "Unix/Linux" had three times as many
vulnerabilities as Windows. The security battle is over, and Windows has
won. Of course, if one actually looks at the list, the story no
longer seems so clear.
Let's examine a few entries:
One could go on for some time, but your editor chose to stop before finishing with the letter "A". The point should be clear anyway: drawing any conclusions from the length of this list makes no sense at all. One might make a reasonable Linux vulnerabilities list by (1) removing the large numbers of entries for BSD and proprietary Unix systems, (2) removing duplicates, and (3) removing proprietary products and other packages not normally shipped or installed with Linux distributions. The resulting list would certainly be less than 20% of the size of the version posted by CERT. One might also be tempted to look at CERT's advisory list for 2005. Of the alleged thousands of "Unix/Linux" vulnerabilities, exactly one (the Snort Back Orifice buffer overflow) merited an advisory from CERT. Every other alert sent out in 2005 was for Windows and other proprietary products. It might have been nice for CERT to mention this when it put up its list of vulnerabilities. One can also point out that most of the vulnerabilities were found as the result of active auditing efforts; they were fixed before anybody exploited them. Many of them are theoretical in nature, and many of them are only exploitable by local users. All vulnerabilities are not created equal. In the end, however, one fact remains: even a list which is 10% as long as CERT's is too long. We can argue relative security all we want (and we should dispute the outright silliness that results from CERT's list), but Linux still is not as secure as we need it to be. When the length of that list gets rather closer to zero, we'll be in a position to brag about the security of Linux. (Log in to post comments)
The CERT vulnerability list Posted Jan 6, 2006 18:05 UTC (Fri) by madscientist (subscriber, #16861) [Link] Good grief. I took the Windows list and the UNIX list and ran them through sort -u just to remove the OBVIOUSLY duplicated ones (without even trying to merge the "foo" and "foo (updated)" lines).
Results: MS 747, UNIX/Linux 1408. Still ridiculous but less than twice as many now... and that's the MOST TRIVIAL cleanup of reports that could possibly be made. As the LWN article says, comparing these two lists as-is is completely silly.
The CERT vulnerability list Posted Jan 6, 2006 19:10 UTC (Fri) by drag (subscriber, #31333) [Link] Not to also mention that your dealing with the entire freaking Unix universe here. There are dozens times more software in Redhat then what you get with Windows, or even Microsoft, for instance. Then when you combine Redhat, Debian, OS X, Mandrake, Fedora, etc etc etc in one big lump your dealing with so much code it'll make your head spin.
And when your dealing with Windows advisories do they add stuff like vunerabilities to the synaptic virus scanner to the list? I doubt it.
That's not to say that comparisions are not valid however...
Like Apache vs IIS 6. It's obvious to anyone who makes a superior product security-wise.. and that's Microsoft. IIS 5 is ancient history compared to when you want to start a new web server today.
Especially when you combine OpenSSL vunerabilities to Apache, like say if you want to run a server that accept payments, then it's even more pathetic comparision... because there is no comparision.
It's actually quite sad. There is a reason why Microsoft has started to eat away at Apache's server numbers.. and it's not just because of domain name registar games and such. It's been a ongoing trend for quite a few months now. Except for a couple bumps in November and in october that I noticed.. Since April 2005 Microsoft's webserver has been showing small gains on Apache in 'Active Webservers' which is what counts.
I fully expect the trend to accelerate through 2006. Apache will continue to loose more and more ground as SSl and Apache vunerabilities pile up and there isn't anything anybody can do about it except fix the problems as they find them and hope to god that Apache project doesn't introduce regressions.. which they will.
Then when you take PHP into considuration... They've had worms! Insecure programming practices by the PHP developers folks, combined by insecure practices by PHP web application development folks, combined by plain lazy webmasters and administrators have caused the first widespread worms to affect Linux servers since Redhat 6/7 days when they decided to follow w2k's example and 'enable everything to make it easy'.
Of course for people were Microsoft == God, then they'll crone on about how IIS 6 has no serious vunerabilities.. which is bullshit. It's just how Microsoft choose how to word vunerabilities so that they don't show up attatched to 'IIS 6 product' in places like secunia.com.
IIS 6 has had no vunerabilities in it's default configuration which is only able to serve static html pages. As soon as you start enabling features you start running into vunerabilities... For instance ASP.NET has had a couple.
But it's still comes out to single digit numbers compared to a few dozen 'LAMP' problems.
The CERT vulnerability list Posted Jan 6, 2006 19:16 UTC (Fri) by madscientist (subscriber, #16861) [Link] > Not to also mention that your dealing with the entire freaking Unix> universe here. There are dozens times more software in Redhat then what you > get with Windows, or even Microsoft, for instance. Then when you combine > Redhat, Debian, OS X, Mandrake, Fedora, etc etc etc in one big lump your > dealing with so much code it'll make your head spin.
Yes, that's what I meant when I said comparing the two lists is silly.
But, the LEAST CERT could have done was uniq the dups!
Apache Vs. MS IIS Posted Jan 6, 2006 20:33 UTC (Fri) by csmiller (guest, #32031) [Link] On,http://news.netcraft.com/archives/2006/01/05/january_2006... image http://news.netcraft.com/archives/2006/01/overallc.gif The graph shows a small swing from Apache to an unknown server. According to NetCraft, > The market share for the Apache web server is down by nearly three percent this > month, due primarily to configuration changes at domain registrar Go Daddy. Its bulk > hosting service includes a front-end system that generates an HTTP redirect when a > site is first accessed — and this redirect is not served by (or, at least, does not identify > itself as) Apache. Once the redirect is followed, or if the site is accessed a second > time, it is then served by Apache. So this change (which, given the large number of > sites hosted by Go Daddy, has not gone unnoticed), has caused a large swing from > Apache to Unknown.
Apache vulnerabilities Posted Jan 6, 2006 21:54 UTC (Fri) by rfunk (subscriber, #4054) [Link] The string of Apache vulnerabilities and regressions is what keeps a lotof people from upgrading from 1.3 to 2, despite the preferences of the Apache team. As for PHP's issues, I'm just amazed at how many current projects still require register_globals=On. I'm thinking 2006 is the year you'll see a lot more people start moving toward Ruby On Rails.
The CERT vulnerability list Posted Jan 6, 2006 18:45 UTC (Fri) by rickmoen (subscriber, #6943) [Link] Even at that, LWN's brief sampling didn't come close to exhausting the number of reasons CERT'slistings are useless statistics soup: They are notionally attempting to compare a hypothetical Windows x86 machine with every conceivable application including specialised utilities against the same on sum of all Unix-family OSes on all possible CPU architectures -- a comparison of no non-fantasy importance, whatsoever.
Debian 'etch" currently has at its immediate disposal this many packages, counting only open
~ $ grep Package: /var/lib/apt/lists/*testing*Packages | wc -l
...and yet, I'd be an idiot to make claims about "Debian security" based on the number of CERT
They're lumping in DoS attacks and cross-site scripting vulnerabilities (neither of which are not
Meaningless, incompetently reported statistics soup, I say.
Rick Moen
The CERT vulnerability list Posted Jan 6, 2006 21:50 UTC (Fri) by iabervon (subscriber, #722) [Link] The thing that bothers me is the assumption that it is worse to have more vulnerabilities. I'd say that for a fixed portion of time for which one is vulnerable, it is better if that is due to a wider variety of vulnerabilities, because it means that an attacker has to try more things, and is therefore more likely to be caught by an IDS or otherwise leave evidence. Would you rather use a system where there are 2000 exploits that each have a low probability of working, or a system where there is only 1 exploit, and it always works?
The CERT vulnerability list: defenses forgotten Posted Jan 6, 2006 22:15 UTC (Fri) by fenrus (guest, #31654) [Link] Linux has many defenses nowadays against vulnerabilities. Examples are NX support, SELinux, randomisation. Several distributions ship additional patches (ExecShield/PaX/Grsecurity). Glibc and gcc also have added defenses, for example double free exploits are no longer possible, and newly compiled stuff (such as FC4 or SuSE 10) makes format string exploits impossible.
Result is that a large chunk of the vulnerabilities are not explo;itable on linux (or really really really hard to improbable), even though the bug is in the software. This list does obviously also not take that into account.
The CERT vulnerability list Posted Jan 7, 2006 3:52 UTC (Sat) by jeleinweber (subscriber, #8326) [Link] The 3rd party spin is not entirely CERT's fault; they placed no interpretation on their list. It was just a year-end summary page with links to all their vulnerability notes from 2005 and counts of how many lines were in the broad categoriues. You can ask why they didn't bother to do any interpretation, but you shouldn't accuse them of pretending that their list proved anything one way or the other about the relative security of the various systems and applications. Idiot third parties who jump to statistically indefensible conclusions deserve whatever mockery, derision, and refutation they suffer by way of rebuttal.The unfortunate truth of the matter is that it's an incredible amount of work to do a sound comparison, and ultimately futile because the results won't actual change anyone's deployment decisions. Which is probably why you can't find any decent comparisons. Academics aren't interested and MBA's can't cost-justify them, so no one does them. If I were doing a high quality comparison, I'd want to precede as follows:
With this background work done, you can start the comparison. For each distinct vulnerability (this is where something like CVE ID's really pay off) see if it is part of your threat model and if it applies to any of your evaluation platforms. If it's not for software you are using, or not part of your threat model, ignore it - just because it is tangentially associated with one layer of your platform doesn't make it relevent. If it does apply, classify the risk in your risk model and see if it makes your severity cutoff. The survivors get counted to the detriment of their affected platforms. Half a man-year later, you have counts of dangerous vulnerabilities per platform and an estimate of the cost of those vulnerabilities being exploited against you. Congratulations - your analysis is now meaningful, defensible, and ready! Unfortunately, it's not very interesting, either to you, or to anyone else. The biggest difference between secure and insecure systems is the quality of the development process and the quality of the systems administration. Carefully done, any of the platforms is good enough to business on. So your organization makes its actual decision on other grounds, such as which one integrates more easily with their legacy technologies, or incurs the least marginal training costs, or has the best industry-specific 3rd party ecosystem of add-ons, or has the best local pool of talent. Your supurb analysis doesn't actually influence the decision at all. It's not completely worthless - you can recyle the threat model and risk assessment pieces to guide your preventative security measures, but overall doing it wasn't cost-effective. So your competitors who didn't bother with this analysis beat you to market or got to undercut your prices. This is why the MBA's won't authorize the resources to do this sort of analysis. Meanwhile, the rest of the your industry is very impressed, but also puzzled. It turns out their threat model is different, or they want different criteria on the risk model, or their threshholds are different, or they disagree with your interpretations of the vulnerabilities, or their software stack doesn't have exactly the same components. So in the end, they can't use your conclusions to guide their decisions either. This is why the academics won't bother: the results don't generalize. The bottom line is that counting security bulletins doesn't tell you anything you should care about much. Statistical morons who pretend it does need to learn to do better, particularly if they are so slipshod as to miscount - which they usually do, because cleaning up the raw data is a lot of work, which they are too ignorant to realize is necessary. As no one has an incentive to produce good comparisons, only bad ones get done. Upon this sea of gloom we can cast a few rays of hope: It is worthwhile to reduce the cost of trying to do good evaluations, because even if they don't generalize, they can offer genuine insight into worthwhile strategic directions. And the cheaper, easier, faster, and more reliable such evaluations get, the better the cost/benefit tradeoff looks. So efforts such as the SANS top-20 lists, or the CERT attempt to get virus names standardized, or the "Common Vulnerability Scoring System" which Cisco is pushing and FIRST sheparding are worthwhile.
The CERT vulnerability list Posted Jan 8, 2006 5:09 UTC (Sun) by joey (subscriber, #328) [Link] Thanks, that was the best analysis I've ever read of why every security comparison anyone is likely to see is bunk. You should write it up into a paper. Seriously. (I included some similar stuff in a paper once but you'd do a much better job based on this post.)
How do you feel about more experiment-based methods of comparing the security of different systems? Things like putting up some honeypots of each system and seeing how long it takes before they are compromised in different ways, or analysing a large existing group of systems.
It's obviously flawed when the systems being compared are quite different, like Windows and Linux, because it will depend not only on what holes exist but what exploits are in the wild and which systems attackers are interested in targeting. However, for more interesting comparisons, such as between different linux distributions, or different versions of the same distribution, it seems like a useful approach.
experiment based comparisons - also futile Posted Jan 12, 2006 3:15 UTC (Thu) by jeleinweber (subscriber, #8326) [Link] > How do you feel about more experiment-based methods of> comparing the security of different systems?
Pessimistic.
If you configure the systems well and monitor them closely, the rate of compromise is nearly zero on all of them, and you get no data. If you configure them badly, you don't get a measure of the inherent security of the systems, you get a profile of the attacking community. The Honeynet project posted a paper on that a year ago. They had the opposite problem from the CERT list - people were misinterpreting their estimated time to compromise figures, and they had to add a disclaimer that their data didn't say that Windows was less secure than Linux just because their honeypots had shorter lifetimes. See:
http://www.honeynet.org/papers/trends/life-linux.pdf
If you abandon honeypots and try penetration tests, you fare no better in terms of figuring out relative security. For example, in the SANS "Hacker Techniques, Exploits & Incident Handling" class (SEC-504) on day 5 the student teams try to break into a mixed network of slightly vulnerable Windows and Linux hosts the instructor sets up. The good teams always end up owning 100% of both kinds, and then slink home quaking in their boots for fear of what could happen to their own networks - which they suddenly realize are even less secure than their instructor's testbed.
The best large scale study comparing relative security of organizations that I've heard of was conducted by Microsoft in the wake of their Nachi/Welchi problems in the summer of 2003. About half of their large corporate customers got nailed badly, and about half didn't. They desperately wanted to know the what the differentiating factor was. It turned out to be the quality of the systems administration, not any specific technology. The survivors had layered defenses, well executed patch processes, real time network monitoring, incident response teams, etc. The victims were sloppier, and paid for it. I don't have references for that, sorry. But Microsoft's emphasis in Windows XP SP2 on firewalls, virus scanning, and automatic updates all on by default was a direct reflection of that learning. It has had an effect too - the miscreants have now turned to phishing, in part because worms are so much less effective. However, once again, these data say nothing comparative about Windows versus Linux or any other platform inherently.
Security is hard, and comparing security is even harder. It's not that you can't get helpful answers to interesting questions - we're learning a lot, and improving. But so are the bad guys.
experiment based comparisons - also futile Posted Jan 12, 2006 12:37 UTC (Thu) by jschrod (subscriber, #1646) [Link] Thanks for both of your posts and the work you put into writing them. They are very well phrased and concise summaries that I will keep for reference. Would you care to give your full name for citation? Googling "jeleinweber" didn't bring in much. :-)
TIA, Joachim
The CERT vulnerability list Posted Jan 7, 2006 4:05 UTC (Sat) by jamesh (guest, #1159) [Link] Note that a lot of the complaints about the "Unix/Linux" vulnerability list can also be levelled at the Windows vulnerability list of that page:
This is just raw data, and you'd need to do a fair bit of analysis before drawing any conclusions.
The CERT vulnerability list Posted Jan 7, 2006 10:49 UTC (Sat) by ccnix (guest, #16105) [Link] Corbet: It's all over the mainstream media ... Huh? Did you really mean that? According to Google News (search:cert+vulnerability) (@10:16 UTC, Jan 7), the 'CERT list' story was picked up by SQL Server Magazine, ZDNet (US, UK, Australia, India), Inquirer, The Register, and Slashdot and a few others. Meanwhile, the same search term revealed the real mainstream (Reuters, Washington Post, ABC, MSNBC, Forbes, SMH, The Standard(HK), and 400 others), is more concerned about the issue of the late release of the "WMF flaw" patch. According to one source, Microsoft was "feeling heat from security experts who said the vendor's response was too slow. Many even advised users to download and install a third-party patch, such was the risk involved." Based on that widespread perception alone, I don't think the mainstream will be publishing 'Windows is better at security than Linux' anytime soon ... .
The CERT vulnerability list Posted Jan 7, 2006 10:55 UTC (Sat) by zorgan (guest, #4016) [Link] I don't understand why everyone is bashing CERT for this. They publishedan index of vulnerabilities, sensibly divided into Windows, Unix/Linux and others (sensibly since most vulnerabilities either affect Windows systems or many Unix variants). Certainly they didn't want to "remove duplicates" because they wanted to link every bulletin where they had provided information on a particular vulnerability. They didn't ask anyone to draw conclusions from this list.
The CERT vulnerability list Posted Jan 8, 2006 0:12 UTC (Sun) by drag (subscriber, #31333) [Link] Well it doesn't seem to be very usefull...
If you want to find out about vunerabilities with Apache webserver your interested in actually Apache webserver, not the hundred or so advisories and patch release commentary from all the different vendors that use Apache.
plus I don't think it's bashing CERT so much as bashing articles that make use of misleading statements and meaningless statistics to generate hype and readership...
(note that I am about to plug something I think is very cool)
Something as a alternative is the Open Source Vunerability Database (at www.osvdb.org) . It's not so much about vunerabilities in oss software as actually having the database as open as possible. It has information about all sorts of software oss or not.
They do rss feeds and such. Also they have setup a API and whatnot so that you can access their database and pull relavent information off of it directly rather then manually scanning through the website listings.
I'd be usefull for building a script that looks for advisories in any of the software your using or are looking at using.. You could integrate it into applications to scan and do network security assessments of computerson your lan.
They also provide handy contact information about various vendors as it can be difficult to find contact information for vunerability issues and patches and such that tend to get buried in websites and such.
I think it's very cool.
If anybody was to write a article comparing actually usefull statistics about vunerabilities and such then I figure using the OSVDB.org would be a great place to start.
The CERT vulnerability list Posted Jan 9, 2006 21:29 UTC (Mon) by crankysysadmin (guest, #19449) [Link] Yes, they're damned if they do and damned if they don't. If they had offered interpretation, whatever camp was shown in a bad light would have complained bitterly about the method of interpretation.
Blaming CERT Posted Jan 12, 2006 9:31 UTC (Thu) by ncm (subscriber, #165) [Link] They could have grouped duplicates, and separated out vulnerabilities in (1) proprietary products and (2) non-core-OS packages. Neither would have been much work, and neither would count as "interpretation", but they would provoke less faulty interpretation.
The CERT vulnerability list Posted Jan 12, 2006 15:30 UTC (Thu) by RobSeace (subscriber, #4435) [Link] This is a decent breakdown of the CERT figures into something moreuseful and meaningful:
hypocritisy; technical and non-technical arguments against Microsoft Posted Jan 12, 2006 20:33 UTC (Thu) by wilck (subscriber, #29844) [Link] Although the calls for scientific soundness and fairness posted here are certainly valid, I see a certain level of hypocrisy in them. Did we call for the same when the raw numbers looked different some time ago? Instead of just question the numbers, we should realize that Microsoft is working hard to improve the security of ots products. If I look at anti-Microsoft arguments that I have been using in the past, most of the technical ones (poor networking stack, crashes all the time, ...) don't apply to recent Windows versions any more. We should expect the same for the strongest technical argument we have today, security. Perhaps, some day, Windows will actually turn out to be more secure than Linux. But even if that came to pass, I'd still avoid Microsoft products. Because the strongest of all anti-Windows arguments, and the one I have always regarded most important, is non-technical, and it's certain that Microsoft will never disprove it: Microsoft products aren't free.
|
|||||||||||||
