Counting vulnerabilities

Mr. Jones found that five Vista vulnerabilities were disclosed in its first 90 days, exactly one of which was fixed by Microsoft. When he looked at Red Hat Enterprise Linux 4WS, the story was a little different: in the first 90 days of RHEL4, Red Hat fixed 181 vulnerabilities and left another 85 without patches. 129 of those vulnerabilities had been disclosed prior to the RHEL4 release. The result is a nice little bar chart showing that RHEL4 was two orders of magnitude worse than Windows Vista for security performance in the first 90 days. Scary stuff.

The numbers posted can be checked, and they are not far out of line. Red Hat published a table showing that a default install of RHEL4 WS suffered from 274 vulnerabilities in its first year of existence. That's a lot of security holes, even after accounting for the fact that a full 52 of them were in Ethereal (now Wireshark).

One could argue that the first 90 days is exactly the period of time one would want to look at if one's goal were to produce the most lopsided result. Before Vista's release, few people had the opportunity to look at it; there was not much outside probing for security holes going on. Vista was initially only available in its business edition, reducing both the scope of the system's functionality and the number of copies distributed. Every component of RHEL4, instead, had been publicly available for months before the system's release. There were no real surprises in RHEL4. The relatively long freeze time involved in the creation of an "enterprise" distribution makes the problem worse; while the world is busily finding (and fixing) security problems in free software, the packages for the upcoming RHEL release are just sitting there waiting to be decreed sufficiently stable. So of course there will be a big pile of RHEL vulnerabilities on the first day of release, and of course Vista will not have the same kind of pile.

Red Hat's response to this situation can be clearly seen on the RHEL4 security updates page. On the day of release, Red Hat put out 27 advisories, many of which fixed more than one vulnerability. For example, the postgreSQL update addresses five different CVE numbers, some of which were clearly worth fixing. First-day fixes also updated php, krb5, cups, KDE, thunderbird, Python, Perl, mailman, and more. Many of these were important fixes, though none of them were deemed "critical" by Red Hat; the first critical updates happened a few weeks later, when bugs in Firefox, HelixPlayer, and Mozilla were fixed.

One could well ask: why does Red Hat not fold these updates into the initial release? If they are good enough to issue on release day, they should be good enough to go directly into the distribution. There are certainly logistics issues here; mirrors would have to be updated and so on. But it's not like the old days when there were thousands of boxed sets to be manufactured. Red Hat could probably find a way to get the first-day updates into the distribution itself. The benefits, however, would be entirely in the area of public relations. The number of deployed RHEL4 systems in the first day (or the first 90 days) will be sufficiently close to zero that the amount of actual exposure caused by the existence of those vulnerabilities is negligible.

In his report, Mr. Jones goes to some trouble to try to filter out some packages which are not available on Windows as a way of heading off criticism that he is not comparing equal systems. But they are still not equal, of course, and never can be. Any default RHEL installation will certainly include Python, for example, and will suffer from Python's vulnerabilities, even if that installation never actually uses Python in a way which makes those vulnerabilities exploitable. Many RHEL4 systems will have installed the vulnerable versions of cvs, xloadimage, mysql, telnet, mailman, gaim, postfix, alsa-lib, vim, gpdf, enscript, Perl, etc.; these are all packages which are missing from a Vista install. The vulnerabilities in these packages are also not exploitable in much (probably a large majority) of RHEL deployments. How many companies deploy RHEL for the purpose of running HelixPlayer, busybox, or elinks?

Then, there's the silly ones. It might be embarrassing that the initial RHEL4 release included a bug with a 1999 CVE number. This vulnerability was in cpio, which neglected to create archive files with the user's umask taken into account. As a result, cpio archives created with the -O option have world read and write permissions granted. This is a bug worth fixing, but it would be amazing if anybody, anywhere, were to actually be affected by this bug. Even so, the cpio vulnerability counts in the total.

Perhaps more to the point, how many vulnerabilities like the cpio hole will ever be disclosed in Vista? No security researcher is likely to bother disclosing a bug like that. If that sort of problem is fixed at all, it will be a quiet part of some service pack update with no public announcement. By so aggressively going after and fixing this kind of security problem, we are causing the number of disclosed vulnerabilities to grow in a way that most proprietary software companies would try to avoid. Finding and fixing these problems remains the right thing to do, though, regardless of who is counting the resulting advisories.

It is also worth pointing out that some of the disclosed vulnerabilities are mitigated by Red Hat's use of exec-shield and SELinux. Red Hat still fixes the bug because it's the right thing to do, but, for some of these vulnerabilities, exploitation is difficult or impossible even without the fix.

The most important points, though, are these: (1) despite the seemingly large number of vulnerabilities, the number of systems actually compromised still seems to be quite low, and (2) this number of vulnerabilities is still far too high, regardless of what any other operating system is doing. It is encouraging that the number of remotely exploitable vulnerabilities is small, but the fact that we are arguably not getting any better at not putting security holes into our code in the first place is discouraging. There is still much to be done in the areas of careful coding, pre-release security auditing, and security-related development tools. Regardless of what one thinks of the methodology of this report, the security bugs that were counted are real; every one of them is a reminder that we can be doing better.