CentOS 5, RHEL 5.6, and security updates

CentOS is forever in catch-up mode. That's because it repackages Red Hat's Enterprise Linux (RHEL) for those who would prefer an enterprise distribution without the costs associated with RHEL. That regularly puts the distribution in something of a pinch, because Red Hat, quite reasonably, follows its own schedule for updates. That pinch is being felt strongly right now with two RHEL releases in quick succession (6.0 followed by 5.6). But it isn't just the distribution developers who are being pinched, as the security updates for CentOS 5 have also been held up by the ongoing work to release CentOS 5.6 and 6.0.

CentOS had already been struggling for a bit in its efforts to put out CentOS 6 after the release of RHEL 6 in November. Then, on January 13, Red Hat released its latest update for RHEL 5, 5.6. At that point, CentOS was faced with a bit of a dilemma: should it focus on 6.0 or work on 5.6 first? The decision was made to work on 5.6 and 6.0 in parallel more or less. That meant that CentOS had two fairly large jobs at hand.

With each Red Hat release, the CentOS developers need to go through the packages and remove any Red Hat-specific elements: artwork, trademarks, %description lines in RPM spec files, and so on. Once that's done, there is a QA process that the packages go through before a final release can be done. Turning RHEL 6 into CentOS 6 is a time-consuming process, but that's also true with 5.6. But there is an additional problem with 5.6: security updates.

Normally, CentOS follows along with Red Hat security updates, releasing its versions as quickly as it can after the RHEL update is released. But 5.6 (or any "point" release of RHEL) comes with a whole slew of updated packages, any of which might have a security update—or be a dependency of a package updated for security reasons. Since there are no CentOS 5.6 packages (yet), these security updates fall into a crack in the CentOS development process. CentOS can either backport the fixes into the 5.5 package, or release an updated 5.6 package along with all of its dependencies, some of which may not have passed the QA process yet.

Except for those updates that Red Hat has marked as "critical", CentOS has chosen to do neither of the above, according to lead developer Karanbir Singh. That may leave its users vulnerable to a number of potentially exploitable security holes. In email, Singh said that the CentOS team is looking at Red Hat's security updates to fix those that are deemed "remotely-exploitable", but that doesn't seem to jibe with what is getting released for CentOS 5. Since the release of RHEL 5.6, there have been no CentOS 5 security updates.

In fact, the last CentOS 5 security update was for the kernel on January 6, a week before RHEL 5.6 dropped. In the interim, Red Hat has released 22 updates, most with "low" or "moderate" impact, but a few that are "important" (two for the kernel, and one each for openoffice.org, krb5, and java-1.6.0-openjdk), and three "critical" bugs (java-1.5.0-ibm, flash-plugin, and java-1.6.0-sun). [Update: As pointed out in the comments (and by Singh), those last three packages are closed source and thus not distributed by CentOS.] There is also a pre-5.6 wireshark vulnerability that has yet to be patched. The full list can be seen here.

It may well be that some of those vulnerabilities only apply to the updated packages that came with RHEL 5.6, but it is extremely unlikely that's true of all of them. The critical Java updates are perhaps the most worrisome, since they come with vague vulnerability descriptions (e.g. "Unspecified vulnerability in the Swing component in Oracle Java SE and Java for Business 6 Update 21, 5.0 Update 25, 1.4.2_27, and 1.3.1_28 allows remote attackers to affect confidentiality, integrity, and availability via unknown vectors."). The critical flash-plugin update is also of concern, but one would guess that there aren't all that many CentOS users running browsers with Flash on a server-oriented distribution.

It's also not at all clear that none of these vulnerabilities are remotely exploitable, as there are, at least, some remote denial of service flaws (which can sometimes be turned into remote exploits). For CentOS installations with untrusted users, there are plenty of locally exploitable flaws in the list. Even without untrusted users, a flaw in a content management system or other web application, for example, may provide an attacker the local access they need to use a local exploit to potentially compromise the entire system.

CentOS is pretty clearly dropping the ball on security updates here, which is probably not what its users expect. While the project is understaffed and is always looking for additional contributors, CentOS 5 users may not be aware that nearly two-dozen security updates (so far) have gone by the wayside while the QA process for 5.6 is ongoing. The CentOS FAQ clearly states that the goal is to have updates available in 72 hours after Red Hat puts them out and, by and large, the project meets that goal—except during the point-release gap.

That gap has stretched longer than the project would like, as Singh notes:

According to Singh, the 5.6 release is imminent ("within the next few days"), which will allow the project to release the updates soon after. There have been complaints in the past about updates that didn't exactly track the upstream RHEL release (i.e. changes for CentOS 5.5 that are not in RHEL 5.5), but doing so in previous releases (e.g. the 5.4 to 5.5 transition) "was the right thing to do" and when there is a "serious threat to user deployed machines, we would do it again", he said.

There has been some discussion of the problem on the centos-devel mailing list, with former CentOS developer Dag Wieers being particularly critical of the delays. He is concerned that users are being misled: "I don't think most of the users ever expected to be without security updates for 10 weeks or more when choosing CentOS, and that is an important characteristic." Singh and others agree that there are things that the project could be doing better, but do not see this as the right time to address those problems. As Singh puts it:

As part of that discussion, though, Singh muddies the waters further about which kinds of security fixes are actually being considered for CentOS 5:

But the reality seems to be rather different, as all manner of vulnerabilities are still languishing in the CentOS 5 tree.

It is a difficult situation for the project. It must necessarily trail the Red Hat releases, and keeping up with security updates while trying to push two releases out is difficult. Doing so would likely push back the releases even further. On the other hand, though, CentOS users may well be unaware that there have been potentially significant updates while they wait for CentOS 5.6. Unless those users follow the RHEL update announcements, they don't even know that there are vulnerabilities they may need to be aware of.

While there are no guarantees about security updates for CentOS (or any other community distribution for that matter), enterprise distribution users tend to expect regular updates, without significant, somewhat arbitrary, gaps. The biggest problem here is really one of communication as the CentOS team should try to make it widely known that security updates are being held back. It probably also makes sense for the project to try to figure out a way to keep up with the update stream even in the point release gap.

Another alternative would be to put CentOS 6 on hold, while focusing on CentOS 5.6. There are, after all, no CentOS 6 users yet, while CentOS 5 has many. It would also be nice to see some of the companies that benefit from CentOS (like various hosting providers, for example) put some effort into helping the project. Those companies are getting an awful lot from CentOS without, visibly at least, putting much back in.