Debian, OpenSSL, and a lack of cooperation

A rather nasty security hole in the Debian OpenSSL package has generated a lot of interest—along with a fair amount of controversy—amongst Linux users. The bug has been lurking for up to two years in Debian and other distributions, like Ubuntu, based on it. There are a number of lessons to be learned here about distributions and projects working together or, as in this case, failing to work together.

Back in April 2006, a Debian user reported a problem using the OpenSSL library with valgrind, a tool that can check programs for memory access problems. It was reporting that OpenSSL was using uninitialized memory in parts of the random number generator (RNG) code. Using memory before it is initialized to a known value is a well known way to create hard-to-find bugs, so it is not surprising that the valgrind report caused some consternation.

Debian hacker Kurt Roeckx tracked the problem down to what he thought were two offending lines of code and posted a question on the openssl-dev mailing list:

There were few responses, but they were not opposed to removing the lines, including one from Ulf Möeller using an openssl.org email address: "If it helps with debugging, I'm in favor of removing them." Unfortunately, as was discovered recently, removing one of the two lines was harmless, the other essentially crippled the RNG so that OpenSSL-generated cryptographic keys were easy to predict. (For more technical details on the bug and what should be done to respond to it, see our article on this week's Security page.)

It turns out, at least according to OpenSSL core team member Ben Laurie, that openssl-dev is not for discussing development of OpenSSL. That may be true in practice, but the OpenSSL support web page describes it as: "Discussions on development of the OpenSSL library. Not for application development questions!" In addition, the address suggested by Laurie (openssl-team-AT-openssl.org) does not appear in any of the OpenSSL documentation or web pages. If it wasn't the right place, it would seem that the OpenSSL developers could have provided a helpful pointer to the right address, but that did not occur.

It probably was not clear that Roeckx was asking the questions in an official Debian capacity, nor that he was planning to change the Debian package based on the answer to his questions. As Laurie rightly points out, he should have submitted a patch, proposing that it be accepted into the upstream OpenSSL codebase. That probably would have garnered more attention, even if it was only posted to openssl-dev. It seems very unlikely that the patch in question would have ever made it into an OpenSSL release.

It is in the best interests of everyone, distributions, projects, and users, for changes made downstream to make their way back upstream. In order for that to work, there must be a commitment by downstream entities—typically distributions, but sometimes users—to push their changes upstream. By the same token, projects must actively encourage that kind of activity by helping patch proposals and proposers along. First and foremost, of course, it must be absolutely clear where such communications should take place.

Another recently reported security vulnerability also came about because of a lack of cooperation between the project and distributions. It is vital, especially for core system security packages like OpenSSH and OpenSSL, that upstream and downstream work very closely together. Any changes made in these packages need to be scrutinized carefully by the project team before being released as part of a distribution's package. It is one thing to let some kind of ill-advised patch be made to a game or even an office application package that many use; SSH and SSL form the basis for many of the tools used to protect systems from attackers, so they need to be held to a higher standard.

Another of Laurie's points, which also bears out the need for a higher standard, is the timing of the check-in to a public repository when compared to that of the advisory. Any alert attacker could have made very good use of the five or six day head start, they could have gotten by monitoring the repository, to exploit the vulnerability. While it is certainly possible that some of malicious intent already knew about the flaw, though no exploits have been reported, alerting potential attackers to this kind of hole well in advance of alerting the vulnerable users is unbelievably bad security protocol.

This is the kind of problem that could have been handled quickly and quietly by all concerned. All affected distributions—though it might be difficult to list all of the Debian-derived distributions out there—could have been contacted so that the advisory and updates to affected packages could have been coordinated. One of these days, one of these problems is going to give Linux a security black eye unless the community can do a better job of working together.