Debian, OpenSSL, and a lack of cooperation

A rather nasty security hole in the Debian OpenSSL package has generated a lot of interest—along with a fair amount of controversy—amongst Linux users. The bug has been lurking for up to two years in Debian and other distributions, like Ubuntu, based on it. There are a number of lessons to be learned here about distributions and projects working together or, as in this case, failing to work together.

Back in April 2006, a Debian user reported a problem using the OpenSSL library with valgrind, a tool that can check programs for memory access problems. It was reporting that OpenSSL was using uninitialized memory in parts of the random number generator (RNG) code. Using memory before it is initialized to a known value is a well known way to create hard-to-find bugs, so it is not surprising that the valgrind report caused some consternation.

Debian hacker Kurt Roeckx tracked the problem down to what he thought were two offending lines of code and posted a question on the openssl-dev mailing list:

There were few responses, but they were not opposed to removing the lines, including one from Ulf Möeller using an openssl.org email address: "If it helps with debugging, I'm in favor of removing them." Unfortunately, as was discovered recently, removing one of the two lines was harmless, the other essentially crippled the RNG so that OpenSSL-generated cryptographic keys were easy to predict. (For more technical details on the bug and what should be done to respond to it, see our article on this week's Security page.)

It turns out, at least according to OpenSSL core team member Ben Laurie, that openssl-dev is not for discussing development of OpenSSL. That may be true in practice, but the OpenSSL support web page describes it as: "Discussions on development of the OpenSSL library. Not for application development questions!" In addition, the address suggested by Laurie (openssl-team-AT-openssl.org) does not appear in any of the OpenSSL documentation or web pages. If it wasn't the right place, it would seem that the OpenSSL developers could have provided a helpful pointer to the right address, but that did not occur.

It probably was not clear that Roeckx was asking the questions in an official Debian capacity, nor that he was planning to change the Debian package based on the answer to his questions. As Laurie rightly points out, he should have submitted a patch, proposing that it be accepted into the upstream OpenSSL codebase. That probably would have garnered more attention, even if it was only posted to openssl-dev. It seems very unlikely that the patch in question would have ever made it into an OpenSSL release.

It is in the best interests of everyone, distributions, projects, and users, for changes made downstream to make their way back upstream. In order for that to work, there must be a commitment by downstream entities—typically distributions, but sometimes users—to push their changes upstream. By the same token, projects must actively encourage that kind of activity by helping patch proposals and proposers along. First and foremost, of course, it must be absolutely clear where such communications should take place.

Another recently reported security vulnerability also came about because of a lack of cooperation between the project and distributions. It is vital, especially for core system security packages like OpenSSH and OpenSSL, that upstream and downstream work very closely together. Any changes made in these packages need to be scrutinized carefully by the project team before being released as part of a distribution's package. It is one thing to let some kind of ill-advised patch be made to a game or even an office application package that many use; SSH and SSL form the basis for many of the tools used to protect systems from attackers, so they need to be held to a higher standard.

Another of Laurie's points, which also bears out the need for a higher standard, is the timing of the check-in to a public repository when compared to that of the advisory. Any alert attacker could have made very good use of the five or six day head start, they could have gotten by monitoring the repository, to exploit the vulnerability. While it is certainly possible that some of malicious intent already knew about the flaw, though no exploits have been reported, alerting potential attackers to this kind of hole well in advance of alerting the vulnerable users is unbelievably bad security protocol.

This is the kind of problem that could have been handled quickly and quietly by all concerned. All affected distributions—though it might be difficult to list all of the Debian-derived distributions out there—could have been contacted so that the advisory and updates to affected packages could have been coordinated. One of these days, one of these problems is going to give Linux a security black eye unless the community can do a better job of working together.

As far as I can tell that "day" has already started, the only question is whether it started
back in 2006 or on may 13, but this is unbelievably bad.

Given that the compromised keys are known the ssh client should be updated with the blacklist
as well, so it can:
1) Change its own key and use the old key as a fallback.
2) Remove the compromised keys from the authorized_keys file on the remote systems as soon as
a user logs in.

With the compromised keys in use in tons of authorized_keys files, many of them on systems
that might not get much attention because they weren't subject to the initial problem, it's
only a matter of time before someone generates the keys needed and owns a substantial part of
the machines running ssh in the world.

I've been a great fan of Ubuntu lately, but I'm going to start looking into a more security
conscious distribution (read: something not based on Debian), maybe even FreeBSD which I hear
is getting to be quite acceptable.

New SSH packages that include such a blacklist have hit both ubuntu and debian repos now.

(Personally I'm not going to use those patches on my non-Debian systems: they slow down
connection with a binary search across a multimegabyte file on every connection attempt, they
eat 4Mb of disk space on / and I know none of my keys are vulnerable. But still, it's probably
good if you don't know that. I just hope that someday we can remove those patches again...)

Hi,

this was the best write-up on the subject I read in the past few days. Many thanks. 

I for one think that my subscription money is well spent when I read this articles, clearly
stating 
each side position.

I would encourage you to open this article sooner to the wide public, so that we can spread
the link.

Best regards,

There is one thing in Ben Lauries post though, which I find highly disturbing. That is that he
expects people to know that the openssl team is reachable by openssl-team@... instead of the
address published in both openssls README and the openssl web pages: openssl-dev (where Kurt
Roeckx asked his question). He complains much that Kurt didn't mail to openssl-team, but I
didn't find any mention of that address in openssl source or the openssl web site.

cu

> But Laurie does later say that the Openssl team needs to make it easier for > people to
communicate things like this to it

To communicate things like _what_?  Security holes?  That's not the issue here -- nobody knew
it was as security hole back in 2006 when it was first discussed.  It just seemed like some
valgrind nonsense and the Debian developer didn't know better (which is why he publically
asked for input).  Personally I'm surprised with Laurie's response -- I didn't previously know
that the Debian developer had discussed this on openssl-dev and gotten a "go-ahead" from an
openssl developer!

The "maintainability" problem is unlikely to be changed anytime soon, from what I gather.
Since I started using OpenSSL in '01 (and likely for a number of years before that), there's
always been a -DPURIFY compile option to disable the one line that remained commented out
after the Debian package was fixed.  The docs specifically say that the use of an
uninitialized buffer is intended to increase entropy, and that you should disable it at build
time if you need a purify- or valgrind-friendly version.

It might be better for distros to use existing flags like these rather than diverging from the
upstream release, at least when such flags are available.  The hassle of a makefile mod vs.
the hassle of patching the source again with each new release seems comparable, if not
weighted in favor of the former.

Unfortunately, fixing this bug does not fix the problems it has caused: Many many insecure
certificates have been created out there. From a security POV, it would be appropriate if
newer upstream OpenSSL versions refused to connect to public servers whose certificates can be
attacked by a man in the middle. Otherwise, there is a false sense of security, which is worse
than no security at all. But I don't think there are plans to implement anything like that,
are there??

I think it is important to understand that whatever the problems of the OpenSSL development
team (and I agree there are quite a few), Roeckx did one very wrong thing.

Roeckx introduced a patch into the Debian package without first trying to get it included
upstream.  Trying to include your patch upstream will expose it to the many eyes that make
bugs shallow, and will give you useful feedback; after you've gone through that, and fully
understood why your patch was rejected, you can make an informed decision whether to include
it in your package.

It is unfortunate that many distribution maintainers (and Debian is far from being the worst
in that regard) find it easier, faster, cheaper to just include random hacks into their
packages without trying to push them upstream.

He discussed it on openssl-dev and got a (weak) go-ahead from an openssl developer.  I don't
see this general trend you seem to be hinting at.

Is this not serious enough to warrant the creation of a new protocol version?

ssh protocol 2.2 or 3?

Yes it will hurt. Hard luck; this was a big mistake.

Anyone blaming 'Debian' is foolish in the extreme; this could have happened in any distro.
It's not about Debian, it's about Linux / GNU/Linux / BSD and friends.

This had nothing to do with the protocol, it was a question of a single implementation's PRNG.

I know.
However this has essentially exposed a massive number of protocol 2 implementations. If I run
a non-debian ssh server I still need to upgrade to an sshd that checks the blacklist right?
Since a debian using user could have sent me her debian-generated weak public key? That
account is now unsafe?

So if I make or allow a protocol 2 connection on a non-debian machine am I safe? Maybe; maybe
not.

So bump the protocol and rest assured that anything accepting or making a protocol 2+
connection was implemented after the faulty PRNG debacle and move on.

Would it also avoid the blacklist - no blacklist lookup needed for protocol 2+ ?

I am not, by any means, a naive user - and yet I can't be sure I've correctly updated all my
systems. The fix is complex and subject to human error.

The *real* problem with the code in question is that it was poorly documented.  If you have to
go and ask upstream to understand the security implications of the patch, you have already
lost.  Two identical lines of code were used in the program in two very different contexts,
and the effect of the code relied on external factors (namely caller-provided buffer data),
that was not documented at that particular part of the code.  Code quality is a big issue for
maintainability.