LWN: Comments on "Debian, OpenSSL, and a lack of cooperation"

Debian, OpenSSL, and a lack of cooperation

nix — Wed, 20 Apr 2011 09:43:37 +0000

Wow. Thy cup of bitterness runneth over?

Debian, OpenSSL, and a lack of cooperation

cce_ — Wed, 23 Mar 2011 05:31:19 +0000

There's a much better technical writeup of exactly what Kurt Roeckx got wrong by Gergely Risko. He didn't just comment out a couple of lines because they told him it was okay.

He ignored working -DPURIFY #ifdefs (and advice that they worked, and to use them) that could've easily solved his problem. Then he commented out code that weren't part of his problem (and weren't surrounded by #ifdef PURIFY, a clear signal that it was a dicey idea) out of sheer ignorance.

The guy had no idea what the code he was editing actually DID, and had no business editing OpenSSL without telling anyone. He notified no one on the OpenSSL list that he was about to commit changes that would affect the security of millions of computers.

Read the thread yourself; they gave him good advice (try -DPURIFY) and he ignored it, then never followed up to show them the patch he recklessly committed. The level of negligence and hubris he showed is nearly criminal.

And even worse, Debian never kicked him off his position maintaining OpenSSL; he continues to maintain it today. In 2009 he was appointed Debian Secretary and he was re-appointed in February 2011. Is this how Debian rewards incompetence? I suppose he meant well.

Debian, OpenSSL, and a lack of cooperation

ketilmalde — Thu, 22 May 2008 06:10:21 +0000

The second best writeup. The best one is definitely the one at xkcd.

Lack of documentation

brinkmd — Mon, 19 May 2008 20:56:36 +0000

The *real* problem with the code in question is that it was poorly documented.  If you have to
go and ask upstream to understand the security implications of the patch, you have already
lost.  Two identical lines of code were used in the program in two very different contexts,
and the effect of the code relied on external factors (namely caller-provided buffer data),
that was not documented at that particular part of the code.  Code quality is a big issue for
maintainability.

Debian, OpenSSL, and a lack of cooperation

lbt — Mon, 19 May 2008 13:21:43 +0000

I know.
However this has essentially exposed a massive number of protocol 2 implementations. If I run
a non-debian ssh server I still need to upgrade to an sshd that checks the blacklist right?
Since a debian using user could have sent me her debian-generated weak public key? That
account is now unsafe?

So if I make or allow a protocol 2 connection on a non-debian machine am I safe? Maybe; maybe
not.

So bump the protocol and rest assured that anything accepting or making a protocol 2+
connection was implemented after the faulty PRNG debacle and move on.

Would it also avoid the blacklist - no blacklist lookup needed for protocol 2+ ?

I am not, by any means, a naive user - and yet I can't be sure I've correctly updated all my
systems. The fix is complex and subject to human error.

Debian, OpenSSL, and a lack of cooperation

mmarsh — Sun, 18 May 2008 18:32:53 +0000

The "maintainability" problem is unlikely to be changed anytime soon, from what I gather.
Since I started using OpenSSL in '01 (and likely for a number of years before that), there's
always been a -DPURIFY compile option to disable the one line that remained commented out
after the Debian package was fixed.  The docs specifically say that the use of an
uninitialized buffer is intended to increase entropy, and that you should disable it at build
time if you need a purify- or valgrind-friendly version.

It might be better for distros to use existing flags like these rather than diverging from the
upstream release, at least when such flags are available.  The hassle of a makefile mod vs.
the hassle of patching the source again with each new release seems comparable, if not
weighted in favor of the former.

Debian, OpenSSL, and a lack of cooperation

mmarsh — Sun, 18 May 2008 18:26:07 +0000

This had nothing to do with the protocol, it was a question of a single implementation's PRNG.

Debian, OpenSSL, and a lack of cooperation

lbt — Sun, 18 May 2008 15:47:12 +0000

Is this not serious enough to warrant the creation of a new protocol version?

ssh protocol 2.2 or 3?

Yes it will hurt. Hard luck; this was a big mistake.

Anyone blaming 'Debian' is foolish in the extreme; this could have happened in any distro.
It's not about Debian, it's about Linux / GNU/Linux / BSD and friends.

Debian, OpenSSL, and a lack of cooperation

giraffedata — Sun, 18 May 2008 02:39:27 +0000

To communicate things like _what_?

That there was a maintainability problem (unclean Valgrind report) that might be worth fixing in the base, and that Debian intended to ship to the world a variation of Openssl with that line removed.

But "communicate things like this to [the Openssl team]" is my wording, and doesn't fairly represent Laurie's statement, which was about bidirectional communication.

Bear in mind that Laurie claims the apparent "go-ahead" from an Openssl developer was a communication gap -- that 1) he meant go ahead and do it in a debugging run, not in production; and 2) his advice was not the official, considered advice of the Openssl team, which was never contacted. I know these conclusions are extremely weak, but they are his position nonetheless.

I have on many occasions asked on a mailing list if a certain patch would be OK, been told by someone yes, and then upon submitting the patch, had it rejected. The explanation is always, "you misunderstood; I just said the general principle was OK, as far as I could see without actually looking." You get a whole different kind of review when it comes down to actually merging code.

Debian, OpenSSL, and a lack of cooperation

jimparis — Sat, 17 May 2008 20:16:35 +0000

He discussed it on openssl-dev and got a (weak) go-ahead from an openssl developer.  I don't
see this general trend you seem to be hinting at.

Debian, OpenSSL, and a lack of cooperation

jimparis — Sat, 17 May 2008 20:14:42 +0000

> But Laurie does later say that the Openssl team needs to make it easier for > people to
communicate things like this to it

To communicate things like _what_?  Security holes?  That's not the issue here -- nobody knew
it was as security hole back in 2006 when it was first discussed.  It just seemed like some
valgrind nonsense and the Debian developer didn't know better (which is why he publically
asked for input).  Personally I'm surprised with Laurie's response -- I didn't previously know
that the Debian developer had discussed this on openssl-dev and gotten a "go-ahead" from an
openssl developer!

Debian, OpenSSL, and a lack of cooperation

jch — Sat, 17 May 2008 10:45:57 +0000

I think it is important to understand that whatever the problems of the OpenSSL development
team (and I agree there are quite a few), Roeckx did one very wrong thing.

Roeckx introduced a patch into the Debian package without first trying to get it included
upstream.  Trying to include your patch upstream will expose it to the many eyes that make
bugs shallow, and will give you useful feedback; after you've gone through that, and fully
understood why your patch was rejected, you can make an informed decision whether to include
it in your package.

It is unfortunate that many distribution maintainers (and Debian is far from being the worst
in that regard) find it easier, faster, cheaper to just include random hacks into their
packages without trying to push them upstream.

Please make OpenSSL clients reject insecure certificates

scarabaeus — Fri, 16 May 2008 14:07:04 +0000

Unfortunately, fixing this bug does not fix the problems it has caused: Many many insecure
certificates have been created out there. From a security POV, it would be appropriate if
newer upstream OpenSSL versions refused to connect to public servers whose certificates can be
attacked by a man in the middle. Otherwise, there is a false sense of security, which is worse
than no security at all. But I don't think there are plans to implement anything like that,
are there??

Debian, OpenSSL, and a lack of cooperation

giraffedata — Fri, 16 May 2008 00:55:43 +0000

But Laurie does later say that the Openssl team needs to make it easier for people to communicate things like this to it (I would think that includes publishing an email address where someone is likely to look for it that reaches people in a position to fix upstream Openssl), and that it probably isn't possibly due to lack of resources.

The fundamental problem being pointed out here might be with distribution of resources. Debian apparently has the resources to maintain a private fork of Openssl, but not to maintain the trunk, whereas the trunk seems to be the more effective place to apply resources in a case like this.

One of these days?

nix — Thu, 15 May 2008 10:41:20 +0000

New SSH packages that include such a blacklist have hit both ubuntu and debian repos now.

(Personally I'm not going to use those patches on my non-Debian systems: they slow down
connection with a binary search across a multimegabyte file on every connection attempt, they
eat 4Mb of disk space on / and I know none of my keys are vulnerable. But still, it's probably
good if you don't know that. I just hope that someday we can remove those patches again...)

Debian, OpenSSL, and a lack of cooperation

incase — Thu, 15 May 2008 10:00:18 +0000

There is one thing in Ben Lauries post though, which I find highly disturbing. That is that he
expects people to know that the openssl team is reachable by openssl-team@... instead of the
address published in both openssls README and the openssl web pages: openssl-dev (where Kurt
Roeckx asked his question). He complains much that Kurt didn't mail to openssl-team, but I
didn't find any mention of that address in openssl source or the openssl web site.

cu

Debian, OpenSSL, and a lack of cooperation

melo@simplicidade.org — Thu, 15 May 2008 09:24:00 +0000

Hi,

this was the best write-up on the subject I read in the past few days. Many thanks. 

I for one think that my subscription money is well spent when I read this articles, clearly
stating 
each side position.

I would encourage you to open this article sooner to the wide public, so that we can spread
the link.

Best regards,

One of these days?

dion — Thu, 15 May 2008 07:14:25 +0000

As far as I can tell that "day" has already started, the only question is whether it started
back in 2006 or on may 13, but this is unbelievably bad.

Given that the compromised keys are known the ssh client should be updated with the blacklist
as well, so it can:
1) Change its own key and use the old key as a fallback.
2) Remove the compromised keys from the authorized_keys file on the remote systems as soon as
a user logs in.

With the compromised keys in use in tons of authorized_keys files, many of them on systems
that might not get much attention because they weren't subject to the initial problem, it's
only a matter of time before someone generates the keys needed and owns a substantial part of
the machines running ssh in the world.

I've been a great fan of Ubuntu lately, but I'm going to start looking into a more security
conscious distribution (read: something not based on Debian), maybe even FreeBSD which I hear
is getting to be quite acceptable.