LWN.net Weekly Edition for July 4, 2002

The OpenSSH vulnerability and the disclosure process

By now, presumably, most of you are running on systems with an updated OpenSSH installed. It has been over a week since the "challenge/response" vulnerability was disclosed; there remains, however, a great deal of controversy over how that disclosure happened. According to one point of view, the OpenSSH team withheld specific information on the vulnerability in order to create fear, bring about massive upgrading, and draw attention away from what is, in the end, an OpenBSD-specific vulnerability. Such a view goes over well in the Linux community, where many users upgraded in a hurry only to find out that they never had been vulnerable in the first place. The real story is more complicated, however; it is worth understanding what was going on and how it reflects on how our security processes work.

The disclosure of a vulnerability is the opening bell of a high-stakes race between crackers, vendors, and system administrators. Crackers will put amazing amounts of time and energy into the rapid creation of exploit tools, which are then distributed to script kiddies and other "black hat" types worldwide. Before those tools are widespread, vendors have to create an updated package, put out an alert, and get system administrators to actually apply those packages. System administrators who lose the race - either because updates were not available, or because they did not apply those updates - run a serious risk of having their systems compromised.

The time window between the disclosure of the vulnerability and the posting of exploit tools can be less than one day.

This grace period before the exploits appear is at the core of why the OpenSSH team acted the way it did. Through careful information and release management, the OpenSSH developers hoped to maximize the amount of time that system administrators had to secure their systems. They wanted OpenSSH users to be able to begin running the race while keeping the crackers sidelined for a little longer.

The first step, thus, was to put out a vague notice that there was a problem, along with an OpenSSH release which contained the problem without actually fixing it. If the OpenSSH team had released a patch which fixed the real problem, it would undoubtedly have been easier for vendors to tell their users if they were vulnerable. It also would have enabled users to secure their systems - if, indeed, they were vulnerable - by simply disabling the challenge/response feature. But it also have given the crackers the information they needed to develop an exploit. Releasing a warning and enabling privilege separation were actions intended to deny the crackers access to that information. As OpenSSH maintainer Theo de Raadt tells us:

In fact, according to the fourth version of the OpenSSH advisory, even telling users about other workarounds would have released too much information:

For most security vulnerabilities, the accepted procedure is to notify vendor security contacts before the community as a whole so that they can prepare an updated package for their users. There are special, "security contacts only" mailing lists which exist for just this sort of notification. In this case, that procedure was not followed; vendors were told no more than anybody else about the nature of the vulnerability. This was a cause for some disgruntlement in the vendor community, which did not like having its response managed in this way. According to Mark Cox, who handles security at Red Hat:

In fact, when the final Red Hat advisory came out, it noted that most users were not vulnerable and provided a patched version of OpenSSH 3.1. Until the disclosure, however, Red Hat (and other distributors) had no option other than preparing a full OpenSSH 3.3 package, fixing the problems, and pushing it onto the users. Keeping the vulnerability information secret most certainly made life harder for distributors. Now that the information is out and the hole closed, distributors like Red Hat can prepare OpenSSH 3.4 packages with full testing prior to release.

The OpenSSH team did not disclose the vulnerability to vendors for a simple reason: they did not trust those vendors to keep the information secret. Quoting Theo de Raadt again:

It does not help, of course, that there are some 80 vendors which ship OpenSSH in some product or other. This remains a disturbing claim, however: the free software security contact mechanism, it is said, is not secure. Then again, perhaps the old Ben Franklin quote applies: three may keep a secret if two of them are dead. It is almost certainly unrealistic to expect 80 vendors to keep something under wraps for very long.

So how can our community function in the claimed absence of a working security infrastructure? Should all vulnerabilities be handled the way this one was? The OpenSSH team claims that this bug was special, for a couple of reasons. One is that OpenSSH is now nearly ubiquitous - there are far more ssh servers exposed to the net than web servers, for example. Thus the vulnerability had to be handled with extra care. The other reason, of course, was that there was a way to protect users against exploits without (immediately) disclosing the nature of the problem. From the OpenSSH advisory:

The real answer, according to Theo, is "fast vendors." In the end, for most users, it is still a matter of how quickly their distributor makes an update available. In this case, the first OpenSSH exploit turned up on Bugtraq 22 hours after the disclosure went out. Opinions certainly differ on the best way to give users a head start, but security in the modern world is still a race.

(As a postscript, the OpenSSH team is recommending that all users upgrade to 3.4, even if they are not vulnerable to this particular problem. It has "lots of other fixes people need.")

Comments (6 posted)

The 2002 Ottawa Linux Symposium

Your editor, tired after a couple of days of Kernel Summit coverage, decided not to produce talk-by-talk coverage from the Ottawa Linux Symposium. Information from some of the talks will show up in LWN over the next week or two; for people wanting the full details the conference proceedings are available online (as a 3MB PDF file).

OLS is increasingly a kernel-oriented event. There were only two GNOME-oriented talks on the schedule this year, and very few others that discussed user-space topics. Kernel topics have always been a big part of OLS, but the kernel is well on the way toward becoming the only topic. Attaching the Kernel Summit to the conference (which might happen again next year) further encourages that trend. That, of course, is entirely acceptable to those of us interested in the kernel. OLS could become the premier worldwide kernel-oriented conference.

Interestingly, the tutorials had a very different orientation, with topics like DocBook and authenticating Windows 2000 users.

Stephen Tweedie talked, in his keynote, of the importance of providing opportunities for hackers to meet face to face. Interactions just go better when you've had a chance to "share a pint" with your collaborators and when you are able to associate a face with the email address. Thus, as a community, we need events like OLS. So it is encouraging to see that OLS attendance was back up this year.

One final note to the joker who thought your editor should win a copy of Running Weblogs With Slash: that's not funny...

Comments (1 posted)

The importance of saying "thanks"

Jon 'maddog' Hall gave a talk at the OLS reception on the first day of the conference. Those who have heard other maddog talks would certainly recognize the collection of "amusing stories from maddog's travels" theme of this one. Mr. Hall did, however, make a new and worthwhile point this time around.

Users of free software (and we all are, in one way or another) often have many things to say to the developers of that software. They send in feature requests and bug reports. They ask where the next release is. They want help making things work. They complain about vulnerability disclosure policies. They post snide comments about the quality of the code or the documentation.

It is relatively uncommon for free software users to simply say "thanks."

Every line of free code is a gift from the developer (or from whoever paid for the developer's effort). Nobody is entitled to free software; it's a windfall, a present from those who created it. All told, it is a gift worth, by most accounts, billions of dollars.

A little gratitude goes a long way. The next time you deal with a developer of a package that you use, consider throwing in a brief "thank you." The developers have earned it.

Comments (8 posted)