The OpenSSH vulnerability and the disclosure process

The disclosure of a vulnerability is the opening bell of a high-stakes race between crackers, vendors, and system administrators. Crackers will put amazing amounts of time and energy into the rapid creation of exploit tools, which are then distributed to script kiddies and other "black hat" types worldwide. Before those tools are widespread, vendors have to create an updated package, put out an alert, and get system administrators to actually apply those packages. System administrators who lose the race - either because updates were not available, or because they did not apply those updates - run a serious risk of having their systems compromised.

The time window between the disclosure of the vulnerability and the posting of exploit tools can be less than one day.

This grace period before the exploits appear is at the core of why the OpenSSH team acted the way it did. Through careful information and release management, the OpenSSH developers hoped to maximize the amount of time that system administrators had to secure their systems. They wanted OpenSSH users to be able to begin running the race while keeping the crackers sidelined for a little longer.

The first step, thus, was to put out a vague notice that there was a problem, along with an OpenSSH release which contained the problem without actually fixing it. If the OpenSSH team had released a patch which fixed the real problem, it would undoubtedly have been easier for vendors to tell their users if they were vulnerable. It also would have enabled users to secure their systems - if, indeed, they were vulnerable - by simply disabling the challenge/response feature. But it also have given the crackers the information they needed to develop an exploit. Releasing a warning and enabling privilege separation were actions intended to deny the crackers access to that information. As OpenSSH maintainer Theo de Raadt tells us:

In fact, according to the fourth version of the OpenSSH advisory, even telling users about other workarounds would have released too much information:

For most security vulnerabilities, the accepted procedure is to notify vendor security contacts before the community as a whole so that they can prepare an updated package for their users. There are special, "security contacts only" mailing lists which exist for just this sort of notification. In this case, that procedure was not followed; vendors were told no more than anybody else about the nature of the vulnerability. This was a cause for some disgruntlement in the vendor community, which did not like having its response managed in this way. According to Mark Cox, who handles security at Red Hat:

In fact, when the final Red Hat advisory came out, it noted that most users were not vulnerable and provided a patched version of OpenSSH 3.1. Until the disclosure, however, Red Hat (and other distributors) had no option other than preparing a full OpenSSH 3.3 package, fixing the problems, and pushing it onto the users. Keeping the vulnerability information secret most certainly made life harder for distributors. Now that the information is out and the hole closed, distributors like Red Hat can prepare OpenSSH 3.4 packages with full testing prior to release.

The OpenSSH team did not disclose the vulnerability to vendors for a simple reason: they did not trust those vendors to keep the information secret. Quoting Theo de Raadt again:

It does not help, of course, that there are some 80 vendors which ship OpenSSH in some product or other. This remains a disturbing claim, however: the free software security contact mechanism, it is said, is not secure. Then again, perhaps the old Ben Franklin quote applies: three may keep a secret if two of them are dead. It is almost certainly unrealistic to expect 80 vendors to keep something under wraps for very long.

So how can our community function in the claimed absence of a working security infrastructure? Should all vulnerabilities be handled the way this one was? The OpenSSH team claims that this bug was special, for a couple of reasons. One is that OpenSSH is now nearly ubiquitous - there are far more ssh servers exposed to the net than web servers, for example. Thus the vulnerability had to be handled with extra care. The other reason, of course, was that there was a way to protect users against exploits without (immediately) disclosing the nature of the problem. From the OpenSSH advisory:

The real answer, according to Theo, is "fast vendors." In the end, for most users, it is still a matter of how quickly their distributor makes an update available. In this case, the first OpenSSH exploit turned up on Bugtraq 22 hours after the disclosure went out. Opinions certainly differ on the best way to give users a head start, but security in the modern world is still a race.

(As a postscript, the OpenSSH team is recommending that all users upgrade to 3.4, even if they are not vulnerable to this particular problem. It has "lots of other fixes people need.")