By now, presumably, most of you are running on systems with an updated
OpenSSH installed. It has been over a week since the
"challenge/response" vulnerability was disclosed; there remains, however, a
great deal of controversy over how that disclosure happened. According to
one point of view, the OpenSSH team withheld specific information on the
vulnerability in order to create fear, bring about massive upgrading, and
draw attention away from what is, in the end, an OpenBSD-specific
vulnerability. Such a view goes over well in the Linux community, where
many users upgraded in a hurry only to find out that they never had been
vulnerable in the first place. The real story is more complicated,
however; it is worth understanding what was going on and how it reflects on
how our security processes work.
The disclosure of a vulnerability is the opening bell of a high-stakes race
between crackers, vendors, and system administrators. Crackers will put
amazing amounts of time and energy into the rapid creation of exploit tools,
which are then distributed to script kiddies and other "black hat" types
worldwide. Before those tools are widespread, vendors have to create an
updated package, put out an alert, and get system administrators to
actually apply those packages. System administrators who lose the race -
either because updates were not available, or because they did not apply
those updates - run a serious risk of having their systems compromised.
The time window between the disclosure of the vulnerability and the posting
of exploit tools can be less than one day.
This grace period before the exploits appear is at the core of why the
OpenSSH team acted the way it did. Through careful information and release
management, the OpenSSH developers hoped to maximize the amount of time
that system administrators had to secure their systems. They wanted
OpenSSH users to be able to begin running the race while keeping the
crackers sidelined for a little longer.
The first step, thus, was to put out a vague notice that there was a
problem, along with an OpenSSH release which contained the problem without
actually fixing it. If the OpenSSH team had released a patch which fixed
the real problem, it would undoubtedly have been easier for vendors to tell
their users if they were vulnerable. It also would have enabled users to
secure their systems - if, indeed, they were vulnerable - by simply
disabling the challenge/response feature. But it also have given the
crackers the information they needed to develop an exploit. Releasing a
warning and enabling privilege separation were actions intended to deny the
crackers access to that information. As OpenSSH maintainer Theo de Raadt
The warning permitted the community to move to privsep, filter
their networks, or disable ssh for a window of time until a new
In fact, according to the fourth version of the
OpenSSH advisory, even telling users about other workarounds would have
released too much information:
We could not alert the community that disabling
ChallengeResponseAuthentication solved the problem, since this
would highlight that the bug is in about 500 out of 27,000 lines of
For most security vulnerabilities, the accepted procedure is to notify
vendor security contacts before the community as a whole so that they can
prepare an updated package for their users. There are special, "security
contacts only" mailing lists which exist for just this sort of
notification. In this case, that procedure was not followed; vendors were
told no more than anybody else about the nature of the vulnerability. This
was a cause for some disgruntlement in the vendor community, which did not
like having its response managed in this way. According to Mark Cox, who
handles security at Red Hat:
At Red Hat we try to backport security fixes if that means they
will have a lower impact on our users and therefore be easier to
install. We try to avoid switching users to new upstream versions
of software where there have been significant non-security fixes
also added. In this particular case we were asked by the OpenSSH
team to switch our userbase onto a brand new release of OpenSSH
that had a significant functionality change, and one that was not
completely working in all cases.
In fact, when the final Red Hat
advisory came out, it noted that most users were not vulnerable and
provided a patched version of OpenSSH 3.1. Until the disclosure,
however, Red Hat (and other distributors) had no option other than
preparing a full OpenSSH 3.3 package, fixing the problems, and pushing
it onto the users. Keeping the vulnerability information secret most
certainly made life harder for distributors. Now that the information is
out and the hole closed, distributors like Red Hat can prepare
OpenSSH 3.4 packages with full testing prior to release.
The OpenSSH team did not disclose the vulnerability to vendors for a simple
reason: they did not trust those vendors to keep the information secret.
Quoting Theo de Raadt again:
It has been shown that these mailing lists do not work, and that
they leak information into blackhat or public forums very
I've seen leaks happen. Last week, the resolver issue was released
extremely quickly (too quickly I think) because leaks started
moving through the FreeBSD and NetBSD communities within hours of
their security contacts being informed.
It does not help, of course, that there are some 80 vendors which ship
OpenSSH in some product or other. This remains a disturbing claim,
however: the free software security contact mechanism, it is said, is not
secure. Then again, perhaps the old Ben Franklin quote applies: three may
keep a secret if two of them are dead. It is almost certainly unrealistic
to expect 80 vendors to keep something under wraps for very long.
So how can our community function in the claimed absence of a working
security infrastructure? Should all vulnerabilities be handled the way
this one was? The OpenSSH team claims that this bug was special, for a
couple of reasons. One is that OpenSSH is now nearly ubiquitous - there
are far more ssh servers exposed to the net than web servers, for example.
Thus the vulnerability had to be handled with extra care. The other
reason, of course, was that there was a way to protect users against
exploits without (immediately) disclosing the nature of the problem. From
the OpenSSH advisory:
We feel that this method of releasing served the community best for
a "contained" vulnerability of this kind. We do not suggest this
is necessarily the correct information release process for all
problems, and as firm believers of full disclosure have never
suggested that, though we believe that disclosure must be carefully
The real answer, according to Theo, is "fast vendors." In the end, for
most users, it is still a matter of how quickly their distributor makes an
update available. In this case, the first OpenSSH exploit turned up on
Bugtraq 22 hours after the disclosure went out.
Opinions certainly differ on the best way to give users a
head start, but security in the modern world is still a race.
(As a postscript, the OpenSSH team is recommending that all users upgrade
to 3.4, even if they are not vulnerable to this particular problem. It has
"lots of other fixes people need.")
Comments (6 posted)
Your editor, tired after a couple of days of Kernel Summit coverage,
decided not to produce talk-by-talk coverage from the Ottawa Linux
Symposium. Information from some of the talks will show up in LWN
over the next week or two; for people wanting the full details
the conference proceedings are available online (as a 3MB
OLS is increasingly a kernel-oriented event. There were only two
GNOME-oriented talks on the schedule this year, and very few others that
discussed user-space topics. Kernel topics have always been a big part of
OLS, but the kernel is well on the way toward becoming the only topic.
Attaching the Kernel Summit to the conference (which might happen again
next year) further encourages that trend. That, of course, is entirely
acceptable to those of us interested in the kernel. OLS could become
the premier worldwide kernel-oriented conference.
Interestingly, the tutorials had a very different orientation, with topics
like DocBook and authenticating Windows 2000 users.
Stephen Tweedie talked, in his keynote, of the importance of providing
opportunities for hackers to meet face to face. Interactions just go
better when you've had a chance to "share a pint" with your collaborators
and when you are able to associate a face with the email address. Thus, as
a community, we need events like OLS. So it is encouraging to see
that OLS attendance was back up this year.
One final note to the joker who thought your editor should win a copy of
With Slash: that's not funny...
Comments (1 posted)
Jon 'maddog' Hall gave a talk at the OLS reception on the first day of the
conference. Those who have heard other maddog talks would certainly
recognize the collection of "amusing stories from maddog's travels" theme
of this one. Mr. Hall did, however, make a new and worthwhile point this
Users of free software (and we all are, in one way or another) often have
many things to say to the developers of that software. They send in
feature requests and bug reports. They ask where the next release is.
They want help making things work. They complain about vulnerability
disclosure policies. They post snide comments about the quality of the
code or the documentation.
It is relatively uncommon for free software users to simply say "thanks."
Every line of free code is a gift from the developer (or from whoever paid
for the developer's effort). Nobody is entitled to free software; it's a
windfall, a present from those who created it. All told, it is a gift
worth, by most accounts, billions of dollars.
A little gratitude goes a long way. The next time you deal with a
developer of a package that you use, consider throwing in a brief "thank
you." The developers have earned it.
Comments (8 posted)
Page editor: Jonathan Corbet
Inside this week's LWN.net Weekly Edition
- Security: OWASP's premier release; TCPA FAQ; Apache worm on the loose
- Kernel: A safe <tt>SCHED_IDLE</tt> patch; taskfile and XBox; improving SCSI
- Distributions: News from Debian, Mandrake, Red Hat, SuSE and Yellow Dog; New distribution - uClibcLinux
- Development: AxKit, Ogg Traffic, mnoGoSearch 3.1.20, KDE 3.0.2, Pygame 1.5,
FLTK 1.1.0rc4, KOffice 1.2beta2 is, KWinTV Rewrite Alpha 1,
Bluefish 0.7, Tracking Software Development Projects.
- Commerce: Why MandrakeSoft will not join UnitedLinux; IBM launches "Linux Virtual Services"
- Press: Disappearing Open Source Vendors, Caldera CEO switchover, Opera in
China, Linux in Biotech, $200K for Linux on the Xbox.
- Announcements: LPI News, LSB v1.2, the Free Software Distribution Project, Debconf 2, LinuxDailyNews, Loads of Linux Links v1.0.1.
- Letters: Security vulnerabilities; user-friendly Linux