LWN: Comments on "OpenSSH 5.0 released"

OpenSSH 5.0 released

shane — Mon, 14 Apr 2008 18:02:27 +0000

To be fair, there are a lot of public forums on the Internet. Every distribution has one, and software developers do not have the time to follow every single possible channel for bug reports.

Normal procedure is for distributions to push bug reports and other patches upstream to the main developers. In my experience distributions tend to be pretty bad about this process, and maybe frustration with this has contributed to the asinine actions from the OpenSSH developers.

OpenSSH 5.0 released

daniels — Thu, 10 Apr 2008 10:55:43 +0000

Wow, what an incentive.  'Don't make any mistake ever, and we won't spend our entire release
announcement calling you idiots.'  Your generosity is matched only by your kindness.

(Of course, Novell employees are idiots when they make a typo introducing one security bug,
but whoever made this mistake is doubtless not an idiot.)

OpenSSH 5.0 released

nix — Tue, 08 Apr 2008 06:00:13 +0000

Well, libxml 2.6 only came out in 2003, so I'm not sure this counts. (Also 
there's not a commitment to pretty much never bump the version again.)

:)

OpenSSH 5.0 released

jengelh — Tue, 08 Apr 2008 01:31:59 +0000

When was the last time there were significant  just the basic parsing, no DOM/SAX voodoo  in
libxml 2.6?!

OpenSSH 5.0 released

nix — Mon, 07 Apr 2008 22:55:24 +0000

I was actually more interested in a WHY. i.e. a simple feature list and 
answers to the question 'you have this feature, why is it there? What was 
it intended for? What's its purpose?' `How do you use it' is, if anything, 
secondary. Right now we can't answer that question because we don't know 
what features *exist* unless we read the entire source base, because many 
of them (e.g. subsystems) are pretty much completely undocumented.

This sort of rationale thing is the sort of question for which an answer 
*must* exist (or why was the feature added in the first place?) but which 
generally the only people who know the answers are the people who added 
the features in the first place :(

(but I agree with the lack-of-manpower part. It's just surprising that, 
given the percentage of security problems caused by unknowing misuse of 
security features, someone on a project as security-obsessed as OpenBSD 
hasn't found the time.)

OpenSSH 5.0 released

djm — Mon, 07 Apr 2008 21:59:44 +0000

No, we don't have a good HOWTO guide. That's because nobody has stepped up to write one - they
don't write themselves...

OpenSSH 5.0 released

nix — Mon, 07 Apr 2008 15:11:51 +0000

Hear, hear. This is the single biggest fault in OpenSSH, to my mind, and 
one I hear constantly from users. It's this great strong 
authentication-and-connection toolkit, but other than the manpages and a 
book you have to buy (so nobody does: ssh is often a small component of 
larger systems, so the documentation should be incorporatable into the 
docs for those systems: but it's in book form, so it isn't), there is *no 
useful documentation*, and for a security system that is *really* 
unfortunate, because if you make an error in configuring a security system 
you tend to introduce insecurity without even knowing it. (This may or may 
not be true of OpenSSH: I can't tell.)

I mean I've been using OpenSSH since it forked from SSH: I read the 
changelogs religiously and know the manpages pretty much by heart, but I 
still have *no* idea what the moduli file is for, what the consequences 
are when it gets changed in some OpenSSH release (there must be 
consequences or 'make install' would overwrite, but there can't be 
consequences or newer fresh installs would have trouble talking to older 
upgraded installs), why I might want to use  a different MAC from the 
default, why I might want to use a different cipher from the default (that 
I *think* I know, but where security is concerned that's not really good 
enough)... I only know how subsystems work because I read the code, and 
when I did I was jumping around babbling about how cool this is, oh look 
you can attach anything that can chatter to stdin/stdout as a subsystem 
and ride on OpenSSH's authorization/authentication/networking layer...

... but nobody who hasn't read the code knows that, and things like that 
really should not lie hidden for ten years because of a total absence of 
useful docs.


*Are* there useful docs somewhere? (The manpages, for these purposes, are 
not useful. There's too much they leave unsaid. Printed books are 
minimally useful: you can't grep them, you never have them around when you 
need them, and you can't point confused customers in Australia at them 
when they're having trouble getting their end of some SSH tunnel up).

Otherwise, well, I'd write some better docs, but that's sort of hard since 
there's too much I don't know. Someone who actually knows SSH has to do 
it, and as far as I can tell that set consists largely of the maintainers.

OpenSSH 5.0 released

dwmw2 — Mon, 07 Apr 2008 14:16:31 +0000

I'm sorry; I didn't mean to mislead. I did say "bugs/RFEs", and it was just an example.

Personally, I count the first two as bugs and only the last as an RFE — I do consider the forwarding of X clients to the "wrong" display to be a bug rather than a missing feature, and it isn't so much of a hack to use our own locally-generated MIT-MAGIC-COOKIE to differentiate between clients. I thought your objection was mostly that we have a similar bug with agent forwarding which is harder to fix, and that fixing one but not the other would be inconsistent, even though the agent forwarding bug is much less often an issue (in my experience, never). But this probably isn't the correct forum for that discussion.

I could perhaps have also included bug #1349, but I haven't been carrying that patch in my builds for so long.

Anyway, those are just in my personal builds. The important thing is that we get everything from the distribution(s) properly considered for merging upstream. There are too many patches outstanding for my liking, for whatever reasons.

OpenSSH 5.0 released

gmaxwell — Mon, 07 Apr 2008 13:25:34 +0000

SRP has lots of scary patents surrounding it.

My understanding is that Stanford University has granted a royalty free license to the basic form of SRP as described in RFC-2945. Stanford has been pretty loudly beating "it's free! it's free!" drum. ... and at least some groups seem to have bought into these claims... For example, GNUTLS includes SRP and I'd expect them to be somewhat patent paranoid.

While I wouldn't be at all surprised to discover that Stanford's claims are inaccurate, it sure would be nice to have a good reference on the patent problems. Any suggestions?

The lack of an automatic PKI in SSH is a serious impediment to security in the real world. While OpenSSH provides all the tools needed for a skilled user to be secure, real users simply do not understand or use them. MITM attacks against SSH work in the real world, not only against unskilled users but against technically competent ones as well. The classic solutions to this class of problems are too centralized, too complex, or simply too labor-intense to address SSH's needs.

In my opinion SRP would result in too large an increase in effective security to be ignored. Whatever patent concerns exist need to be sorted out, because the current state of affairs is harmful to the public.

OpenSSH 5.0 released

djm — Mon, 07 Apr 2008 12:14:05 +0000

I should add that most (all?) of the _bugs_ filed by the Redhat maintainer (Thomas Mraz)
against OpenSSH are closed, mostly because he writes great bug reports and makes his patches
very easy to merge. He is probably the easiest distribution representative to deal with.

OpenSSH 5.0 released

djm — Mon, 07 Apr 2008 12:06:35 +0000

Your commentary is quite misleading: of the three patches you list, only one is a bug (and is
arguably not) - the other two are enhancements. One of these (multiple X11 forwarding) is not
supported by the SSH protocol without hacks, and I have given you quite a reasoned explanation
why we aren't pursuing your approach - that is not unresponsiveness, just a disagreement over
what features should live in the product.

OpenSSH 5.0 released

djm — Mon, 07 Apr 2008 11:59:30 +0000

SRP has lots of scary patents surrounding it.

OpenSSH 5.0 released

jond — Mon, 07 Apr 2008 10:21:23 +0000

To quote that part of the social contract in full:

"We will keep our entire bug report database open for public view at all times. Reports that
people file online will promptly become visible to others."

OpenSSH 5.0 released

dwmw2 — Mon, 07 Apr 2008 10:07:12 +0000

madscientist writes:

You're right about all these errors, but you left out the one specific error I called out as the most egregious one:
x) Red Hat does not report the bug and its fix upstream to OpenSSH back in 2005, when they found it.

This is fairly unfortunate, and definitely not Fedora policy. For both selfish and altruistic reasons, we really do try to merge patches upstream as promptly as possible.

OpenSSH is a bit special here, since it seems so hard to get patches merged. It's very unfortunate that we carry so many patches, but after my own experience with bugs/RFEs #1328, #1329 and #1330, for which I've been building my own packages for years and occasionally trying to merge the patches but getting nowhere, I can't really criticise our OpenSSH package maintainer for that.

Looking through the (unfortunately private) bug report in RHEL bugzilla, it seems that it was originally reported to us with the text "Grrr. This is a *known* sshd bug...", which probably made it seem even less necessary for the package maintainer to chase it to the recalcitrant upstream.

Still, maybe this is a good time for us to improve matters by trying to flush all our pending patches to upstream, and for upstream to start being a little more receptive to them.

OpenSSH 5.0 released

gmaxwell — Mon, 07 Apr 2008 05:44:14 +0000

Another release without SRP support. *Sigh*.

OpenSSH 5.0 released

smoogen — Sun, 06 Apr 2008 22:17:01 +0000

They have one for OpenBSD and one for the rest of the world. The rest of the world covers
other BSD's. The OpenBSD tree is what they are going to worry about the most because they see
it integrating and working with their OS. Theo, Marcus, etc were quite clear that they do not
know Solaris, Linux, etc and really did not have the time or energy to do so. That is up to
people who know FreeBSD, Solaris, Linux, MacOS etc to put what they know into it.

It definitely matters.

smoogen — Sun, 06 Apr 2008 22:13:13 +0000

Thankyou for the multiple explanations djm.

It definitely matters.

djm — Sat, 05 Apr 2008 07:37:44 +0000

Cranking the major version mattered back when we were making major functional improvements or
incompatible change. OpenSSH 2.0 brought in SSH protocol version 2 support for the first time.
OpenSSH 3.0 changed some defaults in a way that could affect some users.

Nowadays OpenSSH is a mature product, so we just increment the version when we do a timed
(~every 6 months) release.

OpenSSH 5.0 released

man_ls — Fri, 04 Apr 2008 22:36:42 +0000

I don't think it counts as "hiding"; it is just "discreetly fixing before revealing"; other innocent parties can get hurt if this kind of information is disclosed. And after all the security team is not supposed to sit on security bugs indefinitely.

It definitely matters.

rsidd — Fri, 04 Apr 2008 15:47:01 +0000

Another example: OpenSSH 1.x -> OpenSSH 2.x.  Of course, they were following the lead of
commercial SSH in this.

It definitely matters.

jospoortvliet — Fri, 04 Apr 2008 15:45:52 +0000

Indeed. I would be much more careful to upgrade from 4.9 to 5.0 than from 5.0 to 5.1 or 5.1 to
5.1.1, yet it seems for OpenSSH you can simply never tell the size of the changes by version
number... Which is pretty annoying.

OpenSSH 5.0 released

madscientist — Fri, 04 Apr 2008 15:18:40 +0000

You're right about all these errors, but you left out the one specific error I called out as
the most egregious one:

x) Red Hat does not report the bug and its fix upstream to OpenSSH back in 2005, when they
found it.

Of course it can be quite difficult to determine whether a bug is a security risk, especially
if that's not how you came across it in the first place.  I certainly don't blame Red Hat for
not recognizing the security implications.  Regardless, however, good community participation
demands that when you find a bug you report it upstream to the developers.  It's true we don't
know for a fact that this was not done, but we can have a reasonable suspicion since there was
no indication of this in the bug report, and the bug was not fixed upstream after all this
time.

It definitely matters.

beoba — Fri, 04 Apr 2008 15:10:05 +0000

In many cases, yes. For example, Python 3.x is not planned to be compatible with 2.x. Same
with PHP 4.x vs PHP 5.x, or Apache 1.x vs Apache 2.x. There are lots of cases where a change
in major version number means "features have changed, you will need to reconfigure".

I'd much rather be able to just follow standard version syntax rather than have to memorize
which incremental updates are the ones that are likely to disrupt things. Who wants to spend
their time memorizing that foo 1.2 -> 1.3 involves a complete rewrite, while foo 1.3 -> 1.4 is
only fixing some documentation?

Heck, even a date-string (foo-YYYYMMDD) would be more useful, because then you'd at least get
some indication of "this is the first release in a year, so there are likely some major
changes".

OpenSSH 5.0 released

madscientist — Fri, 04 Apr 2008 15:07:12 +0000

Actually, I didn't say there were 50+ patches.  What I said was that THIS patch has been
applied to 50+ different Red Hat openssh RPM versions, since it was first discovered in 2005.
I have no idea how many different patches Red Hat is carrying for openssh, although that would
be interesting to look at.

You're right, we don't know whether Red Hat tried to report this upstream, and was rebuffed,
or not.  However, there's no indication (in the bug report etc.) that such an attempt was
made.  I have also heard rumors that it can be difficult to get bugs reported upstream for
openssh, probably because of the odd way it is packaged where the developers maintain only the
OpenBSD port and other people are left to maintain all the other ports: in some cases it might
be tricky to convince the developers that the bug exists in the original OpenBSD version as
well.

Still, as I said, even without the security implications being reported it seems like this is
an obvious bug that they would have been interested to hear about.

um...

elanthis — Fri, 04 Apr 2008 14:00:27 +0000

But does it really matter?  All you really need to know is, "new version is out, it has more
features and/or bug fixes, probably time to upgrade."

OpenSSH 5.0 released

nhippi — Fri, 04 Apr 2008 13:29:16 +0000

> However, to my mind the real fault here lies with XXX

I think it was rather a unfortunate chain of multiple human errors. Blaming a sigle point
misses the whole point, when fiasco could have been avoided had almost anyone in the chain of
events not done their mistake:

x) confusing ipv4/ipv6 behaviour on socket bind

If someone as experienced as openssh developers fall to this, how many other will/have done
the same mistake? This api does not seem to follow the principle of least surprise.

x) openssh developers creating the bug

Not fully understanding the api you are using is quite common source of security problems.
Requesting an tcp listening socket and then telling X to connect to the ipv4 address via
$DISPLAY might seem reasonable to assume safe when ipv6 doesn't pop into your mind.

x) fedora fixing the bug without realizing the the security implications

Has also happened often before. Especially when dealing with software as critical to security
as openssh, every bug should be considered as a security bug until proven wrong. It is
unfortunate that distributions hold bugfixes in their packages without pushing them upstream..

x) reporting the security bug to debian via inproper channel

For a enduser, finding out the proper way of reporting bugs is hard enough. Now it requires
even more to find out that there is a separate channel or security issues. Now documentation
might help but most people don't read them anyway. debbugs could support security bugs better
(as it already supports the security tag the user was properly using..)

x) reporting it to wrong upstream contact

Different person than above, but basically the same mistake.

x) Debian security not watching "security" tagged bugs closely.

Presumably caused by lack of manpower.

x) pushing the update to Debian not via the security channel

While it was somewhat reasonable to expect that people before him in buglog had contacted the
upstream properly, A bigger mistake was to upload to unstable directly instead notifying
debian security team to do security uploads to stable and testing.

While the specific people have probably learned by now and will not repeat their mistake(s),
new people will join and eventually do similar mistakes. It's quite hard to avoid human
mistakes without same time making contributing inhumanly hard.

OpenSSH 5.0 released

jhubbard83 — Fri, 04 Apr 2008 13:16:42 +0000

Here's the Fedora openssh repo, if anyone is interested.
http://cvs.fedoraproject.org/viewcvs/rpms/openssh/

OpenSSH 5.0 released

jhubbard83 — Fri, 04 Apr 2008 13:15:31 +0000

I don't know the real reason so this is just conjecture.  Maintainers will carry around
patches for a package that others find useful.  RedHat/Fedora really do want to patches to go
into upstream so that they don't have to carry them around.  

You've indicated that there are 50+ patches that they have been carrying since 2005.  I don't
know if that is true or not, but that would seem to indicate that it is hard to get patches
accepted in upstream for whatever reason.  The package maintainer has probably accepted that
requests for patches being accepted will fall on deaf ears and has not bothered to attempt to
get the supplier of the patch to send it upstream.  

I couldn't find any links quickly, but I have seen posts in the past about the difficulty of
getting patches into openssh.  At some point people will stop caring and won't attempt it
anymore.  This is not just an issue with OpenSSH but it common to other projects as well.
Con Kolivas's decision to stop providing is a high visibility example.If someone at openssh
really wanted, they could probably pull the source for the major distibutions packages and see
what they're putting in them.

Don't get me wrong I'm not trying to cast blame onto anyone, it's just the way life is and
people make mistakes. Fix it and go on.

OpenSSH 5.0 released

jond — Fri, 04 Apr 2008 13:13:47 +0000

Hiding a bug in this fashion would violate Debian's social contract ("We will not hide
problems").

OpenSSH 5.0 released

djm — Fri, 04 Apr 2008 10:18:15 +0000

Not everyone uses PAM - OpenBSD uses "BSD auth" (derived from BSDI IIRC), which is IMO a much
cleaner and safer design than PAM. Most of the differences lie in missing library routines or
systems calls, or ones with different semantics. [uw]tmp/lastlog handling (actually login
handling in general) is another area of wild divergence.

Yes, we use some tools to merge changes from the OpenBSD tree to the portable tree - it isn't
much of an overhead at all.

OpenSSH 5.0 released

nix — Fri, 04 Apr 2008 08:31:47 +0000

That would be too much like less ;)

(really they're emulating Emacs. 1.3, 1.4, 15, 16, 17... ;P )

emacs 22

coriordan — Fri, 04 Apr 2008 07:46:38 +0000

The GNU Emacs developers were stuck on version 1.x for years, so they eventually just droped
the "1." and went straight from version 1.12 to version 13.  They're up to version 22.2 now
:-)

OpenSSH 5.0 released

Wummel — Fri, 04 Apr 2008 06:55:04 +0000

I wonder why they have to release two SSH branches - one for BSD and one portable for all other Unixes.
On the Portable SSH page they claim a lot of differences in authentication between UNIXes. Isn't PAM now a standard library in pretty much all UNIX implementations? Or is it the crypto stuff that differs?

It seems harder to maintain and verify two codelines instead of a merged one, especially from a security point of view. But the developers may have a magic ingredient for this (some half-automatic syncing mechanism maybe).

OpenSSH 5.0 released

bronson — Fri, 04 Apr 2008 06:36:58 +0000

Wow, two of the first three paragraphs of that press release are spent complaining about
Debian.  Tell us more about good procedure?  Especially as related to discrete communication?
:)

um...

qu1j0t3 — Fri, 04 Apr 2008 05:29:02 +0000

But when "real progress" occurs, that's exactly when you DO bump the major number!

By making the same increment for all changes, whether "one security fix" or "6 months of heavy
development", you're reducing the informational value (at least in common practice) that is
implied, for example, in the bump from 4.9.0 to 4.9.1 compared to 4.9.1 to 5.0.

Disclaimer: Of course reading the version number does not substitute for reading the release
notes. :)

OpenSSH 5.0 released

micah — Fri, 04 Apr 2008 04:06:13 +0000


Its a hard claim to accept that Debian didn't follow good procedure when the bug was reported
in a public forum and existed there for nearly 3 months for all the internet to see. If this
bug was reported via private channels such as vendor-sec and then Debian went about releasing
it to the world anyways, then yes you would be right, but once its out there, its out there.
I'm sorry, but there is no undo on the internet.

OpenSSH 5.0 released

drag — Fri, 04 Apr 2008 03:11:16 +0000

Because the . is just what they've used?

It's not unusual for the number releases to change their level of importance as a project
matures. The x.0 may have mattered more in the past, but it doesn't mean so much anymore.

OpenSSH 5.0 released

abartlet — Fri, 04 Apr 2008 02:10:47 +0000

It is to do so in the release announcement.  Leave such things for a private (or at least not
in the announce mail) discussion with the debian maintainer.

OpenSSH 5.0 released

ttrafford — Fri, 04 Apr 2008 01:25:31 +0000

"we treat our version number as a simple counter"

Any reason it's not just version "50" then?

OpenSSH 5.0 released

nix — Fri, 04 Apr 2008 01:03:46 +0000

I don't know, that might work. Some projects are forever at version 2.6.x, 
and they seem to do all right ;}