OpenSSH bug falls through the cracks

Linux distributions often patch the software they distribute, to fix bugs or add features. Anything they add is pushed upstream to the project responsible for the package—at least in theory. When that theory is not borne out in practice, it can lead to the kind of unhappiness and finger pointing that went along with a recent OpenSSH release. The release notes point at Debian for failing to report it upstream, but the bug was actually fixed much earlier, in Red Hat Enterprise Linux 4 (RHEL4).

The bug in question is rather nasty, allowing a local attacker to hijack X Windows programs of a user who logged in using ssh with X forwarding enabled. Under those circumstances, the ssh client and server arrange that any X programs started on the logged-in machine actually display on the client machine's desktop. This is very useful for running X programs across the internet as the X traffic is encrypted as part of the ssh session.

Due to a broken interaction with Internet Protocol version 6 (IPv6)—the next generation protocol for internet traffic—ssh can get confused about the port number of the X server. If a particular port (which maps to the X DISPLAY environment variable) is not available to be used under IPv4—the protocol in use today—but is available under IPv6, the ssh server will incorrectly set DISPLAY. If it is an attacker's program that is listening on the IPv4 port, it will be able to hijack X programs that get run.

Up until sometime in the last several years, this would not have happened for most Linux boxes because IPv6 was generally not enabled. In that case, the ssh server would recognize that it could not get the port it wanted and try another, eventually setting DISPLAY correctly. Because IPv6 is much newer, these kinds of bugs may exist in other network programs. This bug should serve as a reminder to developers to closely check their IPv6 support.

Clearly, though, the bug fell through the cracks. The OpenSSH team shows its annoyance in the release notes:

It was reported in January to the Debian bug tracking system, but not fixed and released until late March. OpenSSH does releases every six months or so, with 4.9 being released on March 30. Having to turn around another release four days later to fix a problem that was known for a few months could certainly make for annoyed developers. So how did the bug get fixed in Debian, with a Common Vulnerabilities and Exposures (CVE) number being assigned, but without notifying the OpenSSH team?

The Debian bug entry is instructive, because it documents some of the steps that led to the hurried release. In particular, Phil Miller thought he had done the right thing to report the problem in February:

That email must have gotten lost or been eaten by a spam filter as de Raadt would presumably have gotten it to the right people had he seen it. The bug description clearly puts it in the realm of a security problem, but the bug was not classified that way in the Debian system. Had it been, it would have been handled differently, possibly triggering an email to the proper place. But the bug report also shows that Red Hat fixed it in 2005.

It was reported to Red Hat by a customer and got entered into their bugzilla as bug #163732. Unfortunately, that bug report is confidential because it contains potentially sensitive customer information. This makes it difficult to track further. Indications are that it was not seen as a security problem and that it was believed to have been already known as an OpenSSH bug. Apparently no one checked to make sure the OpenSSH folks knew of it though.

Closer cooperation between the OpenSSH maintainers for Red Hat and the upstream team would probably have helped. Red Hat has been carrying the patch along for quite some time. Because the security implications were not clear and the patch is quite simple, it may not have seemed to be all that necessary to get it upstream. Though, there are more than twenty patches listed in the fedora OpenSSH CVS repository for rawhide, which will become Fedora 9.

The OpenSSH team would be well served by paying closer attention to various distribution patches to their code as well. It is certainly plausible that those interested in finding security holes to exploit might start by seeing if any patches floating around for critical services like OpenSSH were useful. By being more proactive, OpenSSH might have found and fixed this bug much earlier. The way this particular bug avoided notice seems to be mostly happenstance; if there is blame to be placed, there is plenty to go around.

RHEL and other "enterprise" distributions have long support cycles which means that the versions of various packages being maintained are well behind the upstream project. It doesn't take very many bug reports getting shot down because they have already been fixed in a more recent version before distribution maintainers lose enthusiasm for making those reports. But it is an essential part of the process. The OpenSSH team has the reputation of being somewhat difficult to work with, which may have helped this particular problem get overlooked.

It is a difficult problem to solve fully. Distributions have their own set of requirements which may be in opposition to those of the upstream project. Those projects may also have policies and procedures that distributions are not up to speed on. The Linux kernel often sees the same kind of conflicts, which is why distributions often maintain their own set of kernel patches for features their customers need. But it is in everyone's best interest to work those problems out so that distributions carry along as few patches as possible while upstream projects do not miss out on bug fixes and features.