Debian changes OpenSSH packaging
In the wake of the XZ backdoor, the Debian project has revisited some of the patches included in its OpenSSH packages to improve security. The outcome of this is that the project will be splitting out support for Kerberos key exchange into a separate set of packages, though not until after the Debian 13 ("trixie") release expected next year. The impact on Debian users should be minimal, but it is an interesting look into the changes Linux distributions make to upstream software as well as some of the long-term consequences of those choices.
Patches revisited
In April, Colin Watson (one of the OpenSSH package maintainers for
Debian) started
a discussion on the debian-devel mailing list, saying that he was
"reconsidering some choices in Debian's OpenSSH packaging
"
because he hoped to configure the package "more wisely
within more or less our current constraints
". Watson did not
elaborate much beyond that, but it was patching OpenSSH to link to libsystemd
that made Debian and Fedora's packages vulnerable to the XZ backdoor,
which was still very fresh in everyone's mind at the time. See LWN's coverage of how the
XZ backdoor works for more detail.
Debian carries nearly 40 patches against OpenSSH, ranging from fairly large patches like the one that enables GSS-API and changes OpenSSH functionality to small documentation patches. Out of all of the patches Debian applies to OpenSSH, Watson identified four for discussion: the patches for libsystemd, SELinux, TCP wrappers, and Generic Security Service Application Program Interface (GSS-API), which is used to provide a vendor-independent method of authenticating against different Kerberos implementations.
At the time, he said that the libsystemd patch (actually called the
systemd-readiness
patch) was "the obvious
thing on everyone's mind right now
" due to its role in the XZ
backdoor. OpenSSH was patched so that it could notify systemd when it
was ready to accept connections, using the libsystemd library, which
in turn had a dependency on the liblzma library that was
compromised.
Watson said it had seemed reasonable to merge the patch when he did it,
but "we do seem to have ended up on the wrong side of history with that
one
". However, he noted that there was work to remove the
dependency on libsystemd itself, and to move the readiness protocol
upstream, "so we'll get rid of that linkage and reduce our patch load
a bit
".
Debian also compiles OpenSSH with support for SELinux, which
OpenSSH developer Damien Miller
said
should be in an alternate package (along with Kerberos/GSS-API) to reduce
exposure for the bulk of users who may not need the feature. However,
Watson said, "my inclination is to leave this [patch] be
", but
invited SELinux experts to weigh in. No SELinux experts turned up to
voice an opinion one way or another, and that
patch remains.
TCP wrappers
The idea of dropping support for TCP wrappers
drew more discussion. TCP wrappers is a system for filtering access to
services, such as OpenSSH, using host-based access-control lists
(ACLs) placed in /etc/hosts.deny and
/etc/hosts.allow. The upstream OpenSSH project dropped
support for TCP wrappers in 2014, just ahead of the Debian 8
("jessie") freeze. Watson said that he had patched it back in
"temporarily
" and then "dropped the ball on organizing a
proper transition
". He also noted that Fedora had dropped the use
of TCP wrappers in 2018.
A few users were reluctant to lose TCP wrappers, even if the project was considered outdated. Marc Haber said it was much easier to use TCP wrappers to protect his system than to use a packet filter. Marco d'Itri objected to removing support for TCP wrappers since it supports use cases like DNS-based ACLs that are not supported by layer 3 firewalls.
Watson suggested
d'Itri could replace the functionality with tcpd as Fedora
suggested when dropping TCP wrappers, and pointed
out that OpenSSH supports DNS-based ACLs without requiring a
separate library. That did not convince d'Itri, who complained
that the tcpd approach would require socket activation of
sshd "which is not the default for good reasons
". He
did not elaborate on this, but there are reports that socket
activation can be exploited to cause a denial of service, and recent changes that
split
sshd into multiple binaries have made Debian's non-default
support for socket activation more fragile.
Watson wanted to get rid of the TCP wrappers dependency because its library, libwrap, pulls in a heavy dependency chain, including support for the little-used Network Information Service (NIS). He suggested refactoring libwrap to reduce its dependency chain, which d'Itri seems to have done. TCP wrappers support does not appear to be on the chopping block at this point—though Watson hasn't explicitly confirmed that as of yet.
GSS-API key exchange
OpenSSH added initial support for GSS-API in version 3.7 back in 2003. But OpenSSH's GSS-API support still requires SSH host keys to authenticate the server to the user, as Simon Wilkinson pointed out in announcing a version of his GSS-API patch for OpenSSH 5.0p1. (The "p" in the release version indicates it is an OpenSSH portable release for systems other than OpenBSD.) Many users of OpenSSH did not want to maintain both a Kerberos infrastructure and OpenSSH host keys, so the patch allows organizations to use a single security architecture rather than managing both.
In 2004, Matthijs Mohlmann requested that Debian add support for Kerberos to OpenSSH. Ultimately, after some discussion and waiting for the patch to mature, Watson merged the patch in 2005. A number of attempts were made over the years to persuade the OpenSSH developers to accept the patch, but they officially turned it down in 2010. Miller said that none of the OpenSSH developers were in favor of the patch:
This is not a slight on Simon's patch, which is of fine quality, but just that a) we don't trust GSSAPI implementations that much and b) we don't like adding new KEX [key exchange] since they are pre-auth attack surface. This one is particularly scary, since it requires hooks out to typically root-owned system resources.
The patch is now maintained on GitHub by contributors from Debian and Fedora, and both distributions carry a version of the patch in their OpenSSH packages.
In his initial message to debian-devel, Watson said that the
GSS-API patch was the largest patch applied to Debian OpenSSH "by
an order of magnitude, and easily the most intrusive in terms of complexity
and exposure
". He added that he had come to regret merging the
patch a few times over the years, but he was also aware that people
depend on the functionality in the patch. "We've had it for nearly
20 years now, so we can take the time to do a proper job that at least
tries not to leave users in the lurch.
"
His proposal was to create special ("dependency-only
")
packages for Debian trixie that would have names to reflect GSS-API
support. For the trixie release, those packages would effectively be
aliases for Debian's OpenSSH client and server packages, which would
still contain GSS-API support. However, that functionality will be
split out into packages with the same name in the Debian 14 ("forky")
release, so users who have the special packages installed would
automatically get the new packages when they upgrade. The patch would
be dropped from the regular OpenSSH packages in forky. The OpenSSH package
NEWS.debian file and trixie's release notes will also inform
users of the changes.
Michael Stone suggested
speeding things up by doing things the opposite way: creating
"nogss" packages that don't contain the GSS-API patch for Debian
trixie, so users could move to the new, non-GSS-API, version
quickly. "Pushing back the ability to install the unpatched version
for a few more years seems suboptimal.
" Watson said
that he worried about churn and inviting more bugs by approaching it
that way, but he'd consider it.
Split
In the end, though, he went with his original approach. On August 30, he announced that he'd created the special openssh-client-gssapi and openssh-server-gssapi packages with the intent to split out GSS-API key-exchange support and authentication in forky. For now, Debian does not have an OpenSSH package without the GSS-API patch. Users who want those features in forky will need to use the -gssapi packages, and users who want to run OpenSSH without the patch will want the openssh-client and openssh-server packages.
Miller said
this was excellent news: "this substantially reduces the amount of
pre-authentication attack surface exposed on your users' sshd by
default
".
I emailed the maintainers of the OpenSSH packages for Fedora, and Dmitry Belyavskiy said that Fedora will be keeping the patch. He added that the maintainers had discussed following Debian's approach, but they do not plan on getting rid of GSS-API support for now or breaking it out to separate packages.
The transition should be painless for Debian users. Those who depend on the GSS-API patch will have years to ensure that their systems are ready for the transition. Meanwhile, it's an interesting case to ponder for Linux distribution packagers thinking about carrying a patch against an upstream project. A patch applied today may still need tending for years, if not decades.
[Thanks to LWN reader A. Karl Kornel who suggested this story.]
Posted Sep 27, 2024 14:17 UTC (Fri)
by djm (subscriber, #11651)
[Link] (12 responses)
Nit: we patched OpenSSH to notify systemd without using the libsystemd library. We would have done it sooner if there was a published specification for the interface, but documentation on how to interact with systemd outside the actual implementation was lacking. That aside, we really welcome Debian's work to upstream their changes - we've accepted almost everything that they've proposed.
Posted Sep 27, 2024 15:46 UTC (Fri)
by jzb (editor, #7867)
[Link] (1 responses)
Posted Sep 27, 2024 20:03 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted Sep 27, 2024 20:51 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Sep 28, 2024 11:56 UTC (Sat)
by fredrik (subscriber, #232)
[Link] (2 responses)
Using a protocol rather than calling a function in a library dependency is of course an excellent way to isolate the calling code from any vulnerabilities or other problems in a library implementation or its transitive dependencies.
In this case, when the protocol is trivial, it is also an easy choice. Or at least it became trivial once the protocol was documented. Before then, perhaps the most obvious way was to use libsystemd? Just because no other option was apparent.
Anyway, it seems to be a good strategy to reduce attack vectors for vulnerabilities, to prefer integration based on protocols over added library dependencies.
Posted Sep 30, 2024 7:27 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link] (1 responses)
Posted Sep 30, 2024 8:40 UTC (Mon)
by cortana (subscriber, #24596)
[Link]
Posted Sep 28, 2024 16:55 UTC (Sat)
by intelfx (subscriber, #130118)
[Link] (3 responses)
This is pretty disingenuous.
The sd_notify interface was stable since it was first ever documented in 2010, and systemd had *always* offered a standalone implementation of that protocol that was supposed to be imported into other projects's sources.
Posted Sep 28, 2024 19:53 UTC (Sat)
by atnot (subscriber, #124910)
[Link] (2 responses)
But nevertheless, I did implement it myself in python around 2020 and don't think I had to try very hard. So clearly it wasn't impossible to figure this out, that seems ahistorical to me. Especially since to my memory the original discussions about integrating sd-notify were rejected primarily on grounds of "double fork is fine" and "systemd bad" rather than some concern for adding one more linker entry being a deal breaker.
Posted Sep 28, 2024 20:01 UTC (Sat)
by intelfx (subscriber, #130118)
[Link]
The sample implementation in the manpage itself — yes, was added very recently.
But even the very first version of the sd_notify(3) manpage was clearly saying that a reference implementation, designed for vendoring into other projects (and thus written to C89 standards without any extensions), was maintained in src/sd-daemon.[ch] within systemd sources (300 SLOC, of which the majority were attributable to other functions that had no bearing on sd_notify() itself and could be left out if not needed).
Posted Sep 30, 2024 9:02 UTC (Mon)
by grawity (subscriber, #80596)
[Link]
Posted Oct 11, 2024 2:59 UTC (Fri)
by turistu (guest, #164830)
[Link]
except that the abstract unix domain socket "support" in that patch is purely perfunctory -- i.e. it does not work, and hasn't been tested at all. Why was it added in the first place?
from the linked patch:
And btw, why doing a
Also, is
Posted Oct 20, 2024 15:18 UTC (Sun)
by hauleth (guest, #174122)
[Link]
From what I know it was documented at least since 2019, as at that time I have implemented `sd_notify` in Erlang, and I was using manpages for specification.
Posted Sep 27, 2024 14:32 UTC (Fri)
by DemiMarie (subscriber, #164188)
[Link] (1 responses)
Posted Sep 27, 2024 18:58 UTC (Fri)
by hmh (subscriber, #3838)
[Link]
https://salsa.debian.org/ssh-team/openssh/-/blob/master/d...
Posted Sep 28, 2024 4:32 UTC (Sat)
by buck (subscriber, #55985)
[Link]
Within the "enterprise" computing environment where Kerberos gained sway back in the early aughts, to have PuTTY (then with a patch from Vintela, which then became Quest) be able to passwordlessly authenticate one to a Linux host, without having to worry about SSH host keys or users' authorized_keys, was a game-changer. And it still connects us to our Linux hosts' remote desktop sessions to this day, with Simon Tatham having integrated the gsskex functionality into PuTTY itself not too long ago:
https://www.chiark.greenend.org.uk/~sgtatham/putty/wishli...
Granted, we are sitting behind many layered defenses and our servers don't offer up preauth attack surface to the pounding that sshd gets, day in and day out, other places, so I would concede that the OpenSSH developers and DJM in particular have perhaps indeed had better things to do over the years than adding such a sizable chunk of code to their preauth concerns. And I am extremely grateful to them and especially DJM's porting efforts, for making SSH synonymous with remote access, and, arguably (though I am not trying to start an argument), playing a crucial part in Linux' rise to dominance as *the* (container) hosting platform, though I guess the OpenBSD guys might think that a backhanded compliment.
Posted Sep 28, 2024 7:05 UTC (Sat)
by ballombe (subscriber, #9523)
[Link] (6 responses)
Posted Sep 28, 2024 8:11 UTC (Sat)
by smurf (subscriber, #17840)
[Link] (1 responses)
Posted Sep 28, 2024 15:32 UTC (Sat)
by ballombe (subscriber, #9523)
[Link]
Posted Sep 28, 2024 8:35 UTC (Sat)
by Heretic_Blacksheep (guest, #169992)
[Link] (3 responses)
There are certainly exceptions to the above, but for the most part, people are effectively just doing the distribution's equivalent to creating a spec file, doing rpmbuild, uploading to the test server, call it a day hoping that someone using the test level packages uses the software to test the patches before they trickle into stable/LTS deployment.
Posted Sep 28, 2024 18:11 UTC (Sat)
by ballombe (subscriber, #9523)
[Link]
Posted Sep 30, 2024 7:30 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link]
Unless you have some serious source that shows that patches from distributions introduce more issues than there are at large in all software, I think this line of thought isn't very useful.
Posted Oct 2, 2024 13:13 UTC (Wed)
by nim-nim (subscriber, #34454)
[Link]
In my experience, many upstreams vendor third-party code without understanding its architecture or reviewing its code, and many upstreams then patch this vendored code without understanding the implications.
If that looks like how distro packager work that’s exactly like it. What upstreams that vendor and fork severely dislike is all the QA processes of a big distro that are used to detect when third party code use is problematic.
Posted Sep 29, 2024 4:42 UTC (Sun)
by grawity (subscriber, #80596)
[Link]
Both use the exact same GSS-API (and therefore both are equally "Kerberos"), only Debian's patch does it much earlier (in place of host auth) so it is considered more risky than the upstream support (which works only as user authentication).
So it doesn't make any sense to say that "OpenSSH added GSS-API in 2003 but refuses to add Kerberos support" because GSS-API and Kerberos are for all intents and purposes the same feature; what OpenSSH refuses to add is the ability to use it at an earlier time during the connection.
Likewise a package that doesn't contain the Debian gsskex patch would not, by any means, be a "non-GSS-API" package – unless the built-in support gets caught in the crossfire and disabled as well (for guilt by association or something).
nit
OpenSSH was patched so that it could notify systemd when it was ready to accept connections, using the libsystemd library
Sorry - wasn't that _after_ the XZ incident, though? Perhaps I should've written "Debian OpenSSH had been patched" or something similar to make that clearer?
nit
nit
nit
How about MPTCP?
Thanks for integrating based on a protocol rather than dependencies
Thanks for integrating based on a protocol rather than dependencies
Thanks for integrating based on a protocol rather than dependencies
nit
nit
nit
No, the entirety of libsystemd used to be standalone from the beginning – it was originally shipped as a self-contained re: nit
sd-daemon.{c,h} intended to be copy-pasted (whole or in parts) into one's own project, for about a year before it became a shared library in ~2011.
nit
Assuming that
if ((path = getenv("NOTIFY_SOCKET")) == NULL || strlen(path) == 0
...
if (strlcpy(addr.sun_path, path,
...
/* Support for abstract socket */
if (addr.sun_path[0] == '@')
addr.sun_path[0] = 0;
...
if (connect(fd, &addr, sizeof(addr)) != 0) {
NOTIFY_SOCKET is "@/foo" that will not connect to the abstract unix socket with the address "\0/foo" (as intended) but to the one with the address "\0/foo\0\0\0...+ other 100 null bytes".
connect() + write() on a single-use datagram socket instead of a simple sendto()?
close(-1) undefined behaviour that should be checked against?
nit
SELinux integration
SELinux integration
GSSAPI key exchange
patch vs package
Sure you need to maintain it and provide security support, but it is also the case for a package.
patch vs package
patch vs package
It may happen that the author is one of distribution developer, but this is incidental to the point I tried to make unsuccessfuly.
Consider the difference between carrying a patch and packaging a fork that includes the patch.
patch vs package
patch vs package
The article does not give the impression that this applies to the Debian OpenSSH package maintainers.
In my field almost all Debian maintainers are also upstream of related packages.
patch vs package
patch vs package
gsskex or gssauth?