LWN: Comments on "Debian changes OpenSSH packaging"

nit

hauleth — Sun, 20 Oct 2024 15:18:22 +0000

> but documentation on how to interact with systemd outside the actual implementation was lacking

From what I know it was documented at least since 2019, as at that time I have implemented `sd_notify` in Erlang, and I was using manpages for specification.

nit

turistu — Fri, 11 Oct 2024 02:59:53 +0000

except that the abstract unix domain socket "support" in that patch is purely perfunctory -- i.e. it does not work, and hasn't been tested at all. Why was it added in the first place?

from the linked patch:

if ((path = getenv("NOTIFY_SOCKET")) == NULL || strlen(path) == 0
...
if (strlcpy(addr.sun_path, path,
...
/* Support for abstract socket */
if (addr.sun_path[0] == '@')
	addr.sun_path[0] = 0;
...
if (connect(fd, &addr, sizeof(addr)) != 0) {

Assuming that NOTIFY_SOCKET is "@/foo" that will not connect to the abstract unix socket with the address "\0/foo" (as intended) but to the one with the address "\0/foo\0\0\0...+ other 100 null bytes".

And btw, why doing a connect() + write() on a single-use datagram socket instead of a simple sendto()?

Also, is close(-1) undefined behaviour that should be checked against?

patch vs package

nim-nim — Wed, 02 Oct 2024 13:13:30 +0000

> They're volunteers with limited time, resources, and skill sets. They never audit the packages they download, and don't understand the software's architecture enough to understand the full impact or implications of changes 3rd party patches introduce.

In my experience, many upstreams vendor third-party code without understanding its architecture or reviewing its code, and many upstreams then patch this vendored code without understanding the implications.

If that looks like how distro packager work that’s exactly like it. What upstreams that vendor and fork severely dislike is all the QA processes of a big distro that are used to detect when third party code use is problematic.

re: nit

grawity — Mon, 30 Sep 2024 09:02:29 +0000

No, the entirety of libsystemd used to be standalone from the beginning – it was originally shipped as a self-contained sd-daemon.{c,h} intended to be copy-pasted (whole or in parts) into one's own project, for about a year before it became a shared library in ~2011.

Thanks for integrating based on a protocol rather than dependencies

cortana — Mon, 30 Sep 2024 08:40:44 +0000

... which has been documented in sd_notify(3) since systemd v1!

patch vs package

LtWorf — Mon, 30 Sep 2024 07:30:21 +0000

Ah, the ever present upstream developers are genetically better than people who do packages.

Unless you have some serious source that shows that patches from distributions introduce more issues than there are at large in all software, I think this line of thought isn't very useful.

Thanks for integrating based on a protocol rather than dependencies

LtWorf — Mon, 30 Sep 2024 07:27:06 +0000

The "protocol" is a static string to send over a socket.

gsskex or gssauth?

grawity — Sun, 29 Sep 2024 04:42:05 +0000

The article seems to be muddling up two different facets of GSS-API support. One mode (gssapi-with-mic) which has been upstream for years, and another mode (gssapi-keyex) which Debian keeps patching in downstream.

Both use the exact same GSS-API (and therefore both are equally "Kerberos"), only Debian's patch does it much earlier (in place of host auth) so it is considered more risky than the upstream support (which works only as user authentication).

So it doesn't make any sense to say that "OpenSSH added GSS-API in 2003 but refuses to add Kerberos support" because GSS-API and Kerberos are for all intents and purposes the same feature; what OpenSSH refuses to add is the ability to use it at an earlier time during the connection.

Likewise a package that doesn't contain the Debian gsskex patch would not, by any means, be a "non-GSS-API" package – unless the built-in support gets caught in the crossfire and disabled as well (for guilt by association or something).

nit

intelfx — Sat, 28 Sep 2024 20:01:44 +0000

> I'm pretty sure that standalone version was in fact added by bluca in response to discussions on this very website?

The sample implementation in the manpage itself — yes, was added very recently.

But even the very first version of the sd_notify(3) manpage was clearly saying that a reference implementation, designed for vendoring into other projects (and thus written to C89 standards without any extensions), was maintained in src/sd-daemon.[ch] within systemd sources (300 SLOC, of which the majority were attributable to other functions that had no bearing on sd_notify() itself and could be left out if not needed).

nit

atnot — Sat, 28 Sep 2024 19:53:58 +0000

I'm pretty sure that standalone version was in fact added by bluca in response to discussions on this very website?

But nevertheless, I did implement it myself in python around 2020 and don't think I had to try very hard. So clearly it wasn't impossible to figure this out, that seems ahistorical to me. Especially since to my memory the original discussions about integrating sd-notify were rejected primarily on grounds of "double fork is fine" and "systemd bad" rather than some concern for adding one more linker entry being a deal breaker.

patch vs package

ballombe — Sat, 28 Sep 2024 18:11:47 +0000

I suppose it depends on the distribution and the package.
The article does not give the impression that this applies to the Debian OpenSSH package maintainers.
In my field almost all Debian maintainers are also upstream of related packages.

nit

intelfx — Sat, 28 Sep 2024 16:55:49 +0000

> We would have done it sooner if there was a published specification for the interface, but documentation on how to interact with systemd outside the actual implementation was lacking.

This is pretty disingenuous.

The sd_notify interface was stable since it was first ever documented in 2010, and systemd had *always* offered a standalone implementation of that protocol that was supposed to be imported into other projects's sources.

patch vs package

ballombe — Sat, 28 Sep 2024 15:32:54 +0000

Refactoring and maintainance are the work of the patch author/maintainer, not of the distribution carrying it.
It may happen that the author is one of distribution developer, but this is incidental to the point I tried to make unsuccessfuly.
Consider the difference between carrying a patch and packaging a fork that includes the patch.

Thanks for integrating based on a protocol rather than dependencies

fredrik — Sat, 28 Sep 2024 11:56:04 +0000

Feels like stating the obvious here, but: Thank you for using that approach in OpenSSH.

Using a protocol rather than calling a function in a library dependency is of course an excellent way to isolate the calling code from any vulnerabilities or other problems in a library implementation or its transitive dependencies.

In this case, when the protocol is trivial, it is also an easy choice. Or at least it became trivial once the protocol was documented. Before then, perhaps the most obvious way was to use libsystemd? Just because no other option was apparent.

Anyway, it seems to be a good strategy to reduce attack vectors for vulnerabilities, to prefer integration based on protocols over added library dependencies.

patch vs package

Heretic_Blacksheep — Sat, 28 Sep 2024 08:35:20 +0000

How many package maintainers actually help maintain the software beyond some relatively superficial patch sets? My experience in interacting with packagers is that many of are trusting to upstream, adding whatever patches necessary to function and build in their distribution's build and runtime environment, and that's about it. They're volunteers with limited time, resources, and skill sets. They never audit the packages they download, and don't understand the software's architecture enough to understand the full impact or implications of changes 3rd party patches introduce. On top of that, a lot of software has become so complex no single packager can fully understand the code logic of larger projects even if they're part of the upstream team.

There are certainly exceptions to the above, but for the most part, people are effectively just doing the distribution's equivalent to creating a spec file, doing rpmbuild, uploading to the test server, call it a day hoping that someone using the test level packages uses the software to test the patches before they trickle into stable/LTS deployment.

patch vs package

smurf — Sat, 28 Sep 2024 08:11:36 +0000

Well, not exactly. The amount of additional work you need to do when Upstream decides to refactor their code differs by at least an order of magnitude. Also patches might introduce security problems or other bugs that don't exist in the original package, or worse got introduced inadvertently by an upstream change that by itself would be perfectly safe.

patch vs package

ballombe — Sat, 28 Sep 2024 07:05:30 +0000

Carrying a patch is exactly the same as carrying a package. You do it because it serves (a subset of) yours users.
Sure you need to maintain it and provide security support, but it is also the case for a package.

GSSAPI key exchange

buck — Sat, 28 Sep 2024 04:32:08 +0000

Just have to say that I am extremely grateful for Simon Wilkinson's patch and his work making it available, for Colin Watson carrying it so many years for Debian, in league with the Red Hat devs, and for the commitment to carry it onward for however long he does.

Within the "enterprise" computing environment where Kerberos gained sway back in the early aughts, to have PuTTY (then with a patch from Vintela, which then became Quest) be able to passwordlessly authenticate one to a Linux host, without having to worry about SSH host keys or users' authorized_keys, was a game-changer. And it still connects us to our Linux hosts' remote desktop sessions to this day, with Simon Tatham having integrated the gsskex functionality into PuTTY itself not too long ago:

https://www.chiark.greenend.org.uk/~sgtatham/putty/wishli...

Granted, we are sitting behind many layered defenses and our servers don't offer up preauth attack surface to the pounding that sshd gets, day in and day out, other places, so I would concede that the OpenSSH developers and DJM in particular have perhaps indeed had better things to do over the years than adding such a sizable chunk of code to their preauth concerns. And I am extremely grateful to them and especially DJM's porting efforts, for making SSH synonymous with remote access, and, arguably (though I am not trying to start an argument), playing a crucial part in Linux' rise to dominance as *the* (container) hosting platform, though I guess the OpenBSD guys might think that a backhanded compliment.

nit

Cyberax — Fri, 27 Sep 2024 20:51:43 +0000

> we've accepted almost everything that they've proposed.
How about MPTCP?

nit

bluca — Fri, 27 Sep 2024 20:03:35 +0000

Yes, that's correct, most generalist distributions patched it to use libsystemd to fix certain issues, for many years. After the xz exploit we started pushing again to get the patch upstream (the bugzilla ticket about this had been open for years too), open coding the integration instead of using libsystemd, and this was merged.

SELinux integration

hmh — Fri, 27 Sep 2024 18:58:05 +0000

Apparently, so that the client can specify the desired SELinux role for an user that is allowed multiple roles.

https://salsa.debian.org/ssh-team/openssh/-/blob/master/d...

nit

jzb — Fri, 27 Sep 2024 15:46:11 +0000

Sorry - wasn't that _after_ the XZ incident, though? Perhaps I should've written "Debian OpenSSH had been patched" or something similar to make that clearer?

SELinux integration

DemiMarie — Fri, 27 Sep 2024 14:32:58 +0000

Why does OpenSSH need SELinux integration beyond what PAM does?

nit

djm — Fri, 27 Sep 2024 14:17:41 +0000

OpenSSH was patched so that it could notify systemd when it was ready to accept connections, using the libsystemd library

Nit: we patched OpenSSH to notify systemd without using the libsystemd library. We would have done it sooner if there was a published specification for the interface, but documentation on how to interact with systemd outside the actual implementation was lacking.

That aside, we really welcome Debian's work to upstream their changes - we've accepted almost everything that they've proposed.