LWN: Comments on "OpenSSH 9.5 released"

Forking

elenril — Sat, 07 Oct 2023 10:12:37 +0000

This has been my experience as well, my patch to implement RFC5014 (IPv6 source address preference, currently only available on Linux) was rejected because OpenBSD kernel does not support this.

Forking

gioele — Fri, 06 Oct 2023 12:51:25 +0000

> For the software I'm used to thinking about, there's no user-specific config (it's system-wide software, so doesn't load from $HOME), and you set argv as part of the service manager's configuration, rather than running it by hand.

I agree, but that does not mean that the software does not already have code in place used to merge ARGV with the built-in config values and those found in /usr.

And even system-wide software sometimes reads from $HOME-like configuration files. For example Apache has directory-specific .htaccess files, cron has user-specific crontabs.

Forking

gdiscry — Fri, 06 Oct 2023 12:07:31 +0000

And you can also add environment variables between argv and the configuration files.

Forking

farnz — Fri, 06 Oct 2023 11:22:27 +0000

For the software I'm used to thinking about, there's no user-specific config (it's system-wide software, so doesn't load from $HOME), and you set argv as part of the service manager's configuration, rather than running it by hand.

Forking

gioele — Fri, 06 Oct 2023 11:20:38 +0000

> Merging then happens at runtime; the software loads its built-in config, then loads the config from /usr (overwriting the built-in config where the two conflict), then loads the config from /etc (overwriting both built-in and /usr configs where the two conflict).

You stopped too early. Then the software loads the user-specific config in $HOME (overriding built-in, /usr and /etc configs). And then it loads other settings from ARGV (overriding built-in, /usr, /etc and $HOME configs).

I do not really understand how people that are used to the decades-old mechanism of ARGV overriding $HOME overriding /etc overriding built-in defaults, are now against having just another level of override in /usr (an additional level that brings many practical benefits).

OpenSSH 9.5 released

bluca — Fri, 06 Oct 2023 10:44:46 +0000

and unlike mosh, you can also have functional scrollback, which is kinda useful

Forking

bluca — Fri, 06 Oct 2023 10:43:09 +0000

Oh but everyone absolutely loves spending hours manually solving merge conflicts due to whitespace or comment changes in multiple thousand-lines-long config files, it's the _superior_ mechanism don't you know /s

Forking

farnz — Fri, 06 Oct 2023 10:32:29 +0000

If it's done sensibly, it works a heck of a lot better than merging conffiles on upgrade. The software has built-in defaults, which are expected to be reasonable. The distribution puts its overrides for the software defaults in /usr, covering cases where the distribution does things differently to upstream, but where upstream shouldn't, can't or won't change its default to match the distribution. Then the syadmin owns /etc completely, and the package manager doesn't touch it at all - it's entirely yours.

Merging then happens at runtime; the software loads its built-in config, then loads the config from /usr (overwriting the built-in config where the two conflict), then loads the config from /etc (overwriting both built-in and /usr configs where the two conflict).

OpenSSH 9.5 released

kilobyte — Fri, 06 Oct 2023 10:14:30 +0000

A longer comparison: mosh uses 0-1 connections, mptcp 1-∞. Mosh can survive a total disconnection, although that protects you only from losing the session's contents, you still sit there doing nothing until you manually deconfigure your landline. MPTCP on the other hand requires at least one flow to work all the time, but hops between them with very low latency. All you notice is a small lag spike.

Mosh is also limited to interactive sessions only. It can't scp, can't tunnel, can't jump host, can't X forward, etc.

It's also a separate program that uses ssh only for initial setup (authentication, setting up the two UDP endpoints) -- after that, mosh doesn't interact with ssh at all. MPTCP on the other hand is a generic TCP extension that's usable with all programs.

They have distinct use cases, just with an overlap for roaming.

Forking

Cyberax — Fri, 06 Oct 2023 06:51:04 +0000

The problem is that GAI can't distinguish between plain TCP and MPTCP. And it can't just default to MPTCP everywhere, because it can subtly break backcompat.

That's why GAI doesn't support transparent IDNs either. And btw, OpenSSH also doesn't support IDNs.

OpenSSH 9.5 released

Shoragan — Fri, 06 Oct 2023 06:49:30 +0000

Yes. But unlike mosh, it can be used with connection sharing and for jump hosts.

Forking

djm — Fri, 06 Oct 2023 02:09:56 +0000

> > So that bug is the one were we literally merged server-side Linux VRF support. Client-side support was rejected for the reasons explained in the bug. The BSD support for VRF is AFAIK exactly as capable as the Linux support, so this doesn't bolster your case that we "routinely" reject patches because they focus on Linux.
> You are proving my point for me: given BSD doesn't have native client support, then Linux is not allowed to have it either, and has to use dangerous workarounds with setuid binaries instead.

No, you're again misrepresenting what described in that bug. Linux client support is in the exact same state as OpenBSD client support - you need to use an external tool to pin a ssh process to an rdomain. This has nothing to do with openbsd vs linux, since their support is basically equivalent and everything to do with not reimplementing things that can be delegated to existing tools. sshd got more extensive support (including Linux support from day one) because the existing external tools couldn't cover the important usecases, not because of any anything to do with it being easier on some platform.

Anyway, I'm done arguing here. If we didn't care about Linux then we could ditch libaudit events, selinux support and a heap of compat support and our lives would be significantly simpler. We put a lot of effort into keeping OpenSSH useful and consistent across a wide range of platforms, so it's pretty frustrating when people denigrate the portability work by saying we don't care.

Forking

djm — Fri, 06 Oct 2023 00:50:12 +0000

> It also can't be cleanly integrated into GAI without breaking legacy apps that do stupid things.

disagree. It could be an optional ai_flags flag and would then need no subsequent special-casing

Forking

bluca — Thu, 05 Oct 2023 22:13:51 +0000

> Wide adoption meaning people actually using it, or at least requesting it. Apart from that PR and the conversation here, I've heard very little demand and have never encountered it at $dayjob.

There's demand enough that people bothered to implement it, and others are complaining that it isn't there. I'm not sure how there could be people using OpenSSH with MPTCP before it becomes available, it's a catch 22. For the protocol itself, MPTCP was used on IOS and I think on some Android phone too, which means there are not only more users of this Linux networking feature than any BSD-specific networking feature openssh has ever implemented, but than all BSD users combined, and by orders of magnitude.

> So that bug is the one were we literally merged server-side Linux VRF support. Client-side support was rejected for the reasons explained in the bug. The BSD support for VRF is AFAIK exactly as capable as the Linux support, so this doesn't bolster your case that we "routinely" reject patches because they focus on Linux.

You are proving my point for me: given BSD doesn't have native client support, then Linux is not allowed to have it either, and has to use dangerous workarounds with setuid binaries instead.

> OpenSSH already supports AF_UNIX for a number of things mostly around forwarding. It doesn't (yet) support it for connections, because nobody has got around to reviewing and merging that patch. Once again, this has nothing to do with it being Linux-specific (it absolutely isn't) and everything to do with it being a niche feature.
> If your point was that we're conservative in adopting new features or that we're slow to examine larger external contributions then I'd probably agree with you. We're a small team with limited bandwidth and our own opinions about what we like to work on. But I don't think you've successfully demonstrated that we have some special bias against Linux.

No, my point was that Linux-specific features have impossibly demanding bars to meet to even be considered, and even then only if there's a BSD equivalent, while the opposite is not the case. I mean, it's your project, so of course you are fully entitled to do whatever you want with it.

Forking

Cyberax — Thu, 05 Oct 2023 21:49:58 +0000

> Yeah, I get that. I don't think that OpenSSH is the project that has to go first though.

SSH is _the_ poster child for MPTCP. It depends on long-lived TCP sessions and doesn't support reconnects. Very few other tools need that support.

> 1. That applications have to fiddle and override ai_protocol makes me think that MPTCP is not well integrated with the system.

Not many things _need_ it. It also can't be cleanly integrated into GAI without breaking legacy apps that do stupid things.

Forking

djm — Thu, 05 Oct 2023 21:41:42 +0000

>>Yeah, we've not implemented this yet. IMO it's reasonable to wait and see if this gains wide adoption before implementing it. New TCP-like networking protocols have come along before (e.g. SCTP) and have not been widely adopted.
>That's reasonable but it hits a chicken-and-egg problem: can't use a feature that the program you're using doesn't support.

Yeah, I get that. I don't think that OpenSSH is the project that has to go first though.

> given how this particular feature lives 99% in the kernel, making the patch nothing but:
> option? IPPROTO_MPTCP : ai->ai_protocol
> plus configuration, which should be simple enough to not take any non-trivial effort from you.

I'd say two things about this:

1. That applications have to fiddle and override ai_protocol makes me think that MPTCP is not well integrated with the system. Why should applications need to do that at all? ai_protocol should be getting set appropriately by getaddrinfo and the application should just use that without having to special-case anything.

2. You're right that this alone is not a big change but once we accept something like this it's very hard to remove it, and 20+ years of small features/options add up to a substantial maintenance and testing burden.

OpenSSH 9.5 released

djm — Thu, 05 Oct 2023 21:31:44 +0000

> What does this mean: "make PerSourceMaxStartups first-match-wins"?
> First match of what?

OpenSSH's configuration language is supposed to be first-match-wins, i.e. the first instance of a directive that is encountered is the one that is used and subsequent instances are ignored. The configuration parser had a bug that broke this for PerSourceMaxStartups and accidentally considered all instances of the directive.

Forking

djm — Thu, 05 Oct 2023 21:28:53 +0000

> It's been a thing for like 15 years, it's used on Linux, OSX and IOS. But of course if by "wide adoption" you mean "it's implemented on BSD", then yeah, in that sense it doesn't have "wide adoption" yet, and probably never will

Wide adoption meaning people actually using it, or at least requesting it. Apart from that PR and the conversation here, I've heard very little demand and have never encountered it at $dayjob.

> No, it does not. Or rather, it supports the BSD version of VRF very well, which I'm sure it's great for both its users, but the openssh client lacks native support for Linux's VRF, and the patch for it was rejected: https://bugzilla.mindrot.org/show_bug.cgi?id=2784 and requires some horrendous workarounds that needs either root or setuid binaries, which is just a great idea as we've seen the other day with the setuid CVE in glibc

So that bug is the one were we literally merged server-side Linux VRF support. Client-side support was rejected for the reasons explained in the bug. The BSD support for VRF is AFAIK exactly as capable as the Linux support, so this doesn't bolster your case that we "routinely" reject patches because they focus on Linux.

> True, I was thinking of AF_UNIX there https://github.com/openssh/openssh-portable/pull/435 AF_VSOCK hasn't been tried yet given it doesn't support AF_UNIX which is much simpler

OpenSSH already supports AF_UNIX for a number of things mostly around forwarding. It doesn't (yet) support it for connections, because nobody has got around to reviewing and merging that patch. Once again, this has nothing to do with it being Linux-specific (it absolutely isn't) and everything to do with it being a niche feature.

If your point was that we're conservative in adopting new features or that we're slow to examine larger external contributions then I'd probably agree with you. We're a small team with limited bandwidth and our own opinions about what we like to work on. But I don't think you've successfully demonstrated that we have some special bias against Linux.

OpenSSH 9.5 released

zdzichu — Thu, 05 Oct 2023 21:12:41 +0000

Ah, like 'mosh' then.

Forking

bluca — Thu, 05 Oct 2023 19:41:42 +0000

Yeah, notoriously there are never upgrades on MicroOS, you just throw the computer away and buy a new one

Forking

kilobyte — Thu, 05 Oct 2023 19:28:21 +0000

It's something for distros that lack any conffile handling. Of course, that means you can forget about upgrades as no changes to configuration files will be merged.

OpenSSH 9.5 released

Cyberax — Thu, 05 Oct 2023 19:24:19 +0000

> Next, with that done, is MP-TCP an improvement in daily use, in terms of user experience?

It's a pretty good improvement.

My standard use-case: I have a laptop plugged into a display, that also has wired Ethernet. I disconnect the laptop and move to a couch, and the SSH sessions survive the switch to WiFi.

Forking

bluca — Thu, 05 Oct 2023 14:21:25 +0000

yes, it has been for years, that's how it's done for image-based OSes like MicroOS, Silverblue and others: https://uapi-group.org/specifications/specs/configuration_files_specification/
SUSE has even built a library for it

OpenSSH 9.5 released

xnor — Thu, 05 Oct 2023 14:14:54 +0000

What does this mean: "make PerSourceMaxStartups first-match-wins"?
First match of what?

OpenSSH 9.5 released

paulj — Thu, 05 Oct 2023 13:32:42 +0000

Right, and how many hosts are running mptcpd?

Next, with that done, is MP-TCP an improvement in daily use, in terms of user experience? How well does MP-TCP switch - i.e. how reliably does it distinguish congestion loss from path failure?

Lab results are one thing, real world use is another. The paucity of real-world use to date makes it hard to say, doesn't it? (If there are users regularly using it, would love to hear their experience).

Forking

job — Thu, 05 Oct 2023 13:17:41 +0000

> default config files in /usr/

Wait what? Is that a thing?

OpenSSH 9.5 released

kilobyte — Thu, 05 Oct 2023 11:52:07 +0000

Kernel support, if you don't need mixing IPv4+IPv6 flows in one connection, is there since 5.6; in Debian land this means two stable releases.

You also need host configuration and (trivial) enabling in program you use.

As for host configuration, you need:
* multiple configured interfaces with an external route (and common networking tools tend to sabotage that, eg. connman if you connect your phone drops the route for wlan0 preferring usb0 over it, including dropping routes you add by hand. Other high-level tools like to do the same.)
* tell the kernel it can use an address as an endpoint. This is not done by default as it may be a privacy issue (the server you talk to will learn your other external IPs -- but hey, that's required for packets to be delivered...). For 99% of us there's a daemon, mptcpd, that adds new interfaces as they pop up.
* note that the other side doesn't need anything, the above configuration matters only if you actually want to use multiple interfaces on _your_ side

As for the program, for trivial enablement it just needs to specify protocol as IPPROTO_MPTCP; the vast majority of programs are not interested in fine-tuning or observing the details. An example for the latter might be eg. a game wanting to catch multiplaying bastards -- old code would log only the initial IP while knowing the rest would make catching cheaters easier (and this shows the privacy issue if it's you who's a cheating bastard...). But as for ssh, what would it even do with such knowledge? An admin who wants to log everything can use `ip mptcp monitor` without altering sshd code.

If you want openssh packages with the openssh-portable PR#335 rebased, take them from me: https://angband.pl/debian/pool/main/o/openssh/ -- applied as-is, ie, disabled in config by default (edit it on both sides!).
There's also mptcpize (LD_PRELOAD) and some eBPF replacer, but these are hacks.

OpenSSH 9.5 released

paulj — Thu, 05 Oct 2023 11:11:53 +0000

How well does MPTCP work, and how widely is it used? There are still some issues with the MP-TCP congestion control, meaning that your statement "the bandwidth gets summed" is often not true - sometimes far from true.

Even if in the Linux kernel, there are not many MPTCP enabled hosts out there, are there?

Forking

kilobyte — Thu, 05 Oct 2023 10:56:21 +0000

> It's been a thing for like 15 years, it's used on Linux, OSX and IOS.

That's not entirely true. There are two incompatible protocols: experimental MPTCPv0 (RFC 6824, 10 years old) and MPTCPv1 (RFC 8684, 3 years old). You can support both, but given how much the old protocol sucked, in Linux it has been ripped out entirely, replaced by MPTCPv1 in kernel 5.6.

I see other platforms that claim MPTCPv0 support: OSX, IOS, IOS (iPhones and Cisco, respectively), some stuff I don't recognize, and... FreeBSD?!? Don't know if that patchset has been merged, but in either case it could be a good starting point for you.

As for usage, MPTCPv0 is used in iOS (the iApple kind) when communicating with its mothership; it enables them to avoid reconnect latency by continuing an existing session instead of having the old one time out then a new one be built up.

I don't know the details behind the decision to drop MPTCPv0 completely, but given how much we suffer by misdesigns in TCP itself, that's probably for the better.

But then, the above talk is about *kernel* support; ssh support can be an one-liner...

Forking

kilobyte — Thu, 05 Oct 2023 10:09:21 +0000

>Yeah, we've not implemented this yet. IMO it's reasonable to wait and see if this gains wide adoption before implementing it. New TCP-like networking protocols have come along before (e.g. SCTP) and have not been widely adopted.

That's reasonable but it hits a chicken-and-egg problem: can't use a feature that the program you're using doesn't support. And given how this particular feature lives 99% in the kernel, making the patch nothing but:
option? IPPROTO_MPTCP : ai->ai_protocol
plus configuration, which should be simple enough to not take any non-trivial effort from you.

Also given how aggressively at least the Linux implementation falls back to regular TCP it might even be reasonable to drop the configuration bits and hard-code the flag (falling back for old kernels).

As for SCTP, yes, it's a good comparison. Both protocols implement similar functionality, and SCTP is far more mature (but lacking lessons we learned since) and not hobbled by having to conform to TCP. The problem is, with idiot ISPs you can't use SCTP on today's Internet:
* 20 years ago, ISPs workers knew well that non-TCP non-UDP are stuff evil hackers use, customers need to be protected. And ICMP is not a protocol but some magic.
* today, ISP workers know that there are only two protocols: TCP and UDP

Thus, if you lack the tuits to implement MPTCP in OpenBSD today, it would be great if you could for now enable it on Linux and see if people actually use it.

Forking

bluca — Thu, 05 Oct 2023 09:40:26 +0000

> > MPTCP
> Yeah, we've not implemented this yet. IMO it's reasonable to wait and see if this gains wide adoption before implementing it. New TCP-like networking protocols have come along before (e.g. SCTP) and have not been widely adopted.

It's been a thing for like 15 years, it's used on Linux, OSX and IOS. But of course if by "wide adoption" you mean "it's implemented on BSD", then yeah, in that sense it doesn't have "wide adoption" yet, and probably never will

> OpenSSH already has better VRF support than most other daemons, including Linux support. I'm not aware of outstanding feature requests in this area, but if they came in then we'd definitely look at them.

No, it does not. Or rather, it supports the BSD version of VRF very well, which I'm sure it's great for both its users, but the openssh client lacks native support for Linux's VRF, and the patch for it was rejected: https://bugzilla.mindrot.org/show_bug.cgi?id=2784 and requires some horrendous workarounds that needs either root or setuid binaries, which is just a great idea as we've seen the other day with the setuid CVE in glibc

> > default config files in /usr/
> Again, I'm not aware of feature requests for this that have been rejected. It seem pretty possible using existing compile-time knobs + Include directives.

It is not possible to use existing knobs for this, and it has been rejected when a SUSE developer approached the project to talk about this RFE. If it wasn't from you, maybe it was from another maintainer. openSUSE has to ship some downstream patches and workarounds because of that to support transactional updates in MicroOS.

> > AF_VSOCK
> Again, nobody has asked for this.

True, I was thinking of AF_UNIX there https://github.com/openssh/openssh-portable/pull/435 AF_VSOCK hasn't been tried yet given it doesn't support AF_UNIX which is much simpler

OpenSSH 9.5 released

farnz — Thu, 05 Oct 2023 08:58:29 +0000

On the other hand, it failed CI - unfortunately, logs are no longer available because of the time since that patch, but that in itself is concerning.

Forking

djm — Thu, 05 Oct 2023 01:57:56 +0000

> They routinely reject non-BSD specific features and enhancements.

Citation needed

> MPTCP

Yeah, we've not implemented this yet. IMO it's reasonable to wait and see if this gains wide adoption before implementing it. New TCP-like networking protocols have come along before (e.g. SCTP) and have not been widely adopted.

> VRF

OpenSSH already has better VRF support than most other daemons, including Linux support. I'm not aware of outstanding feature requests in this area, but if they came in then we'd definitely look at them.

> default config files in /usr/

Again, I'm not aware of feature requests for this that have been rejected. It seem pretty possible using existing compile-time knobs + Include directives.

> AF_VSOCK

Again, nobody has asked for this.

> It's not for lack of trying

Apparently not, because more than half of your examples haven't been tried.

Forking

mpr22 — Wed, 04 Oct 2023 23:33:25 +0000

By "adopt", you mean "reimplement from scratch, because they can't take implementations from the Linux kernel for obvious reasons".

Forking

zwol — Wed, 04 Oct 2023 23:30:26 +0000

If these features are really so great, maybe you should be trying to convince the BSD kernel people to adopt them.

OpenSSH 9.5 released

Sesse — Wed, 04 Oct 2023 22:19:20 +0000

Wow, that patch doesn't exactly strike me as a huge maintenance burden.

Forking

mjg59 — Wed, 04 Oct 2023 19:30:32 +0000

And it wouldn't be unheard of, most of the Windows integration is in a Microsoft fork rather than upstream.

Forking

bluca — Wed, 04 Oct 2023 18:39:03 +0000

They routinely reject non-BSD specific features and enhancements. MPTCP, VRF, default config files in /usr/, AF_VSOCK, and more. It's not for lack of trying, patches do get sent.

Forking

corbet — Wed, 04 Oct 2023 18:34:29 +0000

Might it be worth trying to contribute back to the portable OpenSSH project before forking? (Assuming, of course, that this has not already been tried).

OpenSSH 9.5 released

bluca — Wed, 04 Oct 2023 18:02:02 +0000

OpenSSH is pretty much a BSD project that sort of works on Linux as a side hobby. We really should band together a bunch of mainstream Linux distros and fork it, so that we can modernize it with things like MPTCP, AF_VSOCK and more.