LWN: Comments on "Serious vulnerability fixed with OpenSSH 9.8"

Authenticated port knocking

mb — Sat, 20 Jul 2024 06:51:08 +0000

Yes, it receives the systemd socket and then notifies readiness to systemd. But only if it has been started by systemd.

Authenticated port knocking

mrugiero — Sat, 20 Jul 2024 03:51:37 +0000

> Sure. I'll always be happy to include examples in the documentation. I have no idea what an s6 system is. though :)

It's an alternative init system, supervision-tree based, but with readiness notification and some parts of socket activation. I have a hobby server where I used it for fun and that's the one I'd like to use letmein with.

> Also code changes are ok, as long as they are not too intrusive for the rest of the use cases.

Cool. I don't think they will be needed (and s6 is rather niche, so I wouldn't want to burden you with maintaining code for it), but it's nice to see you're open minded about it if it turns out I'm wrong.

> letmein can run without systemd. If it doesn't see the systemd environment variables, it will create the listening socket on its own. Also see the --no-systemd option. It doesn't create a pidfile or fork into the background like traditional daemons do, though.

No forking needed :)
I'll see if the features it adds are compatible with s6, if not then I'll probably use it without. Is it only to receive the listening socket? I was thinking of notifying readiness as well.

> Looking forward to your contribution. Feel free to create and issue and/or PR.

Thanks! It'll probably be some time till I have free time for that. I will try this weekend but can make no promises.

Authenticated port knocking

mb — Fri, 19 Jul 2024 05:59:38 +0000

Sure. I'll always be happy to include examples in the documentation. I have no idea what an s6 system is. though :)
Also code changes are ok, as long as they are not too intrusive for the rest of the use cases.

letmein can run without systemd. If it doesn't see the systemd environment variables, it will create the listening socket on its own. Also see the --no-systemd option. It doesn't create a pidfile or fork into the background like traditional daemons do, though.

Looking forward to your contribution. Feel free to create and issue and/or PR.

Authenticated port knocking

mrugiero — Thu, 18 Jul 2024 23:48:04 +0000

Nice! Very interesting project. I'm interested in trying it in an s6 system. I think it can make to work with the systemd code (that is, without touching the actual Rust code) to handle notification and fd passing with a little wrapping in execline. Would you be interested in some example files as docs when I do that?

Authenticated port knocking

mb — Sat, 13 Jul 2024 07:15:31 +0000

Ah. Got it. :)
I am the original author and by now I have already stripped the things that I wanted to strip.
I don't plan to remove anything else. See the open issues oh Github for things that I still plan to do.

Authenticated port knocking

mrugiero — Fri, 12 Jul 2024 23:30:25 +0000

I assumed you'd work on a fork for the stripping down. Are you the original author? Or was your idea to do it for yourself and keep it local?

Authenticated port knocking

mb — Fri, 12 Jul 2024 19:13:32 +0000

Uhm, what fork?

Authenticated port knocking

mrugiero — Fri, 12 Jul 2024 18:48:31 +0000

I'd be interested in that fork when it's there.

Automatic async-signal-unsafe checking?

mrugiero — Fri, 12 Jul 2024 18:35:41 +0000

I feel something similar to how OpenBSD only allows syscalls to be called by the libc by using the range of the memory mapping (IIRC that was the trick), maybe signal handlers could be in a specific section of the binary and restrict what it can call based on that. Calling code outside the authorized sections would cause the OS to kill the process.

Automatic async-signal-unsafe checking?

pizza — Wed, 10 Jul 2024 11:14:54 +0000

SDL is a platform abstraction layer. If your application is directly mucking with signals (and/or threads), then it's playing outside of (and potentially interfering with) SDL's sandbox. Given that the reason for using SDL is easy portability, this is counter-productive.

SDL hooks signals so it can clean up after itself properly. That can be disabled at init time.

It also generates signals and expects the application to supply a callback to handle them in a realtime manner -- This is notably used in SDL's audio subsystem, and is also one of the approaches supported by ALSA.

It also uses signals in its internal threading implementation.

All of this was true in the SDL1 days; I don't know if SDL2 or beyond does things differently.

But as I mentioned above, ALSA can use signals as one of three mechanisms for informing applications to refill audio buffers. (the other two being blocking I/O and non-blocking select/poll-based I/O). Which one you use depends on how your application is structured.

Automatic async-signal-unsafe checking?

paulj — Tue, 09 Jul 2024 17:07:52 +0000

By handler I meant "A function to handle the signal, that the user calls from normal context" - not a handler to install directly as the signal handler. I.e., the user handles the signal itself, then later calls the library-specific handlers, from the normal user-code context.

That's really the only sane API for libraries that need to know about signals. I'm trying to understand why that wouldn't work for a library, as pizza suggests.

Automatic async-signal-unsafe checking?

sionescu — Tue, 09 Jul 2024 16:35:59 +0000

That's a completely different thing from a library defining a handler, thus potentially overwriting a handler defined elsewhere. Since signals are a global property of a process, you can't have libraries define their own signal handlers because that's not composable. It's perfectly reasonable to expose a function that the app should call in case of signal.

Automatic async-signal-unsafe checking?

paulj — Tue, 09 Jul 2024 13:01:11 +0000

Why?

In particular, why couldn't it use the "generally the sanest approach for libraries" thing of defining handlers to be called (from non-sigcontext generally) by the user application if a signal was caught?

WireGuard as defense-in-depth

raven667 — Mon, 08 Jul 2024 04:16:10 +0000

I don't know anything technical about Wireguard, but that's a neat property. You might still be able to scan for it though, based on the _lack_ of an ICMP port unreachable response, unless you go out of your way to block ICMP, but doing so can have other unintended consequences so more modern firewalling guides don't recommend it the way they used to.

unprivileged sshd

donald.buczek — Fri, 05 Jul 2024 04:46:28 +0000

> In theory it should be possible to run a SSH daemon without root by having systemd create the session and having it pass back ttys similar to how machinectl shell/run0 works, but I highly doubt the OpenSSH devs would implement something that uses systemd.

Yes, this already works and sshd doesn't need to be aware. For example, we abuse sshd to offer interactive sessions via our cluster scheduler. sshd is running unprivileged, the network socket is already connected externally (so we run it in inetd mode) and keys are created per session and communicated out of band [^1].

[^1] https://github.molgen.mpg.de/mariux64/mxtools/blob/master...

WireGuard as defense-in-depth

aaronmdjones — Fri, 05 Jul 2024 01:30:17 +0000

There cannot be such a WireGuard scanner as you put it. WireGuard does not respond at all, to any unsolicited traffic, unless that traffic resembles a handshake containing its public key. If you don't know a WireGuard device's public key, you can't get it to send anything to you, ever. This is because the public keys are Curve25519 key agreement keys, not Ed25519 signature keys (WireGuard doesn't use any asymmetric signature crypto), so if you don't know its public key, you can't even agree a session negotiation key with it.

WireGuard as defense-in-depth

mussell — Thu, 04 Jul 2024 21:18:05 +0000

Wireguard can run as a userspace daemon which uses TUN/TAP and does not need to run as root. By comparison, OpenSSH needs to run as root with all capabilities, no seccomp filter, and YesNewPrivs due a combination of how awful Unix authentication is and trying to do privilege separation on its own. In theory it should be possible to run a SSH daemon without root by having systemd create the session and having it pass back ttys similar to how machinectl shell/run0 works, but I highly doubt the OpenSSH devs would implement something that uses systemd.

Automatic async-signal-unsafe checking?

sionescu — Thu, 04 Jul 2024 17:59:12 +0000

SDL could be one exception, but also "Dear lord!".

Automatic async-signal-unsafe checking?

pizza — Thu, 04 Jul 2024 16:32:39 +0000

> Libraries shouldn't ever handle signals.

SDL wouldn't function without use of signals.

(for extra fun, it triggers them as well as consuming them...)

Automatic async-signal-unsafe checking?

sionescu — Thu, 04 Jul 2024 16:07:56 +0000

Libraries shouldn't ever handle signals.

Automatic async-signal-unsafe checking?

Cyberax — Thu, 04 Jul 2024 05:33:43 +0000

> The approach of setting a flag doesn't work here, because you're in code that you don't control in a PAM module which cannot check your flag; you have to call _exit().

These days, I would move the timeout to the parent process entirely, and instead I will send heartbeat messages to it.

Automatic async-signal-unsafe checking?

geofft — Thu, 04 Jul 2024 01:52:58 +0000

You are continuing to misunderstand or mischaracterize my proposal. I am not making the proposal of restricting your signal handlers to the entire set of async-signal-safe functions, so I don't know why you are commenting on how large that set is - it is not relevant to my proposal.

My proposal is to permit a narrow and well-defined set of operations, much smaller than the set of async-signal-safe functions. I am not expressing the opinion that all other functions are async-signal-unsafe. There are many other async-signal-safe functions; you should, nonetheless, not call them.

Contrary to what you say, OpenSSH has not tried my proposal. They have tried the proposal you are attacking - of restricting signal handlers to async-signal-safe calls, and paying attention to how that set changes on various OSes. That is the proposal that (as you point out) has failed in practice. We should not adopt it. I am not calling for people to adopt it.

For context, here is the patch in TFA that OpenSSH suggests to fix the vulnerability in existing releases:

diff --git a/log.c b/log.c
index 9fc1a2e2e..191ff4a5a 100644
--- a/log.c
+++ b/log.c
@@ -451,12 +451,14 @@ void
 sshsigdie(const char *file, const char *func, int line, int showfunc,
     LogLevel level, const char *suffix, const char *fmt, ...)
 {
+#ifdef SYSLOG_R_SAFE_IN_SIGHAND
 	va_list args;
 
 	va_start(args, fmt);
 	sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
 	    suffix, fmt, args);
 	va_end(args);
+#endif
 	_exit(1);
 }

It's very clear to see that they're taking the approach of calling or not calling functions depending on whether they are async-signal-safe, and this is exactly what I am claiming one should not do. Under my proposal, there should not be any function calls to other user-written functions in a signal handler - the very existence of this sshsigdie() function is a design flaw, because it is not immediately obvious that it is called from a signal handler and subject to async-signal-safety restrictions. (In fact, it's called indirectly from a macro sigdie(), further obscuring the link between the signal handler and this code.) The idea of doing conditional compilation based on the current OS's async signal safety guarantees is an indication that you are already well down an incorrect path.

My proposal is to change the code in OpenSSH such that the SIGALRM handler is not responsible for ending the child process, it is just responsible for notifying whatever code was being called that the child process should shut down. If a read() from the remote end returns EINTR and a flag is set, that's a sign to call the regular code to exit the process. The handler should not be doing it.

My proposal is that, if you want to use alarm()/SIGALRM for a timeout, your alarm handling should be in the course of how you process all other events - much as it would be if you were using timerfd for a timeout on an OS that has it. If you want to syslog, pass a message to your main program to syslog. The fact that things like timerfd exist is, I would argue, evidence that this is in fact the consensus view. So is the fact that we have had numerous new types of kernel features in the last few decades exposed via pollable file descriptors and none via new signals (not counting the real-time signals, which are a rather different sort of thing from signals like SIGALRM and SIGWINCH).

In fact, it looks like the patch that OpenSSH applied in 9.8 to fix the bug was to do exactly this sort of change - change the signal handler to exit without logging anything, and move responsibility for logging into the parent sshd process.

And this patch points out what's actually going on in the vulnerability: it's a timeout that needs to be able to interrupt a blocking PAM call. The approach of setting a flag doesn't work here, because you're in code that you don't control in a PAM module which cannot check your flag; you have to call _exit(). But by the same token, your proposal of blocking signals outside of pselect() etc. wouldn't work here either: if you only unblocked SIGALRM while you're inside a pselect() call of your own, you would never interrupt the stuck PAM module.

Automatic async-signal-unsafe checking?

NYKevin — Wed, 03 Jul 2024 22:48:42 +0000

Operating systems will happily let you carry out all manner of bad ideas. The fact that something is supported doesn't magically make it into a good idea.

WireGuard as defense-in-depth

Wol — Wed, 03 Jul 2024 19:05:36 +0000

Maybe I'm dense, and we'll need flussence to explain, but if wireguard is running on the firewall (which presumably has no legitimate reason to initiate connections to internal machines), don't you need some other exploit - for example sshd - to compromise an internal machine?

(Yes, once you're in the firewall, compromising other machines is easier ...)

I'm assuming wireguard and sshd are NOT on the same machine ...

Cheers,
Wol

WireGuard as defense-in-depth

mb — Wed, 03 Jul 2024 16:41:47 +0000

>If your argument is that the pre-auth attack surface on Wireguard is smaller than the pre-auth attack
>surface on sshd then that's a reasonable argument to make given sufficient evidence, but otherwise
>there isn't an additional wall here.

Yes. That's exactly why I use an authenticating knocker instead of a VPN. It's only about reducing the external pre-auth attack surface. The knocker is not as small as I'd like it to be, but there's still room for improvement.

WireGuard and/or OpenSSH

gnoutchd — Wed, 03 Jul 2024 16:19:15 +0000

Ah, fair point. If we run WireGuard on the same host as the sshd, and if we assume WireGuard and OpenSSH are equally likely to have remote-root bugs, then I'd agree that it wouldn't buy you much.

However, WireGuard and sshd need not run on the same host. What if you put WireGuard on a gateway device?

Also, I actually would expect WireGuard to be less prone to security bugs than OpenSSH, for two reasons:

WireGuard's protocol and implementation are deliberately very simple, with exactly one cryptosystem and authentication scheme, whereas sshd must carry ~15-20 years of negotiable ciphersuites and authentication schemes.
sshd has to interact with stuff like syslog and pam and nss and systemd, and this has caused trouble several times (e.g. xz-utils, this CVE, CVE-2006-5051). WireGuard's integrations appear to be much simpler.

I've not looked closely at the code of either, though, so I could be wrong.

WireGuard as defense-in-depth

mjg59 — Wed, 03 Jul 2024 15:18:19 +0000

If I've already compromised wireguard, why do I need to compromise sshd? I've presumably already got RCE.

WireGuard as defense-in-depth

Trelane — Wed, 03 Jul 2024 15:10:33 +0000

Exactly. Thank you.

Automatic async-signal-unsafe checking?

rweikusat2 — Wed, 03 Jul 2024 14:18:23 +0000

Parent article: Serious vulnerability fixed with OpenSSH 9.8

>> What precisely is or isn't "nontrivial work" is in the eye of the beholder.
> The comment you are replying to just explained exactly what "nontrivial work" is. It's anything other than setting a > flag, writing one byte into a self-pipe (or the like), or calling _exit(2). These are not examples. They are an
> exhaustive list. Literally any other code under the sun is "nontrivial."

The comment reflects the opinion of the author what's considered "non-trivial". The set of async signal safe function is significantly larger than just write and _exit which very strongly suggests that other people do not share this opinion. For another example, OpenBSD (and also, some Linux distributions) have an asycnc-signal-safe syslog function specifically to enable it to be used safely from asynchronous signal handlers.

WireGuard as defense-in-depth

dskoll — Wed, 03 Jul 2024 12:19:01 +0000

It's true that exploiting Wireguard would be just as good as exploiting sshd, but I suspect there are a lot more SSH scanners prowling the Internet looking for vulnerable servers than there are Wireguard scanners. So in a way, there's a bit of security through obscurity.

WireGuard as defense-in-depth

Wol — Wed, 03 Jul 2024 09:10:19 +0000

Ah.

As I understood it, the target was the *local* machine, which could only be reached by ssh from the remote machine. So if the remote machine was protected by wireguard, then you would need to compromise each in turn.

Cheers,
Wol

WireGuard as defense-in-depth

mjg59 — Wed, 03 Jul 2024 07:34:40 +0000

Wireguard is running on the remote server, so you only need to find a hole in Wireguard to gain access to the remote server. If your argument is that the pre-auth attack surface on Wireguard is smaller than the pre-auth attack surface on sshd then that's a reasonable argument to make given sufficient evidence, but otherwise there isn't an additional wall here. If Wireguard has an exploitable vulnerability, you simply exploit that instead of caring about sshd at all.

WireGuard as defense-in-depth

Wol — Wed, 03 Jul 2024 07:17:07 +0000

> But now you just changed where the hole has to be--wireguard instead of sshd. I don't see what you gain.

Because you misunderstand the problem. He hasn't *changed* where the hole is, he's added a whole new outer wall round the castle. So now you have to find *two* holes, not one.

Before you can exploit sshd (the old situation), you now have to exploit wireguard first - the new outer bailey.

Cheers,
Wol

WireGuard as defense-in-depth

Trelane — Wed, 03 Jul 2024 05:32:10 +0000

But now you just changed where the hole has to be--wireguard instead of sshd. I don't see what you gain.

Automatic async-signal-unsafe checking?

NYKevin — Tue, 02 Jul 2024 21:49:39 +0000

This is what Python does as well.

Annoyingly, it has to acquire the GIL before it can execute the signal handler, and even more annoyingly, it tries to handle SIGINT by default, so (in extreme cases) Python can hang for several seconds between pressing ^C and the process actually terminating, even if the programmer made no attempt to handle any signals. Fortunately, it does not try to handle SIGQUIT, so you can just press ^\ instead of ^C.

Automatic async-signal-unsafe checking?

NYKevin — Tue, 02 Jul 2024 21:40:38 +0000

> What precisely is or isn't "nontrivial work" is in the eye of the beholder.

The comment you are replying to just explained exactly what "nontrivial work" is. It's anything other than setting a flag, writing one byte into a self-pipe (or the like), or calling _exit(2). These are not examples. They are an exhaustive list. Literally any other code under the sun is "nontrivial."

> System call variants exists specifically for this use case, namely the p-variants of select, poll etc. This obviously also requires that all handled signals are blocked while signal handlers themselves are executing as they will have been unblocked by the I/O multiplexing call. An advantage of this is that it doesn't need userspace code for dispatching signals.

This works great, right up until one of your libraries tries to use alarm(2) (or sleep(3), usleep(3), setitimer(2), etc.) to do something interesting, and finds that signals are blocked and it does not work. And yes, that probably does go against the "no nontrivial work" rule in most practical cases, but in userspace, the sad reality is that you do not own all of the code in your process, and sometimes have to live with libraries doing ill-advised things.

(Never mind the even crazier possibility that one of your libraries will decide it is a good idea to manipulate the signal mask!)

Automatic async-signal-unsafe checking?

dskoll — Tue, 02 Jul 2024 20:45:17 +0000

Not sure about Python, but I believe Perl just remembers that a signal has occurred and then calls the appropriate $SIG{'HANDLER'} at a safe time within the evaluation loop. When you control the entire environment, it's easy to impose a policy like this.

Signals are one of the misfeatures of UNIX.

Kinds of fault handlers

riking — Tue, 02 Jul 2024 20:41:45 +0000

The first action of every useful SIGSEGV handler (that might successfully return to retry the instruction) I've seen is to perform a range check on the faulting memory address, the faulting program counter, or both. Either we're touching a special memory region where I know how to map in the memory, or we're running a special code region that wants me to signal failure and jump the PC to the failure branch.

Automatic async-signal-unsafe checking?

Sesse — Tue, 02 Jul 2024 18:08:29 +0000

Valgrind used to have a separate tool for this. I'm not sure if it's still available/maintained, though?

Automatic async-signal-unsafe checking?

rweikusat2 — Tue, 02 Jul 2024 17:35:45 +0000

I fail to see any difference between "just don't call functions which aren't async signal sage" and your elaboration about just that, because that's what it boils down to. What precisely is or isn't "nontrivial work" is in the eye of the beholder. OpenSSH has had two vulnerabilities because of this approach in the last 25 years. And OpenSSH is a high-profile security-related program developed by experts in this area.

An alternate option is to defer handling of asynchronous signals until a time when the process can safely be interrupted. For a program structured around a synchronous I/O multiplexing loop, this will be when the process has again called into the kernel to be notified when more I/O is possible. System call variants exists specifically for this use case, namely the p-variants of select, poll etc. This obviously also requires that all handled signals are blocked while signal handlers themselves are executing as they will have been unblocked by the I/O multiplexing call. An advantage of this is that it doesn't need userspace code for dispatching signals.

Yet another option is to block all signals an application wants to handle and receive signals as input on a signalfd file descriptor. This needs some form of userspace signal dispatching but will work in environments where sigaction isn't easily usable (eg, perl).