A backdoor in xz

Posted Mar 29, 2024 20:10 UTC (Fri) by cjwatson (subscriber, #7322)
In reply to: A backdoor in xz by cjwatson
Parent article: A backdoor in xz

The only problem is that it's a pretty horrible amount of code to have to inline. Look at pid_notify_with_fds_internal ...

Apparently unreleased versions of systemd dlopen liblzma instead, which would have meant it wasn't in sshd's process space.

A backdoor in xz

Posted Mar 29, 2024 20:28 UTC (Fri) by intelfx (subscriber, #130118) [Link] (3 responses)

> The only problem is that it's a pretty horrible amount of code to have to inline. Look at pid_notify_with_fds_internal

I don't think any of that code is needed. OpenSSH as patched only needs sd_listen_fds() and plain sd_notify() which _as used_ can be implemented in about 5-10 lines of C code each.

A backdoor in xz

Posted Mar 29, 2024 20:35 UTC (Fri) by cjwatson (subscriber, #7322) [Link] (2 responses)

sd_notify calls pid_notify_with_fds_internal, though? But if there's a reasonably standard inlined C reimplementation that covers all the necessary API surface, I'd definitely consider it.

A backdoor in xz

Posted Mar 30, 2024 1:12 UTC (Sat) by zdzichu (subscriber, #17118) [Link]

That's the details of a specific implementation. Actual protocol is simple. Env var contains a socket path, you write a short string text string to it. Really few lines of code.

A backdoor in xz

Posted Mar 30, 2024 6:50 UTC (Sat) by intelfx (subscriber, #130118) [Link]

> But if there's a reasonably standard inlined C reimplementation that covers all the necessary API surface, I'd definitely consider it.

Yep, that's why I tried to emphasize "as used". The implementation you see is shared between several mostly-disjoint users (e. g. it is also used to communicate with hypervisors via vsock) and also implements other features of this ad-hoc protocol (such as fd passing) which are not used in openssh.

The usage in openssh (to signal readiness) is covered by writing a fixed, static text string into an AF_UNIX datagram socket pointed to by the $NOTIFY_SOCKET variable.

A backdoor in xz

Posted Mar 29, 2024 21:05 UTC (Fri) by judas_iscariote (guest, #47386) [Link] (13 responses)

It will still be.. selinux requires it. happy now ? Supply chain attacks are not systemd's fault .. :-)
It is more like corporations fault for not paying people to work in things they profit from.

A backdoor in xz

Posted Mar 30, 2024 11:04 UTC (Sat) by fenncruz (subscriber, #81417) [Link] (12 responses)

I agree it's not systemd's fault, but is there something it (and other software) can do to make this attack harder? Like somehow preventing the symbols being replaced by a malicious library?

A backdoor in xz

Posted Mar 30, 2024 12:02 UTC (Sat) by bluca (subscriber, #118303) [Link] (7 responses)

We have already replaced all linked dependencies (apart from glibc and libcap) in libsystemd.so with dlopen (that is activated only if and when the specific API that needs the external library is called, not automatically) in git main

A backdoor in xz

Posted Mar 30, 2024 14:12 UTC (Sat) by smurf (subscriber, #17840) [Link]

Interesting. I was about to thank you for your proactive response to this incident, but a look at systemd's git reveals that this change was done a month ago, in order to reduce systemd's footprint on startup RAM disks. ;-)

A backdoor in xz

Posted Mar 30, 2024 15:27 UTC (Sat) by dskoll (subscriber, #1630) [Link] (1 responses)

I understand the advantages of the dlopen approach, but it still leaves me feeling uneasy. You might get shared libraries that you don't expect dlopened just by making an innocent API call.

It seems to me that the supervisor notification protocol is likely to be used by many programs, and also quite likely that they might not want anything else from libsystemd. Wouldn't it make sense to put the notification client code in its own shared library that has no external dependencies and won't dlopen anything else ever?

A backdoor in xz

Posted Mar 30, 2024 15:52 UTC (Sat) by zdzichu (subscriber, #17118) [Link]

Funny, it was this way until v209 in 2014. sd-daemon was a collection of functions like sd_notify() and so on, it got merged into libsystemd then.

A backdoor in xz

Posted Mar 30, 2024 18:36 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Sorry, but random dlopen()s are even MORE unacceptable. It also prevents very useful security measures like locking the text of the running executable.

A backdoor in xz

Posted Mar 30, 2024 19:14 UTC (Sat) by andresfreund (subscriber, #69562) [Link] (2 responses)

It doesn't prevent that at all? Unless you use text relocations, .text should only be mapped read only. And .got would have been remapped ro at start if you use -z now -z relro. Dlopen() doesn't change any of that?

A backdoor in xz

Posted Mar 30, 2024 19:41 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

I mean, locking down the complete set of executable pages in a process, so that no new code can't get loaded. OpenBSD has mseal() that can do that.

> Dlopen() doesn't change any of that?

Indeed it doesn't (right now), but expanding its usage will make it harder to enable something like mseal() later.

A backdoor in xz

Posted Mar 31, 2024 13:13 UTC (Sun) by bluca (subscriber, #118303) [Link]

You can still do that, but then you lose some features. That seems like a perfectly acceptable trade-off to me.

A backdoor in xz

Posted Mar 30, 2024 16:53 UTC (Sat) by judas_iscariote (guest, #47386) [Link] (3 responses)

Yes, you can prevent symbols from been replaced by something else by various compiler, linker flags and possibly enviroment variables.. it will still be a cat & mouse game because something that has root already has all the power.

A backdoor in xz

Posted Mar 30, 2024 19:05 UTC (Sat) by andresfreund (subscriber, #69562) [Link] (1 responses)

Afaict all the options for doing so were used in this case. The redirection happened just before the got was remapped read only.

I'm somewhat surprised that nobody called for glibc's rtld-audit infrastructure to be removed. That's really what made this attack possible despite relro. As far as I know, it's not used widely.

A backdoor in xz

Posted Mar 31, 2024 13:30 UTC (Sun) by nix (subscriber, #2304) [Link]

Perhaps it should be possible to set some sort of link-time tag to instruct ld.so to disable the LD_AUDIT infrastructure for particular binaries? Not sure that's doable for specific shared libraries, but at least this would let one mark critical system daemons as "hands-off" for this application, so their own libraries can't compromise them like this. It's enough like AT_SECURE or coredump/ptrace prevention that there should probably be one mechanism to turn all this stuff on at the same time... (For userspace stuff latrace and the things that it enables are actually quite useful, but I can't imagine ever running latrace on sshd, and if I did I'm debugging it anyway and would be at the very least foregrounding it and could presumably manually turn auditing back on. For that matter, latrace could be modified to do that to the programs it invokes, since it knows its own use of LD_AUDIT is non-malicious.)

A backdoor in xz

Posted Mar 31, 2024 6:37 UTC (Sun) by epa (subscriber, #39769) [Link]

I think if the symbol-replacing were not allowed, nor arbitrary code execution on *loading* the library, then the attack would be more difficult. The application does not call any functions from xz. An attacker would have to get a backdoor into the library and somehow persuade sshd to call it.