A backdoor in xz

Posted Mar 30, 2024 8:19 UTC (Sat) by DimeCadmium (subscriber, #157243)
In reply to: A backdoor in xz by himi
Parent article: A backdoor in xz

Til they tell the service manager "I'm running!" just before failing. I've had that happen several times, in fact. The only *actual* solution for that problem is monitoring. Notify-by-socket is precisely equivalent to notify-by-fork in terms of reliability.

A backdoor in xz

Posted Mar 30, 2024 9:57 UTC (Sat) by cesarb (subscriber, #6266) [Link] (4 responses)

Another solution is the watchdog: the service has to tell the service manager "I'm still running!" periodically, otherwise it's treated as failed. Then the main effect of the initial "I'm running!" is that the timeout before it is given by TimeoutStartSec, and the timeout after it is given by WatchdogSec, which can be shorter (allowing for services which are slow to start, like heavy Java-based servers).

A backdoor in xz

Posted Mar 31, 2024 1:33 UTC (Sun) by DimeCadmium (subscriber, #157243) [Link] (3 responses)

Til it tells the service manager "I'm still running" but never calls accept again. Notify-by-socket is precisely equivalent to notify-by-fork in terms of reliability.

The solution you are looking for is *monitoring*.

A backdoor in xz

Posted Mar 31, 2024 1:45 UTC (Sun) by intelfx (subscriber, #130118) [Link] (2 responses)

> Notify-by-socket is precisely equivalent to notify-by-fork in terms of reliability.

It might be "equivalent" in an information-theoretical sense (everything that can be achieved with one, is also achievable with the other), but it's absolutely not equivalent in _practical reliability_.

Setting up a proper "notifying" double-fork (which, I remind you, means that the immediate child has to wait for the grandchild to initialize and only then exit, because in most cases the initialization must be completed in the grandchild) is tenfold more _complicated_ and _easier to get wrong_ than simply writing a line into a pre-existing socket that the supervisor has prepared for you.

Even more: all known cases of proper notifying double-fork implementatoin involve creating a temporary pipe or socket between the child and the grandchild, precisely for the reasons described above. As such, we are choosing between a notify-by-socket implemented _once_ and a notify-by-socket implemented _over and over again_ in each daemon. The choice must be obvious, unless you specifically have an irrational axe to grind against systemd.

A backdoor in xz

Posted Mar 31, 2024 6:22 UTC (Sun) by DimeCadmium (subscriber, #157243) [Link] (1 responses)

> As such, we are choosing between a notify-by-socket implemented _once_ and a notify-by-socket implemented _over and over again_ in each daemon. The choice must be obvious,

Indeed it must be, considering that we are discussing the result of everyone sharing a single implementation of it.

A backdoor in xz

Posted Mar 31, 2024 12:25 UTC (Sun) by bluca (subscriber, #118303) [Link]

No, most of us are discussing the result of a multi-year-long sophisticated social engineering attack that preyed on underfunded and overworked unpaid maintainers to inject a complex backdoor. Yes, a handful of people are missing the wood for the trees because they are unable or unwilling to run a simple command to check the attack surface gained by backdooring xz:

$ apt-cache rdepends liblzma5 | wc -l
354

If it hadn't been libsystemd in the middle of the dependency chain, it would have been something else. The exploit was primed and ready to add more backdoors for other arbitrary workflows, with pre-prepared and unused "test files" signatures that we'll now never know what would have attacked.

A backdoor in xz

Posted Mar 30, 2024 14:29 UTC (Sat) by smurf (subscriber, #17840) [Link] (2 responses)

One does not exclude the other. In fact many of my monitoring scripts actively check that the service in question hasn't legitimately been shut down before complaining (too) loudly.

You cannot do a "has legitimately been shut down" check without systemd. (Well, OK, of course I could use or write some other code that does this job, but why would I want to replace one mostly-coherent, widely-used presumed-safe software package with five less-widely-used and poorly-integrated ones? sysV init scripts are of no help here)

A backdoor in xz

Posted Mar 31, 2024 1:33 UTC (Sun) by DimeCadmium (subscriber, #157243) [Link] (1 responses)

Yes you can? OpenRC has done it for longer than systemd has existed ffs

A backdoor in xz

Posted Apr 1, 2024 14:06 UTC (Mon) by farnz (subscriber, #17727) [Link]

I used OpenRC before I used systemd; it does not, as far as I can find (and even today) offer a proper lifecycle check; in particular, the only queries you can ask it are "is this service running", "is this service known and shut down", or "has this service crashed", whereas systemd adds "is this service running but in the process of shutting down" and "is this service running but in the process of restarting" to that list, which is essential for automated remediation of faults - you know that if a service is restarting, a fault is OK, while if a service is shutting down, you should determine if that shutdown is expected, or if you need to alert a human.

A backdoor in xz

Posted Mar 30, 2024 14:48 UTC (Sat) by dvdeug (guest, #10998) [Link] (2 responses)

> Til they tell the service manager "I'm running!" just before failing.

Which is a bug; they should complete all checks that make them fail before reporting success. Yes, bugs are a reality.

> The only *actual* solution for that problem is monitoring. Notify-by-socket is precisely equivalent to notify-by-fork in terms of reliability.

No, there exists many cases where a fork happens and then the program fails before it would have notified the service manager it was successfully running. By the same logic, monitoring is precisely equivalent to notify-by-fork in terms of reliability; monitoring programs can fail to notice a service no longer working as well, except that they add false positives and can report that a system has failed when it's been properly shutdown or had a temporary glitch, as from system overload.

A backdoor in xz

Posted Mar 31, 2024 1:34 UTC (Sun) by DimeCadmium (subscriber, #157243) [Link] (1 responses)

> Which is a bug; they should complete all checks that make them fail before reporting success. Yes, bugs are a reality.

Indeed. But *IT IS THE SAME BUG WHETHER YOU'RE USING SYSTEMD'S NOTIFICATIONS OR FORKING*

I don't understand why I have to explain that so many times only to hear the EXACT SAME (inane) ARGUMENT again.

A backdoor in xz

Posted Apr 1, 2024 11:29 UTC (Mon) by HenrikH (subscriber, #31152) [Link]

for that particular deamon yes, but there are less that have this bug than where the double fork is not reliable (which is 100% of the double fork cases).