Systemd 254 released

Posted Aug 3, 2023 8:33 UTC (Thu) by rschroev (subscriber, #4164)
In reply to: Systemd 254 released by zdzichu
Parent article: Systemd 254 released

I feel some of these conflict with the documentation. From https://www.freedesktop.org/software/systemd/man/systemd....

"Note that the commands specified in ExecStop= are only executed when the service started successfully first. They are not invoked if the service was never started at all, or in case its start-up failed, for example because any of the commands specified in ExecStart=, ExecStartPre= or ExecStartPost= failed (and weren't prefixed with "-", see above) or timed out."

That directly conflicts with your items 1, 2, and 4, and I feel it also conflicts with 6 and 9. And I don't see where it is documented that ExecStop= commands are called when systemd detects that processes have stopped (I haven't read *all* the documentation though, there's quite a lot of it); I feel generally while the documentation does explain what ExecStop does, it doesn't say enough about if and when ExecStop is triggered.

If the documentation is wrong, incomplete or unclear, I don't see how we're supposed to write correct unit files that work in all cases including edge cases. We shouldn't have to read the code to find out.

Systemd 254 released

Posted Aug 3, 2023 10:07 UTC (Thu) by Wol (subscriber, #4433) [Link]

I believe the systemd documentation itself warns of the consequences of "double daemonisation" or whatever it is in SysV scripts. There are various behaviours common (indeed, almost mandatory) under other init systems that are pathological to systemd. It would not surprise me if this binary assumes SysV (or indeed no init system at all) and behaves in a manner systemd neither likes nor expects.

What then triggers the mass killing I don't know. All I know is (1) it only happens if the systemd unit file contains an ExecStop. And (2) iirc the systemd logs actually point the finger straight at this binary!

At some point I need to fix it, but it's a load of reverse engineering I don't have time for :-(

Cheers,
Wol

Systemd 254 released

Posted Aug 3, 2023 10:35 UTC (Thu) by bluca (subscriber, #118303) [Link] (9 responses)

Writing documentation is hard - what is obvious to me as developer, can be entirely opaque to a given user, and it's difficult to tell when what is what. In this case, to me it's perfectly obvious that ExecStop is ran regardless of _how_ a unit went away, because we don't track commands being sent by admins/programs, we track cgroups. So, to me, "Note that the commands specified in ExecStop= are only executed when the service started successfully first." is clear. But to a user, this might be totally confusing and unexpected.

So, long story short, that manpage is here: https://github.com/systemd/systemd/blob/main/man/systemd.... please send a PR to reword so that it becomes clear to you as a user, and I'll happily review and merge it

Systemd 254 released

Posted Aug 3, 2023 11:11 UTC (Thu) by rschroev (subscriber, #4164) [Link] (8 responses)

I can't reword it because I don't have a good enough grasp on how it works. Every new piece of information seems to contradicts the previous one. And when the documentation doesn't match the code, I can't know which one of them (if any) is correct.

> to me it's perfectly obvious that ExecStop is ran regardless of _how_ a unit went away

But *when*? Is it triggered e.g. at the time you do 'systemctl stop', regardless of what happened to the service in the meantime? Or is triggered at the time systemd notices that the service went away? That's a big difference.

> To me, "Note that the commands specified in ExecStop= are only executed when the service started successfully first." is clear.

It seems clear to me too, but my interpretation is contradicted by the list in zdzichu's comment (https://lwn.net/Articles/940224/), which is correct as far as I can see. According to that, the commands in ExecStop= *are* executed even if the service did *not* start successfully, at the moment systemd detects that.

Systemd 254 released

Posted Aug 3, 2023 12:28 UTC (Thu) by Wol (subscriber, #4433) [Link] (3 responses)

I'm guessing that at least one pathological behaviour here is

(1) systemd fires off a process
(2) this process fires off the daemon and exits
(3) systemd sees the process has terminated, and runs ExecStop

That certainly is the sort of behaviour I assumed was behind the double-daemonisation, and why this fork option was added to the unit file - to prevent exactly this mis-understanding by systemd. I must admit that wasn't obvious from said documentation but it was all in there ...

And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Cheers,
Wol

Systemd 254 released

Posted Aug 3, 2023 13:05 UTC (Thu) by gioele (subscriber, #61675) [Link] (1 responses)

> And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Maybe the service was also launching other services or calling other init scripts?

In that case these newly spawn processes will live inside the cgroup of the service and are going to be killed by systemd once the main service is stopped.

Systemd 254 released

Posted Aug 3, 2023 15:37 UTC (Thu) by Wol (subscriber, #4433) [Link]

> In that case these newly spawn processes will live inside the cgroup of the service and are going to be killed by systemd once the main service is stopped.

Oh the joys of people not reading the thread. The killing spree is OF OTHER SERVICES which have nothing whatsoever to do with the service causing the problem ...

Ie something is seriously wrong somewhere. I just need to debug it.

Cjhers,
Wol

Systemd 254 released

Posted Aug 3, 2023 21:14 UTC (Thu) by malmedal (subscriber, #56172) [Link]

> And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Wild guess. Some init-scripts kill process-groups instead of pids, so if it hit the wrong one...

Systemd 254 released

Posted Aug 3, 2023 16:55 UTC (Thu) by jem (subscriber, #24231) [Link] (3 responses)

The linked man page contains the following text for ExecStop: "Commands to execute to stop the service started via ExecStart". This hints that the purpose of ExecStop is to provide the commands to explicitly stop the service, triggered by some external event like systemctl stop.

Looking at the code, it is called as a direct result of systemctl stop, which calls service_stop. If the service state is SERVICE_RUNNING, the service_stop function unconditionally calls service_enter_stop, which in turn executes the command specified in ExecStop (if any).

I don't see why it would be "totally confusing and unexpected" to a user that the commands in ExecStop are not run if the service fails to start. If the service failed to start, what's the point in trying to stop it? You don't try to close a file that you failed to open, either.

Systemd 254 released

Posted Aug 3, 2023 18:09 UTC (Thu) by rschroev (subscriber, #4164) [Link] (2 responses)

> I don't see why it would be "totally confusing and unexpected" to a user that the commands in ExecStop are not run if the service fails to start. If the service failed to start, what's the point in trying to stop it? You don't try to close a file that you failed to open, either.

But according to the source code, the ExecStop commands *are* run even if the service fails to start. Referring to zdzichu's comment somewhere in this thread (see https://lwn.net/Articles/940224/), line 2264 in service.c (in service_enter_running()) calls service_enter_stop() when a service fails to start. service_enter_stop() in turn executes the ExecStop commands.

I agree with you: I don't expect ExecStop to be triggered if a service fails to start. The documentation agrees, if I interpret it correctly. But unless both zdzichu and I are misreading the code, the code does trigger it in that case.

Systemd 254 released

Posted Aug 3, 2023 18:56 UTC (Thu) by bluca (subscriber, #118303) [Link] (1 responses)

They are not:

$ sudo systemd-run --quiet -t -p ExecStop="echo hello" false
$ sudo systemd-run --quiet -t -p ExecStop="echo hello" true
hello

Systemd 254 released

Posted Aug 4, 2023 7:47 UTC (Fri) by rschroev (subscriber, #4164) [Link]

And neither are the ExecStop= commands executed when ExecStartPost failed:

$ sudo systemd-run --quiet -t -p ExecStop="echo hello" -p ExecStartPost="false" true
-> gives no output

So it seems we both did misread the code. Good, that solves my worries.