LWN: Comments on "Systemd 254 released"

Systemd 254 released, exponential growing restart for intervals for Restart=

zdzichu — Tue, 16 Apr 2024 14:32:28 +0000

I written about this feature some time ago, here's the link to my blog:

https://enotty.pipebreaker.pl/posts/2024/01/how-systemd-e...

Two examples and a link to the formula in the source code.

Systemd 254 released, exponential growing restart for intervals for Restart=

jdyrhovden — Tue, 16 Apr 2024 13:36:21 +0000

Based on the recent update, I'm curious about the methodology behind calculating the exponential increase in restart intervals. How is the growth factor determined? Additionally, I would appreciate an illustrative example demonstrating the calculation process for each step.

* The service activation logic gained new settings RestartSteps= and
RestartMaxDelaySec= which allow exponentially-growing restart
intervals for Restart=.

Systemd 254 released

tao — Sun, 06 Aug 2023 17:21:03 +0000

Surely if supporting systemd-sysv-generator requires almost no maintenance, it'd be a great target for newcomers who want a good way to get their feet wet and to get major kudos from the apparently enormous community of people who want to use initscripts instead of systemd units.

Systemd 254 released

anselm — Sun, 06 Aug 2023 12:21:32 +0000

That's why it's a dick move.

I disagree. At some point the training wheels need to come off the bicycle.

Properly-written systemd unit files make it possible for system services to avail themselves of all sorts of convenient systemd features which improve stability, security, and general service handling. Most of this remains unused by services which just keep using SysV init files with autogenerated service unit definitions. Programmers who, in 2023, insist on providing only SysV init files do their users an obvious disservice, and there is no reason anymore for systemd to be complicit in this.

Note that there is nothing at all preventing programmers from providing both a systemd service unit and a SysV init file for their system services. Nobody talks about forcing anyone to give up SysV init support completely. It's just that nowadays most people can do a lot better with proper systemd support rather than legacy SysV init files – and even if software authors are too lazy and irresponsible to acknowledge this, they can still provide a minimal 10-line generator-style service unit that just calls the SysV init script, big deal.

Systemd 254 released

bluca — Sun, 06 Aug 2023 11:12:42 +0000

In case you never noticed, Linux (as in the kernel) breaks compatibility all the time. Every new release something breaks and need changes somewhere, whether it's a netlink message, or a uevent, or an LSM, etc etc. The answer is always 'deal with it'.

Systemd 254 released

Cyberax — Sun, 06 Aug 2023 03:34:53 +0000

> Systemd has been around as the de-facto standard for mainstream Linux distributions for more than a decade now.

Yup. Just like Linux should drop anything that is older than a decade. Those developers having chutzpah not to move all their API use to io_uring in 2023!

Yeah, I understand that it can be at least a couple of orders of magnitude more complicated task. But still, the thing that users like is not breaking their stuff. And this particular change will absolutely break some users' workflows.

Also consider this, supporting systemd-sysv-generator requires almost no maintenance. Yet now many distros will probably have to scramble and keep patches on top of systemd to support this. And it will absolutely require an order of magnitude more aggregate man-hours to do that.

That's why it's a dick move.

Systemd 254 released

anselm — Fri, 04 Aug 2023 16:49:03 +0000

I still have fairly new software (less than 3 years old!) that only has SysV scripts. Yeah, I can easily writer a wrapper script, I know. However, having a tool to do it is nice.

You have that tool now. It's called systemd (or systemd-sysv-generator if you want to be picky; it even comes with its own man page). There's nothing wrong with copying its output against a rainy day.

So yep, removing it is a dick move.

I'd consider it a friendly suggestion to some people to get off their lazy fat behinds.

Systemd has been around as the de-facto standard for mainstream Linux distributions for more than a decade now. If in 2023 there are people who have the chutzpah to distribute commercial software that comes only with a SysV init script, they deserve a stern talking-to. (And if the software in question is FLOSS, here's your chance to do something nice for the community.)

Systemd 254 released

jwarnica — Fri, 04 Aug 2023 14:32:54 +0000

At this point, even if you don't know why the *n* versions of the basic c library are better, you just do it.

At some point the lazyness of simply writing a 5 line ini has to overwhelm any ideological attachments to 1000 line shell scripts.

Just do it.

Systemd 254 released

rschroev — Fri, 04 Aug 2023 07:47:35 +0000

And neither are the ExecStop= commands executed when ExecStartPost failed:

$ sudo systemd-run --quiet -t -p ExecStop="echo hello" -p ExecStartPost="false" true
-> gives no output

So it seems we both did misread the code. Good, that solves my worries.

Systemd 254 released

malmedal — Thu, 03 Aug 2023 21:14:27 +0000

> And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Wild guess. Some init-scripts kill process-groups instead of pids, so if it hit the wrong one...

Systemd 254 released

bluca — Thu, 03 Aug 2023 18:56:34 +0000

They are not:

$ sudo systemd-run --quiet -t -p ExecStop="echo hello" false
$ sudo systemd-run --quiet -t -p ExecStop="echo hello" true
hello

Systemd 254 released

rschroev — Thu, 03 Aug 2023 18:09:26 +0000

> I don't see why it would be "totally confusing and unexpected" to a user that the commands in ExecStop are not run if the service fails to start. If the service failed to start, what's the point in trying to stop it? You don't try to close a file that you failed to open, either.

But according to the source code, the ExecStop commands *are* run even if the service fails to start. Referring to zdzichu's comment somewhere in this thread (see https://lwn.net/Articles/940224/), line 2264 in service.c (in service_enter_running()) calls service_enter_stop() when a service fails to start. service_enter_stop() in turn executes the ExecStop commands.

I agree with you: I don't expect ExecStop to be triggered if a service fails to start. The documentation agrees, if I interpret it correctly. But unless both zdzichu and I are misreading the code, the code does trigger it in that case.

Systemd 254 released

anselm — Thu, 03 Aug 2023 17:21:08 +0000

I'm sorry YOU don't want initscripts on YOUR laptop, bluca. Some others of us would prefer that that compatibility layer stay. Thanks.

Nobody will prevent your init scripts from working. All that changes is that you will have to come up with your own systemd service units that call them. If your service is running now you can copy a simple one from /run/systemd/generator.late for later.

This is independent from the fact that on a systemd-based machine, when you're using init scripts – especially with a very simple (auto-generated) service unit – you're deliberately forgoing a bunch of cool and useful features which systemd provides out of the box, which could make your service easier to control, more stable, and more secure, and which SysV init files generally do not bother with. Of course this is your privilege, but it may be worth considering whether making a dedicated systemd service unit for your service which works without the init script and takes more direct advantage of systemd's features might not be a good idea after all.

Systemd 254 released

jem — Thu, 03 Aug 2023 16:55:51 +0000

The linked man page contains the following text for ExecStop: "Commands to execute to stop the service started via ExecStart". This hints that the purpose of ExecStop is to provide the commands to explicitly stop the service, triggered by some external event like systemctl stop.

But *when*? Is it triggered e.g. at the time you do 'systemctl stop', regardless of what happened to the service in the meantime? Or is triggered at the time systemd notices that the service went away? That's a big difference.

Looking at the code, it is called as a direct result of systemctl stop, which calls service_stop. If the service state is SERVICE_RUNNING, the service_stop function unconditionally calls service_enter_stop, which in turn executes the command specified in ExecStop (if any).

I don't see why it would be "totally confusing and unexpected" to a user that the commands in ExecStop are not run if the service fails to start. If the service failed to start, what's the point in trying to stop it? You don't try to close a file that you failed to open, either.

Systemd 254 released

Wol — Thu, 03 Aug 2023 15:37:47 +0000

> In that case these newly spawn processes will live inside the cgroup of the service and are going to be killed by systemd once the main service is stopped.

Oh the joys of people not reading the thread. The killing spree is OF OTHER SERVICES which have nothing whatsoever to do with the service causing the problem ...

Ie something is seriously wrong somewhere. I just need to debug it.

Cjhers,
Wol

Systemd 254 released

jccleaver — Thu, 03 Aug 2023 14:35:37 +0000

>Unit files are stable. Old and crufty init scripts were known for many things, but being stable is definitely not one of them. You will still be able to use them of course, if you like them - not on systemd though.

Thank you for so clearly demonstrating my original point in this thread.

What you're calling "old and crufty" others call stability. Some of us (many of us) prefer imperative to declarative during important phases of operation because it makes it easier for a human to debug. Even leaving aside the fact that init scripts have been updated less and less due to systemd-migration, init scripts haven't changed much because they *just worked*... This is especially the case on the Red Hat/Fedora side of the ecosystem, instead of the Debian landscape where things actually were a bit more painful. But init script conventions in Fedora-world had been locked down pretty well for a long while.

Regardless, your "you can use them, but since I don't, you're not going to use them on systemd" attitude is *PRECISELY* what earns you such vitriol from admins downstream of your decisions.

I'm sorry YOU don't want initscripts on YOUR laptop, bluca. Some others of us would prefer that that compatibility layer stay. Thanks.

Systemd 254 released

gioele — Thu, 03 Aug 2023 13:05:09 +0000

> And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Maybe the service was also launching other services or calling other init scripts?

In that case these newly spawn processes will live inside the cgroup of the service and are going to be killed by systemd once the main service is stopped.

Systemd 254 released

Wol — Thu, 03 Aug 2023 12:28:16 +0000

I'm guessing that at least one pathological behaviour here is

(1) systemd fires off a process
(2) this process fires off the daemon and exits
(3) systemd sees the process has terminated, and runs ExecStop

That certainly is the sort of behaviour I assumed was behind the double-daemonisation, and why this fork option was added to the unit file - to prevent exactly this mis-understanding by systemd. I must admit that wasn't obvious from said documentation but it was all in there ...

And that's what's probably behind ExecStop being executed in my case (still doesn't explain the killing spree ...)

Cheers,
Wol

Systemd 254 released

anselm — Thu, 03 Aug 2023 11:17:46 +0000

If you don't know what that script is doing, what the daemon it's starting is doing, not it's NOT that trivial.

And that should help convince anyone that allowing arbitrary shell code for daemon startup is, with hindsight, not the greatest of ideas. At least with systemd you know where you stand.

And you don't stand a hope in hell of knocking them out that quick.

For most if not all SysV init scripts it's not a huge problem to come up with simple service units that call them (nobody said you had to get rid of the init script altogether, after all). Systemd even does that automatically, for now anyway. That way you're not taking advantage of many of the helpful and convenient things systemd can do, but it's a start. If you're only interested in keeping the service working as before once the automatic support for SysV init scripts is removed from systemd, it may be all you need to do. You could even take a peek in /run/systemd/generator.late to see what you can find there.

It's when you want to replace the init script completely with a service unit that you need to look at the init script to see what exactly it does, and that of course takes time (especially with some of the more gnarly init scripts out there). I don't think anyone has seriously doubted that.

Systemd 254 released

rschroev — Thu, 03 Aug 2023 11:11:55 +0000

I can't reword it because I don't have a good enough grasp on how it works. Every new piece of information seems to contradicts the previous one. And when the documentation doesn't match the code, I can't know which one of them (if any) is correct.

> to me it's perfectly obvious that ExecStop is ran regardless of _how_ a unit went away

> To me, "Note that the commands specified in ExecStop= are only executed when the service started successfully first." is clear.

It seems clear to me too, but my interpretation is contradicted by the list in zdzichu's comment (https://lwn.net/Articles/940224/), which is correct as far as I can see. According to that, the commands in ExecStop= *are* executed even if the service did *not* start successfully, at the moment systemd detects that.

Systemd 254 released

bluca — Thu, 03 Aug 2023 10:57:24 +0000

There's 2 cases here: either you have nothing at all, and then this change to the generator doesn't affect you in any way, or you have an init script used through the deprecated generator. In the latter case, it is as simple as taking the generated unit and initially just shipping that as-is. That _is_ trivial, it's just a copy and a git add!
Why is it better? Because you can then iteratively improve on that, pick up recommended patterns, add sandboxing, etc etc, so that it can evolve and improve over time, rather than being ossified to the lowest common denominator of whatever silliness was happening back in the 80s.

Systemd 254 released

paulj — Thu, 03 Aug 2023 10:45:49 +0000

In my experience it _is_ pretty simple:

1. Open the init script
2. Find where it launches your daemon
3. Remove the argument telling the daemon to daemonise
4a. If this is the first one you're doing: Write the trivial systemd unit file to ExecStart that script
4b. If not the first, copy the trivial systemd unit file you've already got and change the ExecStart line.

Pretty much every daemon I've ever used, there is an argument to enable or disable daemonisation, cause a) developers want to be able to debug daemons (run under GDB usefully); b) there already are other init systems (inc. various SysVs Unixes in the olden days, like AIX, that already have various process managers; and other hacky homebrew and vendor-hacky process managers) that want processes to not daemonise; and so this is nearly always easy to figure out and set.

Systemd 254 released

bluca — Thu, 03 Aug 2023 10:35:41 +0000

Writing documentation is hard - what is obvious to me as developer, can be entirely opaque to a given user, and it's difficult to tell when what is what. In this case, to me it's perfectly obvious that ExecStop is ran regardless of _how_ a unit went away, because we don't track commands being sent by admins/programs, we track cgroups. So, to me, "Note that the commands specified in ExecStop= are only executed when the service started successfully first." is clear. But to a user, this might be totally confusing and unexpected.

So, long story short, that manpage is here: https://github.com/systemd/systemd/blob/main/man/systemd.... please send a PR to reword so that it becomes clear to you as a user, and I'll happily review and merge it

Systemd 254 released

Wol — Thu, 03 Aug 2023 10:07:15 +0000

I believe the systemd documentation itself warns of the consequences of "double daemonisation" or whatever it is in SysV scripts. There are various behaviours common (indeed, almost mandatory) under other init systems that are pathological to systemd. It would not surprise me if this binary assumes SysV (or indeed no init system at all) and behaves in a manner systemd neither likes nor expects.

What then triggers the mass killing I don't know. All I know is (1) it only happens if the systemd unit file contains an ExecStop. And (2) iirc the systemd logs actually point the finger straight at this binary!

At some point I need to fix it, but it's a load of reverse engineering I don't have time for :-(

Cheers,
Wol

Systemd 254 released

Wol — Thu, 03 Aug 2023 10:00:58 +0000

Bingo.

This thread all started because the OP to whom I replied said that someone else could port 30 or so SysV init scripts to systemd unit files in a few hours. If you don't know what that script is doing, what the daemon it's starting is doing, not it's NOT that trivial. And you don't stand a hope in hell of knocking them out that quick.

And I just gave my experience as an example, where an attempt to start a binary with a systemd unit file blew up in my face spectacularly, precisely because I didn't know what exactly that binary was doing. I know there are landmines. I know I clearly stepped on one. I just haven't debugged which one, yet :-)

Cheers,
Wol

Systemd 254 released

rschroev — Thu, 03 Aug 2023 08:33:40 +0000

I feel some of these conflict with the documentation. From https://www.freedesktop.org/software/systemd/man/systemd....

"Note that the commands specified in ExecStop= are only executed when the service started successfully first. They are not invoked if the service was never started at all, or in case its start-up failed, for example because any of the commands specified in ExecStart=, ExecStartPre= or ExecStartPost= failed (and weren't prefixed with "-", see above) or timed out."

That directly conflicts with your items 1, 2, and 4, and I feel it also conflicts with 6 and 9. And I don't see where it is documented that ExecStop= commands are called when systemd detects that processes have stopped (I haven't read *all* the documentation though, there's quite a lot of it); I feel generally while the documentation does explain what ExecStop does, it doesn't say enough about if and when ExecStop is triggered.

If the documentation is wrong, incomplete or unclear, I don't see how we're supposed to write correct unit files that work in all cases including edge cases. We shouldn't have to read the code to find out.

Systemd 254 released

zdzichu — Thu, 03 Aug 2023 07:35:09 +0000

I disagree, ExecStop can be invoked if is defined and

when sysadmin run systemctl stop unit;
when systemd decides to stop unit;
when unit exited/failed by itself.

By coincidence, systemd is open source so we don't have to guess!

Function service_enter_stop() runs ExecStop= if the section defined. service_enter_stop() is invoked in 10 cases:

line 2264 – service did not start successfully and is not RemainAfterExit=true
line 2294 – ExecStartPost failed
line 2663 – command_next failed? (I don't know this mechanism)
line 2692 – failed to start main task
line 2801 – stop command is invoked
line 3632 – PIDFile was specified but never wrote by the service
line 3688 – Out of Memory condition manifested
line 3815 – both main and control processes exited
line 4004 – another failure of PIDFile
line 4128 – RuntimeMaxSec= was reached

In summary, when ExecStop= is defined, it is run in multitude of cases, including service failure to start*. Not only when administrator requests service to stop.

* - I suspect failures to start are most often caused by wrong Type=. I once wrote a blog note with a table explaning the symptoms of mismatch.

Systemd 254 released

jem — Thu, 03 Aug 2023 06:41:18 +0000

The key thing here is what *triggers* the invocation of the command in ExecStop. The command is run as a result of systemd actively wanting to stop the service, not because it has suddenly detected that the process has exited. The sentence from "man systemd.service" you quoted means that systemd does *not* check whether the program is alive or not, it runs it anyway.

Systemd 254 released

zdzichu — Thu, 03 Aug 2023 06:28:44 +0000

Your understanding is incomplete. Quoting man systemd.service:

Also note that the stop operation is always performed if the service started successfully, even if the processes in the service terminated on their own or were killed.

Nevertheless, Wol's stories about one service stop killing unrelated services are hard to believe in. Unless he wrote ExecStop=/usr/bin/killall…

Systemd 254 released

rschroev — Wed, 02 Aug 2023 23:14:40 +0000

> I suspect the fact that it's backgrounding makes systemd think it's stopped and triggers the exec stop.

If that's what you see then that's what happening, but it goes completely against my understanding of how systemd behaves. It is my understanding that systemd runs the ExecStop commands when it wants to stop the service, not when it detects that the service is stopped.

Systemd 254 released

rschroev — Wed, 02 Aug 2023 23:01:51 +0000

If I understood everything correctly, you're trying to manage third-party software of which you don't really know how it expects to be managed properly. That's always going to be an uphill battle, regardless of which supervisor you use. Writing code, or config files in this case, doesn't work very well when you don't have a specification for what the code is supposed to do.

Systemd 254 released

mjg59 — Wed, 02 Aug 2023 17:18:59 +0000

It came from paulj in https://lwn.net/Articles/939956/ - he's suggesting that you write a systemd unit that just wraps the existing sysv script. You appear to have misinterpreted that as a suggestion that your program was a script.

Systemd 254 released

mb — Wed, 02 Aug 2023 15:09:47 +0000

>If you're a Unix fan, unit files probably felt familiar to you

I'm sorry. That doesn't really make any sense. At all.

And you're also running in circles. Multiple times. We all understand that you do have trouble writing a unit file and didn't succeed so far. But that's far from being the norm.

Systemd 254 released

Wol — Wed, 02 Aug 2023 14:37:30 +0000

Thanks. I think you're pretty spot on.

I believe I've tried "forking = yes".

You do start and stop it with special command line arguments (--start and --stop would you believe :-), and yes with --start it does go into the background.

Beyond that, I need to investigate what's going on. I suspect the fact that it's backgrounding makes systemd think it's stopped and triggers the exec stop. And when it gets that, I suspect it gets confused as to what services are its own and what are not, and sends kills to the wrong processes. But that's a debugging session I need to get into when I have time. At present I just don't use exec stop.

Because if exec stop is enabled, I can guarantee a bunch of random services will fail to start, with systemd reporting they've been killed on startup :-(

Cheers,
Wol

Systemd 254 released

Wol — Wed, 02 Aug 2023 14:21:13 +0000

"not easy", or "hard".

Not necessarily the same thing.

But you can't expect a complete novice at writing them, to churn out several in the first hour, as the OP implied. My first attempt did nothing. I scrabbled around in the documentation, emailed the mailing list, and got exec start to work. It wasn't hard, but it was quite of lot of frustration trying to find out information.

Then as I say someone else added the exec stop and all hell broke loose.

It probably is true that writing unit files is pretty easy. But not to a novice. If I stuck TCL in front of you, even with excellent documentation you'd struggle, and it really is easy.

And it may also seem odd for someone on LWN so much, but I'm not that familiar with (or a fan of) "the Unix way". What little I know is what I've had to learn (and no, I wouldn't put Windows in my "favourite OS" list, either).

If you're a Unix fan, unit files probably felt familiar to you, even when meeting them for the first time. They still feel alien to me.

Cheers,
Wol

Systemd 254 released

jem — Wed, 02 Aug 2023 12:28:49 +0000

We need more details of what the program does. Does it daemonise? If it does, you need to add Type=forking to the unit file, if not, don't add a Type= line. (What happens if the program is started from an interactive shell? Does it go into the background?)

If the program daemonises, it probably writes its PID to a file. From man systemd.service: "If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can reliably identify the main process of the service."

If the program expects to shut down in some special way, like running the program binary with a special command line parameter, add this to the ExecStop= option. The ExecStop= option is not mandatory; the default is for systemd to send a SIGTERM signal to the process, followed by a SIGTERM (if needed), with the assumption that the program catches one of these signals and does a graceful shutdown.

Systemd 254 released

anselm — Wed, 02 Aug 2023 11:46:28 +0000

I am merely observing that THAT IS WHAT HAPPENS.

To be precise, this is what happens to you, or at any rate what seems to have happened to you in one particular instance. It is certainly not the norm.

As I said, the usual way for systemd to support services which only have SysV init scripts is to construct basic systemd service units on the fly that essentially use ExecStart= and ExecStop= to invoke the init script, and that seems to work without a hitch in the vast majority of cases. Taking such an init script as a first approximation to/starting point for a free-standing service unit file would not be the dumbest of ideas (even though systemd prefers services that don't double-fork).

Systemd 254 released

mb — Wed, 02 Aug 2023 11:37:34 +0000

> "Because writing a trivial systemd unit file is not, in fact, trivial?"
> Which it isn't.

Well, it actually is trivial for many many cases.

I really think you are hitting a corner case here and your application is doing something absolutely crazy.
We obviously can't debug that here on LWN.
But please stop saying that writing systemd unit files was hard, just because you have *one* case that might be a bit harder. In general it is easy.

Systemd 254 released

Wol — Wed, 02 Aug 2023 10:43:01 +0000

I'm not blaming systemd at all. I am merely observing that THAT IS WHAT HAPPENS. Why people can't tell the difference between an objective description of reality and a subjective allocation of blame I really don't understand! Sadly the latter seems to be the norm.

Which can make fixing problems EXTREMELY difficult - one only has to look at society to see the trouble this causes :-(

Cheers,
Wol

Systemd 254 released

Wol — Wed, 02 Aug 2023 10:39:10 +0000

> The suggestion was that you write a systemd unit that runs the old sysv script, not that the program you were trying to run was a script.

And where did that come from? Not from me. Yes I think I can see the confusion, but at no point did *I* mention scripts at all. My comment was

"Because writing a trivial systemd unit file is not, in fact, trivial?"

Which it isn't. Writing my first unit file, and getting it to work with a simple ExecStart, wasn't easy. Then someone else added that ExecStop and the killing sprees started.

At some point I need to dig into the code to find out why this perfectly functional daemon does not function correctly with a very simple unit file :-(

Too many people seem to think that *their* normal applies to everyone else ...

Cheers,
Wol