LWN: Comments on "Distributors ponder a systemd change"

Distributors ponder a systemd change

mathstuf — Wed, 29 Jun 2016 13:57:19 +0000

That's my issue with the "but SIGHUP!" arguments that have been used. However, adding new signals is even more unportable than the status quo. Instead I'd rather see a nohup-like tool which stuffs a subprocess into a new PAM session which would basically allow it to persist the session and not get killed. I don't know whether such a thing even works with PAM or not though. Needs investigation.

Distributors ponder a systemd change

mcortese — Tue, 28 Jun 2016 23:14:46 +0000

While I don't have strong feelings for or against this change in systemd, I have something to say about SIGHUP, nohup & disown that many keep promoting in the comments.

In the glory days of UNIX, I would log in to a shell. When starting a user process, the choices were either run until I shut down the shell (i.e. I log out) or keep running forever. The shell would send a SIGHUP on exit, so the choices were actually either obey or ignore such signal. Since 'obey' was the default, 'ignore' had to be specified: nohup was all I needed, back then.

Today I can have several shells open inside one graphical session, and several (graphical or textual) sessions open at once. The behavior I might require from a user process varies from 'run until I shut down the shell', to 'run until I close this session', to 'run until I close all sessions' to 'run forever'. I can't express this variety with nohup. What I need is a replacement for nohup where I can specify the exact 'scope' I want to give to a process. (Ideally, that would be paralleled by new signals besides SIGHUP that express the variety of possible events with the same granularity, but I'm pragmatic enough to understand this will never happen!)

Now, I don't know if systemd-run is the "nohup on steroids" I envision, nor if KillUserProcesses can make up for the missing signals, but pretending that nohup is still the best tool we can hope for is disingenuous!

Distributors ponder a systemd change

ThinkRob — Mon, 20 Jun 2016 19:04:19 +0000

> It's Ok to introduce “bold”, backward-incompatible changes when you are doing experimental work. Once your creation is in use by millions “bold steps” are no longer allowed.

But that's the strength of the distro model isn't it? Upstream projects can experiment, do cool stuff, etc. and the distros make sure that the parts that they ship are configured to suit whatever the goals of the distro (ease of use, niche-specific stuff, whatever.)

And that's exactly what's happening here. systemd switched the default in their upstream repo, and it's up to the distros to determine whether to follow that change immediately, give people some lead time, or say "fuck it" choose to ignore the change forever. All three are valid options depending on the distro's goals.

Distributors ponder a systemd change

zblaxell — Mon, 20 Jun 2016 14:01:43 +0000

To be clear, this was never intended to be a rescue system. We did a pilot project and the results were so successful (QA particularly enjoyed having a much more repeatable testing experience) that we promoted it to production and formally terminated plans to switch to anything else. We deploy everything this way now.

I'm not sure if _any_ shell works, but bash and dash do. Any shell that can trap signals (i.e. all of them) and that uses PID 0 as the argument to waitpid (all of them written after 1987) do this just fine. The kernel blocks most of the fatal signals anyway. If /bin/sh is segfaulting you have big problems and you should probably panic the kernel to stop them from getting worse.

Distributors ponder a systemd change

zlynx — Mon, 20 Jun 2016 07:56:16 +0000

I'd have to double check but I am pretty sure the shells don't do init's job properly. Signal handing and child reaping, if I recall correctly.

You can get away with it for system rescue, but long term?

Distributors ponder a systemd change

zblaxell — Sun, 19 Jun 2016 03:53:17 +0000

My first (and last) encounter with systemd three years ago revolved around this behavior. Some Yocto distribution or other had decided to turn this on by default, and it ruined my day.

Since then I've copied the behavior for myself, in the form of a half dozen five-line shell scripts that replicate systemd's cgroup behavior. Every aspect of the cgroups' lives--how much RAM, CPU, and IO they can use, and making sure processes run, live, and die when they're told to--can be handled this way. It's _awesome_, and it's definitely one of the better ideas coming out of the systemd project.

The other thing I realized was that sysvinit had been ruining my days for years, and systemd was going to continue that pattern. To isolate myself from upstreams that should know better, but make breaking behavior changes anyway, I replaced init with a shell script. It's a little longer than five lines--ranging from 55 to 155 lines of code depending on whether it's a desktop, embedded, or server workload--but I haven't looked back since.

It was a painful transition with a bit of learning curve, but it needs much less attention than before. Apparently the best tools for the task at hand were the Unix shell, the & operator, and some small syscall wrapper programs.

Distributors ponder a systemd change

Wol — Sat, 18 Jun 2016 16:29:24 +0000

> Still, that's the only recent example I can think of where the kernel folks did something with as much potential for catastrophic breakage as this has.

Not recent, but it was a deliberate decision by Linus and it caused chaos ...

Who remembers him ripping the swap optimisation code out of kernel 2.4? Leading to a spate of linux systems crashing as sysadmins discovered "swap should be twice ram" was NOT an old wives' tale ...

Cheers,
Wol

SIGHUP for "session has gone away", not SIGTERM/SIGKILL

Wol — Fri, 17 Jun 2016 16:01:53 +0000

:-)

Certain topics press certain buttons. Gnome brings out one set of posters.Systemd brings out another (and I've noticed systemd tends to attract troll accounts I've never seen before ...)

And databases? Well that tends to get me going :-) It's all about what matters to people. And some people just enjoy sitting in the peanut gallery lobbing rotten tomatoes ... :-)

Cheers,
Wol

Distributors ponder a systemd change

Wol — Thu, 16 Jun 2016 16:18:14 +0000

> He gets angry about the amount of band-aids he is already carrying around, but this is in many ways the fact that the users already have too many of the old around and can not just fork lift fix their infrastructure at his urging.

The problem is that your "band-aid" is Lennart's security hole. All these band aids are unnecessary code, that is more likely than average to harbour bugs (and hide bugs in the other program too), and are dangerous things to leave lying around. And this example is classic - leaving processes lying around because the system can't/won't get rid of them by default is exactly that! If they're buggy enough not to shut down, how many other bugs do they harbour?

Cheers,
Wol

Distributors ponder a systemd change

Wol — Thu, 16 Jun 2016 16:12:09 +0000

> Someone with quite a bit of experience is noted for saying something like "UNIX doesn't have all the good ideas, just most of them".

Well, as someone with quite a bit of non-Unix experience (and no, that's not just Microsoft), ime Unix/linux has quite of lot of stupid ideas too. And given Linus' attitude of "We're posix compliant if posix makes sense", he probably thinks the same.

Simple example - "cp a b". What's that going to do? Oh, and I don't want a long winded answer with a load of "if"s in it :-)

Whereas on Pr1mos, "copy a b" gives me an exact copy of a (security permissions permitting) called b.

And ime, most of these comments seem to be bikeshedding between two camps - the "sooner the better" camp, and the "the right time is never" camp. If it's going to happen, then now is as good a time as any. How long has this change been in the works? Since the dawn of systemd? And if all these programs - screen, tmux, nohup, haven't done anything about it yet, then they're not going to unless something gives them a kick up the bum.

Cheers,
Wol

Distributors ponder a systemd change

geek — Thu, 16 Jun 2016 07:30:37 +0000

hey, what's wrong with Lennart's hair style?

Rant

anselm — Wed, 15 Jun 2016 15:05:09 +0000

On Debian 8 (Jessie), too. Everything is basically as it used to be.

Rant

mathstuf — Wed, 15 Jun 2016 11:56:56 +0000

Huh. The default on Fedora is for 6 VTs.

Rant

jrigg — Wed, 15 Jun 2016 08:57:58 +0000

When I tried systemd on Debian 8 Testing prior to upgrading my systems, Ctrl+Alt+Fn didn't work. It may have changed by the time Debian 8 was released (which wasn't long after I tested it) but I haven't tried it since.

Rant

micka — Wed, 15 Jun 2016 08:16:30 +0000

Debian systems and one ubuntu system.
All have consoles on Ctrl+Alt+Fn. I don't remember having changed a config setting.

Rant

jrigg — Wed, 15 Jun 2016 06:53:52 +0000

Found it: http://0pointer.de/blog/projects/serial-console.html

Looks like you can re-enable additional ttys by changing NAutoVTs= in logind.conf .

Rant

jrigg — Wed, 15 Jun 2016 06:35:37 +0000

It's switched off by default. There's a way to enable it in systemd config but I can't find a link to the info at the moment. I'm using sysvinit-core on my Debian systems which allows it to work.

Distributors ponder a systemd change

mathstuf — Wed, 15 Jun 2016 03:54:31 +0000

> If I start a process nohup, then I know what I am doing and I want it to stay around.

Most of my "nohup" processes are things like image and PDF viewers for opening attachments from mutt so that the viewers don't close when I close the tmux window. I don't think `nohup` means "should persist the session" at all; something stronger needs to be done (I have it on my TODO list to do some experimentation with a "new-session" wrapper application to start a PAM session).

Rant

mathstuf — Wed, 15 Jun 2016 03:00:52 +0000

> Many of us were used to Ctrl-Alt-Function keys giving you consoles. That got disabled too for reasons I don't understand. Well, ok.

Huh? When was this disabled? Have a reference?

Distributors ponder a systemd change

mathstuf — Wed, 15 Jun 2016 02:57:49 +0000

> KillUserProcesses is implemented, and GNOME's problem is solved.

The flag was implemented way long ago (before 2011). This is about changing the default.

Distributors ponder a systemd change

jrigg — Tue, 14 Jun 2016 13:05:18 +0000

>You awareness is incorrect. Systemd is committed to backwards compatibility

According to this, https://www.freedesktop.org/wiki/Software/systemd/MyServi... the way to allow realtime scheduling for users for a specific service is to add ControlGroup=cpu:/ to its [Service] section. The ControlGroup= option was removed in systemd 205 (July 2013) but the document hasn't been changed. That's an example of what I was referring to.

To be fair it's only one specific example, but it did contribute to my decision to stick with sysvinit-core on my Debian systems for the time being.

Distributors ponder a systemd change

Cyberax — Mon, 13 Jun 2016 20:58:46 +0000

> Keeping up with the constantly moving goalposts is another matter. Systemd's method of configuring realtime privileges for users has changed at least twice that I'm aware of (and the relevent documentation at freedesktop.org is two years out of date), for example.
You awareness is incorrect. Systemd is committed to backwards compatibility, so my units from 2012 still work just fine.

Distributors ponder a systemd change

johannbg — Mon, 13 Jun 2016 16:01:54 +0000

In my experience as linux instructor as well as handling the migration of legacy sysv initscripts in the hundreds I agree that the learning curve for systemd is less than the learning curve of both system-v and bash and shell scripting combined but managing to cram into students heads all the different type units ( close to 20 now ) within an hour or two and manage to have them write them as well within that time frame well let's just say you must have higher intelligent and more efficient audience or relatively few in class compared to me and the problem with subtle difference between distribution still exist so the notion that that upstream can use the same type unit file across distribution is not always so ( thou in many cases it will just work ).

Daemon vs socket activation, path used in type units and simply the name of the component ( apache vs httpd for example ) etc still differs between distributions and that problem will never be solved unless unification in the core/baseOS can be achieve so those upstream(s) that actually care and ship initscripts of any kind are still dealing with that issue.

Distributors ponder a systemd change

niner — Mon, 13 Jun 2016 15:32:40 +0000

They could have improved the transition by providing backwards compatibility which would have been possible. It would even be possible still. Instead they required everyone to port their libraries and have all dependencies being ported before one can port one's own code. In hindsight clearly not the winning strategy and I wish they'd have used the chance to at least make a couple more steps forward (think graphemes and GIL) when they insisted on backwards incompatibility.

Distributors ponder a systemd change

micka — Mon, 13 Jun 2016 13:33:40 +0000

Gnome is just an example of such program, but as some people like to hate gnome, the simple fact that the name appeared somewhere makes it the center of grief (something it shares with systemd).
And I suppose not all such processes use dbus.

Distributors ponder a systemd change

paulj — Mon, 13 Jun 2016 13:23:38 +0000

Make the new stuff opt-in though, and don't wilfully break the existing stuff if at all possible.

That's the basics of good system maintenance.

Please don't shaft scientific computing users!

paulj — Mon, 13 Jun 2016 13:17:49 +0000

So much better that I switched back to WindowMaker (running under mate-session) the other year. Cause session-management works - least for the stuff I care more about like non-GNOME/gtk apps, older apps, xterms, etc. and - for all its flaws - it at least isn't heave everything up and over on me every 6 months.

If I really wanted to shake up my desktop, I'd probably just go for Android as my desktop, and use some rootless Xserver for whatever older apps I needed. I'd have tried that already, but I can't just 'yum install android-desktop' on my Fedora desktop and laptop.

Distributors ponder a systemd change

jrigg — Mon, 13 Jun 2016 12:38:09 +0000

Learning the basics of systemd may well only take an hour or two. Keeping up with the constantly moving goalposts is another matter. Systemd's method of configuring realtime privileges for users has changed at least twice that I'm aware of (and the relevent documentation at freedesktop.org is two years out of date), for example.

It's easy to say that's purely a problem for the distros, but system upgrades are painful enough without having to learn several different versions of things in preparation for the next one.

Distributors ponder a systemd change

anselm — Mon, 13 Jun 2016 12:22:08 +0000

In my experience as a Linux instructor, one or two hours of systemd instruction is adequate to provide the basics for people who would otherwise be using System-V init as system administrators. Building on that, it is certainly more feasible to spend another couple of hours teaching somebody how to write a systemd service unit file for a new service and to integrate that into an existing setup, than it is to spend a couple of days teaching them enough shell scripting and distribution-specific minutiae to be able to write a robust System-V init script for a new service and to integrate that into an existing setup, on one single distribution. (The next distribution is going to be subtly, or not so subtly, different.)

For an upstream project, it is reasonable to invest the time to produce a good systemd-based configuration, which by now is likely to be applicable with few if any changes to a large number of platforms, because the effort for that is going to be smaller, in the long run, than the effort required to test and tweak new versions of their application (or application stack) on a huge number of subtly different legacy environments that all require some degree of individual adaptation.

Distributors ponder a systemd change

gb — Mon, 13 Jun 2016 11:34:11 +0000

If there are gnome processes requiring cleanup why in bloody hell this should have influence on anything else? Just ask dbus to list processes connected to it AND 'gnome' and kill em.

System-wide change is clearly not necessary here!

Distributors ponder a systemd change

johannbg — Mon, 13 Jun 2016 10:50:08 +0000

Perhaps end users can cover the very basic of systemd in an hour or two but for upstreams it requires them to have in depth understanding to be able to accept and or write and maintain an proper type unit file. So for upstreams ( and arguably administrators as well ) it takes a much more time both to grasp it and then to fully test it with their application or application stack and or infrastructure and it environment(s).

People that approach and view systemd as a new technology with new concepts adapt to it quicker than those with any background in any legacy init system in which they more often than not approach systemd as an legacy init system and apply legacy init system concepts that are not applicable to systemd ( and expect same and similar outcome or behavior ) like for example the concept of "run levels" which does not exist in systemd but the concept of boot targets does etc.

Distributors ponder a systemd change

Jluis — Mon, 13 Jun 2016 08:23:53 +0000

But kernel makes a true effort to no break userland.

Distributors ponder a systemd change

anselm — Mon, 13 Jun 2016 07:36:58 +0000

That's a lame excuse if ever I heard one.

Learning the basics of systemd takes one or two hours, tops. It's not exactly rocket science. That would cover the various types of unit files, what they contain and where they're located, how service activation works, the systemctl command and its more important subcommands, and an overview of ancillary software such as journalctl or systemd-logind. It should certainly give one enough knowledge to be dangerous and to build upon incrementally as required. There's what I would consider a reasonable primer on systemd in this manual from the tuxcademy project (although I'm biased because I wrote it myself), and Lennart Poettering's blog and the documentation on freedesktop.org are also worth a peek.

Given the importance of systemd in current and future Linux systems, one would be more than justified in considering these two hours a reasonable investment (for some people it would also be worth it just to learn enough about systemd to not appear ignorant in discussions on LWN.net). Think of it as alternative entertainment on an evening when there's nothing interesting on TV.

Distributors ponder a systemd change

jrigg — Mon, 13 Jun 2016 07:14:45 +0000

>No, he suggested that you should use a distribution that provides the required hand-holding if you can't be bothered to learn some pretty cool new stuff like systemd

There's a big difference between "can't be bothered" and "don't have time".

Distributors ponder a systemd change

anselm — Mon, 13 Jun 2016 06:18:58 +0000

No, he suggested that you should use a distribution that provides the required hand-holding if you can't be bothered to learn some pretty cool new stuff like systemd (which would incidentally come in useful in a few other places, too). Not quite the same thing.

Seriously, if you're that worked up about this, we're talking about one switch that you need to flip to make the issue at hand go away, and that's only if your distribution doesn't do it for you already. We can quibble endlessly about whether changing that default was a great idea, and there are reasonable arguments on both sides – but as far as I'm concerned, “That's not how we used to do it in 1980” is one of the less reasonable arguments.

Distributors ponder a systemd change

elvis_ — Sun, 12 Jun 2016 23:27:27 +0000

Did you really just say I shouldn't use Linux because systemd breaks userspace too much? The year of the Linux desktop may have just receded into infinity... I've been using Linux since close to when you were born, my perspective is a lot different to yours I think.

Distributors ponder a systemd change

johannbg — Sun, 12 Jun 2016 19:43:56 +0000

Kdbus is dead and the code that could use it in systemd is being removed to ensure it will never be used so kdbus and systemd is an certain "will not happen in the future".

Distributors ponder a systemd change

davidstrauss — Sun, 12 Jun 2016 15:37:58 +0000

> systemd took over DBus.

systemd *uses* DBus and provides its own DBus client library. It's basically what cURL's relationship to Apache is.

Just in case you're thinking about kdbus, it's not part of systemd, hasn't happened yet, and may or may not happen in the future.

Distributors ponder a systemd change

jspaleta — Sun, 12 Jun 2016 13:31:05 +0000

Exactly... starting to do this to... shutting down process linger as much as I possible can..and only enabling it when I'm sure I need it.
And I really love not having to run bolt-on process reapers.

I'm trying to keep my development system and even my workstation as locked down as my production environment now...and tracking the configuration differences..so I know exactly why I'm relaxing constraints on the dev system. I want to push against production constraints using non-production workloads in unexpected ways and see what breaks. The amount of relearning isn't that bad. I mean its not like relearning to jump to python3... this is minor.

Distributors ponder a systemd change

jspaleta — Sun, 12 Jun 2016 13:14:41 +0000

where did you expect it to be discussed and it wasn't?

There are a hell of a lot of communication channels... is everyone on the same page as to where they expect discussion to happen with regard to upstream changes for any project?

-jef