LWN: Comments on "Systemd vs. Docker"

Container history

JvB — Sun, 08 Mar 2020 15:00:19 +0000

In retrospect, one can see why RedHat acquired CoreOS (https://coreos.com) and now runs with Podman (https://podman.io)

Systemd vs. Docker

MarkRiggins — Sun, 17 Jul 2016 21:55:44 +0000

If the goal is to be able to run services, and clean up zombie processes, then maybe this tool that I'm developing could be useful. It's called "dockerfy" and is a simple binary that when used as the main ENTRYPOINT of a docker image, can start services, reap zombies, pre-run initialization commands (before the primary command starts), edit configuration files, and even inject secrets into containers. For example:

    RUN wget https://github.com/markriggins/dockerfy/releases/download/0.2.4/dockerfy-linux-amd64-0.2.4.tar.gz; \
        tar -C /usr/local/bin -xvzf dockerfy-linux-amd64-*tar.gz; \
        rm dockerfy-linux-amd64-*tar.gz;


    ENTRYPOINT dockerfy 
    COMMAND dockerfy \
        --secrets-files /secrets/secrets.env \
        --template /app/nginx.conf.tmpl:/etc/nginx/nginx.conf \
        --wait 'tcp://{{ .Env.MYSQLSERVER }}:{{ .Env.MYSQLPORT }}' --timeout 60s \
        --run '/app/bin/migrate_lock' --server='{{ .Env.MYSQLSERVER }}:{{ .Env.MYSQLPORT }}' --password='{{.Secret.MYSQLPASSWORD}}' -- \
        --start /app/bin/cache-cleaner-daemon -- \
        --stderr /var/log/nginx/error.log \
        --reap \
        --user nobody -- \
        nginx "-g daemon off;"

Would do the following: - load secrets from the file /secrets/secrets.env, - create an nginx.conf file from a template, - wait up to 60 seconds for a mysql database to start accepting connections - run a database migration against the mysql database, using a secret password BEFORE starting nginx - start a service named 'cache-cleaner-daemon' - start a go routine to reap zombies - run nginx as user "nobody" - tail the /var/log/nginx/error.log to stderr If the database migration fails, then the container will exit without starting nginx. While nginx is running, if the cache-cleaner-daemon dies, the entire container will shut down so the cloud platform can start up another instance. The nginx.conf.tmpl template can use the full features of Go languages templates to inject environment variables and secrets into the nginx.conf file. A template like this:

    server {
      location / {
        proxy_pass {{ .Env.PROXY_PASS_URL }};
        proxy_set_header Authorization "Basic {{ .Secret.PROXY_PASSWORD }}";

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect {{ .Env.PROXY_PASS_URL }} $host;
      }
    }

could generate an nginx.conf file like this:

    server {
      location / {
        proxy_pass http://myserver.com;
        proxy_set_header Authorization "Basic a2luZzppc25ha2Vk";

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect http://myserver.com $host;
      }
    }

Hope this helps -- You can build from source or use a pre-built binary from my latest github release

Systemd vs. Docker

davidlee — Tue, 14 Jun 2016 17:32:42 +0000

I agree with the sentiment that it is NOT Docker's fault that errant or poorly designed applications are being run inside of a container.

The "solution" I am using is call supervisord. If I need something controlled from inside the Docker, I do it myself. I note some of the previous comments made derisive comments about the kinds of init scripts folks like me might come up with. So what? I don't need their permission, nor do I need their acceptance. With four decades of script writing, I think I can write one that will do the job.

Yes, I could use systemd for those applications that install scripts it will use. I like the three-line example for Apache. But it was quite unique in that Apache installs SO MUCH STUFF that the example works. Perhaps Nagios might also work. Or Splunk. Or a myriad of other major applications which are mature enough to do so.

One of my recent docker containers was an interface with Dropbox. Nope, no three-liner there. It required too much to configure and set up. Actually, virtually every Docker container I have designed has required setup and configuration -- and, thank you very much, a carefully crafted startup script.

If I have a docker which needs to manage internal processes, I'll stick with solutions like supervisord (there are more options, but this is the one I have settled on). I'd rather use that on the few docker containers I need it than have the weight of systemd in every single docker container I generate. Imagine a busybox container with systemd running...

Systemd vs. Docker

nye — Wed, 16 Mar 2016 13:56:15 +0000

>"Wow" indeed! If he's that bad, what position does that leave *you* in with your outright attacks on his person?
>Oh wait, I think I know: *plonk*

Wait, what? I think you might have got the two people in this discussion mixed up?

Systemd vs. Docker

flussence — Sat, 12 Mar 2016 22:17:20 +0000

It'd be a good idea to use a small wrapper for both cases actually - either you want to clean up child processes, or you'd want to use seccomp to forbid unexpected ones...

Systemd vs. Docker

ploxiln — Sat, 12 Mar 2016 20:08:49 +0000

Not all services end up creating zombie processes. Some spawn no sub-processes at all (example: nodejs application you write yourself). Others create some sub-processes which create no further sub-processes and manage them correctly (example: nginx and its own worker processes, or a python wsgi application you write yourself and run under gunicorn).

Some services do spawn child processes, which spawn children themselves and then sometimes die without cleaning them up. This does result in zombies. But you can and should avoid such services. If you really need to run one, you can use a 20-some line c utility which will just run the service and reap zombies (linked to in another reply).

Whether systemd has too much functionality for init on a normal desktop is debatable. But for a container? Yes, definitely.

Systemd vs. Docker

evinco — Thu, 10 Mar 2016 09:59:47 +0000

No, it doesn't require that. rkt (for example) is tested on systems that don't have systemd as PID1.

And yeah, the default rkt stage1 implementation uses `systemd-nspawn` internally.

Systemd vs. Docker

mathstuf — Thu, 10 Mar 2016 02:17:33 +0000

Right. Which is why I think it's not really a valid comparison.

Systemd vs. Docker

ibukanov — Tue, 08 Mar 2016 11:36:25 +0000

> just to support stuff like secrets, monitoring, etc

Could you clarify this? It is very trivial to deal with passing secrets to Docker with a minimal shell script as the tool provides all the necessary hooks. As for monitoring, support for logging to syslog or journald with custom tags in Docker 1.8 resolved all the monitoring issues for me.

Systemd vs. Docker

vonbrand — Tue, 08 Mar 2016 10:23:19 +0000

Fork and scrap was the way forward here, why fork? Besides, upstart in practice was only used as a SysVinit drop-in replacement.

Systemd vs. Docker

dberkholz — Tue, 08 Mar 2016 05:58:34 +0000

This isn't a real restriction, if you fork prior to rearchitecture.

Systemd vs. Docker

vonbrand — Mon, 07 Mar 2016 02:22:06 +0000

Upstart style events need to be quite a bit more abstract than "A started" to be useful in case the functionality offered by A can be obtained in different ways... and there it breaks down.

Systemd vs. Docker

cg909 — Mon, 07 Mar 2016 01:41:12 +0000

Using process groups would work for simple daemons, but not for services like sshd.

The main problem is that process groups always belong to a session. So every service that spawns user sessions would also need to break out of the process group.

If you use seccomp filters to ignore setpgrp() and setsid() sshd would fail in spectacular ways as all processes in the process group will share the same controlling terminal and so every process spawned by sshd will receive SIGHUP when a session is closed. Also anything spawning a shell might run into problems as shells use process groups to separate tasks.

You'd need "super process groups" which may span multiple sessions and contain multiple process groups.

And this is what cgroups provide.

Systemd vs. Docker

smurf — Sun, 06 Mar 2016 22:50:15 +0000

Sure, if you have a reasonably complete set of test cases refactoring is easy. ;-)

In this case, about half of Upstart's events translate reasonably cleanly to dependencies (i.e. instead of job B's start being triggered by "A has started", let B simply depend on A). Deprecate the other events aggressively. Now you can hook dependent jobs to states instead of state transitions without changing anything. Next step, add stuff you couldn't even think about in Upstart ("let D run as soon as both B and C are up", B and C being independent of each other). At last you're in a position to add targets. QED.

That being said, I'd guesstimate that doing this requires at least three times the amount of work on the core system, compared to starting from scratch, and a work-in-progress that's definitely worse than Upstart. Thus, one might reasonably conclude that Upstart's copyright assignment policy was a Good Thing in this case because it prevented the systemd people from getting mired in a prolonged Upstart refactoring that may or may not have succeeded.

Systemd vs. Docker

vonbrand — Sun, 06 Mar 2016 17:13:23 +0000

In this case, the path forward was from SysVinit to something new. Upstart never got any significant hold except as a SysVinit lookalike. And systemd also offers SysVinit compatibility, but the switchover to native systemd was surprisingly fast and smooth, something I believe shows they got it right. Many distribution's users fought it nail and tooth in the ubiquitous flamefests, while their developers just moved over silently.

Systemd vs. Docker

reedstrm — Sun, 06 Mar 2016 15:34:21 +0000

Yes, it usually is very mind boggling, and requires thinking about upgrade paths while doing the actual development. It's not something you can bolt on after, and is the source of the classic tension between developers and admins.

There almost always is a way to do it, if you're willing to spend the resources (time as much as money). Unfortunately, actual rearchitecting (as opposed to new feature development) requires someone with a deep understanding of both the old and new systems, to develop the path forward. And patience to complete, as well as put up with any less than perfect intermediate states along the way. Usually a hard sell in today's go go go faster faster world.

Systemd vs. Docker

vonbrand — Sun, 06 Mar 2016 15:17:39 +0000

Reachitecture and incrementally. The mind boggles.

Systemd vs. Docker

hummassa — Sun, 06 Mar 2016 15:16:21 +0000

THIS. But not only this, I will say as one that had a great deal of suspicion towards the whole systemd thing initially: yes, Debian works BETTER and smoother with systemd. I probably will not try Devuan in the near future.

Systemd vs. Docker

anselm — Sun, 06 Mar 2016 12:21:59 +0000

There seems to be this weird notion around that the systemd developers were somehow able to pressurise everybody into adopting systemd against their better judgement. In actual fact, the mainstream Linux distributions came on board exactly because their developers looked at systemd and decided – in some cases after extensive and painful deliberation – that it was an idea worth pursuing, over the available alternatives. Sounds like “market acceptance of their ideas” to me.

Would it have been possible to beef up SysV init or Upstart to do what systemd does? Perhaps, but in spite of a lot of talk about doing this, nobody volunteered to put in the effort – and in the free-software world, working code beats hypothetical yet-to-be-written code any day.

Systemd vs. Docker

hitmark — Sun, 06 Mar 2016 10:06:36 +0000

"If they wanted to remake the Linux ecosystem, fine... They should have done what everyone else has had to do and build a distribution that fits their needs, then let it rise or fall according to market acceptance of their ideas. Instead, they snuck it in under the guise of a small improvement to one specific area that no one thought would cause too much trouble"

Oh so very very much this.

Systemd vs. Docker

hitmark — Sun, 06 Mar 2016 10:04:15 +0000

Yeah it feels like Fedora has become RHs lightning rod for some of their more "devops" oriented employees. A place for them to "go nuts" while RHEL picks up the potentially best bits and package them for corporate/government palates.

Systemd vs. Docker

hitmark — Sun, 06 Mar 2016 10:01:32 +0000

That one requires that systemd is sitting as pid1 on the host system, no?

Also, last time i checked, rkt is specifically built on top of systemd.

Systemd vs. Docker

smurf — Sun, 06 Mar 2016 07:04:37 +0000

That re-architecture could have been done somewhat-incrementally, and IIRC Lennart & Co. even looked into that, but with the copyright assignment requirement this wouldn't have worked. Too many people refuse to sign the stuff.

Systemd vs. Docker

vonbrand — Sun, 06 Mar 2016 03:51:09 +0000

Check out the systemd documents on why it's development started. You'll see that the upstart problems were severe enough to require a complete rearchitecture, not just some bug fixes. The whole basic model (define some "events", and trigger them) is fundamentally impossible to scale, you can't decouple things enough. That is a good part of the reason why no native upstart configurations ever materialized. The "refurbished upstart" might just look an awfully lot like systemd. In a sense, systemd is upstart's better replacement.

Systemd vs. Docker

raven667 — Sun, 06 Mar 2016 03:43:07 +0000

>Upstart was basically dead in the water since nobody seemed eager to take on the work

Right, I was just pointing out my opinion on what the largest factor which made it difficult to pick up the maintainership of Upstart was Canonical's insistence on copyright ownership. Many strong developers (or their employers) insist on owning their own copyrights, with few exceptions like for the FSF. If a strong maintainer had stepped forward and had been effective in getting the restructuring done there never would be a systemd. Even if systemd were written in a world with a non-broken Upstart, it wouldn't gain any traction because it wouldn't be enough better to justify the cost of transition.

Systemd vs. Docker

anselm — Sun, 06 Mar 2016 00:30:59 +0000

The original author of Upstart (who is now working on different projects at Google) is on record as saying that it needed a massive redesign in order to cope with some nasty architectural deficiencies that were discovered in practice. At that point Upstart was basically dead in the water since nobody seemed eager to take on the work, and the Ubuntu people sensibly decided to go along with most other Linux distributions and adopt systemd.

(The fact that the Debian project had just decided to opt for systemd as the default init system on Linux also contributed to this, since if Debian had gone with Upstart then the Ubuntu people would have been able to rely on Debian developers doing much of the work adapting existing packages to native Upstart, instead of having to do all of the work by themselves.)

Systemd vs. Docker

raven667 — Sat, 05 Mar 2016 20:47:35 +0000

>Did anyone actually move to upstart?

I don't think so, there were some high hopes that upstart would be the next gen init system but clearly the native job files arent sufficient in practice and fixing the deficiencies was not possible without Canonical dropping the copyright assignment requirement, for-profit companies often don't like doing free consulting work for other for-profit companies, not even the original author was allowed to submit patches after taking a job working on ChromeOS.

Systemd vs. Docker

seyman — Sat, 05 Mar 2016 06:48:20 +0000

> I'd submit that neither the systemd-as-init nor the systemd-as-collection-of-tools aspects have been anywhere near as "operationally invisible" to sysadmins as the move to upstart was.

Did anyone actually move to upstart? Even Ubuntu never used native upstart job files for the vast majority of their services in their distribution, preferring to run them using upstart's compatibility mode.

Systemd vs. Docker

johannbg — Fri, 04 Mar 2016 22:30:55 +0000

There was never any work done by Red Hatters to migrate anything to native upstart configurations they just implemented it in Fedora functioning just like SysV did thus half implemented as so many other features flagged %100 done by the ( broken ) feature process in the project.

If they would have, it most certainly would never have been ""operationally invisible".

Systemd vs. Docker

mathstuf — Fri, 04 Mar 2016 21:17:52 +0000

> I'd submit that neither the systemd-as-init nor the systemd-as-collection-of-tools aspects have been anywhere near as "operationally invisible" to sysadmins as the move to upstart was.

Not sure this is a fair comparison. The switch to upstart didn't make anything worse, but I also never noticed anything getting better with it either. With systemd, on the other hand, I've seen bugs, but also *vast* improvements in various places, across the board. Certainly, for me, worth the issues I've hit.

supporting musl libc

nwmcsween — Fri, 04 Mar 2016 16:27:22 +0000

The only real required bit is the printf.h support but it exposes libc internals to poke at which is unacceptable but sadly not realized by systemd.

Systemd vs. Docker

evinco — Fri, 04 Mar 2016 14:18:43 +0000

> Because systemd does things in a very unique way, it does not fit easily inside a container.

This is a strange claim; the systemd developers run it in a container all the time (via `systemd-nspawn --boot`) to test it, and it works perfectly well. rkt works similarly.

(By "container" here I'm referring to the most common technical conception, of Linux chroot+cgroups+namespaces. Maybe you have a different definition.)

Systemd vs. Docker

ofr — Fri, 04 Mar 2016 12:16:32 +0000

Yelp has written and open-sourced a "dumb-init" for that exact purpose:
http://engineeringblog.yelp.com/2016/01/dumb-init-an-init...

Systemd vs. Docker

vonbrand — Fri, 04 Mar 2016 11:12:07 +0000

I must second that. Not often, but enough to be annoying, I saw SysVinit (and it's upstart emulation) messing up. To be able to start and stop anything anytime, with no regard to dependencies, meant one had to have a map of running services for all the random machines with specialized tasks in the head, or figure it out on the fly. Not funny.

Systemd vs. Docker

smurf — Fri, 04 Mar 2016 07:41:38 +0000

> value new and shiny things over reliability

I seriously doubt that it's just my experience which shows that, once adapted to, systemd is *way* better at reliability, consistency and whatnot than any alternate system one could name.

Systemd vs. Docker

raven667 — Fri, 04 Mar 2016 02:13:42 +0000

> they are still using EL5 on some systems

I still have plenty of EL5 systems as well, for most purposes its totally fine, feature complete. We will migrate in the future because our developers are getting tired of PERL 5.8.8 and would like to be able to use current libraries without trouble, other than that I can't think of any base OS improvement that is substantial, except for systemd, which I like because I'm tired of crappy init scripts or people just running things out of rc.local.

Systemd vs. Docker

raven667 — Fri, 04 Mar 2016 02:03:15 +0000

When you work with other people who use a system, what you think of as "intended purpose" may not matter all that much, what people want to do and are doing with the system is what matters, in this case running Docker format container services orchestrated through systemd and its existing dependency resolution and service management framework. I'm sorry that you personally think such use is "unnatural" but ultimately your customers decide.

Systemd vs. Docker

raven667 — Fri, 04 Mar 2016 02:00:28 +0000

> systemd dictates the rules and other applications adapt.

The fact that you perceive this from an authoritarian perspective, where one part of the system dictates to subordinate parts and you perceive systemd in a container as a priority inversion, is part of the disconnect, you can't see that different parts of the system, different groups of developers, work together to a common goal without a master/slave, dictator/subject relationship.

Systemd vs. Docker

jcc1 — Thu, 03 Mar 2016 23:04:30 +0000

This view seems made up to me, and I can speak to why personally.

It's quite possible that we run in different circles, but the view is definitely not "made up". Every EL shop I know of is being more cautious with EL7 than with previous transitions.

Mostly false, at least for the cause/effect you're suggesting. Fedora lost substantial users over the F15-F20 period, but it has more than recovered them since (F21 and F22) [3] without reversing those decisions. If those were the decisions rationally driving users away, they would have stayed away.

Although numbers have recovered, that doesn't really speak to whether the then-users stayed away or not. As focus changes, so does audience. I'd suggest that Fedora users who didn't agree with direction things were going went with EL, or perhaps Debian. e.g., from this wonderful thread: https://lists.fedoraproject.org/pipermail/devel/2011-June/152636.html

What tests could show that systemd is "operationally invisible" to most users running distributions with it?

Well, perhaps we're equivocating on "user" here, a not-uncommon issue. For some distributions, "users" are people relying on the distribution to control apps. For others, "users" are the system administrators who are deeply involved in reliability engineering and may or may not have specific ways of setting up systems for their company's needs.

I'd submit that neither the systemd-as-init nor the systemd-as-collection-of-tools aspects have been anywhere near as "operationally invisible" to sysadmins as the move to upstart was. Depending on the needs of non-sysadmin users who don't know what the /etc/fstab file is, it may have been more "operationally invisible", except for the opportunity cost of all the other people having to deal with systemd bugs over the years.

Systemd vs. Docker

johannbg — Thu, 03 Mar 2016 21:38:33 +0000

"rejected as a F14 feature"
I do believe I'm not misremembering anything but then again this was 5 years ago but as I recall it, the reason systemd got rejected in F14 was lack of proper documentation both upstream ( man pages ) and downstream, though I think I, Rahul Sundaram, Lennart and Matthias Clasen manage to take care of most that before F14 got released, lack of clear packaging guidelines and related upgrade path and hand full of bugs somewhere between 5 - 10 which I think Lennart and Kay had fixed before F14 got released but due to a short timeframe and to much of grey area surrounding this FESCo decided to take the safer route ( defer systemd to the f15 release ) to have those things properly settled out, despite the media hype around that feature ( which the feature process basically is ) that had taken place.

"Many of the most active contributors on fedora-devel during that time were NOT Red Hat employees" I'm not sure what you mean by that but the components that make up the RHEL release has always been owned and maintained by Red Hat employees most of which are smart and thus stay away from getting involved in distribution politics so you wont notice them on -devel. ( It's not like it matters decisions that affect those components are made elsewhere than in Fedora ) and the number of active "core" package maintainers as always remained the same despite significant growth in components in the distribution and since stats ( in no relation to what actually matters ) where thrown at devconf this year I'm pretty sure in twenty sixteen we will see alot of hype surrounding much about nothing regarding those stats from the PLL.

"The folks that care about reliability and consistency haven't used Fedora as a server platform for years (unless it's being managed hands-off by the end-user via Plesk or some other control panel) -- they've been using EL or one of its rebuilds. They (we) assumed that Fedora was being reasonable in their choices, not realizing that Fedora had caught the "developer" disease of trying to endlessly fix everything."

Oh really I know alot of people that have used and still use Fedora as server platform in business and mission critical infrastructure so I'm not getting what you are referring to and the core/baseOS has been pretty well maintained.
( We are not talking about Anaconda/Gnome here which get rewritten every cycle with let's throw it over the wall and see what leaks mentality ).

There was no "mass exodus around that time" no more than was happening with other distribution and happens every cycle for all of them ( distribution hoppers jump between distribution which is the feature most or the most hype is about ) and you certainly cannot directly relate that to systemd, Gnome Shell, X, rewrite and or whatever else was supposed to be going on and if you are claiming that by all means show the stats to back that up.