Systemd vs. Docker
Systemd vs. Docker
Posted Mar 1, 2016 18:10 UTC (Tue) by jcc1 (guest, #107291)In reply to: Systemd vs. Docker by ibukanov
Parent article: Systemd vs. Docker
A typical use case for Docker is a container running a single process or perhaps the main process and few workers. For this use cases the extra requirements imposed on PID-1 does not matter at all. For more exotic cases signal handling can be done using a simple C program like https://github.com/Yelp/dumb-init/blob/master/dumb-init.c .
I suppose Red Hat guys are perfectly aware of the triviality of the issue.
They're perfectly aware (and, to be fair, it's more the "Fedora guys" and systemd "cabal" than Red Hat per se), but allowing that would reduce the impact of the Embrace, Extend, Extinguish "take no prisoners" strategy with regard to init systems.
A traditional 20-line C program as "/sbin/init" is verboten. Not just an option, explicitly disallowed and unpackaged. Traditional initscripts used for starting on other services can now no longer be packaged in Fedora -- not even as optional packages, not packaged at all. This rapid split is a source of continual frustration for those of us using versions 5 or 6 of RHEL or any of its rebuilds (CentOS or Scientific Linux) and the EPEL packages.
The systemd cabal are being poor winners and trying to eliminate any vestige of anything not meeting their aesthetics. I wish more people were willing to seriously reconsider Fedora's place as the top of the Red Hat ecosystem. IMO, they no longer deserve it... and haven't since the F15 era.
Posted Mar 2, 2016 16:09 UTC (Wed)
by johannbg (guest, #65743)
[Link] (31 responses)
Red Hat owns the Fedora trademark thus Red Hat polices Fedora as in
In the end of the day Red Hat controls ( not just sponsors ) Fedora from a to z with the end result and the fact that Fedora is neither free and available to all ( for example it's bound u.s. export administration regulations ) nor a community project, there exist only the sold out lie and the illusion of community participation since for as long as your contribution aligns with Red Hat goals or does not interfere with it and their early product implementation and adoption you are a good community member but if it does not, as in competes with it's solution, be it application,application stack or product you work will not stand on equal term with theirs and they will yank it's corporate slave chain that community members have around their nack so they fall back into their place as a second class Fedora citizen member or even go so far as to threaten to out them from the project so claiming that Fedora is not Red Hat or is just wrong and there exist no "Fedora Guys' capable of changing anything let alone the direction of the project.
And FYI there exist no "Embrace, Extend, Extinguish take no prisoners" strategy within the systemd cabal or the systemd community in general and even if it did, nobody from within the systemd community has forced downstreams to adopt systemd including Fedora.
It was quite foreseeable when systemd became the init system in Fedora that eventually it would not ship any more legacy sysv initscript since it was driven by a community member not Red Hat employee but the plan was of the one that handle the various integration/migration bits of systemd in Fedora ( including the migration for legacy sysv initscript ) to have every component of Fedora migrated before that happened but as luck had it Red Hat employees in FESCo thought they knew better than the guy that had been working on it for four years and when that happened that itch to scratch turned into bruise, with those Red Hat employees that thought that they "knew better" failing miserably in completely that work, which in turned forced them to drop all existing un-migrated component, since those component where starting to becoming hindrance to other ( deprecation ) work to be completed.
That should not come as a surprise since it just followed Red Hat patterns of drive by featuring and half implementing things into the distribution just like Casey Dahlin an Red Hat employee responsible for and ran the feature of switching the init system to upstart where no initscript got migrated to native upstart format.
Posted Mar 3, 2016 18:33 UTC (Thu)
by jcc1 (guest, #107291)
[Link] (30 responses)
There was a huge amount of discussion on the Fedora-devel list over this, but the initial presentation and feature set was "a replacement for init (upstart) that does better parallelization". That alone was controversial enough, which led to it being rejected as a F14 feature (along "if not broke, don't fix it" and KISS lines). It got even more heated once the actual problems in the paradigm started impacting people as F15 started taking shape.
Many of the most active contributors on fedora-devel during that time were NOT Red Hat employees. They were pushing systemd for various reasons, but there had always been (and still remains) a desire there to be on the bleeding edge, and value new and shiny things over reliability. That's the problem. At some point, Fedora's leaders and most of its user base stopped being sysadmins and started being developers
The folks that care about reliability and consistency haven't used Fedora as a server platform for years (unless it's being managed hands-off by the end-user via Plesk or some other control panel) -- they've been using EL or one of its rebuilds. They (we) assumed that Fedora was being reasonable in their choices, not realizing that Fedora had caught the "developer" disease of trying to endlessly fix everything.
In short, there was a mass exodus around that time, caused not only by systemd, but also other needlessly disruptive dumb ideas, like mandated TmpfsOnTmp, and the UsrMove. If "Red Hat" corporate were in control like you say, they would have listened to their revenue-generating customers who'd have said that the churn wasn't anywhere near the breakage. Instead, the Freedesktop.org crowd (i.e., the "systemd cabal") prioritized features they wanted on their laptop over any problems for server users.
I have no idea why you're dismissing the concept of an agenda by LP and others -- it's completely salient in the presentations, discussions, and white papers they've been making. It WASN'T *completely* salient in the Fedora 14/15 days, which is why many people are upset about this all. If they wanted to remake the Linux ecosystem, fine... They should have done what everyone else has had to do and build a distribution that fits their needs, then let it rise or fall according to market acceptance of their ideas. Instead, they snuck it in under the guise of a small improvement to one specific area that no one thought would cause too much trouble -- especially in light of the fact that the transition from "traditional" sysvinit to upstart was mostly invisible and painless. (Primarily because, as you said, Fedora/RHEL was *using* upstart, but wasn't really making use of any of its advanced features. Many thought systemd would be similarly handled -- present if you wanted to use the new hotness as a service manager, but otherwise operationally invisible.)
Furthermore, even agreeing with what you'd said about everything else, LP is an employee of Red Hat too. If RH Corporate controls things, then he's quite part of that.
Posted Mar 3, 2016 19:44 UTC (Thu)
by pizza (subscriber, #46)
[Link] (4 responses)
Red Hat's revenue-generating customers don't use Fedora, they use RHEL.
And I may be wasting my breath, but I'll also point out that systemd is far more useful on servers than laptops.
(And, incidentally, RHEL7, with systemd, has now been out for about a year. RH's revenue-generating customers have yet to leave en masse)
Posted Mar 3, 2016 20:04 UTC (Thu)
by jcc1 (guest, #107291)
[Link] (3 responses)
Posted Mar 3, 2016 20:50 UTC (Thu)
by bronson (subscriber, #4806)
[Link]
Not true. You're writing in an omniscient voice but you certainly aren't speaking for me. My team moved to RHEL 7 just for systemd. We're sick of process monitors and their bugs. (it's especially great when you discover your process monitor has a memory leak... good thing there are zillions to choose from.)
So far systemd has been better than I'd hoped. For deploying web services, it's easy to use, mature, and rock solid. Everyone on the team has gotten to know its config -- no need to learn how to work with some guys favorite tool.
Reliable as all hell. No way we'd go back.
> I know a lot of shops that are staying with EL6 for the time being
And I know a lot of shops that are still on 5. Has nothing to do with systemd.
If you were to tone it down a bit, maybe I could understand what you're getting at... Right now, though, it seems like many of your statements are demonstrably false.
Posted Mar 3, 2016 21:30 UTC (Thu)
by pizza (subscriber, #46)
[Link] (1 responses)
Speaking for my employer, they're sticking with EL6 because the vendors of their business-criticial EDA tools haven't certified said tools for use with EL7 yet. Indeed, they are still using EL5 on some systems for similar reasons.
(We're talking about per-seat licensing costs in the upper five figures here. The actual RHEL license price is a rounding error, as is the hardware cost for all but the beefiest of boxes)
Posted Mar 4, 2016 2:13 UTC (Fri)
by raven667 (subscriber, #5198)
[Link]
I still have plenty of EL5 systems as well, for most purposes its totally fine, feature complete. We will migrate in the future because our developers are getting tired of PERL 5.8.8 and would like to be able to use current libraries without trouble, other than that I can't think of any base OS improvement that is substantial, except for systemd, which I like because I'm tired of crappy init scripts or people just running things out of rc.local.
Posted Mar 3, 2016 20:14 UTC (Thu)
by davidstrauss (guest, #85867)
[Link] (17 responses)
This view seems made up to me, and I can speak to why personally. My company runs runs hundreds of servers on Fedora, and I am a systemd committer. Both have been true since about Fedora 18. I have also worked with the primary systemd developers on key issues affecting server infrastructure, including being invited to present on remaining issues at systemd's latest conference [1]. And, in case you're wondering, those remaining issues relate to optimizations at scale, not fundamental design issues with what systemd does.
Moreover, there has been substantial use on the server side by plenty of other people. Some early adopters were shoehorning it into CentOS 6 to use systemd's tools for their own services; one of those companies posted publicly to the Debian mailing list during the init discussions to publicly support moving to systemd. I've also been asked to speak at Facebook on best practices for running services within systemd, which I'm doing later this month [2].
> In short, there was a mass exodus around that time, caused not only by systemd, but also other needlessly disruptive dumb ideas, like mandated TmpfsOnTmp, and the UsrMove.
Mostly false, at least for the cause/effect you're suggesting. Fedora lost substantial users over the F15-F20 period, but it has more than recovered them since (F21 and F22) [3] without reversing those decisions. If those were the decisions rationally driving users away, they would have stayed away. Also, other distributions adopting systemd have not seen any "mass exodus" at the point of their decision, either. So, your claim fails doubly: (1) reversal of effect without alteration of the supposed cause and (2) lack of effect when applying the supposed cause elsewhere.
> Instead, the Freedesktop.org crowd (i.e., the "systemd cabal") prioritized features they wanted on their laptop over any problems for server users.
I'm not even sure how to disprove that systemd developers are in some kind of FreeDesktop.org crowd/cabal.
> Many thought systemd would be similarly handled -- present if you wanted to use the new hotness as a service manager, but otherwise operationally invisible.
Likewise here. You're not providing testable criteria, just other statements that are "not even wrong." [4] How could I falsify your view here? What tests could show that systemd is "operationally invisible" to most users running distributions with it?
[1] https://systemd.events/systemdconf-2015/sessions/challeng...
Posted Mar 3, 2016 23:04 UTC (Thu)
by jcc1 (guest, #107291)
[Link] (16 responses)
Posted Mar 4, 2016 21:17 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
Not sure this is a fair comparison. The switch to upstart didn't make anything worse, but I also never noticed anything getting better with it either. With systemd, on the other hand, I've seen bugs, but also *vast* improvements in various places, across the board. Certainly, for me, worth the issues I've hit.
Posted Mar 4, 2016 22:30 UTC (Fri)
by johannbg (guest, #65743)
[Link] (1 responses)
If they would have, it most certainly would never have been ""operationally invisible".
Posted Mar 10, 2016 2:17 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link]
Posted Mar 5, 2016 6:48 UTC (Sat)
by seyman (subscriber, #1172)
[Link] (12 responses)
Did anyone actually move to upstart? Even Ubuntu never used native upstart job files for the vast majority of their services in their distribution, preferring to run them using upstart's compatibility mode.
Posted Mar 5, 2016 20:47 UTC (Sat)
by raven667 (subscriber, #5198)
[Link] (11 responses)
I don't think so, there were some high hopes that upstart would be the next gen init system but clearly the native job files arent sufficient in practice and fixing the deficiencies was not possible without Canonical dropping the copyright assignment requirement, for-profit companies often don't like doing free consulting work for other for-profit companies, not even the original author was allowed to submit patches after taking a job working on ChromeOS.
Posted Mar 6, 2016 0:30 UTC (Sun)
by anselm (subscriber, #2796)
[Link] (10 responses)
The original author of Upstart (who is now working on different projects at Google) is on record as saying that it needed a massive redesign in order to cope with some nasty architectural deficiencies that were discovered in practice. At that point Upstart was basically dead in the water since nobody seemed eager to take on the work, and the Ubuntu people sensibly decided to go along with most other Linux distributions and adopt systemd.
(The fact that the Debian project had just decided to opt for systemd as the default init system on Linux also contributed to this, since if Debian had gone with Upstart then the Ubuntu people would have been able to rely on Debian developers doing much of the work adapting existing packages to native Upstart, instead of having to do all of the work by themselves.)
Posted Mar 6, 2016 3:43 UTC (Sun)
by raven667 (subscriber, #5198)
[Link] (9 responses)
Right, I was just pointing out my opinion on what the largest factor which made it difficult to pick up the maintainership of Upstart was Canonical's insistence on copyright ownership. Many strong developers (or their employers) insist on owning their own copyrights, with few exceptions like for the FSF. If a strong maintainer had stepped forward and had been effective in getting the restructuring done there never would be a systemd. Even if systemd were written in a world with a non-broken Upstart, it wouldn't gain any traction because it wouldn't be enough better to justify the cost of transition.
Posted Mar 6, 2016 3:51 UTC (Sun)
by vonbrand (subscriber, #4458)
[Link] (8 responses)
Check out the systemd documents on why it's development started. You'll see that the upstart problems were severe enough to require a complete rearchitecture, not just some bug fixes. The whole basic model (define some "events", and trigger them) is fundamentally impossible to scale, you can't decouple things enough. That is a good part of the reason why no native upstart configurations ever materialized. The "refurbished upstart" might just look an awfully lot like systemd. In a sense, systemd is upstart's better replacement.
Posted Mar 6, 2016 7:04 UTC (Sun)
by smurf (subscriber, #17840)
[Link] (7 responses)
Posted Mar 6, 2016 15:17 UTC (Sun)
by vonbrand (subscriber, #4458)
[Link] (4 responses)
Reachitecture and incrementally. The mind boggles.
Posted Mar 6, 2016 15:34 UTC (Sun)
by reedstrm (guest, #8467)
[Link] (1 responses)
There almost always is a way to do it, if you're willing to spend the resources (time as much as money). Unfortunately, actual rearchitecting (as opposed to new feature development) requires someone with a deep understanding of both the old and new systems, to develop the path forward. And patience to complete, as well as put up with any less than perfect intermediate states along the way. Usually a hard sell in today's go go go faster faster world.
Posted Mar 6, 2016 17:13 UTC (Sun)
by vonbrand (subscriber, #4458)
[Link]
In this case, the path forward was from SysVinit to something new. Upstart never got any significant hold except as a SysVinit lookalike. And systemd also offers SysVinit compatibility, but the switchover to native systemd was surprisingly fast and smooth, something I believe shows they got it right. Many distribution's users fought it nail and tooth in the ubiquitous flamefests, while their developers just moved over silently.
Posted Mar 6, 2016 22:50 UTC (Sun)
by smurf (subscriber, #17840)
[Link] (1 responses)
In this case, about half of Upstart's events translate reasonably cleanly to dependencies (i.e. instead of job B's start being triggered by "A has started", let B simply depend on A). Deprecate the other events aggressively. Now you can hook dependent jobs to states instead of state transitions without changing anything. Next step, add stuff you couldn't even think about in Upstart ("let D run as soon as both B and C are up", B and C being independent of each other). At last you're in a position to add targets. QED.
That being said, I'd guesstimate that doing this requires at least three times the amount of work on the core system, compared to starting from scratch, and a work-in-progress that's definitely worse than Upstart. Thus, one might reasonably conclude that Upstart's copyright assignment policy was a Good Thing in this case because it prevented the systemd people from getting mired in a prolonged Upstart refactoring that may or may not have succeeded.
Posted Mar 7, 2016 2:22 UTC (Mon)
by vonbrand (subscriber, #4458)
[Link]
Upstart style events need to be quite a bit more abstract than "A started" to be useful in case the functionality offered by A can be obtained in different ways... and there it breaks down.
Posted Mar 8, 2016 5:58 UTC (Tue)
by dberkholz (guest, #23346)
[Link] (1 responses)
Posted Mar 8, 2016 10:23 UTC (Tue)
by vonbrand (subscriber, #4458)
[Link]
Fork and scrap was the way forward here, why fork? Besides, upstart in practice was only used as a SysVinit drop-in replacement.
Posted Mar 3, 2016 21:38 UTC (Thu)
by johannbg (guest, #65743)
[Link]
"Many of the most active contributors on fedora-devel during that time were NOT Red Hat employees" I'm not sure what you mean by that but the components that make up the RHEL release has always been owned and maintained by Red Hat employees most of which are smart and thus stay away from getting involved in distribution politics so you wont notice them on -devel. ( It's not like it matters decisions that affect those components are made elsewhere than in Fedora ) and the number of active "core" package maintainers as always remained the same despite significant growth in components in the distribution and since stats ( in no relation to what actually matters ) where thrown at devconf this year I'm pretty sure in twenty sixteen we will see alot of hype surrounding much about nothing regarding those stats from the PLL.
"The folks that care about reliability and consistency haven't used Fedora as a server platform for years (unless it's being managed hands-off by the end-user via Plesk or some other control panel) -- they've been using EL or one of its rebuilds. They (we) assumed that Fedora was being reasonable in their choices, not realizing that Fedora had caught the "developer" disease of trying to endlessly fix everything."
Oh really I know alot of people that have used and still use Fedora as server platform in business and mission critical infrastructure so I'm not getting what you are referring to and the core/baseOS has been pretty well maintained.
There was no "mass exodus around that time" no more than was happening with other distribution and happens every cycle for all of them ( distribution hoppers jump between distribution which is the feature most or the most hype is about ) and you certainly cannot directly relate that to systemd, Gnome Shell, X, rewrite and or whatever else was supposed to be going on and if you are claiming that by all means show the stats to back that up.
Posted Mar 4, 2016 7:41 UTC (Fri)
by smurf (subscriber, #17840)
[Link] (1 responses)
I seriously doubt that it's just my experience which shows that, once adapted to, systemd is *way* better at reliability, consistency and whatnot than any alternate system one could name.
Posted Mar 4, 2016 11:12 UTC (Fri)
by vonbrand (subscriber, #4458)
[Link]
I must second that. Not often, but enough to be annoying, I saw SysVinit (and it's upstart emulation) messing up. To be able to start and stop anything anytime, with no regard to dependencies, meant one had to have a map of running services for all the random machines with specialized tasks in the head, or figure it out on the fly. Not funny.
Posted Mar 6, 2016 10:04 UTC (Sun)
by hitmark (guest, #34609)
[Link]
Posted Mar 6, 2016 10:06 UTC (Sun)
by hitmark (guest, #34609)
[Link] (2 responses)
Oh so very very much this.
Posted Mar 6, 2016 12:21 UTC (Sun)
by anselm (subscriber, #2796)
[Link] (1 responses)
There seems to be this weird notion around that the systemd developers were somehow able to pressurise everybody into adopting systemd against their better judgement. In actual fact, the mainstream Linux distributions came on board exactly because their developers looked at systemd and decided – in some cases after extensive and painful deliberation – that it was an idea worth pursuing, over the available alternatives. Sounds like “market acceptance of their ideas” to me.
Would it have been possible to beef up SysV init or Upstart to do what systemd does? Perhaps, but in spite of a lot of talk about doing this, nobody volunteered to put in the effort – and in the free-software world, working code beats hypothetical yet-to-be-written code any day.
Posted Mar 6, 2016 15:16 UTC (Sun)
by hummassa (subscriber, #307)
[Link]
Systemd vs. Docker
Red Hat owns and manages the entire project infrastructure from web sites to build system to bug trackers and so on.
Any changes to it will have to be approved by Red Hat and it's employee and more often than not implemented by them as well.
If these are shared infrastructure resources between Fedora and RHEL ( as in not Fedora only like bugzilla ) those changes get rejected.
Red Hat dictates and decides which direction the project is allowed to take both directly and indirectly through it's employee who view and approve any changes that are requested to be made with their RHEL maintenance hat on encase they are involved with that..
Red Hat decides which applications and products that resides within Fedora are "recommended" and considered "default".
etc. etc. etc.
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Red Hat's revenue-generating customers don't use Fedora, they use RHEL.
Exactly. Had RHEL (or broader EL community) users known what systemd was going to be allowed to turn into, and how deeply it'd be integrated into the base OS, and had a say, we'd have knifed it before it started.
And I may be wasting my breath, but I'll also point out that systemd is far more useful on servers than laptops.
Hardly. Anyone who had a need for a service manager in EL-land had already solved that problem using one of the existing ones out there (daemontools, runit, xinetd, etc.), had scripted their own solution, was using script management for verification, or -- "mu" -- didn't want one running because they relied on the monitoring system to catch something being dead and they wanted to go in an investigate.
For servers, 90% of systemd-as-an-init-system solves problems that servers don't have. 90% of systemd-as-everything-else (autofs, crond, ntpd) solves problems that had already been solved.
Basically the only thing that systemd does well that was something that needed to be done better is cgroup management. But those who were using it heavily had often already decided on their own solutions. The churn of dealing with systemd wasn't worth that benefit.
(And, incidentally, RHEL7, with systemd, has now been out for about a year. RH's revenue-generating customers have yet to leave en masse)
Leave? No. Migrate to RHEL7? Anecdotally, there's a LOT of concern. I know a lot of shops that are staying with EL6 for the time being, and it's not just for the normal reason of "letting things shake out for a year". There are serious concerns with systemd's reliability and the philosophy behind it.
The only places that are all-in with systemd from an EL perspective are those that are fully on board with containers. Even then, those who were most in need of stateless solutions (high performance compute clusters) already had solutions for running EL in stateless modes, and deploying / managing their images.
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
[2] http://www.meetup.com/South-Bay-Site-Reliability-Engineer...
[3] https://youtu.be/qJ3CozFrEvg?t=29m9s
[4] https://en.wikipedia.org/wiki/Not_even_wrong
Systemd vs. Docker
This view seems made up to me, and I can speak to why personally.
It's quite possible that we run in different circles, but the view is definitely not "made up". Every EL shop I know of is being more cautious with EL7 than with previous transitions.
Mostly false, at least for the cause/effect you're suggesting. Fedora lost substantial users over the F15-F20 period, but it has more than recovered them since (F21 and F22) [3] without reversing those decisions. If those were the decisions rationally driving users away, they would have stayed away.
Although numbers have recovered, that doesn't really speak to whether the then-users stayed away or not. As focus changes, so does audience. I'd suggest that Fedora users who didn't agree with direction things were going went with EL, or perhaps Debian. e.g., from this wonderful thread: https://lists.fedoraproject.org/pipermail/devel/2011-June/152636.html
What tests could show that systemd is "operationally invisible" to most users running distributions with it?
Well, perhaps we're equivocating on "user" here, a not-uncommon issue. For some distributions, "users" are people relying on the distribution to control apps. For others, "users" are the system administrators who are deeply involved in reliability engineering and may or may not have specific ways of setting up systems for their company's needs.
I'd submit that neither the systemd-as-init nor the systemd-as-collection-of-tools aspects have been anywhere near as "operationally invisible" to sysadmins as the move to upstart was. Depending on the needs of non-sysadmin users who don't know what the /etc/fstab file is, it may have been more "operationally invisible", except for the opportunity cost of all the other people having to deal with systemd bugs over the years.
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
I do believe I'm not misremembering anything but then again this was 5 years ago but as I recall it, the reason systemd got rejected in F14 was lack of proper documentation both upstream ( man pages ) and downstream, though I think I, Rahul Sundaram, Lennart and Matthias Clasen manage to take care of most that before F14 got released, lack of clear packaging guidelines and related upgrade path and hand full of bugs somewhere between 5 - 10 which I think Lennart and Kay had fixed before F14 got released but due to a short timeframe and to much of grey area surrounding this FESCo decided to take the safer route ( defer systemd to the f15 release ) to have those things properly settled out, despite the media hype around that feature ( which the feature process basically is ) that had taken place.
( We are not talking about Anaconda/Gnome here which get rewritten every cycle with let's throw it over the wall and see what leaks mentality ).
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker
Systemd vs. Docker