| |||||||||||||
Crowding out OpenBSDCrowding out OpenBSDPosted Nov 14, 2012 1:05 UTC (Wed) by josh (subscriber, #17465)In reply to: Crowding out OpenBSD by rleigh Parent article: Crowding out OpenBSD
Have you looked at the list of Linux features that systemd requires? Have you ever read through that list and given serious consideration to how much work it would require to duplicate those features on another platform? I have, and doing even a fraction of it in a remotely acceptable and maintainable way would easily make for a full-time job. Do you happen to know a half-dozen OpenBSD developers with the time and inclination to do this work, continuously? If so, why shouldn't they work on the corresponding Linux-compatible kernel features instead, so that non-systemd programs can use them too?
I think this article highlighted the critical problem: Linux development has accelerated to the point that some other OSes don't have enough sparse developer time to keep up. Yelling "wait up" doesn't change that, it just slows Linux down so that other OSes have a better chance of keeping up with it.
I consider the BSDs a valuable part of the community, and I'd rather have multiple awesome OSes to work with rather than one awesome OS. However, I'd rather have one awesome OS than multiple less awesome OSes.
(Log in to post comments)
Crowding out OpenBSD Posted Nov 14, 2012 1:15 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
picking on systemd for a moment (hopefully without spawning a 1000 post thread)
Part of the question, even on Linux is if systemd should really REQUIRE all those linux specific interfaces, or merely use them if they are available.
Even on Linux, sometimes people don't want to use cgroups and other things that systemd currently requires.
some of the problem that the BSD folks have is due to them just not having the manpower to keep up, but some of it is also due to some developers using new Linux features with no provision for running on a system without those features (be it Linux or *BSD)
Crowding out OpenBSD Posted Nov 14, 2012 1:41 UTC (Wed) by josh (subscriber, #17465) [Link]
When I said "given serious consideration to how much work it would require to duplicate those features on another platform", that allows for the possibility of providing the same features systemd does without using the platform features it currently requirs. Even allowing for that, I think my comment still stands: doing so would require a prohibitive amount of work.
You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash. Upstart handles that "portably" by ptracing daemons that fork and capturing the relevant PIDs; that approach requires a great deal more code and work than using cgroups, because that approach doesn't rely on a platform feature specifically designed to support it.
Now repeat that exercise for the hundreds of systemd features supported by Linux platform features.
Crowding out OpenBSD Posted Nov 14, 2012 22:01 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
> You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash.
Why does the init system have to do this job?
If a daemon exits or crashes, is it really the right thing to just start a new copy? what damage did the failed instance do that may need to be cleaned up?
If a daemon has a complex set of dependancies (or deliberately does something like a double-fork to distance itself), is it really enough to only consider it 'failed' if they all exit? or should there be something more application aware (i.e. a daemon specific watchdog) that does this?
Crowding out OpenBSD Posted Nov 14, 2012 22:31 UTC (Wed) by raven667 (subscriber, #5198) [Link]
>> You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash.
> Why does the init system have to do this job?
Its in the best position to do this as it's what starts the daemons and has a parent/child relationship with them. Process respawning is also available in SysV init as configured by /etc/inittab but that feature is incomplete.
> If a daemon exits or crashes, is it really the right thing to just start a new copy? what damage did the failed instance do that may need to be cleaned up?
I would think that in the general case the answer is very much YES. If you aren't worried about service outages I suppose you could make failure permanent, maybe wrap the daemon startup in a script which disables the service after the daemon exits. Seems kind of silly though.
> If a daemon has a complex set of dependancies (or deliberately does something like a double-fork to distance itself),
Double forking is more an implementation detail needed by other init systems, you don't need or want to double fork with systemd, although things will still work if you do.
> is it really enough to only consider it 'failed' if they all exit?
I'm not sure what happens if you challenge the init system in this way, what's the Right Thing(tm) to do in this case? If the processes maintain a parent/child relationship then you can rely on signals otherwise if the processes take input you can manage their sockets to detect failures.
> or should there be something more application aware (i.e. a daemon specific watchdog) that does this?
I suppose you could do that too, to handle cases where a process is wedged and needs to be restarted, but you solve 99% of the problem with just basic process monitoring using pretty standard techniques.
Crowding out OpenBSD Posted Nov 14, 2012 22:46 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
> I suppose you could do that too, to handle cases where a process is wedged and needs to be restarted, but you solve 99% of the problem with just basic process monitoring using pretty standard techniques.
heh, I think people would argue that the sysV init solved 99% of the problem with even simpler tools.
I think that the simple case is handled well by SysV init, the really complex cases are not handled well by anything short of application specific watchdogs/monitoring/cleanup tools, so the question is if there is enough of a gap between these two to make the cgroups requirement (and overhead) worthwhile.
I've gotten to the point that everything I build is now either a cluster, or at least potentially part of a cluster, so I look at anything with the question in mind of 'how would I do this if it was part of a cluster', and this mindset really makes me question the value of trying to have the init system be more clever in addressing this problem.
One of the most common problems you run into when dealing with clusters isn't "the application exited or crashed", it's "The application is running, but wedged" I just don't see many cases where the cgroups approach to managing the app would have prevented problems (although, I freely admit that it has the 'cool' factor)
Crowding out OpenBSD Posted Nov 14, 2012 23:44 UTC (Wed) by bronson (subscriber, #4806) [Link]
For every cluster deployed, there must be hundreds or thousands of single hosts deployed to serve up wordpress or static content. Simple stuff written by a weekender, maybe with slashdot effects handled by CloudFlare. I think these guys would find systemd's respawning awesome. Much nicer than configuring and tracking down memory leaks in God, that's for sure! http://godrb.com/
Yes, clusters need more development effort and more powerful tools, but don't let that get in the way of the 80% solution.
Crowding out OpenBSD Posted Nov 15, 2012 1:13 UTC (Thu) by wahern (subscriber, #37304) [Link]
We've already had respawning with inetd. The fact that few used this almost universal facility should make one wonder whether it's really a feature developers don't enjoy reimplementing themselves.
That might seem illogical at first, until you consider how a developer's environment is setup. When developing a daemon you absolutely _hate_ centralized process management, because it tends to get in the way of debugging, among other things. Instead, you basically have to implement some sort of simple process management yourself, regardless of how your production environment will look. (If you're a web developer for a large website, imagine having to develop, debug, test using your process to upload and install in production... it'd be a nightmare.)
I usually start my development on OS X, and then port to Linux and the BSDs. Not once have I considered using launchd.
Crowding out OpenBSD Posted Nov 15, 2012 1:45 UTC (Thu) by josh (subscriber, #17465) [Link]
inetd tied process spawning to sockets. systemd doesn't; you can independently choose whether to use socket activation and whether to use respawning.
Crowding out OpenBSD Posted Nov 15, 2012 1:56 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
inetd tied process spawning to sockets, but there was also inittab for daemons that you wanted to have restarted if they exited or crashed.
Crowding out OpenBSD Posted Nov 15, 2012 3:33 UTC (Thu) by josh (subscriber, #17465) [Link]
True, and it seems quite unfortunate to me that distributions stopped using inittab for its intended purpose and started using it to launch scripts instead.
Crowding out OpenBSD Posted Nov 15, 2012 14:47 UTC (Thu) by khim (subscriber, #9252) [Link] inittab worked as intended couple of decades ago. But when daemons started spawning processes left-and-right and started depend on other processes it become useless. You can view systemd group-using core as "inittab done right".
Crowding out OpenBSD Posted Nov 15, 2012 15:34 UTC (Thu) by raven667 (subscriber, #5198) [Link]
Exactly, I see systemd as a return to unix roots, features that init would have had it continued innovating instead of stagnating and standardizing, not some weird departure against the overall style of the system.
Crowding out OpenBSD Posted Nov 15, 2012 22:04 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
parts of systemd could be called implementing init properly, but other parts are far more questionable.
why does device enumeration belong as part of init (udev)?
why does logging belong as part of init (the journal)?
Crowding out OpenBSD Posted Nov 15, 2012 22:44 UTC (Thu) by raven667 (subscriber, #5198) [Link]
>why does device enumeration belong as part of init (udev)?
It isn't part of PID 1 but hardware initialization is a dependency for services being started. The service manager ends up needing to know about hardware state for dependency purposes and therefore shares a lot of infrastructure with udev which is why they are distributed together now.
> why does logging belong as part of init (the journal)?
This is also not part of PID 1 . Whether system logging should be tackled as part of the systemd project is debatable. It is still an optional component though and might solve some real problems.
Crowding out OpenBSD Posted Nov 15, 2012 22:50 UTC (Thu) by raven667 (subscriber, #5198) [Link]
I thought of one more thing as well, the init system is in a prime position for logging both because it is connected to the stdout/stderr of processes that it spawn, where they might dump data, and it has its own needs for logging but by definition is started before any userspace logging service could be running. So there are at least a couple of cases that a PID 1 and related tools should identify and solve.
Crowding out OpenBSD Posted Nov 15, 2012 14:45 UTC (Thu) by khim (subscriber, #9252) [Link] I've gotten to the point that everything I build is now either a cluster, or at least potentially part of a cluster, so I look at anything with the question in mind of 'how would I do this if it was part of a cluster', and this mindset really makes me question the value of trying to have the init system be more clever in addressing this problem. Funny that you are trying to use cluster-aware approach to oppose cgroups. You do remember that cgroups were invented by Google engineers to manage clusters, right? SystemD just applies the same logic to local system management. One of the most common problems you run into when dealing with clusters isn't "the application exited or crashed", it's "The application is running, but wedged" I just don't see many cases where the cgroups approach to managing the app would have prevented problems (although, I freely admit that it has the 'cool' factor) That's strange because I see tons of places where cgroups solve cluster-management problems. What happens with your cluster if some unimportant task start to eat memory or CPU endlessly? With cgroups answer is obvious: it's bound by cgroup's limit. What happens if some task must be killed and restarted because your node is overloaded? Cgroups make sure you can kill the task reliably. Sure, that means that your tasks must be written to be killable, but it makes sense for clusters anyway since failed PSU can do the same thing suddenly and without warning.
Crowding out OpenBSD Posted Nov 15, 2012 20:53 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
I didn't say that cgroups did not have a place or a reason to use them.
I just said that requiring cgroups to properly start and stop a process is using the wrong tool for the job.
Crowding out OpenBSD Posted Nov 15, 2012 16:34 UTC (Thu) by raven667 (subscriber, #5198) [Link]
I wouldn't call cgroups "cool" but I would say that it's a better solution than what's offered in /etc/init.d/functions and ancillary tools like /sbin/pidof. In fact I recently had an outage due to a SysV script that on restart managed to killall rsyslogd but then only start the one instance causing an outage for the other instance on the same host. That kind of unnecessary outage is depressingly easy to have, we can do better.
Crowding out OpenBSD Posted Nov 22, 2012 14:53 UTC (Thu) by TRauMa (guest, #16483) [Link]
> people would argue that the sysV init solved 99% of the problem
In 15+ years of using linux I never came across a sysv init based system that could reliably restart mysql under all typical production circumstances. Now mysql used (?) to be a close to worst case behaving deamon, but it was also very popular, so if no distributor could handle it correctly with the tools given (and the init scripts tended to be very, very long and complex and hard to debug), how can one argue that it's a 99% solution?
Crowding out OpenBSD Posted Nov 22, 2012 19:21 UTC (Thu) by hummassa (subscriber, #307) [Link]
does systemd (and upstart) actually solve this problem?
Crowding out OpenBSD Posted Nov 22, 2012 19:36 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
Yep. SystemD can reliably track mysql's state and kill it. Mysql recovery should then take care of unclean shutdowns.
Crowding out OpenBSD Posted Nov 22, 2012 21:37 UTC (Thu) by paulj (subscriber, #341) [Link]
How does it reliably track mysqls state btw? What would define a bad state requiring restart?
IME just because a process is running doesn't mean it's actually in a good state. You need some kind of heartbeat protocol. Does systemd implement something like that?
Crowding out OpenBSD Posted Nov 22, 2012 22:45 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
No, the common problem with mysql is that it doesn't terminate cleanly. Its shutdown scripts can hang for minutes and _still_ leave hanging processes behind. These hanging processes later interfere with the startup.
SystemD nicely solves this problem. Hooking it up with an external heartbeat monitor should also be quite easy.
Crowding out OpenBSD Posted Nov 22, 2012 23:25 UTC (Thu) by cortana (subscriber, #24596) [Link]
systemd can be configured to require that a service regularly notifies that it is still up, or else it'll be killed & restarted. See http://0pointer.de/blog/projects/watchdog.html for the details.
Crowding out OpenBSD Posted Nov 15, 2012 1:27 UTC (Thu) by josh (subscriber, #17465) [Link]
> Why does the init system have to do this job?
That tangent misses the original point. Given a piece of software (systemd) that implements quite a few features by using Linux-specific platform functionality, I hinted at the amount of additional work required to replace even one of those features with a slightly more portable alternative. Responding to that with "do you really need that feature" misses the point completely.
Crowding out OpenBSD Posted Nov 14, 2012 1:54 UTC (Wed) by Lennie (subscriber, #49641) [Link]
The creator of systemd specifically made the choice created an architecture leveraging the best parts of the Linux kernel. So the way the program works, depends on the Linux kernel.
It means that systemd should be a fairly small program when compared if it needed the right abstractions and the architecture would probably have been different too.
Wayland is probably the same, I'm sure the code base is very small in comparison to X. For example keyboard handling is mostly handled by the kernel I believe.
I was already wondering how long the Debian project could keep the kFreeBSD port alive. For now in Debian they will probably let the user choose to use systemd or some other init. Maybe systemd will eventually the default with the Linux kernel.
kFreeBSD had ZFS and PF, what other user visible ? Will btrfs fix the biggest feature ?
It used to be true that academia developed new TCP/IP extension on the FreeBSD kernel first, this isn't true anymore. This is now Linux too.
Crowding out OpenBSD Posted Nov 14, 2012 4:23 UTC (Wed) by abartlet (✭ supporter ✭, #3928) [Link]
It reminds me very much of OpenSSH, for which we are very much thankful to the OpenBSD project. It too has a focus on just the bare essentials required to run on one OS - OpenBSD in this case - and a 'portable edition' patch for other platforms.
In my own work keeping Samba portable, we have like most large projects, a portability API (libreplace), mostly trying to emulate the C APIs of other OS versions onto Linux APIs where possible. The Kerberos implementation that we work closely with for the AD DC also has libroken, another portability API, and we link against libbsd for strlcpy etc.
I've done a reasonable amount of work in this area recently, and it is certain the portability doesn't come for free. Indeed, we have two major build systems because (in part) we don't want to drop systems that don't have python!
that smell Posted Nov 14, 2012 5:17 UTC (Wed) by ncm (subscriber, #165) [Link] strlcpy? Please tell me you don't actually use that.
that smell Posted Nov 14, 2012 11:15 UTC (Wed) by abartlet (✭ supporter ✭, #3928) [Link]
We do use strlcpy. We moved from safe_strcpy() and the custom wrappers fstrcpy() and pstrcpy() (for fstrings of 256 and pstrings of 1024 bytes) to at least something others have seen before. Natrually, a function called safe_ wasn't really safe (just safe against memory overwrite), but the name made us feel better about it despite having the same issues as strlcpy().
As code is rewritten, most strings are dynamically allocated talloc buffers, but stragglers remain.
Crowding out OpenBSD Posted Nov 14, 2012 9:43 UTC (Wed) by intgr (subscriber, #39733) [Link]
> two major build systems because (in part) we don't want to drop systems that don't have python!
Huh, there are systems powerful enough to build Samba that don't run Python?
Crowding out OpenBSD Posted Nov 14, 2012 13:22 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]
Yes, old big-iron batshit-crazy OSes in locked rooms without Internet access. Apparantly, Volker Lendecke has to provide support for these kinds of customers.
Crowding out OpenBSD Posted Nov 15, 2012 2:53 UTC (Thu) by nix (subscriber, #2304) [Link] The creator of systemd specifically made the choice created an architecture leveraging the best parts of the Linux kernel.cgroups? Best parts? Only from the perspective that it provides a feature that you can't get any other way. Not from the perspective that it's a crawling horror, nor, indeed that it's considered any good by any of the kernel developers. :)
Crowding out OpenBSD Posted Nov 16, 2012 21:34 UTC (Fri) by Tet (subscriber, #5433) [Link] Not from the perspective that it's a crawling horror, nor, indeed that it's considered any good by any of the kernel developers. :)I've heard that claim many times. But I'm too far removed from what's going on in kernel space these days. So if they're as bad as people are claiming, what alternatives does the kernel provide that give the same functionality as cgroups?
Crowding out OpenBSD Posted Nov 18, 2012 15:39 UTC (Sun) by nix (subscriber, #2304) [Link]
None. That's why they get used, even if they *are* a crawling horror internally.
Crowding out OpenBSD Posted Nov 14, 2012 1:51 UTC (Wed) by clump (subscriber, #27801) [Link] Linux development has accelerated to the point that some other OSes don't have enough sparse developer time to keep up. Yelling "wait up" doesn't change that, it just slows Linux down so that other OSes have a better chance of keeping up with it.Well said. I report a lot of bugs in Linux, and I can certainly say that the ones I report for RHEL are looked at far more often than the ones I report for Fedora. It's certainly no conspiracy, it's a matter of resources. To your point, I think the larger issue is that Linux has taken the dominant free OS role. The article rightly points out that certain projects are less concerned with portability than others, though with the success of Linux it's going to be hard for others to keep up.
|
|||||||||||||