Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Posted Nov 12, 2019 13:38 UTC (Tue) by tlamp (subscriber, #108540)Parent article: Debian reconsiders init-system diversity
I loathe the time I had to maintain some init scripts my self, or wanted to see what one exactly executed when. Naturally one could write relatively nice and simple ones, but that did not happen often, and it was still a procedural way of describing something which should clearly be described in schema-based configurations.
I'd guess that it's easier to make a backward comaptible switch from a systemd based one to a new, keeping the configs, than it was switching from sysv to systemd.. Albeit, one needs to support quite some more features, or ship parts of systemd until it's fully replaced.
And for me, that is one of the biggest advantages systemd made, not the faster (parallel) boot, or the pile of additional features.. Just the simple config files where I can *describe* my unit's relation and start/stop behavior in a maintainable way.
Posted Nov 12, 2019 16:33 UTC (Tue)
by BirAdam (guest, #132170)
[Link] (40 responses)
That said, most of this is irrelevant to why most people dislike SystemD (ime). The most common arguments I have heard revolve around security, complexity, and Poettering himself. A lesser heard but still somewhat salient argument has been: Re Hat imperialism. I do not personally find the argument compelling, but I do think it's an argument to be heard. The centralization of so much code into the hands of one guy who clearly doesn't value other people, refers to what came before as "child's tinker toys" and doesn't understand why things were built the way they were isn't a good selling point. Nor is it a good selling point to then say that this is software now largely controlled by a single mega corp.
I say that those aren't selling points, because they aren't. The actual selling point to me for SystemD is that I at least now have some standardization across distributions, which has made things "simpler" from a certain point of view. It's "nice" to be able to use journalctl and systemctl and the like, and have things (for the most part) just work. It's just way worse now when it does not "just work".
The older I get the more I realize that early software systems were built around failure. The idea was that software was shit and therefore you had to expect failure and make it as easy as possible to troubleshoot and fix. Newer software seems to be written from the opposite point of view, make it easier for people to work when it is working and hope it never fails or that those responsible for it when it fails are experts.
As to which of those routes is best is probably better answered by the companies and other organizations that will be consuming the product than by those of us in the industry who apparently just enjoy endlessly arguing about this stuff. Even this though is less relevant than it used to be. People are largely trying to move to outsourcing everything to cloud vendors and having completely managed everything. This then involves more automation, less customization, less performance tuning, etc... I can easily see where the operating system and all of its stuff just "doesn't matter" anymore. Oddly enough, this may also mean that in the end, it's busybox's init that becomes the winner, as that's what Alpine uses, and everyone seems to love containers for deployment.
Posted Nov 12, 2019 16:39 UTC (Tue)
by seyman (subscriber, #1172)
[Link] (3 responses)
Given that the two commands do not do the same things, I seriously doubt that people say that.
Posted Nov 12, 2019 16:41 UTC (Tue)
by BirAdam (guest, #132170)
[Link] (2 responses)
Posted Nov 15, 2019 4:15 UTC (Fri)
by jccleaver (guest, #127418)
[Link] (1 responses)
Coming from RedHat-land, that's one thing that's always bugged me about Debian's system. RedHat had long had the chkconfig and service scripts, which were preferred as the interfaces for both starting and stopping services (via script) as well as canonically turning them on and off on boot.
Among other things, having that interface allowed single locations for expanding featuresets, patching in weird stuff, or dealing with security context switches. Way cleaner than tweaking initscript perms by hand or adding "exit 0"'s in the scripts the way some Debian folks do it. (Of course, that's always there as an option during an emergency -- an important benefit over systemd -- but that's not the preferred way of dealing with services.)
Posted Nov 15, 2019 5:21 UTC (Fri)
by anguslees (subscriber, #7131)
[Link]
The difference I guess is that most *humans* on Debian just invoked the init scripts directly.
Posted Nov 12, 2019 17:17 UTC (Tue)
by jkingweb (subscriber, #113039)
[Link]
It's not about systemctl (after all, service xyz <start|stop|reload> has made a lot of what systemctl offers to users easy for years), but rather what drives it. Given a process with no init script and no systemd unit file that I want to run as a service, there's no question which if those two missing pieces is simpler to write myself. It's not surprising people used to init.d find it simpler and easier, but I wonder how many people familiar with no init systems at all find systemd harder to work with.
Posted Nov 12, 2019 18:24 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
Personally I don't care a whit why sysV init was built the way it was, that was 30 years ago and we might possibly have learned a thing or two in the meantime. I care about whether the new kid on the block does what I need it to do in order to make my job easier. Surprise: systemd does that, very well IMHO. A whole lot of small programs which each replicate one single slice of its features (most of what systemd does isn't exactly new …), each of which needs to be hacked into every daemon's init script, do not.
Posted Nov 12, 2019 23:47 UTC (Tue)
by SEJeff (guest, #51588)
[Link]
Or how about when the process started up and daemonized, but didn't write its pid file out properly or the status method is written poorly (vendor software much?) so now the init script can't stop it? Old versions of tomcat used to be very problematic due to this. In the systemd land, it creates a control group for each service. When it is time to see the status or to stop a service, systemd simply terminates all pids in that control group, a feature native to the kernel. There are no bugs in this because the kernel keeps track of the pids in a given control group, unlik the pid file under /var/run with the wrong pid in it in the olden days.
The restart bits are also super nice, but many other non-sysvinit systems have this. There are so many areas as a systems engineer who has managed tens of thousands of bare metal linux servers over my career that systemd has made it easier. It has caused some problems, it has added some bugs, but overall, it is a step in the right direction. Despite comments like yours, systemd "won" because it is a technically better solution and absolutely no one has written something better to replace it with.
Posted Nov 13, 2019 3:02 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (4 responses)
First, I doubt the claim that Poettering doesn't understand why things were the way they were with init systems. And even if so, why would that be a reason to *not* do systemd? Sure, things may have been fast and loose before, but we're in a different time now. I like reliable service process management. I like being able to not have a long-running email sync step on itself because cron is too dumb to know the previous instance is still running (yeah, lock files, but now I have two problems; OnUnitInactiveSec=15m that knows when I manually checked too is just too damn convenient).
As an analogy, sure, I can understand why C is everywhere the past 50 years. I can see why it can still be useful today. But would I write a file format parser in C today? Hell no. I know myself better than that. Sure, choosing something like Rust loses you things like more obscure architecture support with the current state of the toolchain, but I don't have access to test on those things *anyways* and any C code I'd be writing would be "eh, hope they're normal" hopes and prayers. And C is the kind of language that does not answer kindly to hopes and prayers.
Posted Nov 26, 2019 12:06 UTC (Tue)
by immibis (subscriber, #105511)
[Link] (3 responses)
Posted Nov 26, 2019 13:55 UTC (Tue)
by farnz (subscriber, #17727)
[Link] (2 responses)
Yes. After all, we're going from one monopoly (SysV style init), to another (systemd style init), with a small phase in the middle where distros used multiple new inits (upstart, openrc, systemd) in SysV style compatibility mode instead of their native modes.
It's not like SysV style init was something with multiple compatible implementations for most of its life, after all - why is the SysV monopolization good, while the newer styles are bad?
Posted Nov 27, 2019 17:09 UTC (Wed)
by jwilk (subscriber, #63328)
[Link] (1 responses)
https://sources.debian.org/src/debian-policy/3.1.1.1/poli...
Posted Nov 27, 2019 17:33 UTC (Wed)
by farnz (subscriber, #17727)
[Link]
That's not what that document says, though - it says that SysV init supports two different ways to manage initscripts, and they are only documenting one of the two.
Posted Nov 13, 2019 4:30 UTC (Wed)
by tau (subscriber, #79651)
[Link]
I dunno, I get the opposite impression. Old UNIX systems and mainframes were built to try to deal with every possible failure case, likely because any sort of useful hardware had to be big and expensive back then. Instances were unique and extremely precious. Decade-long uptimes were considered something to be proud of. If the hardware or software failed anyway, well, restore from backup. Hopefully that too is a contingency plan that you've tested recently.
Modern systems look like Kubernetes. Which is also a horribly complex beast, but in different ways. Everything is cheap, redundant, and disposable. You know you can deal with failure because it is constantly happening. Distributed state is managed by consensus, network topology is virtualized, program instances are redundant. It is still possible to induce a system-wide failure severe enough to jeopardize the whole system though, and good luck quickly getting to the bottom of what's going on when that happens.
Posted Nov 13, 2019 5:34 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
I recently had a misfortune to debug a (virtualized) system that depended on a daemon taking at least 1 second to start. It was not a problem on 90-s era hardware, but a virtual system suddenly can start it in less than a tenth of a second.
Early Unix systems were crap. At least all Unix and BSD-based systems, I don't have much experience with proprietary ones. They depended on the system being static, on launch timings remaining stable and on tons of other things.
These days we're moving away from this. Today's software is designed to be built from smaller modules that try to encode their assumptions explicitly and deal with the dynamic environment (with all of its failures). Heck, we are even finally getting a reliable process management API in Linux!
Posted Nov 13, 2019 13:40 UTC (Wed)
by pizza (subscriber, #46)
[Link]
Relatedly, a sizeable portion of the "systemd caused failure in X/Y/Z" reports were actually due to systemd properly checking return codes and statuses instead of blindly carrying on. In other words, the old scripts were already broken, but just happened to function well enough (except of course when they randomly didn't) for most folks to not notice.
Meanwhile, those "failures" were actually much easier to diagnose and fix than they would have been before, because systemd also captured *all* stdout/stderr/syslog into a single log stream and made it easy to isolate the messages (via systemctl status or journalctl -u unitname).
Posted Nov 13, 2019 8:18 UTC (Wed)
by pixelpapst (guest, #55301)
[Link] (2 responses)
Wow, I'm amazed at how well you too made this exact same point in my head with the above commend. Because I too grew up with init-scripts, and in my circles, *the* canonical way to disable services always was stopping the service, then editing the init file and inserting an "exit 0" at the appropriate place, ideally with a comment on the reasoning ¹. Which, of course, is not declarative at all.
Up until reading your comment just now, I had never even considered using a chmod -x for disabling a service. Or that some other admin might have done so. So yeah, much better indeed.
(I'm tempted to insert some rambling on why you'd want a failure to be logged in this situation etc., but the main point is made so well, I'll leave it as is.)
¹ Plus Debian's way of some services being disableable in /etc/default/xyz.
Posted Nov 14, 2019 17:38 UTC (Thu)
by derobert (subscriber, #89569)
[Link]
Don't forget the third way of disabling a service — change around the symlinks in /etc/rc?.d so that they're K* instead of S* symlinks (update-rc.d and several other programs could do this for you on Debian, RedHat had its own programs to do it, so did everyone else.). And the fourth, chmod -x /usr/sbin/daemon ... which thankfully I've only seen on other people's systems :-/ systemd thankfully only has two ways (disable and mask). And systemctl status will tell you about both.
Posted Nov 19, 2019 16:36 UTC (Tue)
by jccleaver (guest, #127418)
[Link]
On RedHat/Fedora/RHEL/CentOS/Scientific Linux all of those other mechanisms were deprecated LONG ago, and you were supposed to use the /sbin/service and /sbin/chkconfig commands for a) controlling current state, and b) controlling boot and S/K## ordering. Initscripts were supposed to be immutable (replaced on update) and all config happened in /etc/sysconfig/<scriptname> or /etc/chkconfig.d/<scriptname>, which were treated as config files and remained on update. I'm not sure when these were introduced but it's got to have been at least 20 years now... Certainly there by the Cartman RedHat Linux release.
All these other mechanisms for doing something, other than as a 3am hack, strike me as incredibly dirty. Can't believe other ecosystems hadn't fixed it by now.
Posted Nov 13, 2019 13:06 UTC (Wed)
by anselm (subscriber, #2796)
[Link] (14 responses)
The sad fact is that there really is no deep philosophical “why” there to understand. The traditional init infrastructure was basically cobbled together out of various pieces that happened to float around the community at the time – the init system itself was cribbed from System V, inetd was adopted from BSD, and so on. There simply was no common overarching design to the whole setup, so it is no surprise that the result is a big mess where no two configuration files have the same syntax, there are four or five unrelated ways in which service processes are started, etc. Contrast this to systemd which for its faults at least has a reasonably unified design and consistent features across the board.
Of course if you've grown up with the big traditional mess it's possible that you're inured to it to a point where you don't notice that it is actually a big mess. But if you're new to Linux then systemd is a lot easier to teach and learn (I used to teach Linux administration for a living so I know what I'm talking about; among my students I have yet to meet anyone who when presented with both the traditional setup and systemd didn't prefer systemd).
If I was at all cynical I'd say that the people who have sunk a lot of time into understanding the traditional system hate systemd mostly because all the cruft associated with the traditional setup goes away. You used to have to work really hard and accept a lot of odd things on faith in order to be admitted to the priesthood and now suddenly it's all reasonably straightforward, internally consistent, well-documented, and easily maintainable? Can't have that.
Posted Nov 13, 2019 13:28 UTC (Wed)
by smurf (subscriber, #17840)
[Link]
There's nothing cynical about that, it's basic human nature to act that way.
The sad part is that it happens on a subconscious level, thus the inability of most people to recognize this mechanism in themselves.
Posted Nov 13, 2019 13:46 UTC (Wed)
by pizza (subscriber, #46)
[Link] (5 responses)
Actually, there is -- the UNIX philosophy is first and foremost about simplicity of implementation, at the potential expense of performance, correctness, consistency, and of course, general user friendliness.
(In other words, the "Do One Thing" mentality -- Not to be confused with "Do One Thing _Well_", because "Well" typically requires much more complex implementations)
Posted Nov 13, 2019 15:31 UTC (Wed)
by anselm (subscriber, #2796)
[Link] (4 responses)
I personally would be reluctant to use the words “System V init” and “simplicity of implementation” in the same sentence, but of course you're right in principle.
The main point, really, is that the traditional setup was built from pieces that all purported to “do one thing” each, but nobody ever took the time or trouble to ensure that the various things being done are configured sanely, or even that if the same thing is being done in various places it is being done in roughly the same way. (Which of course is an issue that is endemic in Unix – look at the various command-line options for specifying field separator characters in sort, cut, awk, join, …)
By contrast, systemd builds on a common design and infrastructure which ensures, for example, that the same features and mechanisms are available regardless of whether a service process is started by (in terms of the traditional setup) /etc/inittab, an init script, inetd, cron, or at – which is something that the traditional setup does not address at all. In this case, systemd is much more about “do one thing well” than the traditional setup ever was, mostly because it includes library support to do that one thing which is used wherever that one thing needs to be done.
Posted Nov 13, 2019 19:27 UTC (Wed)
by rgmoore (✭ supporter ✭, #75)
[Link] (3 responses)
My impression is that SysV init shows one of the major drawbacks of the Unix philosophy. Unix is supposed to provide a nice toolbox of "do one thing and do it well" programs that let sysadmins hack together quick and dirty solutions to their problems when they need to. Over time those quick and dirty solutions become critical components of the system in their own right, but there's no guideline in the Unix philosophy about when it's time to rewrite the hacked together shell script with a properly written C program. Even worse, as your example of init, at, cron, inetd, etc. shows, there's no guideline about recognizing when there are multiple programs doing related things that need to have their shared functionality merged. The net result is that hacks like SysV init scripts hang around long past the point when they should have been replaced by a better written solution.
Posted Nov 13, 2019 19:48 UTC (Wed)
by anselm (subscriber, #2796)
[Link] (2 responses)
I think this has less to do with the “Unix philosophy” and more with the fact that for a number of its crucial early years Unix wasn't a “product” in the sense that other operating systems are “products”; not unlike Linux in the early 1990s, Unix in the 1970s was a fun side project of a few people that they graciously shared with other interested folks. If you compare Unix with, say, VAX/VMS, which is roughly of the same vintage as Unix and, unlike Unix, used to be an actual supported product, you will find that the good people at Digital Equipment, for example, expended a huge amount of effort to make command-line options align between different commands in a way that the Unix community never managed.
When the “Unix toolbox” was assembled in the mid-to-late 1970s, the guiding principle was more along the lines of “here's another neat program”, people didn't really worry too much about consistency, and there was no product manager, “benevolent dictator”, or written human-interface guide to ensure that command-line options that do the same thing are actually identical between different commands.
Posted Nov 13, 2019 22:27 UTC (Wed)
by rgmoore (✭ supporter ✭, #75)
[Link] (1 responses)
And my gut feeling is that this kind of anarchy has become part of the Unix philosophy. The idea of using small "do one thing and do it well" tools fits in perfectly with disconnected developers working without oversight from a master architect. Maybe you can say that this is part of Unix culture rather than Unix philosophy, but it's clear that separate tools developed by people who aren't always talking to each other is deeply embedded in the way Unix is done.
There's a reason it was Unix, the system initially developed by people having fun in their spare time and writing loosely coupled tools, that was eventually replaced by Free Software rather than one of the top-down designed systems. There's no way the GNU tools could have been developed if it weren't practical to rip individual components out of Unix and replace them with alternative implementations. And things like Linux and systemd show that this is possible even with critical components like the OS kernel and init system.
I think this is an underappreciated part of what people who object to systemd don't like about it. You can argue all you like about the systemd project consisting of a bunch of separate tools that just happen to share a repository, but it is still being developed by a group that coordinates closely and has built its tools to cooperate closely. That's very different from the traditionally anarchic Unix approach, and importantly, that close coordination makes it more difficult for somebody to take just the parts they want and replace the ones they don't.
Posted Nov 13, 2019 23:12 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
I don't think that's actually the case. The various pieces of systemd, their interactions and interdependencies are in fact documented quite well (certainly more so than those of other Unix-based tools). I do agree with Lennart elsewhere in this discussion that it's probably not entirely trivial to “replace the ones they don't want” because there's a real risk of reinventing subtle bugs that the official components have fixed a long time ago, and in practice there's likely not a whole lot to be gained from attempting that sort of thing.
In any case I don't think anyone would have objected if the authors of sort, awk, etc. had at some point got together and decided to standardise the names and meanings of their common command-line options. It would definitely have made Unix stronger. So close coordination is not necessarily a bad thing – as a user and educator I would certainly prefer it over gratuitous diversity.
Posted Nov 14, 2019 1:03 UTC (Thu)
by sorpigal (guest, #36106)
[Link] (6 responses)
Am I alone in thinking that this is a feature and not a bug?
A system cobbled together is malleable (you can always cobble something different.) All of the discussion in comments here about how complicated it is to replace logind with elogind in a compatible way is, to me, evidence that the whole systemd thing isn't "cobbled together from parts" *enough*.
I like the feel of a system made out of parts; I'm also a big fan of legos. Maybe it was unintentional, but the messy system that evolved out of the 80s exhibited behaviors I liked even if there's nothing about the "design" that particularly recommends it. A well designed system which preserved the "just a pile of bricks you can unstick and restick any which way" feel of that would be most appealing.
Posted Nov 14, 2019 8:46 UTC (Thu)
by dvdeug (guest, #10998)
[Link] (2 responses)
You can't unstick basic components from Linux or any other major system and restick them any which way. To replace any basic tool with a fundamentally similar tool is going to find problems where the new tool is not 100% compatible with the original, including bugs. To do something interestingly new is going to require lots and lots of work, and ongoing work if you can't convince the world to switch to your new way.
Posted Nov 18, 2019 17:05 UTC (Mon)
by jezuch (subscriber, #52988)
[Link] (1 responses)
Well, it took Debian literally years to make it possible to use dash as the default /bin/sh instead of bash. There was a huge campaign to rid all the scripts of "bashisms" - in fact I don't even know if it has been completed :)
Oh, and the kfreebsd architecture in Debian... Is not faring well, I think.
Posted Nov 19, 2019 8:50 UTC (Tue)
by tao (subscriber, #17563)
[Link]
Posted Nov 14, 2019 11:50 UTC (Thu)
by tao (subscriber, #17563)
[Link]
The traditional UNIX userspace is more like older style Meccano. You can still fit all the pieces together,
The worst trouble I encounter when building LEGO is typically that a piece I need isn't available in that particular shade (yet).
Having different designers isn't a bad thing if they try for consistency on a UI level. Having to memorise
Posted Nov 18, 2019 17:09 UTC (Mon)
by jezuch (subscriber, #52988)
[Link] (1 responses)
It's also brittle. Undocumented. Not discoverable. It fossilizes quickly and a cargo cult develops around it ;)
Posted Nov 26, 2019 12:08 UTC (Tue)
by immibis (subscriber, #105511)
[Link]
Posted Nov 13, 2019 22:52 UTC (Wed)
by mezcalero (subscriber, #45103)
[Link] (5 responses)
Well, so you know it's one thing telling me I don't understand stuff, but I'd really prefer if you'd not put quotes in my mouth I am pretty sure I never said. I googled for "child's tinker toys" now, because it's not really in my active vocabulary, and I hence was curious where you picked up that supposed quote of mine, and all I found was this LWN comment of yours.
Or should I claim that BirAdama refers to his left hand as "Mr. Noseypokey"?
Thank you,
Lennart
Posted Nov 14, 2019 1:33 UTC (Thu)
by BirAdam (guest, #132170)
[Link] (4 responses)
Posted Nov 14, 2019 8:32 UTC (Thu)
by hunger (subscriber, #36242)
[Link]
Posted Nov 14, 2019 10:31 UTC (Thu)
by edomaur (subscriber, #14520)
[Link]
Posted Nov 14, 2019 13:28 UTC (Thu)
by mezcalero (subscriber, #45103)
[Link] (1 responses)
Posted Nov 14, 2019 14:23 UTC (Thu)
by zdzichu (subscriber, #17118)
[Link]
Now, let's forfeit nitpicking and return to technical discussion.
Posted Nov 15, 2019 5:29 UTC (Fri)
by anguslees (subscriber, #7131)
[Link]
With the old sysvinit way of doing things, you only had a global sort-ordering of scripts, and anything else you had to actually execute the script to discover.
Amongst other things, the config file approach gives you all the information you need to "easily" generate a sysvinit script and ordering (if you wanted to migrate in that direction) - whereas the opposite is basically impossible.
If we ignore all the rest, moving the ecosystem to systemd unit files (or something similar) gives us a much more useful description of our system.
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
No, it was not. If we're talking about initscripts here, then pretty much all early systems had some kinds of race conditions in them for example.
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
doesn't understand why things were built the way they were
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
When the “Unix toolbox” was assembled in the mid-to-late 1970s, the guiding principle was more along the lines of “here's another neat program”, people didn't really worry too much about consistency, and there was no product manager, “benevolent dictator”, or written human-interface guide to ensure that command-line options that do the same thing are actually identical between different commands.
Debian reconsiders init-system diversity
I think this is an underappreciated part of what people who object to systemd don't like about it. You can argue all you like about the systemd project consisting of a bunch of separate tools that just happen to share a repository, but it is still being developed by a group that coordinates closely and has built its tools to cooperate closely. That's very different from the traditionally anarchic Unix approach, and importantly, that close coordination makes it more difficult for somebody to take just the parts they want and replace the ones they don't.
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
There are still a lot of scripts requiring bash, but at least any script using /bin/sh should run equally well (except faster) with /bin/sh pointing to /bin/dash.
Debian reconsiders init-system diversity
but sometimes you need to do a bit of bending to get things to fit. An even better simile might be Geomags.
At first everything fits together nicely, but eventually you either end up in a situation where you cannot quite
achieve the angles you want without horrible kludges, or situations where your design collapses under its own weight.
whether I get help from a binary using "-?" "-h", "--help", "help", "--usage", "-u", "-usage", "-help", "/h" or whatever
else just scratches the surface of frustrating. Some people not even bothering to write manual pages for their software
(hi GNU!) makes things even worse.
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity
Debian reconsiders init-system diversity