On the sickness of our community

Posted Oct 9, 2014 23:59 UTC (Thu) by anselm (subscriber, #2796)
In reply to: On the sickness of our community by dag-
Parent article: On the sickness of our community

Lennart Poettering is one of those rare people who aren't afraid to tackle difficult problems that most folks are way too scared to touch. It is safe to say that the traditional setup (System-V init and friends), after 30 years or so, is no longer adequate – a situation that is usually addressed with the equivalent of band aid and baling wire, and that has been waiting for a long time for somebody to do a radical rethink (incidentally something that most other Unix versions, to various degrees, have done ages ago).

The problem with paradigm shifts of this sort is that there are usually lots of people who are heavily invested in the status quo and hence unwilling to consider alternatives even if they are technically superior. The problems start when (a) most major distributions notice that something like systemd is a good idea and start incorporating it in their setups, and (b) the inventors of the new paradigm are convinced they're doing the Right Thing and don't want to bother with people who disagree, especially when they disagree without proposing compromise solutions (preferably with code). (a) means that people will feel “forced” into using the new software simply by virtue of the fact that their favourite distribution made a (hopefully well-considered) decision to adopt it. They could move over to a different distribution but that would mean work. (b) means that, together with people who bitch and moan as a matter of principle, people with legitimate concerns who did take the trouble to see what the new software would do (or not do) for them can be left out in the cold.

It is generally reasonable advice for people in charge of a free-software project to have meaningful discussions with users who do have legitimate concerns. (There can be little meaningful discussion with people whose main contribution is “This software sucks because I don't like it – even though I never looked at it in detail –, and its developers should go jump in a lake”.) This does not mean that one must bend over backwards to accommodate everybody and their pet problem, but that people on both sides of such a discourse should come away from it with an awareness of each other's point of view and the reasons for it, and a reasonable picture of how to proceed (or a rationale why not to proceed, as the case may be). We have lots of open-source software projects where that sort of approach seems to generally work, and in the long run this will accomplish a lot more than people calling one another “assholes” or collecting money to have somebody killed.

Personally I'm convinced that something like systemd is a wonderful idea in principle. It has great potential to standardise various aspects of Linux that have long been neglected and today are notoriously disparate between distributions. With a software project of systemd's scope, there are bound to be dark corners and places where the initial solutions aren't quite right, and that can prompt people to reject the idea as a whole, which is shortsighted because such issues can be identified and fixed. It would be best if all the name-calling could stop and we could all work together to make systemd into something that the Linux community can stand behind for the next 30 years or so.

to post comments

On the sickness of our community

Posted Oct 10, 2014 17:44 UTC (Fri) by Jandar (subscriber, #85683) [Link] (1 responses)

> With a software project of systemd's scope, there are bound to be dark corners and places where the initial solutions aren't quite right, and that can prompt people to reject the idea as a whole, which is shortsighted because such issues can be identified and fixed.

Systems's scope is the main problem for many people criticizing it. If systemd's developer would constrain it to an excellent init and service {starter,monitor} most complaints would go away. Why does it have to have a so large scope that in this bundle of solutions (to problems many don't have) there are dark corners and places where the initial solutions aren't quite right?

On the sickness of our community

Posted Oct 12, 2014 6:10 UTC (Sun) by agrover (guest, #55381) [Link]

Sometimes when you're writing code and you do something new, and a whole set of assumptions that were baked into the old code no longer make sense. I think this has happened repeatedly: one, because init and startup is so central to how the overall system works, and two, because there is a huge amount of baggage that has built up over Unix's lifetime, actually not baggage but just no-longer-correct assumptions about amount of needed flexibility or about the resources on a system. The initial systemd fixed one, but there were a LOT more (and still are).

So many things we hold as foundational about Linux/Unix are actually accidents, or bugs-that-became-features. (see /usr -> /home transition, and dotfiles not showing up in ls). The reason that distros keep adopting it is even as new and possibly-buggy-in-corners, it does more, in less code and complexity overall than what it replaces, and if a problem is found it's going to be fixed.

On the sickness of our community

Posted Oct 14, 2014 4:29 UTC (Tue) by ThinkRob (guest, #64513) [Link] (12 responses)

Personally I have very few problems with systemd as a piece of software. It seems quite well thought out, and while there are certainly plenty of areas where it drives me nuts, the ratio of awesome:annoying is a) within the bounds of my tolerance b) about the same as plenty of other stuff that I consider good.

Where I do have a problem is with systemd as a project, particularly with some of the leaders' behavior. The "debug" flag drama was an example of the sort of behavior that I find rather stupid, but more bothersome was the fact that many people from the systemd community (cabal? ;) I'm not sure what the right word is here... camp, maybe?) seemed to see absolutely nothing wrong, and were seemingly confused why Linus would react the way he did. That worries me, as it's indicative of either a rather pervasive arrogance or a rather poor display of leadership from people who should be in a position to ensure that the project "plays nice with others". Neither one seems particularly healthy from a project that the community is supposed to base, well, basically everything on.

On the sickness of our community

Posted Oct 14, 2014 7:55 UTC (Tue) by johannbg (guest, #65743) [Link] (11 responses)

The "debug" flag drama as you like to call it was fault of the kernel community using an generic term for their debug output called "debug" not kdebug or kernel.debug basically some other term that clearly identifies it not being a generic thus being kernel only and theirs alone since others are *also* using the kernel commandline.

Not a single time did Kay ever step out of bounds in response ( nor anyone other from the systemd community including Lennart ) in that bug report but the same cannot be said about the kernel developers Borislav Petkov and Steven Rostedt followed by him posting that to lkml with Linus responding way out of line followed by a media storm and his fans shitting all over that bug report as an result of that.
( Kay wanted to move this to the mailinglists for further discussion as he clearly stated in that report)

And to this day I have yet to see the kernel community have an architectural discussion about those *generic* namespace kernel parameters which are open to misinterpretation which is the underlying problem that caused this to begin with and change the name of *their* parameters and fix *their* workflow accordingly since misinterpretation like this was bound to happen eventually and bound to happen again. ( If it had not been systemd it would have been something else)

Apparently it's much easier for the kernel community to play territorial pissing matches, break stuff by hiding it, shit over people in bug reports and mailinglists then there is to have civil discussion and simple add "kernel." in front of those *generic* namespace kernel parameters and adjust their workflow accordingly, thus eliminate the underlying problem *for good* and the shortcomings in their own design and at the same time follow what they preach about fixing problems where they belong.

On the sickness of our community

Posted Oct 14, 2014 8:07 UTC (Tue) by johill (subscriber, #25196) [Link] (5 responses)

The kernel command line constitutes ABI in a sense, so gratuitously changing it (by either side - renaming on the one hand or actually using it for something it wasn't used before!) cannot be done.

On the sickness of our community

Posted Oct 14, 2014 8:52 UTC (Tue) by anselm (subscriber, #2796) [Link] (4 responses)

It can. It just takes preparation and time. ”debug” on the command line isn't one of the most widely used-by-other-software parts of the kernel UI (not ABI as there's nothing “binary” about it) to begin with. It's probably more feasible to change the behaviour of “debug” than it is to change the behaviour of open(2).

Having said that, the original issue was a misunderstanding about what “debug” on the kernel command line actually means. Some people insist that it applies to the kernel only while others claim it applies to other basic system components like plymouth or systemd, too. Since it had never been documented properly it is difficult to figure out who is right and who is wrong here; it does make some sense to be able to tell early-boot software to log debug messages, and possibly to do so using one convenient command line parameter rather than half a dozen. The actual problem at hand started because systemd contained a faulty assertion that caused it to write loads of stuff to the “dmesg” buffer. That assertion was promptly fixed.

Anyway, one interesting observation is that in the bug report we find

Kay, Please go die in a fire along with Lennart. Your type is the cancer that is killing any semblence of usability Linux once had.

and

Kay and Lennart: please just go away, disappear from the FOSS community, we don't need you and your crap.

and

Hopefully the FOSS community will wake up and eject these fucktards.

with nothing remotely similar being said by systemd developers, who remain commendably calm and try to get the discussion back on a technical track. If that is supposed to be an example of how systemd developers are rude and pushy while kernel developers are polite, sensitive and deferential, then there seems to be a bit of cognitive dissonance going on here.

On the sickness of our community

Posted Oct 14, 2014 9:33 UTC (Tue) by mpr22 (subscriber, #60784) [Link] (2 responses)

The kernel command line is absolutely an ABI; it is used not only by humans, but by other programs (bootloaders write it, and userspace programs running on top of the kernel read it). That one of the defined properties of the kernel command line is that it looks like text does not make it cease to be an ABI, any more than the defined inputs or outputs for various nodes in /proc being formatted text makes those not an ABI.

On the sickness of our community

Posted Oct 14, 2014 10:22 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

Last time I checked there was no stability promises with regards to the kernel commandline ( not sure it ever can be ) so relying on it in any shape or form cannot be trusted thus you are on your own if something breaks and as far as I know things get added and deprecated all the time.

Cleaning this up ( yes it's mess not just the generic part ) takes preparation, time and adoption period ( an time where both the new and the old parameters syntax are valid )

Once the kernel community has established clear and well defined kernel command line naming policy and clean things up accordingly it could well decide to declared it as an "ABI" but now it is an complete utter mess.

As things currently stand you cannot see a difference between an kernel command line parameter that is bios related,kernel related, pci related etc. Things are being separate by an underscore, an dot. an minus. o_O

The old ( none existing ) naming scheme might have worked back in the day when there were just 5 kernel parameters and everybody kept a local copy of the kernel man pages in their back pocket for what these meant and what they where supposed to do and where they belong but this is the 21 century and things have considerably grown since then so an clear policy like bios.<foo>, kernel.<foo> pci.<foo> etc is much needed to clearly define which parameter belongs where and to avoid anykind of misinterpretation of those parameters.

If the kernel community does not want to clean this up it should not insult and complain when stuff breaks as an result of that.

This whole "debug drama" would not have taken place if these things had been thoroughly thought through from the beginning but I guess people had better things to do at that time and now it's being indicated that they cannot fix it as an result of that. <sigh>

On the sickness of our community

Posted Oct 14, 2014 11:18 UTC (Tue) by anselm (subscriber, #2796) [Link]

Last time I checked there was no stability promises with regards to the kernel commandline

I don't think there are formal stability promises for anything in the kernel. It's just that the kernel people have a good track record of not changing things without cause, and they're trying to keep things that way.

The kernel command line is an example of what happens if people get to add stuff in an ad-hoc manner and with little regard to present consistency or compatibility down the road. No wonder nobody knows exactly what the “debug” option is supposed to mean when it was probably added at 2am in the morning someday in 1993 to solve some immediate issue and never actually thought through. As far as kernel commandline parameters go, we're now basically stuck with them in the same way that we're stuck with “cut -d“, “awk -F”, and “sort -t”.

Could the “debug drama” have been handled better by everyone concerned? Sure. But piling all the blame on Kay Sievers doesn't do it for me.

On the sickness of our community

Posted Oct 16, 2014 20:32 UTC (Thu) by flussence (guest, #85566) [Link]

Can you explain the purpose behind the three quotes you've chosen to post in this context, and whether any of those people have any connection whatsoever to either systemd or kernel development?

Without that context, I interpret the second half of that post as a textbook example of strawman fallacy.

On the sickness of our community

Posted Oct 14, 2014 15:48 UTC (Tue) by ThinkRob (guest, #64513) [Link] (4 responses)

My issue wasn't with Kay's wording or anything like that. I didn't see him be anything but civil. It was more the sentiment behind the action that worried me. Let me try to explain:

> And to this day I have yet to see the kernel community have an architectural discussion about those *generic* namespace kernel parameters which are open to misinterpretation which is the underlying problem that caused this to begin with and change the name of *their* parameters and fix *their* workflow accordingly since misinterpretation like this was bound to happen eventually and bound to happen again. ( If it had not been systemd it would have been something else)

Ok, so you want better naming. Fair enough. (Which, BTW, is what ended up happening, with systemd using a more specific name...)

But like with any API that's been there for a while, there is now an established base of code that depends on certain parameters working in a certain way.

This is where the split in ideology happens.

One side says "we should just start doing things 'the right way' and screw whatever breaks, 'cause it was broken anyways". It seems to me that you're clearly of this mindset, as are many in the systemd community. I understand the appeal, and I too used to be of this mindset.

The other side takes the approach of "here is how people expect it to work, and even if it's not ideal we shouldn't just wholesale break all of their expectations just because we had a better idea". This is where Linus seems to fall. I tend to agree with this more nowadays, having been on the receiving end of one too many "your stuff is now broken 'cause you were doing it 'wrong'" changes, particularly ones where "wrong" meant "not the way I want to do it."

The problem with the first is that very few people can agree on what "the right way" is, so that too ends up in a pissing match. Usually the best politician or the strongest ego wins, and continual battling coupled with CADT means that it becomes very hard to have a system that remains "stable" (in the development/administration sense, not the uptime sense) for more than a couple years.

The problem with the second is that if left unchecked, you wind up with Win32.

Both sides need to be aware of the dangers of their default stance. I've seen awareness of that in the Linux kernel dev. community throughout the years (as evidenced by the many long discussions surrounding various large-scale changes, many of which have been covered here on LWN), but as of yet I've seen precious little awareness of the dangers of the first approach in the systemd community. And *that* is what concerns me.

On the sickness of our community

Posted Oct 14, 2014 17:26 UTC (Tue) by johannbg (guest, #65743) [Link] (3 responses)

"One side says "we should just start doing things 'the right way' and screw whatever breaks, 'cause it was broken anyways". It seems to me that you're clearly of this mindset, as are many in the systemd community. I understand the appeal, and I too used to be of this mindset."

Yes I am of the opinion that things that are broken are supposed to be fixed where they are broken not workaround so if that happens to be systemd then it should be fixed in systemd or if it happens to be the kernel it should be fixed in the kernel.

"The problem with the first is that very few people can agree on what "the right way" is, so that too ends up in a pissing match. Usually the best politician or the strongest ego wins, and continual battling coupled with CADT means that it becomes very hard to have a system that remains "stable" (in the development/administration sense, not the uptime sense) for more than a couple years."

I would argue there are only few areas of stability that actually existing in open source software since the governing mentality is "throw it over the wall and see what sticks" Look at Gnome for example I would argue that it has been in beta state since it got introduced in RHL.

With regards to what concerns you I did not manage to follow what you where referring to as first approach and second half hence I could not put into context so you kinda need to spell it out what actually concerns you.

On the sickness of our community

Posted Oct 14, 2014 20:36 UTC (Tue) by viro (subscriber, #7872) [Link] (2 responses)

Bloody wonderful. Do you suggest we do the same with the kernel-to-userland interfaces? You know, on the theory that interface stability is so rare that nobody would care. Because there's a whole lot of ill-designed crap I would love to be rid of - all the *notify stuff, for starters. And cgroup would be much improved by being ripped out and replaced with something saner. And ioctl-based part of socket API is a disgraceful mess - we wouldn't *need* netdev namespaces if it had only been done right back in early 80s. So's sysv IPC interface. Let's kill them, should we? What, tons of userland code would break? Tough, but serves them right for using bad interfaces. Oh, and sysfs? A walking design mistake, with really unpleasant consequences wrt e.g. containers. Let's take it out as well, while we are at it...

Do you really want that? BTW, I'm absolutely sure that glibc people also can provide a not so little list of misfeatures that won't be missed (and screw those who would miss them)...

On the sickness of our community

Posted Oct 14, 2014 21:07 UTC (Tue) by johannbg (guest, #65743) [Link]

I would not say bloody but quite wonderful yes.

If it's broken in the kernel you fix it in the kernel or where else would you have the kernel-to-userland interfaces fixed and the rest you listed there and or wanted to get rid of?

On the sickness of our community

Posted Oct 15, 2014 6:14 UTC (Wed) by tao (subscriber, #17563) [Link]

It certainly *would* be wonderful if that would be at all feasible. Of course it isn't. But trying to clean up a bit in the swamp of boot-time command-line options (which is a mish-mash of options to the kernel, kernel modules, the boot loader and init) is (in my opinion) quite far from stuff like ioctl, cgroups, *notify, etc.

That said, it's Linus's kernel, and if he decides not to namespace the debug option, then I think that's his prerogative.

PS: It would be kind of nice to have a consistent set of "non-legacy" kernel interfaces (such as you allude to) and an option to disable all the legacy stuff. I wonder how much cleaner & safer software written against such an interface would be.

PPS: Yes, glibc would certainly benefit from removing quite a bit of brainfuck and in cases like gets(3) -- which is not deprecated by LSB, obsoleted by POSIX and removed from C11 -- everyone who do use them should indeed be screwed. But again, wholesale crapectomy would of course be infeasible, unless aiming for a "non-legacy" OS.