On the sickness of our community

Posted Oct 14, 2014 4:29 UTC (Tue) by ThinkRob (guest, #64513)
In reply to: On the sickness of our community by anselm
Parent article: On the sickness of our community

Personally I have very few problems with systemd as a piece of software. It seems quite well thought out, and while there are certainly plenty of areas where it drives me nuts, the ratio of awesome:annoying is a) within the bounds of my tolerance b) about the same as plenty of other stuff that I consider good.

Where I do have a problem is with systemd as a project, particularly with some of the leaders' behavior. The "debug" flag drama was an example of the sort of behavior that I find rather stupid, but more bothersome was the fact that many people from the systemd community (cabal? ;) I'm not sure what the right word is here... camp, maybe?) seemed to see absolutely nothing wrong, and were seemingly confused why Linus would react the way he did. That worries me, as it's indicative of either a rather pervasive arrogance or a rather poor display of leadership from people who should be in a position to ensure that the project "plays nice with others". Neither one seems particularly healthy from a project that the community is supposed to base, well, basically everything on.

to post comments

On the sickness of our community

Posted Oct 14, 2014 7:55 UTC (Tue) by johannbg (guest, #65743) [Link] (11 responses)

The "debug" flag drama as you like to call it was fault of the kernel community using an generic term for their debug output called "debug" not kdebug or kernel.debug basically some other term that clearly identifies it not being a generic thus being kernel only and theirs alone since others are *also* using the kernel commandline.

Not a single time did Kay ever step out of bounds in response ( nor anyone other from the systemd community including Lennart ) in that bug report but the same cannot be said about the kernel developers Borislav Petkov and Steven Rostedt followed by him posting that to lkml with Linus responding way out of line followed by a media storm and his fans shitting all over that bug report as an result of that.
( Kay wanted to move this to the mailinglists for further discussion as he clearly stated in that report)

And to this day I have yet to see the kernel community have an architectural discussion about those *generic* namespace kernel parameters which are open to misinterpretation which is the underlying problem that caused this to begin with and change the name of *their* parameters and fix *their* workflow accordingly since misinterpretation like this was bound to happen eventually and bound to happen again. ( If it had not been systemd it would have been something else)

Apparently it's much easier for the kernel community to play territorial pissing matches, break stuff by hiding it, shit over people in bug reports and mailinglists then there is to have civil discussion and simple add "kernel." in front of those *generic* namespace kernel parameters and adjust their workflow accordingly, thus eliminate the underlying problem *for good* and the shortcomings in their own design and at the same time follow what they preach about fixing problems where they belong.

On the sickness of our community

Posted Oct 14, 2014 8:07 UTC (Tue) by johill (subscriber, #25196) [Link] (5 responses)

The kernel command line constitutes ABI in a sense, so gratuitously changing it (by either side - renaming on the one hand or actually using it for something it wasn't used before!) cannot be done.

On the sickness of our community

Posted Oct 14, 2014 8:52 UTC (Tue) by anselm (subscriber, #2796) [Link] (4 responses)

It can. It just takes preparation and time. ”debug” on the command line isn't one of the most widely used-by-other-software parts of the kernel UI (not ABI as there's nothing “binary” about it) to begin with. It's probably more feasible to change the behaviour of “debug” than it is to change the behaviour of open(2).

Having said that, the original issue was a misunderstanding about what “debug” on the kernel command line actually means. Some people insist that it applies to the kernel only while others claim it applies to other basic system components like plymouth or systemd, too. Since it had never been documented properly it is difficult to figure out who is right and who is wrong here; it does make some sense to be able to tell early-boot software to log debug messages, and possibly to do so using one convenient command line parameter rather than half a dozen. The actual problem at hand started because systemd contained a faulty assertion that caused it to write loads of stuff to the “dmesg” buffer. That assertion was promptly fixed.

Anyway, one interesting observation is that in the bug report we find

Kay, Please go die in a fire along with Lennart. Your type is the cancer that is killing any semblence of usability Linux once had.

and

Kay and Lennart: please just go away, disappear from the FOSS community, we don't need you and your crap.

and

Hopefully the FOSS community will wake up and eject these fucktards.

with nothing remotely similar being said by systemd developers, who remain commendably calm and try to get the discussion back on a technical track. If that is supposed to be an example of how systemd developers are rude and pushy while kernel developers are polite, sensitive and deferential, then there seems to be a bit of cognitive dissonance going on here.

On the sickness of our community

Posted Oct 14, 2014 9:33 UTC (Tue) by mpr22 (subscriber, #60784) [Link] (2 responses)

The kernel command line is absolutely an ABI; it is used not only by humans, but by other programs (bootloaders write it, and userspace programs running on top of the kernel read it). That one of the defined properties of the kernel command line is that it looks like text does not make it cease to be an ABI, any more than the defined inputs or outputs for various nodes in /proc being formatted text makes those not an ABI.

On the sickness of our community

Posted Oct 14, 2014 10:22 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

Last time I checked there was no stability promises with regards to the kernel commandline ( not sure it ever can be ) so relying on it in any shape or form cannot be trusted thus you are on your own if something breaks and as far as I know things get added and deprecated all the time.

Cleaning this up ( yes it's mess not just the generic part ) takes preparation, time and adoption period ( an time where both the new and the old parameters syntax are valid )

Once the kernel community has established clear and well defined kernel command line naming policy and clean things up accordingly it could well decide to declared it as an "ABI" but now it is an complete utter mess.

As things currently stand you cannot see a difference between an kernel command line parameter that is bios related,kernel related, pci related etc. Things are being separate by an underscore, an dot. an minus. o_O

The old ( none existing ) naming scheme might have worked back in the day when there were just 5 kernel parameters and everybody kept a local copy of the kernel man pages in their back pocket for what these meant and what they where supposed to do and where they belong but this is the 21 century and things have considerably grown since then so an clear policy like bios.<foo>, kernel.<foo> pci.<foo> etc is much needed to clearly define which parameter belongs where and to avoid anykind of misinterpretation of those parameters.

If the kernel community does not want to clean this up it should not insult and complain when stuff breaks as an result of that.

This whole "debug drama" would not have taken place if these things had been thoroughly thought through from the beginning but I guess people had better things to do at that time and now it's being indicated that they cannot fix it as an result of that. <sigh>

On the sickness of our community

Posted Oct 14, 2014 11:18 UTC (Tue) by anselm (subscriber, #2796) [Link]

Last time I checked there was no stability promises with regards to the kernel commandline

I don't think there are formal stability promises for anything in the kernel. It's just that the kernel people have a good track record of not changing things without cause, and they're trying to keep things that way.

The kernel command line is an example of what happens if people get to add stuff in an ad-hoc manner and with little regard to present consistency or compatibility down the road. No wonder nobody knows exactly what the “debug” option is supposed to mean when it was probably added at 2am in the morning someday in 1993 to solve some immediate issue and never actually thought through. As far as kernel commandline parameters go, we're now basically stuck with them in the same way that we're stuck with “cut -d“, “awk -F”, and “sort -t”.

Could the “debug drama” have been handled better by everyone concerned? Sure. But piling all the blame on Kay Sievers doesn't do it for me.

On the sickness of our community

Posted Oct 16, 2014 20:32 UTC (Thu) by flussence (guest, #85566) [Link]

Can you explain the purpose behind the three quotes you've chosen to post in this context, and whether any of those people have any connection whatsoever to either systemd or kernel development?

Without that context, I interpret the second half of that post as a textbook example of strawman fallacy.

On the sickness of our community

Posted Oct 14, 2014 15:48 UTC (Tue) by ThinkRob (guest, #64513) [Link] (4 responses)

My issue wasn't with Kay's wording or anything like that. I didn't see him be anything but civil. It was more the sentiment behind the action that worried me. Let me try to explain:

> And to this day I have yet to see the kernel community have an architectural discussion about those *generic* namespace kernel parameters which are open to misinterpretation which is the underlying problem that caused this to begin with and change the name of *their* parameters and fix *their* workflow accordingly since misinterpretation like this was bound to happen eventually and bound to happen again. ( If it had not been systemd it would have been something else)

Ok, so you want better naming. Fair enough. (Which, BTW, is what ended up happening, with systemd using a more specific name...)

But like with any API that's been there for a while, there is now an established base of code that depends on certain parameters working in a certain way.

This is where the split in ideology happens.

One side says "we should just start doing things 'the right way' and screw whatever breaks, 'cause it was broken anyways". It seems to me that you're clearly of this mindset, as are many in the systemd community. I understand the appeal, and I too used to be of this mindset.

The other side takes the approach of "here is how people expect it to work, and even if it's not ideal we shouldn't just wholesale break all of their expectations just because we had a better idea". This is where Linus seems to fall. I tend to agree with this more nowadays, having been on the receiving end of one too many "your stuff is now broken 'cause you were doing it 'wrong'" changes, particularly ones where "wrong" meant "not the way I want to do it."

The problem with the first is that very few people can agree on what "the right way" is, so that too ends up in a pissing match. Usually the best politician or the strongest ego wins, and continual battling coupled with CADT means that it becomes very hard to have a system that remains "stable" (in the development/administration sense, not the uptime sense) for more than a couple years.

The problem with the second is that if left unchecked, you wind up with Win32.

Both sides need to be aware of the dangers of their default stance. I've seen awareness of that in the Linux kernel dev. community throughout the years (as evidenced by the many long discussions surrounding various large-scale changes, many of which have been covered here on LWN), but as of yet I've seen precious little awareness of the dangers of the first approach in the systemd community. And *that* is what concerns me.

On the sickness of our community

Posted Oct 14, 2014 17:26 UTC (Tue) by johannbg (guest, #65743) [Link] (3 responses)

"One side says "we should just start doing things 'the right way' and screw whatever breaks, 'cause it was broken anyways". It seems to me that you're clearly of this mindset, as are many in the systemd community. I understand the appeal, and I too used to be of this mindset."

Yes I am of the opinion that things that are broken are supposed to be fixed where they are broken not workaround so if that happens to be systemd then it should be fixed in systemd or if it happens to be the kernel it should be fixed in the kernel.

"The problem with the first is that very few people can agree on what "the right way" is, so that too ends up in a pissing match. Usually the best politician or the strongest ego wins, and continual battling coupled with CADT means that it becomes very hard to have a system that remains "stable" (in the development/administration sense, not the uptime sense) for more than a couple years."

I would argue there are only few areas of stability that actually existing in open source software since the governing mentality is "throw it over the wall and see what sticks" Look at Gnome for example I would argue that it has been in beta state since it got introduced in RHL.

With regards to what concerns you I did not manage to follow what you where referring to as first approach and second half hence I could not put into context so you kinda need to spell it out what actually concerns you.

On the sickness of our community

Posted Oct 14, 2014 20:36 UTC (Tue) by viro (subscriber, #7872) [Link] (2 responses)

Bloody wonderful. Do you suggest we do the same with the kernel-to-userland interfaces? You know, on the theory that interface stability is so rare that nobody would care. Because there's a whole lot of ill-designed crap I would love to be rid of - all the *notify stuff, for starters. And cgroup would be much improved by being ripped out and replaced with something saner. And ioctl-based part of socket API is a disgraceful mess - we wouldn't *need* netdev namespaces if it had only been done right back in early 80s. So's sysv IPC interface. Let's kill them, should we? What, tons of userland code would break? Tough, but serves them right for using bad interfaces. Oh, and sysfs? A walking design mistake, with really unpleasant consequences wrt e.g. containers. Let's take it out as well, while we are at it...

Do you really want that? BTW, I'm absolutely sure that glibc people also can provide a not so little list of misfeatures that won't be missed (and screw those who would miss them)...

On the sickness of our community

Posted Oct 14, 2014 21:07 UTC (Tue) by johannbg (guest, #65743) [Link]

I would not say bloody but quite wonderful yes.

If it's broken in the kernel you fix it in the kernel or where else would you have the kernel-to-userland interfaces fixed and the rest you listed there and or wanted to get rid of?

On the sickness of our community

Posted Oct 15, 2014 6:14 UTC (Wed) by tao (subscriber, #17563) [Link]

It certainly *would* be wonderful if that would be at all feasible. Of course it isn't. But trying to clean up a bit in the swamp of boot-time command-line options (which is a mish-mash of options to the kernel, kernel modules, the boot loader and init) is (in my opinion) quite far from stuff like ioctl, cgroups, *notify, etc.

That said, it's Linus's kernel, and if he decides not to namespace the debug option, then I think that's his prerogative.

PS: It would be kind of nice to have a consistent set of "non-legacy" kernel interfaces (such as you allude to) and an option to disable all the legacy stuff. I wonder how much cleaner & safer software written against such an interface would be.

PPS: Yes, glibc would certainly benefit from removing quite a bit of brainfuck and in cases like gets(3) -- which is not deprecated by LSB, obsoleted by POSIX and removed from C11 -- everyone who do use them should indeed be screwed. But again, wholesale crapectomy would of course be infeasible, unless aiming for a "non-legacy" OS.