Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 12:59 UTC (Wed) by deater (subscriber, #11746)
In reply to: Kernel prepatch 4.1-rc1 by johannbg
Parent article: Kernel prepatch 4.1-rc1

> Personally I would not be surprised if Greg would delay the next long
> term release ( 4.2+ ) until kdbus is accepted or the kernel community
> sends a clear cut message that it never will accept it and everybody
> starts deleting their work.

Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

The kdbus people are an impatient lot and a bit rude to be flat out demanding their code gets merged. There have been many more cases of much more worthy code that languished for years.

Has anyone been following the struggle the Raspberry-pi mailbox driver people are having trying to get just a simple niche interface merged?

Does anyone remember the GGI/KGI graphics in kernel decision made by Linus that set Linux graphics back by at least 10 years?

Do you know how many years of trying it took to get a performance counter
interface merged into the kernel?

The kdbus people's best hope is probably to hope Ingo Molnar gets interested and re-implements things from scratch. Then it would probably get merged within the month.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 14:29 UTC (Wed) by johannbg (guest, #65743) [Link] (15 responses)

"Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights."

Why do you think there is anything unusual with certain feature having to have landed in the kernel before that kernel is long time supported?

I would think that corporate are *the* deciding factor which kernel becomes long time supported kernel and those corporate decisions being based on features that will have to exist in that kernel so their needs are met and they can build their business upon it.

The company's that contribute the most to the kernel and rely on the long term kernel releases being the most influenced ones in that decision making as in when all their needs are foreseeable to be met in X kernel release, that kernel release is announce to be the next long time supported kernel release ( to allow for everyone else to try to get their stuff merged in due time ) and I can think of two companies on top of my head that are on the top ten list contributing to the kernel that are waiting for kdbus thus as I said I would not be surprised if the next long time supported kernel release would be delayed until kdbus get's merged ( or it's foreseeable that it never will ).

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 18:39 UTC (Wed) by deater (subscriber, #11746) [Link] (2 responses)

> I can think of two companies on top of my head that are on the top ten
>list contributing to the kernel that are waiting for kdbus thus as I said
>I would not be surprised if the next long time supported kernel release
>would be delayed until kdbus get's merged ( or it's foreseeable that it
> never will ).

I don't think Linux kernel development works the way you think it does.
At least I hope it doesn't.

I work on Linux because in the end the people running the show have mostly good taste, and I can trust them to make good technical decisions even if I sometimes don't agree with them.

If I wanted an OS where "corporate decisions" ruled then there are plenty of fairly awful alternative OSes I could be using.

I think the official rules for what it means to have a "stable" release preclude adding major functionality like kdbus, especially if it's not upstream, so I wouldn't hold your breath on it miraculously appearing in 4.2-stable. I think in that case you'd have to fork the code and call it something other than Linux. I guess you could call it SystemDos.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 21:22 UTC (Wed) by johannbg (guest, #65743) [Link]

"I don't think Linux kernel development works the way you think it does."

Afaik the longterm kernel initiative is born out of corporate kernel maintenances, corporate needs with it's maintainers on corporate payroll while the stable kernel release is indented for community driven,fast moving target audience.

- mainline - development
- stable - fast moving and community driven distributions
- longterm - corporates

Greg can clarify if corporates needs don't steer which kernel gets chosen for longterm release and correct me if I'm wrong but then again I thought there was only two longterm release in circulation at any given time and one stable but listed here [1] are 2 stable and 7 longterm and lwn mentions Greg releasing four stable updates here [2] as opposed to two stable and two longterm so there seems to be atleast four "stable" releases in circulation which may or may not include the longterm ones in that number and perhaps total of nine so I'm probably dead wrong how they are maintaining this these days and way off how many maintained kernels are in circulation...

1. https://www.kernel.org/
2. https://lwn.net/Articles/642388/

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:03 UTC (Wed) by MrWim (subscriber, #47432) [Link]

Greg already released an LTSI kernel with kernel assisted DBus acceleration over two years ago: https://lwn.net/Articles/533534/ , so it would seem quite reasonable to wait for kdbus to be merged or abandoned before making another LTSI kernel release.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:04 UTC (Wed) by mbt (guest, #81044) [Link] (11 responses)

What is the huge rush? Why would corporate contributers like to see kdbus so soon?

As has been pointed out in several comments on this page, there are severe performance problems when using D-Bus as a mechanism to transfer big blobs of data. Also, as well pointed out in other comments, the speed improvement when using kdbus is less than an order of magnitude greater than that when using D-Bus. There are inefficiencies in the code (which could be improved without changing the ABI) and, evidently some arise (see http://lwn.net/Articles/642212/) as a result of having to support certain features. Changing such features would indeed entail an ABI change. That in itself is worrying: Linux makes ABI-stability guarantees, so ripping out a badly done component is a no-go. Also, should there be an improved D-Bus interface with the changed feature set, there would most certainly have to be code in the kernel to translate between old and new kdbus flavors. Can you think of a major commercial OS with this kind of problem?

Much has been made of the ten-year history of D-Bus and of the two years of kdbus development. Do you mean to tell me that over all this time no one has done the needed performance tuning that is magically supposed to happen because the code is in the kernel?

This brings us to what I suspect is the Real Reason for all the rush. The comment about the two kernel-contributing companies that want to see a quick merging of kdbus is a telling one. Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed. So how to take advantage of GPL'ed programs and libraries but by the magic of syscalls. Kdbus is the ticket: the can use it to talk to GPL'ed code without violating the license. They can even use GLibC bindings: GLibC is LGPL.

It is handy thing that so many companies are able to take advantage of Linux and contribute to it. We gain from that, but when people want to play proprietary games, we all lose.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:12 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed.

While DBus-the-daemon is GPL'd, that is not true of many (most?) of the client libraries.

So, please, let's keep this discussion fact-based?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:21 UTC (Wed) by mbt (guest, #81044) [Link] (4 responses)

I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. The performance improvement is the biggest reason people cheer for kdbus, but as Linus and several others have pointed out, the argument is specious. If kdbus gets into the kernel and is still too slow, then what?

Might it be better to use the message bus only for control? The cooperating processes could do handshaking over the message bus in order to set up an IPC mechanism much more performant than the message bus could ever possibly be. (There well **always** be overhead in communicating through the message bus; it's unavoidable!) You might as well stick with D-Bus.

From the D-Bus FAQ (http://dbus.freedesktop.org/doc/dbus-faq.html): The overhead comes from a couple of places; part of it is simply "abstraction penalty" (there are layers of code to support multiple main loops, multiple transport types, security, etc.). Probably the largest part comes from data validation (because the reference implementation does not trust incoming data). It would be simple to add a "no validation" mode, but probably not a good idea all things considered.

There are concerns with ABI stability, interactions with kernel components, and security. I'm left wondering why kdbus is so important. Please give us an answer that is better than "just because." I suspect that the true answer is one that the kdbus proponents would like not to admit.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:57 UTC (Wed) by cesarb (subscriber, #6266) [Link] (3 responses)

> I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. [...] I'm left wondering why kdbus is so important.

I recalled the true reason for kdbus *not* being performance, but I couldn't recall precisely what it was (something related to containers IIRC), so I went back to one of its initial announcements, and here's what I found (https://lwn.net/Articles/580194/):

> Beyond that, credential passing is limited, there are no timestamps on messages, D-Bus is not available at early boot, connections to security frameworks (e.g. SELinux) must happen in user space, and there are race conditions around the activation of services.

This old presentation I found (https://archive.fosdem.org/2014/schedule/event/anatomy_of...) goes into more detail.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:47 UTC (Wed) by dlang (guest, #313) [Link] (2 responses)

kdbus doesn't have any connections to security frameworks (dbus does at this point, so this is actually a regression)

As far as race conditions go, examples please? the one provided on the discussion says that services must not exit if there are any outstanding messages to be delivered to them and that's racy today. This seems like the wrong way to do things (want to prevent a service from updating to a new, more secure config, keep sending it messages)

why can dbus not start from initramfs? we've moved so many functions out of the kernel into initramfs, why is dbus more significant than RAID configuration?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:10 UTC (Thu) by cesarb (subscriber, #6266) [Link] (1 responses)

> As far as race conditions go, examples please?

From what I understood, if I am recalling it correctly, the race condition was with inetd-style activation of dbus services. Something like:

1. The monitor process receives a message for the service, and spawns the service daemon.
2. The monitor process hands over the bus endpoint to the service, so any further messages go to the service daemon.
3. The service has been idle for a while, and decides to exit.
4. The monitor process notices that the service has exited, and starts monitoring the endpoint again.

Between 3 and 4, any messages sent to the service get lost.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:32 UTC (Thu) by dlang (guest, #313) [Link]

if there is guaranteed delivery, why would the routing process not detect
that the message it sent to the service didn't get acknoledged?

what happens if the process dies instead of processing the message?

how does the kernel close this race? the kernel is not going to be starting
services and doing the other work that you describe the monitor process
doing.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:28 UTC (Wed) by dlang (guest, #313) [Link] (4 responses)

Since RedHat doesn't sell desktop systems, why would they care about a _desktop_ buss?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:58 UTC (Wed) by d9 (guest, #102181) [Link] (2 responses)

They use it in init and now want to add it to initramfs, apparently they don't like the scripts they currently have there.

http://www.spinics.net/lists/kernel/msg1977814.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:32 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

Better to view it as an reasonable request from an reasonable initramfs maintainer for many years and provide a link to his explanation of it [1] then to list this as some requirement by Red Hat

1. http://www.spinics.net/lists/kernel/msg1977695.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 22:30 UTC (Thu) by artem (subscriber, #51262) [Link]

Even better to view it via this link: http://thread.gmane.org/gmane.linux.kernel/1930358/focus=...

which shows several responses to that request. If you are really interested, the thread also continues here:

http://thread.gmane.org/gmane.linux.kernel/1940497

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:47 UTC (Thu) by HelloWorld (guest, #56129) [Link]

D-Bus isn't and never was a Desktop (only) bus.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 9:52 UTC (Thu) by nye (subscriber, #51576) [Link] (4 responses)

>> Personally I would not be surprised if Greg would delay the next long
>> term release ( 4.2+ ) until kdbus is accepted or the kernel community
>> sends a clear cut message that it never will accept it and everybody
>> starts deleting their work.

>Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

Could you explain why you think this is a bad thing?

When faced with a fairly major and clearly controversial feature, waiting for the outcome of the discussion before committing to a new many-year support promise seems like good engineering practice, no?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 10:34 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (3 responses)

To me, and maybe to deater as well, the comment made by johannbg (»Personally I would not be surprised…«) read as if he expected Greg to hold the next LTS release hostage and force other kernel developers to make an »either/or« decision. That's something that I would consider a blatant abuse of power, and if that happened then I would very much want to see such a maintainer be removed.

However, I find it highly unlikely if not completely ridiculous to think that Greg would do something like that. Or I may have mis-read johannbg; I often have to re-read what he's written in order to really understand what he means.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 11:16 UTC (Thu) by Thue (guest, #14277) [Link]

I don't have any opinion whether Greg would actually do this. But if it was the case, then it doesn't read like hostage taking to me.

Greg works on what he thinks is worth-while. If he doesn't think a long-term kernel without kdbus is worthwhile, because it lacks a critical feature which will be in 4.3, then he is free to use his time in ways that he considers more productive. That doesn't stop somebody else from maintaining 4.2 as a long-term kernel.

People refusing to do work for you for free is not hostage-taking!

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:10 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

I'm not a native English speaker so my meaning might be a bit off at times or usual or probably most likely always ( feel free to communicate with me in my native language, Icelandic to see if you fair any better ) anyway I dont see how this comes of as an somekind of hostage situation for the longterm kernel release that Greg waits with it's release until an final decision has been made with regards to kdbus as opposed to him simply being reasonable if he considers it a deciding factor.

Those corporates invested in kdbus and dependent on the long term kernel ( some of which are heavily contributing to the kernel ) will have to wait another two years if he doesn't so he has an option of an delay it one development cycle ( 2 - 3 months or even two development cycle ) until that gets hashed out vs having them wait for another two years.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:25 UTC (Thu) by mbunkus (subscriber, #87248) [Link]

I stand corrected, and thanks for the clarification; it's definitely clear to me now what you meant.