Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 13:59 UTC (Tue) by johannbg (guest, #65743)
In reply to: Kernel prepatch 4.1-rc1 by d9
Parent article: Kernel prepatch 4.1-rc1

Interesting seeing kdbus discussion now being derailed into benchmarking mud slide between kernel and userspace. ( Did people suddenly run out of gut feelings, magic eight balls and voodoo dolls? ) I thought the general rule of thumb was to first implement the code then optimize it and arguably kdbus needs to be in place *before* people will but any time and effort into optimize the user-space code itself...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:08 UTC (Tue) by d9 (guest, #102181) [Link] (35 responses)

I'm sure you are right and Linus and the other kernel folks have no clue what they are doing.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:14 UTC (Tue) by johannbg (guest, #65743) [Link] (34 responses)

Indeed [1]

1. http://thread.gmane.org/gmane.linux.kernel/1930358/focus=...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:19 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

Look at Havoc responses in that thread...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:28 UTC (Tue) by d9 (guest, #102181) [Link]

I see. Then let's wait until they figure it out.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:26 UTC (Tue) by d9 (guest, #102181) [Link] (31 responses)

I really tried to work with DBus a few years ago. I abandoned it after a coupe of weeks because it was slow and using a lot of CPU cycles. I remember that most people were suggesting using it only for signaling.

I replaced it with TCP sockets over the loopback interface + some basic serialization of my data types. I tested the same program recently on a laptop and it is moving data at more than 50 Gbps. On a laptop!

In my experience, the main reason a program that moves data around is slow is poor serialization. The next is thrashing the CPU cache. The way to solve the problem is benchmark, profile and once you have solid data that points to your bottlenecks you optimize. You don't move your code into the kernel and hope it solves magically all your problems (it won't).

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 15:46 UTC (Tue) by johannbg (guest, #65743) [Link] (30 responses)

There exist no such thing as perfectly bug free optimized code so there is no point in waiting for those white knights to appear, bringing in that holy grail ( for kdbus or something else ) to be implemented in the kernel.

Nor is there any point in talking about any *new* pony farmed implementation that solves all the worlds problem in this area because people need a backward compatible solution to existing implementation which kdbus already does provide or expect optimisation taking place in the area that kdbus is intended to obsolete.

Linus should just break the status quo , stop the kernel communities theatrics, merge this here and now in 4.1 and allow people to move on since evolution will eventually take care of the rest at some point or another.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 16:53 UTC (Tue) by d9 (guest, #102181) [Link] (4 responses)

> stop the kernel communities theatrics

The theatrics in this thread come from your side. Such as: "mostly philosophical", "benchmarking mud slide", "gut feelings, magic eight balls and voodoo dolls", "white knights", "holy grail", "pony farmed". Is this attitude considered appropriate in the fedora community?

> exist no such thing as perfectly bug free optimized code

No, but I disagree with your statement that it is normal for an implementation which has the main goal of *optimizing* the solution to a problem not to come with reproducible benchmarks and profiling that shows that it is working (or at least that it is feasible). Experienced engineers should know that you *cannot* optimize a piece of code as long as you have not *identified the bottleneck*.

Also it is not correct what you said that a working kernel implementation is needed to check if moving the code to the kernel can lead to a significant improvement. There are plenty ways to check if moving the data through the kernel is the bottleneck of DBus.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 16:58 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (3 responses)

> Is this attitude considered appropriate in the fedora community?

No, and Johann has been warned about this attitude on multiple occasions.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 17:19 UTC (Tue) by d9 (guest, #102181) [Link]

I think I (and others in other threads) hold the blame too. Unfortunately it is easy for these discussions to derail and become confrontational.

But in this case (kdbus) I think the solution is quite simple: with patience, benchmark, profile, share numbers with the community, make improvements, repeat. Whatever decision is taken, I'm sure DBus will be improved.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 20:29 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

After I received threats for my simple work on the systemd integration migration or simple association with Lennart ( take your pick why certain individual sent me those systemd releated hate mails ) and even had Red Hat employees threat me on Red Hat/Fedora events with a clear cut message that I should adhere to the whims of the Red Hat Desktop team or I would be outed from the project ( and I aint talking about the KDE part of that team ) my subtlety went out the window so the part of this behaviour being accepted in the Fedora community then the answer to that is yes if you are a Red Hat employee worst case scenario you get a slap on your wrist by your "manager" however that kernel remark comes directly from individual from the kernel community itself which makes one believe they make decisions based on some black magic these days instead of technical hard facts.

" OTOH I am still convinced kdbus is the wrong solution, for the sole
reason called gut feeling. " <--

1. https://lkml.org/lkml/2015/4/21/120

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 21:12 UTC (Tue) by d9 (guest, #102181) [Link]

That's awful and I'm truly sorry to hear you had to go through that.

Unfortunately it doesn't surprise me that there has been so much tension in the Linux community over the last couple of years (but I didn't know things got so far, that's crazy!). A lot of changes happened quickly and many projects were affected or killed. It's not fun to work on/with something for years (possibly as a volunteer) and see that it all gets flushed down the toilet over night. Even more so when you feel things are forced on you by distros.

Something is clearly wrong with the community, although I'm not sure what. Maybe there's too much money and corporate interests involved.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 8:53 UTC (Wed) by jsanders (subscriber, #69784) [Link] (24 responses)

What's the case for for merging kdbus right now? Why can't the implementation mature after discussion over another few months or a year? I read that there's a use case for improving the speed, but nobody appears to actually require that yet. If there were applications which could only use kdbus, but couldn't use normal dbus due to speed, there would be a stronger case for including it.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 9:37 UTC (Wed) by johannbg (guest, #65743) [Link] (22 responses)

There already has been enough time for discussions and the use case for it is not further delay other related (k)dbus work from people and companies who have been waiting for the past two years for it to be accepted into mainline. ( basically you wont see anything emerge using it or improvements around (k)dbus until it has been accepted. classic chicken and egg situation ).

Personally I would not be surprised if Greg would delay the next long term release ( 4.2+ ) until kdbus is accepted or the kernel community sends a clear cut message that it never will accept it and everybody starts deleting their work.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 12:59 UTC (Wed) by deater (subscriber, #11746) [Link] (21 responses)

> Personally I would not be surprised if Greg would delay the next long
> term release ( 4.2+ ) until kdbus is accepted or the kernel community
> sends a clear cut message that it never will accept it and everybody
> starts deleting their work.

Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

The kdbus people are an impatient lot and a bit rude to be flat out demanding their code gets merged. There have been many more cases of much more worthy code that languished for years.

Has anyone been following the struggle the Raspberry-pi mailbox driver people are having trying to get just a simple niche interface merged?

Does anyone remember the GGI/KGI graphics in kernel decision made by Linus that set Linux graphics back by at least 10 years?

Do you know how many years of trying it took to get a performance counter
interface merged into the kernel?

The kdbus people's best hope is probably to hope Ingo Molnar gets interested and re-implements things from scratch. Then it would probably get merged within the month.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 14:29 UTC (Wed) by johannbg (guest, #65743) [Link] (15 responses)

"Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights."

Why do you think there is anything unusual with certain feature having to have landed in the kernel before that kernel is long time supported?

I would think that corporate are *the* deciding factor which kernel becomes long time supported kernel and those corporate decisions being based on features that will have to exist in that kernel so their needs are met and they can build their business upon it.

The company's that contribute the most to the kernel and rely on the long term kernel releases being the most influenced ones in that decision making as in when all their needs are foreseeable to be met in X kernel release, that kernel release is announce to be the next long time supported kernel release ( to allow for everyone else to try to get their stuff merged in due time ) and I can think of two companies on top of my head that are on the top ten list contributing to the kernel that are waiting for kdbus thus as I said I would not be surprised if the next long time supported kernel release would be delayed until kdbus get's merged ( or it's foreseeable that it never will ).

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 18:39 UTC (Wed) by deater (subscriber, #11746) [Link] (2 responses)

> I can think of two companies on top of my head that are on the top ten
>list contributing to the kernel that are waiting for kdbus thus as I said
>I would not be surprised if the next long time supported kernel release
>would be delayed until kdbus get's merged ( or it's foreseeable that it
> never will ).

I don't think Linux kernel development works the way you think it does.
At least I hope it doesn't.

I work on Linux because in the end the people running the show have mostly good taste, and I can trust them to make good technical decisions even if I sometimes don't agree with them.

If I wanted an OS where "corporate decisions" ruled then there are plenty of fairly awful alternative OSes I could be using.

I think the official rules for what it means to have a "stable" release preclude adding major functionality like kdbus, especially if it's not upstream, so I wouldn't hold your breath on it miraculously appearing in 4.2-stable. I think in that case you'd have to fork the code and call it something other than Linux. I guess you could call it SystemDos.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 21:22 UTC (Wed) by johannbg (guest, #65743) [Link]

"I don't think Linux kernel development works the way you think it does."

Afaik the longterm kernel initiative is born out of corporate kernel maintenances, corporate needs with it's maintainers on corporate payroll while the stable kernel release is indented for community driven,fast moving target audience.

- mainline - development
- stable - fast moving and community driven distributions
- longterm - corporates

Greg can clarify if corporates needs don't steer which kernel gets chosen for longterm release and correct me if I'm wrong but then again I thought there was only two longterm release in circulation at any given time and one stable but listed here [1] are 2 stable and 7 longterm and lwn mentions Greg releasing four stable updates here [2] as opposed to two stable and two longterm so there seems to be atleast four "stable" releases in circulation which may or may not include the longterm ones in that number and perhaps total of nine so I'm probably dead wrong how they are maintaining this these days and way off how many maintained kernels are in circulation...

1. https://www.kernel.org/
2. https://lwn.net/Articles/642388/

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:03 UTC (Wed) by MrWim (subscriber, #47432) [Link]

Greg already released an LTSI kernel with kernel assisted DBus acceleration over two years ago: https://lwn.net/Articles/533534/ , so it would seem quite reasonable to wait for kdbus to be merged or abandoned before making another LTSI kernel release.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:04 UTC (Wed) by mbt (guest, #81044) [Link] (11 responses)

What is the huge rush? Why would corporate contributers like to see kdbus so soon?

As has been pointed out in several comments on this page, there are severe performance problems when using D-Bus as a mechanism to transfer big blobs of data. Also, as well pointed out in other comments, the speed improvement when using kdbus is less than an order of magnitude greater than that when using D-Bus. There are inefficiencies in the code (which could be improved without changing the ABI) and, evidently some arise (see http://lwn.net/Articles/642212/) as a result of having to support certain features. Changing such features would indeed entail an ABI change. That in itself is worrying: Linux makes ABI-stability guarantees, so ripping out a badly done component is a no-go. Also, should there be an improved D-Bus interface with the changed feature set, there would most certainly have to be code in the kernel to translate between old and new kdbus flavors. Can you think of a major commercial OS with this kind of problem?

Much has been made of the ten-year history of D-Bus and of the two years of kdbus development. Do you mean to tell me that over all this time no one has done the needed performance tuning that is magically supposed to happen because the code is in the kernel?

This brings us to what I suspect is the Real Reason for all the rush. The comment about the two kernel-contributing companies that want to see a quick merging of kdbus is a telling one. Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed. So how to take advantage of GPL'ed programs and libraries but by the magic of syscalls. Kdbus is the ticket: the can use it to talk to GPL'ed code without violating the license. They can even use GLibC bindings: GLibC is LGPL.

It is handy thing that so many companies are able to take advantage of Linux and contribute to it. We gain from that, but when people want to play proprietary games, we all lose.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:12 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed.

While DBus-the-daemon is GPL'd, that is not true of many (most?) of the client libraries.

So, please, let's keep this discussion fact-based?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:21 UTC (Wed) by mbt (guest, #81044) [Link] (4 responses)

I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. The performance improvement is the biggest reason people cheer for kdbus, but as Linus and several others have pointed out, the argument is specious. If kdbus gets into the kernel and is still too slow, then what?

Might it be better to use the message bus only for control? The cooperating processes could do handshaking over the message bus in order to set up an IPC mechanism much more performant than the message bus could ever possibly be. (There well **always** be overhead in communicating through the message bus; it's unavoidable!) You might as well stick with D-Bus.

From the D-Bus FAQ (http://dbus.freedesktop.org/doc/dbus-faq.html): The overhead comes from a couple of places; part of it is simply "abstraction penalty" (there are layers of code to support multiple main loops, multiple transport types, security, etc.). Probably the largest part comes from data validation (because the reference implementation does not trust incoming data). It would be simple to add a "no validation" mode, but probably not a good idea all things considered.

There are concerns with ABI stability, interactions with kernel components, and security. I'm left wondering why kdbus is so important. Please give us an answer that is better than "just because." I suspect that the true answer is one that the kdbus proponents would like not to admit.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:57 UTC (Wed) by cesarb (subscriber, #6266) [Link] (3 responses)

> I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. [...] I'm left wondering why kdbus is so important.

I recalled the true reason for kdbus *not* being performance, but I couldn't recall precisely what it was (something related to containers IIRC), so I went back to one of its initial announcements, and here's what I found (https://lwn.net/Articles/580194/):

> Beyond that, credential passing is limited, there are no timestamps on messages, D-Bus is not available at early boot, connections to security frameworks (e.g. SELinux) must happen in user space, and there are race conditions around the activation of services.

This old presentation I found (https://archive.fosdem.org/2014/schedule/event/anatomy_of...) goes into more detail.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:47 UTC (Wed) by dlang (guest, #313) [Link] (2 responses)

kdbus doesn't have any connections to security frameworks (dbus does at this point, so this is actually a regression)

As far as race conditions go, examples please? the one provided on the discussion says that services must not exit if there are any outstanding messages to be delivered to them and that's racy today. This seems like the wrong way to do things (want to prevent a service from updating to a new, more secure config, keep sending it messages)

why can dbus not start from initramfs? we've moved so many functions out of the kernel into initramfs, why is dbus more significant than RAID configuration?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:10 UTC (Thu) by cesarb (subscriber, #6266) [Link] (1 responses)

> As far as race conditions go, examples please?

From what I understood, if I am recalling it correctly, the race condition was with inetd-style activation of dbus services. Something like:

1. The monitor process receives a message for the service, and spawns the service daemon.
2. The monitor process hands over the bus endpoint to the service, so any further messages go to the service daemon.
3. The service has been idle for a while, and decides to exit.
4. The monitor process notices that the service has exited, and starts monitoring the endpoint again.

Between 3 and 4, any messages sent to the service get lost.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:32 UTC (Thu) by dlang (guest, #313) [Link]

if there is guaranteed delivery, why would the routing process not detect
that the message it sent to the service didn't get acknoledged?

what happens if the process dies instead of processing the message?

how does the kernel close this race? the kernel is not going to be starting
services and doing the other work that you describe the monitor process
doing.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:28 UTC (Wed) by dlang (guest, #313) [Link] (4 responses)

Since RedHat doesn't sell desktop systems, why would they care about a _desktop_ buss?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:58 UTC (Wed) by d9 (guest, #102181) [Link] (2 responses)

They use it in init and now want to add it to initramfs, apparently they don't like the scripts they currently have there.

http://www.spinics.net/lists/kernel/msg1977814.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:32 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

Better to view it as an reasonable request from an reasonable initramfs maintainer for many years and provide a link to his explanation of it [1] then to list this as some requirement by Red Hat

1. http://www.spinics.net/lists/kernel/msg1977695.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 22:30 UTC (Thu) by artem (subscriber, #51262) [Link]

Even better to view it via this link: http://thread.gmane.org/gmane.linux.kernel/1930358/focus=...

which shows several responses to that request. If you are really interested, the thread also continues here:

http://thread.gmane.org/gmane.linux.kernel/1940497

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:47 UTC (Thu) by HelloWorld (guest, #56129) [Link]

D-Bus isn't and never was a Desktop (only) bus.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 9:52 UTC (Thu) by nye (subscriber, #51576) [Link] (4 responses)

>> Personally I would not be surprised if Greg would delay the next long
>> term release ( 4.2+ ) until kdbus is accepted or the kernel community
>> sends a clear cut message that it never will accept it and everybody
>> starts deleting their work.

>Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

Could you explain why you think this is a bad thing?

When faced with a fairly major and clearly controversial feature, waiting for the outcome of the discussion before committing to a new many-year support promise seems like good engineering practice, no?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 10:34 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (3 responses)

To me, and maybe to deater as well, the comment made by johannbg (»Personally I would not be surprised…«) read as if he expected Greg to hold the next LTS release hostage and force other kernel developers to make an »either/or« decision. That's something that I would consider a blatant abuse of power, and if that happened then I would very much want to see such a maintainer be removed.

However, I find it highly unlikely if not completely ridiculous to think that Greg would do something like that. Or I may have mis-read johannbg; I often have to re-read what he's written in order to really understand what he means.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 11:16 UTC (Thu) by Thue (guest, #14277) [Link]

I don't have any opinion whether Greg would actually do this. But if it was the case, then it doesn't read like hostage taking to me.

Greg works on what he thinks is worth-while. If he doesn't think a long-term kernel without kdbus is worthwhile, because it lacks a critical feature which will be in 4.3, then he is free to use his time in ways that he considers more productive. That doesn't stop somebody else from maintaining 4.2 as a long-term kernel.

People refusing to do work for you for free is not hostage-taking!

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:10 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

I'm not a native English speaker so my meaning might be a bit off at times or usual or probably most likely always ( feel free to communicate with me in my native language, Icelandic to see if you fair any better ) anyway I dont see how this comes of as an somekind of hostage situation for the longterm kernel release that Greg waits with it's release until an final decision has been made with regards to kdbus as opposed to him simply being reasonable if he considers it a deciding factor.

Those corporates invested in kdbus and dependent on the long term kernel ( some of which are heavily contributing to the kernel ) will have to wait another two years if he doesn't so he has an option of an delay it one development cycle ( 2 - 3 months or even two development cycle ) until that gets hashed out vs having them wait for another two years.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:25 UTC (Thu) by mbunkus (subscriber, #87248) [Link]

I stand corrected, and thanks for the clarification; it's definitely clear to me now what you meant.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 9:59 UTC (Wed) by neilbrown (subscriber, #359) [Link]

> What's the case for for merging kdbus right now? Why can't the implementation mature after discussion over another few months or a year?

Most enterprise distros have an "upstream first" policy. So it is really good to get something upstream if you want wide adoption.

I'm sure it would be perfectly OK to merge an imperfect *implementation* and let it mature over the years. The problem is that it is very difficult to merge an imperfect *interface* and let it mature. We aren't really good as maturing interfaces. The "no regressions" policy makes a lot of sense, but it also makes it nearly impossible to experiment with interfaces.

We do merge imperfect interfaces, and live with the consequences. We try really really hard not to merge insecure imperfect interfaces, and that seems to be the main sticking point.