Kernel prepatch 4.1-rc1

[Posted April 27, 2015 by corbet]

The 4.1-rc1 prepatch is out. Linus says: "No earth-shattering new features come to mind, even if initial support for ACPI on arm64 looks funny. Depending on what you care about, your notion of 'big new feature' may differ from mine, of course. There's a lot of work all over, and some of it might just make a big difference to your use cases." What he doesn't mention is that, in the end, kdbus was not merged for this development cycle.

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 10:28 UTC (Mon) by Thue (guest, #14277) [Link] (57 responses)

> What he doesn't mention is that, in the end, kdbus was not merged for this development cycle.

So a summary if I understand it correctly:
* Some people have argued that kdbus should be replaced by (or build on top of) a wholly new IPC design built from scratch. This seems like a waporware argument, and Linus hasn't mentioned it.
* The other argument against kdbus is that it "leaks" metadata about processes, even though the same metadata can be found today in /proc. Almost everybody have complained about the inclusion of capability metadata, but Linux has said that he is perfectly fine with including capability metadata. While he thinks that the commandline metadata should definitely not be included, because people will probably log it unthinkingly, which will lead to information leaks. [1]

So Linus has the final call, and his objections seem easy to fix (AFAICT). I guess it is possible he could still pull kdbus if removing cmdline from the trivial(?) changes he has requested were made? As I understand it, Linus does sometimes merge features slightly beyond the merge window, as long as the maintainer was diligent about proposing the patches inside the merge window.

In my non-expert estimation, the remaining objections to kdbus seem mostly philosophical, so what would be the point of waiting another patch cyclus?

[1] http://lkml.iu.edu/hypermail/linux/kernel/1504.2/04786.html

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 12:06 UTC (Mon) by karath (subscriber, #19025) [Link] (2 responses)

I believe that there are issues raised by Al Viro with missing documentation of locking and he identified a locking pattern that is likely incorrect.

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 13:05 UTC (Mon) by johannbg (guest, #65743) [Link] (1 responses)

Did not David Herrmann address his concerns already?

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 14:34 UTC (Mon) by dvdhrm (subscriber, #85474) [Link]

Yes, I did.

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 14:42 UTC (Mon) by ebiederm (subscriber, #35028) [Link] (10 responses)

For kdbus to productively move forward people need to stop and really listen to the concerns other people have, and not attempt to make a list of boo-boos to be fixed.

Even if you just concentrate on the boo-boos many of the review efforts of kdbus stopped short because of issues that were seen. There is no reason to presume that deeper looks at kbus will not uncover other unaticipated concerns.

All that is safe to say at this point is the conversation is not complete.

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 14:57 UTC (Mon) by Thue (guest, #14277) [Link] (9 responses)

> really listen to the concerns other people have

Well, I tried to list the concern I am aware of. You seem to think my list is incomplete, so please go ahead and elaborate.

> and not attempt to make a list of boo-boos to be fixed.

So making a list of concrete obvious problems to be fixed is bad thing?

> There is no reason to presume that deeper looks at kbus will not uncover other unaticipated concerns.

IIRC then kdbus has been under development for 2 years now. Of course there could be other issues not found yet, but that is the case with all code in the kernel. People have had plenty of time and opportunity to raise issues already.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 13:44 UTC (Tue) by d9 (guest, #102181) [Link]

> please go ahead and elaborate

You can start here:

http://article.gmane.org/gmane.linux.kernel/1939166
http://article.gmane.org/gmane.linux.kernel/1939022

> kdbus has been under development for 2 years now

It is unfortunate that a lot of time and money was spent on it, but that is a silly reason to force its adoption into the kernel, especially when the main justification for its existence (improving DBus performance) appears to be flawed.

"Premature optimization is the root of all evil." Some people learn it the hard way.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:04 UTC (Tue) by d9 (guest, #102181) [Link] (7 responses)

(sorry for double posting)

Another problem that I have with kdbus is that it is middleware. From what I have seen over the last couple of decades, middleware solutions tend to have a short lifespan. There is always a "new" thing, which is so much "better" than the previous one. But eventually they end up hated and abandoned for various reasons (be it performance, security, reliability, change in requirements -- e.g. connecting applications running on multiple hosts instead of a single host, which DBus/kdbus does not address, and probably never will if it morhps into kdbus).

Why add kdbus into the kernel if 10 years from now nobody will use it?

This reminds me of the database products which around 2000 spent a few years trying to integrate XML into SQL databases. It never really worked well, and by the time they were done, XML had already faded off (it was replaced by other methods of serializing data) and people also shifted interest to distributed databases.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:38 UTC (Tue) by pizza (subscriber, #46) [Link] (4 responses)

> Why add kdbus into the kernel if 10 years from now nobody will use it?

Then it can be marked as deprecated and eventually removed...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 15:55 UTC (Tue) by juhl (guest, #33245) [Link] (3 responses)

Once it is in the kernel it becomes ABI that is more-or-less impossible to remove without breaking userspace.
Yes, you can deprecate it and discourage its use but to actually remove could easily prove impossible or a project with a timespan of 10-20 years - and it still has to be maintained during that time.
So the "we can just deprecate and remove it" argument really doesn't hold. It needs to be really solid before getring merged or we'll have a lot of future pain on our hands.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 16:15 UTC (Tue) by nye (subscriber, #51576) [Link] (2 responses)

>Once it is in the kernel it becomes ABI that is more-or-less impossible to remove without breaking userspace.

That doesn't really apply here, since in this case "remove it" really means "move it back into userspace where it is right now".

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 18:33 UTC (Tue) by juhl (guest, #33245) [Link] (1 responses)

Except you'd still have syscalls, ioctls or other gunk lying around. Once the kernel exposes an interface that userspace can invoke that interface has to stay in place and be functional - even if 99% of everyone move to the userspace interface.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 12:00 UTC (Wed) by nye (subscriber, #51576) [Link]

Oh I see. I was thinking in terms of of consumers of an API that don't care about the implementation details beyond "it does what I need, as quickly as I need", in the same way that when I link against glibc I don't care whether or not it has to do some song and dance to get some function to work on some particular kernel (does that happen? I've never had to care).
But that was rather short-sighted - some software will of course end up calling the kernel's own interfaces directly you would at least need an ABI shim to stay around forever.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 15:20 UTC (Tue) by raven667 (subscriber, #5198) [Link] (1 responses)

It is also useful to point out that dbus has been around for 10+ years already and is seeing more usage, not less, so it's probably going to be around for 10+ years more, it didn't just fall off the turnip truck yesterday and the people who develop it haven't either.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 17:08 UTC (Tue) by d9 (guest, #102181) [Link]

That doesn't invalidate the comparison. XML was designed around 1996 and it was essentially a trimmed down version of SGML, which appeared in the late 80's (in turn derived from GML, created in the 60s). It's much older than DBus.

If DBus appears to be gaining more traction, it's great. I hope whatever solution is chosen is flexible enough to survive a long life.

Kernel prepatch 4.1-rc1

Posted Apr 27, 2015 19:21 UTC (Mon) by roskegg (subscriber, #105) [Link] (42 responses)

The 9P protocol isn't vaporware. It has existed a lot longer than dbus.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 8:26 UTC (Tue) by tao (subscriber, #17563) [Link] (41 responses)

It's vapourware in the sense that it doesn't support D-Bus, which is the whole point here. There are several hundred packages in Debian just using the C-bindings for D-Bus. Then there are bindings for C++, python, java, C#, ADA, GO, QT, glib, Ocaml, ...

Yes, 9P is a nice IPC mechanism. Unless you can figure out some magical way to write a D-Bus daemon on top of 9P it's totally irrelevant in this discussion though. Well, not unless you feel up to the task of submitting patches against a thousand or so applications to replace the D-Bus support with 9P support.

D-Bus isn't ideal. But it's the de facto standard IPC mechanism for desktop applications (and quite a few system level components).

Some of the issues that the current implementation of D-Bus suffers from can be remedied, such as performance issues, race conditions, etc. Handwaving about how there exists other IPC mechanisms that are better isn't productive, nor will it solve any problems.

Whether or not Kdbus is the best possible way to improve D-Bus I cannot say -- it *might* be possible to implement a completely new D-Bus server i user space that gives similar performance, etc. What I do know -- from experience developing commercial products that used D-Bus extensively -- is that the current D-Bus server has a lot of issues and that Kdbus would've solved most (all?) of those.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 13:34 UTC (Tue) by d9 (guest, #102181) [Link] (37 responses)

> Whether or not Kdbus is the best possible way to improve D-Bus I cannot say

It's not. It turns out kdbus offers very little performance improvements because most of the DBus bottlenecks are actually in the userspace libraries, not in data copying.

http://article.gmane.org/gmane.linux.kernel/1939166
http://article.gmane.org/gmane.linux.kernel/1939022

Moreover, the kdbus (and DBus) performance is abysmal. But it's good that the kernel community has looked at this (why the actual DBus devs have not benchamrked & profiled it thoroughly is another issue) because now that it is known where the problems are, they can be fixed properly.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 13:59 UTC (Tue) by johannbg (guest, #65743) [Link] (36 responses)

Interesting seeing kdbus discussion now being derailed into benchmarking mud slide between kernel and userspace. ( Did people suddenly run out of gut feelings, magic eight balls and voodoo dolls? ) I thought the general rule of thumb was to first implement the code then optimize it and arguably kdbus needs to be in place *before* people will but any time and effort into optimize the user-space code itself...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:08 UTC (Tue) by d9 (guest, #102181) [Link] (35 responses)

I'm sure you are right and Linus and the other kernel folks have no clue what they are doing.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:14 UTC (Tue) by johannbg (guest, #65743) [Link] (34 responses)

Indeed [1]

1. http://thread.gmane.org/gmane.linux.kernel/1930358/focus=...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:19 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

Look at Havoc responses in that thread...

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:28 UTC (Tue) by d9 (guest, #102181) [Link]

I see. Then let's wait until they figure it out.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 14:26 UTC (Tue) by d9 (guest, #102181) [Link] (31 responses)

I really tried to work with DBus a few years ago. I abandoned it after a coupe of weeks because it was slow and using a lot of CPU cycles. I remember that most people were suggesting using it only for signaling.

I replaced it with TCP sockets over the loopback interface + some basic serialization of my data types. I tested the same program recently on a laptop and it is moving data at more than 50 Gbps. On a laptop!

In my experience, the main reason a program that moves data around is slow is poor serialization. The next is thrashing the CPU cache. The way to solve the problem is benchmark, profile and once you have solid data that points to your bottlenecks you optimize. You don't move your code into the kernel and hope it solves magically all your problems (it won't).

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 15:46 UTC (Tue) by johannbg (guest, #65743) [Link] (30 responses)

There exist no such thing as perfectly bug free optimized code so there is no point in waiting for those white knights to appear, bringing in that holy grail ( for kdbus or something else ) to be implemented in the kernel.

Nor is there any point in talking about any *new* pony farmed implementation that solves all the worlds problem in this area because people need a backward compatible solution to existing implementation which kdbus already does provide or expect optimisation taking place in the area that kdbus is intended to obsolete.

Linus should just break the status quo , stop the kernel communities theatrics, merge this here and now in 4.1 and allow people to move on since evolution will eventually take care of the rest at some point or another.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 16:53 UTC (Tue) by d9 (guest, #102181) [Link] (4 responses)

> stop the kernel communities theatrics

The theatrics in this thread come from your side. Such as: "mostly philosophical", "benchmarking mud slide", "gut feelings, magic eight balls and voodoo dolls", "white knights", "holy grail", "pony farmed". Is this attitude considered appropriate in the fedora community?

> exist no such thing as perfectly bug free optimized code

No, but I disagree with your statement that it is normal for an implementation which has the main goal of *optimizing* the solution to a problem not to come with reproducible benchmarks and profiling that shows that it is working (or at least that it is feasible). Experienced engineers should know that you *cannot* optimize a piece of code as long as you have not *identified the bottleneck*.

Also it is not correct what you said that a working kernel implementation is needed to check if moving the code to the kernel can lead to a significant improvement. There are plenty ways to check if moving the data through the kernel is the bottleneck of DBus.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 16:58 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (3 responses)

> Is this attitude considered appropriate in the fedora community?

No, and Johann has been warned about this attitude on multiple occasions.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 17:19 UTC (Tue) by d9 (guest, #102181) [Link]

I think I (and others in other threads) hold the blame too. Unfortunately it is easy for these discussions to derail and become confrontational.

But in this case (kdbus) I think the solution is quite simple: with patience, benchmark, profile, share numbers with the community, make improvements, repeat. Whatever decision is taken, I'm sure DBus will be improved.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 20:29 UTC (Tue) by johannbg (guest, #65743) [Link] (1 responses)

After I received threats for my simple work on the systemd integration migration or simple association with Lennart ( take your pick why certain individual sent me those systemd releated hate mails ) and even had Red Hat employees threat me on Red Hat/Fedora events with a clear cut message that I should adhere to the whims of the Red Hat Desktop team or I would be outed from the project ( and I aint talking about the KDE part of that team ) my subtlety went out the window so the part of this behaviour being accepted in the Fedora community then the answer to that is yes if you are a Red Hat employee worst case scenario you get a slap on your wrist by your "manager" however that kernel remark comes directly from individual from the kernel community itself which makes one believe they make decisions based on some black magic these days instead of technical hard facts.

" OTOH I am still convinced kdbus is the wrong solution, for the sole
reason called gut feeling. " <--

1. https://lkml.org/lkml/2015/4/21/120

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 21:12 UTC (Tue) by d9 (guest, #102181) [Link]

That's awful and I'm truly sorry to hear you had to go through that.

Unfortunately it doesn't surprise me that there has been so much tension in the Linux community over the last couple of years (but I didn't know things got so far, that's crazy!). A lot of changes happened quickly and many projects were affected or killed. It's not fun to work on/with something for years (possibly as a volunteer) and see that it all gets flushed down the toilet over night. Even more so when you feel things are forced on you by distros.

Something is clearly wrong with the community, although I'm not sure what. Maybe there's too much money and corporate interests involved.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 8:53 UTC (Wed) by jsanders (subscriber, #69784) [Link] (24 responses)

What's the case for for merging kdbus right now? Why can't the implementation mature after discussion over another few months or a year? I read that there's a use case for improving the speed, but nobody appears to actually require that yet. If there were applications which could only use kdbus, but couldn't use normal dbus due to speed, there would be a stronger case for including it.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 9:37 UTC (Wed) by johannbg (guest, #65743) [Link] (22 responses)

There already has been enough time for discussions and the use case for it is not further delay other related (k)dbus work from people and companies who have been waiting for the past two years for it to be accepted into mainline. ( basically you wont see anything emerge using it or improvements around (k)dbus until it has been accepted. classic chicken and egg situation ).

Personally I would not be surprised if Greg would delay the next long term release ( 4.2+ ) until kdbus is accepted or the kernel community sends a clear cut message that it never will accept it and everybody starts deleting their work.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 12:59 UTC (Wed) by deater (subscriber, #11746) [Link] (21 responses)

> Personally I would not be surprised if Greg would delay the next long
> term release ( 4.2+ ) until kdbus is accepted or the kernel community
> sends a clear cut message that it never will accept it and everybody
> starts deleting their work.

Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

The kdbus people are an impatient lot and a bit rude to be flat out demanding their code gets merged. There have been many more cases of much more worthy code that languished for years.

Has anyone been following the struggle the Raspberry-pi mailbox driver people are having trying to get just a simple niche interface merged?

Does anyone remember the GGI/KGI graphics in kernel decision made by Linus that set Linux graphics back by at least 10 years?

Do you know how many years of trying it took to get a performance counter
interface merged into the kernel?

The kdbus people's best hope is probably to hope Ingo Molnar gets interested and re-implements things from scratch. Then it would probably get merged within the month.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 14:29 UTC (Wed) by johannbg (guest, #65743) [Link] (15 responses)

"Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights."

Why do you think there is anything unusual with certain feature having to have landed in the kernel before that kernel is long time supported?

I would think that corporate are *the* deciding factor which kernel becomes long time supported kernel and those corporate decisions being based on features that will have to exist in that kernel so their needs are met and they can build their business upon it.

The company's that contribute the most to the kernel and rely on the long term kernel releases being the most influenced ones in that decision making as in when all their needs are foreseeable to be met in X kernel release, that kernel release is announce to be the next long time supported kernel release ( to allow for everyone else to try to get their stuff merged in due time ) and I can think of two companies on top of my head that are on the top ten list contributing to the kernel that are waiting for kdbus thus as I said I would not be surprised if the next long time supported kernel release would be delayed until kdbus get's merged ( or it's foreseeable that it never will ).

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 18:39 UTC (Wed) by deater (subscriber, #11746) [Link] (2 responses)

> I can think of two companies on top of my head that are on the top ten
>list contributing to the kernel that are waiting for kdbus thus as I said
>I would not be surprised if the next long time supported kernel release
>would be delayed until kdbus get's merged ( or it's foreseeable that it
> never will ).

I don't think Linux kernel development works the way you think it does.
At least I hope it doesn't.

I work on Linux because in the end the people running the show have mostly good taste, and I can trust them to make good technical decisions even if I sometimes don't agree with them.

If I wanted an OS where "corporate decisions" ruled then there are plenty of fairly awful alternative OSes I could be using.

I think the official rules for what it means to have a "stable" release preclude adding major functionality like kdbus, especially if it's not upstream, so I wouldn't hold your breath on it miraculously appearing in 4.2-stable. I think in that case you'd have to fork the code and call it something other than Linux. I guess you could call it SystemDos.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 21:22 UTC (Wed) by johannbg (guest, #65743) [Link]

"I don't think Linux kernel development works the way you think it does."

Afaik the longterm kernel initiative is born out of corporate kernel maintenances, corporate needs with it's maintainers on corporate payroll while the stable kernel release is indented for community driven,fast moving target audience.

- mainline - development
- stable - fast moving and community driven distributions
- longterm - corporates

Greg can clarify if corporates needs don't steer which kernel gets chosen for longterm release and correct me if I'm wrong but then again I thought there was only two longterm release in circulation at any given time and one stable but listed here [1] are 2 stable and 7 longterm and lwn mentions Greg releasing four stable updates here [2] as opposed to two stable and two longterm so there seems to be atleast four "stable" releases in circulation which may or may not include the longterm ones in that number and perhaps total of nine so I'm probably dead wrong how they are maintaining this these days and way off how many maintained kernels are in circulation...

1. https://www.kernel.org/
2. https://lwn.net/Articles/642388/

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:03 UTC (Wed) by MrWim (subscriber, #47432) [Link]

Greg already released an LTSI kernel with kernel assisted DBus acceleration over two years ago: https://lwn.net/Articles/533534/ , so it would seem quite reasonable to wait for kdbus to be merged or abandoned before making another LTSI kernel release.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:04 UTC (Wed) by mbt (guest, #81044) [Link] (11 responses)

What is the huge rush? Why would corporate contributers like to see kdbus so soon?

As has been pointed out in several comments on this page, there are severe performance problems when using D-Bus as a mechanism to transfer big blobs of data. Also, as well pointed out in other comments, the speed improvement when using kdbus is less than an order of magnitude greater than that when using D-Bus. There are inefficiencies in the code (which could be improved without changing the ABI) and, evidently some arise (see http://lwn.net/Articles/642212/) as a result of having to support certain features. Changing such features would indeed entail an ABI change. That in itself is worrying: Linux makes ABI-stability guarantees, so ripping out a badly done component is a no-go. Also, should there be an improved D-Bus interface with the changed feature set, there would most certainly have to be code in the kernel to translate between old and new kdbus flavors. Can you think of a major commercial OS with this kind of problem?

Much has been made of the ten-year history of D-Bus and of the two years of kdbus development. Do you mean to tell me that over all this time no one has done the needed performance tuning that is magically supposed to happen because the code is in the kernel?

This brings us to what I suspect is the Real Reason for all the rush. The comment about the two kernel-contributing companies that want to see a quick merging of kdbus is a telling one. Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed. So how to take advantage of GPL'ed programs and libraries but by the magic of syscalls. Kdbus is the ticket: the can use it to talk to GPL'ed code without violating the license. They can even use GLibC bindings: GLibC is LGPL.

It is handy thing that so many companies are able to take advantage of Linux and contribute to it. We gain from that, but when people want to play proprietary games, we all lose.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:12 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> Vendors are forbidden from linking non-free programs to GPL-licensed libraries. They might try to get around this by using the message bus to communicate. Ah, but drat! D-Bus is also GPL licensed.

While DBus-the-daemon is GPL'd, that is not true of many (most?) of the client libraries.

So, please, let's keep this discussion fact-based?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:21 UTC (Wed) by mbt (guest, #81044) [Link] (4 responses)

I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. The performance improvement is the biggest reason people cheer for kdbus, but as Linus and several others have pointed out, the argument is specious. If kdbus gets into the kernel and is still too slow, then what?

Might it be better to use the message bus only for control? The cooperating processes could do handshaking over the message bus in order to set up an IPC mechanism much more performant than the message bus could ever possibly be. (There well **always** be overhead in communicating through the message bus; it's unavoidable!) You might as well stick with D-Bus.

From the D-Bus FAQ (http://dbus.freedesktop.org/doc/dbus-faq.html): The overhead comes from a couple of places; part of it is simply "abstraction penalty" (there are layers of code to support multiple main loops, multiple transport types, security, etc.). Probably the largest part comes from data validation (because the reference implementation does not trust incoming data). It would be simple to add a "no validation" mode, but probably not a good idea all things considered.

There are concerns with ABI stability, interactions with kernel components, and security. I'm left wondering why kdbus is so important. Please give us an answer that is better than "just because." I suspect that the true answer is one that the kdbus proponents would like not to admit.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 22:57 UTC (Wed) by cesarb (subscriber, #6266) [Link] (3 responses)

> I am indeed looking to the known facts while trying to uncover what appears to be a hidden agenda. So if D-Bus itself already offers a way to avoid GPL violations, my suggested explanation goes away. That leaves me casting around for other explanations. [...] I'm left wondering why kdbus is so important.

I recalled the true reason for kdbus *not* being performance, but I couldn't recall precisely what it was (something related to containers IIRC), so I went back to one of its initial announcements, and here's what I found (https://lwn.net/Articles/580194/):

> Beyond that, credential passing is limited, there are no timestamps on messages, D-Bus is not available at early boot, connections to security frameworks (e.g. SELinux) must happen in user space, and there are race conditions around the activation of services.

This old presentation I found (https://archive.fosdem.org/2014/schedule/event/anatomy_of...) goes into more detail.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 23:47 UTC (Wed) by dlang (guest, #313) [Link] (2 responses)

kdbus doesn't have any connections to security frameworks (dbus does at this point, so this is actually a regression)

As far as race conditions go, examples please? the one provided on the discussion says that services must not exit if there are any outstanding messages to be delivered to them and that's racy today. This seems like the wrong way to do things (want to prevent a service from updating to a new, more secure config, keep sending it messages)

why can dbus not start from initramfs? we've moved so many functions out of the kernel into initramfs, why is dbus more significant than RAID configuration?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:10 UTC (Thu) by cesarb (subscriber, #6266) [Link] (1 responses)

> As far as race conditions go, examples please?

From what I understood, if I am recalling it correctly, the race condition was with inetd-style activation of dbus services. Something like:

1. The monitor process receives a message for the service, and spawns the service daemon.
2. The monitor process hands over the bus endpoint to the service, so any further messages go to the service daemon.
3. The service has been idle for a while, and decides to exit.
4. The monitor process notices that the service has exited, and starts monitoring the endpoint again.

Between 3 and 4, any messages sent to the service get lost.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 1:32 UTC (Thu) by dlang (guest, #313) [Link]

if there is guaranteed delivery, why would the routing process not detect
that the message it sent to the service didn't get acknoledged?

what happens if the process dies instead of processing the message?

how does the kernel close this race? the kernel is not going to be starting
services and doing the other work that you describe the monitor process
doing.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:28 UTC (Wed) by dlang (guest, #313) [Link] (4 responses)

Since RedHat doesn't sell desktop systems, why would they care about a _desktop_ buss?

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 20:58 UTC (Wed) by d9 (guest, #102181) [Link] (2 responses)

They use it in init and now want to add it to initramfs, apparently they don't like the scripts they currently have there.

http://www.spinics.net/lists/kernel/msg1977814.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:32 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

Better to view it as an reasonable request from an reasonable initramfs maintainer for many years and provide a link to his explanation of it [1] then to list this as some requirement by Red Hat

1. http://www.spinics.net/lists/kernel/msg1977695.html

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 22:30 UTC (Thu) by artem (subscriber, #51262) [Link]

Even better to view it via this link: http://thread.gmane.org/gmane.linux.kernel/1930358/focus=...

which shows several responses to that request. If you are really interested, the thread also continues here:

http://thread.gmane.org/gmane.linux.kernel/1940497

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:47 UTC (Thu) by HelloWorld (guest, #56129) [Link]

D-Bus isn't and never was a Desktop (only) bus.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 9:52 UTC (Thu) by nye (subscriber, #51576) [Link] (4 responses)

>> Personally I would not be surprised if Greg would delay the next long
>> term release ( 4.2+ ) until kdbus is accepted or the kernel community
>> sends a clear cut message that it never will accept it and everybody
>> starts deleting their work.

>Wow, I hope if a maintainer did something like that, that they would quickly be stripped of their maintainer rights.

Could you explain why you think this is a bad thing?

When faced with a fairly major and clearly controversial feature, waiting for the outcome of the discussion before committing to a new many-year support promise seems like good engineering practice, no?

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 10:34 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (3 responses)

To me, and maybe to deater as well, the comment made by johannbg (»Personally I would not be surprised…«) read as if he expected Greg to hold the next LTS release hostage and force other kernel developers to make an »either/or« decision. That's something that I would consider a blatant abuse of power, and if that happened then I would very much want to see such a maintainer be removed.

However, I find it highly unlikely if not completely ridiculous to think that Greg would do something like that. Or I may have mis-read johannbg; I often have to re-read what he's written in order to really understand what he means.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 11:16 UTC (Thu) by Thue (guest, #14277) [Link]

I don't have any opinion whether Greg would actually do this. But if it was the case, then it doesn't read like hostage taking to me.

Greg works on what he thinks is worth-while. If he doesn't think a long-term kernel without kdbus is worthwhile, because it lacks a critical feature which will be in 4.3, then he is free to use his time in ways that he considers more productive. That doesn't stop somebody else from maintaining 4.2 as a long-term kernel.

People refusing to do work for you for free is not hostage-taking!

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:10 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

I'm not a native English speaker so my meaning might be a bit off at times or usual or probably most likely always ( feel free to communicate with me in my native language, Icelandic to see if you fair any better ) anyway I dont see how this comes of as an somekind of hostage situation for the longterm kernel release that Greg waits with it's release until an final decision has been made with regards to kdbus as opposed to him simply being reasonable if he considers it a deciding factor.

Those corporates invested in kdbus and dependent on the long term kernel ( some of which are heavily contributing to the kernel ) will have to wait another two years if he doesn't so he has an option of an delay it one development cycle ( 2 - 3 months or even two development cycle ) until that gets hashed out vs having them wait for another two years.

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 13:25 UTC (Thu) by mbunkus (subscriber, #87248) [Link]

I stand corrected, and thanks for the clarification; it's definitely clear to me now what you meant.

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 9:59 UTC (Wed) by neilbrown (subscriber, #359) [Link]

> What's the case for for merging kdbus right now? Why can't the implementation mature after discussion over another few months or a year?

Most enterprise distros have an "upstream first" policy. So it is really good to get something upstream if you want wide adoption.

I'm sure it would be perfectly OK to merge an imperfect *implementation* and let it mature over the years. The problem is that it is very difficult to merge an imperfect *interface* and let it mature. We aren't really good as maturing interfaces. The "no regressions" policy makes a lot of sense, but it also makes it nearly impossible to experiment with interfaces.

We do merge imperfect interfaces, and live with the consequences. We try really really hard not to merge insecure imperfect interfaces, and that seems to be the main sticking point.

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 17:42 UTC (Tue) by HelloWorld (guest, #56129) [Link] (2 responses)

> What I do know -- from experience developing commercial products that used D-Bus extensively -- is that the current D-Bus server has a lot of issues
That sounds interesting, care to elaborate?

Kernel prepatch 4.1-rc1

Posted Apr 28, 2015 19:23 UTC (Tue) by tao (subscriber, #17563) [Link] (1 responses)

The commercial products in question were Nokia's line of Maemo/Meego products (Nokia 770, N800, N810, N810 WiMAX, N900, N950, N9). The issues were mainly performance issues (and a few of my colleagues analysed these issues quite heavily and we did the best we could to work around or fix the issues).

The fact that we sometimes noticed the D-Bus server not being able to keep up with the traffic generated on a mostly "one app, one user" system is rather depressing.

One of my colleagues even wrote his own proof of concept D-Bus server that was blazingly fast. One of the contributing factors was that it dropped support for some horribly "designed" part of the protocol (if I remember correctly it was something related to arrays, but that was quite a while ago, so my memory might deceive me).

Kernel prepatch 4.1-rc1

Posted Apr 30, 2015 12:17 UTC (Thu) by HelloWorld (guest, #56129) [Link]

> One of the contributing factors was that it dropped support for some horribly "designed" part of the protocol (if I remember correctly it was something related to arrays, but that was quite a while ago, so my memory might deceive me).
This sounds like something that may have been addressed already by the new D-Bus marshalling (GVariant marshalling).

Kernel prepatch 4.1-rc1

Posted Apr 29, 2015 9:53 UTC (Wed) by karath (subscriber, #19025) [Link]

I'm generally in favour of the ideas behind systemd and, more recently, kdbus. I'm sure that some of the opposition to merging kdbus are motivated by resistance to the people (person?) behind systemd. However, watching some of the behaviour of some supporters of the current drive to merge, I see them doing themselves long-term harm.

The more thoughtful opponents to merging kdbus in its current form are raising issues that are not so easily and quickly dismissed. For example, on the issue of the meta-data: Andy Lutomirsky suggests that including capabilities in the metadata is a seriously bad idea while having previously given quite a bit of useful code review. On the other hand, Linus says that having the capabilities is not just a good idea, it is a necessity, but the process command line is a no-no. Additionally, Linus wanted attention paid to the translation of capabilities across user namespace boundaries. Clearly this an area where smart people can reasonably disagree and therefore could take some time to resolve.

Al Viro required documentation about the locking before he felt ready to review the code. Greg Kroah-Hartman acknowledged that this documentation was a valid requirement and, while Dave Hermann provided it, the timing was very late in the cycle. Likely Al had his own set of patches to manage, never mind that quality review may take quite a bit of time.

And now Linus is asking difficult questions about dbus and kdbus performance.

kdbus is not likely to go into the kernel in this release cycle, though Linus could override the convention as he occasionally has done in the past. However, some supporters seem willing to burn the trust and respect that the community has for them in their rush to get the immediate result.