User: Password:
|
|
Subscribe / Log in / New account

Good fences make good projects?

Please consider subscribing to LWN

Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

By Jonathan Corbet
November 9, 2011
Back in August, there was a big fight over whether the user-space "native Linux KVM tool" should be merged into the mainline kernel repository. One development cycle later, we've had the same fight with many of the same arguments and roughly the same result. Sequels are rarely as good as the original; that applies to flame wars as well as to more creative works. But there is a core issue here that has relevance well beyond the kernel community: does the separation of projects help the Linux community more than it hurts it?

The proponents of merging the tool into the kernel make a number of points. Having the projects in the same repository makes development that crosses the boundary between the two easier; in particular, it helps in the creation of APIs that will stand the test of time. The project's overall standards help to keep the quality of the tools high and the release cycle predictable. Reuse of code between user-space and kernel projects gets easier. All told, they say, having the "perf" tool in the kernel tree has greatly helped its development; see this message from Ingo Molnar for a detailed description of the perceived advantages of this mode of development. Artificial separation of projects, instead, is said to have high costs; Ingo went so far as to claim that Linux lost the desktop market as the result of an ill-advised separation of projects.

Opponents, instead, say that putting the kernel and the tools in the same tree makes it easier to create API regressions for out-of-tree tools. The reason that perf has a relatively good record on this front, Ted Ts'o said, has more to do with the competence of the developers involved than its presence in the kernel tree. Adding user-space tools bloats the kernel source distribution, puts competing out-of-tree projects at a disadvantage, and, Ted said, creates a number of difficulties for distributors.

The one concrete end result of the discussion was that the pull request for the KVM tool was passed over by Linus who, feeling that he had enough stuff for this development cycle already, did not want to wander into this particular disagreement. It is not hard to imagine that he will get another chance in a future development cycle; it does not seem that any minds have been changed by the discussion so far.

In the middle of this discussion, it was asked whether it would make sense to bring other projects into the kernel - GNOME, for example. It was pointed out that BSD-based systems tend to be developed in this mode - an existence proof that operating system development can work that way. Ted responded (in the message linked above) as follows:

[T]here has speculation that this was one of many contributions to why they lost out in the popularity and adoption competition with Linux. (Specifically, the reasoning goes that the need to package up the kernel plus userspace meant that we had distributions in the Linux ecosystem, and the competition kept everyone honest. If one distribution started making insane decisions, whether it's forcing Unity on everyone, or forcing GNOME 3 on everyone, it's always possible to switch to another distribution. The *BSD systems didn't have that safety valve....)

One could note that BSD does have one safety valve: to fork the entire system. That has happened a number of times in the history of BSD; pointing this out, though, only serves to reinforce Ted's point.

Distributors play a crucial role in the Linux ecosystem; they function as the middleman between most development projects and their users. Most of us, most of the time, do not obtain the software we run directly from those who wrote it; it comes, instead, nicely packaged from our distributor. As they ponder each package, distributors (the successful ones, at least) will be keeping their users' needs in mind. If the package has obnoxious anti-social features or security problems, the distributors will either fix it or leave the package out altogether. The recent Calibre mess is a prime example; aware distributors had already eliminated the worst problems before they were generally known.

Distributors make it possible to change the source of your operating system without having to stop running Linux. Anybody who has been working with Linux long enough has almost certainly switched distributions at least once during that time; the process is not without its disruptions, but the amount of pain is usually surprisingly low. The lack of lock-in in the Linux world has improved life for users and, at the same time, given distributors an incentive to improve the Linux experience for everybody.

The role of the distributors is made possible by the boundaries between the projects. If the entire system were integrated into a single source tree, there would be little space for the distributors to do their own integration work. The lack of independent *BSD distributions makes this point clear. That suggests that too much integration at the project level might not be a good thing for Linux.

So one could make an argument that bringing GNOME into the kernel source tree is probably a bad idea for this reason alone; Linux as a whole may be better served by having the kernel and the desktop environments be separate components that can be combined (or not) at will. That makes it clear (if it wasn't before - your editor can be slow at times, please bear with him) that there is a line to be drawn somewhere; bringing some projects into the kernel source tree may be harmful for Linux even without considering the effects on the kernel itself. But separating the kernel from some user-space projects may have costs that are just as high. There is no consensus, currently, on what those costs are or where the line should be drawn.

All of this implies that the debate over the inclusion of the KVM tool has an importance that goes beyond the fate of that one project. Does (as some allege) the integration between perf and the kernel impede the development of alternatives and hurt the performance tooling ecosystem as a whole? Would the integration of the KVM tool put QEMU at the mercy of a fast-changing, regression-prone API over which its developers have no control? Are we better served by a fence between the kernel and user space that is as well defined at the project level as it is at the API level? Or, on the other hand, does keeping the KVM tool out of the kernel repository slow its growth and hurt the capability and usability of Linux tooling as a whole? And, importantly, what does the reasoning that leads to an answer to these questions tell us about which other projects should - or should not - find a home in the kernel tree?

These issues arise at a number of levels; some distributors, for example, are increasingly taking control of parts of the system through tightly-controlled in-house projects. Android is an extreme example of this approach, but it can be found in more traditional distributions as well. There are clear advantages to doing things that way, but it is worth asking whether that behavior is good for Linux in the long term and just where the line should be drawn. The fences between our projects may have played an important role in both the successes and failures of Linux; decisions on whether to strengthen them or tear them down need some serious thought.


(Log in to post comments)

Good fences make good projects?

Posted Nov 10, 2011 2:36 UTC (Thu) by jlargentaye (subscriber, #75206) [Link]

Maybe this argument was already made during the flamewar but does including NLKT in the kernel tree help kernel development overall? If so, it could be considered. If not, it's weighing down the tree and must be rejected.

Good fences make good projects?

Posted Nov 12, 2011 8:13 UTC (Sat) by steffen780 (guest, #68142) [Link]

The kernel by itself is pretty much useless. Whether or not something is good for the kernel (IMHO) quite obviously needs to take a back seat to whether or not it is good for the users. The kernel is also "weighed down" by the more-or-less strict adherance to API compatibility, yet I don't think anyone would argue that this policy should be dropped simply because the kernel would develop faster.

Or how would you like to run an OS that has just the linux kernel? Yes, this is a stupid strawmen-argument but it's the logical conclusion to your argument. Obviously a line has to be drawn somewhere between kernel benefits vs. wider benefits, but simply ignoring the latter is a rather odd suggestion.

remember how bad ALSA was?

Posted Nov 10, 2011 4:04 UTC (Thu) by dlang (subscriber, #313) [Link]

Things have settled down in the last few years, but remember how bad things were with ALSA due to the kernel portion and userspace portions being maintained and distributed separately?

Ted is right that a lot of it boils down to how good the developers are and how much they work to keep things sane, but ALSA is proof that keeping things separate just because one portion runs in userspace does not eliminate all the problems either.

I can offer another, not kernel-related example

Posted Nov 10, 2011 7:57 UTC (Thu) by khim (subscriber, #9252) [Link]

I had similar experience with our Native Client work.

Initially we had one single tree where all components resided: binutils, gcc, newlib and nacl core itself (which plays the role of kernel in the nacl ecosystem). In early days it was not uncommon to see the change which affected all components at once. I fail to even imagine the nightmare of trying to do separate development.

But times are changing: our ABI stopped changing so much over time (we needed to make it secure and it's kinda hard to do if there are daily changes), amount of changes in GNU toolchain was also reduced and now we use scheme with separate repos. We plan to push changes upstream which will make coordinated changes even harder (but will make life of our users easier), but this is Ok now because platfrom have quite stable ABI which will be supported for a long, long time.

I think it's points out to the same thing ALSA shows us: initial development often needs to change ABI (since nobody knows how "sane ABI" should look like), later projects can be separated. The only remaining question is: should the immature projects be kept in kernel for "incubation period" or perhaps they must live in separate repo along with patches to kernel till they "graduate" and will be ready to live as something separate from kernel? ALSA hints that first approach is not always feasible.

Good fences make good projects?

Posted Nov 10, 2011 11:40 UTC (Thu) by nhippi (subscriber, #34640) [Link]

> Ingo went so far as to claim that Linux lost the desktop market as the result of an ill-advised separation of projects.

Aye, that's we OS/2, *BSD, BeOS, Haiku, reactos etc are such runaway successes on desktop unlike Linux.

But rethinking the kernel/userland border would not be bad. For example I would love to see glibc being developed with the kernel development model and coding style...

Good fences make good projects?

Posted Nov 11, 2011 1:40 UTC (Fri) by HelloWorld (guest, #56129) [Link]

> > Ingo went so far as to claim that Linux lost the desktop market as the result of an ill-advised separation of projects.
> Aye, that's we OS/2, *BSD, BeOS, Haiku, reactos etc are such runaway successes on desktop unlike Linux.
Ingo is saying that the lack of ill-advised separation of projects is /necessary/ to be successful on the desktop, and not that it's /sufficient/. Please learn to distinguish necessary and sufficient criteria in order to start making sense.

Good fences make good projects?

Posted Nov 10, 2011 13:39 UTC (Thu) by davecb (subscriber, #1574) [Link]

I had good experiences with a project that shipped with a distribution but actually had a kernel-specific component (this was actually on Solaris, but the experience is similar).

We switched on the interface version number to allow the kernel/userspace components to develop independently while we were developing, so no-one could suffer from version skew, and cleaned out all old version each time the kernel rev'd. That allowed us to avoid the Windows-style backwards compatibility bugs.

We also used versioned interfaces internally, so we could develop different parts in parallel and not have flag days.

--dave
See also Fearnley on FOSS and continuous maintenance at http://cjfearnley.sys-con.com/node/1204364 and the Stachour article at http://cacm.acm.org/magazines/2009/11/48444-you-dont-know...

Good fences make good projects?

Posted Nov 10, 2011 17:49 UTC (Thu) by ccurtis (guest, #49713) [Link]

I'm not sure I see much good in integrating non-kernel stuff into the kernel tree, but I wonder if it would make more sense if some of these tools were perceived as test tools comprising part of a kernel test suite.

Then, there would be a test tool that exercises some part of the ABI while the user-friendly tool is hosted somewhere else. The test tool could be developed in-tree until the ABI is declared stable, at which point it is forked into an external project while the original tool can be used as a reference implementation for other tool developers.

When that "stable" point is reached may be contentious, but it may also be straightforward. For instance, the first patch that says "Make tool <foo> easier for users to run..." could be a good indicator that it's ready to be forked out.

Good fences make good projects?

Posted Nov 11, 2011 9:19 UTC (Fri) by ebirdie (guest, #512) [Link]

>I'm not sure I see much good in integrating non-kernel stuff into the kernel tree, but I wonder if it would make more sense if some of these tools were perceived as test tools comprising part of a kernel test suite.

I think this is very interesting idea and replying & quoting this just to raise attention and some insightful consideration for it.

The article raised a question, why perf is deemed to indefinite life in the kernel tree? As I have considered perf to be a kernel performance tool for kernel developers and haven't yet seen the userspace tools I use to monitor and tune system performance to use perf for anything, I haven't payed much attention to it. Does perf's life in kernel tree and the perception hinder its worthyness and development of use in userspace? Thus a kernel test & ABI suite/tree like linux-next but to reverse direction seems logical from a sysadmin perspective.

Good fences make good projects?

Posted Nov 11, 2011 11:22 UTC (Fri) by jnareb (subscriber, #46500) [Link]

> The article raised a question, why perf is deemed to indefinite life in the kernel tree? As I have considered perf to be a kernel performance tool for kernel developers and haven't yet seen the userspace tools I use to monitor and tune system performance to use perf for anything, I haven't payed much attention to it. Does perf's life in kernel tree and the perception hinder its worthyness and development of use in userspace? Thus a kernel test & ABI suite/tree like linux-next but to reverse direction seems logical from a sysadmin perspective.

"perf" development in Linux kernel tree makes it hard to package it and use it outside of kernel profiling. I was trying to use it to profile application (with limited success), in addition to other profiling tools that use perf events via PAPI library, and it was a bit difficult to make it work.

Good fences make good projects?

Posted Nov 11, 2011 21:47 UTC (Fri) by bcopeland (✭ supporter ✭, #51750) [Link]

> "perf" development in Linux kernel tree makes it hard to package it and use it outside of kernel profiling.

It seems the way it is handled in ubuntu at least is to package up perf as a script that just tells you to get yet another kernel-specific package that contains the actual binary. Given that, I find it amusing that perf is held as the gold standard for this kind of thing.

Good fences make good projects?

Posted Nov 10, 2011 20:20 UTC (Thu) by jimparis (subscriber, #38647) [Link]

Another argument that was brought up during that thread was that it can be harder for outsiders to work with and contribute to the code when it's part of a larger tree. For example, someone mentioned a feature of perf, and I wanted to take a quick look at perf source. It took me 15 minutes to update my kernel tree and checkout the latest copy. Having a separate git repository for perf would have made that much easier and quicker.

That might not be a practical problem for perf is because many perf users already have a kernel tree that they're working in. But the developer sets don't always overlap like that.

As a real-world example, the monolithic X11 tree was always a bit overwhelming. My friend used to hack a feature he liked into xterm, and it was always a hassle to get the whole tree and figure out how to get just xterm to build. When it was easily broken out into a separate autoconf'ed package around 2005, that became MUCH easier for him to do.

The discussions of pros/cons from the X11 modularization might still be relevant to this situation.

Good fences make good projects?

Posted Nov 11, 2011 8:32 UTC (Fri) by russell (guest, #10458) [Link]

Interesting... I wonder if GNOME would be so broken now ( in my opinion ) if they had been in the kernel. What they did could be considered a massive AP(person)I break.

Good fences make good projects?

Posted Nov 11, 2011 15:32 UTC (Fri) by hadess (guest, #24252) [Link]

Are you waiting for somebody to feed you?

Good fences make good projects?

Posted Nov 13, 2011 7:05 UTC (Sun) by russell (guest, #10458) [Link]

hehe, perhaps when people take away my food I tend to bite :)

Good fences make good projects?

Posted Nov 11, 2011 16:54 UTC (Fri) by teigland (guest, #31876) [Link]

Why not a separate linux-tools.git on kernel.org that parallels linux.git, with the same policies? An interesting question is who would manage such a tree. Part of the appeal of linux.git/tools may be that the tree already has an owner, thus avoiding the thorny process of selecting a new one.

Good fences make good projects?

Posted Nov 11, 2011 17:14 UTC (Fri) by dmarti (subscriber, #11625) [Link]

Also, anything that goes into the current tree counts toward your and your company's stats in the LWN reports. (What's the kernel equivalent of "sabermetrics?)


Copyright © 2011, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds