LWN.net Logo

KS2010: Development process

By Jonathan Corbet
November 3, 2010
2010 Kernel Summit
As the 2010 Kernel Summit headed toward its end, Linus Torvalds and Andrew Morton were given a session slot to talk about how the development process is working for them. The bottom line: things are going well, but there are a number of things which can be improved. Release numbering is not one of them, though.

Linus said that, on the whole, he's very happy with how things are working. He was very happy last year, and things have gotten better since. He does not have any major concerns. That said, he does have some pet peeves.

At the top of his list was pull requests for trees containing commits made within the last hour; those, he said, make him want to kill somebody. There is no excuse for such requests, especially after -rc3 or so - though they may be understandable during the merge window (but he still doesn't like them). He also advised against faking commit dates to get around this issue - something which has apparently "accidentally" happened in the past. A dead giveaway, he says, is when he does a series of pulls and one of the later ones turns out to be a fast-forward merge.

There were questions about emergency fixes or patches which have been extensively tested in other settings. But Linus's point is that he wants developers to have tested the exact tree that they are asking him to pull.

Linus would also like to see more Reported-by, Reviewed-by, and Acked-by tags in patches. And, especially, he would like to see those tags from outside people. A bunch of Reviewed-by tags from people in the same company do not have a lot of value; patches really need outside review. On the other hand, he would like fewer embargoed reports of security bugs the day before a major release. That, it turns out, is what delayed the 2.6.36 release by a week. He won't release a kernel with a known root hole. If you plan to ask for an embargo, he would rather just not be told.

Andrew, instead, said that he has never had any pet peeves - but he did have a few requests. He does a lot less work than he used to; he handles about 1/3 of the patch volume that once went through his tree. But he does still maintain between 100 and 150 subsystem trees. He would like some help, not with all those trees, but with major decisions like the merging of control groups or checkpoint/restart. Merging control groups set the kernel on a multi-year course; he would really like not to have to make those decisions on his own. So he asked maintainers to get out of their subsystem occasionally and look at major patches to other areas of the kernel.

He seemed a little hurt by some recent requests for a separate tree for memory management patches. Those calls were driven by some bugs found in the kmap_atomic() rewrite; the source problem is that code in -mm doesn't appear in linux-next, so it does not get a lot of testing. He understands that his patch management could use improvement; he'll be getting -mm into linux-next shortly. Andrew said that he would like to keep working on the memory management subsystem, an announcement which was received with relief by a number of the people present.

The problem with memory management, Andrew said, is that it has become incredibly complex; at this point, only Hugh Dickins understands the whole thing. It can be very hard to figure out how things work. What is more worrisome is that this complexity is turning up throughout the kernel. In response, he has been pushing harder and harder to get people to comment their code. Nobody ever pushes back; we all agree that it's needed. But, Andrew said, "we still suck at it." We're going to be working on the kernel for a very long time; code we write now should be seen as a sort of communication to some stranger five years from now. So we need to put more effort into properly documenting our work.

Linus jumped in with one other pet peeve: he hates having to be the person to say "no." He finds it a lot easier to just grumble and say "OK, just this time," but it makes him unhappy. According to Linus, most subsystem maintainers are too eager to take new features; he would like them to be more selective. There is one easy way to see which trees make him unhappy: look for those which are pulled late in the merge window. He has a tendency to delay making decisions on merges which look ugly, so unpleasant trees pile up until the end.

Greg Kroah-Hartman jumped in with what was, perhaps, the most pressing development process question: is it time to change the version numbering scheme? Some people are getting tired of the long numbers, and the term "2.6 kernel" does not really mean anything anymore. Alternatives could be going to 3.0 for the next release or using a year-based scheme (so the next kernel would be 2011.0, say). The discussion went on for a while; it became clear that not everybody sees the need for a change. Additionally, there appears to be a desire to see the minor number reach 42. Linus ran a straw poll which was won (42-33) by the "no change" contingent. So the version numbering scheme will not change for now; Greg promised to try again next year.

Arjan van de Ven asked: what is our biggest priority for the next six months? Linus answered quickly: he wants to see the VFS scalability work merged in the very near future or he will be very unhappy. Al Viro said that he is not happy with all of that work, especially the dentry cache scalability changes. Linus allowed that "the code is interesting," but said there do not appear to be any real alternatives.

Ted Ts'o said that the memory management and filesystem people need to get together and figure out how the writeback problem will be solved. Andrew seconded that, saying that he has seen some "drive-by patches," but that there does not seem to be a real plan. Christoph Hellwig described that statement as being unfair: the relevant people, he said, have been talking about the problem and working on it. Ted asked if there was a plan, saying that he certainly hadn't seen one. Christoph responded that the developers have an idea of where they want to go.

There was a brief discussion of the device tree patches. Linus said that he thought device trees were "always a bad idea," that it is far better to simply discover the hardware present on a system. The catch, of course, is that not all hardware supports robust discovery; in that case, there is no alternative to describing the hardware externally. Device trees seem to be the way to go for such situations.

Next: Future summits.


(Log in to post comments)

KS2010: Development process

Posted Nov 3, 2010 23:22 UTC (Wed) by vomlehn (subscriber, #45588) [Link]

Hearing that Linux doesn't like device trees leaves me concerned that he is less focused on the embedded world than I might like. Embedded developers can't afford to stick self-describing buses on all of our hardware and having a standard way to describe hardware is a big plus. Also, seems PCs use something rather like the device tree to pass information from the bootloader to the kernel; is there a conceptual difference to justify the dislike?

It's true we have a lot of work to do on the device tree, but the need is strong.

KS2010: Development process

Posted Nov 4, 2010 2:30 UTC (Thu) by neilbrown (subscriber, #359) [Link]

As I understood it, Linus doesn't like device trees primarily because discoverable buses are just soooo much better. So in overall design decisions, discoverable buses should be considered the norm and device tree is the second cousin that is used when there is no alternative.

There is no suggestion that we must not have device trees, just that they be kept clearly in their place, which is secondary to discoverable buses.

KS2010: Development process

Posted Nov 4, 2010 12:36 UTC (Thu) by etienne (subscriber, #25256) [Link]

IMHO, most of the device tree use I have seen is to define addresses of subcompoments, and I still think that is a job for the linker - having a special extra file as linker command to contain all those addresses simplify the process.
It should be possible to have late relocation for very few fields in the bootloader (at least kernel relocations is possible on ia32/amd64) if you really want a single kernel for different "fat" microcontrollers.

KS2010: Development process

Posted Nov 4, 2010 4:20 UTC (Thu) by swetland (subscriber, #63414) [Link]

I'm perplexed at the desire for device trees.

The idea that you're going to have one kernel that will run on the next spin of the hardware with no changes at all is somewhat laughable in most "real" embedded projects. Things move around and not just simple things like gpio assignments that you can stick in a static structure.

Oops, turns out that you need a level shifter on part of the I2C bus that must be turned on before you initialize the peripherals behind it. Until you start sticking bytecode in your device trees, changes like this will require kernel/driver changes.

Basically device trees make my life harder by putting board configuration in the bootloader or another partition somewhere where it can get out of sync with the kernel.

A lot of people seem to want embedded to look like PCs with everything based on (expensive) enumerable busses and (clunky) BIOSes that set up things before booting. This is incompatible with building cost-effective and power-use-competitive hardware in highly competive embedded/mobile spaces.

KS2010: Development process

Posted Nov 4, 2010 11:00 UTC (Thu) by Klavs (subscriber, #10563) [Link]

I'm a bit confused here. You seem to be against BOTH devicetrees and discoverable buses? and if so - what solution do you think is the best for embedded environment?

KS2010: Development process

Posted Nov 4, 2010 12:01 UTC (Thu) by swetland (subscriber, #63414) [Link]

I love nice enumerable busses -- I just almost never have the luxury of them existing on the embedded platforms I work with. In their absence, I find the existing board/machine file mechanism Linux uses gets the job done and without having to depend on fragile links between data structures embedded in the bootloader or elsewhere and the kernel.

I'd be happy to see improvements to make it easier to maintain board files and such -- they can get a little unwieldy -- but I'm not convinced that a device tree approach actually improves the situation.

I've never worked on a production device (typically shipping between hundreds of thousands and millions of units) where having to compile a new kernel to pick up device-specific hardware support is the major pain point. Certainly I never have a canned, existing kernel that I'm going to just use with no modifications -- power tuning and debugging during stabilization guarantees that we're going to be spinning quite a few kernel builds until things are ready to ship.

KS2010: Development process

Posted Nov 4, 2010 16:41 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]

nobody is expecting that device trees will eliminate the need for you to compile your own kernel for a production system.

it's also not that you will take a kernel compiled for one device and change the devicetree file and load it on another device.

what they will do is make it so that there aren't 70 different subarchitectures in ARM that all have to be tested individually and replace it with a much smaller number with devicetrees identifying the minor details like where a particular piece of hardware is addressed. As such it will make it easier for people to maintain the kernel and make sure that it will work.

Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds