LWN.net Logo

Kernel Summit 2007 - an advance view

By Jonathan Corbet
August 24, 2007
For the past several years, the annual, invitation-only kernel developers' summit has been held immediately prior to the Ottawa Linux Symposium. This year is different, though: the summit is, instead, happening just after LinuxConf Europe in Cambridge, UK. As usual, your editor will be there and will be able to report from the event. The preliminary agenda has been posted, though, as has the list of attendees [PDF]. So it is possible to look forward and get a sense for what is likely to be discussed.

A few months ago, a discussion of interesting topics was held on the 2007 summit list. Many of the usual topics came around; there is always plenty of interesting development work going on in the kernel community. Andrew Morton objected to many of the topics under discussion, though, saying that the summit was not the appropriate venue to talk about them:

My overall take on kernel summit: we spend far too much time talking about technical stuff. There is little benefit in doing this: we conduct technical discussions over email and we do it well, and there are many very good reasons for doing it that way.... We fly halfway around the world to yap on about dentry cache scalability? Spare me, we'd get more done by staying home.

Andrew's conclusion, which was seconded by a number of other developers, was that the process-oriented discussions are always more interesting and useful than the deep technical sessions. Discussions of virtualization, memory management, or device drivers will always be uninteresting to a significant part of the group, and they do not necessarily add much over what can be done with email. But the process-oriented talk affects everybody and is much harder to do electronically.

So this year's agenda is more high-level than in previous years. That does not mean that there will be no technical talk, though. Some of the more technical sessions will cover:

  • Reports from mini-summits. The kernel is a big program, and developers often find that subsystem-specific questions are better addressed in smaller groups. At the summit, attendees from some recent mini-summits (covering power management, filesystems, storage, and virtualization, at least) will report back to the larger group.

  • Real time and scheduler issues are on the agenda because there are some big decisions to make. While much of the real-time tree has found its way into the mainline, some of the more disruptive chunks (sleeping spinlocks, threaded interrupt handlers) remain outside. Also outside of the mainline is the syslets/threadlets patch set. Hopefully some decisions will be made on whether these features should be merged, and, if so, what needs to be done to get them into shape.

  • There are a number of memory management issues out there, including the variable page and variable block size patches, approaches to deadlock avoidance, scalability work, and more. Also on the agenda is the more process-oriented question of why memory management patches are so hard to get into the mainline.

  • Virtualization has fallen off the agenda because most of the kernel-level work in this area has already been merged. The containers developers are just getting going, though, and there are a lot of questions about what their final destination is thought to be. A full containers implementation could impose significant overhead - on developers and on run-time performance - and could prove hard to sell.

That's about it for the serious technical talks; everything else will have a higher-level focus. The summit will start with a panel of distributor kernel maintainers. To a great extent, distributors are the immediate customers for the kernels that the developers put out; those distributors are then charged with getting mainline releases into a condition that allows it to be shipped to users. Distributor kernel maintainers tend to be on the front line when things go wrong; they always hear about all the problems. This panel will be a chance for those maintainers to talk about the quality of the kernels they are getting from the mainline and how things could be made to work better.

Once upon a time, the kernel stood alone and presented services to the system by way of the system call interface. In current systems, instead, users see a view of the system which is created by a whole set of utilities, including the C library, udev, HAL, and more. Interactions between these low-level components and the kernel is not always as smooth as it could be, and, despite the best efforts of the kernel development community, kernel releases have been known to occasionally break utilities like udev. The "greater kernel ecosystem" session will cover these issues and the general question of making the system as a whole work better together. Establishing better control over the user-space API is likely to come up, though the problem remains difficult.

There is a half-hour session on developer relations. The kernel development community is visibly growing, and that is generally a good thing. Ensuring the continued health of kernel development requires bringing in a steady stream of new developers - from all over the world. This session will be the place to talk about how that can be done, and how participation from under-represented parts of the world can be improved.

Andrew Morton gets an hour to pound the table on kernel quality and related issues. There still appears to be a consensus among the developers that the kernel is not getting buggier, but that view is not universally held. Everybody agrees that fewer bugs would be a good thing, though. So topics like bug tracking, fixing the reviewer shortage, possible stabilization releases, and so on, are likely to come up in this session.

Documentation is, inevitably, on the agenda - everybody wants more of it, but, somehow, it fails to just show up on its own. Last year there was some talk of imposing documentation requirements on new patches, but few people took the idea all that seriously. So maybe some different ideas for improving the situation will come about this time around. Also on the list may be the area of managing translations - an area of increasing interest - and standardizing kernel messaging.

Various other process-oriented questions have been swept into a session late on the second day. Are big code cleanups worth it? How can we improve our handling of large patches which affect a number of different subsystems? How do we deal with problematic maintainers? And, in general, is the kernel process going too fast? But perhaps the discussion will be dominated by Andrew Morton's suggestion that the developers form a union and demand a massive pay raise.

There are other sessions on the agenda as well; see the posted version for the full list. Whenever a group of this nature comes together, interesting things are bound to come out of it. Tune into LWN around September 6 for coverage from the event.


(Log in to post comments)

Kernel Summit 2008 - an advance view

Posted Aug 24, 2007 19:08 UTC (Fri) by aleXXX (subscriber, #2742) [Link]

From the email:

- the 80-column rule: is it time to upgrade to 96?

Why 96 ? Is there something special about 96 ?

Alex

Kernel Summit 2008 - an advance view

Posted Aug 24, 2007 20:07 UTC (Fri) by dwheeler (guest, #1216) [Link]

I suspect it's because 96=80+16. Tab stops are traditionally 4 or 8 chars wide, and in the Linux kernel coding convention are 8 chars. So 96 would let you create text with two more indentation levels before you hit today's limits.

I don't know if that a REAL improvement, though. Too many OTHER things don't work as well when you go beyond 80 chars.

Anyway, that's my guess.

Kernel Summit 2008 - an advance view

Posted Aug 26, 2007 11:00 UTC (Sun) by ibukanov (subscriber, #3942) [Link]

> Too many OTHER things don't work as well when you go beyond 80 chars.

For example, with 80 chars per line I can have 2 source windows side by side with a nice readable font on 1280x768 screen of my laptop. That would not work with 96 chars.

Kernel Summit 2008 - an advance view

Posted Aug 25, 2007 19:31 UTC (Sat) by jengelh (subscriber, #33263) [Link]

>- the 80-column rule: is it time to upgrade to 96?

Oh no please spare us. Today we have the right end at column 80. People do occassionally go over that. And that's totally ok[1].

Now, if the right end was extended to 96, people would start writing lines with up to 112 chars in length, and the story re-begins. NO THANKS.

[1] http://lkml.org/lkml/2007/6/23/189 second-to-last paragraph "Dammit, [...]"

Kernel Summit 2008 - an advance view

Posted Aug 27, 2007 6:09 UTC (Mon) by landley (guest, #6789) [Link]

Sigh. I'm sorry I won't be able to participate in that Documentation
discussion. Oh well, let me know what you guys decide to do and I'll see
if any of it's at all useful to what I'm doing...

Kernel Summit 2008 - an advance view

Posted Aug 30, 2007 16:52 UTC (Thu) by AJWM (guest, #15888) [Link]

The 80-columns of course is legacy from punch card (Hollerith card) days, which was inherited by most dumb terminals (24 or 25 lines of 80 chars -- which xterm by default duplicates). There was also a brief period where a computer manufacturer (I think IBM, but could be wrong) introduced a 96-column punch card (different shape overall, not just 20% wider).

The relationship could also derive from typing: using skinny (1/4") margins to get 8" lines on an 8.5x11 page (US letter size), a courier typewriter at 10 chars/inch would fit 80 chars to the line (and 6 lines/inch). A "elite" typewriter, at 12 cpi would fit 96 chars on that same line (and 8 lines/inch).

If you really wanted to go to wide lines, there's the 132 columns (characters) that was standard for line printer output back in the day.

Kernel Summit 2008 - an advance view

Posted Aug 30, 2007 17:20 UTC (Thu) by jschrod (subscriber, #1646) [Link]

You should note that typography research recommends to use at most 60-70 characters per line for readability. Taking indentation into account (but also not forgetting about comments) that translates to ca. 80-90 characters in source files.

There's more to this limit than punch cards.

Joachim

Kernel Summit 2008 - an advance view

Posted Aug 30, 2007 19:12 UTC (Thu) by AJWM (guest, #15888) [Link]

Oh, I'm not advocating >80 char lines in source code. Heck, I used to program a lot in APL, 20 characters ought to be more than enough for anyone ;-)

80 column terminals

Posted Sep 1, 2007 0:21 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

It's definitely from Hollerith cards. People didn't use the early terminals to type memos, so even if there were a custom of typing with 1/4" margins (which I'm pretty sure there never was), it would not have influenced terminal design. What they did with terminals was type out (or type in) datasets that otherwise were stored on Hollerith cards: columns of numbers, source code, etc.

Block sizes on tape and disk were also multiples of 80 bytes for a long, long time for the same reason.

For those who haven't heard the story of why a Hollerith card is 80 columns: It's just what happened to fit on a card of that size with machinery of the day. And the size of the card was determined by the size of US currency, because there were storage boxes and such for that size.

Kernel Summit 2008 - an advance view

Posted Aug 24, 2007 20:58 UTC (Fri) by Tara_Li (subscriber, #26706) [Link]

Headline says 2008, e-mail says 2007. Eh?

Kernel Summit 2008 - an advance view

Posted Aug 24, 2007 21:01 UTC (Fri) by corbet (editor, #1) [Link]

I get ahead of myself sometimes...fixed now.

Kernel Summit 2007 - an advance view

Posted Aug 26, 2007 12:30 UTC (Sun) by error27 (subscriber, #8346) [Link]

I don't think slowing down is a good idea. It would just make things languish in git trees for a little longer but it wouldn't result in many fixed bugs.

Kernel Summit 2007 - an advance view

Posted Aug 26, 2007 17:22 UTC (Sun) by mingo (subscriber, #31122) [Link]

I don't think slowing down is a good idea. It would just make things languish in git trees for a little longer but it wouldn't result in many fixed bugs.

Yeah - or worse, it would force developers to go elsewhere.

I think the right approach to high-flux development is three-fold:

1) decrease the latency of getting the latest kernel to the user. 2-3 months release cycle pushes human limits but is doable. 2-3 _days_ is doable latency for the -git kernel, for bleeding-edge testers: today a fully packaged up rpm kernel of latest -git is yum-upgradable 1-2 days after Linus commits a change into his tree.

2) increase the likelyhood of users to report back a bug and for them to test fixes. This one has many aspects: good debugging infrastructure (many automated debugging features, automatic bisectability), meaningful (and early enough) debug output, and rewards to testers who report back, responsive maintainers, etc. This is where we have the biggest technical deficiencies right now.

3) maintainers must push back on unrobust code, as a function of how stable the _previous_ release was. If the previous release was too unstable, be stricter in the next release. If the previous release was too boring, risk more changes. 2-3 months of a delay to a feature is not the end of the world - with a release cycle longer than that it can be lethal to motivation.

The biggest problem is the fundamental lability of this dynamic system: if the kernel gets buggier then users go away, due to that the kernel gets _even more_ buggier (and kernel developers wont even notice initially), then developers go away too, etc. It quickly spirals out of control and detaches the kernel developer community from the user community. That is what happened with Linux 2.5 and it took years to fix up and get Linux 2.6.0 out of the door.

Sticking to the "90 days max" release cycle is key. That is what keeps us all honest. If we mess up then the only true recourse is to release a buggier kernel, and the punishment for that is much more direct (and unpleasant to the kernel developer) "your kernel is buggy" (or "your subsystem is buggy") impression than a "slip in the release date" action is - so it's a far more efficient method of feeding back the true quality of the kernel back to the developer community and thus keeping the quality of the kernel up high enough.

Kernel Summit 2007 - an advance view

Posted Aug 27, 2007 11:03 UTC (Mon) by balbir_singh (subscriber, #34142) [Link]

Hi, Ingo,

Good suggestions, I agree with all of the three. But, I might add that it might be a good idea to have

1. A feature reviewer for every new feature added (as a backup for the
contributor of the feature). That way we ensure that at-least three
people understand the code, the contributer, the feature reviewer
and the maintainer. Currently, the maintainer is burdened with too
many features to handle.
2. I like the way boost works with respect to processes
(http://www.boost.org/more/formal_review_process.htm)

Kernel Summit 2007 - an advance view

Posted Aug 28, 2007 11:53 UTC (Tue) by pointwood (guest, #2814) [Link]

=== CUT ===
2) increase the likelyhood of users to report back a bug and for them to test fixes. This one has many aspects: good debugging infrastructure (many automated debugging features, automatic bisectability), meaningful (and early enough) debug output, and rewards to testers who report back, responsive maintainers, etc. This is where we have the biggest technical deficiencies right now.
=== CUT ===

I'm just a mere user, but I would like help out if/where I can. I have no idea how to test a new kernel on my systems, but if it was made easy (LiveCD's?) and/or there was a guide that provided me with relevant info (what to report back, etc.), then I'd be happy to test new kernels. I bet much of it can be automated. Create a mailing list used for announcing new kernels to be tested. I think that would get you a lot more testers and get the kernel tested on a lot more hardware.

Kernel Summit 2007 - an advance view

Posted Aug 30, 2007 10:53 UTC (Thu) by intgr (subscriber, #39733) [Link]

There indeed appears to be no documentation on kernel testing for
non-developers; Linus is always asking for more testers of release
candidates, but it is often not straightforward without guidance on getting
the kernel installed and reporting problems.

Publishing LiveCDs for every release candidate would really be a waste,
and I am not aware of any. If compiling it all manually is over your head,
don't bother -- debugging and reporting problems also requires quite a bit
of technical know-how which you are unlikely to be prepared for.

This is the "short" guide to testing kernels: follow Linus's release
candidates; kerneltrap.org often reports -rc1 releases earlier than LWN
because it's not bound by the weekly schedule, but LWN is more consistent
-- the latest kernel release candidate is always reported on the kernel
page.

Preferably you would use git because it allows regressions to be bisected
to the patch that caused the regression:
http://www.kernel.org/pub/software/scm/git/docs/v1.4.4.4/...

If that's too complicated for you, get tarballs from www.kernel.org; build
the kernel (look for a howto/tutorial for your distro and take your time,
configuration might take a while for the first time); use it for everything
you normally use your computer for. When problems occur, write to LKML
(linux-kernel@vger.kernel.org) with the details of your hardware, your
configuration, any suspicious messages in dmesg, and what could have
caused the problem. Be sure to mention that it's a regression, and the
last kernel version that worked. More details at:
http://www.kernel.org/pub/linux/docs/lkml/reporting-bugs....
If you can be bothered, use the git-bisect feature to track down the
erroneous commit (see the link about git above).

Even if the kernel runs fine, keep your eye on dmesg for OOPSes -- these
are definitely bugs regardless of whether they cause you problems.

If there is no release candidate, you can try running the -mm kernel tree;
it contains experimental patches, many of which are probably going to be
merged into the mainline sooner or later. Be aware though, that it is more
likely to have serious bugs.

Kernel Summit 2007 - an advance view

Posted Aug 30, 2007 18:57 UTC (Thu) by aegl (subscriber, #37581) [Link]

"Even if the kernel runs fine, keep your eye on dmesg for OOPSes -- these
are definitely bugs regardless of whether they cause you problems."

If you are regularly testing development kernels, it is really useful to save the dmesg(1) output from every kernel that you boot. I added this to my /etc/rc.local

REL=`uname -r`
TSTAMP=`date +%Y%m%d%H%M%S`
dmesg -s 100000 > /dmesg/$TSTAMP-$REL

Then when I notice something odd happening I can check whether anything new and interesting showed up in the boot messages.

I've spotted a lot of issues by simply running diff(1) on the current and previous dmesg output.

Kernel Summit 2007 - an advance view

Posted Sep 1, 2007 14:12 UTC (Sat) by kreutzm (guest, #4700) [Link]

Or use a tool to scan your log files (like logcheck). When I try out a new kernel, I immediately get all new lines which I can then add to the log check rules after I acknowledged them (and possibly took other action).

Kernel Summit 2007 - an advance view

Posted Aug 31, 2007 1:41 UTC (Fri) by rddunlap (guest, #27065) [Link]

Have you seen the "Linux Kernel Tester's Guide" ?

http://www.stardust.webpages.pl/files/handbook/handbook-e...
or just
http://www.stardust.webpages.pl/files/handbook/handbook-e... for
the non-rc1 version.

Copyright © 2007, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds