LinuxCon: Kernel roundtable covers more than just bloat

If you have already heard about the kernel roundtable at LinuxCon, it is likely due to Linus Torvalds's statement that the kernel is "huge and bloated". While much of the media focused on that soundbite, there was quite a bit more to the panel session. For one thing, Torvalds definitely validated the impression that the development process is working better than it ever has, which has made his job an "absolute pleasure" over the last few months. In addition, many other topics were discussed, from Torvalds's motivations to the lessons learned in the 2.6 development series—as well as a bit about bloat.

The panel consisted of five kernel developers: Torvalds, Greg Kroah-Hartman of Novell, Chris Wright of Red Hat, Jonathan Corbet of LWN, and Ted Ts'o of IBM (and CTO of the Linux Foundation) sitting in for Arjan van de Ven who got held up in the Netherlands due to visa problems. James Bottomley of Novell moderated the panel and set out to establish the ground rules by noting that he wanted to "do as little work as possible", so he wanted questions from the audience, in particular those that would require answers from Torvalds as "he is sitting up here hoping to answer as little as possible". Bottomley was reasonably successful in getting audience questions, but moderating the panel probably took a bit more effort than he claimed to be looking for.

Innovative features

Bottomley began with a question about the "most innovative feature" that went into the kernel in the last year. Wright noted that he had a "virtualization slant", so he pointed to the work done to improve "Linux as a hypervisor", including memory management improvements that will allow running more virtual machines more efficiently under Linux. Corbet and Ts'o both pointed to the ftrace and performance counters facilities that have been recently added. Tracing and performance monitoring have both been attacked in various ways over the years, without getting into the mainline, but it is interesting to see someone approach "the problem from a different direction, and then things take off", Corbet said.

Bottomley altered the question somewhat for Kroah-Hartman, enquiring about the best thing that had come out of the staging tree that Kroah-Hartman maintains. That seemed to stump him momentarily, so he mentioned the USB 3.0 drivers as an innovative feature added to the kernel recently, noting that Linux is the first OS to have a driver for that bus, when hardware using it is still not available to buy: "It's pretty impressive". After a moment's thought, though, Kroah-Hartman pointed out that he had gotten Torvalds's laptop to work by using a wireless driver from the staging tree, which completely justified that tree's existence.

Ts'o also noted the kernel mode switching support for graphics devices as another innovative feature, pointing out that "it means that the X server no longer has to run as root—what a concept". He also suggested that it made things easier for users who could potentially get kernel error messages in the event of a system hang, without having to hook up a serial console.

Making it easy for Linus

Torvalds took a "different tack" on the question, noting that he was quite pleased with "how much easier my job has been getting in the last few months". He said that it is a feature that is not visible to users but it is the feature that is most important to him, and that, in the end, "it improves, hopefully, the kernel in every area".

Because subsystem maintainers have focused on making it "easy for Linus" by keeping their trees in a more mergeable state, Torvalds has had more time to get involved in other areas. He can participate in more threads on linux-kernel and "sometimes fix bugs too". He clearly is enjoying that, especially because "I don't spend all my time just hating people that are sending merge requests that are hard to merge".

Over the last two merge windows (including the just completed 2.6.32 window), things have been going much more smoothly. Smooth merges mean that Torvalds gets a "happy feeling inside that I know what I am merging — whether it works or not [is a] different issue". In order to know what he is merging, Torvalds depends on documentation and commit messages in the trees that outline what the feature is, as well as why people want it. In order to feel comfortable that the code will actually work, he bases that on his trust of the person whose tree he is merging to "fix up his problems afterwards".

Motivation

The first question from the audience was directed at Torvalds's motivation, both in the past and in the future. According to Torvalds, his motivation for working on the kernel has changed a lot over the years. It started with an interest in low-level programming that interacted directly with the hardware, but has slowly morphed into working with the community, though "I shouldn't say 'the community', because when anyone else says 'the community', my hackles rise [...] there's no one community". It is the social aspect of working with other people on the kernel project that is his main motivation today, part of which is that "I really enjoy arguing".

Torvalds's technical itch has already been scratched, so other things keep him going now: "All of my technical problems were solved so long ago that I don't even care [...] I do it because it's interesting and I feel like I am doing something worthwhile". He doesn't see that changing over the next 5-10 years, so, while he wouldn't predict the future, there is a clear sense that things will continue as they are—at least in that time frame.

Malicious code

Another question from the audience was about the increasing rate of kernel contributions and whether that made it harder to keep out malicious code from people with bad intentions. Kroah-Hartman said that it is hard to say what is malicious code versus just a bug, because "bugs are bugs". He said he doesn't remember any recent attempts to intentionally introduce malicious code.

Torvalds pointed out that the problem has never been people intentionally doing something bad, but, instead, trying to do something good and unintentionally ending up causing a security hole or other bug. He did note an attempt to introduce a back door into the kernel via the BitKeeper repository 7-8 years ago which "was caught by BitKeeper with checksums, because they [the attackers] weren't very good at it". While that is the only case he is aware of, "the really successful ones we wouldn't know about". One of Git's design goals was to keep things completely decentralized and to cryptographically sign all of the objects so that a compromise of a public git server would be immediately recognized, because it didn't match others' private trees, he said.

Performance regressions

Bottomley then turned to performance regressions, stating that Intel had been running a "database benchmark that we can't name" on every kernel release. They have found that the performance drops a couple of percentage points each release, with a cumulative effect over the last ten releases of about 12%. Torvalds responded that the kernel is "getting bloated and huge, yes, it's a problem".

"I'd love to say we have a plan" for fixing that, Torvalds said but it's not the case. Linux is "definitely not the streamlined, small, hyper-efficient kernel that I envisioned 15 years ago"; the kernel has gotten large and "our icache [instruction cache] footprint is scary". The performance regression is "unacceptable, but it's probably also unavoidable" due to the new features that get added with each release.

Audio and storage

In response to a question about professional audio, Torvalds said that the sound subsystem in the kernel was much better than it is given credit for, especially by "crazy" Slashdot commenters who pine for the days of the Open Sound System (OSS). Corbet also noted that audio issues have gotten a lot better, though, due to somewhat conflicting stories from the kernel developers over the years, audio developers "have had a bit of a rough ride".

A question about the need for handling memory failures, both in RAM and flash devices, led Ts'o to note that, based on his experience at a recent storage conference, there is "growing acceptance of the fact that hard disks aren't going away". Hard disks will always be cheaper, so flash will be just be another element in the storage hierarchy. The flash hardware itself is better placed to know about and handle failures of its cells, so that is likely to be the place where it is done, he said.

Lessons learned

The lessons learned during the six years of the 2.6 development model was the subject of another question from Bottomley. Kroah-Hartman pointed to the linux-next tree as part of a better kernel development infrastructure that has led to more effective collaboration: "We know now how to work better together". Corbet noted that early 2.6 releases didn't have a merge window, which made stability of those releases suffer. "What we've learned is some discipline", he said.

In comparing notes with the NTFS architect from Microsoft, Ts'o related that the core Windows OS team has a similar development model. "Redmond has independently come up with something almost identical to what we're doing", he said. They do quarterly releases, with a merge period followed by a stabilization period. Microsoft didn't copy the Linux development model, according to the NTFS architect, leading he and Ts'o to theorize that when doing development "on that scale, it's one of the few things that actually works well". That led Bottomley to jokingly suggest a headline: "Microsoft validates Linux development model".

Torvalds also noted that the development model is spreading: "The kernel way of doing things has clearly entered the 'hive mind' when it comes to open source". Other projects have adopted many of the processes and tools that the kernel developers use, but also things like the sign-off process that was added in response to the SCO mess. Sign-offs provide a nice mechanism to see how a particular chunk of code reached the mainline, and other projects are finding value in that as well.

Overall, the roundtable gave an interesting view into the thinking of the kernel developers. It was much more candid than a typical marketing-centric view that comes from proprietary OS vendors. Of course, that led to the "bloated" headlines that dominated the coverage of the event, but it also gave the audience an unvarnished look at the kernel. The Linux Foundation and Linux Pro magazine have made a video of the roundtable available—unfortunately only in Flash format—which may be of interest; it certainly was useful in augmenting the author's notes.