ELC/LFCS2009: A tale of two panels

Two kernel panel sessions were held last week in San Francisco, one for each of two conferences sharing facilities—and participants. In both cases, the kernel developers making up the panel were asked about various kernel features and developments, both from a historical and future perspective, but each had its own focus as well. The Embedded Linux Conference (ELC) panel was, unsurprisingly, focused on topics of most interest to the embedded community, while the Linux Foundation Collaboration Summit (LFCS) panel looked at more mainstream kernel concerns.

ELC: Embedded Linux Kernel Features and Developments

Besides the venue, the panel sessions also had another thing in common: LWN Executive Editor Jonathan Corbet, who moderated the LFCS panel and was a member of the ELC version. The ELC panel was moderated by CE Linux Forum (CELF) architecture group chair Tim Bird, while embedded maintainer David Woodhouse and Matt Mackall, developer of the SLOB memory allocator (along with various other kernel tasks), rounded out the panel. Bird asked most of the questions, but the audience also got the opportunity to ask a few too.

One of the themes of the discussion—as well as Woodhouse's earlier keynote—was the convergence of features between so-called "big iron" (servers and mainframes) and embedded devices. Corbet was amused to see "highmem" support recently added for ARM processors, noting that it was a controversial feature at the time it was added for servers; supporting a full 1GB of RAM on a 32-bit processor was once a "big iron" problem. Mackall also pointed to SMP and NUMA support moving into the embedded architectures. But things are not only moving in that direction, Mackall said, there is recognition from the big iron developers that there is value for their systems in some of the embedded features too.

Bird asked the panel about the proliferation of embedded distributions and whether that was a good or bad thing. Woodhouse said "fragmentation doesn't have to be bad"; it's only bad when a distribution doesn't work well with the various upstreams and goes off and does its own "weird things". Multiple distributions are one of the "great strengths of Linux", Corbet said, as it provides playgrounds where folks can experiment with different approaches.

Mackall pointed to a lack of community involvement in the various embedded Linux distributions, noting that the most successful desktop distributions were those with a strong community. In the mobile space, the distributions are "coming from the top down", he said, for any of them to be successful, they need to get community feedback.

The impact and usefulness of new "social networking" sites for Linux developers—like MontaVista's meld and the LF's relaunch of the Linux.com community—was another question Bird put to the panel. Woodhouse didn't really see the need, but "communication is always good". Mackall was concerned that these other services not become a "substitute for talking to the Linux kernel community through its normal channels". Corbet noted that there is value in "small town environments", but there is a risk that they can become inbred. "It rarely leads to good things" when a small community gets headed off in their own direction, he said.

One of the more interesting exchanges centered around the question of what a developer who just has a small amount of time can do to assist the larger community. The discussion spread out from there, though. Woodhouse stated that every developer needed to make sure that what they are working on can go upstream even if their managers "need to be whipped to allow you to do that". But Mackall wanted to "back up a step" and ensure that developers are running Linux on their desktop.

Mackall said that developers should be running Linux at home as well; if they are going to work with Linux, they should "live it". Making it work on a laptop is a good exercise; if it doesn't work, figure out why and fix it. He has seen too many embedded Linux developers with Windows desktops who don't understand Linux well enough to properly develop on it. "They don't have the Linux mindset", he said.

Those thoughts were echoed by Woodhouse as he related an anecdote about some embedded developers who would FTP a file to the Windows box, edit it using Notepad, then FTP it back to the Linux machine. It is not efficient to do things that way, he said. Doing the development on Linux will lead to a better result, Mackall said. Doing everything on a Linux desktop will help that, Mackall pointed out, "you should read your mail on it too".

Towards the end of the hour-long session, Bird asked "have we won?", is embedded Linux unstoppable or "is it possible to lose?". Mackall and Corbet had similar thoughts, worrying about the proliferation of devices running Linux that could not be modified by their users. "We haven't won until I can put my code on my phone", Mackall said. Corbet echoed that: "If we end up populating the world with locked-down Linux systems, then we've lost".

In closing, Bird noted that embedded Linux has made an "awful lot of progress". This is the fifth year for ELC and he has been working on embedded Linux for 17 years, over that time, "things have gotten way better", he said.

LFCS: The Linux Kernel: What's Next

Corbet opened the panel by having the participants introduce themselves to an audience of around 400 people. The panel consisted of X.org project lead Keith Packard of Intel, Andrew Morton the kernel "odd job man" from Google, USB maintainer Greg Kroah-Hartman of Novell, and Ted Ts'o of IBM who is currently on loan to the LF as its CTO. After that, Corbet got started by asking Kroah-Hartman about the -staging tree.

Approximately one-third of the code that was merged as part of the 2.6.30 merge window came in via the -staging tree, which Kroah-Hartman maintains. Corbet said there was a lot of confusion about the tree and asked for an explanation. Basically, it is a collection of drivers that used to live outside of the tree, Kroah-Hartman said, consisting of bad code with bad API usage and other major problems barring their acceptance into the mainline, in other words, "crap". But there is a lot of hardware in use that requires those drivers and the code was not getting improved out of the tree, so moving them gives a centralized location where people can get them and hopefully improve them.

Kroah-Hartman said that there were several drivers that had graduated from -staging and into the mainline, so the process seemed to be working. "If you want to get involved in the kernel, that's a good place to start", he said. He noted that there are TODO files in each driver's directory listing the kinds of changes needed before the driver will be accepted into the mainline.

Corbet mentioned that he had been going to conferences for years hearing about all the great things that were going to be done in the Linux graphics area, but that we had now reached a point where much of it had actually been done. He asked Packard to fill the audience in on what had been done and "why it's cool". Packard described how X.org had "turned the graphics stack upside down" by moving the device configuration out of user space and into the kernel.

By doing that, X becomes just an API for existing applications, and other APIs such as OpenGL or Wayland can be considered, he said. Support for Intel graphics is good, and there is lots of work going on for Radeon (ATI) chipsets, but NVIDIA is "not helping at all". He pointed out that Fedora 11 will be shipping with the Nouveau driver for NVIDIA hardware because it has surpassed the free 'nv' driver in capabilities. He also noted that moving the configuration and initialization into the kernel allows people to experiment with graphics acceleration without spending an inordinate amount of time figuring out how to initialize the hardware.

Next, Corbet asked Ts'o about the status of the ext4 filesystem. Ts'o reported that Fedora and Ubuntu would be shipping it in their next releases that are coming within a few months. He said that the user community was growing and "to be brutally honest, that will sometimes find bugs". He said one goal is to get it into the next round of enterprise distributions. He also noted that ext4 is a temporary solution, based on BSD FFS, which is technology from the 70s. Btrfs, nilfs, and others were where the interesting filesystem development is happening. All of those make it an "exciting time" to be a filesystem developer.

Morton responded to a query about the linux-next tree by saying that it is working out well, overall, as a place for integration and testing. But, he said that he was "a bit disappointed with the uptake it has", especially from a testing perspective. Fewer maintainers are taking advantage of the opportunity to integrate and test using linux-next than he would like to see. It is often the case that when there is a problem that shows up in Linus Torvalds's tree, it is because the code never made into linux-next.

From the audience, ftrace developer Steven Rostedt asked about the pressure to merge new code upstream into the mainline, but that there is major resistance to certain things—he mentioned SystemTap and utrace—being merged. He wanted to know what can be done to resolve that. Morton responded that for device drivers or supporting new architectures, the path is easier, but that the two examples Rostedt gave touch core kernel code. Morton likened the utrace battle to an "incestuous family struggle", but noted that the code needed improvement before it could go in.

One of the reasons that utrace didn't make it into the kernel was a lack of an in-kernel user of the code, Rostedt noted. Morton responded that having an in-kernel user for a feature is a "nice checkbox", because it gives the kernel community a means to test the code. But, Kroah-Hartman pointed out that "changing core kernel code is hard, and it should be". Ts'o also pointed out that several core kernel developers are helping out with utrace, which should significantly smooth its path into the mainline.

That discussion led Corbet to ask about tracing, noting that there were several tracing solutions that were still out of the tree, but that ftrace got new tracers added for each kernel release. Morton would like to "see evidence that people are using them and getting good results". Both he and Ts'o pointed at the lack of documentation for various tracers, saying that adding that and making the tracing more usable would help get more of that code into the mainline kernel.

The recently proposed nftables packet filtering subsystem was raised by Corbet as an example of a place where a user-space interface—the existing iptables—might be supplanted. He asked how that transition could be accomplished. Morton called it a "pretty traumatic transition" that would require a compatible set of tools, with several years of warning along with buy-in from the distributions. That takes three to four years according to Kroah-Hartman. Ts'o called the packet filtering interface more of an administrative interface that didn't have to be kept as stable as others, but that the iptables command does need to be stable.

All of that led Packard to complain about the difficulties of keeping the current user-space interface for X servers while moving modesetting into the kernel. According to Packard, there are exactly two users of the interface, both of which are under his control, so why does he need to provide backward compatibility? Ts'o said that the problem would be for distribution users who wanted to upgrade their kernel. Because the distribution might use an old X server, that interface—which Packard described as "open /dev/mem"—needs to be maintained. Kernel hackers want as much testing of new kernels as they can get, so any barrier to that testing is problematic.

At the end of the session, LF Executive Director Jim Zemlin announced the first ever LF "Unsung Hero" award, which he then presented to Morton. He explained that Morton is an avid car racer, so the LF arranged for him to have a day at the track as a reward. It was no surprise that there was much applause for Morton—one of the few people actually able to follow the linux-kernel mailing list. He also reviews an incredible amount of the code that ends up in the kernel.

These sessions provide an interesting view into the thinking of the members of the panel—one not easily derived from just keeping up with the technical side of Linux development via the LWN Kernel page or even by sifting through linux-kernel. They also give attendees a look at what's coming in the future that can be hard to discern, though Corbet's Linux Weather Forecast is helpful there. In the end analysis, though, the biggest benefit may just be putting kernel developers and users together in a fairly informal setting so that both sides get a better feel for the other. Faces and personalities don't necessarily jump out of the normal communication channels, so panel sessions like those that went on in San Francisco are useful well beyond their technical content.