NLUUG/ELCE: Embedded Linux and the community

As one of two embedded maintainers for the Linux kernel, David Woodhouse is in an excellent position to see where the community is failing to keep up its end of the bargain. At the recent co-located NLUUG and Embedded Linux conferences, his keynote on the second day made it very clear what areas he sees that need improvement. We fairly regularly hear about things that companies should be doing—see the report on Harald Welte's first day keynote—but the community should certainly keep an eye on its behavior as well. In his presentation, Woodhouse notes multiple projects that are not upstreaming their changes; he also notes things that individuals could do to make Linux better.

He started by pointing out that "it's not entirely clear what 'embedded' means", as there are many kinds of devices that have embedded attributes. Things like headless, handheld, low power, small size, limited ram, or limited persistent storage tend to be a part of the description of embedded devices, but there is "no real definition that I'm aware of that makes any sense".

Woodhouse then went on to see if he could define what an "embedded maintainer" is and does. He doesn't see the role as chasing patches to get them included upstream, it is more of an advocate role. Keeping an eye out for stupidity in the kernel using Bloatwatch and other tools as well as encouraging people—in various companies as well as in different parts of the community—to work together on solutions to problems they have in common are all part of the job.

From Woodhouse's perspective, companies are "getting a lot better" in terms of their Linux support. Less promising is the community: "We suck, really". He looked at a number of community embedded projects—like OpenWrt, Maemo, Moblin, and OLPC—to see how well they work with upstream; what he found was rather discouraging.

By looking at several concrete criteria, such as how many unsubmitted local kernel patches there were, how accessible their source is, and how old the kernel is that the project is using, Woodhouse is judging those projects the same way that companies are measured. Of the four projects that he looked at, only one, OLPC, was "mostly OK", the rest varied from "less good" to "FAIL".

Moblin for example, only had 23 outstanding patches, but those were against kernel 2.6.24. OpenWrt had a better kernel version, 2.6.27, but had 160 outstanding patches, plus an extra 425 files weighing in at 125,000 lines of code, which prompted a "sorry!" from an OpenWRT developer in the audience. OLPC has just a few outstanding patches against 2.6.27.4, while Woodhouse couldn't even find the kernel source for Maemo.

Getting work upstream is extremely important. Running older kernels and backporting fixes and features may seem like it saves time, but "it never works in the long run, it's a false economy". Woodhouse listed the usual suspects as reasons to get things upstream: code review, compile testing, updates for kernel API changes, and automated bug checking. He also mentioned the Kernel Janitors, whose efforts are generally useful, even though they are "often a little misguided, sometimes they don't engage their brain before sending patches". All of these benefits only come from getting code into the mainline.

The theme of the talk is summed up in one statement: "Divergence is pain". Any time that your code is not current with the most recent kernels or your patches are not making their way upstream, it should be felt as pain because diverging from upstream will end up causing exactly that. The pain may not be felt until later, but Woodhouse wants developers to recognize the problems caused by divergence so that they are averse to it right from the start.

Looking at the reasons why code is hoarded is instructive, he says. One of the reasons that is often heard, as well as Woodhouse's opinion, are summed up in a bullet point on one of his slides: "too hard to write decent code get code accepted". Another reason is that there is not enough time in the schedule for getting code merged. Many "see it as an extra part of the process after the driver is complete", which is the wrong way to look at it. Drivers and other features should be shared early on the appropriate mailing list so that any problems are dealt with near the beginning of development.

An issue related to code quality is that many times drivers are developed for ancient versions of the kernel, but that really shouldn't be a barrier as any "decent code will port relatively easily". Sometimes there is resistance to changes by the upstream developers. An example he noted was a feature that allowed multicast to be optionally removed from the IPv4 networking stack. It saved a fair amount of space for embedded devices that did not need that functionality, but David Miller and other networking developers were not very interested. This is where the embedded maintainer role can come into play as Woodhouse can step in to try to help convince the upstream developers.

Woodhouse had specific suggestions for making the situation better. "For a start, put everything in git trees" as it allows others to look at and test the code. Each feature should have its own topic tree that gets pulled into the main tree and developers should regularly assess the outstanding code to determine if it is ready to be moved upstream. Working with the upstream developers, getting them involved, and getting them to care about the feature or driver is crucial. In cases where a logjam develops, call on Woodhouse or Andrew Morton, they "can't promise any miracles, but often it can help".

Something that Woodhouse would like to see more developers do is to adopt a driver. There are countless drivers in SourceForge and elsewhere that are not upstream, so he suggests that folks "pick one driver, just tidy it up and make it acceptable upstream". Incidentally, Woodhouse is no fan of SourceForge: "I don't think I wrote 'don't use SourceForge' on any of the slides, but pretend that it's there". He mentioned the -staging tree as a possible destination for adopted drivers, though he is skeptical of the tree, "but it exists, we should see if we can get something from it".

Woodhouse summed up his talk with a simple statement: "We need to work better as a community before we can point fingers at companies who don't play nicely". It is certainly true that the community needs to set a good example for companies to follow. By highlighting some of our failures, Woodhouse has done the community a great favor, we can and, with luck, will do better.