Conference (ELC) organizer Tim Bird surveyed the embedded Linux landscape in
a talk he gave at the conference. He looked at new and proposed kernel
features that embedded developers might be interested in as well as issuing
a "call to arms" to those developers to get more involved with the
rest of the community. This talk is a regular feature at ELC to help the
embedded community stay on top of the "ripping" speed of
There have been four new kernels since last year's conference, and Bird
listed the interesting features for the embedded space in each of
2.6.30-33, as well as noting that LogFS had finally made it into the kernel
in 2.6.34, something that he was concerned might not ever happen. The
speed of kernel development is amazing, he said, and the great thing about
it is that "even while I am sleeping in my bed, people are pounding
away on it".
He pointed to a few "patches to watch" that may be coming in
new kernels, specifically the kbuild CROSS_COMPILE option, which
will make it easier to build for multiple architectures. He also noted
Arnd Bergmann's asm-generic
patches that are geared towards making it easier to add new architectures
to the kernel—without propagating the bugs and quirks from existing ones.
Bird then looked at different "technology areas" to point out interesting
features or work going on in those areas. Boot time is a "hot
topic" right now; it would have been in the past if the embedded
community was more involved in mainline kernel development. The Moblin five second boot effort really
kickstarted that work. He noted that he has a Sony (his employer) video
camera that boots Linux in 1.5 seconds; "I'm very proud of
that", he said.
Several new kernel features are available to help reduce boot time,
including asynchronous function
calls, which allow some parts of device initialization to run in
parallel. There is also scripts/bootgraph.pl to help visualize
where boot time is being spent.
Devtmpfs was also noted as a
way to decrease boot times, with some seeing a 0.6 second reduction on desktops. Bird
said that there needs to be some testing done on the embedded side to see
how much it can help there. He also listed two patches that speed up
symbol resolution for module loading by getting rid of the current linear
search. One switches to a binary search and the other uses a hash table.
For Bird's use cases, he always statically links in drivers, but has heard
that more embedded developers are going the loadable module route.
Greg Kroah-Hartman piped up that he needed one of those two patches for MeeGo, but that
the submitters had disappeared. There was general agreement that
contacting them and getting something upstream would be good.
Several different filesystems for embedded use cases were listed by Bird.
Squashfs has been out of the mainline for years, but was merged in 2.6.29,
and has since been improved by others in a "classic case"
demonstrating the advantages of mainline code. Ubifs is also in the
mainline and folks at Toshiba have been characterizing its performance,
which they reported on at the CE Linux Forum (CELF) Japan Jamboree. It has
"really slow mount times" in some cases, which CELF would like
to fund someone to fix.
LogFS is "way better optimized" for certain flash devices and
has fast mount times, he said. He noted that AXFS, the advanced
execute-in-place (XIP) filesystem, had kind of disappeared, so it didn't
appear to be on track for mainlining. He has been
playing with AXFS at Sony to try to further decrease boot time.
Bird also noted that the VFAT
patent avoidance patches had not made it into the mainline. It would
be useful for some embedded devices, he said. Most embedded developers
work around the patent by disabling VFAT and using 8.3 filenames, which is
somewhat unfortunate. Another thing he is keeping an eye on is VFS-based
union mounts, which would allow embedded developers to stop creating
"filesystems with weird links" between them as is currently common.
Power management and realtime
The runtime power management code has been merged, which will allow suspending
and resuming individual system components to reduce power consumption.
There is ongoing work on asynchronous suspend/resume, which Bird said he
didn't know very much about, but it's "gotta be really cool".
An audience member helped out by saying that it is in some ways like the
asynchronous initialization code (for faster boot), but "in the other
The RT_PREEMPT patchset "continues its slow march into the
kernel", with threaded
interrupt handlers being merged in 2.6.30 and preparatory work for the future sleeping spinlocks merge that
went into 2.6.33. There
are still some big kernel lock issues (BKL) to be resolved and CELF may fund some
work in that area.
The slide for kernel size and memory use had a picture of a "hybrid
Winnebago", which is the image Bird has of the kernel today. It
just keeps growing in size. To help embedded developers make better use of
limited memory, there is the smem tool that was funded by
CELF. He has used it in a few projects this year and it "has been
Various compression methods have been added to compress the kernel image in
different ways. LZMA can be up to 30% better than gzip, and LZO is not as
good at compression, but is much faster. There are tradeoffs dependent on
processor speed and I/O bandwidth that make it more difficult to pick the
right compression method, as Dirk Hohndel pointed out.
The ramzswap device (also known as compcache) allows in-memory
compressed swap. It is "really cool" but the maintainer only
was able to benchmark on desktop systems. It would be good if someone
could do some benchmarking on embedded systems, Bird said.
Tracing and security
Ftrace now has support for dynamic probes that came in 2.6.33, and the perf
tool can place and use those probes as well. There is tracing of
kernel variable access and modification available now. The perf "diff" mode can show the
performance differences between two runs, and also came in 2.6.33.
The TOMOYO merge in 2.6.30 was "a big deal" because it finally was able
to get path-based security into the kernel. NTT Data is now adding TOMOYO
rules to Android. Bird is in favor of a diversity of choices for security
as it gives people a chance to demonstrate which is the best solution for
various use cases. As part of that, CELF funded a study [PDF] for applying the
Smack security module in a television use case and found that the overhead
was higher than they expected.
CELF contract work
CELF has funded various projects over the last year including smem,
out-of-memory notifications in cgroups, SquashFS, the Smack analysis,
device trees for ARM, and the -ffunction-sections work to put each
kernel function in its own section to assist with dead code removal. Going
forward, CELF has an open project
proposal plan that will start funding new projects in the next few
weeks. It is also sponsoring Matt Mackall to be one of the two Linux kernel
embedded maintainers (David Woodhouse is the other).
A call to arms
Bird ended his talk with a list of things that embedded developers can do
to work better with the community. At the top of that list was "work
at top of tree". He realized that when he gives these talks, he is
generally talking to people who aren't using the kernels he talks about
because embedded folks tend to pick a kernel and stick with it. "It's
difficult" to work with the most recent kernel, but it's worth it.
"Version gap is the single biggest problem" in embedded Linux.
He suggested that embedded developers beat up on their board vendor to get
board support packages using the latest kernels and to do their testing on
boards that are already supported in the mainline.
The other suggestion he had was not to "wait for others to test new
features", and instead to do the testing themselves. He listed a
number of things that need testing in the mainline: LogFS, Ubifs mount
times, ramzswap, runtime power management, and so on. "Post the
results to the elinux.org
wiki" or come to the next conference (October 27-28 in
Cambridge, UK) and tell him about it.
to post comments)