It is something of a tradition to have a "State of Embedded Linux" talk at
each Embedded Linux Conference (ELC), and the recently concluded ELC
Europe did not disappoint. In his keynote, MIPS architecture
maintainer Ralf Baechle looked at the "pain points" for
embedded developers, as well as what was being done to address them. He
also looked to the future and made some predictions of what was coming for
the embedded Linux landscape.
Baechle started working with MIPS Linux in 1993 or 1994, but was using
Linux on x86 even earlier than that. He started off his talk by reporting
on two embedded Linux summits that were
recently held. One was at LinuxCon Japan and another was held in
two days before his talk. There were a large number of companies
represented at the summits, and "a lot of the big players".
There were 16 attendees at the Tokyo summit and 12 at the one in Cambridge.
"fairly good representation of the industry", he said, along
with a bunch of architecture maintainers and users.
The summits were organized to talk about problems, Baechle said, as
"the good stuff doesn't need to be talked about". The
meetings were held "off the record" so that the discussions could be
The attendees identified a number of pain points for embedded Linux.
The first is the problem with "IP blocks", which are particular components
that are licensed for use in system-on-chip (SoC) devices. A typical SoC
"consists of a number of licensed IP blocks", and it is very
hard for the kernel to determine which blocks are
supported by included on a particular
SoC. In addition, developers often don't know that a particular IP block
is supported, so drivers and other support code get developed multiple
times. There is a plan to maintain a list of these IP blocks in a wiki,
along with their support status and device tree bindings, Baechle said.
Another problem area is "legal pain", mostly surrounding the
GPL. That has caused developers to look at alternatives to glibc because
it has moved they fear it moving to the LGPLv3. In addition, the GPLv3 has been interpreted by
an unnamed company as being targeted at voiding its patents. Baechle
doesn't agree with that interpretation, but GPLv3 certainly makes some
companies uncomfortable. Android avoids all GPL code where it can, he
said. Also, the BusyBox lawsuits have caused some consternation in the
embedded world because of the demands for Makefiles and installation
instructions. Not everyone interprets the GPL to require those things, but
as yet, unresolved.
There is also a fair amount of "kernel pain" in the embedded
community, starting with the "huge version gap" among the
kernels used in embedded Linux devices. Kernels from 2.6.11 up through
recent kernels were mentioned as being used, and "not even 2.4 is
really really dead", he said. But, Linux is finding its way into
more and more products. There is a large company that has made it a policy
to put Linux in any of its products that will need to be supported for more
than 10 years.
Another part of the kernel pain is the large amount of out-of-tree code
that embedded Linux developers are working with. Part of the problem comes
from multiple groups within companies, each with its own fairly small set
of patches. There is little communication between those groups, so that
causes a "huge group of patches to build up" within the
company. But the single largest patchset that is carried around by
embedded Linux developers is the RT_PREEMPT patchset and the summit
participants "really would like to see it go upstream", he
said. There may be an effort among the participating companies to try to
help make that happen.
But, there were not only pains discussed at the summits, there was also
discussion of various things that had been added to the kernel recently,
many of them with the support of the CE Linux Forum (CELF). SquashFS, which is a compressed
read-only filesystem, was merged, as was LZO
support for it. LZMA support for the filesystem made it as far as -next
before that particular implementation was rejected by Linus Torvalds.
There is hope for the YAFFS2
flash filesystem to be merged as it is now being cosponsored by CELF and
A way to remove unused functions from kernel builds
(i.e. -ffunction-sections support) for saving
space is getting close to being merged as well, though it is currently held
up by some PA-RISC linker problems. Using that can result in savings of
around 7% of the kernel size, he said. While the merger of CELF and the
Linux Foundation was not known until the Cambridge summit, Baechle
expressed optimism that it would be good for the embedded Linux community.
Linaro presented itself at one of the summits. It considers itself a
"community facing group", he said, that is working to reduce
pain in the ARM world. It has 70 full-time engineers doing open source
work. Right now, Linux can "at best produce one image per SoC
family", which results in some projects needing as many as 50
images, all of which are slightly different variations. Linaro wants to
reduce that pain so that companies can "differentiate themselves not
by fixing random bugs, but by adding new features".
One thing that may help reduce the proliferation of slightly different
variations is the device tree work. Device trees describe the buses,
devices, memory, interrupts, and so on for a particular SoC. That tree gets
passed to the kernel at boot time, which will allow kernels to support more
SoCs within a single image.
It is currently being used by Power PC across
all of its platforms and MIPS is using it as of 2.6.37-rc1. Baechle said
that ARM maintainer Russell King is "not quite convinced"
about device trees, but he believes that King eventually will be.
Virtualization is a hot topic in the embedded Linux world these days, but
it is "not going to be for everybody". Systems that are too
resource constrained will not be interested in virtualization, but others
will be. He went through various virtualization technologies available for
Linux including containers, Xen, KVM, and two proprietary solutions from
Wind River and MontaVista. Each has its place, but containers for OS-level
virtualization and KVM for full virtualization are likely to be the
dominant players for embedded devices, at least partially because they are
part of the mainline
Baechle sees virtualization as a game changer for larger embedded systems.
For example, high availability systems can use a pair of guests that can
fail over to each other. That will also allow in-service software upgrades.
Alternatives to glibc were next on the agenda. Embedded developers are
looking for those alternatives because glibc is "the size of an
aircraft carrier". It complies with all of the standards but that
comes at a heavy price. uClibc is one alternative, but the problem is that
it is "yet another API" that application developers need to
But Embedded GLIBC (EGLIBC) offers
an alternative for embedded developers that doesn't require a separate
API. It is a variant of glibc that is maintained by CodeSourcery and is
friendly". Unlike glibc (whose maintainer "says 'embedded
crap' frequently"), it can be configured without some
features, which leads to a reduction in code size, while still allowing
applications that don't use those features to run without modification.
In many cases, the same application can run
on the desktop or the embedded device and there aren't two different
toolchains required. EGLIBC is another game changer, according to Baechle,
though it is not for the smallest systems. But it simplifies development
which leads to "instant ISV [independent software vendor]
In a look at the mobile distribution space, Baechle was clearly impressed
with MeeGo. He thinks that it will be a "fairly hot commodity in the
future" because it uses the typical Linux software stack. Android,
on the other hand, "feels alien", though Google does a good
job with its development tools. Because MeeGo is stewarded by the Linux
Foundation, it is in more neutral hands than Intel's would be, he said.
The "working upstream" policy of MeeGo is very important, he said. That policy is
increasing the pressure on other embedded Linux community members to get
their code upstream. MeeGo has the most push from the industry and a
tremendous amount of money behind it. He is optimistic about its future,
saying that "MeeGo is going to change the game a little bit".
The embedded world is changing, Baechle said.
"Embedded" used to be a synonym for "resource-constrained", with functionality
that was reasonably easy to implement. But, modern devices are multi-functional
that share a lot of technology with desktop and server systems. There are
devices using the NUMA code to get good performance from multiple memory
banks, for example. SMP was originally developed for servers, moved into
the desktop world, and is now being used by embedded devices.
In wrapping up his talk, Baechle looked into his crystal ball and made a
few predictions. Over the next year or so, he believes that three more
architectures will get merged, as will YAFFS2, but that the RT_PREEMPT
patchset won't be. The
pressure to work upstream will continue to increase which will lead
embedded companies to rethink how they handle source code and how they put
together their development teams.
"Feature-wise, Linux has become rather mature [and] very
stable", but "the complexity of the code has increased quite
dramatically over the last few years", Baechle said. There has been
progress made everywhere in the kernel, with no one feature that stands
out. That is likely to continue over the next few years, and we will be
seeing Linux in even more devices.
to post comments)