ELCE: The state of embedded Linux

It is something of a tradition to have a "State of Embedded Linux" talk at each Embedded Linux Conference (ELC), and the recently concluded ELC Europe did not disappoint. In his keynote, MIPS architecture maintainer Ralf Baechle looked at the "pain points" for embedded developers, as well as what was being done to address them. He also looked to the future and made some predictions of what was coming for the embedded Linux landscape.

Baechle started working with MIPS Linux in 1993 or 1994, but was using Linux on x86 even earlier than that. He started off his talk by reporting on two embedded Linux summits that were recently held. One was at LinuxCon Japan and another was held in Cambridge, UK two days before his talk. There were a large number of companies represented at the summits, and "a lot of the big players". There were 16 attendees at the Tokyo summit and 12 at the one in Cambridge. The summits had "fairly good representation of the industry", he said, along with a bunch of architecture maintainers and users.

The summits were organized to talk about problems, Baechle said, as "the good stuff doesn't need to be talked about". The meetings were held "off the record" so that the discussions could be candid. The attendees identified a number of pain points for embedded Linux.

The first is the problem with "IP blocks", which are particular components that are licensed for use in system-on-chip (SoC) devices. A typical SoC "consists of a number of licensed IP blocks", and it is very hard for the kernel to determine which blocks are supported by included on a particular SoC. In addition, developers often don't know that a particular IP block is supported, so drivers and other support code get developed multiple times. There is a plan to maintain a list of these IP blocks in a wiki, along with their support status and device tree bindings, Baechle said.

Another problem area is "legal pain", mostly surrounding the GPL. That has caused developers to look at alternatives to glibc because it has moved they fear it moving to the LGPLv3. In addition, the GPLv3 has been interpreted by an unnamed company as being targeted at voiding its patents. Baechle doesn't agree with that interpretation, but GPLv3 certainly makes some companies uncomfortable. Android avoids all GPL code where it can, he said. Also, the BusyBox lawsuits have caused some consternation in the embedded world because of the demands for Makefiles and installation instructions. Not everyone interprets the GPL to require those things, but it is, as yet, unresolved.

There is also a fair amount of "kernel pain" in the embedded community, starting with the "huge version gap" among the kernels used in embedded Linux devices. Kernels from 2.6.11 up through recent kernels were mentioned as being used, and "not even 2.4 is really really dead", he said. But, Linux is finding its way into more and more products. There is a large company that has made it a policy to put Linux in any of its products that will need to be supported for more than 10 years.

Another part of the kernel pain is the large amount of out-of-tree code that embedded Linux developers are working with. Part of the problem comes from multiple groups within companies, each with its own fairly small set of patches. There is little communication between those groups, so that causes a "huge group of patches to build up" within the company. But the single largest patchset that is carried around by embedded Linux developers is the RT_PREEMPT patchset and the summit participants "really would like to see it go upstream", he said. There may be an effort among the participating companies to try to help make that happen.

But, there were not only pains discussed at the summits, there was also discussion of various things that had been added to the kernel recently, many of them with the support of the CE Linux Forum (CELF). SquashFS, which is a compressed read-only filesystem, was merged, as was LZO support for it. LZMA support for the filesystem made it as far as -next before that particular implementation was rejected by Linus Torvalds. There is hope for the YAFFS2 flash filesystem to be merged as it is now being cosponsored by CELF and Google.

A way to remove unused functions from kernel builds (i.e. -ffunction-sections support) for saving space is getting close to being merged as well, though it is currently held up by some PA-RISC linker problems. Using that can result in savings of around 7% of the kernel size, he said. While the merger of CELF and the Linux Foundation was not known until the Cambridge summit, Baechle expressed optimism that it would be good for the embedded Linux community.

Linaro presented itself at one of the summits. It considers itself a "community facing group", he said, that is working to reduce pain in the ARM world. It has 70 full-time engineers doing open source work. Right now, Linux can "at best produce one image per SoC family", which results in some projects needing as many as 50 images, all of which are slightly different variations. Linaro wants to reduce that pain so that companies can "differentiate themselves not by fixing random bugs, but by adding new features".

One thing that may help reduce the proliferation of slightly different variations is the device tree work. Device trees describe the buses, devices, memory, interrupts, and so on for a particular SoC. That tree gets passed to the kernel at boot time, which will allow kernels to support more SoCs within a single image. It is currently being used by Power PC across all of its platforms and MIPS is using it as of 2.6.37-rc1. Baechle said that ARM maintainer Russell King is "not quite convinced" about device trees, but he believes that King eventually will be.

Virtualization is a hot topic in the embedded Linux world these days, but it is "not going to be for everybody". Systems that are too resource constrained will not be interested in virtualization, but others will be. He went through various virtualization technologies available for Linux including containers, Xen, KVM, and two proprietary solutions from Wind River and MontaVista. Each has its place, but containers for OS-level virtualization and KVM for full virtualization are likely to be the dominant players for embedded devices, at least partially because they are part of the mainline kernel.

Baechle sees virtualization as a game changer for larger embedded systems. For example, high availability systems can use a pair of guests that can fail over to each other. That will also allow in-service software upgrades.

Alternatives to glibc were next on the agenda. Embedded developers are looking for those alternatives because glibc is "the size of an aircraft carrier". It complies with all of the standards but that comes at a heavy price. uClibc is one alternative, but the problem is that it is "yet another API" that application developers need to support.

But Embedded GLIBC (EGLIBC) offers an alternative for embedded developers that doesn't require a separate API. It is a variant of glibc that is maintained by CodeSourcery and is "embedded friendly". Unlike glibc (whose maintainer "says 'embedded crap' frequently"), it can be configured without some features, which leads to a reduction in code size, while still allowing applications that don't use those features to run without modification. In many cases, the same application can run on the desktop or the embedded device and there aren't two different toolchains required. EGLIBC is another game changer, according to Baechle, though it is not for the smallest systems. But it simplifies development which leads to "instant ISV [independent software vendor] happiness".

In a look at the mobile distribution space, Baechle was clearly impressed with MeeGo. He thinks that it will be a "fairly hot commodity in the future" because it uses the typical Linux software stack. Android, on the other hand, "feels alien", though Google does a good job with its development tools. Because MeeGo is stewarded by the Linux Foundation, it is in more neutral hands than Intel's would be, he said.

The "working upstream" policy of MeeGo is very important, he said. That policy is increasing the pressure on other embedded Linux community members to get their code upstream. MeeGo has the most push from the industry and a tremendous amount of money behind it. He is optimistic about its future, saying that "MeeGo is going to change the game a little bit".

The embedded world is changing, Baechle said. "Embedded" used to be a synonym for "resource-constrained", with functionality that was reasonably easy to implement. But, modern devices are multi-functional that share a lot of technology with desktop and server systems. There are devices using the NUMA code to get good performance from multiple memory banks, for example. SMP was originally developed for servers, moved into the desktop world, and is now being used by embedded devices.

In wrapping up his talk, Baechle looked into his crystal ball and made a few predictions. Over the next year or so, he believes that three more architectures will get merged, as will YAFFS2, but that the RT_PREEMPT patchset won't be. The pressure to work upstream will continue to increase which will lead embedded companies to rethink how they handle source code and how they put together their development teams.

"Feature-wise, Linux has become rather mature [and] very stable", but "the complexity of the code has increased quite dramatically over the last few years", Baechle said. There has been progress made everywhere in the kernel, with no one feature that stands out. That is likely to continue over the next few years, and we will be seeing Linux in even more devices.