UDS from an embedded hacker's perspective

The Ubuntu Developers Summit (UDS), held November 16-20 in Dallas, while kicking off the development cycle for the next Ubuntu release, "Lucid Lynx", had a surprising amount to interest a kernel hacker with embedded tendencies. The Summit covered a wide range of topics from low level kernel details, to best community practices, but the ARM netbook support sessions were particularly interesting. At this UDS, the Ubuntu ARM developers set out to enable support for many ARM machines in a single distribution, a difficult task due to the lack of a standard firmware interface on ARM systems; a familiar problem to embedded developers. This report covers the solutions debated at UDS — including Kexec bootloaders and the flattened device tree — and the choices made for the next Ubuntu release.

Ubuntu has supported the ARM architecture since the 2008 Intrepid Ibex (8.10) release, but the relative lack of consumer hardware has effectively made it interesting only to developers. During the Lucid cycle we can expect that to change as Canonical is working with ARM netbook OEMs to provide full support for the new devices that are widely anticipated to appear on the market in the new year.

However, support for a wide range of ARM devices is complicated by the absence of any form of a firmware interface standard for ARM systems. The vast majority of ARM designs are embedded systems with no expectation that the end user will install their own software. General purpose ARM computers are historical rarities; the notable exceptions being the original Archimedes, and the Corel Netwinder. As such, unlike the x86 architecture where an IBM PC-type BIOS is mostly a given, device manufactures can (and do!) implement whatever firmware interface best meets their needs. Every device has a different method for booting the OS. Additionally, since firmware provides little if any information about the hardware, the kernel must be hard coded with device addresses and configuration information.

The current situation requires a different method for each system to boot an installer and multiple kernel images, each hard coded for a specific machine. For a device vendor who only maintains software loads for a handful of devices this situation is not particularly onerous. For general purpose distributions like Ubuntu it is problematic. The maintenance load of both installer and kernel images for every single hardware platform from every single vendor is completely unmanageable. As such, a number of the sessions in the mobile track at UDS this year were devoted to solving the ARM boot problem.

Bootloaders and Firmware

Getting firmware to behave was the theme behind two sessions on Tuesday. The first covered Oliver Grawert's script for creating a bootable image for a Freescale Babbage i.MX51 board with RedBoot firmware. From a purely pragmatic viewpoint, the script is absolutely necessary to create a usable install image but it certainly is not portable. Every board requires a different script for convincing the firmware to boot from the right place. So while it is a current necessity, it is not a viable solution in the long term.

The ARM Soft Bootloader session led by Michael Casadevall on Tuesday afternoon proposed a more complete solution. Rather than depend on the capabilities of firmware executed by the device at power up, he suggested using a second stage boot loader that is built around the kernel and uses Kexec to boot the real OS image in the manner of Petitboot for the PS3. The idea is that the kernel can be used to sidestep firmware differences and provide a consistent boot interface to the installer. It also eliminates the current need to write device drivers twice; once for the kernel, and a second time for the firmware.

For most developers the firmware is not actually interesting, it is just a necessary hoop to jump through to boot an operating system. By adopting the kernel as the boot loader, it eliminates all the (arguably wasted) effort spent developing drivers for firmware, freeing up time for other development. However, there are concerns about the soft bootloader approach. The most significant of these is how much time booting a second kernel will add to the boot process.

Discussion in the Soft Bootloader session was interesting, and is well summarized in the blueprint. It is worth a read for anyone doing firmware work. At the end of the session it was decided that Soft Bootloader is the preferred approach but it still requires some investigation and engineering. Most likely it will not be ready for the Lucid release but is a Lucid+1 candidate.

Flattened Device Tree

Several sessions discussed the problem of hard coded machine information in the kernel which requires the kernel to be modified for each new device. I was asked to lead two sessions on the Flattened Device Tree (FDT). The first was an overview of the FDT approach — how it is implemented in PowerPC and how it can be used to solve several of Ubuntu's ARM problems. The Flattened Device Tree has been discussed elsewhere with more eloquence, but in brief it is a data structure that is passed to the kernel at boot time which describes the hardware. Instead of relying on hard coded platform device tables, the kernel reads the data structure to determine what devices to register and how they are configured.

While no firm decisions were made in the first session, it was generally agreed that device trees solve the problem at hand and should be pursued. Proof of concept work is proceeding which should demonstrate the device tree on an ARM platform in mid-January.

The overview sparked the scheduling of a second Flattened Device Tree session quite unrelated to ARM platforms. Fixing Hardware Quirks in Device Trees discussed the idea of using the same FDT data structure to capture machine specific quirks that are currently hard coded in the kernel. For example, ALSA codec routing information is wildly different from board to board but changing it (for instance when trying to uncover why sound does not work on a user's brand new machine) requires a kernel recompile. If a binding could be written for the device tree that encodes the codec configuration, then it may be possible to fix non-working audio by dropping a new device tree blob into sysfs somewhere instead of asking the user to replace the kernel. Research is needed to decide if this is a viable approach, but it seems promising. Jeremy Kerr has volunteered to investigate it further during the Lucid cycle.

Other Topics

Other interesting discussions for ARM and kernel developers included Colin Watson's Cross Building Ubuntu session, the decision to drop ARMv6 support to take advantage of the performance gains in ARMv7, and the decision to use the 2.6.32 kernel version with Ext4 remaining as the default filesystem (the exception being the ARM kernels which will be allowed to lag if the vendor supplied trees are not yet at the 2.6.32 baseline).

On the other end of the spectrum, the work going into the Quickly application framework was presented during Monday's plenary session. It is intriguing and appears to be a promising approach to simplify Linux application development.

Summit Organization

Sessions at UDS were limited to small groups and the focus was on discussion and making decisions about the next release. Only a handful of 15 minute time slots per day were devoted to formal presentations during the plenary sessions in the ballroom. Participation was open; and to provide for those who were unable to attend, audio from all the meeting rooms was Icecast to the Internet. One of the two projectors in each room was dedicated to showing an IRC channel, allowing anyone in the world to listen in and participate. The IRC channel worked well for those offsite, but also had advantages for those who were onsite. It was often used as a "back channel" for making comments without interrupting the conversation at hand.

With 12 sessions going at any one time, far more was happening at UDS than any one person could track. This report gives a taste from the embedded Linux perspective, and is not intended to be a complete review. Anyone interested in knowing more about what was discussed at UDS is strongly encouraged to browse through the posted schedule for notes on most sessions' discussion and decisions. In addition video that was taken at the conference has been posted for public access.