Consumer ARM boards and the bleeding edge

There were several talks at SCALE 13x in Los Angeles that dealt with the practical side of embedded Linux development. One of the most well-received sessions was Stephen Arnold's look at the state of kernels and free-software graphics drivers on "open" consumer ARM devices. Arnold is a longtime Gentoo developer who also works with the Yocto project. He presented a rundown of the experiences other users can expect with ARM kernels and graphics on development boards and common single-board computers (SBCs), including the reliability of vendor-supplied drivers and the availability of Linux distributions.

Arnold's session was part of the "open hardware" track at SCALE, and one of the central questions addressed in his talk was how open many of the so-called "open hardware" ARM products really are. As most members of the community are aware, popular devices like the Raspberry Pi may be advertised as open, but the graphics chips on which they rely may have no free drivers at all. Those that do enjoy support from a free driver may have only partial functionality, or they may have an old, unmaintained driver that is difficult to port to a more recent kernel. If updating or rebuilding the vendor's kernel is impossible, after all, some of the key advantages of having an open device are all but lost.

ARM wrestling

Arnold led off with a quick tour through his personal embedded-hardware inventory, which he said he had amassed into "a collection of almost every ARM board ever." Hardware has changed so much over the years, he said, that today's boards exist in a blurry region between traditional embedded and desktop-class devices. Multi-core CPUs, more powerful GPUs, per-core floating-point units, and accelerated video processing are all becoming the norm.

Nevertheless, there are still significant differences to be found. Just within recent (ARMv7) generation products, he pointed out, boards like the Udoo and Cubox-i come with Vivante GPUs, Samsung Chromebooks and Sunxi "TV dongles" come with Mali GPUs, but the Acer Chromebook and Jetson TK1 board come with NVIDIA's Tegra K1—which is, essentially, a desktop-class graphics unit featuring up to 192 CUDA-capable cores.

Arnold also took a few minutes to explain some of the differences between the various instruction sets that users were likely to encounter on open-hardware ARM products and how that might affect writing software for them. The big divide, he said, is between devices that include the NEON instruction set and those that do not. Most ARMv7 boards include NEON, but a few (such as the Trim-Slice line of low-power boards) do not. As with ARMv6 devices (such as the original Raspberry Pi), the Trim-Slice boards use the older vector floating point (VFP) instruction set instead of NEON. The various video-driver projects often need to make use of NEON or VFP in order to provide responsive performance, so knowing which instruction set is supported is important.

The Five Blobs

Returning to the topic of graphics drivers themselves, Arnold looked at the five current GPU families for which ARM vendors are shipping binary-blob drivers: the iMX.6, the VideoCore IV, the Mali, the Tegra K1, and the OMAP SGX. There are projects working on open-source drivers for each GPU family, he said—in some cases, more than one. The Tegra family is the closest to having a complete solution, with both the OpenTegra/grate project and Nouveau support actively under development and producing working builds. Since the Tegra K1 is essentially a desktop GPU, this is perhaps not surprising.

The VideoCore IV (used in the Rapsberry Pi) has decent support with fbturbo, he said. That is what ships with the Raspbian distribution, but users may also want to look at the Weston for Raspberry Pi work that can take advantage of the GPU's hardware video scaler. The fbturbo driver also works with the OMAP, Vivante, and Mali GPUs, via the omapfb, lima, and etna_viv projects. Arnold noted that there is one other GPU family available, the Adreno, which has a free-software driver project, but he has not added the hardware to his collection yet, and thus has yet to test it himself.

If the user does stick with a vendor-supplied driver, though, there are still other factors to be considered—starting with what kernel the vendor ships. Typically, the kernels supplied by the vendor are a single release with a lot of patches applied—and the branch in question is likely to be old. Arnold said that the oldest kernel in his ARM collection is Linux 2.6.31.14, which shipped with a Fujitsu LifeBook and has never been updated. The newest kernel he has seen is 3.14. Usually, these kernels have had minimal back-porting done (if any), which means they may have security vulnerabilities, and attempting to forward-port the vendor's driver to a new kernel can take a long time.

Furthermore, the vendor kernels are often brittle: they may not use device trees, they may compile with lots of warnings (or even generate errors), or they may only compile with specific modules (or only when the kernel is built in monolithic form). Sometimes the kernel .config file is missing required configuration options, or the options included are clearly incorrect. A do-it-yourself kernel in most PC distributions is hard to mess up these days, he said, but if you change one thing in a vendor-supplied ARM kernel, you may find that USB stops working entirely, or that the firmware for the network or audio chips fails. Out of the current crop of ARM devices, he called the CuBox (with its iMX.6 GPU) and the Raspberry Pi the easiest to work with.

The other element to watch out for is the bootloader. Most vendors use their own fork of U-Boot; like the vendor-supplied kernel, it can be old and hard to work with. Some vendors that are "really on the ball" may have a newer U-Boot branch, since there has been considerable work upstream to consolidate the many U-Boot branches. If so, the user can take advantage of more recent bug fixes and optimizations.

Vendors tend to have only one supported bootloader configuration, he said, with the most common being booting from an SD card with two partitions (one for / and one for /boot). Manually changing this configuration is possible, but users who head down that path should be on the lookout for unusual boot options in UEnv.txt or boot.scr files. He also cautioned that he has seen different bootloader options even from the same vendor on two products that use the same system-on-chip (SoC), so an abundance of caution is warranted.

Status reports

Arnold then gave a rundown of the differences between the mainline kernel and the vendor kernels supplied for common ARM devices. Device trees are the main difference; whether or not a working device tree is available for a recent mainline kernel varies from device to device, and improving that situation is the subject of ongoing work by the community. He advised users looking to deploy a mainline kernel on their ARM device to check the Linux on ARM page at the EEwiki to see if there is a recent how-to guide. Robert Nelson from DigiKey maintains patches for a long list of vendor kernels and U-Boot forks at that wiki.

The Linux graphics stack itself is also in flux, Arnold said, with the migration from X to Wayland, the legacy driver model going away, and fluctuations in the OpenGL realm all complicating factors. He reported on his personal tests performed on his ARM hardware collection, calling out the Tegra 20/30 and Vivante hardware as working well with recent X.org releases. OMAP/PowerVR devices and Mali-based devices are also usable, although they depend on TI's open-source framebuffer code and fbturbo, respectively. He has recently given up entirely on the Efika MX product line, he added, which only works with "ancient kernels and closed-source drivers."

Even if the user decides to update from the vendor-supplied kernel and graphics drivers, it may be difficult to find a fully working Linux distribution for any particular ARM board. Arnold's final status report was a look at the distribution offerings for the various open-hardware ARM products in his collection. Typically, he said, the vendor will supply an Android OS option plus one other, traditional Linux distribution for each device. These distributions, of course, are typically limited to the vendor's legacy kernel and binary-blob drivers.

But some ARM products stand out from the crowd for their ability to support multiple distribution options. The Raspberry Pi offers the most, via its "New Out Of the Box Software" (NOOBS) system, which includes six distributions. The BeagleBone and BeagleBone Black officially support several flavors of Yocto and OpenEmbedded Linux, as well as Debian, Ubuntu, and Gentoo. Various Chromebooks support multiple mainstream distributions (Ubuntu, Debian, Gentoo, Arch, and Fedora being the most common). Udoo boards support an Ubuntu variant (udoobuntu), Android, OpenElec, and occasionally other distributions as well.

Pulling yourself up by your bootstraps

Ultimately, however, most users get interested in bootstrapping their own distribution at some point. For those people, Arnold said Yocto was the most reliable option. If there is a vendor board-support package (BSP), he said, Yocto can run on it. Yocto can also run on a modest desktop Linux machine, building a full image and handling build dependencies automatically. If there is not a BSP for your board, he said, you can probably create your own for Yocto.

Gentoo is the next option; Arnold maintains the Gentoo overlays for most of the ARM devices, and new stage3 builds are created every few weeks. For newcomers who may not be ready to migrate their desktop to Gentoo in order to do a native Gentoo build, there are other options, such as performing the build in a virtual machine. Last but not least, Debian and Ubuntu are always available: "if you can boot it, you can debootstrap it," he said.

In the brief question-and-answer period at the end of the session, Arnold agreed with one audience member that the pace of new ARM hardware devices was becoming problematic. There are new chipsets every year, but it typically takes the free-software community two or three years to fully implement software support. In the meantime, users are often left with just the vendor-supplied kernels, graphics drivers, and distributions to work with. So users can "run Linux" on their open-hardware ARM boards, but the situation is far from ideal.