Creating a Fedora ARM distribution part 1: History

The Fedora distribution is often associated with laptops and desktops using x86 processors. These systems are cheap, powerful, and readily available to developers, and so it would naturally follow that they would be well supported. But Fedora has long supported systems based upon architectures other than the venerable x86. Such systems include those using PowerPC (and its bigger cousin, POWER), SPARC, and even mainframe processors. Collectively, these alternatives are known in the Fedora nomenclature as secondary architectures, and they have historically tended to target more niche users (the number of readers owning a mainframe is likely small).

What makes the ARM port different than many of the existing Fedora secondary architectures is that ARM-based hardware is, like x86, cheap, readily available to developers, and is increasingly becoming more powerful. It has not yet reached the level of a high-end x86 desktop or laptop, but decent performance can be had from an embedded board or netbook system costing under $200. The latest systems use dual core ARM Cortex-A9MP processors such as the OMAP4 from Texas Instruments, or the Tegra2 from Nvidia, with more-capable processors on the horizon. Each of these systems feature around 1GB of RAM, a decent amount of storage options, and good network connectivity, along with a host of peripheral devices.

The availability of inexpensive ARM-based hardware gives rise to an interest in distribution support from those who wish to ship such hardware to end users and developers. Companies like Marvell have been active in the early support of Fedora as a means to enable popular "Plug" Computers such as the SheevaPlug, GuruPlug, and DreamPlug built using Marvell's ARMv5-based Kirkwood and later processors. Interest in a Fedora port to the ARM architecture does not come only from those selling microprocessors. Projects such as OLPC (One Laptop Per Child) already use Fedora on x86 and will have an easier migration path to ARM if they can remain using the same distribution.

A history of Fedora ARM

The Fedora ARM project has had several beginnings over the years. At first, the work was mostly driven by hardware manufacturers or those owning specific ARM-based systems with an interest in running Fedora on them. Those were good efforts, but their narrow focus meant that the project remained a fairly niche effort. Today, the Fedora ARM project is independent of any one hardware company, software company, or even target system. Instead, it is formed from a diverse (and growing) group of developers with shared interests in ARM technology. The team includes faculty and students of Seneca College in Toronto, employees of Red Hat, members of the existing Fedora community, and individuals from several continents. Some are using embedded ARM boards like the PandaBoard, TrimSlice and GuruPlug, others are using netbooks and next generation OLPC prototypes, but all have one collective interest in seeing ARM recognized as a first-class citizen within the Fedora family.

Fedora ARM had its earliest beginning in the form of an initial project sponsored by Marvell. Marvell makes the Kirkwood chip and its successors, which power many of the popular "plug computing" devices on the market. The development was based largely on still earlier work done by Lennert Buytenhek, a (former) Marvell employee who had made substantial progress on his own time. This led to several successful releases of various packages and filesystem images in the Fedora 8-12 timeframe, sufficient to gain a small following among those using Marvell's devices. Many early problems were solved at this time, including adding the necessary support within Fedora's package management tools (RPM, Yum, and so on) to recognize ARM architectures, as well as various small (and some not so small!) patches to core distribution packages to handle the differences between ARM and x86. Although Marvell discontinued its project following Fedora 12, it proved very valuable as a starting point for the effort that would follow.

The project as it exists today began in early 2010 when Chris Tyler of Seneca College in Toronto and a small group of his students picked up where Marvell had left off. Together with members of the Fedora community (and including the valiant efforts of Fedora release engineer Dennis Gilmore), Seneca created and hosted an instance of the Koji build system software used to build all Fedora packages, along with a large number of ARM-based build systems. Just like with other secondary architecture teams, the Fedora ARM project builds every software package using systems running on the target architecture rather than (for example) cross-compiling for a specific target architecture on x86. The rationale behind this approach is complex and best saved for another article, but it is a strategy supported by those working on the Fedora ARM port. Builds are performed using Koji, a distributed build management system, which drives individual builder systems running a build-in-chroot tool known as mock.

Building Fedora 13 for ARM

With the new build infrastructure, the Seneca team was able to orchestrate the mass building of Fedora 13 packages, taking the exact versions of source packages from the existing primary architecture (x86) as their input. In the case of some packages that had never been built for Fedora 13, even on the primary architecture, older versions (fortunately available from the Marvell effort) were used instead. Once the building of packages had been completed, an instance of the Fedora signing server was setup for ARM and a signed set of packages was released. Fedora 13 thus represented the most complete release yet, both in terms of packages, and in terms of the overall release management process, following very closely that used by the primary x86 architecture. The release was well received by the small (but seemingly growing) base of users, and was installed on popular embedded boards as well as various ARM netbooks.

The Fedora 13 release lacked two key components that will be eventually required for ARM to gain more widespread use amongst Fedora users. The most obvious omission was a standard Linux kernel package, of the kind that is present for every other architecture. A lot of work is ongoing in the wider Linux ARM community to allow a single binary kernel to support many different systems (through the use of device tree and similar platform device enumeration technologies, as well as industry standardization efforts), but in the interim it is still necessary to make some choices during kernel build that limit the ability for a distribution to ship a one-size-fits-all kernel. Instead, the need of a shorter term solution lead to several kernels being made available for Fedora 13 (based upon one source kernel package), each targeting a family of processors, and having names like kernel-omap, kernel-tegra2, and so on.

The Fedora ARM user had to manually choose the correct kernel for their system, but the number of users was still relatively small and was mostly confined to developers or those willing to experiment. Besides, the Fedora 13 release lacked an installer, so installation typically involved building a filesystem containing the desired packages and kernel combination. The lack of a generic installation option was in part because nobody had gotten around to getting the Fedora installer built properly and in part because of the lack of widely deployed platform standards for ARM systems, meaning that there was no one way to install and configure a system to boot Fedora automatically. Many embedded ARM systems (but by no means all) use U-Boot, however there are many flavors of U-Boot today, each of which is configured differently, and none of which has a standard mechanism for upgrading the kernel from an installed system.

In the future, installation of generic operating systems like Fedora should improve as newer standards such as UEFI become more prevalent on ARM. This will allow for a full, general-purpose installer such as Fedora's Anaconda to support ARM to the same extent that it supports x86 today. In the interim though, the Fedora ARM project has chosen to make a few assumptions around the use of U-Boot and provided a means to specify the location of the images required to boot through the means of a configuration file. This allowed the project to at least provide several kernel packages that could be configured to install for a range of possible target systems and bootloader choices. A number of filesystem images were shipped for specific systems in lieu of having a full installer such that it is possible to download a Fedora 13 image built for one of the popular embedded boards and get going quickly.

Fedora 13 for ARM is (still) technically the current release, although all development is now focused on getting Fedora 15 finished. For those who just want to quickly investigate an older release, Fedora 13 is available in the form of a filesystem that can be downloaded via the "Releases" section of the Fedora ARM wiki. It can be unpacked onto an SD card (for example) and installed onto a target system, along with the appropriate kernel for the chosen target. A number of pre-created images are also available, for example targeting the Texas Instruments PandaBoard (and using device tree during the boot process). Following installation, standard tools such as yum can be used to install various additional packages.

Reflections upon the Fedora 13 release

Fedora 13 for ARM was released and subsequently installed on many different systems. For a (short) period of time, updates were made available based on builds being done for the primary architecture, but the fun was relatively short-lived as Fedora primary moved on to the F15 release, which meant that, under Fedora policy, support was officially discontinued for the older F13 release. This gave the Fedora ARM team an opportunity to reflect upon the realities of continuing to trail the primary architecture. In the longer term, this would hinder the ability to reach the goal of becoming a primary architecture by virtue of being too out of date for consideration.

Fortunately for the Fedora ARM project, relief would come in the form of a mass rebuild of packages for the primary architecture. Whereas many releases of Fedora will include packages from older releases that have not been touched in the interim, various changes at the core distribution level necessitated that every package be rebuilt for Fedora 15 on all architectures. This meant that there was no need to carry any Fedora 14 (or earlier) packages in the final Fedora 15 ARM release provided that the set of Fedora 15 packages could be successfully rebuilt without them (or by building only interim throwaway Fedora 14 packages to aid in the mass rebuild effort). Therefore, the project had an opportunity to largely skip Fedora 14 and head full steam ahead toward a Fedora 15 release. Mass rebuilds certainly benefit secondary architecture releases and it is hoped that they will become Fedora policy.

Another reason for skipping over Fedora 14 and focusing on Fedora 15 alone came in the form of growing interest in supporting newer ARMv7-based processors. The existing Fedora 13 packages had targeted ARMv5, which was fine in 2007 when the project first started, but was increasingly becoming a performance issue by 2010. In the interceding years, there had been first the ARMv6 architecture (with SMP support) and then ARMv7. With ARMv7 came not only core architectural improvements such as more heavily-optimized instruction streams and virtualization extensions but also mandated support for VFPv3 (Vector Floating Point) hardware. To accompany this requirement, ARM introduced a newer Application Binary Interface (ABI) revision to the Procedure Call Standard for the ARM Architecture (AAPCS) specific to ARMv7 and higher systems that improves performance (especially when using floating point), but at the expense of a one-time break with backward compatibility.

A new ARM Architecture

Typically, ARM architecture improvements are backward compatible since ARM recognizes the value in supporting the code deployed for the billions of ARM-based devices in the wild. It would have been possible for Fedora to build ARMv5 packages that would support both older and newer processors, but at the cost of losing some of the architectural improvements and ABI changes that can be leveraged by targeting only v7. Thus, Fedora 15 could have continued to use software floating support as had been the case historically and left the newer ABI for the future. But instead, it was decided to capitalize on the mass rebuild being performed in Fedora 15 and treat ARMv7 as an entirely new architecture. A new architecture featuring a new ABI that was not compatible with older ARM processors would require a build (bootstrap) of every package in any case. Treating ARMv7 as a new architecture also allowed Fedora to progress while continuing to support older ARM processors through a separate ARMv5 build.

Breaking backward compatibility is never an easy choice, as anyone who has weighed the merits of the newer x32 ABI can likely attest, but there are times when its impact can be reduced. This was especially true in the case of the Fedora ARM project, which doesn't (yet) have an installed base of many millions of users and so could afford a one time inconvenience to the (mostly developer) user base in order to reap the many benefits of having a standardized base going forward. The newer ABI, known informally within the community as the "hard float" ABI has quite a following and the various Linux distributions have agreed to work together to develop a stronger base of inter-distribution binary compatibility. It has also been agreed to pursue standardization in the Linux Standard Base around ARMv7.

Moving to a new ABI is in general a non-trivial operation because it means that all of the software in the entire Fedora distribution must be rebuilt from scratch. In this respect, moving to the new ABI is similar to moving to a new architecture, which is how the transition to ARMv7 was approached. Fortunately, in this case there was already a working Linux kernel because only the userspace linkage of applications was actually being changed. The kernel itself only cares about the system call interface ABI, which wasn't affected by the change. Therefore, in this particular case (though this is not true in general of new architecture ports) it was possible to run both of the userspace ABI versions on the same kernel, even concurrently. Fedora can do this using different user-space chroots, while some other distributions are experimenting with something they call "multi-arch" which uses different dynamic linker prefixes according to the ABI version. Fedora ARM will aim to be compatible with this approach by virtue of the cross-distribution standardization effort (though it may in the short term use a symlink approach targeting one arch with the standard prefixes in the absence of full multi-arch support).

[The second part of this series will be available in the near future.]