User: Password:
Subscribe / Log in / New account


Creating a Fedora ARM distribution part 2: Bootstrapping

October 21, 2011

This article was contributed by Jon Masters

[Editor's note: this is the second of a two-part series on the creation of a Fedora distribution for the ARM architecture, contributed by Red Hat developer Jon Masters. Part 1 covers the history leading up to the current effort; this part will look at how the Fedora ARM distribution was bootstrapped and where things will go from here.]

Like most distributions, Fedora uses binary software packages (RPMs in this case) to manage installed software. These packages are built using complex sets of build dependencies (other software packages), some of which are not explicit dependencies but rather implied through their fixed presence in the standard "buildroots" (chroot environments containing a basic set of packages) used in the Fedora build infrastructure. All of these build dependencies have to initially come from somewhere. Ordinarily, the build system takes care of managing this, picking up dependencies as required, but in the case of a complete bootstrap there are no pre-built dependencies, so they must be created from scratch. Not only that, but build dependencies must be built using a minimal "bootstrap" configuration that avoids circular source dependencies on packages not built yet, and also provides enough basic functionality to rebuild the full set of dependencies using standard tools.

Currently, there is no convenient way to automatically bootstrap Fedora from scratch. Individual packages lack the level of data that would be required, such as a special set of configuration options sufficient to build a minimal "bootstrap" version of the core packages (free of circular dependencies) that could then be used to rebuild the normal configuration. Fedora is not alone in missing support for automated bootstrap, which is currently being investigated by a number of distributions. One of the problems has been that a full bootstrap remains such a rare event that it is difficult to justify the level of effort required to maintain support for automatically bootstrapping in the future (as opposed to making a large number of package changes for one bootstrap effort). Fedora may well eventually support automated bootstrap (a topic to be fully explored). In the meantime, bootstrapping for ARMv7 involved many successive stages.

The first involved building a limited cross-compiling Fedora 15 gcc toolchain for use on an x86 host and targeting the "hard float" ARMv7 ABI. This toolchain was then be used to build a native ARMv7 "hard float" toolchain and associated minimal dependencies necessary to run that toolchain within a chroot environment on an ARM system. The output of stage 1 was a minimal root filesystem, sufficient for use as a chroot (although not bootable), along with a set of bootstrap scripts that can be used in the future should it ever become necessary to repeat this exercise. DJ Delorie wrote the majority of the stage 1 bootstrap scripts, which are available in Fedora's ARM git repository at git:// These bootstrap scripts have since been used by other distributions and developers to experiment with ideas of their own around bootstrap, including several comparative studies of the different toolchain options that might be used.

Stage 2 of the bootstrap process used the minimal chroot environment from stage 1 (which contained a native "hard float" gcc toolchain) to build up a small set of packages sufficient to run a native "hard float" version of rpmbuild, a tool used to build RPM packages from source. The goal was not to get a version of rpmbuild sufficient to build every package in the distribution, but rather sufficient to rebuild the packages from this and the previous stage as "real" RPMs. The choice to build only a minimally useful version of rpmbuild was important. It meant that it was possible to reduce the number of build-time dependency packages that would have to be built manually from source to a minimal set. Further, the extensive use of GNU autotools in core distribution packages meant that many of those dependencies would automatically configure away advanced features not necessary at this stage. Stage 2 resulted in the creation of a directory containing a number of "recipe" scripts, one per package dependency, that could run in order using a numbered script prefix.

Using git as a filesystem

Perhaps the most novel thing about stage 2 was the use of git early on. This author had an idea that, since git supported distributed development, more Fedora developers could collaborate to build a minimal root filesystem sufficient for building further bootstrap packages if that filesystem were itself a git repository, along with the scripts used to create it. A git repository was created that initially contained the stage 1 filesystem contents (the "hard float" toolchain components). To this was added a "/stage2" directory, containing a "recipe.d" subdirectory. The latter housed a number of individual scripts necessary to build particular packages from sources also contained within the same filesystem. The use of git to manage the entire work-in-progress chroot meant that all prospective developers would need to do would be to clone the filesystem, and then make it available to an ARM target system as a chroot (typically by having a second root-owned clone that was NFS-exported to a suitable development system). It was ugly, but it worked.

Each time a developer added a new recipe for an RPM build requirement in "/stage2/recipe.d", they were able to commit that recipe along with all of the sources for a given package, its populated build directory after running the build for the package, and the subsequent changes to the filesystem after installation. Each git commit thus represented one more step toward having a usable set of dependencies to build rpmbuild. In cases where there were not direct build interdependencies between the basic set of packages, developers working on stage 2 were able to parallelize the effort by building packages in their own git branches prior to issuing a standard "pull" request to the maintainer of the filesystem. This had the benefit of providing a git history for general posterity. The output of stage 2 was a working rpmbuild.

Once an rpmbuild binary was available and it became possible to build RPMs, it was on to the next stage. Stage 3 involved taking the filesystem from stage 2 (which now contained enough binaries and dependencies to support running minimal RPM builds of the software that had been built from source up until that time) and rebuilding its contents as regular RPM packages, then adding sufficient additional dependency packages to build the standard "yum" and "mock" Fedora packaging tools. A new top level directory, "/stage3" was created and within it various builds were performed using rpmbuild, with the output source and binary packages preserved. After each build successfully completed, its build directory was removed (it was not necessary to preserve since the source and binary RPMs contained enough data to reproduce it if that became necessary). The resulting package was installed into the same filesystem, overwriting any files that had previously been manually built and installed during stage 2 with real RPM versions. The resulting changes to the filesystem then formed yet another git commit for easy tracking.

During stage 3, developers could once again parallelize various builds that were not inter-dependent, commit the result of those builds into their own git branches, and then submit pull requests for their branches to be merged into the filesystem. The whole git approach would at first appear to be error prone, but it did not prove to be that way in reality since the packages being built were largely independent and did not modify the filesystem in ways that could not be trivially resolved during a git merge. Eventually, the total size of the git repository containing the root filesystem exceeded 4.6GB (which git handled with ease) and all of the necessary packages sufficient to build and use the "yum" and "mock" utilities were available. This included both perl and python as well as many plugins. Yum is a utility used to manage installing software from repositories of packages. It can be driven by Mock, which is a tool that creates chroots of installed packages and then uses those to build other packages in a trusted and reproducible fashion.

The git approach to bootstrapping seemed to pay off. It was retired following the completion of stage 3 because there were sufficient RPM packages available and infrastructure to use yum and mock directly to build other packages. In addition, at this stage it became possible for Dennis Gilmore to use the mock support to build a few more packages and to then create a minimal bootable Fedora 15 "hard float" image for installation onto a few PandaBoards. Images were also created with alternative kernels for other ARM systems, including the ComuLab TrimSlice (based on the Nvidia Tegra-2). These images were made available to developers who then had a choice of booting the images or using them as chroots on an existing Fedora 13 installation (the result being a chroot within a chroot for mock builds done on such latter systems). The images were collectively then used for stage 4 of the bootstrap.

The stage 4 images included a set of scripts written by Dennis Gilmore and subsequently enhanced by many others, including Henrik Nordström. These scripts implemented a poor man's distributed build system (for various reasons, it was not possible to run a full Koji build system yet). A dedicated virtual server machine was created to host various source and successfully built packages in a "stage4" yum repo. Then, each developer with a build system running the stage 4 image was able to automate the process of retrieving a package from the server, building it in mock on their ARM system, and contributing the results back to the overall repository of built stage 4 packages. Using this approach, over 13,000 binary "armv7hl" (Fedora's name for the ARMv7 hard-float, little-endian architecture) packages were built over a period of a couple of weeks. These were combined with the existing "noarch" (meaning architecture independent) packages from the primary architecture (x86) to create an initial set of Fedora 15 ARM packages.

Stage 4 was the largest stage by far since it targeted generally building the entire distribution set of packages. At this point, it was discovered that many source packages required some tweaking. Some did not build for ARM at first, requiring usually trivial fixes and the occasional patch be committed into the Fedora git repositories and built first for the primary architecture. In the case that ARM-specific changes that were not yet in the official Fedora git repositories, special package versions were built containing obvious and compatible changes to the release number (such as appending ".arm0") in such a way that updates later built from the fixed git repository would not conflict. A tool was later written by DJ Delorie to automatically track down those packages that required fixes yet to be committed in order to ensure all Fedora ARM packages were fully in sync with the original primary architecture set.

During stage 4, it had been decided generally to build the Fedora 15 package set as it was on the day Fedora 15 was released for the primary architecture, not including all of the updates that might have been built in the interim (unless those updates contained necessary ARM or non-ARM fixes). This is because the intent was to first bootstrap the entire distribution to the same point as Fedora 15 had been at release on the primary architecture. This known combination of packages could then be used to rebuild the entire package set in the stage 5 final Koji mass rebuild. Updates would then be built from the primary architecture using the known good Fedora 15 base. Henrik Nordström in particular assisted greatly at this stage in building packages and fixing problems with the stage 4, even though at times he did not always agree with the pedantry of targeting non-updates packages first.

And that brings the reader up to the present day. As of this writing, a core set of Fedora 15 packages has been built and the distribution is usable on ARMv7 systems (at least experimentally). The final stage is to rebuild everything one more time using the standard build infrastructure in a known good configuration, using the official ARM Koji system. Fedora protocol is that each package be built exactly once (for official builds) for a given release and at that time it should be built for all variants of the particular architecture. This means that it is necessary to wait for a corresponding initial set of ARMv5 packages to be built prior to completing the stage 5 final mass rebuild of Fedora 15. Since ARMv5 is not a new architecture, and is not changing ABI, all that is required there is a stage 4-like rebuild using mock. This is currently being performed using many of the same builders that bootstrapped ARMv7, under the control of a new set of scripts named "moji", written by Jon Chiappetta of Seneca College.

Once a mass rebuild of the entire Fedora 15 package set has been completed using Koji, an initial release of Fedora 15 will be made available. It is hoped that this will happen prior to the final release of Fedora 16 on the primary architecture (in early November), or at least soon thereafter. It should be noted that until Fedora ARM is a primary architecture, and while it is playing a constant game of catch-up, no release will be entirely finished. Instead, it will be good enough for the criteria established for that particular release. This means that there will be some (non essential) packages that do not make the initial release of Fedora 15. These will follow the release as there is time to work on making them available.

Following the Fedora 15 release, attention will turn to the next phase. It has been proposed that the team skip Fedora 16 and target getting rawhide (the development build of Fedora) built as the next step, using a tool known as koji-shadow to automatically rebuild all rawhide package builds within a reasonable period of their being built on the primary architecture. Whether "reasonable" is one day or several days will have to be decided, but it is clear that, by targeting rawhide as quickly as possible, Fedora ARM can benefit from following along closely as Fedora 17 and 18 are released.

The future of Fedora ARM

The Fedora ARM project has gained greater cohesion over the past 6 months as more involvement from many of the parties involved has lead to the solving of a number of challenging problems around package building, architecture bootstrap, and so on. Clearly, one of the primary (pun intended) goals of Fedora ARM is to reach primary architecture status as effectively as possible. That won't happen over night, and, in fact, no secondary architecture has ever been promoted to primary yet, so there is a lot left to determine around what exactly the requirements will be and how that will be done.

Primary architecture status would afford many benefits to Fedora ARM, such as requiring all Fedora packagers to support ARM (rather than relying on good will in fixing bugs - although the assistance given so far has generally been excellent), and being able to define blockers and targets for individual Fedora releases. As a secondary architecture, Fedora ARM must play a constant game of catch-up, rebuilding packages that have already been built on the primary and hoping that not too many issues will arise to slow down the overall release. As a primary, the entire distribution will wait if an issue arises that causes a package somewhere to fail to build on ARM. At the same time, primary architecture status brings benefits to Fedora in terms of having more than one first-class architecture and will allow Fedora to better compete for the growing numbers of users who have ARM based systems.

Getting to primary architecture status will take some time. At a minimum the process will likely require that Fedora ARM very closely track the building of every primary architecture package and produce a build shortly thereafter (and eventually in parallel). Tools such as koji-shadow are already being deployed to automatically track primary architecture builds and initiate secondary ones in this fashion. In addition, the systems used to build packages will need to be more rugged than those that are available today. The current builder hardware is sufficient for a lab environment, but it is occasionally plagued with issues such as faulty SD cards, overheating boards, and so forth. It is hoped that newer, more suitable hardware will appear over the coming months to solve that issue.

Beyond a plan for becoming a primary architecture, a number of specific recommendations are likely to feed out of the current efforts of Fedora ARM project back into the planning around general Fedora releases. They are likely to include a call for mandatory mass rebuilds every one or two Fedora releases. A mass rebuild means that every package in the entire distribution is rebuilt from scratch, and it makes the life of secondary architectures (or new architectures) much easier because those secondary architectures need build only one release worth of package dependencies.

More about Fedora ARM

To find out more about the Fedora ARM project and to get involved, visit the Fedora ARM secondary architecture wiki pages. Here you can find directions on joining the "arm" Fedora mailing list, and joining the #fedora-arm IRC channel on the Freenode IRC network.

[ Jon Masters is a Principal Software Engineer at Red Hat, where he works on the Fedora ARM project. Jon is co-author of Building Embedded Linux Systems, and is currently writing a book on porting Linux to new architectures. ]

Comments (27 posted)

Brief items

Distribution quotes of the week

One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them.
-- Colin Watson

I'm giving up on cars. Every few years I test-drive a car, to see if they've reached the point where they are usable to every day people, and every time I am disappointed. Sure, maybe coverall-wearing mechanics and uber-elite NASCAR drivers can figure out how to operate them, but they're just too damned complicated for your average joe.

To start off, there are literally hundreds of different kinds of cars, and they're all different. How do car makers expect a non-expert to be able to select one? Most manufacturers even make different "models" of their car, so it's not enough to just say you want a Ford, now you have to decide which Ford you want. I don't know ahead of time whether I'm going to want to move furniture, go off-roading or cruise the Autobahn, why should I have to pick one? Why can't they just make one car that does everything?

-- Michael Hall

Comments (7 posted)

OpenELEC 1.0 Released

After more than two years of development, OpenELEC 1.0 has been released. OpenELEC is a Linux distribution for home theater PCs, and is based on XBMC. "OpenELEC is an embedded Linux distribution that aims to allow people to use their Home Theatre PC (HTPC) in the same manner as any other device attached to your TV - like a DVD player or Sky box. Turn on your box, and OpenELEC is ready and waiting in less than 10 seconds - as fast as some DVD players. A simple remote control is all you will need. On top of that, one need not worry about updating OpenELEC, as the software will self-update automatically when connected to the Internet and an update is available. Installing OpenELEC takes less than 5 minutes and requires zero Linux experience. After that, one need merely add one's media files and preferred addons, and OpenELEC will 'just work.'"

Comments (4 posted)

openSUSE 12.1 RC1 available

The openSUSE project has announced the first release candidate for openSUSE 12.1. "The next release of openSUSE is expected to bring a large number of improvements and changes. Many of these are the 'usual' updates any Linux distribution offers. These include the latest Firefox, GNOME 3.2 and KDE's Plasma Workspace 4.7. Under the hood, we have Linux kernel 3.1 and we expect to be the first to ship Google's new programming language Go. We also overhauled our boot procedure introducing systemd and Grub2 (still in testing) and of course we'll ship the latest developer tools and libraries as well as all the sysadmin goodies openSUSE is known for!"

Full Story (comments: none)

Canonical announces five years of support for Ubuntu 12.04 LTS desktops

Instead of the usual three-year support for a long-term support (LTS) release of Ubuntu on the desktop, Canonical has announced that 12.04 LTS ("Precise Pangolin") will be supported for five years. That is in keeping with the support period for the server version of an LTS release. "April 2012 will see the fourth LTS release of Ubuntu. LTS releases have become particularly popular with Ubuntu business users. Canonical's own survey data shows over 70% of server users are deployed on LTS versions of the product. Bringing this extended support to the desktop is a response to similar popularity in businesses of the desktop LTS releases. [...] The first two years of the LTS period will benefit businesses by including hardware updates (through regular point releases) allowing them to keep up to date with the latest hardware upgrades. Maintenance updates will continue for a further three years. Businesses can now rely on always running an LTS version regardless of their hardware refresh rate."

Comments (25 posted)

Yocto 1.1 released

The Yocto project has sent out a press release announcing the availability of version 1.1. New features include multilib support, an improved graphical configuration system, initial support for the x32 ABI, Btrfs support, and more. The Yocto project will also be taking over management of the eglibc C library fork.

Comments (3 posted)

Distribution News


Upcoming elections for FAmSCo, FESCo, and the Fedora Board

It's election time for the Fedora Project, with seats open on the Ambassadors Steering Committee, the Engineering Steering Committee, and the advisory board. Nominations are open until November 5, 2011.

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

Comments (none posted)

Desktop dreams: Ubuntu 11.10 reviewed (ars technica)

Ars technica has a review of Ubuntu 11.10. "Ubuntu 11.10, codenamed Oneiric Ocelot, prowled out of the development forest earlier this month. In our review of Ubuntu 11.04, released back in April, we took a close look at the strengths and weaknesses of the new Unity shell and compared it with GNOME 3.0. In this review, we're going to revisit Unity to see how much progress it has made over the past six months. We will also take a close look at the updated Software Center user interface and the transition from Evolution to Thunderbird."

Comments (none posted)

Andy Rubin: Android 4.0 to be open sourced by year end (The H)

The H reports that the source code for Android 4.0 ("Ice Cream Sandwich") will be released a few weeks after the recently announced Galaxy Nexus smartphone starts shipping, according to Google's Andy Rubin. "Code-named 'Ice Cream Sandwich' (ICS), Android 4.0 was first revealed yesterday (19 October) alongside the new Nexus device, at a joint Google and Samsung event. The new version of the mobile operating system includes features from both the current phone version, 2.3.x "Gingerbread', and the tablet version, 3.x 'Honeycomb'. While it will reportedly work on both large- and small-screen devices, it was only demonstrated on the Galaxy Nexus smartphone."

Comments (45 posted)

Puppy Linux 5.3 "Slacko" based on Slackware 13.37 (The H)

The H takes a look at the latest release of Puppy Linux.

The Puppy Linux developers have announced the release of version 5.3 of their independent Linux distribution, a new edition code-named "Slacko". According to Puppy Linux founder Barry Kauler, Slacko Puppy (coordinated by Mick Amadio) is to be the project's "new flagship Puppy Linux" and is the recommended "'first stop' for all newcomers".

Like the other versions of Puppy Linux, Slacko was built using the Woof build tool, which, the developers say, can build a Puppy release from the binary packages of any other distribution. Slacko Puppy 5.3 is based on Slackware Linux 13.37 binary packages and the Linux kernel, and includes access to the Slackware package repositories. The new edition uses the Frisbee network manager and has built-in support for a number of multimedia formats. Other software can be installed via PET (Puppy's Extra Treats) packages and packages in the Slackware repositories.

Comments (none posted)

7 Years of Ubuntu: Ubuntu Then and Now (

Joe 'Zonker' Brockmeier looks at how Ubuntu shaped the Linux landscape over the last 7 years. "Today marks seven years since Mark Shuttleworth announced Ubuntu 4.10. Dubbed "Warty Warthog," Ubuntu 4.10 was a major departure from other Linux distros at the time. Seven years, and 15 releases, has brought a lot of change. Let's take a look at how Ubuntu has grown from the new kid on the block to one of the most popular Linux distributions ever. It seems like just yesterday when Ubuntu made its debut. Today, it's hard to imagine a time when Ubuntu wasn't a major part of the Linux landscape. When Ubuntu first appeared, though, it was not obvious to all that it would succeed."

Comments (4 posted)

Page editor: Rebecca Sobol
Next page: Development>>

Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds