Updating and rebuilding Android

This is one in a series of articles on working with the Android Developer Phone (ADP1) device. In the previous episode, your editor went through the process of updating the software on the ADP1. This time around, we'll look at the latest software builds, then take a beginning look at the process of actually building new software for the device.

Your editor started by testing out the Android 1.5 images provided by HTC, the manufacturer of the ADP1. The provision of these images is a nice step forward by HTC; thus far, ADP1 owners have felt somewhat left out when new versions of the firmware have been released. This time around, they have the new software at about the same time as everybody else. The 1.5 update is done in the usual way: use the "adb" tool to copy it to /sdcard/update.zip on the phone, then reboot into the recovery image to actually install the new code. Two such iterations are required this time around; there is an update to the (closed-source) radio code which must be applied first.

So what's new in Android 1.5? The biggest user-visible feature is almost certainly the on-screen keyboard. It's no longer necessary to open the keyboard to send a quick text message. The on-screen keyboard is somewhat painful for your fat-fingered editor to use when the phone is in the portrait orientation, but it works better in the landscape mode. One has to wonder, though, what inspired the Android developers to dedicate a significant chunk of scarce screen space to a "smiley" key. There are plenty of characters which would have been rather more useful in that position.

Beyond that, the 1.5 release includes "Latitudes" support for those of you who want to continuously report your real-world location to the Google mother ship. There are simple screen effects which come into play when switching between applications and orientations. Holding the power button gives quick access to "airplane mode." The camera is quite a bit more responsive. The zoom icons are smaller and more discrete. GPS acquisition is said to be faster; your editor has not had a chance to test that claim, but it would certainly be a welcome improvement. The orientation-awareness (turn the phone on its side and the display switches to landscape mode) that has been a feature of the JesusFreke builds for a while is now part of the core platform. And so on. Mostly small stuff, but it's enough to make for a nicer feel to the platform overall.

Speaking of the JesusFreke builds, the JFv1.50 build, based on 1.5, is also available; your editor promptly installed it. This build is basic Android 1.5 with a number of additions, including multitouch support, tethering support, an augmented init daemon, a whole set of busybox-based command-line utilities, and more; see the full list for the details. As usual, these builds add a number of nice features to the phone; anybody who is interested in really playing with the device will likely prefer the JF version of the software.

Remaking JF builds

Playing with new builds is fun, but this is free software. The real fun comes from rebuilding the software from the source, perhaps with specific changes. There are two levels at which this can be done. The first is to use the JesusFreke "build environments." Essentially, the build environment is a tarball containing the modifications made to create the JF image, along with the necessary scripts. There's a new kernel containing multitouch and unionfs support, along with the patches needed to create it. Busybox is found there, as are a number of other useful diagnostic tools, an ssh client, and more.

To create a new build, it is necessary to get the associated official build, place it within the build environment, then run make. With any luck, the end result is an update.zip file ready to be flashed into the phone.

One of the interesting things your editor learned from looking at the images (and from talking to Mr. Freke) is that the JF builds do not actually involve rebuilding much of the Android system. It's mostly a matter of unpacking an official build and making a few creative substitutions. The kernel has been remade, as has the browser application (to support multitouch zooming). Everything else is just a matter of shuffling files around. So the JF build environments can be useful for somebody else wanting to do the same kind of manipulations, but more extensive changes require building the system at a lower level.

Building Android from source

Remaking Android from the source code turns out to be a bit of a challenge. What follows here is a brief set of instructions derived from the Android "building for Dream" page, some hints helpfully provided by GeunSik Lim, and a fair amount of painful experience. In summary: most of the code needed to rebuild the platform is available, but (1) it's not a quick or simple process, and (2) there are a few pieces missing.

There's a number of tools which must be installed on a Linux system to rebuild the Android platform. These include flex, bison, git, and the Sun Java system. Beyond that, one must grab the repo tool. Repo is Google's answer to the problem of managing a whole set of related repositories; essentially it is a tool which sits on top of git and manages a whole set of git repositories in parallel. Once repo has been installed, the meta-repository is set up with a command like:

    repo init -u git://android.git.kernel.org/platform/manifest.git

This command pulls down the manifest file describing all the repositories needed to build the platform. Note that if a branch other than the trunk is desired, it must be obtained during this stage with the -b option; repo apparently cannot switch branches in an existing source tree.

One then obtains the code by running "repo sync" and going out for coffee.

Incidentally, when you go out for coffee, you need not hurry back. It's entirely possible to fly to a different continent, harvest the coffee by hand (after waiting for it to reach the perfect ripeness), fly home, and roast it yourself. You'll still probably have time for a second cup before the downloading of the source is complete. You are, after all, not just downloading a huge pile of source files. You are, instead, downloading over 100 independent git repositories, each containing a long trail of history - about 2.4GB worth of stuff. It takes a while. And, needless to say, some disk space.

To make things worse, you still don't have all the source; there are a few components of the binary platform for the ADP1 which have not been released as free software. You cannot download those binary components from anywhere; instead, what's needed is to obtain them from a working phone. To that end, the file vendor/htc/dream/extract-files.sh contains a script which will pull the needed components from a USB-connected ADP1 device. These components vary from files containing mixer settings to programs for controlling Bluetooth, the GPS receiver, firmware for the wireless network adapter, a camera control library, and more. The dream directory also contains a binary driver module (wlan.ko) for the WIFI adapter, despite the fact that said driver is open source and included in the distribution.

After that, it's a matter of copying build/buildspec.mk.default to buildspec.mk in the top-level directory, editing it to set TARGET_PRODUCT to htc_dream, and typing make. And going out for more coffee, of course. At the end of the process, with luck (a fair amount of luck may be required), there will be new system and boot images which can be flashed into the phone with the fastboot tool. A reboot will run the new code.

At that point, of course, there are some surprises to be found. One is that the newly built software is lacking a number of features found in an official build. The reason for this is simple: several of the applications which run on Android phones are not open source. These include the Gmail client (which your editor will happily do without), Maps (which cannot be done without, at least until AndNav progresses a little further), and more. These applications can generally be recovered by grabbing the associated package files from an official build and slipping them into the build environment. See this article for a terse description of how that is done.

It took your editor a little while to figure out another little surprise: despite the fact that the Android source tree includes a kernel repository, the build process does not actually build the kernel. One might think that it would be hard to miss something the size of a kernel build, but ... did your editor mention that the Android source tree is big? The Android build system which goes with this source tree is quite a piece of work; there must be people working full time on it, and they probably hate their lives.

Trying to figure out what is happening in an Android build requires digging through many thousands of lines of makefile rules. What your editor finally discovered is that the build system simply pulls a binary kernel from a special "prebuilt" repository (that repository also contains a cross-compiling toolchain for the creation of ARM executables). The kernel source tree, seemingly, is just there for show. Using something other than the prebuilt kernel requires making it separately and pointing a build-system variable at the location of the result.

[PULL QUOTE: It's clear that even people who remake Android are not, in general, expected to remake the kernel. END QUOTE] It's clear that even people who remake Android are not, in general, expected to remake the kernel. The kernel repository pulled down by the repo command does contain the Android-specific patches, but it lacks nice things like branches (even "master" is missing) or tags. There are some remote branches with names like korg/android-msm-2.6.27 which contain lines of development for various kernel versions; the 2.6.27 one appears to be, as of this writing, the one which is best supported on real hardware. But, within those branches, there are (unlike the situation with the rest of the Android code) no tags associated with releases. Nothing in the repository will tell a developer which kernel was shipped with a given version of Android.

So it's hard to build a kernel which corresponds to the one found within an official release. But not impossible: most of the official releases include the git commit ID in the kernel version. So by digging down into the settings menus, your editor was able to determine that the HTC 1.5 build came from commit 8312baf. After checking out that commit, one can do a make msm_defconfig to configure the kernel properly. Then it's just a matter of setting the ARCH and CROSS_COMPILE environment variables and doing the build. If you have a 32-bit Linux environment, the prebuilt ARM toolchain provided with the Android source does the job just fine.

Once the kernel build is done, it's possible to build a new set of firmware images which can be loaded into the device with fastboot. That's easy to say, but it can be harder to do; the sources from the repository often do not build, and it's not always easy to get all the pieces together to make a working image for the ADP1. Making it possible for people outside of the core Android project to build and install the platform appears to be an afterthought, at best.

Android and the ADP1

In truth, Google does not really support the ADP1 as a system people can develop and run on; this situation was somewhat explained by Jean-Baptiste Queru, who is easily the most helpful Google developer on the mailing lists:

So, while the ADP1 is one of the most open cellular phone platforms yet to appear, it does not, yet, represent a fully-open system in the way the OpenMoko phone does. Most people wanting to do things with this device are likely to end up starting with the official, binary builds and tweaking things around the edges, much like as has been done with the JF builds.

That said, there is a lot of fun to be had with this device. It's fully hackable at the kernel level now, and more hardware information is becoming available, which raises the hope of gaining more control over the low-level system in time. About the time the ADP1 becomes fully obsolete and unobtainable, we should have it figured out pretty well. With any luck at all, at least one of the devices which replaces it will be more open from the outset.

---

(Log in to post comments)