This is one in a series of articles on working with the Android Developer
Phone (ADP1) device.
In the previous episode
editor went through the process of updating the software on the ADP1.
This time around, we'll look at the latest
software builds, then take a beginning look at the process of actually
building new software for the device.
Your editor started by testing out the Android 1.5
images provided by HTC, the manufacturer of the ADP1. The provision of
these images is a nice step forward by HTC; thus far, ADP1 owners have felt
somewhat left out when new versions of the firmware have been released.
This time around, they have the new software at about the same time as
The 1.5 update is done in the usual way: use the "adb" tool to copy it to
/sdcard/update.zip on the phone, then reboot into the recovery
image to actually install the new code. Two such iterations are required
this time around; there is an update to the (closed-source) radio code
which must be applied first.
Sidebar: USB cables
If, in the process of pushing updates to the ADP1, you get failures with
"protocol error" messages, you're not alone. It turns out that the device
is sensitive to noise introduced by low-quality USB cables; one needs a
well-built cable for this task. Note: the cable packed with the ADP1 does
not qualify as "well-built."
So what's new in Android 1.5? The biggest user-visible feature is almost
certainly the on-screen keyboard. It's no longer necessary to open the
keyboard to send a quick text message. The on-screen keyboard is somewhat
painful for your fat-fingered editor to use when the phone is in the
portrait orientation, but it works better in the landscape mode. One has
to wonder, though, what inspired the Android developers to dedicate a
significant chunk of scarce screen space to a "smiley" key. There are
plenty of characters which would have been rather more useful in that
Beyond that, the 1.5 release includes "Latitudes" support for those of you
who want to continuously report your real-world location to the Google
mother ship. There are simple screen effects which come into play when
switching between applications and orientations. Holding the power button
gives quick access to "airplane mode." The camera is quite a bit
more responsive. The zoom icons are smaller and more discrete.
GPS acquisition is said to
be faster; your editor has not had a chance to test that claim, but it
would certainly be a welcome improvement.
The orientation-awareness (turn the phone on its side and the display
switches to landscape mode) that has been a feature of the JesusFreke builds
for a while is now part of the core platform.
And so on.
Mostly small stuff, but it's enough to make for a nicer feel to the
Speaking of the JesusFreke builds, the JFv1.50
build, based on 1.5, is also available; your editor promptly installed
it. This build is basic Android 1.5 with a number of additions, including
multitouch support, tethering support, an augmented init daemon, a
whole set of busybox-based command-line utilities, and more; see the full
list for the details. As usual, these builds add a number of nice
features to the phone; anybody who is interested in really playing with the
device will likely prefer the JF version of the software.
Remaking JF builds
Playing with new builds is fun, but this is free software. The real fun
comes from rebuilding the software from the source, perhaps with specific
changes. There are two levels at which this can be done. The first is to
use the JesusFreke "build environments." Essentially, the build
environment is a tarball containing the modifications made to create the JF
image, along with the necessary scripts. There's a new kernel containing
multitouch and unionfs support, along with the patches needed to create
it. Busybox is found there, as are a number of other useful diagnostic
tools, an ssh client, and more.
To create a new build, it is necessary to get the associated official
build, place it within the build environment, then run make. With
any luck, the end result is an update.zip file ready to be flashed
into the phone.
One of the interesting things your editor learned from looking at the
images (and from talking to Mr. Freke) is that the JF builds do not
actually involve rebuilding much of the Android system. It's mostly a
matter of unpacking an official build and making a few creative
substitutions. The kernel has been remade, as has the browser application
(to support multitouch zooming). Everything else is just a matter of
shuffling files around. So the JF build environments can be useful for
somebody else wanting to do the same kind of manipulations, but more
extensive changes require building the system at a lower level.
Building Android from source
Remaking Android from the source code turns out to be a bit of a challenge.
What follows here is a brief set of instructions derived from the Android
"building for Dream" page, some hints helpfully provided by GeunSik
Lim, and a fair amount of painful experience. In summary: most of the code
needed to rebuild the platform is available, but (1) it's not a quick
or simple process, and (2) there are a few pieces missing.
There's a number of tools which must be installed on a Linux system to
rebuild the Android platform. These include flex, bison, git, and the Sun
Java system. Beyond that, one must grab the repo tool. Repo
is Google's answer to the problem of managing a whole set of related
repositories; essentially it is a tool which sits on top of git and manages
a whole set of git repositories in parallel. Once repo has been
installed, the meta-repository is set up with a command like:
repo init -u git://android.git.kernel.org/platform/manifest.git
This command pulls down the manifest file describing all the repositories
needed to build the platform. Note that if a branch other than the trunk
is desired, it must be obtained during this stage with the -b
option; repo apparently cannot switch branches in an existing
One then obtains the code by running
"repo sync" and going out for coffee.
Incidentally, when you go out for coffee, you need not hurry back. It's
entirely possible to fly to a different continent, harvest the coffee by
hand (after waiting for it to reach the perfect ripeness), fly
home, and roast it yourself. You'll still probably have time for a second
cup before the downloading of the source is complete. You are, after all,
not just downloading a huge pile of source files. You are, instead,
downloading over 100 independent git repositories, each containing a long
trail of history - about 2.4GB worth of stuff. It takes a while. And,
needless to say, some disk space.
To make things worse, you still don't have all the source; there are a few
components of the binary platform for the ADP1 which have not been released
as free software. You cannot download those binary components from
anywhere; instead, what's needed is to obtain them from a working phone.
To that end, the file vendor/htc/dream/extract-files.sh contains a
script which will pull the needed components from a USB-connected ADP1
device. These components vary from files containing mixer settings to
programs for controlling Bluetooth, the GPS receiver, firmware for the
wireless network adapter, a camera control library, and more. The
dream directory also contains a binary driver module
(wlan.ko) for the WIFI adapter, despite the fact that said driver
is open source and included
in the distribution.
After that, it's a matter of copying build/buildspec.mk.default to
buildspec.mk in the top-level directory, editing it to set
TARGET_PRODUCT to htc_dream, and typing make.
And going out for more coffee, of course. At the end of the process, with
luck (a fair amount of luck may be required), there will be new system and
boot images which can be flashed into
the phone with the fastboot tool. A reboot will run the new code.
At that point, of course, there are some surprises to be found. One is
that the newly built software is lacking a number of features found in an
official build. The reason for this is simple: several of the
applications which run on Android phones are not open source. These
include the Gmail client (which your editor will happily do without), Maps
(which cannot be done without, at least until AndNav progresses a little further), and
more. These applications can generally
be recovered by grabbing the associated package files from an official
build and slipping them into the build environment. See this
article for a terse description of how that is done.
It took your editor a little while to figure out another little surprise:
despite the fact that the Android source tree includes a kernel repository,
the build process does not actually build the kernel. One might think that
it would be hard to miss something the size of a kernel build, but ... did
your editor mention that the Android source tree is big? The Android build
system which goes with this source tree is quite a piece of work; there
must be people working full time on it, and they probably hate their lives.
Trying to figure out what is happening in an Android build requires digging
through many thousands of lines of makefile rules. What your editor
finally discovered is that the build system simply pulls a binary kernel
from a special "prebuilt" repository (that repository also contains a
cross-compiling toolchain for the creation of ARM executables). The kernel
source tree, seemingly, is just there for show. Using something other than
the prebuilt kernel requires
making it separately and pointing a build-system variable at the location
of the result.
It's clear that even people who remake Android are not, in general,
expected to remake the kernel.
It's clear that even people who remake Android are not, in general,
expected to remake the kernel. The kernel repository pulled down by the
repo command does contain the Android-specific patches, but it
lacks nice things like branches (even "master" is missing) or tags. There
are some remote branches with names like korg/android-msm-2.6.27
which contain lines of development for various kernel versions; the 2.6.27
one appears to be, as of this writing, the one which is best supported on
real hardware. But, within those branches, there are (unlike the situation
with the rest of the Android code) no tags associated
with releases. Nothing in the repository will tell a developer which
kernel was shipped with a given version of Android.
So it's hard to build a kernel which corresponds to the one found within an
official release. But not impossible: most of the official releases
include the git commit ID in the kernel version. So by digging down into
the settings menus, your editor was able to determine that the HTC 1.5
build came from commit 8312baf. After checking out that commit,
one can do a make msm_defconfig to configure the kernel
properly. Then it's just a matter of setting the ARCH and CROSS_COMPILE
environment variables and doing the build. If you have a 32-bit Linux
environment, the prebuilt ARM toolchain provided with the Android source
does the job just fine.
Once the kernel build is done, it's possible to build a new set of firmware
images which can be loaded into the device with fastboot. That's
easy to say, but it can be harder to do; the sources from the repository
often do not build, and it's not always easy to get all the pieces together
to make a working image for the ADP1. Making it possible for people
outside of the core Android project to build and install the platform
appears to be an afterthought, at best.
Android and the ADP1
In truth, Google does not really support the ADP1 as a
system people can develop and run on; this situation was somewhat explained by Jean-Baptiste Queru, who is
easily the most helpful Google developer on the mailing lists:
Yes, the ADP1 situation is currently unfortunate. We've had to pick
priorities, the priority went to open-sourcing code out of Google,
as that's something that only Googlers can do.
The truth is, ADP1 isn't a phone from Google. While Google has some
influence on it (and provides a number of proprietary apps), It's
neither manufactured nor distributed by Google, and that puts
limits on the ways Google can support it (and espcially on how
Google can not redistribute some of the ADP1-specific files).
So, while the ADP1 is one of the most open cellular phone platforms yet to
appear, it does not, yet, represent a fully-open system in the way the
OpenMoko phone does. Most people wanting to do things with this device are
likely to end up starting with the official, binary builds and tweaking
things around the edges, much like as has been done with the JF builds.
That said, there is a lot of fun to be had with this device. It's fully
hackable at the kernel level now, and more hardware information is
becoming available, which raises the hope of gaining more control over the
low-level system in time. About the time the ADP1 becomes fully obsolete
and unobtainable, we should have it figured out pretty well. With any luck
at all, at least one of the devices which replaces it will be more open
from the outset.
Comments (24 posted)
Google uses Python for many of its engineering projects, from internal
server monitoring and reporting to outward-facing products like Google
Groups, so it is no surprise that the company wants to improve Python
application performance. A group of Google developers is working on a new
optimization branch of Python dubbed Unladen Swallow, with
the goal of a five-fold speed increase over the trunk. It will achieve
that goal by adding just-in-time compilation and a new virtual machine
design, all while retaining source compatibility for Python application
Unladen Swallow's lead developers Collin Winter, Jeffrey Yasskin, and
Thomas Wouters have long been core developers for the CPython project, the reference
implementation and most widespread interpreter for the Python language.
All three are Google employees, and others contribute their "twenty percent
time" to Unladen Swallow, but the group insists that it is a Python
project, not an effort owned by Google.
Winter said the origin of the idea dates back to his work on the
web-based code review tool Mondrian,
when the team's attempts at optimization repeatedly hit limitations in
CPython, such as the Global Interpreter Lock (GIL), the mutex that prevents
concurrency on multiprocessor or multi-core machines. While researching
potential speed-ups and changes, Winter and the other Google engineers
eventually decided that the long-range ideas they had in mind were
significant enough to warrant making a separate branch. Plus, doing so
would give them the chance to stress-test their ideas before trying to roll
them back into CPython.
The Concept: a bird's eye view
The core of the Unladen Swallow team's planned improvements
is to remove performance bottlenecks in the Python virtual machine (VM)
design, leaving the rest of the interpreter — not to mention the
substantial runtime library — relatively untouched. The long-term
plan is to replace CPython's existing stack-based VM with a
register-based VM built with code from the Low
Level Virtual Machine (LLVM) project, and to add a just-in-time
compiler (JIT) on top of the new VM. Other performance-based improvements
are welcome at the same time, and the team has several in store based on
their talks with heavy Python users.
Using a JIT will speed up execution by compiling to machine code, thus
eliminating the overhead of fetching, decoding, and dispatching Python
opcodes. "In CPython," Winter explained, "this overhead
is significant; some minor tweaks were made to CPython 2.7 that netted a
15% speed-up with relatively little work."
Adding the JIT presents a good opportunity to switch from a stack-based
VM to LLVM's register-based design, which Winter said will net its own
performance benefits. The merits of stack- versus register-based VMs is an
ongoing debate, but Winter cites a 2005 study
[PDF] from the Lua project showcasing
the empirical benefits of the register-based design.
Unladen Swallow is based on Python 2.6.1, which is not the most recent
release. Python 3.0 was released in
December of 2008, implementing the backward-incompatible
3.0 version of Python. Because the majority of Python code in the wild
— and in use at Google — is still written for Python 2.x, the
Unladen Swallow team decided to focus its efforts on the earlier version
where more benefits would be felt. By using the CPython source as its
base, Python users can expect Unladen Swallow to retain 100% source
Still, Winter said, the team does keep in close contact with Python
designer Guido van Rossum (himself a Google employee) and other members of
the CPython team. "In our discussions with Guido and others about
how and where to merge our changes back into CPython, the idea has been
proposed that Unladen Swallow should merge into 3.x. 3.x is the future of
the language, and if 3.x is significantly faster than 2.x, that's an
obvious incentive to port applications and libraries to 3.x. None of that
is set in stone, and Guido may well change his mind."
The team has set a tight development schedule
for Unladen Swallow, making quarterly milestone releases. The
first release, 2009Q1, was limited in scope, aiming for a 25 to 35% speed
increase over vanilla CPython by making less than drastic changes to the
code. The changes include a new eval loop reimplemented using vmgen, several improvements to the garbage
collector — better tracking long-lived objects so that the garbage
collection can make fewer collection runs — and to the data
serialization module cPickle, which the developers said will benefit web
applications in particular. Several obscure Python opcodes were also
removed and replaced with functionally-equivalent Python functions, which
reduces code size without affecting performance.
Unladen Swallow 2009Q1 is available as source code only for the time
being, and can be checked out as a branch
from the project's public Subversion repository. No specific compilation
instructions are provided because this release closely follows the upstream
CPython, but the developers do recommend building in 64-bit mode in order
to take the fullest advantage of the performance increases.
Since speed of execution is the goal, the team performs regular benchmarks
on the code. The thirteen benchmark tests in the suite are based on
real-world performance tests designed to highlight practical application
tasks, particularly for web applications. The results
of the tests on Unladen Swallow 2009Q1 versus CPython 2.6.1 are posted on
the project wiki; Unladen Swallow ranges from 7.43% faster to 157.17%
faster, beating CPython on every benchmark.
Work is underway now on Unladen Swallow 2009Q2, which will focus on
replacing the existing CPython VM with an equivalent built using LLVM.
Elsewhere in the ecosystem
Other open source projects have sought to improve Python application
execution using some of the same ideas. Psyco was an earlier JIT for
Python, but which was later superseded by the PyPy project.
PyPy's primary goal is not performance, though, rather it is to build a
Python implementation in Python itself. Stackless Python implements
concurrency through the use of its own scheduler and special primitives
called "tasklets." Finally, the Parrot project is implementing Python on
its own register-based VM.
In some ways, Unladen Swallow is more ambitious than these other
projects, particularly when you consider the rapid pace of development laid
out in the road map. On the other hand, Unladen Swallow starts from the
CPython 2.6.1 code base, and incorporates many CPython developers, which
greatly improves the chances that its changes will one day be blessed as
the official CPython release. Many of the 2009Q1 changes have already been
to CPython, and the door is still wide open for the 3.0 series should the
JIT and VM performance deliver real-world performance increases anywhere
close to the expected 400 percent.
Comments (18 posted)
Your editor has long been a user of the Tomboy
Tomboy makes it easy to gather thoughts, organize them, and pull them up on
demand; it is, beyond doubt, a useful productivity tool. But all is not
perfect with Tomboy. Some people have complained about its faults for a
while; Hubert Figuiere, instead, chose to do something about it in the form of the Gnote
utility. So now, of course,
people are complaining about Gnote instead.
So what are the problems with Tomboy? For your editor, at least, the
biggest issue is the simple sluggishness of the tool. It is a large
program which takes quite some time to start up. If one tries to bring up
a note on a system which has been used for other things, there can be a
lengthy delay before the menu from the taskbar icon appears. Rightly or
wrongly, users blame this fundamental slowness on Tomboy's use of the Mono
platform. Now, of course, use of Mono brings in a whole host of other
complaints, but we need not consider those here. The simple fact is that
Mono adds a great deal of baggage to what should be a small and simple
application. A basic note-taking tool should not be a heavyweight
Gnote is a reimplementation of Tomboy's functionality using C++ and GTK+.
In a sense, though, it is not an independently-developed application.
Instead, Gnote is a straightforward port of the original C# code to C++. So
it's not just a Tomboy work-alike; it's a true clone. There are
advantages to this approach; it makes use of the experience which has gone
into Tomboy development, and compatibility with the Tomboy file format is
not a problem. This approach enabled Hubert to put together a working
application in a relatively short time.
Some distributors (Fedora, at least) have made Gnote packages available.
Your editor played with the Rawhide version of Gnote for a bit; it is, in
general, indistinguishable from Tomboy. It does seem more responsive,
especially when the system is under some memory pressure. Annoyingly, it
does not (unlike Tomboy) dismiss notes when the escape key is hit. Beyond
that, though, Tomboy users (at least those who do not use plugins) will be
hard-put to tell the difference between the two.
It is said that imitation is the sincerest form of flattery; if that is
true, one would expect the Tomboy developers to be flattered indeed. But a web
log entry about the Tomboy 0.14.1 release made it clear that
"flattered" may not be the operative word in the Tomboy camp:
Our stance on Gnote is that it is counterproductive to maintain
identical software in two languages. It will be harmful to the
community, especially as these two apps inevitably diverge. It will
result in duplication of effort, duplication of bugs, and a lot of
wasted time for those who are trying to add value to the user
Tomboy is not going away, and it will continue to be developed on
the extremely productive Mono/GTK# language platform. Anyone
thinking about distributing Gnote should consider the impact on
users and their data. When we develop, we should always be asking
ourselves, "is this adding value for our users?"
It should not come as a surprise that this language inspired a lot of
responses, on the original site and elsewhere. Reading through the
discussions, your editor has come to a couple of conclusions:
- The Tomboy development community obviously sees Gnote as a threat. It's
not entirely clear why that should be. If these developers are paying
attention to what they are doing, and if the Mono platform is as
"extremely productive" as they say, they should have no trouble
staying ahead of Gnote. Beyond that, the existence of other, interoperable
applications should serve to grow the community as a whole.
- Gnote clearly has added value for some users. There is a
significant community out there which does not want to have Mono on
its systems. One may or may not agree with their reasoning, but one
cannot argue with the fact that these users exist; simply dismissing their
concerns is unlikely to change their minds. Mono-free, Tomboy-like
functionality adds value for those users.
It is evident that some developers and users in the Tomboy camp think that
the creation of Gnote is an improper thing to do. The creation of a new
application by translating code into another language seems unoriginal at
best, and a misappropriation of the code at worst. The fact that the code
has been relicensed (from LGPL 2+ to GPLv3) in the translation process has not
helped. But it should be remembered that both the translation and the
relicensing are acts which are allowed
by the license chosen by the Tomboy developers. The LGPL license text
packaged with the Tomboy code reads:
You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library.
Other parts of Tomboy carry GPLv2+ or BSD licenses; it's actually quite a
mixture, but they all allow conversion to GPLv3. So Hubert has only done
what the original developers explicitly allowed him to do; about the only
complaint one might make is that he appears to not have carried the
copyright attributions over into his translated code. That, probably, is
an omission which needs to be fixed; it would be hard to argue that Gnote
is not derived from Tomboy.
Beyond that, there are concerns that Gnote will take developer attention
away from Tomboy. That could be true, but chances are that any developers
working on Gnote (and it's not clear that there are any, beyond Hubert)
are unlikely to have been working on Tomboy previously. There is also
concern about what happens if and when Tomboy and Gnote diverge from each
other. That could well happen; Hubert does not appear to have promised to
mirror Tomboy forever. Should things go that way, Gnote really just
becomes another fork; it will live or die on its own merits.
It will take time to know whether hacking on Gnote is a wise use of Hubert's time.
But it is a clear example of what is good about free software: a developer
with a specific itch was able to scratch it (for himself and for others)
without having to start from the beginning. Criticisms of Gnote
would seem to be misplaced, and attempts to suppress it (and telling
distributors that distributing Gnote is a threat to their users' data can
only be seen as such an attempt) even more so. Free software means letting
others take your code in directions you may not always approve of.
Comments (120 posted)
Page editor: Jonathan Corbet
Next page: Security>>