LWN: Comments on "Updating and rebuilding Android"

Clean Build Systems

man_ls — 2009-05-09T22:25:00+00:00

The Gentoo people have got to a point where a relatively ignorant newcomer can follow the manual and get a running system in a couple of days. If our grumpy editor (being a kernel hacker and all) feels that the Android build process is painful then it clearly needs more polish.

This on-screen keyboard you speak of

erwbgy — 2009-05-09T08:52:30+00:00

T-Mobile UK sent out the update yesterday and using Jon's tip I now have an on-screen keyboard.

Updating and rebuilding Android

nbd — 2009-05-08T10:31:47+00:00

Really simple:
If the system lets you get away with having to do less changes to a package to get it working, then *of course* there are going to be fewer changes. People want to get stuff working, and make only the necessary changes.
Or are you really saying that build system and structure have *no* influence on the result? My experience says otherwise...

About making things public and involving kernel people: Yes, that would have helped for getting changes upstream, but it would not have helped to get the product out faster. The Kernel developer community is focusing on getting it done right, while their own development team has to focus on getting it done fast (or at all). Considering the amount of pressure on getting stuff to the market fast, there's only so much time you can spend on figuring out the right approach...

So how would getting things in patches have been far worse, if the patches themselves are under version control? Having patches under version control means that you have *at least* the same amount of information available that you have now, but with an extremely easy way of structuring things even more if you feel like it, which (using quilt) costs virtually no extra effort.

Updating and rebuilding Android

dlang — 2009-05-08T10:18:13+00:00

you don't merge back from the main tree to the feature branch.

but the point is that I don't believe that the people who created the commits that you are complaining about would have done any better if they were patches. there was nothing in the tool that prevented them from doing things better. there are lots of kernel developers who would have helped them develop things (and comment them) if they had asked, but they weren't willing to reveal things.

getting things in patches would have probably been _far_ worse, as there would probably just be one big patch for each program (that _is_ the common thing to do when companies fork programs and then release their changes)

Updating and rebuilding Android

nbd — 2009-05-08T10:09:23+00:00

What you describe still doesn't quite cover the amount of feature separation that I'm talking about. Yes, when you start implementing a feature separately, you still have some degree of separation by keeping it in a branch, but once you start pulling stuff into your master branch and merging back and forth between the master branch and the feature branches, that becomes lost too.
Or what about switching to a newer upstream version? Sure, you could go ahead and first rebase every single feature branch individually, then try to merge them back in the same order that they were merged initially (if you even kept that information and assuming that you didn't touch the same code parts in different branches), but again that's much more expensive than keeping a set of simple, readable patches around in the first place.

To sum things up:
- it is possible to do this with git, but it's more work and requires more discipline
- devs at google did not have time to do this part right, thus the result is a set of forks that due to their structure are not easy to merge back upstream
- even with patches or commits from less experienced developers, the same chaos does not happen in openwrt packages
- in openwrt you can get away with not having to fork a lot packages
- in android you have to fork every single package and if it's just for replacing the makefile with an Android.mk file
- in openwrt you have to pay less attention to how a package works internally, most autoconf based stuff can be handled with very few changes compared to a template makefile (often only changing the package name, description and download location)

In my opinion those differences are quite significant

Updating and rebuilding Android

dlang — 2009-05-08T09:47:06+00:00

the impression I get from watching projects that use git extensively is a lot of people.

remember that branching and merging is very cheap with git, keeping the separate development parts in separate branches lets you test them individually. it's only after you are sure that they are a permanent part of your central branch that you throw them away.

there are people who create separate branches for every different topic that they work on, even if they think it's only going to be a single patch.

Updating and rebuilding Android

nbd — 2009-05-08T09:38:49+00:00

Who actually does that?

Updating and rebuilding Android

dlang — 2009-05-08T07:26:26+00:00

no, you do your two different changes in different branches and then merge the two togeather.

if you need to change one of the features you make a change in that brance and do another merge.

Updating and rebuilding Android

nbd — 2009-05-07T23:47:24+00:00

Here's an important difference:

If you're working on two separate features in an OpenWrt package, the build system generates a quilt patch stack for you to edit, which makes it very easy to keep things separated. This means the system actively encourages you to keep things organized by not forcing extra work on you to do the right thing.

If you maintain such a tree in git and you want to make changes to a feature that you already committed, you're either forced to rebase (which is bad for everybody that's pulling from you), or you have to live with the fact that the grouping of changes is lost in the process.

It does not only depend on the people working on it, but also what the tools encourage you to do, and especially what the common workflow of the tools you work with is.

Updating and rebuilding Android

dlang — 2009-05-07T23:34:24+00:00

you can have a patch with a lousy explination, just like you have have a git commit with a lousy changelog. it's not a technical problem.

if you have the same people producing the commits and the patches they will probably have the same quality.

Updating and rebuilding Android

nbd — 2009-05-07T23:10:23+00:00

It is different in that the OpenWrt patch structure encourages splitting up changes in a way that makes review for upstream merging easier.
Looking at random Google-forked git directories, all I see is uncommented auto-import commits without a readable commit message, without any indication as to what the changes are meant for or why they were added.
Stuff like that does not usually happen in OpenWrt-maintained patchsets.
I looked at jpeg, libxml2, bzip2 and a few others, and I found no useful information that could help with upstream merging at all.

Another issue is that in the android build system, you cannot build packages without making changes to them (you have to at least throw in some Android.mk files to replace the Makefiles), while in OpenWrt you can leave many packages as-is and use them with our package Makefiles and no patches at all

Updating and rebuilding Android

dlang — 2009-05-07T22:42:35+00:00

the android people are accessing everything through git, so their 'forking' is not significantly different from your 'maintaining patches' (you have less stuff to move around, they use a tool that helps them merge with the original and provides them with some safety in terms of their changes still being correct)

This on-screen keyboard you speak of

fb — 2009-05-07T19:02:56+00:00

US Cupcake upgrade announcement http://forums.t-mobile.com/tmbl/board/message?board.id=An...

"End of the next week"

[...]

England should be getting it already http://www.t3.com/news/android-1-5-update-available-today...

[...]

FWIW, At the XDA forums there are (relatively) simple recipes to turn a G1 into a ADP.

Clean Build Systems

wookey — 2009-05-07T18:48:05+00:00

indeed - as these things go, that's relatively painless. (I know - I maintain the build system for balloonboard and it's a very hard thing to do well).

We (well, Jim Rayner actually) just built android for the balloon (which incidentally does produce a genuinely free platform, including GSM, although calling it a 'phone' would be a bit of stretch for most people).
http://balloonboard.org/balloonwiki/AndroidBalloon

This on-screen keyboard you speak of

corbet — 2009-05-07T16:22:36+00:00

"firmware (1.1)" means that you're running version 1.1 of the firmware.

This on-screen keyboard you speak of

felixfix — 2009-05-07T16:19:50+00:00

I tried both the contacts program and an independent program, neither of which did much other than move the cursor when I tapped.

Which of the various settings values corresponds to the 1.5 build? There are firmware (1.1), baseband (long weird string), kernel (2.6.25-01845), and build number (kila-user 1.1 plat-rc33 ...).

This on-screen keyboard you speak of

corbet — 2009-05-07T15:23:13+00:00

Tap on a text-entry widget with your finger with the device closed to get the on-screen keyboard.

I'm not sure about 1.5 for G1 handsets. I thought that T-Mobile was rolling it out in the US, but I'm far from certain of that.

This on-screen keyboard you speak of

felixfix — 2009-05-07T15:20:56+00:00

I have the non-ADP version of this phone. How does one get the 1.5 version -- is this still in development and not for release to customers? If I already have it, how does one activate the on-screen keyboard? I haven't seen any difference in apps which need a keyboard -- they still stare dumbly at me until I slide the keyboard open.

Updating and rebuilding Android

nbd — 2009-05-07T12:41:13+00:00

[shameless plug]
I'm working on integrating the Android software stack in OpenWrt. Much of the core of Android 1.5 is already running, but some work is still left to be done until it's usable.

Why am I doing this? Easy:
- OpenWrt is much easier to keep consistent across multiple platforms, because of the way it keeps kernel patches and configs (that's right, we don't fork entire kernel trees).
- OpenWrt allows you to integrate new packages by writing a small makefile and optionally throwing in some patches (which it also helps you create), so there's no need to fork the whole thing.
- OpenWrt allows you to shortcut parts of the build system, so you don't have to spend minutes (or hours, depending on your build machine's performance) waiting for a huge recompile just because you happened to change a core library, which forces a rebuild of build tools, which forces a rebuild of everything those tools generated.

My other goal is to keep the entire thing free of dependencies on external binaries where possible, and turn the Google-forked external libraries into OpenWrt makefiles and patches.

I'll try to keep people updated on my progress on my blog:
http://nbd.name/blog/?p=36
http://nbd.name/blog/?p=48
[/shameless plug]

Updating and rebuilding Android

fb — 2009-05-07T10:38:03+00:00

Great article! I had been curious about how to build my own distribution, and now I can give it a try.

You should IMVHO at least mention that the most active image builder for the G1 these days is Haykuro. Another piece of information missing is that most of the G1/ADP hacking activity seems to be centered around XDA forums.

Updating and rebuilding Android

bcopeland — 2009-05-07T10:16:33+00:00

The easiest way to update the kernel, in my experience, is to build it from the msm tree as our editor suggests, then take the HTC-supplied 1.5 initramfs (you can get it from the boot.img in the update zip file and unpack it with split_bootimg.pl). That way you don't have to bother with building any of the android userland.

After that you can simply turn on the phone while holding the back button and plugged into USB, then do 'fastboot boot your_bzImage their_boot.img-ramdisk.cpio' which will do a one-time boot of your kernel, then revert to the installed one on the next power cycle.

The android init setup seems to be somewhat tied to the kernel version so matching versions is good -- the 1.1 init will not work with the 2.6.29 kernel, for example. 1.5 does, sort of.

Clean Build Systems

dlang — 2009-05-07T06:48:02+00:00

remember, this isn't just building a single application, this is building an entire distribution (albeit a small one)

the fact that it was as easy as Jon made it sound to recreate (almost) the entire distribution is impressive.

even gentoo (which focuses on this area) takes as much, if not more effort to build a system from source.

as far as the time it takes to build, I suspect that the google people only rebuild their particular app when they change it, and they also probably have build farms of high-speed machiens (or at least a LOT of medium speed machines ;-) so that it takes less wall clock time.

Clean Build Systems

alex — 2009-05-07T06:35:16+00:00

It's sad to see the build system is such a prickly mess. Most
successful FLOSS projects have long since known that a hairy build
system is a barrier to entry for potential new contributors to the
code base.

That's to say nothing of the state there QA must be in. Being able to
automatically build a new image from source every time something is
checked in is basic QA functionality. I would of expected better from
the high flying whizz kids at the big G.

Updating and rebuilding Android

vmlinuz — 2009-05-07T04:16:28+00:00

Just got one minor comment:

One has to wonder, though, what inspired the Android developers to dedicate a significant chunk of scarce screen space to a "smiley" key.

When you define a text entry widget in an Android application, you can hint to the system what sort of text is going to entered, and different keyboards are used for different purposes. If it's defined as a "short message" widget, it'll have a smiley button, if it's defined as a "URI" it'll have a .com button, and so on. It's actually a pretty neat feature, as a developer, because it gives your application a nice level of polish for essentially no work on your part.