ELC: Android and the community

Greg Kroah-Hartman delivered some "tough love" to Android in his keynote at this year's Embedded Linux Conference (ELC). He is very clearly excited about Android and what it can do—uses it daily as his regular phone—but is unhappy with Google's lack of community engagement. There is hope that things will change, he said; there has been a fair amount of "introspection" at Google that he hopes will lead it in a more community-oriented direction.

CE Linux Forum architecture group chair and ELC organizer Tim Bird introduced Kroah-Hartman by noting how many kernel subsystems he maintains: eight. The "amount of work that Greg does in the Linux kernel is really hard to comprehend", Bird said. In addition, he has a knack for "involving people in the community". Kroah-Hartman is "omnipresent" in kernel circles and "when he sees a problem, he just comes up with a solution for it". It would seem that the keynote was targeted at solving the "Android problem".

As part of his disclaimer, Kroah-Hartman noted that he comes at the problem as an experienced kernel developer who also has a background in the embedded space. While he works on MeeGo for Novell as his day job, "they don't even realize I'm here giving this talk". He also noted that he is only looking at the Android kernel, as the user space is "weird" and won't be part of his talk.

Five years ago, phone manufacturers approached the kernel developers because they wanted to use Linux on phones. They needed a stable kernel and X server, along with a free Java. The latter was the sticking point because, at that time, there was no free Java—and it isn't something that the kernel hackers could provide. But Android changed that, providing the third piece that the phone makers need.

He pointed out that all of the complaints he was about to make about Android could be fixed "tomorrow" and that the Android user-space applications wouldn't have to change at all. But it's not something that the kernel community can do alone, Google needs to be involved to change the libraries that make up the platform.

Right moves

There were several things that Google did right, Kroah-Hartman said, starting with its choice of Linux. In an aside, he noted that all phone manufacturers bring up their phones using Linux, including Apple with the iPhone; "a little-known fact". He also lauded Google for following the kernel license, which is something that Palm didn't initially do with WebOS, he said. He pointed to android.git.kernel.org as a "wonderful site" that contains all of the Android code in easily accessible Git repositories. But "that's all the good".

Wrong moves

The list of things that Google got "wrong" starts with android.git.kernel.org because it is so disorganized and chaotic. There are six different kernel trees available there with 33 separate branches. There are three branches of 2.6.34, which is "not even released yet". The trees go back to 2.6.25 and some of the older trees are still being updated. Some of the branches are for different hardware, but some aren't.

There are also things like an "old stale Linus tree" in the collection, along with two standalone drivers in a separate repository. One of those drivers has 13 branches, "not all for just one kernel version". There is also an empty repository—with four branches. "It's a mess", he said, and Google is "burying the code in public". That's a common thing for companies to do; "they are doing it well, and it's crap".

He looked at a branch based on 2.6.34-rc2, noting that there were 283 files changed, with 47,715 lines added and 363 lines removed. Half of that was driver code, 30% for filesystems, 15% for architecture-specific code, and the "really scary one" was the 5% that were changes to core kernel code. That "changes the kernel in ways that make it not Linux", he said.

Tons of drivers

Kroah-Hartman then went through a long list of drivers that were in Google's changes, but were not upstream. For most of them, there is "no reason this code needs to be out of the kernel". Some of the changes like the pmem driver, which the Google developers have publicly called "voodoo", and a logging infrastructure that should be done in user space, are not good candidates for the upstream kernel, but the vast majority is.

The big problem with all of this code is that it implements features and fixes bugs that others in the community need. Google rewrote the USB gadget code because the kernel version couldn't easily support multi-function gadgets in 2007, but never got it into the mainline. So, Samsung had to fix the mainline gadget code. The Android developers also fixed a "bunch of bugs" in the Bluetooth code, which is a "nasty protocol to get right", but they never pushed it upstream.

There are lots of small changes in various locations that make it much harder for those who want to port Android to new hardware, he said. Missing a three-line change in the inotify code will cause some applications to fail. There is also a special Android-specific /proc file type that needs to be ported into any kernel bound for Android. And on and on.

Wakelocks

One of the big problems with much of the Android code is wakelocks, which are an Android-specific power-management feature. There are wakelocks in "every Android driver", but wakelocks are not in the mainline. Kernel developers have tried tearing the wakelocks out of the drivers, but then the code diverges quickly. The right solution is to get wakelocks upstream, which has gotten close to happening, but hasn't.

Wakelocks are a good example of how Google is ignoring the community, Kroah-Hartman said. The wakelock code would be submitted for inclusion, there would be some discussion and then the Android developers would "disappear for six months". When they returned, the process would start over. The solution is to "submit and be persistent", to answer the questions, and participate in the discussion. Wakelocks were "one patch submission away from being accepted", he said, but the developers disappeared again.

Ignoring the community

Google is allowed to ignore the community, but "it's sad". He took the Android drivers into the staging tree and Google said it would help get them into mainline shape, but they didn't. So Kroah-Hartman had to remove them, which means that no one else gets the benefit of those drivers. When the drivers were in the staging tree, Fujitsu was working on fixing bugs in the Android low memory killer to use on their "big supercomputer stuff", but now that work has been dropped.

The community wants the code, but Google thinks that it's unique and doesn't need to work with the community. "Don't think you are unique, you aren't", he said. He was not picking on Google, because "this applies to everyone". Ironically, the Google server folks would like to use some of the changes that are in the Android tree, but they aren't upstream.

It's just a fork

Google's response has been that "it's just a fork, no big deal". Every embedded shop has done a fork and it is abiding by the license, "so who cares?". But Kroah-Hartman wants to see the Android platform succeed and Google is making its partners' jobs harder by making them depend on extra code that needs to be ported to their hardware. In order for the platform to succeed, Google needs to work with the community. If it wants to do it alone, "that's fine", but "they don't and we don't want them to". He noted that Qualcomm, a company not noted for community engagement, had approached the kernel developers complaining about the Android situation, which is a glaring indicator of a problem.

That was the end of the "fire and brimstone" portion of Kroah-Hartman's talk. Looking to the future, he said there is "hope" (accompanied by a slide of the Android logo in the style of Shephard Fairey's iconic Barack Obama "Hope" poster). There will be a meeting very soon where the kernel hackers and Android developers are going to "lock themselves in a room and hash this all out", he said.

For vendors using Android, Kroah-Hartman recommended that they "push on Google to make these changes", and that he will be "pushing as hard as I can on the kernel side". He pointed out that it takes less time to get the code upstream than will be spent maintaining it moving forward. Android is "not an end-of-life project", with 50 new handsets announced at the last trade show. "It's not that much work, truly", he said, and there are plenty of consultants that would be willing to help if the Open Handset Alliance wanted to fund this work.

It was noted that no one from Google attended the talk, and an audience member asked how that could be. Kroah-Hartman seemed a bit disappointed that there were no Google representatives, and did say that they had been invited. Perhaps it was the "long drive from the South Bay" that kept them away, he noted wryly. It may also be that memories of his Canonical bashing from the 2008 Linux Plumbers Conference linger; Google folks may not have wanted to sit through something like that.

The Android user space is "divergent from everything you ever thought about Linux user space", which is "great", he said. The idea that you can run something entirely different on top of the kernel makes it very interesting. Google could replace Linux with BSD or something else and all of the applications would still run. But Kroah-Hartman would like to see "Linux succeed for this market" and Google has followed the kernel license as they should; "I want to reward them for that". He has offered to help before and reiterated that offer. It's clear that he is a big fan of the platform, and is just trying to help fix the problems that he sees.