Greg Kroah-Hartman delivered some "tough love" to Android in his keynote at this year's Embedded Linux
Conference (ELC). He is very clearly excited about Android and what it can
do—uses it daily as his regular phone—but is unhappy with Google's
lack of community engagement. There is hope that things will change, he
said; there has been a fair amount of "introspection" at
Google that he hopes will lead it in a more community-oriented direction.
CE Linux Forum architecture group chair and ELC organizer Tim Bird
introduced Kroah-Hartman by noting how many kernel subsystems he
maintains: eight. The "amount of work that Greg does in the Linux
kernel is really hard to comprehend", Bird said. In addition, he
has a knack for
"involving people in the community". Kroah-Hartman is
"omnipresent" in kernel circles and "when he sees a problem, he just comes
up with a solution for it". It would seem that the keynote was
targeted at solving the "Android problem".
As part of his disclaimer, Kroah-Hartman noted that he comes at the problem
as an experienced kernel developer who also has a background in the
embedded space. While he works on MeeGo for Novell as his day job,
"they don't even realize I'm here giving this talk". He also
noted that he is only looking at the Android kernel, as the user space is
"weird" and won't be part of his talk.
Five years ago, phone manufacturers approached the kernel developers
because they wanted to use Linux on phones. They needed a stable kernel
and X server, along with a free Java. The latter was the sticking point
because, at that time, there was no free Java—and it isn't something
that the kernel hackers could provide. But Android changed that,
providing the third piece that the phone makers need.
He pointed out that all of the complaints he was about to make about
Android could be fixed "tomorrow" and that the Android
user-space applications wouldn't have to change at all. But it's not
something that the kernel community can do alone, Google needs to be
involved to change the libraries that make up the platform.
There were several things that Google did right, Kroah-Hartman said,
starting with its choice of Linux. In an aside, he noted that all phone
manufacturers bring up their phones using Linux, including Apple with the
iPhone; "a little-known fact". He also lauded Google for
following the kernel license, which is something that Palm didn't initially
do with WebOS, he said. He pointed to android.git.kernel.org as a
"wonderful site" that contains all of the Android code in
easily accessible Git repositories. But "that's all the good".
The list of things that Google got "wrong" starts with
android.git.kernel.org because it is so disorganized and chaotic. There
are six different kernel trees available there with 33 separate branches.
There are three branches of 2.6.34, which is "not even released
yet". The trees go back to 2.6.25 and some of the older trees are
still being updated. Some of the branches are for different hardware, but
There are also things like an "old stale Linus tree" in the
collection, along with two standalone drivers in a separate repository.
One of those drivers has 13 branches, "not all for just one kernel
version". There is also an empty repository—with four
branches. "It's a mess", he said, and Google is "burying
the code in public". That's a common thing for companies to do;
"they are doing it well, and it's crap".
He looked at a branch based on 2.6.34-rc2, noting that there were 283 files
changed, with 47,715 lines added and 363 lines removed. Half of that was
driver code, 30% for filesystems, 15% for architecture-specific code, and
the "really scary one" was the 5% that were changes to core
kernel code. That "changes the kernel in ways that make it not
Linux", he said.
Tons of drivers
Kroah-Hartman then went through a long list of drivers that were in
Google's changes, but were not upstream. For most of them, there is
"no reason this code needs to be out of the kernel". Some of
the changes like the pmem driver, which the Google developers have publicly
called "voodoo", and a logging infrastructure that should be
done in user space, are not good candidates for the upstream kernel, but
the vast majority is.
The big problem with all of this code is that it implements features and
fixes bugs that others in the community need. Google rewrote the USB
gadget code because the kernel version couldn't easily support
multi-function gadgets in 2007, but never got it into the mainline. So,
Samsung had to fix the mainline gadget code. The Android developers also
fixed a "bunch of bugs" in the Bluetooth code, which is a
"nasty protocol to get right", but they never pushed it upstream.
There are lots of small changes in various locations that make it much
harder for those who want to port Android to new hardware, he said. Missing
a three-line change in the inotify code will cause some
applications to fail. There is also a special Android-specific
/proc file type that needs to be ported into any kernel bound for
Android. And on and on.
One of the big problems with much of the Android code is wakelocks, which are an
Android-specific power-management feature. There are wakelocks in
"every Android driver", but wakelocks are not in the
mainline. Kernel developers have tried tearing the wakelocks out of the
drivers, but then the code diverges quickly. The right solution is to get
wakelocks upstream, which has gotten close to happening, but hasn't.
Wakelocks are a good example of how Google is ignoring the community,
Kroah-Hartman said. The wakelock code would be submitted for inclusion,
there would be some discussion and then the Android developers would
"disappear for six months". When they returned, the process
would start over. The solution is to "submit and be
persistent", to answer the questions, and participate in the
discussion. Wakelocks were "one patch submission away from being
accepted", he said, but the developers disappeared again.
Ignoring the community
Google is allowed to ignore the community, but "it's sad". He
took the Android drivers into the staging tree and Google said it would
help get them into mainline shape, but they didn't. So Kroah-Hartman
had to remove them, which means that no one else gets the benefit of those
drivers. When the drivers were in the staging tree, Fujitsu was working on
fixing bugs in the Android low memory killer to use on their "big
supercomputer stuff", but now that work has been dropped.
The community wants the code, but Google thinks that it's unique and
doesn't need to work with the community.
"Don't think you are unique, you aren't", he said. He was not
picking on Google, because "this applies to everyone".
Ironically, the Google server folks would like to use some of the changes
that are in the Android tree, but they aren't upstream.
It's just a fork
Google's response has been that "it's just a fork, no big
deal". Every embedded shop has done a fork and it is abiding by the
license, "so who cares?". But Kroah-Hartman wants to see the
Android platform succeed and Google is making its partners' jobs harder by
making them depend on extra code that needs to be ported to their
hardware. In order for the platform to succeed, Google needs to work with
the community. If it wants to do it alone, "that's fine", but
"they don't and we don't want them to". He noted that
Qualcomm, a company not noted for community engagement, had approached the
kernel developers complaining about the Android situation, which is a
glaring indicator of a problem.
That was the end of the "fire and brimstone"
portion of Kroah-Hartman's talk. Looking to the future, he said there is
"hope" (accompanied by a slide of the Android logo in the
style of Shephard Fairey's iconic Barack Obama "Hope" poster). There will
be a meeting very soon where the
kernel hackers and Android developers are going to "lock themselves
in a room and hash this all out", he said.
For vendors using Android, Kroah-Hartman recommended that they "push
on Google to make these changes", and that he will be "pushing
as hard as I can on the kernel side". He pointed out that it takes
less time to get the code upstream than will be spent maintaining it moving
forward. Android is "not an end-of-life project", with 50 new
handsets announced at the last trade show. "It's not that much work,
truly", he said, and there are plenty of consultants that would be
willing to help if the Open Handset Alliance wanted to fund this work.
It was noted that no one from Google attended the talk, and an
audience member asked how that could be. Kroah-Hartman seemed a bit
disappointed that there were no Google representatives, and did say that
they had been invited. Perhaps it was the "long drive from the South
Bay" that kept them away, he noted wryly. It may also be that
memories of his Canonical
bashing from the 2008 Linux Plumbers Conference linger; Google folks
may not have wanted to sit through something like that.
The Android user space is "divergent from everything you ever thought
about Linux user space", which is "great", he said.
The idea that you can run something entirely different on top of the kernel
makes it very interesting. Google could replace Linux with BSD or
something else and all of the applications would still run. But
Kroah-Hartman would like to see "Linux succeed for this
market" and Google has followed the kernel license as they should;
"I want to reward them for that". He has offered to help
before and reiterated that offer. It's clear that he is a big fan of the
platform, and is just trying to help fix the problems that he sees.
to post comments)