GUADEC: porting GNOME to Android [LWN.net]

By Nathan Willis
August 15, 2012

Rightly or wrongly, GNOME is regarded by many casual users as "a Linux desktop environment." It also runs on various BSD and Solaris flavors, of course, and there has been considerable discussion recently of developing a "GNOME OS" deliverable suitable for tablets and other devices. But at GUADEC 2012, there was yet another spin on the redefinition of GNOME: porting several underlying GNOME frameworks to Android.

A team of developers from Collabora hosted the relevant session, which was titled D-Bus, PulseAudio, GStreamer and Telepathy In the Palm of your Hand. Reynaldo Verdejo Pinochet acted as ringmaster for the other speakers, and provided an introduction to the overall effort. As many others have observed, GNOME and other open source projects are facing a "form-factor challenge", he said. Desktop approaches to user interfaces do not work as well on small-screen devices, and mobile device users exhibit different usage patterns: rapidly interleaving multiple tasks rather than camping out in one application.

The open source developer community is currently failing to provide a bridge from the desktop to mobile devices, he said. Meanwhile, the system-on-chip vendors are turning their full attention to Android — a platform into whose development the community has no say. Collabora's solution, he said, is not to "fix" Android or to "fix" the users, but instead to enable the GNOME project's technology to run on Android. As a result, developers and users will have more choice, and the GNOME components will continue to be relevant to both.

The bulk of the session was then taken up by a series of short status updates from Collabora developers working on "smallifying" frameworks used by GNOME applications. Covered were Android ports of PulseAudio, Wayland, Telepathy, and GStreamer. Despite the title, D-Bus was not discussed separately in the talk; Verdejo said in an email that the D-Bus port was undertaken as part of the Telepathy effort. Verdejo closed out the session with general information for application developers looking to port GNOME applications to Android.

Porting stories

Arun Raghavan spoke first, explaining the PulseAudio port for Android. Android's native audio subsystem is called AudioFlinger (AF). It is a software mixer like PulseAudio, he explained, but it pales in comparison to PulseAudio for features. "It's not as nice as PulseAudio, and you can look at code if you don't believe me." In particular, PulseAudio provides flexible audio routing (including network routing), dynamically-adjustable latency (which prevents glitches, saves power, and simplifies the work required from applications), and support for passing compressed audio (in contrast to AF, which provides only PCM audio transport).

Although the project's original goal was to install PulseAudio in parallel with AF, Raghavan said, that quickly proved impossible due to how integrated AF is into Android. So instead, Raghavan focused on building and installing PulseAudio as a complete replacement for AF, using the Galaxy Nexus phone as the target platform. First, he successfully compiled PulseAudio and got it to run and play sound on the Galaxy Nexus. He then started work on using PulseAudio to emulate AF's playback API "AudioTrack," a task that he said is almost complete. Still in progress is emulation of AF's policy framework (which controls volume settings, when playback is allowed, and so forth). After that, the next item on the to-do list is "AudioRecord," the AF audio capture API.

Next, Daniel Stone presented the Wayland port. Once again, the goal was to offer Wayland as an alternative to a built-in Android service — in this case, SurfaceFlinger. SurfaceFlinger provides buffer transport functions analogous to Wayland, and composition functions like those of Wayland's reference compositor Weston. Collabora's port is mostly complete, and is currently able to run Wayland demos. Still in progress is integrating Wayland into the Android stack so that it can run side-by-side with SurfaceFlinger, a challenge that includes both running Wayland and SurfaceFlinger applications at once, and handling clipboard events, task switching, and other functions. Still to come is styling, which would make Wayland applications seamlessly match the SurfaceFlinger look.

The Wayland work ran into the usual porting problems, he said, but there were special difficulties as well. First, Wayland relies on several kernel features too new to be found in the Android kernel (such as timerfd and signalfd). Second, the Android graphics drivers are "terrible" at buffer management in comparison to the desktop. Whereas Mesa provides full EGLImage sharing through the Direct Rendering Manager (DRM) and seamless zero-copy transfers (which Wayland can take advantage of through the wl_drm extension), EGLImage is an "absolute crapshoot" on Android. The quality of the implementation varies drastically between hardware vendors, and the graphics drivers are almost always closed.

Alvaro Soliverez presented the port of the Telepathy communication framework. Telepathy is modular, supporting a long list of pluggable back-ends (from XMPP to SIP, to proprietary networks like MSN) and bindings for a number of programming languages and toolkits. Soliverez's port picked the subset most useful for Android: the Gabble XMPP module, link-local XMPP with Salut (using Avahi service discovery), and Java bindings.

The port is fully functional, he said, and the patches have been sent upstream to Telepathy. Android users can download and install the Telepathy service as a stand-alone application. In addition to the protocols mentioned above, the account management and status management functionality works, and the GLib bindings are in place for developers. Ports of the rest of the framework are underway, he said, although assistance would be welcome.

Verdejo discussed the GStreamer port. GStreamer is already successfully in use on other platforms, where it can serve as a bridge to the operating system's native multimedia framework (such as QuickTime on Mac OS X or DirectShow on Windows). The GStreamer Android port is intended to be a complete port of the framework, putting Android on par with the other platforms. This required the development of two new elements to interface to Android's media layer: GSTSurfaceFlingerSink and GSTAudioFlingerSink. These are sink elements, which allow applications' GStreamer pipelines to play their output on Android.

But the Android platform "is crippled on purpose", he said, offering no real native development story. Instead, all of the emphasis is placed on writing Dalvik applications. Consequently, Collabora decided to build GStreamer as a system-level service as well. It developed two other elements, GSTPlayer and GSTMetaDataRetriver, which can function as replacements for Android's built-in media playback functionality. Up next are new elements to handle direct multimedia output over OpenGL ES and OpenSL ES, and capture elements for camera and microphone input. The OpenGL ES and OpenSL ES elements will eventually replace the existing Android GStreamer sinks so that GStreamer pipelines will not need to depend on the Android API layer. All of the GStreamer work is being pushed upstream and will be included in the GStreamer SDK.

Developing applications

~~Verdejo~~ Raghavan closed out the session with general advice for other developers interested in porting their software to Android. There are two general approaches to consider, he said: the native development kit (NDK) and the complete system-integration route. The NDK is Google's official offering, and it places some requirements on the developer — specifically, all of the application's functionality must be in libraries, which are then accessed by Java code using JNI. The libraries must also be bundled with applications.

The alternative route is to develop the software integrated into the Android source, which means it must subsequently be distributed as a custom Android build. The system integration route does allow one to make the software functionality available to the entire platform, he said, rather than just to specific applications (which is a limitation of bundling the library with an NDK application).

Whichever route one chooses, he said, there are "gotchas" to look out for. Android exhibits not-invented-here (NIH) syndrome, he said, right down to the basic C library: it uses Bionic, a "bastardized" version of libc with missing APIs and hobbled features. Android also uses unique Git repository and patch management tools written by Google, he added. But the main sticking point is Android's specialized build system, which relies on Android-specific makefiles. Maintaining those alongside the makefiles needed for another platform can be a considerable headache.

To simplify the process, Collabora created a tool called Androgenizer. Developers need to set up only a single top-level Android.mk file for the project and Android target rules in Makefile.am. The target rules call Androgenizer, which generates the makefiles Android expects (using Google's macros and Android-specific variables). Androgenizer can be used to generate makefiles appropriate for either NDK or system-integration Android builds, by checking for an environment variable called ANDROGENIZER_NDK.

The Androgenizer tool does not make building for Android trivial, he said, but once you figure it out, it only takes about 30 minutes for each new project. Although the process is not hard, Verdejo encouraged anyone working on an Android port to get in touch, saying the company is interested in helping.

All things considered, the ports of these and other frameworks from desktop GNOME (there are several other projects on display at the company's Git repository) offer an interesting take on the desktop-versus-mobile "problem." Typically the massive increase in mobile Internet devices is talked about as a sign that desktop projects are doomed; Collabora's GNOME-to-Android ports show that developers do not need to take that as a foregone conclusion, even for a tightly-controlled platform like Android.

[The author would like to thank the GNOME Foundation for travel assistance to A Coruña for GUADEC.]

Index entries for this article
Conference	GUADEC/2012

to post comments

GUADEC: porting GNOME to Android

Posted Aug 16, 2012 19:01 UTC (Thu) by jmorris42 (guest, #2203) [Link] (1 responses)

The problem is there is no easy way to distribute new software to Android devices. Heck, just getting an updated Android is a crapshoot unless you have a very popular device. Ask the cyanogen mod guys.

With a PC we have a open and understood boot process and fairly standardized hardware. Removing the OEM preloaded OS and installing a free OS on a PC is therefore fairly easy. Each Android device has to be undertaken as a new port. Without even a plan to crack that that problem there isn't much point trying to make a tablet or phone version of any free OS.

GUADEC: porting GNOME to Android

Posted Aug 17, 2012 2:58 UTC (Fri) by Ford_Prefect (subscriber, #36934) [Link]

This is just how the ARM world works (at least for now). Nevertheless CyanogenMod _does_ run on a very large number of devices. Heck, hardware manufacturers have even become a lot friendlier, making it possible to unlock bootloaders and actually supplying hardware to devs to help the port.

I disagree with you about there not being a point in try to make a phone/tablet version -- we don't have to target every device, just as many as we can (and we can happily just aim for Google's reference devices to start with). More devices will end up being supported by the community if we provide a compelling reason to use our platform.

GUADEC: porting GNOME to Android

Posted Aug 17, 2012 18:37 UTC (Fri) by alankila (guest, #47141) [Link] (8 responses)

Sounds like veritable nightmare combination of ideas to me. In literally every way, it is going to make Android worse. I can't even take this article entirely seriously, but let it be said that I couldn't personally be in greater disagreement about everything proposed here.

1. Autotools could be back. Android was fine without it. In general GNU makefiles are awesome and can do lots of things -- as a case in point, Android can build everything with just GNU makefiles. Can't other people just start writing those instead, and rather drop autotools entirely? If you have taken a look at Android.mk files, they tend to just list the module name, the source files, and some boilerplate that tends to be very common in all Android.mk files. Is this really too much to maintain?

2. PulseAudio. Despite the popular theory of it being superior, PA doesn't support OpenSL (audio effects on streams). Until it does, I am going to be bitterly opposed to any hint of it replacing AF. Android is one of the very few, if not the only, Linux-based systems that can do digital sound processing in user friendly way that anybody can use. The sound-producing application doesn't need to be aware of effects being applied, and nice GUIs to do it have be designed.

3. GStreamer. Android already has a media framework. I think gstreamer is responsible for videos playing without sound on OS X. I know this is an anecdotal punch under the belt, but IMHO, gstreamer has never worked properly for me. On Android, it can't be plausibly be distributed as common component to be used by multiple applications, anyway, so unless you like multi-megabyte application bundles, GStreamer remains a complete non-starter, regardless of its merits of faults.

4. Wayland. It's not vaporware, but also not ready for prime-time any time soon. The fact this plan requires complicated interaction with random vendor-provided GPU drivers that are only built for use with SurfaceFlinger makes me think that there's no hope with trying to make multiple graphics systems work simultaneously on one screen. What exactly is the story for needing Wayland at all? For instance, if GTK+ applications must run on Android, why can't GTK+ talk with SurfaceFlinger?

I know what I write here is going to come across as needlessly harsh, but the common theme I read here is "more fragmentation for android". It's not like Android wants or needs any of this. And if people seriously want to replace AF or SF with PulseAudio or Wayland, surely it would make most sense to talk with Google's Android team about it? The only consolation I have is that I can't see any of this work going anywhere, so it can be safely ignored.

GUADEC: porting GNOME to Android

Posted Aug 17, 2012 19:47 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

PulseAudio certainly supports system-wide and per-stream effects. It's just that nobody has written the required plugins.

GUADEC: porting GNOME to Android

Posted Aug 18, 2012 17:35 UTC (Sat) by alankila (guest, #47141) [Link] (2 responses)

Last time I looked, the plugins couldn't be loaded dynamically or configured remotely. Maybe this has changed, but it's far cry from the productized experience that AF offers with OpenSL.

For security reasons, random DSP plugins can't be done on Android either, but the system must at boot-up time declare static list of files to be used for such purposes. But all ROM modders can change this list, so there's that. Afterwards, applications that wish to have their audio stream managed publish their session id to the wider system to work with, and DSP-related applications hear these broadcasts and can use RPC shims to connect to the main mediaserver process to add and remove these effects on a per-session basis.

It is obvious that it takes quite some effort to get from "can be done in principle" to something that can be distributed to end users. My belief is that PA's DSP story is still in the "can be done in principle" stage...

GUADEC: porting GNOME to Android

Posted Aug 18, 2012 18:05 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> For security reasons, random DSP plugins can't be done on Android either,
Nope. DSP is often done in _hardware_ on Android (for instance, on Galaxy S phones) and sometimes sound stream doesn't even touch the CPU. So that's the reason for non-dynamic AF plugins.

GUADEC: porting GNOME to Android

Posted Aug 18, 2012 21:16 UTC (Sat) by alankila (guest, #47141) [Link]

Oh, some DSP probably is. The sort of thing I have in mind is not. I am not aware of any mediaplayer-based sound streams that would not touch the CPU, although as far as I know phone conversations occur without the main CPU being able to interface with either of those streams.

In any case, Android vendors can provide vendor-specific implementations of AudioEffects, and I suppose it might be possible to hardware-accelerate more of the AudioFlinger API. There's a lot of maybes to this, because last time I looked there was but one digital stream that everything got mixed to, and whatever "DSP" was involved was concerned entirely with the mechanics of generating analog audio from that single PCM data.

So there's multiple kinds of things called "DSP" here. The fact that custom DSP plugins are not possible is obvious given that they must be loaded to a root-owned process implemented in C which must not crash and must be as realtime as possible. So random code, or offloadng processing by some IPC approach is probably out.

GUADEC: porting GNOME to Android

Posted Aug 17, 2012 22:24 UTC (Fri) by daniels (subscriber, #16193) [Link] (3 responses)

You seem pretty vehemently opposed to all of this, so I don't think there's much/any point trying to convince you, but for the record ...

1) Static Makefiles don't support feature detection and configuration in the same way autotools do. Obviously they're not perfect, but they're a damn sight better than just writing plain static Makefiles. Converting the build system of every single project we've been working with is obviously a non-starter for any viable project.

2) This is completely possible with PulseAudio, and many vendors who've deployed it have written their own custom plugins to process streams, some extremely complex, and all without any need for custom knowledge on behalf of the applications. It also provides a number of very real benefits such as drastically reduced latency from source to hardware, and strong power savings. You can read more about this here: http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinge...

3) StageFright has a number of issues, and is not particularly well-suited to more than just basic media playback (and even then ...). GStreamer, on the other hand, has been deployed in all major distributions for years, as well as part of uncountable devices (both consumer/gadget and enterprise), some of which have some hugely taxing requirements. I'm sorry to hear that it anecdotally doesn't work for you on OS X, but it certainly seems to work very well for tens of millions of people. In our Android port, it can indeed be used as part of the base system, because we're developing a branch of AOSP to use as a base for integration, rather than something you can just install from the Google Play store.

4) Wayland isn't, as you note, 1.0 yet (although it is 0.95, with a frozen core protocol), but it's something we're very interested in, as you can see from its commit stats. We think it has a lot to offer in terms of fluid and beautiful UIs which people particularly want in the mobile space, as well as media presentation. We're getting a lot of interest from both hardware vendors and their customers, so a port definitely seemed prudent.

These are all proven, solid technologies (with the exception of Wayland, which is obviously still maturing), which people have deployed and want to continue deploying on a wide range of hardware. Hardware vendors are now turning their attention to Android and only doing base enablement for the Android stack, and the number of hardware vendors who actually care about getting a proper Linux stack up on their hardware is small and diminishing every day. We don't want Android to become a completely separate silo to generic Linux, so we're trying both to offer people with Android-only hardware a much better stack, and also to use the interest from Android deployments to keep that stack more relevant so it can continue to grow and improve for everyone.

It would be nice if that weren't the reality, but what can you do?

Most of these technologies will never become mainstream Android, partially because of licensing concerns, but also partially because the Android development model is based around being almost entirely in-house within Google, rather than the traditional inclusive open source model.

Anyway, as you say, none of this will affect you in any way, so no need to be worried about it. :)

(Obviously I'm speaking for myself and not necessarily for Collabora here. One disclaimer from my brief presentation that Jake left out of the article was that I wasn't actually working on the Wayland port: it was all done by Pekka 'pq' Paalanen, but unfortunately he couldn't make it to GUADEC, so I spoke on his behalf.)

GUADEC: porting GNOME to Android

Posted Aug 17, 2012 22:56 UTC (Fri) by daniels (subscriber, #16193) [Link]

Nate, not Jake. Sorry!

GUADEC: porting GNOME to Android

Posted Aug 18, 2012 17:56 UTC (Sat) by alankila (guest, #47141) [Link] (1 responses)

1. At least autotools insists on running programs to do the feature detection which tends to be a nonstarter for cross-compiling use cases. In any case, Android's notion of the world is monolithic so there's but one configuration, one compiler, all libraries provided by the project itself if you need them, etc. So there's no features to detect and/or configure. Makefiles, though support if statements, so they are not entirely static in nature.

2. Lower latency, power usage, and freedom of sample rate are not useless features, and favor PA -- at least in some use cases. But until it has a comparable DSP story, it is inferior in other practical ways. I'm just bringing this out every time PA gets mentioned against AF so that people would keep in mind that PA is not a superior replacement in every way currently. That article you linked also mentions the OpenSL effects because I pointed them out last time too.

3. Okay. I suppose it ought to work for the basic use cases. I guess I've been really unlucky. No points for planning on distribute a modified android, though. That's precisely the sort of fragmentation Android doesn't need... ;-(

4. Does this imply that Wayland doing android-like UI is better at the job than SurfaceFlinger? Or is this just a general comparison against X? Of particular note is the fact that android 4 apparently managed to add hardware accelerated UI composition where SF apparently composes the application's UI by overlapping various buffers. It sounds fairly optimal already...

GUADEC: porting GNOME to Android

Posted Aug 19, 2012 11:14 UTC (Sun) by smcv (subscriber, #53363) [Link]

> At least autotools insists on running programs to do the feature
> detection which tends to be a nonstarter for cross-compiling use cases

It is possible for a configure.ac to require this, but that feature is relatively rarely-used (most Autoconf checks only require preprocessing, compiling or sometimes linking a program for the embedded architecture, not actually running it).

When a check can't be done without running a program for the embedded architecture (e.g. dbus checks for support for Linux/Solaris-style abstract Unix sockets), in a properly written configure.ac it's possible to tell configure what the result will be, which stops it from actually trying the check. For instance, when building for Android you know that the kernel does support abstract Unix sockets, and there's a configure option to say so.

GUADEC: porting GNOME to Android

Posted Aug 18, 2012 0:04 UTC (Sat) by swetland (guest, #63414) [Link]

Is there any interest in just getting GTK+ (or X if you want to go lower level) working on top of SurfaceFlinger surfaces and arrange for the appropriate libraries to be built in an NDK-compatible way so one could port existing apps such that they'd run on any Android system from Gingerbread forward (which is quite a lot of devices in the field)?

I may be misunderstanding the goals, but it would seem like there are simpler paths to getting existing Linux apps running on Android than replacing big chunks of platform internals...

GUADEC: porting GNOME to Android

Posted Aug 25, 2012 14:02 UTC (Sat) by nilesh (guest, #37042) [Link]

Isn't Ubuntu for Android taking care of a lot of the low-level stuff that could be useful here as well? Granted, that's work in progress and not yet available, but still there could some code-sharing possible here.