The LPC Android microconference, part 1

The Linux Plumbers Android microconference was held in Seattle on August 20th and looked at a number of topics needing coordination between various players in the Android ecosystem. It was split up into two separate sessions; this summary covers the first three-hour session. Topics covered the state of the staging tree, USB gadgets and ConfigFS, running mainline on consumer devices, partitions and customization, a single binary image for multiple devices, Project Ara, and kdbus.

The microconference started after lunch, following the Graphics microconference, where there was also quite a bit of Android-related discussion. The Android microconference was held in two parts: in the afternoon and evening. In total, it went on for about six and a half hours until 8pm.

State of staging

After a brief outline of the schedule by microconference lead Karim Yaghmour, the sessions started with a review of the Android-related code in the staging tree and a quick discussion of what to do with it all, led by staging maintainer Greg Kroah-Hartman. Ashmem, timed_gpio and timed_output, the low-memory-killer, the sync framework, and ION were all listed, and one by one they were addressed.

Ashmem has close parallels with memfd, which is really only missing the memory-unpinning feature of ashmem. Memory unpinning, which marks memory as eligible to be discarded by the kernel, was unsuccessfully submitted upstream via the "volatile range" patches. While that effort has stalled out, it seems like ashmem going upstream as-is would be somewhat duplicative, and adding something like memory unpinning to memfd (generically or via an ioctl()/shrinker interface similar to what ashmem uses) would be preferred. The outcome of the discussion was to leave ashmem in staging for now and to revisit it again next year.

For the timed_gpio and timed_output drivers, it was claimed that no one was using them, and that the LED-trigger infrastructure should be sufficient to replace them. The Android developers mentioned that the Android vibrator hardware abstraction layer (HAL) was still using them, but shrugged about removing them. Everyone seemed to be OK with dropping it from the kernel, if only to see if anyone would complain, but many worried that vendors would just re-add it instead of moving to LED-triggers.

The mempressure notifier functionality was merged upstream to replace the low-memory-killer back in 3.10, and the Google developers did work to create a user-space low-memory killer daemon that uses it. Unfortunately, it was found not to perform suitably on real devices and the Google developers fell back to using the low-memory-killer again. Not enough debugging has been done to determine why exactly the mempressure notifiers aren't working well. So work is needed, and removing the in-kernel low-memory-killer would be premature. That said, the in-kernel low-memory-killer isn't problem-free, and has been getting bug reports and patches to change its heuristics from various different vendors. Since there is not a clear maintainer, there was a call for who might review submitted patches. Riley Andrews and John Stultz volunteered themselves.

Next up was the sync framework. As had been discussed in the Graphics microconference (Etherpad notes), sync points/fences still needed to be integrated with much of the atomic mode-setting work that was recently merged upstream. There is a need for an open-source user space before changes are merged into DRM; specifically, folks would like to see an open-source HWComposer implementation for Android that uses KMS/DRM (though there is some dispute over if that exists already or not), where that sort of sync integration would take place. So, at the moment, while it's clear it's on a number of folks' to-do lists, it's not clear that anyone is working on de-staging the code yet, so it will remain in place for now.

Finally, the ION memory allocator was discussed. As noted in the Graphics microconference, a few different proposals have been made, but there is no clear decision on what to do with them or which to prioritize. Since no real resolution was found through that discussion, folks explained there would be a mini-BOF the next day to try to hash things out. The core issue with ION is that, while it is used to solve the problem of how to allocate memory to share between different devices with different constraints, it requires that the applications using it have a detailed understanding of the constraints of the hardware, which makes it not very useful for writing applications that support different devices. There is also the problem that there aren't good examples of upstream users that need ION, but it continues to be needed for a number of out-of-tree solutions (which are difficult to merge without ION-like functionality properly upstream). New patches are being submitted against ION that add new features and try to establish things like device tree bindings, which Kroah-Hartman has been stalling for now. It shows that the longer it takes for an alternative solution to be merged, the more entrenched ION potentially becomes.

Kroah-Hartman also asked why nothing new has been added to staging, and it was explained that, while there is a fair amount of out-of-tree code used by Android, the remaining parts are not as independent as what has historically been put into staging. For the most part, what is left is stuff that modifies existing in-kernel code in a way that isn't generic enough or suited yet for upstream.

So the main outcome of the discussion was that timed_gpio would be dropped from staging, and that users should move to the LED-trigger infrastructure, or complain loudly.

USB gadgets and ConfigFS: status & future

The next topic was the "ConfigFS gadget driver" (slides [ODP]), which is a gadget driver that is controlled using ConfigFS. The Android developers are starting to use this driver as they are migrating away from the Android gadget driver, which was a forerunner to the ConfigFS gadget (see the summary from Plumbers 2013 and these slides [PDF] for further background). Andrzej Pietrasiewicz, who is the ConfigFS gadget maintainer from Samsung, outlined why runtime dynamic gadget functionality is useful: because it allows various functionality to be used in different combinations, unlike the statically configured gadgets that previously had to be defined at compile time. Pietrasiewicz then covered a bit of detail on how one configures the ConfigFS gadget to support various "functions" through the ConfigFS filesystem interface. One issue he brought up was that since ConfigFS modifications happen in multiple steps, there isn't one single moment when resources are allocated and bound in the kernel. Instead, allocations and binds are done at various different points as the gadget is configured and enabled. Pietrasiewicz sees this as somewhat problematic, and thinks that all the resources needed for a function should be generated when the directory is created in the ConfigFS mount.

At this point, Badhri Jagan Sridharan, a developer on the Android team who is working on migrating Android to use the ConfigFS gadget, agreed that resource allocation during the mkdir() operation would make more sense and require less time to switch between configurations. He had a few questions about Android's migration to ConfigFS, and asked if there were any guidelines for allocating instances versus functions and when to bind. Pietrasiewicz replied that it all depends on the function, and that there aren't any guidelines currently.

There were some further questions on details, but the session was running over time, so Sridharan and Pietrasiewicz carried on further conversation in the hallway track.

Barriers to running mainline on form-factor devices

The next session was a hurried presentation (slides [PDF]) by John Stultz from Linaro. The benefits of running mainline on off-the-shelf consumer devices (i.e. form-factor devices), while not particularly compelling to end users, are mostly for kernel and android developers. It provides a way to do continuous testing so that upstream issues can be caught early. It also brings more awareness of the unique aspects of mobile environments to upstream developers. The main barriers to doing this break down to hardware-related and software-related issues.

For hardware, the core requirements are an unlockable bootloader and access to a serial UART, which usually requires breaking the case open and soldering. Google has an interesting solution with its headphone debug UART cables on Nexus devices, but not all vendors implement it. A serial alternative mode for USB-C would be nice, but still requires standardization. Binary blobs, while not actually hardware, do effectively constrain the hardware that can be used with upstream kernels.

On the software side, the out-of-tree Android patchset has classically been a blocker but, since 3.4, enough of the Android tree has been merged upstream that limited testing can be done. Also, over the last few Android common kernels there's been steady decline in the amount of code out of tree. There are still a few substantial features out of tree but, for the most part, it's all fairly easily forward-ported so its not as much of a problem as before.

The biggest issue right now, as Tim Bird and others have been talking about, is probably lagging vendor system-on-chip (SoC) support upstream. For the Nexus device kernels that Google releases, we're looking at 1-2 million lines of out-of-tree code, and non-Google trees are apparently as bad as 3 million lines. Even so, a few vendors have been getting much better at upstreaming recently, so hopefully with this year's device releases we'll see if things have improved.

Given the above constraints, the 2013 Nexus 7 seems like a good device to target, and work is in progress. Currently the device is booting from MMC, has a working serial port, USB ConfigFS gadget support for ADB, and MTP support is there as well (with some quirks). GPIO button support for power and volume buttons as well as touch-panel input seem to be working as well. All of this is with around 32 patches, most of which are simple device-tree changes or Android build-integration changes.

Display support was hoped to have been done by now, but isn't; however, Bjorn Anderson and Rob Clark have that already working on the Sony Z3 (which is a related device). Apparently Anderson also has WiFi and the modem working on his device as well. Thanks are due to the folks who have been actually writing and upstreaming the device support required. So far, this has mostly been an integration effort, not a development one, which is due to everyone who did the hard work to make that possible.

There are still a number of areas that need work, such as display, battery charging, power management, WiFi, sensors, camera, and so on. And it's interesting that those issues are common sticking points for most SoCs, which the upstream community should take to heart. What we have upstream may be incomplete or too difficult to work with; we really need more folks working in these areas.

Benefits are already being seen from the effort, as it's helped crystallize which out-of-tree patches are most critical to get merged, and has provided a testing ground for the ConfigFS gadget transition.

Android, partitions, and customization

The next session (slides [PDF]) was by Rom Lemarchand from the Google Android team and covered some of the changes the team made for the Android One effort to ship common kernels and disk images for multiple devices. In the ideal world, there would be one user-space image per architecture, the bootloader would detect the device, populate the device-tree tables so that a common kernel with no out-of-tree patches would boot, and all would be well. But, in reality, every device has customized user space and kernels; vendor kernels have huge diffs from mainline and lots of functionality is being pushed out to proprietary user-space logic or even to trusted OS environments.

For the Android One effort, it was decided to have a single kernel and user space for every device "family", which is basically a set of devices using the same SoC with minor changes in hardware like sensors or cameras. For Android One, the /system partition should only have truly generic architecture-specific code. Since it previously contained all binaries and assets for specific devices, a common /system partition is achieved by keeping the device-specific data in separate partitions.

The /vendor partition that was introduced with the Nexus 9 provides a way for SoC-specific drivers to be included, like graphics drivers or core power-management libraries. Android One introduced a /odm partition that is meant to contain device-specific drivers like the sensor HAL, etc.

Finally, the /oem partition was introduced to store things like background graphics, ringtones, and other OEM-level customization for the device. This allows the different partitions to be updated independently (though the /oem partition is not verified and thus is not able to be updated in the current scheme), and allows each partner involved to do the customization wanted at the right level.

It was asked if the /odm partition could contain kernel modules. Lemarchand said that it could and the immediate follow-up question was who signs those modules. He clarified that while the original design manufacturer (ODM) signs the /odm partition and can do over-the-air (OTA) updates independently, any kernel modules included in that partition have to be signed by Google to be loaded since the kernel uses signed modules. Thus vendors can't create their own signed kernel modules.

Mark Gross from Intel also brought up a pain point that he has: in order to test with new kernel modules, he has to re-flash the system partition with those modules, which causes dm-verity to fail. He'd like to see a untrusted partition for kernel modules, since those are already signed. It was asked if being able to load custom modules remotely via ADB would help; Mark said that it would be interesting, but probably not helpful since the whole kernel and all the modules would need to be updated and tested together.

Running a single Android binary image on multiple devices

The next talk (slides [PDF]), which continued on the theme, was by Samuel Ortiz and covered the work done by Intel's Open Source Technology Centre on the Intel Reference Design for Android (IRDA) effort. He said that IRDA was similar to Android One, but for x86 tablets. The goal was to minimize changes to the Android Open Source Project (AOSP) code and to support fairly small hardware differences like changes to GPS, WiFi, and Bluetooth.

The first major issue the project ran into is that, with the current build system design, changing any single device results in changes in the user-space images. And there are many issues in trying to solve this. For instance, kernel modules can help, but Android's insmod lacks the module-dependency handling normally found in modprobe. There's no easy way to dynamically select different HALs, so for devices that have different types of GPSes, you have to have different HALs for each one. For subsystems like WiFi, there are different wpa_supplicant binaries needed to support different vendors. And some things require configuration values or permission files generated at build time. So the desire is to find some way for this to all be done dynamically at boot time, rather than at build time.

IRDA's approach is to use an autodetection daemon that uses ACPI tables to detect the hardware on the device, then, for that hardware, it triggers predefined actions. The project modified libhardware, so that it talks to the daemon to determine which HAL should be used for the device. For some hardware, like GPUs, the drivers need to be in specific locations, so its image includes multiple HAL drivers in different directories, then its daemon bind mounts the appropriate HAL directory into the right place in the file tree. It also includes a FUSE filesystem driver for /etc/permissions/ that generates the right XML permissions files for the device at runtime.

He ran through some examples of how this works for specific devices, and also outlined some areas where the most trouble was seen. Dynamic HAL selection is probably the biggest issue and something that would help would be to have better reference HALs in AOSP, as most vendors are shipping majorly hacked up implementations, usually based on ancient versions. The project has worked on improving Bluetooth support upstream so that the AOSP reference HAL can work more generically and there isn't the need for vendor-specific Bluetooth HALs. Solving kernel-module-dependency resolution would be nice to have as well.

Yaghmour, of Opersys, asked if this worked when devices disappeared and reappeared at runtime, or if it was mostly for boot time only. Samuel clarified that it could be used with dynamic changes at runtime, but various XML permissions files need to be removed. Yaghmour noted that parts of the framework are unhappy when devices disappear.

It was asked which of the reference HALs Ortiz saw as worst overall. To that, he replied "the binary ones" and wished that the reference HALs were more robust, allowing hardware vendors to not waste time with implementing bad custom HALs. He called out GPS, Bluetooth, NFC, and Graphics specifically as being bad and noted he couldn't point to any one AOSP reference HAL being widely used. Though Brown noted that the Audio HAL was fairly good and, while vendors do tend to copy it, it's reused for the most part, so that's progress.

Dmitry Shmidt from Google asked if the probing and autodetection impacted boot time. Ortiz replied that it varies but can be on the order of 500ms or so, which, for the Android boot time, is quite a small percentage.

Adapting Android for Ara

Karim Yaghmour was up next with his talk (slides [PDF]) about Project Ara and how it's adapting Android for the project's needs. After clarifying that he doesn't speak for anyone but himself, he began by covering the overall goals of Ara, which are to introduce more variety into the hardware ecosystem and to provide a platform for hardware developers that is something like what the mobile app platform is for software developers.

Yaghmour provided an overview of some of the most interesting technologies being used by Ara, starting with the MIPI UniPro interconnect that is used as the bus to connect the modules. He also covered the endoskeleton frame, the contactless capacitive connectors used by the modules, as well as the Electro-Permanent-Magnets (EPMs). The endoskeleton connects all the modules and basically acts as a UniPro switch, allowing for even direct module-to-module communication. The EPMs are currently used to hold the modules to the endoskeleton, but the news of the day was that the EPMs weren't working out for the project. Each module can either consume or provide charge, allowing for battery modules to be included with other module functionality, and allowing for multiple batteries to be used together. Finally, each module can have a custom printed cover, which allows for even further user-directed design.

Yaghmour noted that UniPro supports a number of standard protocols, but for Ara there was quite a lot that needed to be developed. So the project created Greybus, which provides the protocol that handles setting up the switch, module notifications, power-management, and standard class protocols for a number of different devices (camera, audio, Bluetooth, GPS, etc.). Since initially there weren't devices implementing these class protocols, it also provides "bridged PHY" native protocols like i2c, USB, UART, SDIO, and so on, which can be run over Greybus. Yaghmour also noted that, while hardware isn't out yet, there is the gbsim simulator for Greybus, which folks can check out now if they're interested.

There were some questions on how Greybus handles dynamically adding buses that cannot be probed like i2c, and to what extent the existing drivers needed to be changed. After some confusion about the question, Yaghmour said that the module provides a manifest of what it contains, and the related Greybus driver creates the platform device based on that knowledge of the module, so that the existing driver can be reused with as little change as possible. It was also noted that the plan is that the devices would use the class drivers, so the bridged PHYs are sort of a short-term solution until class-based hardware is available.

Yaghmour then talked about how Ara was extending Android to handle dynamic device changes at runtime. He clarified that, after some various proposals were run by the Android team, the Ara project settled on adding the dynamic device handling in the HAL layer, so that changes to the framework layer could be minimized. It will then work on each HAL layer to try to extend it so the framework can handle when devices are unplugged. For the most part, this means it supports the HAL interfaces for devices even if the device isn't present. So, for the camera when the camera isn't loaded, it provides a "no camera" screen to the interface. This allows iterative changes through each subsystem, which will likely have more success than trying to change all the subsystems at once.

There was a comment from Marcel Holtmann, the Bluetooth maintainer from Intel, that this seems problematic since, for some functionality like Bluetooth, the pairing is bound to the physical hardware address of the radio. Yaghmour agreed and noted that if you replace your Bluetooth module with a different module, you would have to re-pair your devices.

There was also a question of where the sensors live; Yaghmour answered that they are usually paired with other devices. An example is that the rotation sensor is included with the screen, which was useful because it allowed a known orientation in space (i.e. which way is up), whereas if it were on other modules, it could be inserted in different slots or with a different orientation, so the system would have to figure out the module orientation relative to the display to understand the sensor data.

Yaghmour then gave a brief recap of the previous sessions, noting that there are a number of different approaches being used to try to make Android more generic and to allow it to support more devices without custom builds. The first, by Lemarchand, handled supporting different devices at flashing time by using a combination of generic and hardware-specific partitioning. The second, by Ortiz, talked about supporting different devices at boot time from a single disk image, using bootloader/firmware data to do bind mounts and FUSE filesystems. And, finally, his talk covered handling different devices at runtime through extending the HAL layer to support hotplugging standard classes of devices.

Integrating kdbus in Android

The last talk (slides [PDF]) of the afternoon session was from Pierre Langlois from ARM about his summer project of trying to run Android's libbinder over kdbus. Langlois covered some potential benefits of the project, which included alleviating the concern of duplicating work and maintainership with two new interprocess communication (IPC) mechanisms being introduced into the kernel, the curiosity of whether it could work, and what each side might be able to learn from trying to run libbinder over kdbus. Being a summer project, Langlois did limit the scope of his work to just see if it was possible to make libbinder run over kdbus; security implications or performance were not a focus.

He provided a basic outline of how binder works, with a simple example of how one might add two numbers together using binder. He also covered how the remote services are named, registered, and found through a special process called the ServiceManager that all processes register with. Langlois then dived into details of the binder kernel driver, and how it maintains per-process memory pools, worker threads, reference counts, and dispatches data from one process to another. The kernel driver also has a special call to identify the ServiceManager process. All of the binder kernel interfaces are abstracted out by the libbinder library.

He then covered an overview of kdbus and what it provides, noting that kdbus keeps track of the service naming internally in a registry, which can be probed by processes to find services, and can also can provide notifications. His assessment was that it was sufficient to implement binder transactions, so Langlois implemented libkdbinder, which is a drop-in replacement for libbinder that is implemented using kdbus.

There are some conceptual differences that had to be resolved. Since kdbus provides a service-name registry, Langlois got rid of the ServiceManager component in Android and provided compatibility calls into the kdbus service-list functionality. Also, since binder normally handles thread pools to service requests, and kdbus does not manage worker threads, libkdbinder needed to spawn off and manage the threads itself. With these issues worked out, he was able to successfully parcel up binder transactions, send them with kdbus synchronous messages, and unparcel the returned results.

So far, he has a working proof of concept that passes the binder test cases found in AOSP. This progress actually surprised some of the Android developers, who quickly asked if it supported some of the more obscure binder features. Langlois noted that, while the really obscure weak-references feature, which doesn't actually seem to be used in binder, isn't implemented, most of the other functionality is there.

He then mentioned that he did try booting Android with libkdbus to see if it would work, but ran into one unexpected problem: kdbus seemed to fail when passing ashmem file descriptors over it. It seems that kdbus only allows for sealed memfd file descriptors to be sent, which is more restrictive than what Android uses for ashmem file descriptors shared over binder. Folks in the room were a little confused, as they thought it really ought to work, but there wasn't anyone who could speak authoritatively on why this issue was a problem. Langlois thought that probably kdbus needs to be extended to handle ashmem file descriptors, but had run out of time to debug the issue further. He suspects that after the ashmem issue is resolved, there will probably be other blockers that pop up, but in the end he thinks getting it working to feature parity is entirely doable.

For future work, Langlois sees reviewing the security implications as a high priority. There were also questions of allowing for more than one bus, since running binder names on a desktop kdbus might cause problems. Folks in the room seemed confident that it would be easily doable to implement a separate binder kdbus instance.

Andrews, who is one of the binder maintainers from Google, was a little skeptical that performance parity would be possible, as binder is well-optimized for its use case. One of his current tasks is reworking the binder kernel driver to break apart some of the locking that is causing lock contention on devices with a higher CPU count. One thing he did note is that in doing his rework, he's planning to greatly extend the binder test cases, so that he can be confident he doesn't break anything, and that work will likely provide better test cases for libkdbinder to be tested against.

The code isn't yet released, but Langlois and Serban Constantinescu from ARM were working to get the code released soon and possibly plan to hand the project off to Linaro. Stultz pressed that getting this implementation out and being able to provide analysis of the performance difference or any restrictions (like the ashmem issue) that kdbus has would be important. Since, if there any changes that might be needed in kdbus, it would be particularly useful to know about them soon, as it could affect the conversation with regard to kdbus getting merged.

And with that, the first session was over, and folks took a break. The second session will be summarized in an upcoming article.

[Thank you to all the presenters for their discussions, Karim Yaghmour for organizing and running the microconference, and Rom Lemarchand for helping get so many of the Google Android team to attend.]