The LPC Android microconference, part 2

OP-TEE and Trusty

Jens Wiklander from Linaro shared an overview on secure-world implementations, OP-TEE and Trusty in particular (slides [PDF]). Jens works on OP-TEE, which provides an open-source trusted execution environment (TEE) that has been a valuable reference implementation for research and education; most other TEE implementations are proprietary. He noted that Google also has its Trusty TEE environment that is also open source, and that he would try to provide some comparison between the two where he could in his presentation.

He provided an outline of how TEEs and the secure-world applications interact with the kernel and user space, applicable to both OP-TEE and Trusty. OP-TEE's implementation, in particular, tries to implement a generic TEE API that user-space applications and the TEE supplicant can interact with; it forwards commands to the secure world via secure monitor calls (SMCs). Jens noted that Trusty's approach is similar, but it uses a virtio driver to communicate with the secure world.

One interesting aspect of the secure world is that there isn't necessarily any persistent storage that it can use, so OP-TEE implements secure storage by having trusted applications in the secure world encrypt and sign data that they want to persist. It passes that data out through the kernel to the TEE supplicant, which stores the encrypted data in a database in the filesystem.

Another aspect of the TEEs that he covered is how they do scheduling. Entry into the secure world is normally done via an SMC or via an FIQ interrupt but, for OP-TEE, the work is done only via SMC calls, so all the scheduling done in the secure world is done as part of a command or request from the kernel in the normal world. Trusty, instead, includes an integrated scheduler which is triggered by the FIQ. Communication between the TEE and normal world is done via shared memory, which is implemented via a reserved contiguous memory region. That region must then be set up with the same cache settings in both secure and non-secure world.

OP-TEE currently supports a set of devices, including a number of arm and arm64 devices. He also covered the xtest test suite that OP-TEE uses for validation, which can be extended to support the GlobalPlatform TEE compliance test suite.

Rom Lemarchand asked how much of the GlobalPlatform API is supported with OP-TEE. The answer was that both the internal and client APIs are supported. There was also a question as to which crypto IP blocks are supported; currently none are. When asked if secure memory is supported on any of the supported devices, Jens's answer was all except HiKey. Rom was asked if he could further contrast Trusty and OP-TEE to call out anything that wasn't covered. He clarified that Trusty doesn't support GlobalPlatform at all, and that Trusty needs crypto block and secure memory functionality to work. Jens noted that, while OP-TEE is scheduled by the Linux side, Trusty can take some cores away for exclusively secure execution. At that point the discussion died down and the session moved on.

Multiple devices in AOSP

Rob Herring from Linaro discussed efforts to improve support for multiple devices in the Android open-source project (AOSP) (slides [PDF]). His larger goal is to try to consolidate the hardware abstraction layers (HALs) that Android uses, and allow vendors to develop kernel support for a device once and have it work on Android, ChromeOS, as well as traditional Linux distributions. Vendors could then avoid the effort of developing custom HALs when they don't bring new value. This would make upgrading to newer Android releases easier, allow devices with mainline kernel support to "just work", and would allow for an upstream community for Android devices to grow.

Currently if one wants to bring up Android on a new device, the seemingly "standard" approach is to copy over the device directory for a system that seems similar and rename all the files and variables with the new product's name. This trivial renaming of code has the unfortunate side effect of making diffs between implementations impossible. Then one might scan through other devices to find any needed missing functionality and copy-and-paste those in before building and testing. This is pretty poor. Determining what's actually different between two builds for different devices is quite complicated; Rob recalled a time where he tweaked the build system to generate multiple megabytes of configuration logs just so he could diff two builds. Another problem facing the Android ecosystem is how to quickly upgrade devices to new Android releases, which now come out monthly. This problem compounds if you're trying to support more than one device.

The solution he's working on is trying to allow multiple devices to be supported from a single build target, ideally moving to supporting devices of the same architecture with a single filesystem image. He has implemented this using Kconfig, which is used to generate BoardConfig.mk and device.mk files. This provides a nice UI to allow configurations to be more discoverable, and allows higher-level features that otherwise require multiple configuration settings to be wrapped under a single element.

His build target currently supports DB410c, HiKey, Nexus 7, RaspberryPi 3, and QEMU (for x86_64, arm, and arm64). He gave a brief demo of the Kconfig options and outlined what he sees as the next steps: trying to push this to AOSP, adding any missing Kconfig options as device supports grows, supporting custom compilers or compiler flags, integrating a kernel build, as well as supporting other features like malloc() or filesystem selections. He then provided some input for folks who want to contribute, specifying that there should not be device-specific configurations but, instead, more fine-grained configuration options as needed, along with using device-specific default configuration ("defconfig") files to provide an easy way for folks to get a build for a specific device.

He also outlined some drawbacks with Kconfig. It doesn't integrate nicely with the top-level Android build, so some stale files can be left in the build tree when a configuration changes. The defconfig files also have to be kept current or they can grow stale as new configurations are added. The Google developers in the crowd thought this effort was interesting, but they suggested that Rob talk with the developer who is reworking the Android build system to get his feedback.

HiKey in AOSP

John Stultz provided an update on support for HiKey boards in AOSP (slides [PDF]). HiKey is an arm64 board that was included as a target in AOSP back in March. Since then, work has continued and many features added, including moving to the 4.4 kernel and updating to Android 7.0 (Nougat). Recent work has been done on integrating the energy-aware scheduler, which is a collaboration between folks at Google, ARM, and Linaro. Integration of OP-TEE and Trusty so that they can coexist as build-time options has been worked on. Since most Android devices don't have to deal with dynamic peripherals on buses that cannot be probed, work has been done on creating a device-tree overlay manager so that different device-tree fragments can be selectively applied, depending on boot arguments. Since downloading and building Android can take multiple hours (or days), pre-built factory images have been created so that developers can more easily get started using HiKey boards with the latest Android release.

There has been a lot of work related to HiKey in the generic Android common.git kernel tree: adding new features, cleaning up unused ones, and prepping for an android-4.9 branch. John pointed out Amit Pundir's talk [YouTube] on the status of the Android common tree, as Amit couldn't present it in person. He also mentioned that HiKey was a target for Rob Herring's generic-build effort discussed earlier.

Since HiKey will be supported for another year and a half in AOSP, there is still a fair amount of planned work on things like moving forward to the android-4.9 kernel (and beyond), finishing the OP-TEE/Trusty integration, enabling A/B style updates, and enabling memory reductions. John considers this work useful, because devices released over the last few years often use a kernel version that is one or sometimes two years old, and stick with them.

HiKey's immediate migration to the latest LTS-based Android kernel allows for testing and validation to help make those kernels more stable and reliable. That should result in less work to do when vendors do migrate to newer kernels, hopefully allowing it to be done more quickly. Further, since HiKey development follows upstream kernels as well, testing and validating mainline has caught a number of regressions, allowing them to be reverted or fixed quickly, keeping those issues out of future LTS releases.

Many of the kernel changes needed for HiKey have been upstreamed recently, with a handful of patches related to USB, HDMI-audio, and HDMI-output reliability issues being pushed upstream but not yet merged. Of course, there is still the large issue of Mali GPU support being based on proprietary user-space code, so upstreaming that kernel driver (which makes up by far the majority of the out-of-tree HiKey code) isn't an option, but the dream of an open-source Mali driver persists.

John emphasized why this HiKey effort is so important: it's because we really have two separate communities. One of those is the AOSP and hardware vendor community, which focuses on shipping a fully enabled, single device quickly, where the specific kernel version used doesn't really matter that much. The other is the upstream kernel community, where the priorities are on long-term maintenance and common solutions that allow multiple devices to work with a single binary, utilizing the mainline or linux-next. These different focuses both make sense, but they also make communication between the two communities difficult. So HiKey enables a small but useful area of overlap by providing an affordable device that works against the latest upstream kernels and is also supported by the latest AOSP user space. Thus, the device becomes a shared concern between the two communities, and hopefully will bring them closer together, allowing for more prototyping and validation between the communities.

Systems-level programming in Java

Elliott Hughes from Google shared recent developments to enable systems-level programming using the Java language in AOSP (slides [PDF]). He talked a bit about his own history with the Android project, working on native (C/C++ based) tools and libraries, as well as ART, but specifically his early efforts focused on cleaning up much of the native code in Android.

The libcore code was initially written in C and suffered lots of leaks and issues. His group switched to C++ and used tools to help clean up a lot of leaks and crashes, but still left a fair amount of native code, which can be problematic due to the semantic differences between Java and Unix. So instead the developers decided to expose POSIX to Java, leaving the native code to just marshal the data back and forth from the POSIX API calls. This allowed for easier debugging in Java, but also allowed for more flexibility.

With a few examples he showed how simple the code was, along with how some of the complexity POSIX imposes with error conditions returned via the global errno variable is handled. This code has started seeing use outside of the libcore code, as it allowed folks to avoid having to write their own Java Native Interface logic (which is difficult to do correctly), and soon normal apps in the Play Store wanted to use these interfaces too. So with Android Lollipop (5.0), Google introduced android.system, which makes POSIX generally available.

Some design choices were needed in its implementation; for example in the translation of C types (such as choosing when to make a char * a String and when to make it an Object) and creating Java classes to match C structures. Another example was how to handle errno, using exceptions which return the errno value as well as the string name of the function, which provides nice output for debugging like "open failed: ENOENT (No such file or directory)". The group also provided OsConstants, which allows the Java code to look much like C.

He covered a number of other miscellaneous design choices made. For example, functions always return new structures rather than filling in existing ones, as it's safer and not that expensive. All of this functionality is used in areas like networking code for DHCP, or dup() and lseek() usage in media code, extended-attribute usage, etc. Elliott concluded by saying that, while this isn't a particularly new feature, he mostly wanted to use this talk to advertise it, as he thinks it will be useful.

Using Clang in AOSP

The last speaker of the day was Bernhard (Bero) Rosenkränzer from Linaro, speaking about the use of the LLVM Clang compiler in AOSP (slides [PDF]). Since the release of Nougat, Clang is now used instead of GCC to build Android's user space. This effort required mostly small changes like making the code stick closer to C standards, but there were some more difficult areas like variable-length arrays in structures and differing semantics in the definition of extern. Several real bugs were found and fixed thanks to Clang's extra warnings.

The performance of the system is unchanged, but build times are nicely reduced. GCC is now only used for the kernel and some older device HAL implementations. However, work to get the kernel building with Clang is also happening. Bero and others at Linaro have been working on getting the HiKey kernel building with Clang. That work is a little outdated and will be updated soon to 4.9. While some of the patches needed are on their way upstream, there are still quite a few bad hacks involved.

Bero went over a number of the issues they encountered and workarounds they needed in this project and pointed to a Docker image he developed to make it easy for folks to get the proper toolchain environment and kernel source to try it themselves. Currently, the system runs well, but there are a few odd bugs like crashes in Firefox. While Bero wasn't sure if there was much interest from Google in moving to Clang for the kernel, the Google developers in attendance did say there was interest, but they had been working on this in earnest for just a few weeks, and suggested they sync up and collaborate to avoid duplicate work.

Now that the Clang patches needed to build AOSP user space have been merged, Linaro is working to stay on top of the latest Clang. He outlined the testing efforts done to build AOSP master with the latest Clang snapshots using continuous integration. He pointed out some of the new warnings and errors that Clang 4.0 is producing and noted that, while they aren't seeing too many compiler bugs in testing development branches of Clang, there are the occasional issues that have to be reported. As for the future of GCC in AOSP: AOSP is using GCC 4.9 and Bero suspects that is not likely to change. However, Linaro continues to do testing building AOSP master with GCC 6.x releases; since different toolchains report different warnings, building with both can help uncover issues that one toolchain might miss. However, he worries that the Android build system might someday depend too strongly on Clang plugins, preventing this effort from being useful.

Conclusion

With that, after the microconference concluded, somehow on schedule despite the huge number of sessions. Many thanks to all the speakers and organizers for putting a great session together. Its really great to see all the participation between community members and Android developers. Its quite likely there will be another Android microconference at next year's Plumbers, which we look forward to.