LWN.net Weekly Edition for July 21, 2022

Welcome to the LWN.net Weekly Edition for July 21, 2022

This edition contains the following feature content:

• Leaving python-dev behind: the Python project's development list appears to be approaching the end of its long life.
• Long-lived kernel pointers in BPF: a new mechanism to allow pointers to be stored between BPF program invocations.
• Sharing page tables with msharefs: a revised patch series trying to address the problem of page-table overhead with massively shared memory areas.
• The BPF panic function: is a BPF program "safe" if it is allowed to crash the system?
• Android apps on Linux with Waydroid: a tool that can run Android apps on non-Android systems.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Leaving python-dev behind

It was not all that long ago that Python began its experiment with replacing one of its mailing lists with a forum on its Discourse discussion site. Over time, the Discourse instance has become more and more popular within the Python community. It would seem that another mailing list will soon be subsumed within Discourse as the Python steering council is planning to effectively retire the venerable python-dev mailing list soon.

History

Back in 2018, both Fedora and Python were experimenting with Discourse instances; both are quite active at this point. Discourse is an open-source web forum project that says it aims to "reimagine what a modern Internet discussion forum should be today, in a world of ubiquitous smartphones, tablets, Facebook, and Twitter". But Fedora and Python currently still have mailing lists as well.

As part of the experiment for Python, the core-developer-only python-committers mailing list was switched to Discourse for a few months as a test. That was during the upheaval in the Python world that stemmed from Guido van Rossum's resignation as its benevolent dictator for life. The announcement of the experiment was deemed a bit of an overreach at the time, but the discussion of the new governance model for the language did largely happen in the Committers forum on Discourse.

These days, the python-committers list still exists, but it mostly receives announcements and the like; some discussions still take place there, but most, it would seem, are done on the Discourse site. There are plenty of other forums on that site, including two that overlap the main mailing lists where development discussions take place. The Core Development forum overlaps the function of the python-dev mailing list, while the Ideas forum serves the same purpose as the python-ideas mailing list.

Up until recently, there was no real indication that changes might be in the works, but an April query about PEP discussion that was posted to python-dev by Victor Stinner may have been the first real public indication that changes were afoot. He asked that new PEPs be announced on python-dev since he did not go to Discourse often and several PEPs of interest had slipped by without notice because they failed to be posted there. He also noted that it is "sometimes hard to keep track of everything happening around Python development" in part because of all of the different ways the project's developers communicate:

The discussions are scattered between multiple communication channels:

• Issues
• Pull requests
• python-dev
• python-committers
• (private) Discord
• Discourse
• (public) IRC #python-dev

Sometimes, I [am] already confused by the same topic being discussed in two different Discord rooms :-) It's also common that some people discuss on the issue, and other people have a parallel discussion (about the same topic) on the related pull request.

He noted that there are some in-person events, too, that make it even harder to keep up for those who cannot attend. Petr Viktorin replied that PEPs should be posted to python-dev, "but not necessarily right after they're published". They may be discussed elsewhere before coming to python-dev, for example. The intent, Viktorin said, is that PEPs should only be submitted to the steering council, of which he is a member, "after all relevant discussion took place", which includes python-dev.

Choosing one

Jean Abou Samra noted that the split in the location for discussions is confusing to him, so he asked about any plans "to retire either Discourse or the mailing list and use a unified communication channel". Gregory P. Smith, who is also a steering council member, replied:

We feel it too. We've been finding Discourse more useful from a community moderation and thread management point of view as well as offering markdown text and code rendering. Ideal for PEP discussions. Many of us expect python-dev to wind up obsoleted by Discourse as a result.

That led to some predictable grumbling about Discourse and the problems with following a web-based forum in comparison to a mailing list. That divide comes up whenever changes of this sort are announced or discussed, but newer developers generally seem to be uninterested in learning the "joys" of participating on mailing lists. In part, that is because the relevance of email as a communication mechanism has fallen almost completely off the radar for many. But the writing seems to be on the wall—for Python at least. As Christopher Barker put it:

But if Discourse has been adopted, I guess it's time for us curmudgeons to bite the bullet and start monitoring it -- and frankly, this list (and python-ideas) should probably be retired, or turned into an announcement-only list -- having the current split is the worst option of all.

For discussions on the development of CPython, the python-dev mailing list has been the place to go for two decades or more. The mailing list page has archives going back to April 1999. In fact, one of the earliest messages archived is from Van Rossum asking whether the python-dev archives should be public or private. Obviously, "public" was the decision.

On July 15, Viktorin posted a message that would seem to be bringing that history to a close. On behalf of the council, he said:

The discuss.python.org experiment has been going on for quite a while, and while the platform is not without its issues, we consider it a success. The Core Development category is busier than python-dev. According to staff, discuss.python.org is much easier to moderate.. If you're following python-dev but not discuss.python.org, you're missing out.

The Steering Council would like to switch from python-dev to discuss.python.org.

His message recognized that not everyone finds the Discourse forum software running at discuss.python.org to be easy to follow and use either; it contained several suggestions for alternate ways to interact with it, including "mailing-list mode". The message was also soliciting feedback on whether a permanent switch would "pose an undue burden to anyone". A final decision on the switch had not been made, so the council wants to ensure that it is "aware of all the impact".

No one has really raised any concrete problems of that nature, though Barry Warsaw mentioned the possibility of "accessibility or native language concerns" for Discourse. He also noted that he supports moving to Discourse, which "might seem odd coming from me"; Warsaw is one of the lead developers of the GNU Mailman mailing-list manager system and was a big part of the Mailman 3 effort. "Discourse is not without its issues, but then again, the same can be said about email."

While there were posts with the usual negative opinions of web forums versus email, they were fairly muted. To an extent, it would seem that there is a generational change going on in the Python community; the older developers are either adapting, perhaps via mailing-list mode, or kind of just bowing out. For those looking for more information, mailing-list mode is briefly mentioned in the "Following Python's Development" section of the Python Developer's Guide. But mailing-list mode is not able to disguise one problem that Discourse discussions have: no threading. Ethan Furman said:

I follow each (sub)thread through to it's end, as it keeps a logical flow, but Discourse has everything linear which means that as I read it the conversation keeps jumping around, making it hard to follow.

Warsaw agreed that the lack of threading is problematic, but that feature has fallen by the wayside in today's discussions:

[...] I definitely prefer threaded discussions. Unfortunately though, much like top posting <wink>, I think that horse is out of the barn, what with other forums like GitHub being linear.

As might be guessed, based on the "wink", Warsaw had top-posted his reply. Viktorin noted that he has just had to accept the linear nature of Discourse discussions; things have changed and there is likely no going back:

[...] if python-dev was used by everyone, rather than almost exclusively by people who prefer e-mail (and presumably use threading mail clients), we'd get mangled threading anyway from all the non-threaded clients.

I mean, I could grumble about threading and bottom-posting and plain-text messages and IRC all day, but realistically, I'm not likely to convince anyone who's not into those things already.

That's where things stand at this point. It seems likely that a final decision to switch away from python-dev will be coming soon and that the venerable mailing list will be reconfigured sometime thereafter, "eventually switching to auto-reject incoming messages with a pointer to discuss.python.org". That will be a sad day for some—and effectively a non-event for (many?) others. For those of us who cut our teeth on threaded, text-only, bottom-posted discussions, it is completely mind-boggling that Kids These Days (tm) do not see the advantages of such ... discourse—but that seems to be the way of things.

Comments (110 posted)

Long-lived kernel pointers in BPF

The BPF subsystem allows programmers to write programs that can run safely in kernel space. All memory accesses and function calls in BPF programs are statically checked for safety using the in-kernel verifier, which analyzes programs in their entirety before allowing them to be loaded. While this allows the kernel to safely run BPF programs, it heavily restricts what those programs are able to do. Among these constraints is a rule that programs cannot store pointers into BPF maps for use (such as dereferencing them or passing them to the kernel in kfunc and BPF helper invocations) at a later time. A patch set by Kumar Kartikeya Dwivedi adds this capability to BPF.

Interacting with kernel pointers in BPF programs

Some operations are always safe to execute from within BPF. For example, the bpf_probe_read_user() and bpf_probe_read_kernel() helpers allow BPF programs to safely read user and kernel memory respectively by registering a page-fault handler to catch any faulting accesses. When BPF programs receive pointers from event handlers, for example, they can safely dereference them without having to worry about whether they will cause a crash, though they do have to ensure that the read succeeded by checking the returned error code.

BPF programs can also receive pointers from BPF helpers and kfuncs. BPF helpers are functions defined in the kernel that provide the core APIs that can be invoked by any BPF program. Kfuncs are also functions in the kernel that can be invoked by BPF programs but, unlike BPF helpers, their APIs do not need to be applicable to all types of BPF programs. To make a function available to BPF programs as a kfunc, it must be aggregated into one or more BPF Type Format (BTF) kfunc-sets, which are then registered with the BPF subsystem.

Kfunc-sets can also specify properties about their kfuncs that inform the verifier about how they need to be invoked in order to ensure safe execution in BPF programs. One such property specifies that the kfuncs in a kfunc-set will return an "acquired" pointer that must be passed to another kfunc that is part of a kfunc-set that can release it. Pointers that are subject to this constraint are called "referenced pointers" within the BPF community.

When loading BPF programs, the verifier will enforce this contract, and reject any program that fails to release a referenced pointer before returning, or which passes a pointer that was not previously returned as a referenced pointer to a BPF helper or kfunc. Note that the implementation of the "acquire" and "release" semantics of a kfunc is completely opaque to BPF, and is entirely up to the developer implementing the kfunc. The only thing that BPF requires is a guarantee that an acquired pointer will remain valid until it is released.

Extending BPF usability with referenced pointers

The ability to ensure that a kernel pointer is valid affords several advantages to BPF programs. The first and perhaps most straightforward is that BPF programs no longer need to use probed reads to dereference the pointers. Probed reads use the exception table mechanism used to safely read user memory from kernel space, and while they have nearly the same performance as a normal load instruction for successful reads, they impose a tax on the programmer by requiring them to always check if a read was successful. Avoiding probed reads allows a simpler programming model which can significantly cut down on the raw amount of code needed in BPF programs to satisfy the verifier.

In addition to providing simplification, referenced pointers also improve the extensibility of BPF by allowing BPF programs to safely pass those pointers back to the kernel in subsequent kfunc and BPF helper function invocations. While the kernel could use a mechanism such as copy_from_user() to read pointers received from BPF programs, it is less complex and less error prone to, instead, provide a guarantee to the kernel that pointers received from BPF programs are safe to read. This guarantee also makes it possible to export many internal kernel functions to BPF programs without modifying them.

While referenced pointers are a powerful tool for extending the kernel using BPF, a significant limitation of the feature is the requirement that all of the interactions between BPF programs and kfuncs take place in a synchronous context. Every time it needs to get a referenced pointer from the kernel, a BPF program must invoke a kfunc, and then release the pointer in another kfunc invocation before returning. This may have performance implications, as having to call two kfuncs is quite a lot of overhead relative to performing a single memory read. This workflow is also somewhat orthogonal to the traditional mechanics of reference counting, wherein pointers are stored in a data structure with the intention of being safely accessed at a later time. What would instead be useful is to store kernel pointers in a map, allowing them to be accessed whenever the program requires, possibly over multiple separate calls.

kptrs – storing kernel pointers in BPF maps

Dwivedi's patch set adds this capability via a new feature called "kptrs". A kptr is a strongly typed pointer that is received from a kfunc or BPF helper function and which may be stored into and retrieved from BPF maps throughout the run time of a program. Kptrs may be either ordinary ("unreferenced") or referenced pointers. Unreferenced kptrs have no guarantee of validity and are highly restricted in how they can be used; like normal pointers in BPF programs, they can only be accessed using a probed read. They also cannot be passed to the kernel via a kfunc or BPF helper function, as the pointers may reference invalid memory. Referenced kptrs, on the other hand, may be safely dereferenced by BPF programs, and passed to the kernel via kfunc or BPF helper function invocations.

From the BPF subsystem's point of view, a referenced kptr always has exactly one reference associated with it. In order to transfer a referenced kptr between different contexts, a new bpf_kptr_xchg() helper function was added that atomically swaps ownership of the reference between a map value and a local pointer. If the reference is transferred from a map value to a local pointer, the semantics enforced by the verifier are the same as for references returned by kfuncs in a synchronous context: the verifier will ensure that the reference is either transferred back to a map value or released via a call to a kfunc before the current execution context returns. On the other hand, if the kptr reference is stored in a map, the current execution context can safely return without releasing it.

If the kptr is never transferred back out of the map with bpf_kptr_xchg() and manually released, it will be automatically released when the program is unloaded and the map is destroyed. In order to enable this automatic releasing mechanism, Dwivedi extended the kfunc subsystem to allow developers to specify a kfunc destructor function that should be used for a given type.

Some of the use cases for kptrs were discussed in an earlier revision of the patch set, with Dwivedi describing the most common one as being for performance:

The common use case is caching references to objects inside BPF maps, to avoid costly lookups, and being able to raise it once for the duration of program invocation when passing it to multiple helpers (to avoid further re-lookups).

Storing a referenced kptr in a map obviates the need to invoke a kfunc every time the pointer is required, which provides performance benefits and reduces complexity. BPF programs need only make an initial kfunc invocation to first get the pointer and, after storing it in a map, can simply load it from that map and dereference it directly when it's needed.

Referenced kptrs also provide strong safety and correctness guarantees to developers. It is a ubiquitous paradigm in managed-object frameworks that a reference should only be owned by a single context, and the semantics of kptr reference handling bear a striking resemblance to std::boxed::Box in Rust, and std::unique_ptr in C++ in that regard. In Rust, a Box is a pointer to a heap allocation that can only have a single reference at any given time, and which is automatically freed when that reference goes out of scope. When the Rust program is compiled, the Rust compiler will verify that only a single reference can ever exist to the Box by applying Rust's ownership model. If the Rust program compiles, you have a guarantee that the memory pointed to by a Box is always valid and only has a single reference. In C++, some verification takes place at compile time by, for example, prohibiting an std::unique_ptr from being copied, but problems can still arise at run time. For example, a user could invoke std::unique_ptr::release() twice, and would receive a nullptr on the second invocation.

Kptrs seem to draw inspiration from both languages. On the one hand, the BPF verifier provides the compile-time guarantees that are afforded by the Rust compiler by analyzing BPF programs to ensure that there is only ever a single owner of a reference, and that the reference can never be leaked. On the other hand, bpf_kptr_xchg() closely matches the semantics of std::unique_ptr::swap(), so the mechanics of using the feature will feel more like C++. Managed-object leaks and use-after-free bugs are a common and pervasive source of pain when correct accounting is the responsibility of individual developers. Providing the guarantees of Rust's ownership model, and the semantics of C++'s std::unique_ptr APIs, to C and kernel development using BPF and the verifier therefore seems powerful indeed.

It will be interesting to see if the advantages afforded by these features will motivate more development to take place in BPF as opposed to the kernel.

Future kptr types

While we have been referring to both unreferenced kptrs and referenced kptrs as just "kptrs", they are actually represented as two different types in BTF. If BPF programs wish to use a kernel pointer as an unreferenced or a referenced pointer, they must annotate it with the BTF type tag "kptr" or "kptr_ref" respectively. It makes sense to enforce separate types for each kptr variant, as the verifier needs to use BTF to know a kptr's type, and then enforce its safe use accordingly.

While the current implementation of kptrs only enables the unreferenced and referenced variants, a natural question is whether the implementation could be expanded to include other types of pointers as well. In the first version of the patch set, Dwivedi proposed adding variants for per-CPU and user-space kptrs. Alexei Starovoitov responded by asking what the use case was for storing per-CPU pointers in maps, but Dwivedi did not have a concrete use case in mind. It was decided to drop the feature until a more concrete use case appears, so for now we will have to wait and see if that happens

This patch set is currently in linux-next, and so will presumably be merged during the next development cycle. In the meantime, it will be interesting to see how kfuncs and kptrs will be used to extend the kernel in ways that currently are not possible with BPF.

Comments (14 posted)

Sharing page tables with msharefs

A page-table entry (PTE) is relatively small, requiring just eight bytes to refer to a 4096-byte page on most systems. It thus does not seem like a worrisome level of overhead, and little effort has been made over the kernel's history to reduce page-table memory consumption. Those eight bytes can hurt, though, if they are replicated across a sufficiently large set of processes. The msharefs patch set from Khalid Aziz is a revised attempt to address that problem, but it is proving to be a hard sell in the memory-management community.

One of the defining characteristics of a process on Linux (or most other operating systems) is a distinct address space. As a result, the page tables that manage the state of that address space are private to each process (though threads within a process will share page tables). So if two processes have mappings to the same page in physical memory, each will have an independent page-table entry for that page. The overhead for PTEs, thus, increases linearly with the number of processes mapping each page.

Even so, this cost is not normally problematic, but there is always somebody out there doing outlandish things. As described in the cover letter from the patch series:

On a database server with 300GB SGA [Oracle system global area], a system crash was seen with out-of-memory condition when 1500+ clients tried to share this SGA even though the system had 512GB of memory. On this server, in the worst case scenario of all 1500 processes mapping every page from SGA would have required 878GB+ for just the PTEs. If these PTEs could be shared, the amount of memory saved is very significant.

Sharing those PTEs is the objective of this work, which was discussed at the Linux Storage, Filesystem, Memory-Management, and BPF Summit in May. At that time, Aziz was proposing a new system call (mshare()) to manage this sharing. The current patch set has changed this interface and now requires no new system calls at all.

Even without the system call, it is still necessary for processes to explicitly request the sharing of page tables for a range of memory. The current patch set provides yet another kernel virtual filesystem — msharefs — for that purpose; it is expected to be mounted on /sys/fs/mshare. The file mshare_info in that filesystem will, when read, provide the minimum alignment required for a memory region to be able to share page tables.

The next step is to create a file under /sys/fs/mshare with a name that means something to the application. Then, an mmap() call should be used to map that file into the process's address space. The size passed to mmap() will determine the size of the resulting shared region of memory. Your editor's reading of the code suggests that providing an explicit address for the mapping is advisable; there does not appear to be any mechanism to automatically pick an address that meets the alignment requirements. Once the region has been mapped, it can be used just like any other memory range.

The purpose of creating such a region is to allow other processes to map it as well. Any other processes will need to start by opening the msharefs file created by the first process, then reading a structure of this type from it:

    struct mshare_info {
	unsigned long start;
	unsigned long size;
    };

The start and size fields provide the address at which the region is mapped and its size, respectively; the new process should pass those values (and the opened msharefs file) to its own mmap() call to map the shared region. After that, the region will be mapped just like any other shared-memory area — with a couple of important exceptions, as will be described below.

A process's address space is described by struct mm_struct; there is one such structure for each process (other than kernel threads) in the system. The msharefs patch set changes the longstanding one-to-one relationship between this structure and its owning process by creating a new mm_struct structure for each shared region. The page tables describing this region belong to this separate structure, rather than to any process's mm_struct. Whenever a process maps this region, the associated vm_area_struct (VMA) will contain a pointer to this special mm_struct. The end result is that all processes mapping this area will share not just the memory, but also the page tables that go along with it.

That saves the memory that would have gone into duplicate page tables, of course, but it also has a couple of other, possibly surprising, results. For example, changing the protection of memory within that region with mprotect() will affect all processes sharing the area; with ordinary shared memory, only the calling process will see protection changes. Similarly, the memory region can be remapped entirely with mremap() and all users will see the change.

It appears that use of mremap() is actually part of the expected pattern for PTE-shared memory regions. The mmap() call that is required to create the region will populate that region with anonymous memory; there is no way to request that file-backed memory be used instead. But it is possible to use mremap() to dump that initial mapping and substitute file-backed memory afterward. So applications wanting to use shared page tables with file-backed memory will have to perform this extra step to set things up correctly.

The developers at the LSFMM session were clear that they found this whole concept to be somewhat frightening. So far, the reaction to this patch series has (from a memory-management point of view) been relatively subdued, with the exception of David Hildenbrand, who is pushing for a different sort of solution. He would rather see a mechanism that would automatically share page tables when mappings are shared, without requiring application-level changes. That would make the benefits of sharing more widely available while exposing fewer internal memory-management details.

Automatic sharing would need to have different semantics, though; otherwise applications will be surprised when an mprotect() or mremap() call in another process changes their mappings. Though it was not stated in this version of Aziz's patch posting, the sense from the LSFMM session was that the altered semantics were desirable. If that is the case, fully automatic sharing will not be possible, since applications would have to opt in to that behavior.

Either way, it looks like this particular patch set needs more work and discussion before it can find its way into the mainline. Until then, applications depending on sharing memory between large numbers of processes will continue to pay a high page-table cost.

Comments (20 posted)

The BPF panic function

One of the key selling points of the BPF subsystem is that loading a BPF program is safe: the BPF verifier ensures that the program cannot hurt the kernel before allowing the load to occur. That guarantee is perhaps losing some of its force as more capabilities are made available to BPF programs but, even so, it may be a bit surprising to see this proposal from Artem Savkov adding a BPF helper that is explicitly designed to crash the system. If this patch set is merged in something resembling its current form, it will be the harbinger of a new era where BPF programs are, in some situations at least, allowed to be overtly destructive.

As Savkov notes, one of the major use cases for BPF is kernel debugging, a task which is also often helped by the existence of a well-timed crash dump. By making the kernel's panic() function available to BPF programs, Savkov is trying to combine the two by allowing a BPF program to cause a crash — and create a crash dump — when it detects the conditions that indicate a problem that a developer is looking for. Savkov is seemingly not the only one wanting this capability; Jiri Olsa noted that he has gotten a request for this feature as well.

Making panic() available to BPF has some obvious hazards, so one would expect that there would be some guard rails put into place. In this case, the first step is a new flag, BPF_F_DESTRUCTIVE, that must be provided when a program that will invoke destructive operations (such as a panic() call) is loaded. If this flag is not present, the BPF verifier will reject the loading of a program that contains calls to any destructive helper functions, of which panic() is the only one (so far).

Even then, the panic() helper function is only available to tracing programs. It makes little sense, after all, for an infrared decoder to be able to panic the system, though this restriction will prevent a complete implementation in BPF for remote controls featuring a "panic" button. Then, there is a new sysctl knob (kernel.destructive_bpf_enabled) that must be set to a non-zero value; otherwise the panic() call will not be allowed. Even when the sysctl knob has been set, the process on whose behalf the BPF program is running must have the CAP_SYS_BOOT capability.

All told, it seems unlikely that a BPF program will panic the system by mistake.

There does not appear to be much in the way of opposition to this patch, though there were some questions about the details. For example, Song Liu disliked the sysctl knob "as it is global, and the user can easily forget to turn it back off". Alexei Starovoitov also said that a sysctl is not called for in this situation; the CAP_SYS_BOOT check should be enough, he said. Starovoitov also questioned the need for a full panic of the system, given that there are more direct ways to create a crash dump. Savkov replied that panic() is "more versatile" and that the system's response to a panic is configurable by the administrator. He did agree to remove the sysctl knob.

Starovoitov also suggested implementing the functionality as a kfunc rather than as a BPF helper. The reasoning here is that kfuncs are deemed to be unstable and can be removed at any time if they turn out to be a bad idea, while removal of BPF helpers is harder. It is worth noting, of course, that this position on the removability of kfuncs has, thus far, not faced the test of an irate user whose application depends on a kfunc that has just been removed.

In a later response, Starovoitov questioned the "versatility" of panic(), and said that lower-level functions should be provided to BPF programs instead. Thus, there should be one to create a crash dump, one to send a message to the console, one to halt the system, one to reboot, and so on. That way, he said, the program itself can decide what should happen rather than depending on a specific configuration of the system.

There is clearly another revision of this patch set coming in the future, and it may look significantly different than it did the first time around. But it also seems clear that there is a use case for this sort of "destructive" functionality in BPF programs. The BPF system is quickly growing beyond packet handling and information gathering and heading toward a point where arbitrary types of kernel functionality are available to BPF programs. It's not clear where all of this leads, but it seems likely to be interesting.

Comments (1 posted)

Android apps on Linux with Waydroid

It is not uncommon for users to want to run a program targeted to one operating system on another type of system. With the increasing prevalence of smartphones, Android has become the world's most widely used operating system. So users may want to run Android apps on Linux systems in order to get access to a game or other app that is not available in a Linux version or to develop mobile apps on their desktop system. The Waydroid project provides a way to run those apps on Linux, which means they can run on a variety of devices, including Linux-based smartphones like the PinePhone.

Waydroid is similar in concept to the Windows compatibility layer Wine. The fact that Android runs on the Linux kernel makes properly running Android apps on other Linux systems much simpler than doing so for Windows software. It is not possible to simply run Android apps directly on a regular Linux operating system, though, because they depend on a different user-space environment. However, by using kernel features such as namespaces, it is possible to run the entire Android user space in a container on a Linux system. This is the technique used by Waydroid; it runs a complete Android system in a container in much the same way that it is possible to, for example, run Debian in a container on Fedora. That allows Waydroid to have better performance than it would have running in a virtual machine or an emulator.

Waydroid runs a custom build of the LineageOS Android distribution. It has all of the software features of LineageOS, though it does not emulate all device features. For example, cameras and telephony features are not supported; WiFi and Bluetooth cannot be configured from within Waydroid either. Networking is supported, however; Waydroid always shows an Ethernet connection, which actually routes traffic through the host. Audio input and output both work using the host's configured audio paths. Other than these and a few other minor differences, Waydroid is mostly similar to a regular Android device without Google apps.

Hardware support

Waydroid supports 32-bit and 64-bit x86 and Arm. 64-bit Arm has the best app support, because it is the architecture used by the vast majority of regular Android devices, but many apps do also work on 64-bit x86. I did not test any 32-bit devices, though. Many apps are written entirely in Java and/or Kotlin, both of which compile to architecture-independent Java virtual machine (JVM) bytecode; these apps work on all architectures without any extra effort from the developer. Other apps include native code compiled from languages like C or C++; these apps must be compiled for each platform, but many developers still build for x86 because most Chromebooks, many of which have x86 processors, also support Android apps.

Intel and AMD GPUs, as well as the GPUs integrated into most Arm SoCs, are supported for hardware graphics acceleration. NVIDIA GPUs are not supported (other than the GPUs in Tegra Arm SoCs), but Waydroid does support software rendering as a workaround.

I tested Waydroid on my PinePhone (64-bit Arm, running DanctNIX with Phosh), two laptops (Dell Inspiron and Lenovo IdeaPad 3, both x86-64 running Arch Linux), and my tablet (Microsoft Surface Go 2, also x86-64 with Arch); all of the devices have touchscreens. I used Wayland on all four, because Waydroid requires it.

Unsurprisingly, the overall experience is best on the phone. The experience with Waydroid on the PinePhone is not much different from using a regular Android phone, other than the limitations of Waydroid that are not present in normal Android devices, such as telephony and the camera not working. Because the PinePhone's hardware is slower than most other Android devices, I disabled user-interface animations. This is an issue with the hardware, however, not with Waydroid. After disabling animations, Waydroid is almost as responsive as an actual Android phone.

Waydroid also works quite well on the laptops. Because most apps are not optimized for use with a keyboard and mouse, I use the laptops' touchscreens much more in Waydroid than I do with regular Linux software. Quite a few apps are designed to support keyboard and mouse input for compatibility with Chromebooks, so those tend to work even better. The experience would be much worse on a desktop or a non-touch laptop, but Chromebook-optimized apps would still work well.

On the tablet, touch gestures did not work properly (a swipe was registered as a long tap at a single point), though they work fine in Linux itself; this made Waydroid almost unusable on the tablet. Because of this problem, I did not do much testing. Surface devices often have problems with Linux, though, so it is likely that this is a device-specific issue (possibly even specific to my software setup) rather than a general problem with Waydroid on tablets.

Waydroid does not work with the default kernel provided by some distributions because it requires the binder and ashmem modules. It appears that Ubuntu and Debian both provide these modules by default, while Fedora and Arch do not. I did not check any other distributions. On the laptop and tablet, I installed linux-zen, which is an alternative kernel available for Arch that does provide the modules. The default kernel used in DanctNIX on the PinePhone provides them, so I did not have to replace its kernel.

The process for installing and running Waydroid varies depending on the distribution. I followed the instructions on the Arch wiki for the laptops, so I installed waydroid and waydroid-image from the Arch User Repository (AUR) after installing the Zen kernel. After that, Waydroid had to be initialized with "sudo waydroid init" and the waydroid-container service needed to be enabled and started for systemd. The Waydroid documentation has instructions for installing it on other distributions.

Waydroid has two modes, multi-window mode and full user interface (UI) mode. When multi-window mode works properly, Android apps are integrated into the desktop as if they were Linux desktop apps. On all four devices, however, multi-window mode has several bugs that made it difficult to use, so I only use full UI mode. This runs the entire Android UI in a single window.

Waydroid creates .desktop files for every app installed, including the default system apps, and this cannot be disabled. These desktop entries launch the apps in multi-window mode. If you only use full UI mode, however, they just create unnecessary clutter in the menus. The icons can be hidden by adding Hidden=true to the end of each waydroid.*.desktop file in ~/.local/share/applications. Deleting the .desktop files is futile, because Waydroid will simply create them again the next time it is started.

App support

As would be expected, app support is best on the PinePhone, both because it is an Arm device and because most Android apps are primarily designed for phones. On the laptop, most apps are usable, although mouse support is incomplete in many apps; the touchscreen works fine.

One of the most significant differences between Waydroid and a typical Android device is its lack of Google apps. This is certainly beneficial for privacy, but it does have some drawbacks. Many apps cannot be installed or will not work properly without Google apps and services.

The Google Play Store is not available, significantly limiting the number of apps available to install. Many apps that would otherwise work fine in Waydroid, especially proprietary ones, cannot easily be installed because they are only distributed through the Play Store.

F-Droid works well in Waydroid, and does have good mouse support. The vast majority of apps listed with no "anti-features" will work, and many with anti-features also work. The anti-feature most likely to cause problems is "non-free dependencies": often, the non-free software that this anti-feature refers to is Google Play Services. This can cause problems ranging from no push notifications or missing maps to apps that do not even open. Of course, apps that depend on unsupported hardware features will not work properly regardless of whether or not they have any anti-features.

It appears to be possible to make some of these apps work by installing microG, but I did not test this due to concerns that using it may violate the Terms of Service for Google Play. Aurora Store is an alternative Play Store client that most likely works in Waydroid and could be used to install many Play-Store-only apps; I did not test it either for the same reason.

One limitation of Waydroid is that when a link is clicked in an Android app, there is no option to open the link in the host browser without installing additional software. There is an open issue on the Waydroid repository for this, but as a workaround until this feature is added, I wrote a Python script and Android app to add it. The Android app is installed in the Waydroid container and set as the default browser (although it is not really a browser, it is configured to appear in the list of available browsers), while the Python script runs on the host OS. When a link is clicked in an Android app, the "browser" connects to the Python script, which then opens the real browser on the host.

Waydroid development takes place in a GitHub repository. The project's Web site lists three members of the development team, and GitHub currently shows 25 contributors to the main repository. The latest release is v1.2.1, which came out in April, but there has been quite a bit of development since then. Overall, releases have been somewhat sporadic; v1.1.0, the first release listed on GitHub, was published in September 2021, followed by v1.1.1 two days later. The next release, v1.2.0, came out a month after that, then there were no releases between October and April.

Conclusion

Overall, despite some issues and drawbacks, Waydroid is a useful way to run Android apps on Linux, especially on non-Android Linux phones. Like any software, it has some bugs, but most of its problems are caused by the inherent differences between a computer running only or almost only FOSS and a smartphone with large amounts of proprietary software; services considered essential on most Android devices are missing in Waydroid. And, when it is used on a desktop or laptop, the input devices (keyboard and mouse) are fundamentally different from the touchscreens that Android (and most of its apps) are primarily designed for. The Android UI and many apps already have good keyboard and mouse support for compatibility with Chromebooks, but it is still quite evident that Android is primarily designed for smartphones.

Unfortunately, devices without Google Play Services and the Play Store are so rare that there is little incentive for developers to avoid using Play Services or to publish their apps in alternative channels; the primary exceptions are developers of FOSS and/or privacy-focused apps. Of course, some Linux users would not want to use other apps anyway, so this may not be an issue for a lot of Waydroid users.

Even with these limitations, Waydroid significantly expands the range of software available to Linux users, especially those with Linux smartphones. Of course, it is not an ideal solution, just as Wine is not an ideal solution to the shortage of Linux desktop software; it would certainly be better to have more native mobile-Linux apps. Overall, however, Waydroid is quite useful to Linux phone users who do not want to be limited to the few native apps designed for Linux phones. Waydroid is definitely worth trying on any device where one wants to be able to run Android apps.

Comments (21 posted)