LWN.net Weekly Edition for October 5, 2023

Welcome to the LWN.net Weekly Edition for October 5, 2023

This edition contains the following feature content:

• BPF and security: Alexei Starovoitov looks back at how BPF and security have interacted in the Linux kernel over the last ten years.
• Impressions from the GNU Project's 40th anniversary celebration: a report from the GNU Project's celebration in Biel, Switzerland.
• Security policies for GNU toolchain projects: some projects are trying to better define what constitutes a security bug in order to head off spurious CVE-number assignments.
• Revisiting the kernel's preemption model, part 2: a way to simplify the kernel's handling of preemption and make it work better too.
• Linux ecosystem contributions from SteamOS: how supporting gaming has made Linux better overall.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

BPF and security

The eBPF in-kernel virtual machine is approaching its tenth anniversary as part of Linux; it has grown into a tool with many types of uses in the ecosystem. Alexei Starovoitov, who was the creator of eBPF and did much of the development of it, especially in the early going, gave the opening talk at Linux Security Summit Europe 2023 on the relationship between BPF and security. In it, he related some interesting history, from a somewhat different perspective than what is often described, he said. Among other things, it shows how BPF has been both a security problem and a security solution along the way.

Universal assembly

BPF is something like the vi editor, Starovoitov began, people either love it or they hate it, but both are simply sequences of commands. There are lots of definitions of BPF, but the one he wanted to use for the talk is that it is "a universal assembly language". It is the first strictly typed assembly language; there is no "pointer to memory" in BPF, all pointers are to specific types.

Because it is universal, it goes way beyond the use case of user space telling the kernel what to do; there are now hardware devices that send BPF to the kernel to describe how to use them. There are also user-space-to-user-space applications where the kernel is not involved at all; one application is telling another, perhaps on the other side of the world, what to do.

He is often asked about the difference between BPF and WebAssembly. BPF is not a sandboxed environment like WebAssembly or JavaScript in a browser; sandboxed environments do not know what code they are going to run so they have to restrict the execution environment. They create a boundary, but that slows down the performance due to all of the run-time checks.

BPF, instead, is statically verified so there are "practically no run-time checks"; the only checks are for things that the verifier cannot statically determine. The major difference is that the intent of a BPF program is known before it is executed, which is not at all the same for sandboxes, which have to run arbitrary code. He often sees comments (on LWN in particular) about adding WebAssembly to the kernel; his response is that those developers should "bring it in". He believes there is enough room in the kernel for WebAssembly or some other sandbox environment in addition to BPF.

When eBPF was first proposed in 2013, it was called "internal BPF" (iBPF); it eventually moved to "extended BPF", thus eBPF. In addition, eBPF has itself been extended multiple times; in LLVM, these newer instruction sets are different CPU models that are selected with the ‑mcpu option using values v1 to v4. The v4 support was added in July 2023; it is the instruction set that GCC now also supports.

Starovoitov said that his next slide (number 9 in his slides) was the most important one in his presentation. Each project should have a mission statement, he said. BPF has two parts to its mission, first, "innovate" and, second, "enable others to innovate". The project continues to satisfy his "thirst for innovation", but it also enables others to do new things. One of his favorite parts of working on BPF is helping people who post on the mailing list with something new they are trying to do; that is "the best moment of being a kernel maintainer".

He thinks that attitude for the project shows in the growth of the BPF development community within the kernel. He displayed a graph of unique developers by month since the beginning of 2019, which shows a general rise from around 50 to well over 100 in that span, while the BPF team at Meta, which he is part of, has been fairly flat at around ten or fifteen developers over that same span.

Tracing and networking

"BPF has roots in tracing", he said. The first hook that an eBPF program could be attached to was for kprobes and uprobes; tracepoints came next, then function entry (fentry) and exit (fexit). People often think that BPF can do anything within the kernel, but it is actually quite restricted. For tracepoints, the BPF program can read any kernel data, but it cannot modify it at all. Networking BPF programs can read and modify the packet data and drop packets, but they cannot modify the kernel state. The restrictions on what various types of BPF programs can do are based on the use cases for those programs.

He gave some examples of BPF tracing. Android uses BPF programs to track network usage based on the contacted host, he said. So if a user wants to see their Facebook use on their phone versus YouTube use, say, they are getting that information by way of a BPF program. The PyPerf program uses BPF to profile Python programs. Beyond that, BPF programs can be attached to user-space programs using uprobes such that any invocation of a program will have the BPF program attached. That can be used to see how much time GCC spends processing include files versus compiling the code; each invocation of GCC in a parallel make will be instrumented correctly to gather that data.

There are multiple networking use cases for BPF as well. He noted that the express data path (XDP) feature of the kernel came about as a way to fend off distributed denial of service (DDoS) attacks. At one point, Facebook was under a DDoS attack of 500Gbps that was mitigated by putting a BPF program at the network-driver level using XDP. Absorbing the attack that way provided a 10x improvement over earlier DDoS-mitigation techniques, he said.

Security

The ability to attach BPF programs to Linux security module (LSM) hooks (also known as BPF-LSM) is a recent addition to the kernel that is being used to "prevent interesting security attacks", Starovoitov said. As with other BPF program types, the BPF programs that can be attached to LSM hooks (or to system calls) have specific capabilities that are different from those of tracing, networking, or other program types. Those programs can read arbitrary kernel data and deny operations, but they can also sleep, which is something new for BPF programs. That means the program can cause a minor fault if the user-space address it is accessing has been swapped out, so it is not possible to evade these hooks by referring to swapped-out memory.

Unfortunately, the BPF-LSM programs are generally not publicly available, unlike those for, say, tracing and networking. At least 90% of the programs that he knows about in those realms are freely available; in particular, many of the large internet companies are working together on things like DDoS protection, sharing their code, and learning from each other. The same is true in tracing, but the story for BPF-LSM code "is not that good"

The BPF feature set for networking, tracing, and, even, security is pretty much set at this point; they are 95% done, though, of course, the last 5% takes lots more time, he said. But there are still new things being added to BPF, including the recently landed BPF for human interface device (HID) feature. This allows BPF programs to modify how HID devices, such as keyboards and mice, are seen by the kernel, to correct problems (quirks) or change the behavior in some way.

Starovoitov said that he is excited about the extensible scheduler class that would allow BPF programs to perform scheduling functions in order to test out new scheduling algorithms. There are always niche use cases where a more-specialized scheduler is needed, especially in cloud workloads where the schedulers in the virtual machines end up fighting with the hypervisor scheduler. So far, at least, pluggable scheduling using BPF has been rejected, though that was not mentioned in the talk.

Unprivileged BPF

The original Berkeley Packet Filter (BPF) instruction set was created 30 years ago; it lives on in Linux as "classic BPF" (cBPF) and is used by tcpdump and seccomp(). Uses of cBPF are unprivileged, so eBPF followed suit: two of the 32 BPF program types can be used without privileges, and both only allow reading packet data and dropping packets. One of those, BPF_PROG_TYPE_SOCKET_FILTER, has been completely unused, he said; all of the other program types always required root privileges.

That was fine for the first few years of eBPF's existence in the kernel, he said—until 2017. That was when Jann Horn of Project Zero wrote some BPF code that demonstrated speculative-execution problems, which eventually became known as Spectre v1. Modern CPUs all do speculative execution, but the side effects of their mispredictions still remain in the caches, so they cannot be hidden. As was seen at the end of 2017 (and in the years since), those side effects can be turned into security vulnerabilities.

The hardware vendors response to these problems was to recommend that any possible branch mispredictions be stopped by adding "load fence" (lfence) instructions throughout the code. Microsoft followed this advice and changed its compilers to emit those instructions all over the place; Windows and other tools were then rebuilt.

The hardware vendors requested that the Linux kernel do the same, but kernel developers had other ideas, he said. The lfence instruction is a big hammer, with major performance implications, so it was decided that the kernel would manage speculation by steering it in safe directions rather than turning it off with lfence. It took a lot of work to convince Intel and Arm that the technique was feasible, but it resulted in a better fix for the problem. The array_index_nospec() macro from that patch is used 240 times in the kernel today, some of them in extremely hot paths, such as looking up indexes in the file-descriptor table. The impact of using lfence instructions instead would have been huge.

BPF was used in Horn's exploit, so changes were needed there as well. BPF cannot use the macro directly, but it makes an equivalent change to avoid Spectre v1. Only a few months later, though, Horn came back with a Spectre v2 exploit that used the BPF interpreter, which caused some additional concern about the security story for BPF. The exploit was not actually loading BPF code into the kernel; the speculative execution was using the interpreter on BPF instructions that lived in user space.

The solution was to avoid having the interpreter code in the kernel executable. BPF code can either be interpreted or run with the just-in-time (JIT) compiler, so the BPF_JIT_ALWAYS_ON option was added to always enable the compiler for BPF and remove the interpreter. While BPF was changed to avoid this problem (which is, of course, really in the CPU hardware), he believes that any interpreter in the kernel could be used in this way; there are at least three other interpreters, so the kernel is still not fully safe, Starovoitov said.

It is interesting how the perception of the BPF JIT compiler has changed over the years, he said. In 2011, there was a JIT spraying attack against it, which made some kernel developers wonder whether JIT compilation had any place in the kernel. That problem was addressed at the time, but now the JIT compiler has to be enabled to avoid Spectre v2. The BPF developers have also found that the JIT compiler recovers performance that was lost to retpolines, which are another Spectre v2 mitigation.

In 2019, the BPF developers decided to get smarter with the verifier to avoid other speculative-execution problems. Daniel Borkmann worked on verifier changes to detect and avoid these problems. To do so, the verifier simulates both normal and speculative execution, which is "unique in the industry, no other static-analysis tool can do this kind of speculative analysis".

Along came Spectre v4, which was mitigated with a handful of lines of code in the verifier to sanitize the stack. But other Spectre variants kept showing up, so it was eventually decided to add a configuration option to disable unprivileged BPF completely. The two program types that could be used without privileges were "extremely niche use cases with a number of users less than the number of fingers on a hand"; it simply was not worth the continued pain to the BPF community to keep trying to support that feature, Starovoitov said. The BPF_UNPRIV_DEFAULT_OFF option defaults to "on" so that distributions do not allow unprivileged BPF programs, though administrators can override that choice.

CAP_BPF

Over the years, there were persistent calls to split out some BPF permissions from the root privileges (actually CAP_SYS_ADMIN) needed to perform nearly all BPF actions. The CAP_PERFMON capability was added by the perf subsystem, but it has been adopted by BPF as well; it allows reading kernel memory. The CAP_BPF capability was added to govern various types of BPF operations; it can be combined with CAP_PERFMON to allow loading useful tracing programs or with CAP_NET_ADMIN to allow loading useful networking programs—either without requiring CAP_SYS_ADMIN.

There is a perception problem with CAP_BPF, however, he said; it is not clear what it is actually meant to govern. Part of the problem is that BPF is not constrained by namespaces; if you can look at kernel memory, you can look at all of kernel memory, not just that in a single container. CAP_BPF is meant to work like CAP_SYS_MODULE, which is the capability required to load a kernel module; that capability effectively gives permission to crash the kernel because malicious (or buggy) modules can do just that.

But verifier bugs can lead to BPF programs crashing the kernel, which should be expected as a possibility, but is treated as a security hole instead, Starovoitov said. So every verifier bug gets a CVE, which is a real problem. He noted a mid-September LWN article on the topic of "bogus CVEs", which is a problem for the BPF project as well. Bugs are fixed, but get CVEs filed for earlier kernel versions where backports have not been made; sometimes those CVEs even reference the self-test code that BPF runs to ensure the bug remains fixed. The existence of a CVE then creates a panic to fix the older kernel.

Some security startups are using BPF in strange ways. He noted one unnamed startup that complained about what BPF can do and the dangers to systems inherent in the existence BPF; that was all done to sell the startup's product, of course. The strange piece is that the product was using BPF to protect against all of the BPF problems it was railing against. "In the end, they were saying 'BPF is bad, use BPF to protect from BPF'."

He began running short of time, so he started quickly moving through the rest of his talk. He noted that one of the few BPF-LSM programs that is open source is one that is used by systemd to prevent the mounting of filesystem types based on allow and deny lists. He showed a few other examples of how systemd is using BPF to enforce various security policies. In some cases, it is using the BPF-LSM hooks, but in others it is using other BPF program types (e.g. networking and tracing) to do its job.

Starovoitov said that he believes all kernel modules should really be written as BPF programs. The advantages of kernel modules is that they can be written as arbitrary C code, with full access to the kernel's symbols, but that means they can also crash the kernel due to a bug. For BPF programs, safety is built into them via the verifier. In addition, BPF programs are more portable than kernel modules. That portability is an underestimated advantage, especially for companies with large fleets of systems, some long tail of which will have a variety of different kernel versions.

He closed with a quick note that he thinks the version of C that is used by BPF is a better version of C for kernel programming. The safety that has been built into the flavor of C that can be verified is a better choice for kernel programming overall. His slides showed a few buggy constructs that could be avoided, but he was not able to get into any of the details, though some were mentioned in a talk from a year ago. One suspects that may not be an opinion that is widely held outside of the BPF community—at least yet.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Bilbao for LSSEU.]

Comments (8 posted)

Impressions from the GNU Project's 40th anniversary celebration

On September 27, 1983, Richard Stallman announced the founding of the GNU project. His goal, which seemed wildly optimistic and unattainable at the time, was to write a complete Unix-like operating system from the beginning and make it freely available. Exactly 40 years later, the GNU project celebrated with a hacker meeting in Switzerland. Your editor had the good fortune to be able to attend.

An anniversary like this, one might think, would be an occasion for a fair amount of introspection and planning for the future. That was not the case here, though. There was almost no looking back at GNU project history, little evaluation of strategy or tactics, and no planning for the coming years. The GNU project, it seems, is happy with what it is and feels no need to talk about where it is going as a whole.

The one exception, perhaps, was Panos Alevropoulos, who works on the Free Software Foundation's efforts to end software patents. Some good things have happened in those 40 years, he said; free software exists for almost every purpose, free social networks (including the fediverse) are rapidly growing, the quality of the software is high, and it is supported by a community of passionate people. On the other hand, he said, many features and capabilities still come to proprietary software first, there is no 100% freedom-respecting hardware, no prospect of regulatory action against digital rights management or software patents, and free software is still either unknown to or misunderstood by the general public. In many ways, he said, the project has failed to meet its goals.

This failure, he said, shows that a strategy centered around the development of software is not enough. What is needed is a way to make free software the default for more users. That might be done by creating applications that achieve industry-standard status; VLC, he said, is a good example of how that can work. Working to ensure that software developed with public money is free (something that, as Matthias Kirschner emphasized earlier in the day, the Free Software Foundation Europe is pushing hard for) is another step. Eventually, he said, consideration could be given to banning proprietary software entirely; not everybody in attendance was convinced that the GNU project should venture into advocating for changes to copyright law, though.

The discussion moved on quickly and Alevropoulos's talk did not echo through the rest of the meeting.

The ostensible highlight was an address by Stallman himself. His appearance may have been a shock to many in the room; he disclosed that he has been fighting cancer, and he looks the way cancer patients often look. That fight, he said, is going well, and there is every reason to be optimistic; he added that the community is going to have to put up with his presence for many years yet.

Stallman's talk did not look back to his original announcement at all. Instead, he wandered over a number of topics, seemingly disconnected from each other. For example, Red Hat's changes with regard to access to (and redistribution of) source code were deemed to be "nasty", but he sees no basis for a copyright suit against the company. He said that he had no conclusive answer on whether Red Hat's policy violates the GPL, but it is clear that it is antisocial; Red Hat should change its approach.

He also spent a little time on generative artificial intelligence and the concerns that these systems might constitute a violation of the copyrights on the works used to train them. There are many uses of these systems that run counter to freedom, he said, and those uses should be illegal. But the way to get there is through legislation, not through licensing. Restrictions on activities should be the result of democratic processes and not one person adding rules to a license; no one person should have that power.

The part of the meeting that seemed to resonate most with the attendees, though, was the considerable amount of time given to the presentation of a number of projects that are being developed under the GNU umbrella. The utilities that the GNU project is most widely known for — Emacs, the compiler toolchain, command-line utilities — were notably absent from this gathering; there was almost no overlap with the attendees of the GNU Tools Cauldron, held just a few days before, for example. Much of the energy in the GNU project, today, is focused on rather different projects:

• Martin Schanzenbach presented the GNU Name System, an attempt to create an alternative to the domain name system that lacks central servers and control points.
• Florian Dold talked at length about Taler, an electronic payments system that is trying to solve current problems while avoiding many of the mistakes made by others. There are no offline payments, no blockchains, no unregulated radical privacy, no smart contracts, and no reliance on big tech companies. Taler is meant to provide buyer-side anonymity while maintaining transparency and auditability on the seller side. The talk featured cameo appearances from a member of the Swiss parliament who has tried to create an awareness of Taler in that forum, and from the founder of the NetzBon regional currency, which is looking at integrating Taler.
• Luis Falcon discussed GNU Health, a hospital information system with active deployments worldwide. This project is working to make state-of-the-art capabilities freely available while, of course, adding protection for the privacy of medical information.
• Sébastien Blin works on Jami, a communications platform focused on privacy and a distributed implementation.
• Luca Saiu is developing GNU Jitter, a system for the creation of highly efficient virtual machines.
• Tobias Platen presented Libre-SOC, a project that is working toward the creation of a fast, secure, and 100% free system-on-chip.
• Mohammad-Reza Nabipoor was arguably closest to the GNU project's roots with his presentation of GNU poke, an editor for binary data that has been covered here in the past.

The conclusion to be drawn from all of this is that, without actually saying so, the GNU project has moved on from the task of creating a free operating system. That has been done, and done well. But there is a long list of other problems — urgent problems — that can be addressed with free software, and today's GNU developers are putting their effort into many of them.

There are plenty of easy criticisms to be made regarding the GNU project and its founder; those have been reiterated many times, and it is your editor's hope that readers will avoid doing so yet again in the comments here. But, those criticisms notwithstanding, it is true that, 40 years ago, Richard Stallman saw something more clearly than the rest of us did, expressed a compelling vision of a better world, and changed how we deal with technology. Perhaps free software would have eventually found success without the GNU project, perhaps not, but there is little doubt that it would have come much later. We have all benefited hugely from the GNU Project, and its 40th anniversary is worth celebrating.

Much of the GNU Project's stated mission is arguably obsolete, by virtue of having been accomplished, even if we never did quite get that Empire game that Stallman promised. But there is still a lot of work to do. The GNU Project needs to refocus itself on current problems while continuing to pursue its goal of software freedom. As can be seen, GNU developers are busily doing exactly that; arguably, they are leaving the GNU Project behind in the process. We desperately need freedom and privacy-protecting solutions to a wide range of problems, far beyond the operating system.

The GNU Project has laid a good foundation but, as Alevropoulos said early in the day, it has not created a world where free software solutions are the default. It has not, yet, made the case for free software to the world as a whole. Some new thought into how to solve that problem, along with an intensification of the energy being put into projects like those described above, could cause the GNU Project's next 40 years to far overshadow the last 40.

Meanwhile, we at LWN, like many others in the community, hope that Stallman's health situation continues to improve and that he is able to see the GNU project's course over the coming years.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel to this event.]

Comments (90 posted)

Security policies for GNU toolchain projects

While the CVE process was created in response to real problems, it's increasingly clear that CVE numbers are creating problems of their own. At the 2023 GNU Tools Cauldron, Siddhesh Poyarekar expressed the frustration that toolchain developers have felt as the result of arguing with security researchers about CVE-number assignments. In response, the GNU toolchain community is trying to better characterize what is — and is not — considered to be a security-relevant bug in its software.

Fuzzing the binutils utilities has, he began, become a popular exercise; it is an example of the "fuzzing epidemic" that is happening more widely. Fuzzing is good, he took pains to say, but what happens afterward is not. Researchers have started filing for CVE numbers, often for bugs which are not, in truth, security problems. The "infection" is spreading from binutils into the rest of the toolchain ecosystem. The whole CVE system, he said, is broken. People will report issues and get CVE numbers, but nobody involved has any real understanding of the context in which these bugs are found. As a result, a lot of engineering time goes into rebuilding packages, backporting fixes, and so on, all for problems that are not seen as valid security issues.

Poyarekar would like to create a better focus for security efforts and channel that work into a more helpful direction. Doing so requires creating a better understanding of what constitutes a security issue. In short, a security issue is a bug that allows a user to do something that they would otherwise be unable to do. Issues can be divided into a few subcategories; there are, for example, "direct vulnerabilities" that affect the integrity of the system as a whole. Other types include security features (such as hardening) that do not work as they should, or design flaws that make exploits easier. The first two types tend to get fixed, often after the assignment of a CVE number. Design flaws can get CVEs from zealous researchers as well, but tend to result in little more than "hand wringing".

Setting the context

When it comes to security issues, he said, context matters. What is the use case under which any given bug can be exploited? He is working to set the context for toolchain projects, which involves defining usage models, followed by the safety expectations in each case. GCC, for example, cannot be expected to be safe in every context. Procedures for security issues can only be established after this context has been set.

On the topic of runtime libraries, Poyarekar noted that Florian Weimer has done the work to define a security context for the GNU C library; it is spelled out in the project's top-level SECURITY.md file. That policy has already worked to enable better triage of security issues. In general, runtime libraries must provide the highest standard of security. But here, too, it varies: passing an uninitialized pointer to memcpy() is not a glibc security issue, he said; neither is any use of functions like gets().

Additionally, not every loadable shared object should be seen as this type of runtime library. Libraries like libiberty and libbfd, for example, are not intended to be used in arbitrary situations; they are expected to be robust, but are not meant for secure contexts. Bugs in these libraries should not be seen as security issues.

Carlos O'Donell asked what happens if this definition, as established for a specific library, doesn't meet his needs. For example, he said, it is generally accepted that there is no way to prevent regular expressions from consuming an arbitrary amount of memory, so it is understood that a program cannot accept arbitrary regular expressions from an untrusted source and still be secure. If he wants, instead, to be able to securely evaluate user-provided regular expressions (and expects the library to support that), how does he get the policy changed? The answer was that it would require a discussion among the developers, who will evaluate such requests on a case-by-case basis.

Another context is analysis programs (the objdump utility, for example). There is an expectation that such programs can accept arbitrary input, and they should be expected to be robust. But they cannot be expected to be secure in the face of untrusted input; in such cases, they must be used within a sandbox. An audience member protested that an "all bets are off" attitude does not work, and that even misbehavior must be within limits; a bug should not result in privilege escalation, for example. Poyarekar agreed, but said that, as an example, exposure of a private SSH key is probably within those limits.

Turning to compilers, he stated that they can be expected to produce output for any valid input, but the validation is generally limited to ensuring syntactic and semantic correctness. Any further guarantees are a matter for the specification of the language involved. A compiler should be robust in the face of untrusted input; if it crashes, it's a bug, but not a security bug. Anybody who expects to feed such input to a compiler and maintain the integrity of their system is naive. Compilation of untrusted code should be done in a sandbox; that is what the Compiler Explorer does.

Authority

Heading toward a conclusion, Poyarekar said that the first thing GNU toolchain projects can do is to define a security policy; researchers will generally expect to find it laid out in a top-level SECURITY.* file in a project's repository. That should help to clarify that some types of bugs are not seen as security issues and, hopefully, head off some spurious CVE assignments.

Beyond that, though, projects can play a more active role in the triage of security issues. CVE numbers are handed out by the CVE numbering authorities (CNAs), often with little investigation; the state of the process is "dismal", he said. There is no CNA designated for most projects, meaning that a CVE number can be assigned by anybody that a researcher decides to contact. That CNA may do no research at all and assign a CVE number anyway; once the assignment is made, responding to it is painful.

David Edelsohn pointed out that some researchers are exploiting this process to claim credit for the discovery of more CVE numbers, and that they are especially partial to CNAs that are inclined to assign high severity levels to the CVEs they issue. It is a fully gamed system, he said. Poyarekar agreed, and said that the solution is for GNU toolchain projects to create their own CNA to gain some control over the process.

Glibc's security policy has been in place for a while, he said, and binutils has adopted one as well. Binutils has also allowed Red Hat to serve as its CNA, which has slowed the stream of spurious CVE numbers. There is still work to be done for the other components in the toolchain, though. GDB is an open question; it seems that a number of CVEs that are rejected for binutils are now being refiled against GDB instead. There is a policy under development for GCC now, and a CNA for glibc is being set up as well.

He wrapped up the session by saying that the toolchain projects need to gain better control over the CVE process for their own sanity, and that he is looking for contributors to help with that task.

[Thanks to the Linux Foundation for supporting my travel to this event.]

Comments (94 posted)

Revisiting the kernel's preemption model, part 2

In last week's episode, a need to preempt kernel code that is executing long-running instructions led to a deeper reexamination of how the kernel handles preemption. There are a number of supported preemption modes, varying from "none" (kernel code is never preemptible) to realtime (where the kernel is almost always preemptible). Making better use of the kernel's preemption machinery looked like a possible solution to the immediate problem, but it seems that there are better options in store. In short, kernel developers would like to give the scheduler complete control over CPU-scheduling decisions.

How we got here

The turn in the discussion was driven by this message from Thomas Gleixner, which started with a review of how things got to this point. Initially, no preemption of kernel code was supported at all, as was the case with older Unix systems. As problems were observed, cond_resched() calls were sprinkled into code paths where the kernel was observed to (or suspected of) running for too long and causing problems. Each of those calls is a signal to the scheduler that it can switch to another thread if need be.

Later, the kernel gained "voluntary preemption", which turned hundreds of existing might_sleep() calls into additional scheduling points. Those calls were placed as a debugging aid to catch cases where a potentially sleeping function was called from a non-sleepable context; they indicate a place where it is possible to reschedule, but were never meant to indicate a good place to reschedule. These calls were pressed into service because they were already there and convenient for this purpose.

Later yet, full preemption made the kernel preemptible at arbitrary points, and realtime preemption made even (most) code holding locks preemptible.

This progression is, Gleixner said, an example of the wrong way to be approaching this problem:

The approach here is: Prevent the scheduler to make decisions and then mitigate the fallout with heuristics.

That's just backwards as it moves resource control out of the scheduler into random code which has absolutely no business to do resource control.

He pointed out that the realtime preemption work had run into a similar problem years ago. Making kernel code preemptible, even when it is holding locks, can be good for latency (which is the point of the realtime work), but it can impose a cost in terms of throughput. When preemption can happen at any time, locking contention, in particular, becomes more acute. Often, one thread will cause another to become runnable, at which point the new thread will preempt the first. But if the first is holding a lock that the new one needs, the result will be an immediate block and another context switch. That hurts performance.

In such cases, it would be better to avoid doing the preemption while the first thread is holding the lock. For the realtime case, this was solved through the introduction of "lazy preemption". This code, which has not landed in the mainline kernel, seeks to avoid excessive preemption when (and only when) one non-realtime task would preempt another. If the task that is currently running is holding locks, then the scheduler will set a "lazy preempt" flag rather than preempting that task immediately. Once the locks are released, the preemption can occur. This change restored much of the performance that had been lost for throughput-oriented tasks, Gleixner said, without hurting response time for the realtime tasks.

The problem that is being discussed now is similar: enabling a fully preemptible kernel without hurting the performance of throughput-oriented tasks, many of which do better with voluntary preemption (or no kernel preemption at all). The solution, Gleixner said, can be a variant of the same approach that was taken for the realtime work.

Lazy preemption for the mainline

Some brief background may be helpful for the understanding of the proposed scheme. When the kernel is configured for full preemption, it maintains a "preemption count" that, in short, tracks how many things are preventing preemption of the current task at any time. Its operation is described in this article, which includes this diagram:

Whenever something happens to prevent preemption, such as a call to preempt_disable() or the arrival of an interrupt, the appropriate subfield of the preemption count is incremented. When that condition no longer holds — preempt_enable() is called, for example — that count is decremented. Whenever the count drops to zero, the kernel knows that it can jump into the scheduler to decide which task has the strongest claim on the CPU at that moment.

Calling into the scheduler is expensive, though, so it is best avoided if there is no need to change the running task. Avoiding those calls is the purpose of the "reschedule needed" bit. That bit has an inverted sense; if it is set, then rescheduling is not needed. As long as that bit is set, the preempt count will not be zero, and no calls into the scheduler will be made. When something happens that calls for rescheduling, such as waking a higher-priority task, the bit can be cleared and, as soon as the rest of the count drops to zero, the scheduler will be invoked.

In a non-preemptive kernel, about the only thing that can force rescheduling is the expiration of the current task's time slice. This behavior can be good for a throughput-oriented workload, allowing tasks to run uninterrupted for relatively long periods of time. There are limits, though, and letting tasks run beyond their time slice can lead to latency problems. Long-running kernel code can cause that to happen; avoiding this problem is the motivation for placing calls to cond_resched() in long-running kernel functions. Since the kernel cannot be preempted, it must choose to give up the CPU in such situations.

Gliexner's proposal is meant to preserve this behavior in a fully preemptible kernel. In this world, the existing no-preemption and voluntary-preemption modes would be removed from the scheduler; only full preemption would remain. The maintenance of the preempt count would happen always, as it does with the PREEMPT_DYNAMIC mode used by most distributions now. But there would be one little tweak: if the system is configured to favor throughput, code paths that would normally clear the "reschedule needed" bit would only do so if the current time slice is exhausted. Otherwise, a separate "reschedule eventually" bit, that is not contained within the preempt count, would be set instead.

This change will cause the current task to continue executing for as long as it has some of its time slice left, even if there is another task with the priority to preempt it. There are just a couple of places where that can change; one is on return to user space from a system call, where preemption can occur even in current no-preemption kernels. The "reschedule eventually" bit will be checked there, and might result in a switch to a different task.

The other point where a task might be preempted is when the scheduler interrupt happens; if the scheduler notices that the time slice has expired, it will check the "reschedule eventually" bit. If that bit is set, then the "reschedule needed" bit will be cleared, the preempt count will go to zero, and preemption will happen at the next opportunity.

If, instead, the system is configured for low latency, as might be done for desktop use, for example, the "reschedule eventually" bit will not be used and the kernel will be fully preemptible.

Reworking the scheduler in this way, Gleixner said, would allow the removal of a lot of code that implements the other preemption modes (and which, seemingly, is not heavily used). It would allow the removal of something like 1,400 cond_resched() calls, and would put the scheduler fully in charge of CPU-scheduling decisions. If this solution can be made to work, it looks like a significant improvement over what the kernel does now.

Making it work

Can it be made to work? Gleixner, after having bashed out a proof-of-concept implementation and measured its performance, thinks so: "If this concept holds, which I'm pretty convinced of by now, then this is an opportunity to trade ~3000 lines of unholy hacks for about 100-200 lines of understandable code". Linus Torvalds agreed: "I think you more than proved the concept".

There is, of course, some ground to cover between a proof-of-concept implementation and a reworked scheduler that can be part of a production kernel release. A couple of obstacles (at least) lie in the way. One is that there are four architectures (alpha, hexagon, m68k, and um (user-mode Linux)) that do not support the preempt-count machinery; as things stand, they would be unable to support the new scheduler. Matthew Wilcox was quick to suggest that this problem qualifies those architectures for removal, but there is probably not much in the way of adding preempt-count support instead.

The other problem is that the CPU scheduler is a subtle and complex beast, and Gleixner is not actually a scheduler developer. There will surely be performance issues that will emerge from a change like this, and they will require the right sorts of skills to resolve. Gleixner has let it be known that he is not the person with those skills:

That's as much as I wanted to demonstrate and I'm not going to spend more cycles on it as I have already too many other things on flight and the resulting scheduler woes are clearly outside of my expertise.

Though definitely I'm putting a permanent NAK in place for any attempts to duct tape the preempt=NONE model any further by sprinkling more cond*() and whatever warts around.

So, in other words, the path to a simpler and better scheduler has been laid out, but to get there somebody else is going to have to do the work to push the job through to completion. As of this writing, nobody has stepped forward to take this role. That will likely change, but one should not expect to see a reworked scheduler in the immediate future; this is the kind of change that can take a while to settle into a stable solution. When that happens, though, the payoff should be significant.

Comments (15 posted)

Linux ecosystem contributions from SteamOS

The SteamOS Linux distribution is focused on gaming, naturally, but the effort to build it has resulted in contributions to multiple areas in the Linux ecosystem. Alberto Garcia has been working on SteamOS and came to Bilbao, Spain to describe some of those contributions at Open Source Summit Europe 2023. There are some obvious areas where a gaming-focused OS might contribute upstream, such as graphics, but the talk showed contributions in several other areas as well.

García introduced himself as an employee of Igalia, which happened to turn 22-years old on the day of his talk. He is also a longtime FOSS and Debian developer; he has previously worked on projects like QEMU and Maemo. Lately, he has been working on SteamOS and getting Linux working on the Steam Deck consumer hardware. One of the things he likes about free-software development is the way that new projects end up contributing to older projects that are not necessarily directly connected; SteamOS fits that model well.

Steam Deck

The Steam Deck is a handheld gaming system that was released by Valve, the company behind the Steam game-distribution service, in 2022. The current version of the operating system for the device is SteamOS 3; there were two previous versions, but those do not work on the Steam Deck and have been discontinued. He finds the Steam Deck interesting because it is an example of "a successful consumer device with standard Linux components".

His talk would cover a bunch of different contributions from SteamOS, many of them from his colleagues at Igalia, but the talk was not meant to be an exhaustive list. In addition, there are contributions from other companies and developers, as well as from regular users. García said that he tried to give credit to the contributors in the talk and in his slides, but he apologized in advance to anyone that he had missed.

He showed a picture of the SteamOS user interface, but said that, for the purposes of the talk, it was an uninteresting full-screen interface that allows buying and playing games. Under the hood, though, SteamOS is a Linux distribution that is based on Arch Linux. It uses standard Linux components, with a few customizations, on a filesystem that is based on the Filesystem Hierarchy Standard (FHS). It has a user space with GNU tools and utilities, systemd as its init, and D-Bus for inter-process communication. Unlike many other devices, though, access to the underlying OS is completely unlocked by default; "the user has complete access to the OS, there's nothing hidden". Users can run a shell as root on the Steam Deck.

There are two modes in SteamOS: gaming and desktop. Most users never leave gaming mode, which gives them access to their Steam games, allows configuring things like the network, and more. But the desktop mode is available; he showed a screen shot and said "if this looks like a regular KDE Plasma session, it's because that's what it is exactly". In desktop mode, users can install anything they want, including web browsers, tools, and non-Steam games; in fact, the Steam client allows running non-Steam games that are installed on the Deck. Meanwhile, adding a keyboard, mouse, and display allows using the Steam Deck in desktop mode as a regular desktop computer.

Unlike, say, Android's developer mode, desktop mode is not hidden away at all. The regular power menu has a "Switch to Desktop" option alongside things like "Shutdown" and "Restart". That mode can be accessed by anyone.

Proton and Wine

The use case for the device is to play games on Linux, but there is an inherent problem with that: most Steam games only run on Windows. Those games are never going to get Linux versions, so the Proton compatibility layer is used to allow the games to run on SteamOS. Proton was developed by Valve in 2018 using a collection of different open-source tools. The main components of it are Wine, translators to convert Direct3D to Vulkan, and GStreamer. It is an actively developed project that is available in the Proton GitHub repository.

Wine is a project to run Windows applications on Linux, but it is not an emulator; instead the Windows APIs are implemented using POSIX and/or Linux APIs. It is developed by the community, but the main driver behind the project is a company called CodeWeavers, which works in a partnership with Valve on Wine for Proton. All of the components of Proton are customized for SteamOS and the Steam Deck, but all of those changes are contributed back to the upstream projects.

When implementing Windows APIs on Linux, there are two possibilities; the first is that the API is similar to that of Linux, which means there is no problem. Those APIs can be directly provided in Wine; "there's maybe a little overhead, but it works fine". If there is no equivalent Linux API, Wine needs to implement it, which can result in high overhead; there are some APIs that cannot be implemented in user space as well. Some developers have been looking at which APIs are problematic to implement, but are used frequently in games, in order to add features to Linux to help bridge the gap.

Kernel features

For example, Windows has a synchronization function that is used by a lot of games, WaitForMultipleObjects(). There is no direct equivalent in Linux, so it had been implemented using futex(), but futex() is restricted to a single object. The solution is a new futex() API, which was proposed because the existing API was hard to use and extend. That proposal led to the addition of the futex_waitv() system call in 5.16. The full new API is still a work in progress, but futex_waitv(), which was developed by André Almeida and others, is sufficient to emulate the Windows API call.

A well-known feature of Windows filesystems is that they are case-insensitive so applications expect that "data.bin" and "DATA.BIN" refer to the same file—which is not generally true for Linux filesystems. The "slow solution" to this problem was to implement the case-insensitivity in Wine; when a file lookup failed, Wine would have to look for case-variant versions of the name. The faster solution was to make ext4 case-insensitive, which was developed by Gabriel Krisman Bertazi "from our friends at Collabora". Daniel Rosenberg later added the feature to F2FS for Android, García said.

He quickly went through a few other kernel features, some of which are still in progress, that came from SteamOS, including the user-space adaptive spinlock work by Almeida and Mathieu Desnoyers. Spinlocks are difficult to implement reliably in user space, but changes to restartable sequences have provided a path to do so. Another change was to support mounting two Btrfs filesystems with the same fsid; this was done so that systems with A/B partitioning schemes can mount both filesystems at once when that is needed. Guilherme Piccoli did the work on the Btrfs feature, as well as working on the handling of split locks so that the kernel slowdown applied to those operations can be disabled. Games frequently use split locks, so slowing them down in gaming-centric workloads only leads to unhappy users.

Graphics

There are many more kernel improvements, García said, but then turned to contributions in another critical part of a gaming device: graphics. The RADV Vulkan driver for AMD GPUs was developed by Valve and others; it is an open-source driver that is part of the Mesa project. It is an alternative to the official AMD drivers; RADV is what most Linux distributions are shipping, thus what most people are using, he said.

A related contribution is the ACO shader compiler for AMD graphics, which was developed by Valve and released in 2019. It reduces stuttering and increases the frames-per-second (FPS) rate. There are also some Vulkan extensions for hardware-accelerated video, which needs encoding and decoding support in the graphics drivers and multimedia frameworks, such as for GStreamer; the work on that is still in progress.

The Linux Direct Rendering Manager (DRM) subsystem exposes a small set of color properties; proposals to extend the color API have been made, but none has been adopted. AMD GPUs have additional color capabilities, so a driver-specific color API for AMD devices has been developed; among other things, it will allow displaying high-dynamic-range (HDR) content. This capability, which was developed by Melissa Wen, Joshua Ashton, and Harry Wentland, will be part of the upcoming SteamOS 3.5 release.

GPUs are complex beasts; they can crash, but there is no standard way to report a problem of that sort to user space. There is work in progress to address that lack by standardizing GPU-hang reporting and handling, as well as creating a standard for what compositors do after a GPU reset. Almeida presented on his work in that area at Open Source Summit North America in May.

Valve has developed an open-source micro-compositor, gamescope, in order to solve problems with games trying to use resolutions that are different than the native resolution of the display. "Instead of changing resolutions every time for every game, gamescope tricks the game into thinking it is running in the resolution that the game requested." It then converts the output to conform to the native resolution; it can also do frame-rate limiting for reducing power consumption. Gamescope runs the Steam Deck game user interface and it is available in most Linux distributions.

OS and Flatpak

Part of the work on SteamOS is the same as with any distribution: ensuring that all of the components work well together, handling updates seamlessly, and so on. But there are parts of what SteamOS is doing that are of interest to the rest of the community. SteamOS 3, which is what García has personally been working on, is a customization layer on top of Arch Linux; almost all of the packages come directly from Arch, without being changed or even rebuilt. The Arch Linux philosophy is for packages to be as close to upstream as possible; downstream patches are not applied "unless it is really necessary". SteamOS has adopted the same philosophy; when there is a problem with a package, it is fixed upstream whenever possible.

Unlike Arch, SteamOS has an immutable, read-only root filesystem; it has an A/B partitioning scheme, so there are two versions of the root filesystem available, one of which is the active root. When an update is done, the inactive partition is updated; if all goes well, the update partition becomes the new active root. Users are not expected to do anything that changes the root filesystem, though they have the ability to do so since they have the privileges to remount the root filesystem as read-write; any changes will be lost at the next OS update, however. "So what is the way to install new software?"

Steam games are obtained from the store, "so Steam takes care of everything". For desktop applications, the suggested installation mechanism is to use Flatpak. It is also used by other distributions with read-only root filesystems, such as Fedora Silverblue and Endless OS.

Flatpak is a framework for sandboxed desktop applications that are isolated from the host OS; the isolation level can be adjusted depending on the needs of the application. Flatpak is also a distribution-independent packaging system; it allows the application developers to create a binary package that can be installed on any distribution that supports the framework. Flatpak applications can be distributed from anywhere, but the Flathub repository is "the Linux app store"; there are more than 2000 applications available there and it has become the primary distribution site for some applications (e.g. Bottles). SteamOS developers are working as part of the Flatpak community to fix bugs and add features.

He noted that using Flatpak as the primary distribution mechanism for desktop mode in SteamOS was something of a Trojan horse, as shown in a Mastodon post (and in the image on the left). When application developers add support for the Steam Deck, they are indirectly supporting the Linux desktop. Any effort that is put into improving the SteamOS version is immediately usable on the Linux desktop as well.

Portals are the mechanism used by Flatpak applications to interact with the host, though Portals are not Flatpak-specific. They provide a D-Bus interface to access things outside of the Flatpak sandbox for opening files and URLs, taking screen shots, changing settings, and more.

The Portal API is implemented by a desktop-specific backend, but not all of the backends work on all of the desktop environments, which can cause a variety of different problems. The way to choose the backend was fairly limited and that affected the Steam Deck because it has two separate desktop sessions active, one for each mode. Some work by Emmanuele Bassi gives a better way to choose Portals and backends; it was recently released in xdg-desktop-portal 1.18.0 and will start appearing in major distributions soon.

There is also work on bug fixes and new features for KDE that have been contributed by the SteamOS developers. That includes handling updates with many packages, improving detection of new icons, and better udev-event handling. Beyond that, there are lots of other contributions to things like NetworkManager, Pipewire, ALSA, and more.

García wrapped up by reiterating that SteamOS is making lots of contributions to make desktop Linux better and it is also providing an incentive for desktop-application developers to target Linux with their software. In the Q&A, he was asked about installing other types of packages; Flatpak is only for desktop applications, not for command-line tools. He said that there is no official support for installing those tools, yet, but that people are experimenting with various ways to do so.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Bilbao for OSSEU.]

Comments (23 posted)