|
|
Log in / Subscribe / Register

Leading items

Welcome to the LWN.net Weekly Edition for June 25, 2026

This edition contains the following feature content:

This week's edition also includes these inner pages:

  • Brief items: Brief news items from throughout the community.
  • Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Free-threaded Python: past, present, and future

By Jake Edge
June 22, 2026
PyCon US

Probably the biggest change for Python over the last five years or so is the advent of the "free-threaded" version of the language, which removes the global interpreter lock (GIL) and allows multiple threads to run in parallel in the interpreter. At PyCon US 2026, held in Long Beach, California in mid-May, longtime CPython core developer (and current steering council member) Thomas Wouters gave a talk about the feature. He looked at the motivation behind the GIL-removal efforts, some history, the current status of the free-threaded interpreter, and provided a prediction on where it all leads.

He began by noting that he has been doing CPython core development for about 25 years at this point and has been on the steering council for five of the last six years. The steering council is the body that determines the path forward for language features, including free threading. Beyond that, he works for Meta on the free-threaded interpreter and other things. While it was not entirely relevant to the talk, he noted that he has three cats, while putting up slides to show them. "In an alternate universe, there's a version of this talk where I use my cats as my slides", he said to laughter and applause.

Motivation

But, this is not that talk, he said; "this is a boring talk". While he noted that most in the audience probably already knew, he quickly introduced threads. They provide a way to execute multiple things at the same time in a single process and its address space using separate "threads of control". The primary reason that threads exist is for performance; memory access is slow, so threads are a way to give the CPU something else to do while waiting for memory.

[Thomas Wouters]

The same benefit can come from using multiple processes, but there is more overhead and the address spaces are not shared. There are some additional reasons, beyond performance, that a language like Python needs threads. For example, continuing to execute the main program while calling into blocking APIs or when interacting with a third-party library that requires using threads to access it, "commonly databases".

The GIL is "how CPython decided to support threads" a long time ago, predating his involvement with the language. The GIL protects Python objects and their reference counts, which are used to determine if the objects are still in use; it also protects CPython internals and is "the most efficient way that CPython can support threads". The GIL does not protect a user's Python code because the user does not have control over when the interpreter releases and reacquires the GIL. It "mostly does not protect" C and C++ extensions either, even though it is clearer when an extension call might release the GIL, "but it's still very easy to unintentionally rely on the GIL and end up not being safe".

"Fundamentally, threads are hard"; they are complicated and the GIL does not make them any easier, Wouters said, even though it seems like it does at times. But the GIL also makes threads less useful, which is why there have been efforts to remove it from Python over the years. When the GIL was first added, multi-CPU systems were rare; now his phone has eight cores.

For a long time, the alternative has been to rewrite parts of the program in C, though for the last few years, that has been changed to Rust, which is better. That answer does not always work, however; it means that much of the application needs to be reworked to not only switch the code to the new language, but also to switch the data into that language's domain.

There are other options, such as using multiple processes, but that typically requires more memory and copying data between the processes. Subinterpreters will allow a single process to have multiple interpreters, each with its own GIL, that can be run in separate threads, but there some problems with using that approach. The interaction with third-party libraries, especially those that are not written with Python in mind, "isn't there yet" and there is no good way, so far, to copy data between the subinterpreters. There is also asyncio, which is effectively the same idea as green threads. He is a big proponent of asyncio and thinks it is the best way to do network I/O, "where threads can easily lead you wrong".

But sometimes those solutions do not provide the performance needed, he said. "Without the GIL, multi-threaded solutions to those things can offer higher throughput, lower memory use, and lower latency." But, as mentioned, threads are difficult. That is because sharing data is hard to do. CPUs and compilers optimize for speed and, because memory access is slow, they do various things like caching, prefetching data, and reordering memory accesses.

The interaction between those techniques and threads is "extremely complicated". If a thread writes a value, when does another thread see it, and see all of the value, not just half of what was written? If a thread writes two things, in what order does another thread see them? Handling those problems requires things like memory fences, atomic operations, and memory models, which he was not going to cover. His takeaway: "Shared, mutable data in threads is bad."

The problem for Python is that everything is an object—and everything is shared. Every object has a reference count, which is a small bit of mutable data in every object, "so everything is shared mutable data" for Python. The CPython C API relies on the GIL in multiple ways, Wouters said. For example, PyDict_GetItem() returns a borrowed reference to an item in a dictionary, which is not a problem if the GIL ensures that the object does not change while the code, say, increases the reference count so that it has its own (not borrowed) reference. Without the GIL, the object can change at any time, even though it probably will not frequently cause a problem, but code that is changing the reference count is relying on the GIL for correctness.

Locks can be used to avoid these kinds of problems, but they are expensive and would need to be used all over the place. Python has lists, dictionaries, tuples, code objects, and so on that are "everywhere, the whole process is full of these"; replacing reference counting with a lock would be "extremely expensive". Even just making PyDict_GetItem() use a lock would be expensive and the locks would be present in all Python code, even if it is not using threads. "We can't slow down the interpreter by 50% in order to make multiple threads executing at the same time that much faster", especially since there are no users out there running Python workloads with multiple threads, "because they can't".

History

There have been efforts to remove the GIL going all the way back to Greg Stein's 1996 patch for Python 1.4; it replaced reference counting with a fine-grained lock and added other locks to various parts of CPython. It was not successful because the single-threaded-performance loss was not acceptable.

In 2013, Trent Nelson started PyParallel, which was a completely different model that did not use operating-system threads. It had a different API to create threads from within Python to do various tasks; it deferred reference counting and garbage collection until those threads were done. It wasn't a general solution because it did not work with anything that already used operating-system threads.

In 2015, Larry Hastings started the Gilectomy project, which he worked on for a few years. He looked into novel approaches of dealing with the reference count, specifically, but also into what else would be entailed in removing the GIL from the language. He "pretty much gave up", though there were some other avenues he wanted to explore, Wouters said.

A few years ago, "Sam Gross, who works at Meta, thought he could solve most of those problems and he did so" in a No-GIL fork of CPython 3.9. "That's what we have now, more or less." Gross wrote PEP 703 ("Making the Global Interpreter Lock Optional in CPython"), which described various techniques needed in CPython to support free threading and remove the GIL.

For example, objects in Python now have an "owner" thread, which simply means that the thread has fast access to the object and its reference count; other threads can still access the object but they must take a slower path, which uses atomic operations on a separate, shared reference count. In addition, some reference-count operations are deferred until garbage-collection time and then multiple operations are handled at once; that constitutes a semantic change, however, because objects that would have been reclaimed immediately when the count reached zero will be deferred until the next garbage-collection run. "It's a small difference, it's probably acceptable."

There are also speculative reference-count operations when accessing list and dictionary members. An item object will have its reference count incremented and then CPython will check to see if it is still present in the list or dictionary; "that sounds really weird" and perhaps dangerous since the object may have been destroyed, he said. There are safeguards and it is done under controlled circumstances; it is not for general use, but is "magic in the interpreter itself".

"Quiescent-state-based reclamation" is used to ensure that objects do not disappear out from under other threads. When lists and dictionaries are freed, their memory is not reused immediately so that other threads can still access the reference counts on the objects. The memory is only cleared when the language knows that it is safe to do so, which is generally when garbage collection is done, because all of the threads have reached a quiescent state at that point.

[Thomas Wouters]

Fine-grained locks were added as well; "some things just need a lock". A new garbage collector was added because the existing one was complicated. The memory allocator was switched to mimalloc, which is a thread-safe allocator that also provides the the hooks needed for the quiescent-state-based reclamation.

Putting all of that together results in lock-free lists, dictionaries, and "other important types", he said, which makes access fast, especially for the owning thread; access from other threads is still "pretty fast". When performing actions like changing the type of an object, there is a need to stop all threads to ensure that is done correctly; "we stop the world, then we make the changes, then we run them all again".

There are also per-object locks that provide critical sections. These are not regular locks, "they are a special type of lock that's deadlock-free". He noted that those who had worked with locks would likely find that term to be an oxymoron. It works for CPython, though, because the semantics of the GIL are recreated in the critical section. The existing calls for acquiring and releasing the GIL are repurposed to track which threads are not blocking in an operation so that the "stop the world" operation can complete; threads that are going to block should already be releasing the GIL before doing so.

The end result is that "in free-threaded Python, thread semantics are almost the same as in GIL-ful Python". It avoids the "scary memory model, atomics, memory-order issues, you don't have memory fences, none of that is important, Python just behaves like Python".

Wouters noted that the term "free threaded" did not come from Gross, but from the steering council. It had been called the "No-GIL" Python, but the council did not want to end up having people talk about the "No No-GIL Python". Wouters said that "free threading" was not an industry term, it simply is a Python term that means that the GIL has been removed. It is also a bit of a misnomer, because free threading in Python is much less free than in C or C++, where you can write all over memory without restriction, or even Rust, which is better in that regard, but still has freer threading than Python.

Status

Python 3.13, which was released in October 2024, had experimental free-threading support. It was "relatively slow, with 20-40% slowdown on single-threaded workloads". It had thread-safety issues and some scalability problems, where adding more threads could make things go slower. "But it proved that it worked."

For 3.14, from October 2025, free-threaded Python was "much safer" and "much faster" with a 0-10% slowdown for single-threaded workloads. "I'm still personally shocked that we got to 0% slowdown, that's on Arm hardware, there's some magic going on there, I don't know what it is, but it's amazing." On Linux using GCC, it's usually around 5%, though it can go up to 10%, particularly with older compilers.

He was not part of the steering council that approved PEP 779 ("Criteria for supported status for free-threaded Python"), which he co-authored, for 3.14. It described what needed to be done to remove the "experimental" tag from the free-threaded build and turn it into a supported feature for the language. The PEP acceptance announcement removed the experimental designation and described what the free-threaded developers needed to do moving forward.

Coming in October this year is Python 3.15, which will have a single, stable ABI for both GIL and free-threaded versions of the interpreter. That means extension developers can build their code once and it will load and run on Python 3.15 or later that are using either the GIL or free threading. There are also a lot of scalability improvements for free threading coming in 3.15.

Extension developers may be daunted by the prospect of adding free-threading support, but developers at Quansight have put together a guide to Python free threading. It includes information on porting extensions to the free-threaded interpreter. "It's not that hard."

C global variables need to be protected because the GIL is no longer doing so—if it ever actually was. "You can also just get rid of your C globals, that's not a bad idea." Critical sections can be used to protect mutable data in Python objects. If many threads will be accessing an object at once, other kinds of locks may be desirable, though that can be left as a later optimization.

In addition, extension modules need to declare that they support free threading. There are a number of ways to do that; "check the guide". It is important not to remove any of the existing GIL-related calls in the extension as those are used by the free-threaded interpreter. Adding some multi-threaded tests is also a good idea, Wouters said; "if you have threading bugs, they will show up, because you no longer have the GIL accidentally protecting you some of the time".

For regular Python code, there is probably not much—or anything—that needs to be done. Unless there are already multi-threading bugs in the code, however, since the free-threaded version will make them more likely to occur. Once again, adding some simple tests will likely "expose them that much quicker". The free-threaded interpreter makes it easier to find those kinds of bugs.

Free threading will make using threads in Python more attractive, though, which may also expose bugs in existing code. Making Python code scale well may require more extensive changes. Having many threads accessing the same objects is a problem that Python cannot solve directly. "Sharing objects between threads is still problematic for performance." There are some possible solutions, including the ft_utils package, which has some scalable container types.

Future

There is still work to do on free-threaded Python, Wouters said. There are performance and scalability improvements to be made in the interpreter and in third-party packages that already support free threading. Beyond that, working on supporting free threading in more third-party packages is needed, though it is not necessary to consider free-threaded Python a success. "It's already a success", with support by more than 50% of the top 360 binary wheels on the Python Package Index (PyPI). Much of the credit for that goes to Meta and Quansight, which not only worked on that support, but also helped educate package authors; there were, of course, other contributors to that effort, he said.

While he is on the steering council, "I'm not saying this with any authority whatsoever": he believes that the free-threaded version will become the default Python in an upcoming release. He suspects that the Python project will lag behind Linux distributors, such as Red Hat and Debian, and large companies like Meta, that will have already enabled free threading as the default. His prediction is that it will happen sometime after 3.16 (due October 2027) and before 3.20 (October 2031).

At some point after that, the GIL-based version will go away entirely, he thinks. "This is far into the future, this is next decade, that's what I expect. But, you know, I might be surprised." That ended his talk, but he still had a few minutes for questions.

One of those was about mixing modules that support free threading with those that do not. Wouters said that modules without support for free threading will currently cause the interpreter to emit a warning and re-enable the GIL. That can be worked around with a flag to always disable the GIL, "but that is playing with fire, so you probably only want to do that for testing".

Hastings stepped up to fill in "why Larry Hastings abandoned the Gilectomy", which was met with widespread laughter. He had been trying to use some techniques that were arguably somewhat down the path that Gross eventually took, but ran into huge scalability and performance problems that Hastings found to be unsolvable. He said that Gross had kindly informed him that the technique used for reference counts in the free-threaded interpreter "had not been invented" when Hastings was working on Gilectomy—to more laughter. Wouters got the last word by noting that he had considered putting that anecdote into the talk but did not want to imply that Hastings was not smart enough to invent it himself.

[ I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance with my trip to Long Beach for PyCon US. ]

Comments (5 posted)

AURpocalypse now: a look at the recent AUR attacks

By Joe Brockmeier
June 19, 2026

The Arch User Repository (AUR) has been subjected to a sustained attack recently. The attacker, or attackers, have spun up a series of new accounts then used them to adopt orphaned packages and push malicious updates that would install malware on users' systems. It is unclear how many users were compromised in the attack, but the maintainers were playing Whac-A-Mole for several days to respond to each newly compromised package. The project has turned off the AUR's new-user registration, for now, but it is unclear what its long-term response will be or if the AUR can be secured without major changes to its existing collaboration model.

Why AUR is especially vulnerable

Arch Linux offers official repositories of software, such as core and extras, that are overseen by the Arch Linux Developers and Package Maintainers (see the official contributors categories for more on the hierarchy). These packages are vetted by maintainers and available for download in binary format using the pacman package manager.

The AUR, on the other hand, is a repository for software that has not yet made its way into the official repositories and may never do so. Pacman does not use the AUR repository directly, so users typically turn to a separate AUR helper application, such as paru or yay, for searching the AUR for software, downloading the PKGBUILD files, resolving dependencies, as well as compiling, installing, and updating software.

The AUR is maintained by Arch's Package Maintainers—they respond to requests to orphan or delete packages, and may move packages from AUR to the official extras repository, for example—but there is no formal review process for a package to enter the AUR, nor for any updates to it. The AUR contains the user-contributed PKGBUILD files that are needed to compile software from source; there are no binary builds provided. Currently there are more than 107,000 packages in the AUR, including nearly 14,000 that are currently orphaned and up for grabs.

AUR user registration is typically wide open to anyone who wishes to sign up—currently there are more than 141,000 registered users. Any person who has a registered user account on the AUR can adopt and make changes to orphaned packages. There is no review process or vetting that takes place when a user seeks to publish a new package or adopt an orphaned one; a registered user only has to click "Adopt Package" on the orphaned package's page and ownership of the package is automatically granted.

Use at your own risk

Arch users are expected to exercise care when working with packages from the AUR; they are warned that AUR PKGBUILD files are "completely unofficial and have not been thoroughly vetted. Any use of the provided files is at your own risk". In theory, users will review PKGBUILD files before building or installing the software; it's unlikely that this happens in practice, particularly when a user has already installed a package and is performing an update.

While Arch Linux does not provide builds for packages in the AUR, it does allow "-bin" PKGBUILD files, which are used to download prebuilt binaries from other locations. For example, users can install the LibreWolf fork of Firefox using the librewolf-bin package from the AUR rather than having to build it from source. This is not just convenient for open-source software with long build times; the AUR policies allow proprietary software as well, which is unlikely to be distributed in source form. Of course, this means that users have to be willing to trust that the AUR package maintainer is not providing anything malicious in the binaries.

Arch is not the only distribution that has a service for providing "use at your own risk", unreviewed, user-submitted content; Fedora has Copr, the openSUSE project has the Open Build Service (OBS), and Ubuntu has Personal Package Archives (PPAs). Each of those services allow a person to sign up without any review process and build packages for download by other users of the distributions.

However, there are important differences between those services and the AUR. They provide a build environment that is similar to the ones used for the official distribution packages, and do not allow pre-built binaries or proprietary software. The model for Copr, OBS, and PPAs is that a user creates a project under their own user namespace; users have to add each repository from one of those services separately.

For example, niri creator Ivan Molodetskikh maintains a Copr repository for Fedora users who want to run the tiling Wayland compositor. To install niri from Copr, a user has to enable that repository specifically. It is possible for other Copr users to create a similar project under their own namespace, but it is not possible for another user to take over Molodetskikh's repository unless they compromise his credentials. A would-be attacker could create a malicious fork on Copr and try to lure Fedora users to add that package repository to their system instead, but the attacker cannot simply pick up an orphaned Copr repository to compromise users who have already added it.

The AUR, on the other hand, is much more relaxed about ownership; the PKGBUILD files are all maintained under the AUR namespace. The rules state that, when a new maintainer takes over an AUR package, they are supposed to add their own information as maintainer and then list the prior maintainers as contributors. That, however, is taken on trust and (as seen with the current attack) can be easily abused.

Attacks

The AUR has been subject to a number of attacks over the years that took advantage of orphaned packages. In 2018, three packages were changed to include data-harvesting malware. There were two attacks last year; on July 18 Quentin Michaud sent out an advisory that three packages, all "-bin" packages for web browsers, were uploaded with remote-access trojan (RAT) malware. In late July, there was another attempt, quickly detected; a "google-chrome-stable" package was uploaded with another RAT.

Each of those attacks were small in scale. Not so the most recent attack, which seems to have started (and was detected on) May 27; Fabio Loli reported that the "plex-media-player" package had been updated to install a malicious npm package "crypto-javascript". The PKGBUILD had been orphaned and then adopted by a brand-new account. Several new packages with similar names had also been uploaded by new accounts, all with the same attempt to lure users into installing the malicious npm package.

On June 11, Mark Wagie reported that a new maintainer had adopted the "gnome-randr-rust" package and changed the PKGBUILD file to add a dependency on npm and then use it to install a malicious package called "atomic-lockfile". The attacker also changed the Contributor stanzas in the PKGBUILD to replace the email addresses of prior maintainers—but not the names. The Sonatype blog has an analysis of the package; it included an eBPF program that would attempt to exfiltrate a wide range of information from a user's machine including GitHub credentials, SSH information, browser cookies, data stores from chat applications like Slack, Discord, and more.

The attacker(s) went after hundreds of orphaned packages instead of a select few as with the attacks last year. Jonathan Grotelüschen, one of Arch's Package Maintainers, started a thread on June 11 for people to report compromised packages.

I contacted Grotelüschen by email to ask about the attack; he said that the attacker was creating accounts that would adopt a few packages and compromise them in a similar fashion. New user registration was stopped on June 11 and then re-enabled after the project added Anubis to try to foil the attacker's mass account registrations. That did not work, so registration was disabled again on June 12.

There was another wave of attacks on June 13, conducted with four accounts that had been created before registration was shut down. The attacker switched to using the Bun package manager to attempt to install malicious packages instead of npm, and then attempted to obfuscate the "bun install" command to evade scripts created to scan PKGBUILD files for strings with "npm" or "bun":

    # post_install() {
    #   $'\x63'"d" "/"'t'"m"'p' && "b"'u''n' 'a'"d"'d' 
    $'\141\x6e''s'"i""-"$'\143''o''l''o''r'$'\x73' 
    'n'"e"'x'"t""f"'i''l''e''-''j''s'
    # }

While the obfuscation attempt would evade scripts looking for an install command, any users who actually review the PKGBUILD file would hopefully find it suspect.

Response

In total, more than 1,500 packages are known to have been affected. Grotelüschen said that only about 20 new packages were created, the rest were orphaned packages adopted by the attacker. As of this writing, registration is still disabled. Grotelüschen said that the maintainers "keep the AUR as clean as we can, but at the scale of the AUR, our chance at catching absolutely everything is very small".

He added that "the attack exclusively affected the user-managed content, and the official repositories were unaffected". Furthermore, the AUR packages come with the disclaimer that they are "use at your own risk". Users are responsible for reading PKGBUILDs to decide for themselves if the content is safe to use, Grotelüschen said.

That policy, however, seems to be at odds with the practices of real-world users. It is likely that many users treat the AUR as just another package repository, and rarely if ever read a PKGBUILD file before installing or updating a package. Even careful, security-minded people are unlikely to review every single update to a PKGBUILD, and attackers only need to get lucky once. The AUR's policies may not be sufficient for today's security threats.

The project is working on making it harder to create new accounts, he said, but "the AUR will not require real names or even government id verification". He noted that there are discussions on the AUR mailing list about ways to prevent or detect similar attacks, but the ideas are coming from users rather than Arch maintainers and developers.

Because it's 2026, several ideas of course center around using LLMs, such as Thomas Stromberg's suggestion to use a project he's working on called Atomdrift, which uses "tiny local deterministic AI models, retrained constantly based on recent attacks and threat feeds". Andreas Reichel said he had built something similar called aurscan, which can use Claude, Codex, or local LLMs.

Lukas Grumlik thought that a change in how orphaned packages are handled might help. Instead of allowing anyone to adopt an orphaned package, he proposed that such packages be locked and set to a read-only state. Users could still download the PKGBUILD, but adoption would require "a proper request to take it over, explaining why it needs updating", which would hinder bots from mass takeovers. Reichel replied that such a strategy would discourage well-intentioned people who want to fix packages in good faith, but fail to stop "a state sanctioned criminal syndicate with at least 20k USD deep pockets" and access to LLMs "because my LLM Agent has all the time it needs and will write better explanations anyway".

Another approach would be to build defenses into the AUR utilities rather than the AUR itself. Fidel Ramos suggested that paru display "a loud warning" when installing or upgrading a package that had been orphaned and then adopted. That would likely require changes to the RPC interface for the AUR in order to expose the information needed. Josephine Pfeiffer has opened a merge request that would provide that information.

Jo Guerreiro, maintainer of yay, said that there had been many feature requests asking for changes along the lines of scanning for "npm install yyy", delayed time updates, or maintainer-change tracking. However, he argued that the next wave of malware would change tactics, "with all detection scanning fed into its generation cycle as 'iterate until it is not detected'". There was no way for yay to keep up with attackers, and he wanted to avoid "security theater".

However, he did note that the latest release of yay would now display the last modification time of a PKGBUILD. Recent modifications do not mean that users should not trust a package, but that they should be more careful and review the package. Older packages are not necessarily more trustworthy, he said, "but it is a reason to be more confident that it has been reviewed by the community".

Be careful out there

Considering the open-door policy for submissions and changes in the AUR, it's almost surprising that it has taken this long for an attacker to conduct a campaign at this scale. Nothing good lasts forever, though, and even if the person or persons behind this attack decides to move on, there will almost certainly be copycats.

Slowing new signups may help to some degree, but malicious actors will keep looking for ways to take advantage of AUR users who let their guard down in reviewing their PKGBUILD files. The AUR is an impressive example of community collaboration; it is unfortunate that there is always someone looking to abuse any system built on trust.

Comments (32 posted)

Fedora: 2FA, or not 2FA, that is the question

By Joe Brockmeier
June 24, 2026

Compromised accounts are one of the most common ways that attackers can sneak malware into the open-source supply chain. One way to reduce account compromise is for projects to require two-factor authentication (2FA) or multi-factor authentication (MFA), but that is easier said than done. However, Fedora is currently discussing putting 2FA requirements in place soon, following an an alleged account compromise that led to an AI agent causing a number of problems for the project. After some discussion, Fedora will begin by requiring packagers in the "provenpackager" group to enable 2FA within the next three months or so.

"Rather embarrassing"

Fedora took notice of the agent's activities in May, but it seems to have had access much earlier; how much earlier is unclear. Its contributions appear to have been benign, if unhelpful, but there is no reason to expect that Fedora's luck will hold if another account is compromised. On June 11, Daniel P. Berrangé replied on Fedora's development mailing list in the thread about the unsupervised AI agent. He pointed out that Fedora had considered mandating 2FA two years ago, after the XZ backdoor, but had yet to do so. If the episode was truly a case of password compromise leading to account takeover, "then this is rather an embarrassing situation for Fedora".

Of course, Fedora is not alone; none of the other major Linux distributions with community contributors have 2FA requirements, and it does not seem that there is widespread infrastructure support for it, either. Debian's Salsa collaboration platform does not support enabling 2FA as far as I can tell, nor does openSUSE's Open Build Service (OBS). Ubuntu's single-sign-on service, which is used for its Launchpad collaboration platform, does allow users to enable 2FA, but it is unclear whether there is any requirement for Ubuntu contributors to actually do so. However, Debian and Ubuntu require signing packages with an OpenPGP signature before upload. (Update: Salsa apparently does support 2FA. Apologies for the error.)

Michael Catanzaro said that he uses 2FA for "basically everything *except* Fedora", even though his Fedora account would be of high value to an attacker. "Compromise a Fedora packager and you can push malware more or less directly to users." However, he argued that Fedora is not ready for 2FA.

His complaint was that he did not want to use 2FA until GNOME's online accounts feature supported Kerberos ticket renewals; Fedora uses Kerberos authentication for some infrastructure, such as its koji build system. Catanzaro followed up to say that he was pretty sure he'd made the same objection before, perhaps several years ago, and no progress had been made since. "Unfortunately we're all busy with our usual work, and nobody has been prioritizing these problems. So: basically the usual explanation for how things happen in open source projects."

Support for Kerberos 2FA in GNOME's online accounts feature seems to have been implemented years ago, but has not yet been accepted. Alexander Bokovoy said that he had submitted a merge request to enable MFA Kerberos authentication, but it was stuck on the GNOME side. The feature was submitted in June 2024, and it has been reviewed extensively since, but not yet merged. Catanzaro pointed out that the request's status was still set as "Draft"; perhaps it will see the light of day soon once Bokovoy clicks the proper button.

Make it mandatory first

The problem with waiting until Fedora was fully ready for 2FA, Berrangé said, was that no one was motivated to prioritize 2FA improvements because its use was not mandatory; if it were mandatory, someone would make it a priority to fix the remaining problems. "Or we could continue ignoring the problem until another Fedora account's credentials are compromised and does greater damage that causes Fedora significant reputational harm."

Stephen Smoogen replied that he would love it if that would fix things, but the reality is more complicated:

Anything which slows down builds or makes an already complicated system worse, gets pushed down the queue over higher priority items. Pretty much every time it is said "This time will be different and you have this top priority to work on this over other items" gets a week later "I know we said that, but we really need to make sure that this compose gets out for X reasons so drop whatever you are doing and help".

He noted that the Fedora infrastructure team had been trying to replace its Nagios monitoring with "anything else" since 2009, and other critical infrastructure almost as long, but the priority of those projects does not take precedence over producing the builds for Fedora's twice-yearly releases.

Adam Williamson thought that Fedora should make people use 2FA even if some things wouldn't work immediately; the prevalence of real-world attacks making use of account compromises outweighed the user experience. In addition, he argued that making people use an open-source project is a way to ensure that it gets better. "We've been sitting on our hands saying 'yeah, well, mandatory 2FA would be great but somebody needs to fix <long wishlist> first' for years now. It clearly isn't working. We need to do something else." He added that he had been using 2FA for Fedora work for years, and thought it was fine. The Fedora wiki's instructions for Kerberos authentication cover using 2FA.

At a minimum, he wanted to mandate 2FA be used by members of Fedora's provenpackager group; anyone in that group can commit changes to any package, not only the ones that they own. Thus, a compromised account belonging to someone in that group could do a great deal of damage. He said that Fedora also needed to enable SSH authentication for the project's Forgejo-based collaboration platform, Fedora Forge, require use of SSH keys by proven packagers, then extend that requirement to all packagers.

The forge is hosted on an OpenShift instance, and it is not currently possible to access Git repositories on the platform via SSH because the OpenShift cluster is behind a proxy that is not directly connected to the internet. On June 13, Kevin Fenzi said that he had been working on enabling SSH access, "slowly in the background", and had made some progress but there was yet more work to be done.

Simo Sorce objected to requiring 2FA, on the basis that some people would use the same application for their password and to generate 2FA tokens. "Sadly whenever you improve security you find someone that self-defeats it better." That did not deter Williamson, though; he replied that 2FA could still be useful in many situations.

Convinced

On June 12, Catanzaro said that he had been convinced it was time for Fedora to require 2FA. Gary Buhrmaster wondered what the process was to make 2FA a requirement. Berrangé said that the last time it was proposed it had been submitted to the Fedora Engineering Steering Committee (FESCo).

That was in March 2024, when Miroslav Suchý submitted a proposal to make 2FA mandatory for all Fedora packagers. FESCo accepted the proposal, but watered it down; instead of requiring that all packagers enable 2FA, FESCo changed the policy to say that those in the provenpackagers group "SHOULD" enroll in 2FA. It would consider changing the policy to "MUST" once the user-experience for 2FA was improved.

Little has changed since then to improve the 2FA experience. Williamson followed up the current thread on June 15; he said that he had spoken to Fenzi at Flock and learned that "it's technically difficult to really implement this" at the moment. It would be possible to enforce use of 2FA by monitoring whether actions, such as commits to Fedora's DistGit system, had made use of 2FA. "Then we can go yell at people who didn't follow the rule".

That practice would not scale well for large groups, such as Fedora's provenpackagers group, he said. "So I didn't file the ticket yet for that reason. I'll talk more with Kevin about what might be a way to move forward here." FESCo member "Maxwell G", however, said that he had filed a ticket with FESCo to require 2FA for proven packagers. He acknowledged in the ticket that it was an open question how to implement it, so he was not yet proposing that all packagers be required to use 2FA.

In the proposal's discussion, Fenzi noted that there were 114 people in the group, and 59 of those people did not have 2FA enabled. Maxwell G said that it would be possible to check for 2FA enrollment when adding new people to the group, and set a date to remove people from the group if they had not enrolled in 2FA by the flag date.

FESCo decision

The topic was discussed during the FESCo meeting on June 23 (summary, log). One sticking point during the discussion was Fedora's lack of a recovery mechanism if a user loses the device used for 2FA. Many services offer backup codes that can be used if a user does not have access to their one-time password (OTP) application or hardware.

Both Neal Gompa and Michel Lind said that they used 2FA for other services—but avoided it for Fedora because there was no backup recovery method. Fenzi said that, if all else failed, a person who lost access to their OTP device could email Fedora's admin group and recover their account that way. That is not an ideal solution, but one would hope it would not be necessary often.

After discussion, FESCo decided to require members of the provenpackager group to enable 2FA. The vote was seven in favor, none against; Gompa cast a "+0" vote to indicate his unhappiness with the current 2FA recovery support. "I wish I could be more supportive of it but I feel we're setting people up for failure".

There will be an email to the Fedora development list (draft) as well as an email directly to the members of the group to notify them of the new requirement. There will be a grace period of three months before the requirement would be enforced. After that, users will be removed from the group if 2FA is not enabled. Once it is enabled, they would need to file a request with the infrastructure team to be added to the group again.

The addition of 2FA for Fedora's 114 proven packagers is a small step in adding layers of security to the project, but it is progress nonetheless. By deciding not to make perfect the enemy of the good, the project might save itself and its users some headaches down the road. It may also ensure that the project is able to work out the rough spots and eventually require 2FA for all packagers.

Comments (39 posted)

The first half of the 7.2 merge window

By Jonathan Corbet
June 18, 2026
The 7.2 merge window started with the 7.1 kernel release on June 14. As of this writing, just over 7,000 non-merge changesets have been pulled into the mainline for the next kernel release. Many of the core subsystems have been pulled at this point, meaning that most of the changes that can be expected in 7.2 have now come into focus.

The most significant changes merged so far include:

Architecture-specific

  • More i486-specific support has been removed from the kernel; among other things, this includes the removal over 13,000 lines of code needed to emulate the floating-point unit on processors that lacked one.
  • The AMD Geode processor, used in the OLPC XO-1 computer, has been marked as orphaned.
  • The Intel Trusted Domain Extensions (TDX) feature is implemented as a special software module loaded from flash memory. Changes in 7.2 include the introduction of a new device type for the management of this module and the ability to replace the module (to install security updates, for example) in a running system. This documentation patch contains some more information.
  • The s390 architecture has gained Rust support.
  • The arm64 kernel is being changed to remove its data regions from the all-of-memory linear map as a hardening tactic. While the necessary work has been done, the actual change to effect the removal has been reverted for now, pending a solution to a KVM regression.
  • PowerPC was the only architecture in the kernel with a hardware-optimized MD5 hash implementation. Given the weakness of MD5 in general, this code makes little sense and has been removed.

BPF

  • BPF programs attached to tracepoints are now allowed to fault, enabling them to reliably access user-space memory.
  • The bpf() system call has been extended with "common attributes" support. Initially, this feature is used to control logging for a number of BPF subcommands; see this merge commit for (some) more information.
  • The limit of five parameters to BPF functions and kfuncs has been lifted by allowing additional parameters to be placed on the stack; see this merge commit for more information.
  • It is now possible for kernel code to safely access BPF arenas without having to worry about page faults; see this commit for details.
  • BPF hash maps have a fixed number of buckets set at creation time. The new resizable hash map type removes that limitation; see this merge commit for more information.
  • It is now possible to quickly attach the same BPF program to multiple tracepoints. Documentation for this feature is absent, but the patch series includes a number of self tests showing how the feature can be used.

Core kernel

  • The implementation of /proc/interrupts has been reworked for better performance; see this merge commit for more information. "/proc/interrupts was subject to micro optimizations for a long time, but most of the low hanging fruit was left on the table".
  • The cache-aware load-balancing patch series has been merged. With these changes, the scheduler tries to group processes that share resources into the same cache domain, with, hopefully, an increase in performance.
  • The addition of support for sub-schedulers in sched_ext continues; see this merge commit for the current status.

Filesystems and block I/O

  • Filesystems can now provide information about their case-sensitivity via the file_getattr() system call. Two new flags are implemented: FS_XFLAG_CASEFOLD indicates that file-name lookups are case-insensitive, while FS_XFLAG_CASENONPRESERVING indicates that the specific case of new file names will not be preserved. Among other things, this feature supports Windows NFS clients, which expect case-insensitive behavior.
  • The openat2() system call supports a new flag called O_EMPTYPATH. It exists for a specific purpose: to allow a process to reopen an O_PATH file descriptor to gain access to the file behind it. With O_EMPTYPATH, an empty path name can be given to openat2(), and the given file descriptor will be used to find the file to (re)open.
  • Also new to openat2() is the OPENAT2_REGULAR flag, which will cause an open to fail if the target is not a regular file. The return code in the failure case is a new (to Linux) one: EFTYPE. This flag is meant to be a hardening feature to prevent programs from being manipulated into opening special files.
  • The XFS filesystem has had support for zoned storage devices since the 6.15 release; that feature is no longer marked experimental in 7.2.
  • Btrfs filesystems now use large folios by default. Support for "huge" folios (up to 2MB) has been added as an experimental option.
  • The new dm-inlinecrypt device-mapper target can support block devices with inline-encryption capability. Documentation can be found in Documentation/admin-guide/device-mapper/dm-inlinecrypt.rst.

Hardware support

  • GPIO and pin control: Waveshare DSI-panel GPIO controllers.
  • Graphics: Focaltech OTA7290B panels, Novatek NT35532-based DSI video mode panels, and CHIPWEALTH CH13726A-based DSI panels. Meanwhile, the driver for the venerable Hercules ISA graphics card has been removed after probably not having worked for decades.
  • Hardware monitoring: MPS MP2985 dual loop digital multi-phase controllers, Luxshare LX1308 DC/DC power modules, Analog Devices MAX20830 step-down DC-DC switching regulators, Analog Devices MAX20860A step-down converters, Analog Devices LTC4283 negative voltage hot swap controllers, Delta E50SN12051 power modules, ARCTIC fan controllers, Murata D1U74T power supplies, and Microchip Technology EMC181X/33 multichannel low-voltage remote diode sensors.
  • Miscellaneous: ASPEED AST2700 interrupt controllers, Qualcomm IPQ6018 pulse-width modulators, Loongson-2 Fast Speed I2C adapters, SG Micro SGM3804 voltage regulators, Spacemit K1 SPI controllers, Qualcomm Gunyah watchdog timers, and Andes ATCWDT200 watchdog timers.
  • Networking: Alibaba Elastic Ethernet adapters, NXP NETC Ethernet switches, Airoha AN8801 Gigabit PHYs, and Realtek 8922AU USB wireless network adapters.
  • Sound: Texas Instruments TAS675x quad-channel audio amplifiers, Everest Semi ES9356 codecs, and Cirrus Logic CS42448/CS42888 codecs.

Miscellaneous

Networking

  • The implementation of the TCP authentication option, which was added to the kernel for the 6.7 release, has been changed to use the new libcrypto library, making the code simpler and more efficient. Support for a number of (presumably) unused algorithms has been removed. See this merge commit and this documentation patch for more information.
  • The number of subflows supported for multi-path TCP connections has been increased from eight to 64.
  • Work to improve the scalability of the networking stack by reducing use of the rtnl_lock ("the big networking lock") continues.
  • Removals in the network subsystem include support for ISA and PCMCIA-based ARCnet interfaces, PCMCIA-based Bluetooth adapters, Chelsea inline TLS accelerators, more ATM support (see this merge commit for details), and the AppleTalk protocol.

Security-related

  • The slab allocator has gained the ability to use Clang allocation tokens to separate the placement of different types of allocated objects. That makes it harder to exploit a buffer overflow for one type of object to corrupt objects of a different type. See this commit for more information.
  • The AF_ALG mechanism, which allows the kernel's cryptographic layer to offload operations to special-purpose hardware, has been implicated in a number of recent security problems. Given that there is rarely reason to use this feature in the first place, it has been deprecated with an eye toward a future removal. Meanwhile, support for hardware accelerators has been removed, leaving only software implementations available. A number of drivers for random-number generators using the obsolete rng_alg framework have also been removed.

Internal kernel changes

  • There is a new bh_submit() function to submit a buffer head for I/O, and the b_end_io callback pointer has been removed. This change improves both performance and security; see this merge message for details.
  • The iomap layer has gained support for fs-verity. This feature was added without the accompanying documentation updates, which must certainly be coming soon.
  • There is new documentation for aspiring filesystem developers: Documentation/filesystems/adding-new-filesystems.rst gives a set of considerations and requirements for the addition of new filesystems to the kernel.
  • The minimum version of LLVM needed to build the kernel is now 17.0.1.
  • The new, rigorously undocumented kconfig-sym-check build target will check Kconfig files for references to symbols that are never defined.
  • The Rust "zerocopy" crate has been brought into the kernel source. This crate provides low-cost memory-manipulation primitives, with the intent of isolating unsafe code. See this merge commit for more information.
  • The new "higher-ranked lifetime types" for Rust code better encapsulate the connection between driver lifetimes and the devices they are bound to; see this merge commit for more information.
  • The nolibc library now supports the OpenRISC and 32-bit PA-RISC architectures; it has also gained alloca(), assert(), creat(), and ftruncate() implementations.
  • There is a new error-injection framework for testing at the block-layer level; see Documentation/block/error-injection.rst for some more information.

The 7.2 merge window can be expected to remain open through June 28. There are just over 6,000 changesets sitting in linux-next, meaning that there is still a fair amount of code waiting to move into the mainline. As always, LWN will post a summary of what that code contains after the merge window closes.

Comments (none posted)

A helper library for BPF arenas

By Daroc Alden
June 24, 2026
LSFMM+BPF

BPF arenas are areas of memory (potentially shared with user space) where programs have free reign to build their own data structures, unburdened by the verifier's bounds checks. Many of those data structures are potentially usable in multiple programs. Emil Tsalapatis brought his work on libarena, a library containing generic utilities for use in BPF arenas, to the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. Although the library is already available as part of the kernel, it is still in its early stages and he has more work planned.

Tsalapatis works on sched_ext, a project that has seen him write a number of components based on BPF arenas that he believes could be reused. In particular, he sees potential for having a universally agreed-upon memory allocator for arenas, and a set of common data structures that use it. He also wants to see about adding better debugging capabilities to arenas; arenas may obviate the need to fight with the verifier, but that also means that there can be ordinary memory-safety problems that, while not a threat to the kernel, still need to be found and fixed.

[Emil Tsalapatis]

There are downsides to using arenas for everything, Tsalapatis admitted. Since BPF programs can write to arenas at any time, unchecked, the verifier cannot trust pointers to kernel objects stored in an arena to remain valid. Therefore, the verifier bans storing pointers to kernel objects in arenas at all; those pointers still need to be stored in other types of BPF map. Even with that caveat, he sees focusing on arenas as being worth the hassle. The end goal, he says, is to make BPF C as easy to write as normal C, and arenas are a big part of that.

He envisions libarena as providing a standard library for BPF programs. It would be plain C code that is compiled and linked into BPF programs in the normal way, not an additional kernel interface to maintain. That also means it is optional — small BPF programs would not need to link against it unless they used its features. The library source code would live directly in the kernel tree, and be tested against the verifier to make sure the algorithms and data structures it provides do verify. Since libarena corresponds to a kernel version, it can also make use of the latest verifier features without worry.

One member of the audience questioned how libarena would support migrating programs across kernel versions. BPF programs that use kernel interfaces can use "compile once — run everywhere" (CO-RE) relocations for that purpose, but these wouldn't work with code that is actually linked into BPF programs. Tsalapatis said that libarena wouldn't be supported on kernel versions prior to its introduction, but that with some care it could be made forward-compatible, so that people can build against the libarena version from the oldest kernel version that they wish to target, with no need for CO-RE relocations.

The questioner was doubtful; they are currently trying to fix a problem where a program verifies on the 6.9 kernel but not the 6.19 kernel, so forward compatibility is a real problem. Perhaps programs could link in the version of libarena corresponding to the version of the kernel in use, Tsalapatis suggested. The hope is to have a stable enough interface that linking different versions for different kernels should not cause problems.

He then explained how he envisioned programmers making use of the library. The source lives in the kernel tree, but he wants to periodically export it to a separate Git repository so that users can add it as a self-contained Git submodule, the same way that libbpf works. The build system would call into libarena's makefile to produce libarena.bpf.o. The main BPF program would include libarena's headers and link against the object.

Another audience member asked whether this would require old kernels to be added to BPF's continuous-integration testing, but Tsalapatis reiterated that backward compatibility was not a priority. Alexei Starovoitov suggested worrying about that for the 7.5 kernel, noting that libarena was in the early stages of development.

As of the 7.2 kernel, libarena will be available for experimental use. It contains a buddy allocator and some basic scaffolding for future work, but no actual data-structure implementations yet. It also has the hooks needed to work with Clang's address sanitizer, ASAN, although only with the in-development version of Clang. The allocator is usable, handling all sizes of allocation, but not particularly fast yet. Tsalapatis found writing the allocator to be a good case study for arenas, uncovering a lot of edge cases that needed to be addressed.

In the future, libarena's allocator might end up using the design of mimalloc instead of a plain buddy allocator. Mimalloc was designed for use with functional-language runtimes, and performs well with multithreading and short-lived allocations, two features that Tsalapatis expects to be important for performance in the BPF programs of the future. It's also a simple and efficient design, he said.

Previously, he had experimented with using a slab allocator for its reduced fragmentation, but he found it clunky for arbitrary programs. For sched_ext, it worked well because there were only "like three types" of structure needed, but that's not true of typical programs.

In the future, Tsalapatis would like to offer red-black trees, B-trees, bitmaps, and Lev-Chase queues (also known as Chase-Lev queues) as part of libarena at a minimum. Even if people start using large language models to write BPF programs, he thought there was value in having manually coded data structures in libarena that worked with the verifier. "In my experience, I've found that seeding agents with good code lets them one-shot hard problems."

He then went over what the ASAN hooks do. When an arena is created, pages are mapped lazily — touching a new page causes a page fault. The BPF program can listen for these events and use them to also populate a metadata region containing ASAN's information about which areas of memory are okay to access. Allocating a section of memory marks it as safe with an eight-byte granularity, and freeing it removes the mark. The compiler augments pointer dereferences to check the state of the metadata before loading or storing a value. The work needs the in-development version of Clang because the compiler previously assumed that the metadata memory was always in address space zero, but BPF arenas use address space one, so Tsalapatis had to add a flag for that. (Clang numbers the available address spaces, while GCC names them.)

One audience member asked whether Tsalapatis had considered including code in libarena to navigate some of the kernel data structures that are difficult to navigate correctly. He had considered it, he said, but thought that it was better to keep the scope of libarena small for now, and focus on providing the core things that every program would need. At that point, time for the session ran out, but work on libarena has continued since the talk, with some of the planned-on data structures now available to use.

Comments (4 posted)

Suspending and resuming BPF programs

By Daroc Alden
June 19, 2026
LSFMM+BPF

BPF programs can be used to extend many aspects the Linux kernel, but BPF programs must run to completion in the same context that they began. Kumar Kartikeya Dwivedi is working on changing that by allowing BPF programs to be expressed as coroutines. He spoke about his work at the 2026 Linux Storage, Filesystem, Memory-Management and BPF Summit. While still experimental, the change promises to make long-running BPF tasks significantly easier to write.

Frequently, a single logical task is spread across both space and time, he explained. Execution jumps between different locations, computations can be suspended, and so on. Being able to express this in BPF would make some kinds of extensions to kernel functionality much easier to write. For example, consider the task of collecting stack traces. The kernel has tracing facilities to gather combined stack traces for kernel and user-space code. It's more efficient to collect the user-space portion of the stack trace right before the kernel performs a context switch back to user space. When a stack trace is requested, the kernel part runs immediately, then the computation is suspended, and the user-space part of the collection runs later. Adding BPF into this workflow requires splitting a single logical operation across multiple independent functions, because there is currently no way to suspend an executing BPF program.

Similar problems come up when implementing user-space networking. Attempting to send a packet with sendmsg() eventually makes a call to qdisc_run(), which can also do work for other threads, sending something that they queued. This is another case of follow-up work being done at a different time and place than the main task. People have experimented with writing whole applications in BPF, Dwivedi said, and that turns up even more of the same kind of problem.

[Kumar Kartikeya Dwivedi]

Historically, BPF programs have used hooks and callbacks. There's nothing wrong with that approach; it can express all of the semantics that kernel developers need. It does force programmers to write custom suspend/resume logic and break their program up over multiple functions, however. Dwivedi's solution is to introduce coroutines to BPF: functions that can be suspended, transfer control back to the kernel, and then be resumed later. He then went over his design for BPF coroutines in the kernel.

The best solution would be stackless coroutines, akin to Rust's asynchronous functions or C++'s coroutines, he said. In these languages, the compiler is responsible for rewriting straight-line code into a form that can be suspended and resumed. The advantage is that, from the BPF verifier's point of view, not much would need to change. Resuming a coroutine would look like a normal indirect function call, and the compiler would handle the details of saving and loading intermediate values. The verifier would still need to ensure that the overall program's control flow is valid, and may need some changes to handle code that is split across kernel contexts.

Dwivedi went into a bit of detail about how C++ implements coroutines to illustrate the interface that the verifier would see. In short, he said, C++ creates a structure that contains two function pointers that can be used to resume() or destroy() the coroutine. resume() uses an index recording where the coroutine last suspended, stored in that same structure, to choose which code to execute using a switch statement. Any variables that need to be saved across suspension points are stored in the same structure; any that don't need to be saved across suspension points remain on the stack, and are implicitly discarded when the task is suspended. The verifier's normal check that kernel resources are freed instead of just forgotten about prevents BPF programs from doing that with any locks or reference-counted structures. The two function pointers are always the first two elements of the coroutine structure, so that generic code doesn't need to know how big the coroutine's frame is or how it is laid out.

This is pretty similar to how Rust compiles asynchronous functions, except that it uses its type system to convey the resume() and destroy() callbacks. The Rust compiler can be easily coerced to produce a structure with the same layout as C++, however, so the verifier will probably only need to handle that one layout. From the point of view of tracking the types of BPF values, the coroutine's associated structure can be handled in the same way as the stack: values are spilled to it and loaded from it, but it doesn't need special handling.

The things the verifier will have to do to make sure a coroutine is safe, beyond verifying the constraints that apply to all BPF code, include making sure the resume() and destroy() function pointers are not overwritten, making sure that the index only takes on valid values, and making sure that it's always legal to call destroy() when the coroutine is suspended. The verifier will also need to check that locks aren't held across suspension points, but that check can reuse the verifier's logic for determining that locks are released before a function returns. By the same logic, the verifier may need to invalidate map values held across suspension points, depending on their types.

Andrii Nakryiko asked how the verifier is supposed to ensure that the coroutine doesn't enter an infinite loop, perhaps by setting the index back to an earlier value. The body of the coroutine is still verified, and at every suspension point the verifier checks the call to resume(), so the verifier can identify any loops in the same way that it finds existing loops, Dwivedi said. Nakryiko asked about subtler loops, but Dwivedi pointed out that it was already possible to have two BPF programs arm each other's timers, and that such infinite loops don't cause an actual problem as long as they don't cause the kernel to become unavailable. For that matter, even with the verifier limiting BPF programs to less than one million verified instructions, there's nothing preventing a user from attaching a 999,999 instruction BPF program that makes lots of expensive kfunc calls to every available kernel hook and slowing the system to a crawl. The point of the limit is to prevent infinite loops that might cause deadlocks, so that the system can continue to make forward progress, not to prevent BPF programs from wasting CPU time.

As long as the verifier checks that, from every suspended state, it is always valid to destroy() the BPF program (in case it is unloaded) and resume() it, the program can't subvert BPF's safety guarantees. Even if there is a clever way to set up an infinite loop, it still won't be able to deadlock the kernel, since every time the coroutine is suspended it will have to release any held locks and give control back to the kernel.

Dwivedi has a prototype implementation in progress, but there is still more work to be done. He wants to extend the BTF debugging information for programs that use coroutines, for example. His prototype also had to enable aggregate return types from functions to make his test C++ programs work correctly, so that will need verifier support. Adding Rust support should not be too much more difficult.

A more experimental idea for the future is to allow suspended computations to switch between user space and kernel space. Dwivedi had one of his students work on a prototype for that, but it is definitely not ready yet. If that ever does work, it will enable applications to perform setup in user space, then transition to the kernel and use native BPF capabilities, before potentially switching back to user space. That kind of interface would blur the distinction between user space and the kernel — but it will be a long time before it becomes a reality, if it ever does.

In the nearer term, Dwivedi intends to polish up his current work to prepare it for the kernel. While incorporating coroutines into BPF will not technically enable anything new, his hope is that it will make it easier to integrate BPF programs into the parts of the kernel that can't easily be simplified down to a single hook or callback.

Comments (9 posted)

KASAN for JIT-compiled BPF code

By Daroc Alden
June 23, 2026
LSFMM+BPF

Alexis Lothoré has been working to add support for the kernel's memory-access checker, KASAN, to just-in-time-compiled BPF code. He spoke about that work at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. KASAN support is needed, he said, to help catch bugs in the BPF just-in-time (JIT) compiler. KASAN is a great tool for catching memory-management problems in the kernel, but only in code that can be monitored by it.

KASAN can identify both use-after-free bugs and out-of-bounds accesses, Lothoré said, using either software or hardware memory tagging, depending on what the hardware supports. The generic software implementation reserves a section of memory to act as a bitmap tracking whether accesses to each byte of main memory are permitted. At build time, the compiler augments all of the pointer dereferences in the kernel with calls to special __asan*() functions that check whether the referenced memory is in the right state. The kernel's various allocators are hooked to update the state of the bitmap, and to insert "red zones" before and after allocations that make it easier to catch buffer overruns.

[Alexis Lothoré]

Previously, the BPF JIT emitted pointer dereferences directly, with no calls to the appropriate __asan*() functions. So, adding support for KASAN should be as simple as patching the JIT to emit those calls where appropriate. That picture is slightly complicated by the fact that BPF programs can access different areas of memory than the main kernel code, including BPF maps, arenas, global variables, and stack memory. In particular, the BPF stack is currently treated as one big memory area, but it could potentially be modified to have red zones between each variable, to help KASAN detect incorrect accesses in more detail, Lothoré said.

He has been working on a patch to add KASAN support for JIT code since the beginning of the year, on x86 to begin with. For now, he has been focusing on the LDX and STX BPF instructions, which are basic loads and stores, respectively, even though there are other instructions that can also access memory. His code also ignores loads and stores targeting the stack for the time being. Even with only those two instructions targeted, the overhead is still pretty severe. His patch essentially turns one instruction into twelve — but Lothoré believes that can be simplified with a bit of effort.

The bulk of the overhead comes from saving and restoring registers, which José Marchesi observed may not be needed if the __asan*() functions could be inlined. Lothoré agreed that it could help in principle, but pointed out that compilers are much smarter than his code, and naive inlining would be worse. A member of the audience asked why Lothoré was doing this in the JIT, instead of adding KASAN instrumentation in the compiler or the verifier. He explained that this would require exposing a stable KASAN API, and result in BPF programs needing to be compiled differently for kernels that enable KASAN, which would be an additional pain point when setting up testing with KASAN.

Another person wanted to know whether using KASAN with the BPF JIT had turned up any bugs so far. Not yet, Lothoré said, although that is mainly because his patch hadn't landed yet, and therefore it had not been run in the large number of environments in which the kernel is tested with KASAN. The instrumentation does work, though, because KASAN can generate a report that covers BPF code.

Someone else wanted to know how to tie a KASAN report back to the BPF program that caused it. The KASAN error message will include the BPF function name, Lothoré said. Alexei Starovoitov added that if the program was compiled with BTF, you should also be able to see the file name, line number, and so on. Cupertino Miranda asked where the __asan*() functions were defined; Lothoré explained that they are part of the kernel, not part of the compiler's user-space address sanitizer.

The next steps include expanding the number of BPF instructions that his code handles, particularly atomic instructions. He wasn't sure whether those should count as a KASAN load, store, or a new kind of operation. Starovoitov opined that the atomics are unlikely to be used in a way that conceals a verifier bug, and that complete coverage of every case was unlikely to be worth it, compared to just getting coverage of basic loads and stores. Despite the limited nature of the current patch set, Lothoré's work was met with general approval by the assembled developers; it may not be long before KASAN is available for BPF code as well.

Comments (none posted)

Single-hop block replication with RMR and BRMR

June 18, 2026

This article was contributed by Haris Iqbal

How can cloud providers efficiently supply durable virtual block devices? Remote Direct Memory Access (RDMA) provides a way for servers in a cluster to share chunks of memory, but there still needs to be a protocol that operates on top of RDMA to provide the guarantees expected of a block device. The kernel's RDMA transport library (RTRS) provides a way to send messages via RDMA. I presented about two new components built on top of RTRS at the 2026 Linux Storage, Filesystem, Memory Management and BPF Summit: Reliable Multicast over RTRS (RMR) and Block device over RMR (BRMR). These modules, which I am working on with Jia Li, could be a way for cloud providers to expose durable block devices with as little overhead as possible. To accomplish that, however, we need some discussion and feedback from the community before sending the modules upstream.

The problem

The patches are aimed at cloud providers, which generally have a compute tier and a separate storage tier connected by an RDMA fabric. To keep the service stable, a block device on a compute host typically has to be replicated synchronously to at least two storage nodes, preferably using only a single network hop for each to ensure good performance. Failures of all of the usual kinds (a network interruption, a disk replacement, a storage-node reboot, or a full crash of the compute client) must be recovered from without operator intervention. Of course, the patches are also useful for non-cloud-provider organizations that need this kind of robust block-level replication.

CPU and RAM on a compute host are revenue generators. Every cycle spent on replication or resynchronization is a cycle that cannot be sold to a customer. So the design constraint we work under is: keep the compute host out of the resynchronization data path.

The two existing in-tree options each fail one of those constraints. The Distributed Replicated Block Device's (DRBD) conventional configuration has writes go through a designated primary: a two-hop write path. MD-RAID 1 (the kernel's multiple-device RAID at level 1) on top of a network block device gives single-hop writes, but resynchronization of a degraded leg has to read from the healthy leg on the compute host and write that data to the recovering leg. A "leg" is a replica copy that writes are fanned out to, one per storage node in the replication set. This is two hops again, and the resynchronization bandwidth uses up some of the compute host's budget.

Architecture in two modules

RMR is the transport. Its client-side API takes a scatter-gather list and the pool of connections that designates which legs to send the data in the list to. RMR does not know anything about the client using it — in this case the block devices. BRMR is a thin block-device adapter on top: it exposes /dev/brmrX via the kernel's block multi-queue (blk-mq) layer and owns the on-disk pool metadata. RMR has no on-disk format of its own.

Each module has a client/server split. The compute host runs the RMR and BRMR clients. Each storage node runs an RMR server, a BRMR server, and an RMR client connected to its storage peers. That RMR client residing on storage servers lets recovery traffic flow storage-to-storage on a dedicated RTRS session, single hop, without touching the compute host.

A pool is the unit of replication. One BRMR device maps to one RMR pool with N client-server sessions, one per storage node. Each storage node is identified within the pool by a member ID. Sessions can be added, removed, disassembled, and replaced at run time through sysfs. The state of the session indicates whether it is NORMAL (which means healthy), FAILED, or RECONNECTING. There is no "primary" leg. In steady state, every NORMAL session serves reads round-robin and accepts writes via multicast fan-out from the client.

The dirty map

In order to track I/O operations that have been missed due to a network outage or failure, RMR uses a dirty map that contains entries for all storage members in a pool. Each storage member holds a copy of the dirty map. I/O operations missed by a storage member are recorded on the dirty map held by the other available storage members. When a storage member returns from any kind of failure, as part of the reconnection process, it receives the latest dirty map from other storage members. It can then figure out what needs to be resynchronized.

RMR's dirty map contains a two-level page table. A first-level page (FLP) is an array of pointers to second-level pages (SLPs). Each SLP is one page and holds 4096 one-byte entries. Of that byte, one bit is the dirty bit, one is a "syncing" filter bit, and six are reserved for future flags.

The byte-per-chunk layout was chosen because of the need to track map entries that are added by the client module in special cases, like when a storage back-end is replaced. Currently, the back-end replace feature is disabled since it needs some work. Once it is enabled, the necessity of byte-per-chunk needs to be reviewed again. We may move to a bit-per-chunk design later.

Up to 256 FLPs of 512 entries each point to 4096-byte SLPs, which lets the map track just over half a billion chunks. At the maximum chunk size of 1MB, that is 512TB per pool. When fully populated, the map uses 512MB of memory for this metadata. Chunk size is a power of two from 128KB to 1MB, so per-pool capacity ranges from 64TB to 512TB and calculating which chunk an address falls in can be done with a single shift.

The map is more than a bitmap. Alongside the dirty bits, each per-member map also carries an XArray of active sync entries, with one entry for each dirty chunk that is currently being acted on. Each entry holds a reference count and a wait list. When a sync is in flight for a chunk, an incoming I/O operation that lands on that chunk parks on the wait list and is woken once the sync completes. This design lets a chunk be resynchronized while serving I/O operations, with no global locking or cross-node concurrency coordination.

Every time a session transitions from NORMAL to FAILED, it bumps map_ver, a 64-bit monotonic version number. In-flight I/O operations carry the version number, so, during recovery, the surviving node with the highest map_ver is the authoritative source. If there are ties, the two nodes should have the same map, so one is picked arbitrarily. The current code treats the high bit of map_ver as a "replace" flag for the back-end replace feature.

Data path

Writes from the compute client are issued in parallel to every session in NORMAL state. An I/O operation is acknowledged to the upper layer only after all the sessions have acknowledged it; be it a success or a failure. When a write fails on one member, every surviving member of the pool sets the affected chunks dirty in its own copy of that member's map. This is the map_add protocol operation.

Once a session has gone non-NORMAL, every subsequent write to the remaining available members carries the failed member's ID along with the data payload. The members receiving the write then add this information into their copies of the dirty map. That is the map_add piggyback. An explicit map_add remote procedure call (RPC) is only needed for the in-flight I/O operations that were already on the wire when the session failed.

Reads are handled in a round-robin fashion across NORMAL sessions, so steady-state read throughput scales with the number of replicas. On the server side, an incoming I/O operation checks the local dirty map for the target chunk. If the chunk is clean, the I/O operation goes straight to the backing store.

The dirty case is more involved. The server marks the chunk as syncing and issues a synchronous read from a peer over its storage-side RMR client. Concurrent I/O operations to the same chunk park on a wait list and are woken when the sync completes.

A background sync thread on the recovering server pool does the same thing in bulk. The number of parallel operations it performs is bounded by sync_queue_depth, a module parameter that defaults to 32 and is capped by the pool's RTRS permit count (the number of in-flight RDMA operations RTRS preallocates per session, which defaults to 512).

Failure recovery

The talk covered four failure cases. The first three are variants of a single flow. A network or local I/O failure moves the affected session to FAILED. Dirty bits propagate via the map_add piggyback. When the network recovers, a link event is generated, which causes the session to reconnect (and transition to the RECONNECTING state). The recovery worker then probes the session to check if the store can still serve I/O operations, and, on success, the session goes through the map-update process where its dirty map is repopulated from a NORMAL peer, before returning to NORMAL.

A storage-node crash followed by reboot is the same flow as above. When the storage node comes back up, since the RMR metadata is present in the backing store, it expects a join/rejoin message. It then goes through the map-update process before returning to NORMAL state. A maintenance-mode entry is the operator-driven version of the same path, and is intended to be used for, well, maintenance purposes.

The fourth case is the compute client crashing mid-write. When the client dies, some legs may have received a given write and others may not, and the client itself can no longer tell the survivors which chunks were in flight. If left as is, then the storage nodes that received the write and wrote it to their backing stores would hold different data than the ones that did not receive this write. This is where last_io comes in.

Each storage node keeps a last_io array of length queue_depth, indexed by the I/O request's mem_id (the slot index the request holds in the queue, in the range [0, queue_depth-1]). Every successful write records its chunk ID at that slot, so at any moment each storage node holds a record of the most recent queue_depth chunks it acknowledged. If the same chunk appears in multiple nodes' last_io, only the first node processed wins. The remaining nodes find it already dirty and skip it.

There is one weakness here. If the compute client crashes mid-write and a storage node also crashes before recovery completes, the in-memory last_io of that storage node is lost, and the inconsistency is no longer detectable. This is the same situation as if there was no last_io being recorded. The mitigation is to write last_io to the backing store. BRMR's persistence path exists in code, but finding the proper consistency guarantee is still on the to-do list.

An open question is whether last_io should be written atomically with the data write, or as a follow-up metadata write. That choice has real performance implications. An audience member observed that the answer depends on the crash-consistency contract RMR/BRMR makes to the layer above it.

Why a new module?

Why not just extend DRBD or use MD-RAID 1? This was the first question raised on the mailing list when the LSFMM proposal went out. DRBD's resource and role model assumes that the writer is one of the data copies, and that one peer is the serializer. Those assumptions sit deep in DRBD's request path, state machine, and on-disk format. MD-RAID 1 over a network block device has single-hop writes, but pays for it with two-hop resynchronization through the compute host.

The multicast transport RMR by itself is reusable beyond block storage. A key-value store or an S3-style object store on the same fabric would want the same delivery guarantees. Keeping RMR free of block-layer assumptions preserves that option.

DRBD 9: a wrinkle during the conference

On the mailing list, Philipp Reisner pointed out that DRBD 9 already supports a configuration that looks very close to RMR. In DRBD's terms, it is a diskless primary connected to multiple storage nodes, with the networking abstracted into pluggable transport modules: TCP, load-balanced multipath TCP, and RDMA. The diskless primary fans writes out to the storage peers in parallel on the mesh, which is one hop on the write path. DRBD 9's mesh connections also let one secondary resynchronize from another secondary directly, bypassing the primary on the recovery path.

So the two properties RMR guarantees, single-hop writes from a writer without a local copy and peer-to-peer resynchronization without the writer in the loop, both look available in DRBD 9. A comparison run against DRBD 9's diskless-primary mode is planned as a followup. The configuration only surfaced on the mailing list during the conference, and the relevant code has not been shared by its developers at LINBIT. A pre-talk evaluation was not possible.

Whether RMR should stay separate, or whether the right move is to lift more of DRBD 9's machinery and adopt its on-disk format, is the question the comparison will have to answer. There is also the option to move entirely to DRBD 9, but that is only viable if DRBD 9's performance matches what MD-RAID 1 over a network block device and RMR/BRMR deliver today.

Above the transport, not through it

Leon Romanovsky raised two architectural points with me during the conference, and later wrote them up on the mailing list. The first was a layering point. Replication and dirty-tracking are not RDMA-specific concerns. The block layer has had them for decades in MD-RAID, device-mapper mirror (DM-mirror), and DRBD. A new replication engine should therefore live above the transport, with RDMA as one possible substrate. A user should be able to swap RTRS for TCP, nvme-tcp, or anything else without rewriting the replication logic. RMR, by living in the RDMA stack, looks transport-coupled even if the code itself avoids block-layer assumptions.

The current code honors part of that split. RMR proper has no block knowledge, and BRMR is the layer that handles blk-mq and the on-disk format. But the dirty map, the state machine, and the recovery worker all live inside the RMR client and are reachable only through it.

Pushing dirty-tracking and the multi-leg state machine above the transport layer can be done in multiple ways. The simplest is to lift those pieces into a data-replication module in the block layer, and leave the RTRS bindings as a thin shim around it.

The second point was about the kernel review process. A kernel module with a single in-tree consumer is a hard sell. RMR should be able to name at least one other prospective in-kernel user, and ideally accommodate it from the start. Li and I hope the work attracts the Linux community and the industry, and that someone else finds a use for it.

The original design intent for RMR-as-transport was that the same reliable multicast layer should be usable under a key-value store or an object store on the same fabric. But no second consumer is in development anywhere that we know of.

Where the work goes from here

There are a few open threads. We would like to address the last_io persistence question, pursue Romanovsky's suggestion of a layering refactor, and perform a comparison against DRBD 9's diskless-primary mode. We will post the results on the mailing list as a follow-up.

The code is GPL-2.0-or-later and under active development. Give it a try if the project interests you. An easier VM-based setup will be posted on the documentation site soon. Contributions are welcome. Feel free to reach out to us at haris.iqbal@ionos.com and jia.li@ionos.com if you have thoughts, questions, or just want to find out more about BRMR/RMR.

Comments (7 posted)

Reports from OSPM 2026, day one

By Jonathan Corbet
June 22, 2026
OSPM
The Power Management and Scheduling in the Linux Kernel Summit, which still goes by the historical acronym OSPM, was held in Cambridge, UK, in mid-April. As has become traditional, the presenters at that event have since written summaries of their sessions, and this work has kindly been made available to LWN for publication. The first day's sessions covered a wide range of topics, including idle-state selection, user-space schedulers with sched_ext, lock-holder preemption, and much more.

(See also: coverage from day 2 and day 3.)

Success criteria for CPU idle state selection — Rafael J. Wysocki

In the first session of the conference, cpuidle subsystem maintainer Rafael Wysocki discussed issues related to measuring the quality of decisions made by cpuidle governors.

He started by recapitulating the design and purpose of the Linux kernel's CPU-idle-time-management code, which is to do nothing efficiently. It is based on the observation that, if a logical CPU, which can be a symmetric multi-threading (SMT) thread or a core if SMT is not enabled, becomes idle (that is, there are no tasks to run on it), there is an opportunity to stop it. That will reduce the processor's power consumption, which generally allows some energy to be saved. Of course, that requires hardware support, but the majority of modern processors provide it.

The CPU scheduler directs idle CPUs to execute a special piece of code, called the idle loop, whose role is to utilize processor capabilities to reduce power, if possible. For this purpose, the idle loop invokes the cpuidle subsystem in every iteration. That subsystem consists of three parts: the core that coordinates operations and provides an interface to user space, the driver that uses the processor interface (platform-specific in the majority of cases) to stop the given CPU and reduce the processor power, and the governor responsible for deciding how far the driver can go with the processor power reduction.

Each iteration of the idle loop first checks if the CPU still has no tasks to run and if so, it invokes the cpuidle governor to make its decision, which includes deciding whether to stop the scheduler tick on the given CPU to prevent it from being woken up unnecessarily. Next, it calls into the driver to change the processor state in accordance with the decision made by the governor. The CPU running that code, which is idle from the scheduler's perspective, is stopped and the processor power can be reduced. Eventually, the CPU receives a wakeup event (for example, an interrupt) and it goes back to the scheduler or enters the next iteration of the idle loop.

The decisions made by the cpuidle governor are based on the information about the processor's capabilities, which is supplied by the cpuidle driver in the form of a list of so-called "idle states". Each of those states represents a configuration with reduced power that the processor can be put into after stopping the given CPU. States are characterized by two parameters: the target residency (the minimum time to spend in the given idle state for which selecting it makes sense) and the exit latency (the worst-case time the processor will take to allow the CPU to execute instructions again). The list of idle states supplied by the driver is sorted by both the target residency and exit latency in increasing order. The idle states with lower values of those parameters are referred to as "shallower", whereas the idle states following them in the list order are referred to as "deeper".

The problem is that the governor does not actually know how long the CPU will be idle, but it needs to select an idle state to match the upcoming period of CPU idleness, which is referred to as the idle duration. Doing that accurately every time would require a crystal ball, the supply of which is quite limited. So the governor has to resort to using statistics, and its decisions will never be perfect, though the existing governors strive to make accurate decisions in the majority of cases. The question is then which side it is better to err on, the "shallower" one, or the "deeper" one.

Clearly, if the target residency of the selected idle state exceeds the idle duration (that is, an overly deep idle state is selected), it will hurt both energy efficiency and performance, because more energy could be saved by using a shallower idle state and the latency introduced into the workload would be lower. On the other hand, if the target residency of the selected idle state is too short, it will, again, hurt energy efficiency, because selecting a deeper idle state would allow more energy to be saved, but performance will not suffer. In theory, choosing a shallower state should even improve performance because the exit latency of the shallower idle state is lower than the exit latency of the deeper one.

This seems to indicate that erring on the shallower side is slightly better and that, according to Wysocki, was the consensus in 2019 when that topic was last discussed at the OSPM Summit. However, more recently, it turned out that there were systems where selecting overly shallow idle states also hurt performance.

The exact mechanism leading to that result on the systems in question (Chromebooks based on exotic x86 SoCs) has not been identified precisely. It has been observed, though, that using deep idle states on them allows the frequency of non-idle CPUs to go higher than when shallow idle states are used, and the busy CPUs are power-constrained. This most likely means that the power reduction resulting from idle states does not actually lower the overall energy usage of the system; instead, it allows the energy saved by idle CPUs to be utilized to crank up the frequency of the CPUs that are still executing instructions. In other words, what was designed as an energy-efficiency optimization has become a power-distribution mechanism, which had never been anticipated.

With hindsight, this should have been expected, though, because power-constrained systems are abundant. In fact, the vast majority of client platforms shipping today are power-constrained, mostly due to thermally challenging form factors, so this problem may actually be more widespread than it would seem.

This appears to pose a challenge to CPU-idle-time management in Linux; Wysocki admitted that he did not really know how to address it at the moment. Nevertheless, he thought that bringing it to everyone's attention would be useful.

Video: Success criteria for CPU idle state selection - Rafael J. Wysocki (OSPM26)

Asymmetric packing for energy-efficient scheduling on Intel processors — Ricardo Neri

At OSPM 2025, Wysocki presented an energy-aware-scheduling (EAS) implementation for Intel hybrid processors. His energy model is based on the observation that E-cores (slower CPUs focused on energy efficiency) are more efficient than P-cores (faster, performance-oriented CPUs) across a wide portion of the power-performance curve. A fundamental limitation is that EAS in its current form requires prospective operating-point computations to be carried out at wakeup time, which is incompatible with hardware-controlled performance scaling, the predominant frequency scaling mechanism on Intel platforms.

The presented alternative approach avoids per-wakeup cost computations and instead operates in the load balancer. It uses asymmetric task packing, giving E-cores higher placement priority while relying on the existing capacity-aware scheduling to keep heavy tasks on P-cores. The results from schbench showed comparable energy consumption to EAS with significantly lower 99th percentile latency.

Vincent Guittot noted that EAS latency stems from its packing behavior and can be mitigated by tuning the scheduler slice time.

Peter Zijlstra opposed introducing a parallel energy-efficiency mechanism, arguing instead for extending EAS. He suggested removing the incompatibility with hardware-controlled performance scaling, addressing packing-induced latency, and scaling to systems with large CPU counts.

Video: Asymmetric packing for energy-efficient scheduling on Intel processors - Ricardo Neri (OSPM26)

Rotating task scheduler in the Linux fair scheduling class — Pierre Gondois

Some workloads split their work into as many threads as there are CPUs on a platform. The workload completes only once all threads have finished. However, if the work partitioning is static, meaning each thread has the same amount of work to execute, and the platform is asymmetric (also known as heterogeneous multi-processing or HMP), the thread running on a big CPU will finish earlier than the one running on a little CPU. Some performance improvement could be had with a smarter load distribution.

This highlights a more generic issue in the Linux fair scheduler: forward progress is not monitored across CPUs. A similar issue exists on SMP systems; with three long-running tasks running on two CPUs, the two tasks sharing one CPU will progress more slowly than the third task, which runs alone on the other CPU.

The load balancer balances fair tasks between CPUs. For long-running tasks, load is analogous to the task's nice value: the lower that value, the higher the load, and the more running time a task should receive. By balancing the load between CPUs, tasks should receive approximately the amount of forward progress they deserve.

In the examples above (long-running tasks on HMP, and three long-running tasks on two SMP CPUs), the load balancer doesn't detect and cannot solve the imbalance. Indeed, balancing is done from a CPU point of view rather than from a task point of view.

The most accurate way to distribute forward progress fairly between tasks would be to use a global virtual run time. However, this solution does not scale. It would require a global red-black tree that would be accessed concurrently by all CPUs on the platform.

The approach presented was to reinstate the initial contribution-scaled per-entity load tracking (PELT) implementation. The current scale-invariant PELT implementation makes it possible to estimate the size of a task uniformly, independently of the capacity of the CPU doing the estimation. The old contribution-scaled PELT, instead, measures the amount of instruction throughput received by each task.

By adding another balancing mechanism that relies on the old throughput-based PELT implementation to estimate a forward-progress metric and swaps long-running tasks that don't progress at the same rate, the HMP-specific case was improved. The amount of time saved is platform specific. Unfortunately, the extra logic required, the HMP-specific aspect of the solution presented, and the fact that most multi-threaded workloads use dynamic partitioning make this solution unsuitable for upstream. Scientific workloads are better candidates for static partitioning, but are unlikely to run on HMP.

Video: Rotating task scheduler in Linux Fair scheduler class - Pierre Gondois (OSPM26)

What's missing in sched_ext? — Andrea Righi

Sched_ext (SCX) is a Linux kernel framework that allows the CPU scheduler to be replaced at run time using BPF programs. While SCX has matured quickly and is now widely deployable, several architectural and integration challenges remain under active discussion.

The long-troubled ops.dequeue() callback, broken until the 7.1 kernel release, has finally been fixed, giving BPF schedulers reliable visibility into when tasks are removed from their control by the core scheduler. That change is especially important for schedulers maintaining custom task queues or implementing their own accounting mechanisms in BPF.

Another major improvement is the introduction of a dedicated deadline server for SCX tasks. Previously, aggressive SCHED_FIFO/SCHED_RR or heavily loaded fair-class workloads could starve SCX entirely, often triggering watchdog failures blamed on the BPF scheduler itself. The new mechanism reserves CPU bandwidth for SCX tasks and also enables "partial mode", where SCX and fair-scheduler tasks can safely coexist on the same CPUs without static partitioning.

Some cleanup work is still needed around the deadline-server infrastructure itself. In particular, the kernel currently relies on statically configured bandwidth reservations made at boot time. The next step is to automatically register and unregister those reservations when an SCX scheduler is enabled or disabled. The discussion also touched on moving the debugfs interface used to configure values for the run time and period for the different deadline servers into sysfs (since debugfs may be unavailable on systems with secure boot or kernel lockdown). Zijlstra argued that it was too early to commit to a stable ABI before the FIFO control-group rework is better defined.

The discussion also covered future work around hierarchical scheduling. Initial support for per-control-group sub-schedulers which landed in the 7.1 release, allowing different scheduling policies to coexist across containers or workloads. The question was raised whether this hierarchy belongs in the kernel at all, with concerns that coordinating multiple schedulers with different latency and bandwidth constraints may ultimately require a single global scheduling model.

Integration with proxy execution remains unfinished as well. The next step is to resolve the remaining configuration conflicts, allowing distribution kernels to be built with both proxy execution and sched_ext support enabled simultaneously, removing the current mutual-exclusion constraint.

Another unresolved problem is evaluation: SCX now ships with a growing number of schedulers, but developers still lack good tools or benchmarks to explain why one scheduler performs better than another for a given workload. As the subsystem moves beyond experimentation into broader deployment, that question may become as important as the remaining kernel work itself.

Video: What's missing in sched_ext? - Andrea Righi (OSPM26)

Sched_ext overheads and caveats — Christian Loehle

Loehle introduced sched_ext from a scheduler developer's perspective, including sched_ext_ops callbacks, local and global dispatch queues (DSQs), and custom DSQs. Terminal DSQs such as the local and global DSQs, hand control back to the core sched_ext machinery, while custom DSQs let the BPF scheduler retain ordering and placement control. Much of the real policy often ends up in ops.dispatch(), while wakeup-side callbacks, such as select_cpu() and enqueue(), need to stay cheap.

Loehle then covered overhead measurements. A simple futex wait/wake test showed that callback structure matters a lot. Inserting a task directly into the local DSQ from select_cpu() could beat the fair scheduler on that microbenchmark, while always going through both select_cpu() and enqueue() was significantly slower. The default empty sched_ext path was much faster still, showing the cost of BPF callbacks rather than policy. Further measurements looked at the cost of tracking task state for PELT-like accounting: tracking of runnable and quiescent states was visible on hackbench and CPU-bound tests, run-time callbacks added little more, and tick() callbacks were more expensive, especially at 1000Hz.

The later part of the talk was about where the current interface is still uncomfortable to use. Implementing PELT, capacity-aware scheduling, and EAS requires accurate knowledge of runnable, running, sleeping, migrating, and stolen-time states, which becomes hard once tasks disappear into terminal DSQs, especially the global DSQ. Misfit migration can be approximated with mechanisms such as scx_bpf_reenqueue_local(), or by expiring a task's slice from tick(), but these approaches add races and bookkeeping problems. Core scheduling and gang scheduling can also be expressed with custom DSQs, cookies, inter-processor interrupts, and locking, but coordinating sibling CPUs or switching a gang atomically remains awkward; new 7.1 features such as paired enqueue/dequeue callbacks and SCX_ENQ_IMMED help but do not solve everything.

The discussion was brief and mostly about the practical consequences of those caveats. One topic was cache-line bouncing on shared DSQs, where DSQ granularity becomes part of scheduler design, for example by using a queue per-cluster domain. Another topic was whether the fastest benchmark represented a real policy; it was clarified as essentially the default empty sched_ext scheduler using the global DSQ. The EAS discussion covered stale per-CPU accounting, races when re-enqueueing local queues, and whether iterating the local DSQ would be any better. The final discussion topic was scheduler testing; sched_ext cannot directly test fair-scheduler-only code, but it can create deterministic scenarios for shared infrastructure such as PELT, core scheduling, or future mechanisms like proxy execution.

Video: Sched_ext overheads and caveats - Christian Loehle

FlexGuard vs. time-slice extension: handling lock holder preemption — Victor Laforet

FlexGuard (SOSP'25) is a non-heuristic synchronization technique that uses eBPF to monitor context switches and detect critical-section preemptions. When a lock holder is preempted, FlexGuard proactively transitions waiting threads from spinning to blocking, freeing CPU resources to quickly resume the preempted critical section. By reacting to actual execution events rather than static thresholds, FlexGuard can improve performance by up to six times compared to POSIX mutexes.

The time-slice extension approach, often discussed in the Linux community, instead aims to prevent lock-holder preemption altogether by giving tasks a little CPU time to complete a critical section. Starting from Linux 7.0, a thread can use rseq() to efficiently notify the kernel when it holds a lock. Rather than preempting such a thread, the scheduler allows it to run until the lock is released (for at most 5µs by default, or up to 50µs), preserving forward progress. After a comparative evaluation of both solutions, on database indexes, LevelDB, and other benchmarks, we find that the two approaches address different bottlenecks and complement rather than replace each other.

There was a discussion about realtime tasks and the chance of livelock. However, spinlocks should not be used with realtime tasks. And time-slice extension is not available with PREEMPT_RT anyway. Another discussion was about the fact that a spinlock should never be used by a large number of threads. Spinlocks should instead be fine-grained. This would allow for the use of basic spinlocks instead of queue locks, which are FIFO and have problems when the next waiter is preempted. The time-slice extension feature has been designed around fine-grained locks. However, some widely used software still uses broad locks and benefits from the increased throughput of queue-based spinlocks.

Video: FlexGuard vs. Time-Slice Extension: Handling Lock Holder Preemption - Victor Laforet

Comments (none posted)

Reports from OSPM 2026, day two

By Jonathan Corbet
June 24, 2026
The Power Management and Scheduling in the Linux Kernel Summit, which still goes by the historical acronym OSPM, was held in Cambridge, UK, in mid-April. As has become traditional, the presenters at that event have since written summaries of their sessions, and this work has kindly been made available to LWN for publication. The second day's sessions covered a wide range of topics, including device frequency scaling, using time-slice duration for CPU selection, scheduling domains on multi-cluster Arm systems, the LAVD scheduler, and more.

(See also: coverage from day 1 and day 3).

Devfreq for uncore DVFS — Jie Zhan

The talk covered work on building uncore dynamic voltage and frequency scaling (DVFS) — encompassing L3 cache, interconnects, and memory controllers, but excluding I/O and GPUs — on top of the kernel's existing devfreq framework. The motivation is concrete: on the chips studied, uncore accounts for around 41% of SoC power consumption at idle and around 17% under a CPU-bound workload. A SPECpower run with uncore DVFS enabled saved 7–35% SoC power in the 0–50% load range without measurable throughput loss. The goal is reducing power with minor performance impact, particularly on server SoCs where no generic upstream solution exists.

After reviewing devfreq's structure and its five existing governors, the discussion focused on two challenges. First, frequency-scaling strategy: the simple_ondemand governor produced severe ping-pong between the minimum and maximum frequencies in experiments because it scales down too aggressively and the reported "load" itself scales with frequency. The proposal is a tweaked proportional/integral (PI) governor — using a proportional term for fast upscaling and an integral term for smooth downscaling — which stabilized frequency around 900 MHz with no throughput loss. Second, governor-driver coupling: devfreq passes only one type of governor-specific data at device registration, so adding or switching to a new dynamic governor isn't really supported. The maintainer's guidance was simple: send patches and discuss them on-list.

The audience engaged actively. Several attendees argued for trying a full PID controller rather than PI, noting the derivative term could anticipate spikes, though tuning was acknowledged as notoriously hard and workload-dependent. One attendee challenged the "good result" claim: with frequency parked around 900 MHz while load spiked to 60–80%, the governor may be preventing the hardware from reaching peak throughput — SPECpower can adjust its own load, but other applications or benchmarks could suffer under the same setup.

A recurring suggestion was using the existing interconnect framework with bandwidth and quality-of-service hints tied to CPU frequency, since it can aggregate constraints across CPU, display, and image signal processor. The counterpoint was that this interconnect is largely static today and that reactive load detection always lags — a scheduler-level hint at task enqueue (and a forthcoming latency-based QoS proposal referenced for the next day) may suit L3/uncore better. Another question was about per-scheduling-entity tracking; the response noted that related-CPU topology is already attached to the uncore device, so scheduler utilization could be pulled from there.

Three open questions closed the talk: defining common uncore event descriptions so governors aren't per-driver, pulling hints from CPU or I/O devices (scheduler utilization, enqueue-time signals), and restructuring the governor-driver data interface to decouple tuning parameters from a single governor.

Video: Devfreq for uncore DVFS - Jie Zhan (OSPM26)

Using AMU/PMU to reduce useless power consumption in CPUFreq — Hongyan Xia

Hongyan Xia described the fact that real CPU capacity often does not scale linearly with frequency. Depending on the workload, the bottleneck might not be in CPU capacity but rather in other parts of the system, mostly cache and DRAM contention. However, schedutil is unaware of such factors and will only try to raise CPU frequency to boost performance.

Xia attempted to use counters in the Arm64 activity monitoring unit (AMU) and performance monitoring unit (PMU) to inform the CPU-frequency governor when the CPU power is not the bottleneck and when raising the CPU frequency is not useful. His presentation included how to identify the most helpful AMU/PMU counters in such scenarios and how to use a linear-regression model with the proper counters to limit CPU frequency, achieving almost the same performance level with improved energy efficiency.

Video: Using AMU/PMU to reduce useless power consumption in CPUFreq — Hongyan Xia

Using the slice duration of fair tasks in CPU selection — Vincent Guittot

Vincent Guittot presented some results about using the EEVDF slice duration to order task scheduling while not breaking fairness. The custom-slice feature can be used to theoretically run tasks with short slices first, but figures show that the current scheduler still suffers significant latency outliers.

For his tests, Guittot used the cyclictest benchmark to evaluate the scheduling latency of short-running tasks and hackbench or rt-app to overload the system. Cyclictest sets an 8ms slice whereas hackbench and rt-app use a 20ms slice in order to emulate short, interactive tasks versus background activities. While latency remains acceptable on idle systems, performance degrades under stressed conditions, such as those generated by hackbench. These workloads involve high frequencies of wakeups, sleeps, and migrations. In overloaded scenarios, the maximum latency for a short-running task, as simulated by the Cyclictest benchmark, can reach approximately 9ms (near its full slice duration) even when it should be prioritized and the 99.9th percentile of latency often exceeds 4ms.

This suggests that the current EEVDF implementation fails to preempt running tasks in a number of corner cases:

  • Negative lag preservation: tasks often stay in a queued state to pay back their overconsumption of CPU time (measured as "negative lag"). However, if new tasks are enqueued during this period, an existing task's negative lag can actually increase. The proposed fix ensures that, when a task re-enqueues, its negative lag is at most what it was when it went to sleep, preventing it from being unfairly penalized for system activity that occurred while it was inactive.
  • Next-buddy shortcut: the scheduler sometimes uses the buddy mechanism to favor a specific task or group, which bypasses the EEVDF's selection of the most eligible task. By clearing the buddy if EEVDF would have picked a different task, the scheduler ensures that preemption remains consistent for shorter-slice tasks.
  • Delayed-dequeue bias: the delayed-dequeue tasks, which wait on the run queue to become eligible, often have short deadlines and can easily be the next to be picked to run. They can thus prevent shorter-slice tasks from preempting the current task. To address this problem, those delayed-dequeue tasks are actually dequeued when checking for wakeup preemption.

With these corner cases fixed, the 99.9th percentile of latency moves below 700µs, except for one case, but some cases still need further studies:

  • Newly enqueued entities: between the preemption check and the pick of the next entity, new tasks can be enqueued, moving the average vruntime and making the preempting task not eligible anymore. Using the next buddy to force picking the task that triggered the preemption needs further study.
  • Permanent positive lag: while negative lag can be decayed with delayed dequeue, the positive lag lasts forever and gives an unfair advantage. Decaying the positive lag according to the sleep duration is under consideration. Tracking the uncontended time of a CPU to reset all lags is another option to evaluate.
  • Disabling run to parity improves the maximum scheduling latency because the scheduler can switch tasks more aggressively and keep minimum lag to all tasks, but at the cost of system throughput. Furthermore, any task that needs to run for more than 0.7ms (the default base time-slice value for the fair scheduler) will be preempted before finishing its work, delaying the time to finish a short, but longer than 0.7ms, computation.

In addition to all previous changes, the logic for CPU selection should follow the sequence:

  1. Idle CPU Search: an idle CPU remains the best target, offering immediate execution and avoiding the overhead of preemption.
  2. Minimum slice comparison: if no cores are idle, the scheduler compares the minimum slice duration of all enqueued tasks on the target CPU. The goal is to place the waking task on a core where its slice is shorter than the minimum, maximizing its chance of preempting the current task.
  3. Energy-aware scheduling (EAS): On mobile platforms, the scheduler must weigh these latency gains against the energy cost of waking up a specific power domain or shifting to a more expensive core.
  4. A push callback mechanism: since the window between selecting a CPU and the task actually being enqueued is not atomic, a new task might arrive and steal the eligibility of our waking task in the interim. The push callback acts as a fallback: if the task arrives on a CPU and finds it is no longer the first to run, it is immediately placed in a list to be pushed to a better core.

Video: Using the slice duration of fair tasks in CPU selection — Vincent Guittot

Rethinking multi-cluster scheduling domains for Arm64 servers — Dietmar Eggemann

The discussion focused on how well the current Linux scheduler topology fits modern, 64-bit Arm server systems. While these platforms share the same Linux task scheduler kernel code with x86 systems, their hardware looks quite different. Many Arm64 servers use a flat, mesh-based design with a large number of CPUs and a distributed, non-uniform cache system, rather than the hierarchical layouts the scheduler traditionally expects.

One of the main concerns is that these Arm64 systems often expose only a single large scheduling domain. This limits the scheduler's ability to make effective wakeup and placement decisions. Even though the system is fully cache-coherent, latency across the mesh is not uniform, and this information is not currently visible to the scheduler. As a result, potential locality benefits are not being utilized.

Initial experiments suggest that scheduling-domain size can influence performance, with a tradeoff between the overhead of searching for idle CPUs and the quality of placement decisions. The impact is highly workload-dependent, and early exit conditions from the task-wakeup functions make these effects difficult to observe and reason about.

A number of possible directions were discussed, including splitting large domains into smaller clusters, making better use of firmware-provided topology information, and exploring ways to derive structure from latency measurements between CPUs or memory of the mesh. Each of these comes with its own challenges, particularly around complexity, stability, and avoiding regressions across different workloads.

Overall, there was agreement that the lack of exposed and usable topology in large Arm64 systems is a limitation for the scheduler today. Ongoing work on this topic will further explore the approaches mentioned above to try to find a robust and generally applicable solution that will require coordination between hardware, firmware, and OS teams.

Video: Rethinking multi-cluster scheduling domains for Arm64 servers — Dietmar Eggemann

Evolving sched_ext: resource control, topology awareness, and energy efficiency for modern systems — Changwoo Min and Gavin Guo

Changwoo Min and Gavin Guo gave a joint talk covering two improvements to scx_lavd — the Latency-criticality Aware Virtual Deadline scheduler built on top of sched_ext. When Min first presented scx_lavd at OSPM 2025, it was a gaming-focused scheduler aimed at improving Windows games running on Linux through SteamOS, with waker/wakee frequency as its primary hint for task urgency. A year later, the project is broader, expanding scx_lavd into a potential default fleet scheduler, and that expansion has highlighted two parts of the scheduling problem. The first is support for control-group-v2 CPU-bandwidth control — the cpu.max interface — which multi-tenant systems (containers, VMs, cloud workloads) need in order to enforce hard CPU quotas. The second is a load balancer that understands both task-size heterogeneity and CPU-capacity heterogeneity.

Sched_ext cpu.max — moving the work off the critical path

Min opened with a quick recap of what cpu.max is supposed to do: it allows administrators to specify a (quota, period, burst) tuple per control group, and tasks in an over-quota group are throttled, meaning that they are dequeued and parked until the next period boundary. He quickly reviewed the three design aspects of the kernel's existing implementation that motivated his approach toward improving its performance.

The first is task selection. The kernel mirrors the control-group hierarchy as a nested red-black tree, so picking the next task to run becomes a walk down that nested tree. As Min explained, the cost of task selection grows linearly with the depth of the hierarchy.

The second is throttle detection. The current kernel implementation uses what Min called a "synchronous pull model" — each CPU borrows a 5ms slice from a central, per-control-group quota pool, consumes it locally, and pulls more when it runs out. When the local budget is exhausted and the group itself has no more quota, the CPU must walk up the control-group hierarchy to find a source of budget. The result is that the throttle check on every dispatch is expensive, touches global memory, and again increases with hierarchy depth.

The third is replenishment. The kernel uses two timers per group — a period_timer that refills the quota every period, and a slack_timer that returns unused local budget to the global pool asynchronously. The total number of timers in the system therefore grows linearly with the number of groups. The sched_ext cpu.max library Min built reorganizes all three concerns around a single principle: get the expensive work off the dispatch path. The library is exposed as lib/cgroup_bw and can be linked into any sched_ext scheduler; scx_lavd is its first consumer.

For task selection, the library keeps all unthrottled tasks in the regular dispatch queue (DSQ). Only throttled tasks are moved aside, into a per-control-group backlog task queue (BTQ). Task selection on the unthrottled DSQ therefore stays O(log N); the nested red-black tree disappears entirely. The throttle check on the hot path becomes a single flag read — no locks, no atomics, no hierarchy walk.

That last simplification is only possible because the library trades the kernel's accurate and immediate single-period enforcement model for what Min called eventual bandwidth control across multiple periods. A control group is allowed to overspend for up to one accounting interval; overspend is captured as debt and exactly subtracted from the next period's budget. Min was careful to emphasize that long-run-average usage still converges exactly to the configured quota.

To keep the detection latency small, the library arms an adaptive accounting timer. Each control group maintains an exponentially-weighted moving average of its consumption rate; the timer predicts when the group will hit its limit and fires early enough before the group is throttled.

Finally, the library normalizes every control group's period to a fixed, 100ms window. With every group using the same time unit, a single replenishment timer can serve the entire system — total timer count drops from two per group to two for the whole machine.

Min showed results from a 2-socket, 96-core AMD EPYC machine (192 CPUs) running stress-ng --cpu. He explained the scheduler overhead, measured as kernel-time CPU cycles during a pure user-space workload. As control-group depth increased from one to  32, EEVDF's overhead grew from roughly two CPU-equivalents to roughly five, while scx_lavd stayed flat at about two. At a control-group depth of 32 under a load sweep, EEVDF overhead spiked above ten CPU-equivalents at 125% load while scx_lavd remained near two.

The first question came from Andrea Righi, who pointed out that the BTQ is currently built on top of arena task queues, which, in turn, are built on BPF arenas. On older kernels that lack arena support, this becomes a portability problem; Righi asked whether the same machinery could be built on DSQs. Min said it should be possible in principle, but that DSQs currently lack an API for moving a task directly from one DSQ to another — the only existing transfer path is from a BPF DSQ to a local DSQ for execution. Iterating across DSQs would work, he said, but is expensive. Adding a direct DSQ-to-DSQ task-move kfunc to sched_ext would let BTQ shed its arena dependency.

An audience member asked whether any of these ideas could be contributed back to the kernel. Min replied that he was open to that discussion, but had wanted first to confirm that the design worked as intended; with several months of evidence that the overhead and accuracy numbers behave as expected, he said collaborators interested in improving the kernel's cpu.max implementation were welcome.

Another audience member asked whether the comparison had been run against the post-rework EEVDF. Around September 2025, the kernel's cpu.max throttling was reworked so that throttling is deferred to the user-space return boundary, with debt carry-over already implemented on the kernel side. Min acknowledged that his baseline predated that rework, and agreed that the EEVDF overhead numbers would be lower against the current kernel. The depth-independence of throttle detection and the elimination of per-group timers should remain wins regardless, but the specific overhead delta will need to be remeasured.

Another audience member asked whether the hierarchical accounting was inherent, or whether a flat model could avoid the bottom-up and top-down tree walks altogether. Min's answer was that the control-group interface is hierarchical, so the accounting has to be; the hierarchical work, however, runs only in the background accounting timer, not on the dispatch hot path.

Task-size-aware load balancing in scx_lavd

Guo then took over to describe the second improvement: a new load balancer for scx_lavd. He began by reminding the audience that scx_lavd is a domain DSQ scheduler — each last-level-cache (LLC) domain has a single DSQ shared by all of its CPUs, and domains are assigned "stealer" or "stealee" roles every 10ms based on their queued load relative to the system average. When a CPU's local run queue drains, ops.dispatch() first tries to steal CPU time from a remote stealee, then takes a task from the local domain's DSQ, and finally falls back to a force-steal from any nearby DSQ.

The current balancer has three weaknesses. Its load metric is the sum of utilization and the scaled queue length, which Guo said was obviously problematic — a domain with ten short-lived tasks looks busier than one with two long-running tasks, even though the latter is doing more work. There is no migration budget, so many CPUs race to drain the same overloaded domain in one round, with only probabilistic gating to limit the resulting thundering herd. And migration is task-type-blind: large tasks can land on small cores.

The new design replaces the load metric with queued_load_invr + util_invr, where queued_load_invr is the sum of task sizes in the domain rather than the count of tasks, and util_invr captures how busy the domain's CPUs currently are. The "invariant" qualifier matters on heterogeneous systems: run time is scaled by CPU capacity and frequency, so loads on P-cores, E-cores, and LP-E-cores are directly comparable. Each domain is then given a capacity-proportional fair share. For example, a domain with 40% of the system's capacity is expected to carry 40% of the total queued load; big-core domains, having more compute capacity, naturally carry more.

Migration is bounded by a symmetric 50% budget. A stealee allows half of its excess load above fair share to be migrated out; a stealer accepts half of its deficit below fair share. Closing only half the gap in any single round avoids the thundering herd problem that would occur if the full imbalance were corrected at once — the stealee would simply become the next stealer.

Guo showed schbench-wakeup-latency results on two machines, with six runs each. On a heterogeneous, 14-CPU Meteor Lake system, p99 dropped from 5,867µs (scx_lavd main) to 5,613µs (−4.3%), and p99.9 from 9,899µs to 9,195µs (−7.1%). On a homogeneous, 192-CPU AMD EPYC 9R14, p99 dropped from 5,867µs to 5,741µs (−2.1%), and p99.9 from 7,297µs to 6,777µs (−7.1%). The capacity-aware aspect of the design helps more on heterogeneous platforms, as expected, but the migration budgeting and task-size metric — which are independent improvements — transfer cleanly to the homogeneous case.

An audience member followed up with a question about how capacity is computed on SMT systems. The Meteor Lake box, he pointed out, is not the pure capacity-asymmetric model that EAS targets — it is also an SMT system, and scx_lavd currently treats the four logical CPUs in the P-core domain as fully independent. He asked whether it is fair to assume they have the same capacity and are independent. The answer was that they are not independent; SMT siblings impact each other. Once both siblings of a physical core are busy, the effective capacity of each drops, and the relationship is workload-dependent (two compute-bound siblings interfere more than one compute-bound and one memory-bound). Guo agreed that the current model is an overcount and said he wanted to enhance it in the future. However, in a domain-based scheduler, the problem matters less than performance overhead, as the load balance is based on the domain capacity instead of the CPU capacity.

Ricardo Neri then explained how EAS-style schedulers handle the same problem. CPU capacity is not used for SMT systems, he said — busy/idle transitions are too fast and the overhead of tracking sibling state would be prohibitive. Instead, scheduling on SMT cores is done by priority; a CPU's priority is proportional to its capacity, but SMT siblings are assigned the lowest priority and are populated last. On Meteor Lake configurations where SMT is disabled, the question does not arise; where SMT is enabled, EAS relies on priority alone. Guo said this was exactly the kind of input he had hoped to get from the talk, and added that he would welcome more suggestions from Intel and Arm engineers on what else the model should capture.

The discussion closed with Guo previewing the next step: selective migration. Today, when a stealer pulls from a stealee, it always picks the head-of-DSQ task — which may not be the best candidate to migrate. The plan is to peek at the next two-to-eight tasks in the DSQ and choose based on task size (matched to domain capacity), latency criticality, cache locality, and waiting time (so long-waiting tasks can be rescued to bound tail latency), while skipping tasks that are still cache-hot using a task-hot guard, analogous to the completely fair scheduler's task_hot(), with a roughly 500µs threshold.

Video: Evolving sched_ext - Changwoo Min and Gavin Guo

Steam deck on large servers (continued) — David Dai and Ryan Newton

David Dai's portion of the presentation explored the performance of the scx_lavd scheduler on large servers, specifically analyzing the waker/wakee heuristics in production environments. In a user-facing web service involving a complex chain of wakeups, scx_lavd successfully identifies threads with frequent short wakeups as highly critical. Prioritizing these tasks reduces tail latencies during periods of burst traffic or CPU contention. However, a tradeoff exists: scx_lavd relies on shared dispatch queues per last-level cache, which increases task migrations, negatively impacts L1/L2 cache locality, and results in a small throughput penalty.

The heuristics yielded unintended results in a second case study involving a caching service. This service purposefully delays read tasks to batch them together, lowering their wake-up frequency compared to writer tasks. Consequently, scx_lavd assessed the writers as more latency-critical, creating a heuristic inversion. During the Q&A, audience members like Peter Zijlstra and Guittot suggested that developers could resolve latencies in EEVDF by manually setting the time slice for tasks based on their specific work cycle, a solution Dai acknowledged that he could try and test.

Another issue identified in the caching service was the impact of software interrupts. Cache workers could spend up to a quarter of their running time processing these interrupts instead of their primary tasks. To mitigate this, Dai proposed categorizing CPUs into "steady" CPUs and "turbulent" CPUs. When an audience member asked how to define a turbulent CPU, Dai explained that the current threshold categorizes any CPU spending 15% or more of its time processing interrupts as turbulent, which accounted for roughly a third of the cores in their tests. By calculating a "preemption vulnerability" score that combines a task's latency criticality and utilization, the system can steer highly vulnerable and critical tasks away from turbulent CPUs. This latency-aware placement improved overall tail latencies and better balanced the load across cores.

Ryan Newton's segment of the talk shifted the focus to creating testing environments that move scheduler development away from production systems. His goal is an "abstract, fix, test" loop where production traces are simplified into portable reproducer workloads. This pipeline involves taking production traces, creating a simplified Rust program, running it in a smaller virtual machine topology, and then translating it into a JSON-based workload specification using RT-app. Ultimately, these specifications are executed in SCX-SIM, a user-space simulator.

SCX-SIM does not use a live kernel; instead, it emulates kernel functions via C stubs. This allows SCX-SIM to be bitwise-deterministic and portable, operating three to ten times faster than real time while simulating multiple cores on a single core. Furthermore, SCX-SIM employs controlled concurrency testing. Although the simulator executes sequentially, it artificially interleaves chunks of task execution at controlled, randomized preemption points to expose race conditions and test for concurrency bugs. Addressing simulator accuracy during the Q&A, Ryan noted that, while SCX-SIM introduces randomized delays, to model context-switching overhead for example, , it lacks memory and cache modeling. An audience member suggested recording real instructions-per-cycle penalties during task migrations using hardware performance monitoring unit counters to feed back into the simulator, which Newton agreed would be a highly valuable addition.

The simulation workflow incorporates AI coding agents to help close the gap between the simulator and production. AI agents iteratively adjust the simulator's calibration parameters — such as wake frequency and time-slice distributions — to ensure its output traces closely match actual Perfetto traces from production. Additionally, agents are directed to write unit tests for specific branches of the scx_lavd scheduler, which has increased the test code coverage up to 77%. The audience showed interest in this AI-driven approach, prompting discussions about open-sourcing the prompts and configurations so the broader community could collaborate on simplifying production traces. Finally, when questioned about the human's role in this loop, Newton concluded that humans are primarily needed to establish external guardrails that prevent AI hallucinations and to correct the AI when it makes errors

Video: Steamdeck on large servers (cont.) - David Dai and Ryan Newton (OSPM26)

Hierarchical constant bandwidth server: current state and future challenges — Yuri Andriaccio

This talk presented the latest updates of the patch set that is aimed at replacing the realtime group scheduler with the hierarchical constant bandwidth server (HCBS) mechanism. The patch set was originally presented at OSPM 2025, and has since been sent to the kernel's mailing list to gather comments on its implementation. At the time of this talk, the latest proposed version was RFC v4.

The talk first focused on what the hierarchical constant bandwidth server is, and why it matters. HCBS reworks realtime group scheduling using deadline servers, introducing them to the control-group-v2 world (and dropping v1 support), significantly reducing code footprint, and reusing existing subsystems. Other practical improvements are focused on the realtime soundness of the scheduling algorithm, better control on bandwidth allocation, and the possibility of the execution of unprivileged FIFO or round-robin realtime tasks.

The implementation details are not different from what has already been discussed at OSPM 2025. The general idea consists of allocating a number of deadline servers and run queues for each control group, one for each physical CPU. The servers provide the bandwidth reservation for each CPU, and, whenever they are picked for execution, they recursively invoke the FIFO/RR scheduler on the new control-group-specific run queues.

HCBS is constantly being worked on, with its latest version (at the time of this talk) based on kernel version 6.18. It has been actively reviewed by the scheduler maintainers and other contributors. A growing set of HCBS-specific tests is also constantly being updated and executed to make sure that critical code sections are tested, and that temporal and isolation guarantees are provided.

The talk covered some of the issues that arose during the development of the patch set, like integration with CPU-hotplug mechanism and frequency scaling. While they were just a broad idea in 2025, multi-CPU control groups were further investigated and discussed in this talk. Given that the new HCBS mechanisms are going to be only usable with v2 control groups, it was suggested to integrate the cpuset and CPU controllers to implement partial reservations — to allow execution of FIFO/RR tasks on a subset of the CPUs, while also updating the admission tests for this setup. The original multi-CPU idea also features different budget and period reservations on a per-CPU basis, but integration with existing subsystems is still an open problem.

Another discussion arose about the current meaning of the sched_rt_{runtime/period}_us sysfs knobs. These originally were used to specify the maximum bandwidth allowed for realtime tasks and to implement the realtime throttling mechanism. That throttling was removed in kernel version 6.12, in favor of fair deadline servers. Since then, the sched_rt settings only limit the maximum allocatable bandwidth in the deadline scheduling class, but do not affect FIFO/RR tasks. The accepted solution was to just update the default bandwidth for deadline entities to be 100%, which has been recently merged in the sched/tip branch of the kernel.

A final discussion arose on the meaning and possible substitutes of the current deadline-scheduler admission test. That test in fact does not guarantee that every task will respect its deadlines, instead it only guarantees that the response time of deadline tasks is not unbounded.

Video: Hierarchical CBS: current state and future challenges - Yuri Andriaccio (OSPM26)

Comments (1 posted)

Page editor: Joe Brockmeier
Next page: Brief items>>


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds