LWN.net Weekly Edition for July 24, 2025

Welcome to the LWN.net Weekly Edition for July 24, 2025

This edition contains the following feature content:

• Understanding Debian's security processes: a DebConf presentation on how security updates get out to users.
• When free-software communities unite for privacy: what the Tor project is up to.
• Deep immutability for Python: a proposal to improve the efficiency of Python programs that use subinterpreters.
• Scheduler medley: time-slice extension, sched_ext deadline servers, and LRU batching.: three proposed improvements for the kernel's CPU scheduler.
• QUIC for the kernel: after more than a decade, the QUIC network protocol may finally see support in the Linux kernel.
• How to write Rust in the kernel: part 3: a look at some core kernel abstractions for Rust code.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Understanding Debian's security processes

Providing security updates for a Linux distribution, such as Debian, involves a lot of work behind the scenes—and requires much more than simply shipping the latest code. On July 15, at DebConf25 in Brest, France, Samuel Henrique walked through the process of providing security updates to users; he discussed how Debian learns about security vulnerabilities, decides on the best response, and the process of sending out updates to keep its users safe. He also provided guidance on how others could get involved.

Henrique introduced himself as a member of Debian's security-tools packaging team; he has been a Debian developer since 2018, maintains a several packages, and serves as a mentor for newcomers trying to learn how to package software. He is also a senior system development engineer for the Amazon Linux security team, but he noted that his contributions to Debian are made as a volunteer and not something he does during work hours.

Henrique presented a similar security talk at DebConf24 in Busan, South Korea, called "Fixing CVEs on Debian: Everything you probably know already"; the video and slides are online. That talk, he said, would be a good reference for anyone who wanted to contribute security fixes to Debian.

This year, though, he wanted to give a more user-focused talk because he had noticed that even experienced users and contributors don't understand the way that Debian handles security vulnerabilities. Many users do not understand why Debian backports security fixes, and they often do not know about the Debian backports repository, which contains packages from Debian testing built for the current stable release.

CVEs and vulnerability information

Most of the time, when a discussion comes up about computer vulnerabilities, we're talking about those that have been enumerated as Common Vulnerabilities and Exposures (CVEs). Not always, but "there's a big overlap there". In fact, there are many different areas of security, and he wanted to acknowledge the areas he was not going to talk about. For example, how Debian manages its infrastructure and the way it builds packages are important to the security of the distribution—but he intended to focus solely on vulnerabilities in Debian packages for the talk.

The CVE program was created in 1999 by MITRE, with sponsorship by the US government. The main thing MITRE does, he said, is to assign CVE numbers and coordinate announcement of vulnerabilities by maintaining a public database. CVEs describe a single vulnerability even across platforms, so if a vulnerability affects Firefox on Windows, Linux, and macOS, it will still have a single CVE ID. Anyone can request a CVE ID; there is a process, and it should involve reporting a vulnerability to the developer of the software first, but there are no special credentials required to request a CVE.

Then there are organizations that enrich the data, such as NIST, which maintains the National Vulnerability Database (NVD). It provides a numerical score to rate a CVE's severity, called the Common Vulnerability Scoring System (CVSS). Vendors often use CVSS in service-level agreements to determine how quickly a patch for a CVE must be released. Many vendors, including Canonical, Red Hat, and SUSE, also release their own assessments of a vulnerability's severity that is specific to their products.

Henrique noted that Debian also provides vulnerability information via its security tracker. He showed a page from the tracker for CVE-2023-4911, which describes a buffer overflow in the GNU C Library (glibc). The page contains the CVE description, links to various vulnerability databases, various notes on the CVE, and a list of Debian packages with each one's status related to the vulnerability. For example, the page lists the glibc packages for current Debian releases, which ones are affected, and the version of the package that contains a fix for the vulnerability.

Debian users can get security updates for a release for up to five years from the Debian project, and an additional five years in some cases. Henrique explained that each release receives three years of support from Debian's security team and an additional two years of updates from its long-term support (LTS) project.

He advised the audience that it should update to the newest release after five years; if that was not feasible then there is the extended long-term support (ELTS) from Freexian. While ELTS is a commercial offering, the company makes updates available to everyone for free. However, there are limitations to the ELTS; Freexian only supports a subset of architectures and packages for each Debian release, depending on what its customers request. For example, its Debian 9 ELTS only supports the x86_64, i386, and armhf architectures. The base system packages for Debian 9 are supported for the life of the ELTS, but other packages may not be.

Triaging: anyone can do it

Day and night, weekdays and weekends, the Debian security team is dealing with "the firehose" of CVE information. There were 40,274 CVEs published in 2024, so there is a lot to sift through. Henrique said that the team has automated the import of security data that is queued up for triage by anyone: "if anybody is interested in this, talk to me, and I can teach you". The team has to determine which versions of a package may be affected; that involves "very cool investigative work", such as reading the vulnerability description and searching for the commit that introduced the vulnerability.

After determining the versions of software that are affected, then it's a matter of researching whether it affects a package in Debian. For example, a vulnerability may only be present on a hardware architecture that Debian does not support. That is a bit rare, he said, "because Debian runs on everything".

There are also security embargoes, which are "the most fun thing" about security work, Henrique said, possibly sarcastically. There are a few embargo processes that Debian has to deal with, he said, but they all involve representatives of different organizations coordinating to get fixes ready for when a vulnerability is published.

The most famous embargo group, according to Henrique, is the distros mailing list. It includes representatives from "all the big distros", and access to the list is severely limited. Members of the list may handle security problems themselves, or work with the maintainers of affected packages. It is safe to assume that there is an embargo in effect right now, he said, "there's one going on at all times".

Backports

Henrique circled back to his statement near the beginning of the talk, "people don't understand why Debian backports security fixes" rather than shipping the latest version of software that contains a security fix.

He described a scenario where Debian is shipping version 4 of a project in the stable release, and version 3 in the old stable release. A security vulnerability is announced in the software, and the upstream project fixes it with version 5. Users will think they need version 5, but do not understand that it might break their whole workflow. Just shipping the latest release of software to provide a security fix "is just not suitable for Debian stable".

Instead, the project will try to identify the minimal fix for the problem and backport it to the version currently in each Debian release. If a vulnerability is severe enough, Henrique said, Debian may release a Debian Security Advisory (DSA) to announce it to users and expedite release of the fix to the security repository. (Users can find the most recent DSAs on LWN's Debian security alerts page as well.) If it is a less severe vulnerability, then it will go through the normal proposed updates mechanism and may take up to two months to be released with the next point release for Debian stable.

So far, he had primarily talked only about Debian stable releases. Henrique said that there is no official security support for the experimental, unstable, and testing versions of Debian. However, he said that vulnerabilities in those releases are "all fixed in a timely manner, because we care that vulnerabilities are fixed quickly" even if it the releases are not officially supported. He pointed out that the fix for the XZ backdoor was released for unstable and stable at the same time and took about 24 hours to reach the testing distribution. No promises, he said, but the experimental, unstable, and testing versions of Debian are "more secure than a lot of distros out there".

Sometimes Debian decides not to fix a CVE because it is either too risky, too complex, or the vulnerability is not severe enough to require a fix. For example, Henrique said that some software has support for different network protocols and when the upstream finds a vulnerability it might decide to just remove support for a protocol rather than implement a fix. "That is valid from the upstream's perspective, but we can't just yank out a feature" in a stable release. At least, not unless the severity is too great, "we do an analysis on which is worse". If it's really bad, then it's removed. If not, then the fix will wait until the next release.

In some cases, Debian will decide not to bother with a fix because the vulnerability is essentially impossible to exploit in real-world circumstances. He used the example of finding buffer overflows using fuzzing to target an application's internal functions. "From outside an app, it's impossible to trigger" the vulnerability, Henrique said.

Source packages

Debian has a tooling problem, he said, where a vulnerability is only present in the source package and is not built into the binary package—but Debian's security tooling currently has no way of distinguishing the two. So, to users, it looks like the binary has a security flaw as well. The good news is that there is a fix in the works, contributed by "the Freexian folks" and waiting for Debian's security team to review.

Once that fix is available, Henrique said, he would be able to flag the CVEs he knows are false positives. "I have done some investigation, and roughly 20% of the CVEs in Debian container images" are false positives due to vulnerabilities in source packages that are not present in binaries.

By the numbers

The signal-to-noise ratio of CVEs is pretty low when one considers how many CVEs actually affect Debian. Out of more than 40,000 CVEs in 2024, Henrique said that 2,046 CVEs were fixed and 255 DSAs were issued for Debian stable. Only 56 CVEs that were relevant to Debian, or about 0.01% of all CVEs issued in 2024, had an entry in the US Cybersecurity and Infrastructure Security Agency's known exploited vulnerabilities (KEV) catalog. That does not mean there are no exploits of other CVEs, of course, but Henrique noted the ratio of KEVs to CVEs is an important indicator of actual security vulnerabilities compared to CVEs issued.

People who want to research CVEs on their own should start with Debian's security tracker, Henrique said. He also suggested checking the NVD for extra references and suggested reading "Hacker News, LWN, all those spaces" to keep up with security vulnerabilities. If a vulnerability is serious enough it will have its own name and logo, he said. It may seem silly or funny to brand vulnerabilities, but it works. "Nobody will recognize a CVE number a few years later, but everybody remembers Spectre."

He urged users to pay attention to all mitigating factors for a vulnerability, because it may only be a problem in certain situations or in non-default configurations. While it's critical to apply security fixes, eventually, "you need to understand if you are actually exploitable at that time". He also said people should be "very critical of descriptions in CVEs", as they often imply things that are not true. For example, just because a version of a project is not listed in a CVE does not mean it is not vulnerable.

Understand risk

Users should also be wary of Debian derivative distributions, Henrique said. He did not name any distribution specifically, but said that distributions that pull from Debian may have security problems that Debian does not.

He advised the audience to update their machines frequently. "The real risk is not zero days; it's machines that have not been updated in a long time". He also coached users to pay attention to "unofficial stuff on machines", such as IDE plugins, editor extensions, and other software from "random corners of the internet". If there is a vulnerability in software that is installed from third-party repositories, will users even notice? Will they pull the fix? "I'm not saying don't do this; I know it's required. But understand how you're changing your risk when you do it". If something is unused, it should be uninstalled.

In summary, he said that the Debian security tracker is the canonical source of information for Debian vulnerabilities. Just because a package is affected, he said, does not mean that a system with the package is affected. "A lot of CVEs are just noise".

Q&A

There was a little time left over for questions, so I asked Henrique if there were any practices from other Linux distributions that he would like to see Debian adopt. He said that he did not like that Debian over-reported on CVEs with false positives, but that was changing "with some help from Freexian", and he expected to have fewer false positives in the future.

Another member of the audience asked what his recommendation would be for security scanners that look for software version numbers to determine if a system is vulnerable, which means that Debian is sometimes falsely flagged. He said that he loves that question: "any security scanner that is good enough is reading Debian's data instead of guessing versions; tell your vendor their scanner is broken."

The slides from the talk are available on Henrique's website. Video from the talk should be made available by the DebConf25 video team soon.

[Thanks to the Linux Foundation, LWN's travel sponsor, for funding my travel to Brest for DebConf25.]

Comments (4 posted)

When free-software communities unite for privacy

At DebConf25 in Brest, France, the talk "When Free Software Communities Unite: Tails, Tor, and the Fight for Privacy" was delivered by a man who introduced himself only as intrigeri. He delivered an overview of the Tor Project, its mission, and the projects under the umbrella. He also spoke about how the organization depends on Debian, and plans for the software it delivers.

It is entirely fitting that a talk on protecting user privacy and anonymity would be given by a speaker who does not reveal their full name in person or online. The Tor Project is a non-profit organization with a global community of volunteers who work together to produce "a lot of software", intrigeri said.

He did not cover all of the software produced by the organization, and when one says "Tor" some disambiguation is necessary. Tor may refer to several things. It could refer to the organization itself, or the Tor network, which is the overlay network that runs through Tor network relays operated by volunteers, using a technique known as onion routing. Internet traffic is routed through Tor relays to obfuscate a user's location and destination to deter network surveillance or traffic analysis from determining what a user is doing online.

It could also refer to the Tor software used to connect to the Tor network or the software to set up and run Tor relays.

The project also provides the Tor web browser and Tails, which is a Debian-based operating system meant to be run from a USB thumb drive. Intrigeri has been a Debian developer for many years and runs the Tor Project's Tails team. The project's GitLab instance hosts all of the software it provides.

The problem that Tor is trying to solve with all of this software is that internet service providers (ISPs), commercial interests, or governments are able to identify and track users' activity on the internet. Encryption alone is not enough, he said, to provide anonymity. It does not hide who is talking to whom or who is visiting a web site; encryption only masks the content of communications, not the participants.

Tor tries to provide ways for users to communicate and use the internet anonymously, without fear that they'll be spied upon. For some users—such as whistleblowers, victims of domestic abuse, and dissidents in oppressive countries—the preservation of anonymity is not only about protecting one's rights on principle, it is a matter of safety.

ISPs or government entities like the US National Security Agency (NSA) can figure out who is visiting a site or which users are talking to one another if those connections are made directly over the internet. Even when communications are encrypted, a user's "social graph" may be exposed via metadata. Here he displayed a slide that quoted former NSA director Michael Hayden, who said: "We kill people based on metadata."

In practice, people rarely care about anonymity, he said, so it is necessary to position Tor appropriately for the audience. If he talks to a private citizen, he says "I work on privacy software". If he is speaking to a business, on the other hand, he says that Tor works on network security. When talking to governments, he describes Tor as providing "traffic-analysis resistance", and for human-rights activists, "Tor provides reachability". Here intrigeri used the example of people in Iran or Russia trying to communicate with people in other countries.

If Tor only worked for one of the use cases, such as user anonymity, then "the other three would try to fight it". By addressing the interests of individuals, businesses, governments, and human-rights activists, "all of these people who would not be aligned, or would be adversaries, have some interest in working together".

Evil relays

Many people use commercial virtual private networks (VPNs), which relay traffic through a remote server in an attempt to provide online privacy. But there is the possibility, he mentioned, of an "evil relay"—such as a VPN provider that tracks users or provides information to other parties. Even if the VPN provider is trustworthy, he said, observers can use timing analysis to determine who is connecting to what.

Tor solves this by using multiple relays. One relay knows who is connecting, but "no single relay has all the information" about users and their connections. Even then, he said that using Tor at the network level only hides a user's IP address; there is still information in the network packets that can leak information the user does not want leaked that can uniquely identify them. "As soon as you're unique, you're traceable." That is why the Tor Project provides the Tor Browser, which is Firefox-based and tries to ensure that it does not send information that can uniquely identify a user by fingerprinting their browser.

Some users may think that is what Chrome's incognito mode or Firefox's private browsing mode do; it is not. "That does not do much, it leaves fewer traces on the hard drive, that's about it", he said. For the threat model Tor considers, "it does nothing at all". To emphasize the point, he put up a slide with a cartoon mocking a Chrome user trying to use incognito mode to hide their browsing habits.

Tails and Tor

Of course, some internet activities take place outside the web browser. Users who need anonymity beyond web browsing can look to Tails. Intrigeri said that Tails provides a digital-security toolbox with safe defaults for working with sensitive documents and more. He described it as "amnesia by default" since Tails is meant to be run from a USB stick and leave no traces on the system it runs on.

Tails was first announced in 2009 under the name Amnesia, but the project was not part of the Tor organization until last September. The announcement said that talks began between the two organizations in 2023 because Tails had "outgrown its existing structure". Intrigeri said that the Tails team can now focus on improving Tails and leaves fundraising and running the organization to the Tor Project. "I'm glad. I was not very good at it." Tails is now more sustainable and able to focus on its core mission thanks to combining efforts.

He is happy with the merger, and not just because he has been able to leave fundraising behind. He said that being part of the Tor organization means that the collective is now better equipped to react to censorship and decide what piece of the software stack is best to help solve the problem. Some problems may be best solved with improvements to the Tor Browser, some with Tails, and some at the Tor network level. There are also better training and outreach opportunities now that the organizations have combined.

Thanks, Debian

He noted that speaking at DebConf was a natural fit for Tails and Tor because the organization owes so much to Debian; in addition to being the base distribution for Tails, Debian is the base for Tor's infrastructure as well. Currently, Tor has more than 100 hosts running Debian, and he wanted to say "thank you".

Intrigeri also described Tails as a "gateway to free software". Many users of Tails have never used Linux before they reach for the distribution; "before they were just using macOS or Windows". After experiencing Tails, he said, users may want something similar for day-to-day usage when anonymity is not required. It is not a one-way street, however. "A bunch of people become contributors in free software through Tails", and the Tails project contributes back to Debian as well. For example, members of the Tails team help to maintain AppArmor in Debian, as well as the puppetserver package, and others.

What have you done for me lately?

Lately, Tor has been working on a number of new projects or enhancing existing ones. He said that Tor has been focusing on a "comprehensive approach to anonymity and privacy protections", which includes meeting users where they are. For example, many users' primary method of connecting to the internet is via their phone, not a desktop or laptop. In countries with heavy censorship, where the Tor network itself is blocked, users can connect to relays via bridges, which are Tor relays that are not listed in the Tor directory and use obfuscation to make it hard to detect as a relay.

Until recently, though, connecting to a bridge on a mobile phone was difficult and cumbersome. In April, Tor announced the addition of Connection Assist for Android in Tor Browser 14.5. This feature made it possible for Android users to more easily connect to Tor bridges, something that had only been available on the desktop version of Tor Browser.

He also discussed some of the anti-censorship tools from the project that had been introduced or improved lately, including WebTunnel and Snowflake. WebTunnel, which was announced last March, mimics encrypted web traffic (HTTPS) to fool observers into thinking that a user is simply browsing the web. In reality, it acts as a secret bridge to the Tor network. Users visit the get bridges for Tor page, copy the bridge information, and then use that to configure the Tor Browser to use WebTunnel to hide the fact that they are using the Tor network.

Snowflake is another technology to help "give censorship the slip" where Tor is blocked. Users who cannot connect to Tor ordinarily can use Snowflake to connect via WebRTC; to observers it appears as if the user is simply making a video call. Users whose internet access is uncensored use the Snowflake extension to act as a Tor relay, with the inbound connection coming via WebRTC, allowing Snowflake users to piggyback on their connection. Last year, Tor added a Snowflake extension for Chrome browser users that allows users to share their connection with other users; extensions were already available for Firefox and Microsoft Edge.

The organization also spends a fair amount of time trying to optimize its services and evade not only censorship but attempts to infiltrate the networks. Intrigeri mentioned that one of the ways that users in Russia get information about Tor bridges is via Telegram and the Tor Project realized that there was an unusual amount of new accounts on Telegram requesting bridge information. The purpose of this was likely to find Tor bridges and block them. To prevent that, the project started to send new accounts to one set of bridges and direct older Telegram accounts to a different set.

After the talk I spoke with intrigeri briefly. He said that the project has to be conservative in its response to new tactics in the arms race between Tor and its opponents in order to keep an upper hand. If the project pushes out several new features or practices to thwart censorship before they are needed, its opponents can start trying to subvert them immediately. By waiting, the project can extend the lifetime of its defense.

On the financial front, intrigeri mentioned that the Tor Project had worked to "align with a new set of funders globally", with a mix of funding coming from individual giving, international grants, and "value-aligned companies". There was a bit of applause in the room when he added that the organization was now receiving 20% of its funding from the US government; that was down from 50% at one point. That meant that "the last ten months have been less stressful for Tor than for other organizations".

Keeping data safe

Even though Tails is designed to leave no data behind on the computer it runs on, users still need to store documents and other data. Tails is meant to be a "safe for sensitive material", but there is a challenge: an operating system that runs from a USB stick is especially prone to data loss because USB sticks tend to fail. Cheaper USB sticks, which are often all that is available to lower-income users globally, fail faster. It is also not safe to assume that users have substantial technical education or access to backups.

That means that the project has been working on, and has outstanding goals for, additional safeguards of user data. For example, with the 6.0 release, Tails added error detection for reading and writing to USB sticks. The project also has documentation for data recovery. He said that there is work toward warning users about the dangers of unplugging the USB stick before shutting down Tails and a plan to add a better data-backup feature.

Like it or not, he said, "people use smartphones", and they use messaging apps on smartphones. The project has conducted interviews with a variety of users—digital-security trainers working with human-rights organizations, journalists investigating state surveillance, environmental activists, and others—as a result it realizes that there is a need to provide messaging applications in Tails for communicating with people who only use smartphones. "'Install this app so we can talk' is not ideal; a stumbling block to have a conversation can be a problem." There is no one-size-fits-all solution, but Signal is the "top priority" application to support, as well as Wire and WhatsApp.

As he came to the end of his allotted time, he reminded the audience that its support matters and encouraged people to provide support in the form of running Snowflake proxies, Tor relays, monetary donations, and keeping Debian great.

During the question period, I asked whether the Tor Project provided any legal resources to those who ran Tor proxies, since it was possible that there is some legal risk in doing so. He said that the real risk is in running an exit node—the one that connects directly to whatever sites or resources that Tor users are connecting to. It is possible to operate a Tor relay that is not an exit node, so users concerned with legal risk can opt not to run an exit node. In answer to the direct question, he said that Tor does not provide any legal resources, but there are groups that "bring people together and pool resources, including legal ones", that might be available to support Tor users and proxy operators.

As Q&A time wound up a plane passed overhead rather loudly. As the noise subsided, he deadpanned that it was odd: "usually it's a black helicopter". It's hard to imagine a better exit line than that for a talk on Tor and its assorted projects.

[Thanks to the Linux Foundation, LWN's travel sponsor, for funding my travel to Brest for DebConf25.]

Comments (2 posted)

Deep immutability for Python

Python has recently seen a number of experiments to improve its parallel performance, including exposing subinterpreters as part of the standard library. These allow separate threads within the same Python process to run simultaneously, as long as any data sent between them is copied, rather than shared. PEP 795 ("Deep Immutability in Python") seeks to make efficient sharing of data between subinterpreters possible by allowing Python objects to be "frozen", so that they can be accessed from multiple subinterpreters without copying or synchronization. That task is more difficult than it seems, and the PEP prompted a good deal of skepticism from the Python community.

When threads concurrently access data, care must be taken to avoid race conditions. Historically, Python solved this problem with the global interpreter lock (GIL), which allows only one thread to access Python objects at a time. The Python community has been working to remove the GIL for several years, in order to make multithreaded code more efficient. Most recently, that has resulted in the creation of "free-threaded Python", a separate experimental build of the language that lacks a GIL. Now, users of free-threaded Python are faced with the problem of manually synchronizing threads, as users of other languages are. Subinterpreters offer a less error-prone alternative: separate, isolated memory spaces each with their own GIL. Objects can be sent between them using queues that perform a deep copy of the object to keep the subinterpreters isolated.

PEP 795 proposes an alternate solution: to make some data immutable prior to sharing it between subinterpreters, avoiding the need for a copy. It could also unlock future optimizations around reference counting and garbage collection. Trying to change an immutable object would raise an exception. This is part of a long-term research project to introduce "fearless concurrency" to Python, which the authors of the PEP presented at the Python Language Summit in May 2025. The summit was not recorded, but the authors made a separate recording of their presentation. The full proposal, of which PEP 795 is just the first step, is an ambitious change to Python's object model. The PEP itself was first circulated in June, after the summit. MemHive, which we covered in August 2024, implemented a somewhat similar idea as a library.

In order to provide complete protection from data races, it must not be possible for two subinterpreters to have simultaneous access to mutable data, even data nested deep inside the internals of a data structure. Therefore, making an object truly immutable requires transitively freezing every object reachable by the object being frozen. In Python, this would include the object's class, function objects for any methods, and possibly entire modules. One immediate objection to the proposal when it was discussed on Python's forum was that this would make it difficult to know exactly what would be frozen, both for humans and for type checkers. The PEP does provide an isfrozen() function for testing the immutability of an object at run time, but this isn't sufficient for static checks.

Cornelius Krupp gave an example of code that he believes should always work, because it represents part of the existing informal contract between users and library authors:

    if isinstance(l, list):
        l.append(1)

His belief is that if a library is given a list, it should expect to be able to modify it, which won't be the case if it is frozen. Loïc Simon disagreed, pointing out that it is possible to make that code fail in Python today by creating a subclass of list. Stephen Rosen suggested that, in code that makes use of deep immutability, the new correct way to write this would be:

    if not isfrozen(l) and isinstance(l, list):
        l.append(1)

But he acknowledged that "code which was correct has been rendered incorrect", which should be avoided unless "there's a commensurate benefit". The authors of the PEP thought that there was, but reviewers were less sure, with the potential for ecosystem disruption being a reoccurring point of concern.

One difficulty that came up was freezing function objects. Suppose that a function references a variable from an enclosing scope; if the function is frozen, should that variable become frozen as well? If so, this could end up freezing key parts of the Python interpreter, such as the dictionary that contains global variables. The PEP adopts a compromise position. When freezing a function object, the interpreter will make a copy of the storage location of all non-local variables before continuing with the freezing process. This ensures that the actual data reachable from a frozen function is immutable, but any captured variables can be reassigned in the outer scope (invisibly to the frozen function).

    >>> x = 0
    >>> def references_x(): return x
    ...
    >>> x = 1
    >>> print(references_x())
    1
    >>> freeze(references_x)
    >>> x = 2 # No error
    >>> print(references_x())
    1

Freezing functions at different times can potentially produce multiple, slightly different copies of shared global state. This isn't quite enough to actually avoid data races, though, as Krupp pointed out. This function can mutate a global variable in a way that the rest of the program can see, even when "frozen":

    def counter():
        __import__  # Needed because `import` doesn't work otherwise
        import sys
        sys.counter += 1

        return sys.counter

That function uses Python's import machinery to get access to the (mutable) sys module and store data there. The reference to __import__() is needed because Python's import statement is internally translated to a call to the __import__() built-in function. Without the reference, the variable's value doesn't get copied when the function is frozen, resulting in a run-time error. Even if this hole in the PEP's prototype implementation were closed, it's hard to see how to prevent frozen functions from getting access to shared mutable objects entirely, given Python's flexibility.

Tobias Wrigstad, one of the authors of the PEP, acknowledged the problem, but thought that the PEP was a step in the right direction, even if it didn't offer perfect immutability. Ben Hsing suggested that if the PEP wasn't going to be able to correctly freeze every type of Python object anyway, that it would be better to freeze a limited subset of objects in order to make the proposed mechanism less disruptive. He proposed making it so that freezing an object does not freeze its class, freezing a function does not freeze its referenced variables, and so on. Joao S. O. Bueno thought that it would make more sense to introduce lazily frozen proxy objects, instead.

That was the main sticking point of the proposal during the discussion: if there were still going to be ways to mutate an "immutable" object, why try to freeze an entire interconnected set of arbitrary objects? One suggestion was to focus on making immutability opt-in: let each class declare itself as mutable or immutable, and check that immutable classes only refer to other immutable classes. That would eliminate the problems for type checking (unless, of course, the programmer abuses Python's flexible class system to bypass the mechanism).

Wrigstad wrote a lengthy justification for why the PEP authors chose their particular approach — an explanation that might have been nice to include in the PEP itself. In brief, introducing immutable objects is just the first step in a plan to provide fine-grained, region-based locking for Python subinterpreters. The difficult cases that people were bringing up would be ameliorated by falling back to run-time locks for mutable state contained within an "immutable" object.

Reviewers found this a useful explanation of the intent of the PEP, but were still not wholly satisfied, saying that there were still open questions about how many common types would behave. They also suggested that the entire plan for region-based locking should be introduced as a single PEP. Paul Moore explained it like this:

If allowing frozen objects to be shared is a critical part of the justification for this PEP, it needs to be part of this PEP, otherwise you're asking the [steering council] to approve just the cost, with no associated benefit.

There is definitely an appetite for safer concurrent programming in Python, however. "Asaf Reich" expressed enthusiastic support for the idea, despite the many problems with the current proposal. PEP 795 has not been submitted to the Python steering council, and is unlikely to be adopted in its current form if it were. With the potential performance improvements, user enthusiasm, and a research team with a detailed plan, it does seem likely that this idea will return to Python in the future, however.

Comments (17 posted)

Scheduler medley: time-slice extension, sched_ext deadline servers, and LRU batching.

Decades after its creation, the Linux CPU scheduler remains an area of active development; it is difficult to find a time slice to cover every interesting scheduler change. In an attempt to catch up, the time has come to round-robin through a few patches that have been circulating recently. The work at hand focuses on a new attempt at time-slice extension, the creation of a deadline server for sched_ext tasks, and keeping tasks on isolated CPUs from being surprised by LRU batching.

Time-slice extension

The kernel is able to ensure that it will not be preempted whenever that would be especially inconvenient, for itself or for a task that it is running. User space, instead, has fewer options in that regard. As a result, it is possible for a user-space thread to lose the CPU while holding a lock that another thread needs, delaying the latter unnecessarily. Since lock-hold times are (or at least should be) short, delaying the preemption of a thread for a brief period while it holds a lock could improve the performance of the system overall.

Earlier this year, Steve Rostedt posted a patch series adding the capability for a thread to indicate that it is running in a critical section by way of a special field in the rseq structure that is shared between the kernel and user space. In Rostedt's implementation, a counter was used, and any non-zero value stored there would indicate that a critical section was being executed. If the kernel encountered such a value at a time when it was planning to preempt that thread, it would, if possible, give that thread an additional 50µs of running time. The use of a counter was intended to support nested critical sections on the user-space side. Entry into a critical section would increment the counter, while exiting would decrement it; when the counter reached zero, the task would be known not to be running in a critical section.

That patch set generated a fair amount of conversation and disagreement over how it should be implemented, with the end result that the work stalled and was never seriously considered for mainline inclusion. At the same time as Rostedt was working on his patch, though, Prakash Sangappa had been working on a similar patch. Unlike Rostedt, Sangappa has continued with this work; version 6 of his series was posted at the beginning of July.

Sangappa initially did not use struct rseq, opting instead to create a separate memory region shared with user space. That has changed over the evolution of the patch; the current version uses a one-bit flag in that structure. The maximum deferral time in this series is 30µs, but there is also a sysctl knob that can be used to change that value or, by setting it to zero, to disable the feature entirely. Otherwise, on the surface at least, the patch is similar to Rostedt's version.

Rostedt, though, had tied his mechanism to the relatively new lazy preemption scheduling mode; that decision had proved controversial. Sangappa's patch, instead, applies to tasks running in any scheduling mode. Thomas Gleixner complained about that design, saying that it would allow a normal-priority task to delay a realtime task, which "is fundamentally wrong". Gleixner said that preemption deferral should only happen in situations where lazy preemption would apply — when there is not a high-priority task waiting for the CPU.

Peter Zijlstra, though, disagreed, saying that the potential delay for a realtime task is less than what could already happen when an ordinary system call is made. Additionally, he said, tying deferral to lazy preemption would prevent realtime threads from using it, but the feature might be just as useful for those threads. Gleixner acquiesced to this point of view, but said that the feature needs a mechanism to enable or disable it on a per-process basis; Rostedt agreed with that request.

Nobody seems to disagree with the fundamental objective of this patch series, though, so it is mostly a matter of getting agreement on the implementation. It is hard to say how many more rounds will be required to reach that point, but eventually it seems that this feature should find its place in the mainline kernel.

Deadline servers for sched_ext

The extensible scheduler class (sched_ext) allows custom CPU schedulers to be loaded from user space as a set of BPF programs. Those custom schedulers can implement whatever special algorithm is needed to ensure that a given workload runs efficiently, often optimizing the system in ways that the in-kernel scheduler is unable to achieve. If, however, a realtime task runs for 100% of the CPU time, sched_ext tasks will be unable to run and all that work will have gone for nothing; the problem gets even worse if one of the blocked sched_ext tasks represents the system administrator trying to regain control.

The same problem applies to tasks running in the SCHED_NORMAL class, of course, and various attempts have been made to solve it over the years. The current solution is deadline servers, special kernel processes that run in the deadline-scheduling class, which outranks even the normal realtime classes. A deadline-server thread is configured to run for a maximum of 5% of the available CPU time; it uses that time to run SCHED_NORMAL tasks in the usual way. This mechanism ensures that a minimum of CPU time remains available for non-realtime tasks, but allows realtime tasks to use that time if there are no other requests for it.

The solution, as implemented by Joel Fernandes, is to add a deadline server for sched_ext tasks too, ensuring that they are not crowded out of the CPU entirely. Given that the infrastructure for deadline servers exists at this point, the amount of work required to effect this change is relatively small; still, six versions of the series have been posted so far. That said, comments on the work seem to be winding down, so it may be getting close to ready.

LRU batching on isolated CPUs

There are some workloads that really do not want to be interrupted; they need 100% of the time on the CPU(s) where they have been placed. The kernel supports those applications with its CPU-isolation feature. When a CPU is isolated, all of the normal kernel housekeeping work, including interrupt handling, is moved to other CPUs. The scheduler tick is disabled, and the designated task begins running there. As long as that task does not make a system call, it should not have to compete with any other work for that CPU.

CPU isolation is rather poorly documented within the kernel; there is information to be found in this 2020 LWN article and this article series on the SUSE blog.

Memory-management tasks, such as working through folios on the least-recently-used (LRU) lists, qualifies as the kind of work that should not be done on isolated CPUs. But, as Frederic Weisbecker points out in this patch series, the isolation in current kernels is not yet perfect. Specifically, the memory-management subsystem maintains a set of per-CPU arrays of folios that have been marked for LRU processing. Folios can be added to those arrays quickly, then processed at leisure when the kernel concludes that a good time has been reached.

In current kernels, though, one CPU can, at times, make that decision for another. If the memory-management subsystem notices that there are folios in another CPU's arrays awaiting attention, it can remotely trigger that processing, even if the target CPU is running in the isolated mode. That will result in the workload losing access to the CPU while the LRU arrays are drained, breaking the promise that had been made.

Normally, if the workload is running in user space, one would not expect folios to end up in the per-CPU arrays to begin with. There can be problems with pages left over when the CPU enters the isolated mode, though. Additionally, even the most user-space-intensive task may need to make a system call every now and then, and those can end up queuing more folios for processing.

Weisbecker's solution is to stop the remote triggering of LRU processing on isolated CPUs. Instead, a per-CPU flag will be set, and the target CPU will perform this processing on return from the next system call. That will still delay the isolated workload, but that delay will happen when the task makes a system call and delays are already expected. That should allow this work to be done at a time when it will not create problems for the isolated workload.

There have been some comments on the implementation that suggest the need for a least one more revision of this series. Once this work goes in, though, the kernel will have the basic infrastructure for the deferral of work on isolated CPUs; it would not be surprising to see other kernel housekeeping work end up being moved into that structure as well.

Comments (none posted)

QUIC for the kernel

The QUIC transport-layer network protocol is not exactly new; it was first covered here in 2013. Despite carrying a significant part of the traffic on the Internet, QUIC has been anything but quick when it comes to getting support into the Linux kernel. The pace might be picking up, though; Xin Long has posted the first set of patches intended to provide mainline support for this protocol.

QUIC was created to address a number of problems that have been observed with TCP on the modern Internet. The three-way handshake at the core of the TCP connection protocol adds latency to connections, causing the next cat video to be that much slower to arrive. TCP was not designed to support multiple simultaneous data streams; it suffers from head-of-line blocking, in which a dropped packet brings everything to a halt. All told, TCP does not perform as well as one might like for that all-important web-browsing use case.

TCP also transmits much of its connection metadata in the clear, where any party between the endpoints can read it. That can result in information leaks. But middleboxes on the Internet also make free use of connection information to filter out anything that does not match their idea of how a TCP connection should work. The result is protocol ossification — the inability to make any changes to the TCP protocol because the result will not survive transmission across the Internet. Attempts to improve TCP, such as multipath TCP, have to be carefully disguised as ordinary TCP to function at all. TCP has become almost impossible to improve.

QUIC is an attempt to address all of these problems. A streamlined connection-setup process eliminates the three-way handshake, making the establishment of connections faster. The protocol is built on top of UDP, and is designed with multiple streams in mind; the loss of one UDP packet will not affect any streams that did not have data in that packet. QUIC-specific transport data is contained within the UDP packets, and is always end-to-end encrypted, so middleboxes have no chance to inspect it. If UDP packets can get through, anything that QUIC does can get through as well.

The QUIC protocol is specified in RFC 9000, with some tweaks made in RFC 9369. The protocol is supported by a lot of software — particularly web browsers — found on a typical Linux system and is said to handle a majority of the connections to Google's servers, but the implementation is entirely in user space. This approach was taken to speed the development and distribution of QUIC; the people at Google who were pushing it did not want to have to wait until operating-system kernels with QUIC support were widely distributed. At this point, though, the evolution of the protocol has slowed, and minds are naturally turning toward kernel implementations, which hold the potential for better performance while making QUIC easily available to a wider range of applications.

The patch set aims to integrate QUIC as naturally as possible into the kernel. There is a new protocol type — IPPROTO_QUIC — that can be used with the socket() system call in the usual way. Calls to bind(), connect(), listen(), and accept() can be used to initiate and accept connections in much the same way as with TCP, but then things diverge a bit.

Within QUIC, TLS is used to manage authentication and encryption. Establishing a TLS session can involve a lot of complex, policy-oriented work involving certificate validation and more. As with the existing in-kernel TLS implementation, QUIC pushes that problem out to user space. Once a connection has been made, each side must handle the TLS handshake before the data can start flowing. The sendmsg() and recvmsg() system calls are used to carry out that setup; the libquic library and tlshd utility (from the ktls-utils project) can be used to handle that task. Once TLS setup is complete, data can flow normally between the endpoints.

It is worth noting that QUIC caches the results of the TLS negotiation on both sides of the connection. Once two systems have successfully connected, subsequent connections can skip most of the setup work, allowing data to be transmitted with the first packet.

QUIC is meant to be fast, but the benchmark results included with the patch series do not show the proposed in-kernel implementation living up to that. A comparison of in-kernel QUIC with in-kernel TLS shows the latter achieving nearly three times the throughput in some tests. A comparison between QUIC with encryption disabled and plain TCP is even worse, with TCP winning by more than a factor of four in some cases. Long offers some potential reasons for this difference, including the lack of segmentation offload support on the QUIC side, an extra data copy in transmission path, and the encryption required for the QUIC headers.

This performance gap will likely shrink over time. One of the motivations for getting QUIC into the kernel is to be able to take advantage of hardware-based protocol-offload functionality; that functionality does not really exist yet, but it seems clear that the network-interface vendors are interested in providing it. As QUIC gains the hardware support that TCP benefits from, and as the in-kernel implementation is further optimized, it should see some significant performance gains.

The current level of performance seemingly has not reduced interest in this implementation, though. There is an outstanding pull request [update: pulled on July 17] adding QUIC support to the Samba server and client implementations, and interest in adding that support for the in-kernel SMB and NFS filesystems. There is a repository adding kernel-based QUIC support to curl. Once QUIC is in the kernel, chances are that other applications will gain support as well.

That may take a little while, though. The posted series adds just over 9,000 lines to the kernel, and it is only the low-level support code; another series with the rest of the implementation is promised, but has not been posted as of this writing. The process of reviewing all that code has just begun, and can be expected to take some time; consider that the Homa protocol implementation remains unmerged after 11 revisions posted over nine months. QUIC may have a quicker experience than that, but one still should not expect to see it in the mainline before sometime in 2026 at best.

Comments (2 posted)

How to write Rust in the kernel: part 3

The interfaces between C and Rust in the kernel have grown over time; any non-trivial Rust driver will use a number of these. Tasks like allocating memory, dealing with immovable structures, and interacting with locks are necessary for handling most devices. There are also many subsystem-specific bindings, but the focus of this third item in our series on writing Rust in the kernel will be on an overview of the bindings that all kernel Rust code can be expected to use.

Rust code can call C using the foreign function interface (FFI); given that, one potential way to integrate Rust into the kernel would have been to let Rust code call kernel C functions directly. There are a few problems with that approach, however: __always_inline functions, non-idiomatic APIs, etc. In particular, C and Rust have different approaches to freeing memory and locking.

During the early planning phases, the project proposed adopting a rule that there should be a single, centralized set of Rust bindings for each subsystem, as explained in the kernel documentation. This has the disadvantage (compared to direct use of Rust's FFI) of creating some extra work for a Rust programmer who wishes to call into a new area of the kernel, but as more bindings are written that need should go away over time. The advantage of the approach is that there's a single set of standardized Rust interfaces to learn, with all of the documentation in one place, which should make building and understanding the bindings less work overall. The interfaces can also be reviewed by the Rust maintainers in one place for safety and quality.

Allocating memory

Like C, Rust puts local variables (including compound structures) on the stack by default. But most programs will eventually need the flexibility offered by heap allocation and the limitations on kernel-stack size mean that even purely local data may require heap-allocation. In user space, Rust programs use automatic heap allocations for some types — mainly Box (a smart pointer into the heap) and Vec (a growable, heap-allocated array). In the kernel, these interfaces would not provide nearly enough control. Instead, allocations are performed using the interfaces in the kernel::alloc module, which allow for specifying allocation flags and handling the possibility of failure.

The Rust interfaces support three ways to allocate kernel memory: Kmalloc, Vmalloc, and KVmalloc, corresponding to the memory-management API functions with similar names. The first two allocate physically contiguous memory or virtually contiguous memory, respectively. KVmalloc first tries to allocate physically contiguous memory, and then falls back to virtually contiguous memory. No matter which allocator is used, the pointers that are exposed to Rust are part of the virtual address space, as in C.

These three different types all implement the Allocator interface, which is similar to the unstable user-space trait of the same name. While the allocators can be used to directly create a [u8] (a sized array of bytes; conceptually similar to how malloc() returns a void * instead of a specific type), the more ergonomic and less error-prone use is to allocate Box or Vec structures. Since memory allocation is so common, the interfaces provide short aliases for boxes and vectors made with each allocator, such as KBox, KVBox, VVec, etc. Reference counted allocations can be made with Arc.

The choice of allocator is far from the only thing that kernel programmers care about when allocating memory, however. Depending on the context, it may or may not be acceptable to block, to swap, or to receive memory from a particular zone. When allocating, the flags in kernel::alloc::flags can be used to specify more details about how the necessary memory should be obtained:

    let boxed_integer: Result<KBox<u64>, AllocError> = KBox::new(42, GFP_KERNEL);

That example allocates an unsigned 64-bit integer, initialized to 42, with the usual set of allocation flags (GFP_KERNEL). For a small allocation like this, that likely means the memory will come from the kernel's slab allocator, possibly after triggering memory reclamation or blocking. This particular allocation cannot fail, but a larger one using the same API could, if there is no suitable memory available, even after reclamation. Therefore, the KBox::new() function doesn't return the resulting heap allocation directly. Instead, it returns a Result that contains either the successful heap allocation, or an AllocError.

Boxes, as smart pointers, have a few nice properties compared to raw pointers. A KBox is always initialized — KBox::new() takes an initial value, as shown in the example above. Boxes are also automatically freed when they are no longer referenced, which is almost always what one wants from a heap allocation. When that isn't the case, the KBox::leak() or KBox::into_raw() methods can be used to override Rust's lifetime analysis and let the heap allocation live until the programmer takes care of it with KBox::from_raw().

Of course, there are also times when a programmer would like to allocate space on the heap, but not actually fill it with anything yet. For example, the Rust user-space memory bindings use it to allocate a buffer for user-space data to be copied into without initializing it. Rust indicates that a structure may be uninitialized by wrapping it in MaybeUninit; allocating a Box holding a MaybeUninit works just fine.

Self-referential structures

The kernel features a number of self-referential structures, such as doubly linked lists. Sharing these structures with Rust code poses a problem: moving a value that refers to itself (including indirectly) could cause the invariants of this kind of structure to be violated. For example, if a doubly linked list node is moved, node->prev->next will no longer refer to the right address. In C, programmers are expected to just not do that.

But Rust tries to localize dangerous operations to areas of the code marked with unsafe. Moving values around is a common thing to do; it would be inconvenient if it were considered unsafe. To solve this, the Rust developers created an idea called "pinning", which is used to mark structures that cannot be safely relocated. The standard library is designed in such a way that these structures cannot be moved by accident. The Rust kernel developers imported the same idea into the kernel Rust APIs; when referencing a self-referential structure created in C, it must be wrapped in the Pin type on the Rust side. (Some other pointers in the kernel API, notably Arc, include an implicit Pin, so the wrapping may not always be visible). It might not immediately cause problems if Pin were omitted in the Rust bindings for a self-referential structure, but it would still be unsound, since it could let ostensibly safe Rust driver code cause memory corruption.

To simplify the process of allocating a large structure with multiple pinned components, the Rust API includes the pin_init!() and try_pin_init!() macros. Prior to their inclusion in the kernel, creating a pinned allocation was a multi-step process using unsafe APIs. The macro works along with the #[pin_data] and #[pin] macros in a structure's definition to build a custom initializer. These PinInit initializers represent the process of constructing a pinned structure. They can be written by hand, but the process is tedious, so the macros are normally used instead. Language-level support is the subject of ongoing debate in the Rust community. PinInit structures can be passed around or reused to build an initializer for a larger partially-pinned structure, before finally being given to an allocator to be turned into a real value of the appropriate type. See below for an example.

Locks

User-space Rust code typically organizes locks by having structures that wrap the data covered by the lock. The kernel API makes lock implementations matching that convention available. For example, a Mutex actually contains the data that it protects, so that it can ensure all accesses to the data are made with the Mutex locked. Since C code doesn't tend to work like this, the kernel's existing locking mechanisms don't translate directly into Rust.

In addition to traditional Rust-style locks, the kernel's Rust APIs include special types for dealing with locks separated from the data they protect: LockedBy, and GlobalLockedBy. These use Rust's lifetime system to enforce that a specific lock is held when the data is accessed.

Currently, the Rust bindings in kernel::sync support spinlocks, mutexes, and read-side read-copy-update (RCU) locks. When asked to look over an early draft of this article, Benno Lossin warned that the current RCU support is "very barebones", but that the Rust developers plan to expand on it over time. The spinlocks and mutexes in these bindings require a lockdep class key to create, so all of the locks used in Rust are automatically covered by the kernel's internal locking validator. Internally, this involves creating some self-referential state, so both spinlocks and mutexes must be pinned in order to be used. In all, defining a lock in Rust ends up looking like this example lightly adapted from some of the Rust sample code:

    // The `#[pin_data]` macro builds the custom initializer for this type.
    #[pin_data]
    struct Configuration {
        #[pin]
        data: Mutex<(KBox<[u8; PAGE_SIZE]>, usize)>,
    }

    impl Configuration {
        // The value returned can be used to build a larger structure, or it can
        // be allocated on the heap with `KBox::pin_init()`.
        fn new() -> impl PinInit<Self, Error> {
            try_pin_init!(Self {
                // The `new_mutex!()` macro creates a new lockdep class and
                // initializes the mutex with it.
                data <- new_mutex!((KBox::new([0; PAGE_SIZE], flags::GFP_KERNEL)?, 0)),
            })
        }
    }

    // Once created, references to the structure containing the lock can be
    // passed around in the normal way.
    fn show(container: &Configuration, page: &mut [u8; PAGE_SIZE]) -> Result<usize> {
        // Calling the mutex's `lock()` function returns a smart pointer that
        // allows access only so long as the lock is held.
        let guard = container.data.lock();
        let data = guard.0.as_slice();
        let len = guard.1;
        page[0..len].copy_from_slice(&data[0..len]);
        Ok(len)
        // `guard` is automatically dropped at the end of its containing scope,
        // freeing the lock. Trying to return data from inside the lock past the
        // end of the function without copying it would be a compile-time error.
    }

Using a lock defined in C works much like in show() above, except that there is an additional step to handle the fact that the data may not be directly contained in the lock structure:

    // The C lock will still be released when guard goes out of scope.
    let guard = c_lock.lock();
    // Data that is marked as `LockedBy` in the Rust/C bindings takes a reference
    // to the guard of the matching lock as evidence that the lock has been acquired.
    let data = some_other_structure.access(&guard);

See the LockedBy examples for a complete demonstration. The interface is slightly more conceptually complicated than C's mutex_lock() and mutex_unlock(), but it does have the nice property of producing a compiler error instead of a run-time error for many kinds of mistakes. The mutex in this example cannot be double-locked or double-freed, nor can the data be accessed without the lock held. It can still be locked from a non-sleepable context or get involved in a deadlock, however, so some care is still required — at least until the custom tooling to track and enforce kernel locking rules at compile time is complete.

This kind of safer interface is, of course, the ultimate purpose behind introducing Rust bindings into the kernel — to make it possible to write drivers where more errors can be caught at compile time. No machine-checked set of rules can catch everything, however, so the next (and likely final) article in this series will focus on things to look for when reviewing Rust patches.

Comments (28 posted)