LWN.net Weekly Edition for September 10, 2026

Welcome to the LWN.net Weekly Edition for September 10, 2026

This edition contains the following feature content:

• Stabilizing Rust's never type: after more than two years of work, and a small breaking change, Rust will soon have a stable never type.
• CERN's migration path from CentOS Linux to Debian: a look behind the scenes at how CERN is planning to move to Debian to control the world's largest particle accelerator.
• Recent work in memory tiering: does a new design proposal make sense for tiered-memory systems?
• Deterministic testing for multithreaded Python: Larry Hastings explains how his blanket project can help provide better testing for multithreaded Python code.

• Fixing the TCMalloc regression with RSEQ operations: a look at Olivier Dion's work on the restartable-sequences API to bring TCMalloc back into the fold.
• Typst makes big strides: the most recent release of the typesetting system brings support for variable fonts, MathML, and more.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Stabilizing Rust's never type

A function's return type is supposed to indicate the kind of data that it produces. Rust's "never" type, which is denoted by an exclamation mark ("!"), is the type the language uses to mark a function that never returns and other places where a value can never occur. For a long time, the never type was used internally by the compiler, but was considered an unstable feature. On August 24, after more than two years of work, Rust-compiler-contributor "waffle" finally managed to stabilize the type. It took so long, in part, because it involved a small breaking change to previous Rust editions, which the compiler maintainers needed to ensure did not impact much real code.

(Note: Rust also uses exclamation marks to indicate calls to macros. The way the syntax is constructed, a place where it is valid to use the never type is not a valid place to put a macro invocation and vice versa.)

Why a never type?

There are two reasons that Rust has a never type, one practical and one philosophical. The practical reason is that it allows for more efficient generic code. For example, consider the FromStr trait in the standard library, which is used for types that can be instantiated from a string:

    trait FromStr: Sized {
        type Err;
        fn from_str(s: &str) -> Result<Self, Self::Err>;
    }

FromStr::from_str() either returns a converted result, or a custom error type. For example, attempting to convert "foo" into an integer will return a ParseIntError. But some types have an infallible conversion. For example, it is always possible to convert a string into a ByteString. That implementation of FromStr could set Err to be the never type. Then the compiler would know that the error branch of the returned Result is never present, and could optimize out all of the code that touches it or checks for it.

    impl FromStr for ByteString {
        type Err = !;
        fn from_str(s: &str) -> Result<Self, !> { ... }
        // Keeps the same generic interface,
        // but generates code equivalent to:
        // fn from_str(s: &str) -> Self { ... }
    }

The philosophical reason involves correct type inference. In Rust, constructs such as if statements and while loops are expressions; their results can be assigned to a variable. The compiler needs a type to infer for the result of an infinite loop, if the programmer writes one. That shouldn't come up often in real code, but it turns out to simplify type inference to be able to treat that case uniformly, rather than adding special rules to handle it.

In particular, the never type has a useful property for simplifying code: it automatically coerces to any other type. This sounds strange, but it is safe, since the never type represents the "result" of a computation that will never produce a value. So, anywhere that the code claims to have a value of the never type, the compiler knows that it can't possibly reach that code, and therefore it's safe to ignore it. This is a form of type-system-driven dead-code elimination.

For both of these reasons, Rust programmers have wanted to be able to use the never type in stable versions of the language. Making that happen required resolving a particularly thorny corner case.

Never fallback

Because of the way that conversions from the never type to other types are implemented, the compiler can sometimes end up in a situation where it cannot naively infer the concrete type of an expression. Consider this example, which defines an anonymous function (using ||, which is like Python or LISP's lambda) that never returns, and then calls it in a way that expects a concrete error type (using the ? operator):

    let function_that_never_returns = || { loop {} };
    function_that_never_returns()?;

That infinite loop is given a type of ! which is then implicitly converted to whatever the function is supposed to return. But since the function is defined locally and not given an explicit type, the compiler does not have sufficient information to say what that type is. The problem could be fixed by giving the function an explicit return type:

    let function_that_never_returns = || -> Foo { loop {} };

Since such an annotation would only be required in cases where the function cannot return anything, however, it would be a bit pointless to require the programmer to assign a fictitious type to it. So, the compiler includes a special rule: if, after all other type inference has been done, there is still an ambiguous type that cannot be determined, just assume that it should be the designated fallback type. Prior to the 2024 edition of Rust, that fallback type was () (the unit type, which has exactly one possible value). In the 2024 edition, the fallback type was changed to ! itself, essentially canceling out the implicit conversion. In the compiler internals, the never type still gets converted to an unknown type and then falls back, but from the programmer's perspective the behavior is identical to having the never type only undergo implicit conversion when required for the types to make sense.

That change of behavior was, technically, a breaking change. Type inference for some code could change, which could in turn cause compilation errors. That is the purpose of Rust's edition system: allow breaking changes in the front-end design of the language without breaking older code or requiring the whole ecosystem to update at once. In this case, however, there were reasons to want the new behavior backported to old editions.

Never infallible

For many years, the standard library has had an Infallible type to work around the unstable nature of the never type. It served the same semantic purpose as the never type, but did not have any special compiler support. Therefore, code using it would be technically correct but suboptimal (such as having an extra layer of tags in an enumeration or emitting dead code), because the optimizer would not always be able to remove references to Infallible. It was planned that, when the never type was eventually stabilized, Infallible would become a type alias for ! and all that old code would silently become more efficient. However, people pointed out a handful of ways that redefining Infallible had accidentally been made into a breaking change. Since ! has implicit conversions, changing the definition of Infallible could result in existing code needing additional type specifiers in order to type check.

Luckily, changing the definition of Infallible and changing the default fallback type, while both breaking changes, nearly cancel out. Any code that refers to the standard library's Infallible type by name would continue to work; it is only places where type inference is implicitly expected to produce Infallible that pose a risk of breaking existing code. With Rust's lack of implicit conversions in most cases, that will most often come up in places where the never type used to be implicitly converted to Infallible. If Infallible is made to be a type alias of the never type, then those places may experience never-type fallback, which would, in turn, change the inferred type and cause a compilation error if the never fallback type were not updated at the same time.

With both changes occurring simultaneously, the Rust maintainers believed that almost all existing Rust code would continue to compile — but "almost all" is not a reassuring qualifier when dealing with backward-incompatible changes. The Rust community does have a solution to this in the form of crater, which can download and compile all publicly available Rust libraries from crates.io in search of code that is broken by a compiler change.

Never say never

Waffle ran crater in April and found that, while there were 3,300 crates negatively impacted by the change, only seven were fully broken, with the rest broken by depending on old versions of libraries that had since been fixed. In the latter case, the problem would theoretically be fixable by releasing backported fixes for a handful of core libraries. This is not an accident; Rust has been emitting a warning whenever code triggers never-type fallback in a way that will break with the new change since 2024, so most libraries had plenty of time to update of their own initiative. The most common remaining error observed by crater is code that calls a generic function without enough type information for the compiler to pick a specific return type. Consider this function:

    fn foo<T: Default>() -> Result<T, Error> { ... }

It returns either a value of some caller-chosen type T that must implement the Default trait, or an error. If it is called without specifying a value for T, however, then type fallback can kick in:

    // No type specified.
    foo()?;

Previously, this would have made the compiler assume that T should be (), which implements Default, and so the code compiles. After this change (and on the 2024 edition), the compiler assumes that T should be !, which doesn't implement Default, and therefore causes a compilation error. The fix is to explicitly specify the type that foo() should return, either in the call or by pattern-matching assignment.

    foo::<()>()?;
    // or
    () = foo()?;

Even though it's not a complicated change, the Rust maintainers were not willing to break 3,300 crates. Waffle was asked to work with the maintainers of common libraries to backport simple changes like the above (making a new patch version, which many Rust build environments will pick up automatically), in order to reduce the number of libraries depending on broken dependencies. Several library authors were willing to make the backports, but some refused on the grounds that those old versions were past their end of life. Those maintainers pointed out that users could stay on an older version of Rust or update to the maintained version of the library. Even so, the successful backports addressed 1,553 of the failing crates.

After fixing a handful of related problems to reduce the number of broken crates even further, the Rust maintainers eventually agreed that even though there would still be some broken code it was worth making the change to simplify the language. So, starting in Rust 1.100, the never type will be stable and Infallible will be a type alias for the never type. Users who find that this breaks their code have a few options:

• Stay on Rust version 1.99.
• Update their dependencies to supported versions that include a fix for the problem.
• Add a patch to explicitly specify the return types of affected function calls.

On the one hand, this is a breaking change, and people may see code that had remained stable and working suddenly fail to compile. That could be seen as a violation of Rust's commitment to backward compatibility. On the other hand, the problem is relatively rare, there are multiple simple ways to fix it, it has been warned about for years, and it has always been part of the plan for the language. Additionally, the Rust maintainers worked directly with the community to find and address the breakage, even going so far as to help backport fixes to long-dead versions of popular libraries. So, the whole process could also be seen as an affirmation of Rust's commitment to backward compatibility.

In the future, people learning the language will hopefully find the never type just a little less special. Either way, most users of Rust will probably not be affected at all, but never say "never".

Comments (42 posted)

CERN's migration path from CentOS Linux to Debian

The European Laboratory for Particle Physics, usually just called CERN, is not only the birthplace of the World Wide Web, it is home to the Large Hadron Collider (LHC), the world's largest and highest-energy particle accelerator. As such, its computing environment is both truly unique and of great interest to people outside of CERN who hope to find lessons applicable to their own computing needs. The upcoming migration of some of CERN's systems from CentOS Linux to Debian, which was the topic of a talk at the recent MiniDebConf Winterthur 2026, is of particular interest.

Federico Vaga and Nikos Tsipinakis, both CERN employees, delivered the talk "Controlling CERN's Accelerators with Debian" (video and slides) on August 30. I was not fortunate enough to attend in person, but watched the video of the talk shortly after.

Powering up

Vaga has worked at CERN since 2014; he described himself as a software engineer and FOSS enthusiast "with little free time to make concrete contributions". Tsipinakis has been with CERN for about five years as a system engineer. He said that he is an open-source maintainer for "a few things", has been a Debian user for about ten years, and a Debian package maintainer for about seven years. He said that he hoped to become a full Debian developer at some point in the future.

CERN's main goal was not to invent things like the Web, he said, but "to operate the biggest accelerator complex on the planet", and to host experiments involving the LHC. He showed a slide with pictures of people standing near some of the collider's machinery: "You can judge yourself the size by looking at the people walking there. These are really huge systems to make discoveries about things that are more than invisible, [they are] subatomic." The systems are generating a large amount of data, which is then streamed to CERN's data centers: "We store just part of what gets actually fetched from the sensors because otherwise it would be too much". In the end, petabytes of data are stored for analysis.

However, Vaga said that he would not be talking about CERN's data center, which primarily uses Red Hat Enterprise Linux (RHEL) or the RHEL derivative, AlmaLinux. Instead, he would be focusing on the computers and operating systems that actually run the chain of accelerators and then smash particles "to see what's inside". The accelerator complex is distributed in an area of more than 43 square kilometers in size, and is 100 meters underground at its deepest point. The bulk of the accelerator is underground since it would be challenging to have to travel from one side to the other using surface streets: "We would need crossroads for beam particles and cars not to collide among themselves".

He put up a slide, shown below, with a map of the area where the complex is located. A large blue circle highlights the collision track of the LHC, the Geneva International Airport is highlighted in a red rectangle for comparison. The smaller blue circle is the second-largest machine in CERN's complex, the Super Proton Synchroton (SPS).

Since this is a "very special system" there is no factory that churns out parts that are suitable; CERN has a lot of custom equipment to operate the complex that is designed in-house and then produced in collaboration with industry partners. There are more than 36 kilometers of beam pipes, with 17,000 devices scattered throughout the complex in total and about 70,000 cables connecting those devices to about 2,200 computers in order to operate the devices remotely. Those numbers continue to grow, Vaga said, because CERN always has new needs. As with the parts that make up CERN's accelerator equipment, the PCI cards and such that control all of the devices are usually designed at CERN as well. CERN is more than its hardware, cables, and pipes, he said. It is also made of people.

We have a very heterogeneous population. We have software engineers, like we are, system administrators, we have mechanical engineers, chemical engineers, we have civil engineers to design the caves for the tunnels, but also our own buildings. So we need to do a bit of everything. And of course, scientists, physicists, eventually they need to analyze all this data and tell us the meaning of life.

Because CERN has such a diversity of people with various competencies, and such a big machine with specialized needs, it strives for stability. "So we need a very stable system, and it's sometimes challenging to keep it stable."

CERN and Linux

CERN began using Linux in 2008, "with the realtime patches on top"; prior to that, it used Microware OS-9 and LynxOS, both real-time operating systems. At first, CERN started with its own Linux distribution based on RHEL 5 and made in collaboration with Fermilab, which is a 6,000-acre particle physics and accelerator laboratory based in Batavia, Illinois. That became Scientific Linux based on RHEL 6.

Eventually, Vaga said, it was decided that it would be better to contribute to an upstream project like CentOS Linux rather than rebuilding their own, "because there was nothing very special about it". Pat Riehecky of Fermilab announced the end of Scientific Linux in April 2022, in favor of CentOS Linux 8. "We will collaborate with CERN and other labs to help make CentOS an even better platform for high-energy physics computing."

After that, Vaga said, "we were more or less on autopilot with RHEL derivatives". Unfortunately, the life of CentOS Linux 8 was cut short: Red Hat announced that it was ending CentOS Linux in favor of CentOS Stream. That raised some questions about what to do next. There are thousands of distributions, he said, but which one would fit CERN's needs? "It was important to judge what is the level of control and maintenance that we can afford in operating and configuring an operating system and how well it integrates in the current infrastructure".

CERN first considered RHEL-based systems. CentOS Stream was the first obvious candidate; it didn't guarantee the same stability, but it was not as cutting edge as Fedora. It was decided to start with CentOS Stream 9, with the idea of moving to 10. "However, we knew that we could not afford to have more surprises, so the first thing we did is to make our infrastructure more portable to other distributions and having Debian as a plan B solution."

RHEL to Debian

Debian quickly became plan A, however. The final straw that broke the camel's back, he said, was "a simple compiler flag". Specifically, Red Hat's decision to optimize RHEL 9 for the x86-64-v2 microarchitecture, and then x86-64-v3 for RHEL 10. Turning on gcc's -march=x86-64-v2 flag would have major consequences for CERN.

Vaga's analysis of the situation found that 47% of CERN's systems, which have older CPUs such as the Intel Core Duo line, would be unable to run the v2-optimized binaries, and the systems would not boot. "You get a message from glibc, 'I cannot go further than telling you I cannot go further'." Another 17% of CERN's systems were unable to boot using v3-optimized binaries. That meant that nearly 65% of its systems would not work.

Simply upgrading to newer systems is not really an option for CERN. PCs made before 2010 were generous with PCI slots, but today's are not. "You are lucky if you get one on your desktop" systems, he said. Industrial systems might have two slots, "if you pay a bit more", but that's still not enough. "This means that some systems that must be reconverted, one system might need three computers, and this occupies [more] rack space than it was occupying before."

CERN also uses VMEbus single-board computers that would be rendered obsolete by CentOS. "This is our workhorse for the control system". CERN has designed a new generation, "but the production might be slower because of a war somewhere, because of AI somewhere else, and the price increases, and so on and so forth". As Tsipinakis explained later in the presentation, the specifications for CERN's electronics are public, "but I'm not sure someone else would have a use for them other than if they want to make a particle accelerator". It also means that CERN has a number of custom kernel drivers for its hardware, which makes maintaining the stability of its systems "much more difficult" than if it only had custom user-space applications, since any bug in the drivers could affect the overall stability of its systems.

So CERN would be faced with upgrading a huge number of systems, and with much greater logistical problems than most organizations face when swapping out old equipment for new. CERN's systems, he reminded the audience, are distributed over 43 square kilometers, often underground. After loading a vehicle with the new computers, "you go underground, you change, you re-cable, and perhaps you [mistakenly] swap two cables, and you will discover this only later at commissioning time, and this might happen more often" than it would through normal maintenance.

Even worse, adding new systems might require additional rack space, which might require actually excavating additional space to make room. It would also require two electronic engineers to redesign control boards, two software engineers to develop new drivers, and two additional technicians to physically replace systems. "It starts creating a mess, and meanwhile all these people must be kept happy, because otherwise they cannot do their job. So it's very challenging." Vaga said that the estimated total cost of upgrades would be about 5.4 million Swiss Francs, or about $6.67 million.

In the end, it made more sense to migrate away from RHEL-based systems to Debian, "a universal operating system that tries to work on as much hardware as possible", for computers controlling CERN's accelerator. CERN will now use Debian, but build its own kernel from kernel.org sources and with all of the needed drivers built in.

The long shutdown

The change to Debian-based systems also requires CERN to revise the way it provisions systems. CERN's systems are diskless, with the GRUB bootloader delivered over FTP and other components delivered over HTTP, he said. All of the artifacts are stored in a GitLab package registry, and there is a configuration service that indicates the operating system version, kernel version, and packages that a system needs.

Tsipinakis said that, once the decision to switch to Debian was made, the first thing they had to do was to look at the release schedule. The accelerator is running 24 hours a day, seven days a week, and no updates can be provisioned during that time. There are short "technical stops" a few times a year, he said, but those gaps in the schedule are not long enough to accommodate a major distribution upgrade. "So you might have a five-year plan", he told the audience, "But at CERN, we have a minimum of a 15-year plan, so we know the state of the machine all the way up to 2041". He displayed a slide, shown below, with the long-term schedule for the CERN accelerator complex.

The green portions of the graph indicate when the accelerator is in operation. The blue portions are short-term technical stops for maintenance. Red portions are "what we call the long shutdown, which is when we perform the big maintenance of replacing hardware, major software updates, and so on". Until now, CERN has been running CentOS Linux 7, and the plan is to deploy Debian in early 2027 and keep it as stable as possible until the end of 2033 when the accelerator will again enter a long shutdown.

He then went into a discussion of Debian's release cycle, its long-term-support (LTS) cycle, and the extended LTS (ELTS) project managed by Freexian. The regular lifetime of a release is three years, with an additional two years as an LTS release, and five years beyond that as an ELTS. That meant that CERN could stick with Debian 13 ("trixie") through 2035 if needed. "So this was really valuable for us because if we find out that we cannot do major updates during the run, this is a very valuable backup plan, so we can stay with the same distribution for the entire period."

CERN's layered infrastructure

Now that CERN had mapped out a plan that aligned Debian's releases with its own schedule, it had to figure out how to actually build and deploy Debian on its infrastructure. Until now, since CERN had a homogeneous environment with its infrastructure entirely based on RPM-based systems, most of the people at CERN had little or no experience with Debian. Tsipinakis said they approached Freexian and asked for help. "So they sent us Seb (Sébastien Delafond), who did a four-day-long Debian training" that included package building, hosting APT repositories, managing build repositories, and more.

Tsipinakis noted that CERN's infrastructure is split into three layers and not all of those systems are being moved to Debian. The first layer is what it calls "consoles", which are "basically desktop computers in various rooms about CERN". These are used by physicists or the accelerator's operators, and have applications written in Java or Python used to send commands to the accelerator. "It can be, 'I need to ramp up the magnet strength to this, or increase the intensity of the beam to that'". Those systems are running AlmaLinux 10, with the Xfce desktop. "They will stay as is."

The next layer lies within CERN's data center; after an operator sends a command to the accelerator it is logged, authenticated, and analyzed in the data center to ensure that it's within the safety parameters of the system. He said these systems are currently running RHEL 9, and will likely be migrated to AlmaLinux 10.

What is being migrated to Debian is the third layer, Tsipinakis said: the systems that lie 100 meters underground.

They are paradoxically called the frontend computers. However, if you are an electronic engineer, the computer that you plug your magnet into is the frontend, and everything else is the back-end. So these frontend computers, being migrated to Debian, they are responsible for receiving the commands from the data center and then implementing those commands with hardware signals.

So you receive a command, okay, ramp the magnet strength to this value. This means they will set the command to a power supply to ramp the amperage to this value that will increase the magnet strength. So they're basically the interface between the electronics world and the software world.

Build and deploy

Before introducing Debian, CERN's "legacy" infrastructure—designed in 2005, before the first run of the LHC—had a single bare-metal hardware server operating as its boot server. It provided GRUB and kernel-boot configurations over FTP, and the root directory and applications over NFS. He said that this model had some advantages, such as knowing that every system has the same root directory. It also had a big disadvantage.

This server is the Holy Grail of the data center. You do not approach this server, you do not breathe on this server, because if this server goes down, a lot of people at CERN will have a very long day, because it means all the 2,000 embedded systems suddenly lost their root drive

Tsipinakis noted that CERN's infrastructure was "kept simple on purpose"; it was designed to have as few moving parts as possible. Now, "we have new technologies, we can do something better". Since most of CERN's infrastructure will remain on RHEL-like systems, they had to figure out how to compile Debian software on RHEL. He said that the Embedded Linux Build Environment (ELBE) stood out, "you run the script, and now your entire environment, GCC, and all the other build tools you need are coming from the Debian image, so you can basically compile Debian software on Red Hat".

It also meant introducing a Kubernetes-based, multi-server boot system to replace the "big bare-metal server we cannot touch", and introducing Debian package repositories for applications served via GitLab package registry.

Challenges

There were a few challenges along the way, Tsipinakis said. The first thing CERN ran into was that there is no standard tooling for automating the building and publishing of packages. Yes, he said, Debian has software it uses for building and publishing packages, such as buildd and the Debian Archive Kit (dak): "but these are very specialized Debian [tools], and I think people know that if you want to run dak on your systems, it is not easy to set up or easy to use".

He thought that Debusine, a continuous-integration platform for packaging and distribution-related workflows for Debian-based distributions, looked very promising. Sadly, he said, "it has some blockers that at this point we cannot support in our infrastructure" though he did not elaborate on what the blockers were.

He also complained that many of the tools do not support having multiple versions of the same packages in a repository. CERN wants to be able to distribute different versions of the same packages to different systems. Tsipinakis noted that there is a branch of reprepro, which is a tool for managing APT repositories, that supports multiple package versions; however, it is still in Debian's experimental repository.

CERN already has an extensive infrastructure around building RPM packages, based on Fedora's Koji build-system software. Since Debian is not currently the largest part of CERN's package-building infrastructure, he said, "why don't we bring Debian to the RPM infrastructure?" That is, in the end, what they did. Tsipinakis gave a quick shout-out to Erick Andrei Vilcica; he did the work to create a Koji plugin to allow it to build Debian packages and create repositories for them.

The final takeaway, Tsipinakis said, is that it is a mistake to assume that if you have a working system on one version of a distribution that it will keep working if you update to the next major version. "This is exactly the reason that we are here." Not being bound to a specific distribution, and having the ability to switch, is important. He added that the accelerator would shut down on September 1, a few days after the talk was given, and that all 2,200 of CERN's embedded industrial computers would be migrated to Debian trixie after that.

Q&A

The first question came from former Debian Project Leader Andreas Tille, who wanted to know if it would be possible to hold the next MiniDebConf Switzerland at CERN. Vaga said that it would "definitely be an option", but it would require going through the bureaucracy first. He said he would "come back with news" at a later date.

Another attendee wondered if they had considered rebuilding CentOS Stream without the v2 and v3 compiler flags. Vaga said that was discussed with the AlmaLinux team, but "no one wanted to become responsible for it" at the time. "It's not that you remove the flag and it's all fine; you need to go through a validation process, and our main goal at CERN is doing particle physics". AlmaLinux has since announced the availability of a version of AlmaLinux 10 built with the v2 compiler flags, though it does not offer a version without v2 or v3 flags.

One person wanted to know why CERN was concerned with keeping its frontend servers up-to-date; couldn't it run a "vintage" version of an operating system? "Is there some security concern that you need to address?" Tsipinakis confirmed that the systems were "not on the public Internet and not even on the CERN network". They were in an entirely different network domain, which was more secure because "you have to go through two layers" to get to them. Nevertheless, he said it was still good to keep patch levels up-to-date, though not "at the same level as the bleeding edge". Security updates for those systems happen about once a year during a technical stop.

Vaga added that updating systems was also important so that CERN could use more modern technology. Up until a few months ago, he said, everything was running CentOS 7, which meant that they were stuck with GCC version 4. "You are in an environment that is frozen in the past." Some people had begun to maintain their own compiler in order to move on, and so forth.

The final remark came from Bastian Germann, the maintainer of reprepro. He said that the experimental branch of reprepro still needed more testing, and that he had received a flood of emails complaining that it broke people's repositories the last time he uploaded it to Debian's unstable repository. He said he would love to have CERN's input if it could switch to the experimental branch. Tsipinakis said that was the plan and that they would get back to him after they had done so.

Comments (18 posted)

Recent work in memory tiering

Tiered-memory systems are built with multiple types of memory, each of which has different performance characteristics. In addition to the usual DRAM, a tiered system might also provide faster high-bandwidth memory or slower CXL memory. On these systems, the placement of memory allocations has a significant effect on the performance that a workload will obtain. While work on tiered-memory improvements has been ongoing for years, it feels like the pace has slowed a bit recently. Even so, there are a few efforts underway, but they are facing questions about whether the tiering design makes sense.

Hot-page promotion

"Demotion" refers to the act of moving a page of data from a faster tier to a slower one, while "promotion" is movement in the opposite direction. A page would be most likely to be demoted if the kernel concludes that it is not being actively used; promotion would happen when memory that is in use happens to be in slow memory. Demotion decisions are relatively easy (it is a similar decision to swapping a page out) and works reasonably well, but detecting hot pages in slow memory has proved to be more difficult. So either the kernel ends up leaving hot pages in slow memory, or it promotes pages that turn out not to be hot after all; in both cases, the result is worse performance.

Bharata B Rao has been working on an infrastructure for hot-page detection and promotion for some time now; LWN covered this work in 2025. This patch series is built around the idea that there may be multiple sources of data about which pages in slow memory are seeing high activity and should be promoted. The kernel might generate that data by marking pages as being inaccessible and tracking the resulting faults when a workload tries to use them, for example. CXL systems can also support access tracking and reporting, providing relatively fine-grained information without adding CPU overhead. Rao's proposed "pghot" mechanism integrates tracking data from the available sources and uses it to decide which pages should be promoted to faster memory. A new kernel thread, called kmigrated, uses this data to attempt to move the hottest pages.

The previous article covered the third version of the series; there have been five new versions since then. The core tracking machinery remains similar in concept, though various improvements have been made. This subsystem now can use profile data from AMD's Instruction-Based Sampling and has gained a new "precision mode". By default, pghot uses one byte of memory to track each lower-tier page; in precision mode, that overhead expands to four bytes and, among other things, enables tracking of which NUMA node is accessing each page. That information allows kmigrated to try to promote hot pages to the node where they are being used. That, in turn, enables pghot to (optionally) take over much of the work that is currently handled by the kernel's NUMA-balancing code.

Andrew Morton started the discussion of the current version by asking whether DAMON, which is already in the kernel, could perform this function. Rao answered with an LLM-generated comparison of the two mechanisms, which concluded that DAMON could perhaps do the job, but only after being tuned for the specific workload, and only after the workload had run for some time. DAMON author SJ Park was somewhat unimpressed by this response, asking Rao to actually learn about DAMON before going any further. In a separate response to Morton, Rao described some differences he saw between pghot and DAMON; Park disagreed with some of them, but also said that some pghot-specific features (such as integrating data from multiple sources) were currently being implemented for DAMON.

Matthew Wilcox had a different view of the entire patch set and the ideas that it is implementing:

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

Gregory Price quickly responded that Meta is "actively interested in this work and helping to drive the design of it". Yongting Lin also expressed support, albeit more lukewarmly, for this work. So there does appear to be some interest in this capability, even if it is not widespread.

David Hildenbrand pointed out that, even after eight revisions, there are no review tags for any of the patches; he described the series as "far away from being mergeable". This work, he said, has not been perceived as having even a medium level of priority, so the memory-management maintainers have not yet taken a hard look at it. The question of whether this use case can be adequately handled by DAMON also remains without a definitive answer. So it would appear that pghot is likely to be stuck on the ground floor for some time yet.

Tier-aware memory control groups

In the pghot discussion, Price mentioned a problem that comes up on tiered-memory systems: the first workloads to run fill the faster memory with their data; the workloads that come later then end up in slower memory because that is all that is left. Pghot, he said, would allow the system to rectify this problem over time. But there is also interest in preventing it from coming about in the first place.

Control groups with the memory controller (often referred to as "memcgs") would seem like the logical solution to this problem, but the memory controller is not currently designed to handle tiered memory. It can limit a group to a certain amount of memory overall, but cannot control where the memory is placed. So, naturally, the solution would be to add awareness of tiering to the memory controller. There are currently two separate patch sets that accomplish that goal.

This series from Joshua Hahn takes a relatively simple approach to configuration. If a given system has 25% of its memory as DRAM and 75% as slower CXL memory, then any given control group will be able to use 25% of its memory limit for DRAM, and must use the rest in CXL. That will prevent the first workload that shows up from filling the system's DRAM, leaving memory available for others. The hard and soft limits are both scaled in this way, allowing a workload to use more DRAM (if so configured) when contention for memory is low.

Implementation of the limits is handled in a number of places. Most obviously, perhaps, is at allocation time, when the system will allocate memory from tiers where there is still space available when possible. If that is not possible, and an allocation is placed in a tier with no space remaining, then targeted reclaim will be kicked off to remove some pages from that tier. Various other actions (such as page migration) that can exhaust a tier's limit will also start reclaim. Promotion of pages to faster tiers will be slowed when the limit is reached.

Meanwhile, Qiqi Liu has posted a separate solution to this problem. Like Hahn's series, Liu's sets limits for each tier that match the proportions of each memory type in the system. In this series, though, the administrator can adjust those limits if needed. Liu's solution does not implement the low limits (the minimum amount of memory that is guaranteed to the group), and is based heavily on reclaim to enforce the limits.

Either series seems capable of solving the core problem, but the maintainers will certainly not want to merge both of them. Michal Hocko asked about the differences between the two, while expressing doubt about the goal of adding tier-level control. Hahn, too, asked about the differences, noting that the ability to set per-tier limits in Liu's series seemed to be the biggest of those. Liu responded with a more detailed look at the two, and concluded by saying that Hahn's approach seemed like the best one to carry forward. So, while the question of whether the kernel should really gain tier-level control in the memory controller still needs to be answered, there does appear to be agreement among the interested developers on how to get there.

Comments (3 posted)

Deterministic testing for multithreaded Python

Python's support for multithreaded programs has improved considerably over the last few years with the advent of the "free-threaded" version of the language. But testing multithreaded programs is notoriously difficult, because the underlying host system determines the thread-execution ordering, which adds an element of non-determinism. At PyCon US, Larry Hastings gave a talk (YouTube video) about his blanket project, which is meant to provide mechanisms for deterministic testing of multithreaded Python code.

He began with a scenario where attendees were working on a new Python library, probably destined for the Python Package Index (PyPI). They had heard about "this no-GIL thing"—free-threaded Python which does not have a global interpreter lock (GIL)—and wanted to ensure their new library would work in a multithreaded environment. So they look into it and realize that "all I need to do is add a couple of locks"; he said that would likely suffice in many situations.

But sometimes life becomes more complicated and a race condition is found. That is a solvable problem, he said; often it just requires adding some code to recognize that the condition is occurring and handle it in some fashion. Once that is done, another problem rears it head, however: test coverage.

In 2026, projects should have a unit-test suite with 100% coverage, Hastings said. But that is hard to do when there is a line in the code that only runs rarely, under weird circumstances that are completely outside of the control of the developer. He showed a test-coverage report with a 99% that "is sitting there mocking you"; "you can't sleep at night, you've failed, your life is over". The answer is that "you are going to get it to 100% because you are going to use something called 'blanket'".

Blanket

Blanket is a new library, so new in fact that he had done the initial release the night before the talk. Why that name? "Because it is coverage using threads", he said, to groans, laughter, and applause.

Using the Python threading module is "giving yourself a problem", he said. While that is "a little glib", there are seven synchronization primitives in the module (e.g. threading.Lock(), threading.RLock(), and threading.Semaphore()) but they are all non-deterministic. For example, if three threads all want to acquire the same lock, the order in which they do so is unknown—it depends on the scheduler in the operating system. "You cannot predict what order your threads are going to run in."

The solution is to use blanket's deterministic replacements of the same seven primitives that are all available from the module's Scenario object (e.g. scenario.Lock()). That will make the program deterministic and allow the developer to control the order of events. It will "make any race condition reproducible on demand".

He showed a short Python program that appears in the "Quickstart" section of the blanket documentation. It creates three threads (using threading.Thread objects), each of which is going to acquire() and release() a Lock (implicitly using a with block) and then wait() on a Barrier object. A barrier is initialized with a number of waiters, when that number of threads have called wait() for the barrier, all of the waiters are allowed to run—in some order. For his example, the threads each use the worker() function as the thread; it makes the synchronization calls and reports that it acquired the lock and that it is past the wait:

    import random
    import threading
    
    lock = threading.Lock()
    barrier = threading.Barrier(3)

    def worker(name):
        with lock:
            print(f"worker {name} got the lock")
        barrier.wait()
        print(f"worker {name} is past the barrier")

The three threads, named "A", "B", and "C", are added to a list, which is shuffled (using random.shuffle()), used to start() the threads, and, finally, used to await their completion using join():

    A = threading.Thread(target=worker, args=('A',))
    B = threading.Thread(target=worker, args=('B',))
    C = threading.Thread(target=worker, args=('C',))

    threads = [A, B, C]
    random.shuffle(threads)
    
    for t in threads:
        t.start()
    for t in threads:
        t.join()

As can be seen in successive runs of the program, the order of which threads acquire the lock and get past the barrier is different each time, which he emphasized with four slides showing different orderings. For example:

    worker C got the lock
    worker A got the lock
    worker B got the lock
    worker B is past the barrier
    worker C is past the barrier
    worker A is past the barrier

There are six separate orderings for the lock acquisition and six for getting past the barrier, which makes for 36 different possible orderings of the test. If a race condition only happens in one of those orderings, it is difficult (or painful) to reproduce it reliably, but blanket can help.

He went through the program, modifying it to use blanket. The first step is to import blanket and then to create the scenario object, lock, and barrier:

    scenario = blanket.Scenario()
    lock = scenario.Lock()
    barrier = scenario.Barrier(3)

The worker(), thread creation, and shuffling operations are all the same from the earlier version of the code. The idea is that the worker() threads still do exactly what they did before, but will be using blanket versions of the lock and barrier. It is in the latter part where things change more:

    lock_api = scenario.api(lock)
    barrier_api = scenario.api(barrier)

    with scenario:
        for t in threads:
            t.start()
        list(lock_api.relay(B, A, C))
        lock_api.unblock(lock.release, C)
        with barrier_api.cycle(C, A, B):
            pass
    for t in threads:
        t.join()

The lock_api and barrier_api objects provide a "control surface" to manage the behavior of the underlying object. The code then "enters the scenario" via the with scenario: block and starts the threads as before. The next four lines are where the major differences lie; they use the high-level blanket API.

The relay() function of the lock_api object returns an iterator that steps through its thread arguments, releasing the lock (if it is held), then acquiring the lock, in the thread-order specified. It actually yields each thread after the acquisition step, but that is not used in the code, which just calls list() to iterate. That does mean that the final lock release is not done, which is what the unblock() call handles. The cycle() call on the barrier_api object prepares the threads for waking up after the barrier; more complicated things can be done inside that with block, but by default the order of the arguments governs the order in which the threads are woken up after the barrier.

The most important thing to note from the code is that the lock should be acquired in the BAC order and the barrier should be exited in the CAB order. As might be guessed, his next slide showed exactly that (as did the next three for emphasis).

Scenario

An object of the Scenario class is at the top level in blanket (e.g. scenario = blanket.Scenario()) and all seven of the threading synchronization primitives have blanket equivalents that can be reached from the object (e.g. lock = scenario.Lock()). One interesting thing about the primitives in blanket is that they are classes that are bound to the scenario instance as a bound inner class. He likens those to methods, which are likewise bound to their instance, and believes they solve a lot of problems, though "I have been using these for years and nobody seems to care".

The wrapper objects that blanket returns look the same as those returned by threading, which he calls "masquerading". He has added a way to tell the difference for debugging purposes, however.

    >>> t_lock = threading.Lock()
    >>> b_lock = scenario.Lock()
    >>> t_lock
    <unlocked _thread.lock object at 0x7c62de036d10>
    >>> b_lock
    <unlocked _thread.lock object at 0X7C62DE1CE510>

The ID for a blanket lock is printed in upper case, so that is a "tell", he said. Otherwise, both t_lock and b_lock are instances of threading.Lock, though of course only b_lock is an instance of scenario.Lock.

The scenario provides three levels of APIs: low-level, medium-level, and high-level. For the most part, users will stick with the high-level API, Hastings said, but they need to understand the other two levels, as well. The high-level abstraction is convenient, "but it is important to understand the low-level abstractions underneath you because all abstractions are leaky sooner or later".

The "transaction" is one of the "foundational, low-level concepts" that underlie blanket. It is "an object that represents a particular call to a particular method on a particular primitive at a particular time". If a thread calls Lock.acquire(), the transaction represents the thread, the lock object, and the call type (acquire). Transactions are created frequently, "under the covers"; he showed what one "looks like" by creating a scenario.Lock(), acquiring it, and retrieving an entry from the log that the scenario object maintains:

    <Lock.acquire RETURNED 'MainThread'
    start_time=377537.982058657 result=True
    blocking=False for Lock 1 0X79C14A4D2660>

It shows that the transaction is for an acquire, it is in the RETURNED state, ran on the main thread, and started at a specific time. The result holds the return value or the exception if one was raised. The transaction is not currently blocking and it was done on Lock 1, which is an internal name that blanket automatically gives to a lock. Locks can be renamed by test suites, which makes it easier to read the test output, he said.

Transactions are state machines that only operate in a forward direction. "If you are in a particular state, you know you will never visit a previous state." Outside of a scenario, a transaction simply moves from the COMMIT state, meaning that the underlying method has been called, to the RETURNED state. But inside the scenario (i.e. in a with scenario: block), transactions switch from "unregulated" to "regulated". There are two additional states each inserted ahead of the original two, so BLOCKED comes before COMMIT and PAUSED comes before RETURNED.

The two new states, which are called "parking states", are what allow blanket to control the behavior. They are entered before a blocking call to the underlying synchronization primitive in order to await permission to continue. The way tests work with blanket is that the main thread acts as the scheduler for the worker threads that are using the synchronization primitives. The main thread is running the unit-test suite, orchestrating the order of operations between the worker threads from within the scenario. After the test completes, the main thread exits the scenario and checks the assertions for the test case.

The most surprising thing he found out while developing blanket was that he could simply wrap the existing synchronization primitives and not have to simulate or reimplement them. All that turned out to be needed was the two stopping points: block and pause. Blocking before every method call is "the bread and butter of blanket"; it allows controlling what happens. "You don't need to control the internal state, just who calls the lock, in what order." That implicitly controls the state of the lock, Hastings said.

So blanket uses the real threading primitives, but wraps them. It blocks before making each call, allowing blanket to establish the order for the threads to call into the primitive. It also pauses after the call, which is what allows blanket to adjust the order of which threads resume after using a Barrier or similar primitive.

The other fundamental concept in blanket is scenario.wait(), which is a method of the scenario object that is modeled after "a wonderful API from Win32" called WaitForMultipleObjects(). The wait() method is passed a number of "signaling objects" that represent a binary signal that is either true or false. If all of the objects passed are false, the caller sleeps until at least one of them becomes true, which causes wait() to return a set of the passed-in objects that have signaled.

There are multiple kinds of signaling objects that can be passed, starting with a thread handle, which will signal if a transaction is running on it; that allows the main thread to see what the transaction is. The thread handle can be wrapped in a Terminated() object, so it will signal if the thread terminates. The Call() object can be used to signal when a specific thread is calling a particular method on a given primitive (e.g. Call(thread, lock.acquire)). A transaction handle can be passed to wait for the transaction to complete; they can be wrapped in Paused() to signal only when the transaction is in that state. The Not() wrapper can be used to negate the signal being returned, as well.

Low to high

Hastings presented a simplified version of his starting example in order to show the levels of the blanket API. The first part of the example remained the same through five separate rewrites:

    import blanket
    scenario = blanket.Scenario()
    lock = scenario.Lock()

    def worker(name):
        with lock:
            print(f"worker {name} got the lock")

    A = scenario.thread(worker, 'A')
    B = scenario.thread(worker, 'B')
    C = scenario.thread(worker, 'C')

Instead of using a lock and a barrier, the simplified version simply acquires and releases the lock, while reporting that it did so. The threads he uses in the example are slightly different as well; they are "managed" threads using scenario.thread(), which wraps the underlying threading.Thread objects. A distinguishing feature of the scenario.thread() objects is that they are started (i.e. thread.start()) when the scenario is entered and joined (thread.join()) when it is exited. The first low-level version was:

    with scenario:
        for thread in (B, A, C):
            for i in range(2):
                scenario.wait(thread)
                tx = scenario.transaction(thread)
                tx.unblock()
                scenario.wait(tx)

That produced the expected output (as would all of the examples): "B", then "A", then "C" acquire the lock. It steps through the threads in order, then does a series of operations twice, once for acquire and the second for release. It waits for the thread to enter a transaction, retrieves the transaction, unblocks it so it can continue, and waits for the transaction to complete. That is expressed a bit more directly in the next version:

    lock_api = scenario.api(lock)

    with scenario:
        for thread in (B, A, C):
            lock_api.unblock(lock.acquire, thread)
            scenario.wait(lock.release)
            lock_api.unblock(lock.release, thread)

This version uses the lock API object to unblock an acquire operation on the thread. It then waits for a release operation (on any thread, but the test only has a single possibility) and unblocks it. The next version is at the middle level of the blanket API:

    lock_api = scenario.api(lock)

    with scenario:
        for thread in (B, A, C):
            d = scenario.Driver(thread)
            d()
            d.skip()
            d()
            d.finish()
            d()

It uses a Driver object to push the thread through the various states; each call to d() drives the thread to the next point where a decision is needed. skip() and finish() calls provide the decisions to unblock the acquire and release operations. The next middle-level-API version uses a driver underneath:

    lock_api = scenario.api(lock)
    
    with scenario:
       for thread in (B, A, C):
          scenario.skip(thread, lock.acquire, lock.release)

The scenario.skip() call is passed a thread and multiple operations and will drive the thread to complete those operations by, effectively, running the driver as in the previous example. His last version, which uses the high-level blanket API is quite terse:

    lock_api = scenario.api(lock)

    with scenario:
        for thread in lock_api.relay(B, A, C):
            pass

That uses the relay() mechanism that was employed in the example at the start of the talk. It drives the threads through the acquire and release operations on the lock. These high-level APIs are tailored to the type of the primitive, so relay() is used for Lock and RLock objects, while cycle() is used for Condition, Event, and Barrier objects (as was also seen in the first example). Semaphore and BoundedSemaphore have an allocate() function to drive them through their acquisition and release.

Moar features

After that whirlwind tour of blanket, Hastings went through a list of other features, starting with timeout management. That allows programs to cause timeouts to happen (with expire()), cause them not to happen (with disregard()), or restore the timeout to its original state (revert()). He also briefly described "raw handles", which are synchronization objects that avoid the management blanket provides; the scheduler can interact with the underlying primitives that way.

Each blanket primitive (e.g. scenario.Lock()) is actually made up of four separate objects: the object masquerading as a threading primitive, the API object (e.g. scenario.api(lock)), the raw handle (scenario.raw(lock)), and an internal-only core object. Each of the blanket methods needs to acquire a single lock before they do their job. That lock lives in the Scenario core ("score") object and is called score.lock. That lock is a bit like the GIL, but it is lightly contended; it is a deliberate tradeoff to make blanket deterministic, he said.

He completed the list with two different code-injection mechanisms that blanket supports. scenario.inject() can be used to monkey-patch blanket into a module that directly uses the threading primitives, so that the module will use blanket instead. "Some galaxy-brained individual" who is writing their own synchronization primitives can use blanket.injector() to insert calls into the Python bytecode of those primitives, which will allow blanket-style testing of the custom synchronization mechanism.

Hastings completed his prepared talk with the credo of blanket, which is:

Your test should be effectively single-threaded. If it isn't, you haven't blanketed hard enough. Slow it down.

Before he took questions, he had a few extra slides to present. The first was about whether there are other technologies similar to blanket. "Only kind of"; there is a "rich area of computer-science research" into something called "stateless model checking" (SMC). It is like blanket, but instead of writing a scheduler, SMC systematically or stochastically explores the different combinations of ordering between the various synchronization mechanisms. As he understands it, one of the main focuses of current research is on how to reduce the size of the space that needs to be explored, "because it explodes quickly". He mentioned CHESS, Shuttle, and Loom as examples. The closest to blanket is perhaps Coyote, Hastings said.

His final slide was a picture of Mickey Mouse as a sorcerer from or inspired by Fantasia. As might be guessed, he used that as the backdrop to describe his use of "AI" (presumably LLMs) on the project. It was his first time working with those tools; "it has been an enormous amount of work and I was enormously more productive". The project has been "pair-programmed with AI"; "the tests are kind of vibe testing and the docs are kind of vibe docking". It was so productive, in fact, that he rewrote blanket five times along the way, making it simpler and improving the API as he did. Along those lines, Hastings gave a presentation at EuroPython 2026 (YouTube video) about blanket 2.0, which he has not, as yet, shipped.

There was a question about performance, but that is not something he cares much about because speed is not really a goal of the project, Hastings said. Blanket is meant for unit tests, not for running in production; if there are obvious inefficiencies, he will fix them if he can without harming the "intentionally not fast" use case. As the credo notes, the idea is to slow things down.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance with my trip to Long Beach, CA for PyCon US.]

Comments (3 posted)

Fixing the TCMalloc regression with RSEQ operations

The restartable sequences feature is one of the stranger corners of the kernel's user-space interface; it provides a way for user space to carry out simple lockless operations and be informed if it is preempted over the course of an operation (and must, thus, restart). Work merged in the 6.19 release to improve the performance of restartable sequences broke the TCMalloc allocator, which was relying on an undocumented (and unintended) kernel behavior. Now, Olivier Dion is proposing an addition to the restartable-sequences API that will bring TCMalloc back into the fold; it does not make the restartable-sequences API any less strange, though.

As a quick reminder of the problem: the restartable-sequences API is built around struct rseq, which is shared between user space and the kernel. The cpu_id_start field of that structure is maintained by the kernel; it contains the number of the CPU on which the owning thread is executing. It is documented as being read-only for user space. The TCMalloc library, though, wrote to this field anyway, setting it to a sentinel and using it, by way of some clever (to be polite) data-structure layout tricks, to notice when the kernel had reset it to a valid CPU number. That reset would indicate that the execution of the thread had been interrupted at some point. One of the optimizations in 6.19 eliminated the unconditional setting of that field, defeating this trick.

Until now, the problem has been worked around by detecting users of the older restartable sequences API and preserving the previous behavior, at a significant cost in both code complexity and performance. Using restartable sequences in this mode is also incompatible with how the GNU C Library uses it, causing other performance problems. For more details on the story so far, see this article.

It is clearly desirable to bring TCMalloc up to the current form of restartable sequences, restoring compatibility with the C library and performance as well. Dion's solution takes the form of "RSEQ operations", which is essentially the ability to program a set of operations for the kernel to carry out every time it returns to user space. An operation can be set up to restore the unconditional setting of the CPU number for TCMalloc in a way that does not hurt performance for any thread that is not using this new feature.

The simplest of the operations that are defined in this series is RSEQ_OP_RESET, which resets a word in user-space memory to a specific value. To set up such an operation, user space allocates and initializes a copy of this structure:

    struct rseq_op_reset {
	struct rseq_op_node	node;
	__u64			src;
	__u64			dst;
	__u32			len;
    };

The src field holds a pointer to the value to be stored, or simply zero if zero should be stored; dst is a pointer to where that value should be stored, and len is the length of the destination, either four or eight bytes. The node field is a structure of its own:

    struct rseq_op_node {
	__u64 next;
	__u64 prev;
	struct {
	    __u8  type;
	    __u8  reserved[7];
	};
    };

Here, type should be set to the operation type (RSEQ_OP_RESET in this case); everything else must be zeroed and not touched again thereafter.

There are two other operations defined as well. RSEQ_OP_RESET_WITH_STRIDE_CPUID sets a value into an array indexed by the current CPU ID, while RSEQ_OP_RESET_WITH_STRIDE_MMCID indexes by the current virtual-CPU ID instead. Either uses this structure:

    struct rseq_op_reset_with_stride {
	struct rseq_op_node	node;
	__u64			src;
	__u64			dst;
	__u64			dst_stride;
	__u32			len;
    };

The added dst_stride value is multiplied by the CPU ID to obtain the actual address to store the value into.

Once one of these structures has been set up, it can be registered with the new PR_RSEQ_OP operation to prctl(). If the second prctl() argument is PR_RSEQ_OP_REGISTER, the new operation will be registered; a value of PR_RSEQ_OP_UNREGISTER will unregister an existing operation. Once an operation has been registered, it will be executed on each return to user space. There is a limit of 2,048 operations total; one imagines that should be enough.

There is an interesting quirk to this API. Normally, with a registration operation of this type, one would expect that the describing structure would be copied into the kernel and used there. In this case, though, the data is left in place, and the next and prev pointers in the rseq_op_node structure are used to link the containing structure into a list maintained by the kernel. On each return to user space, the requested operations will be carried out by traversing the list, which remains in user-space memory. Unlike the rseq structure, the operations structures are not mapped into the kernel's address space; the normal primitives for access to user-space memory are used as needed.

This seems like a curious decision; all of those user-space accesses will surely slow the operations down, and user space will have the ability to change the contents of those structures (including the pointers) at any time. Quite a bit of care is taken in the code to validate this list and its contents at every step (increasing the cost further), but it still feels like dangerous territory. It would seem that the ability for user space to make changes is actually a desired feature in this case; the ability to change the value to be stored or the location where that value should go is evidently considered to be worth the cost.

Given that this series is, in the end, adding another virtual machine to the kernel (albeit an extremely simple one), one might wonder why Dion did not just implement the ability to run a BPF program on return to user space and be done with it. The patches do not address that question, but a plausible answer is that loading BPF programs requires privilege, while restartable sequences does not. Forcing every program that needs RSEQ operations to run with CAP_BPF would create problems of its own.

The patch series is so new that the bits are not yet dry; it is at the RFC stage and has not, as yet, received a lot of comments. Dmitry Vyukov asked about whether it would be better to keep the operations structures in the kernel. Florian Weimer, instead, worried that the feature "would turn into another bytecode interpreter over time, basically reimplementing BPF". As of this writing, Dion has not responded to these questions. There is definitely interest in changes that can enable TCMalloc to use the current API and, eventually, allow the removal of the compatibility code; how much this particular proposal will have to evolve before being merged remains to be seen, though.

Comments (11 posted)

Typst makes big strides

Typst is a system for typesetting documents into various formats: PDF, SVG, PNG, and, in progress, HTML. It is adept at handling technical material, and is often considered to be an eventual LaTeX replacement. We last looked in on Typst a year ago, when it had reached version 0.13. A new version, 0.15, was released in June with lots of new features, including support for variable fonts, MathML, multiple bibliographies, and more. Typst is free, Apache-2.0-licensed software, programmed in Rust.

Variable Fonts

Typst now has support for variable fonts. Most fonts are distributed in a set of files containing their glyphs in different weights, in variations such as italic, bold, and so on. A recent development in the world of typography is the advent of variable fonts, which can contain all their variations in a single file. This both saves space and can permit greater flexibility on the part of the author or designer.

Font features and variations are chosen by setting the value for an "axis"; each axis changes some aspect of the rendered glyphs. There are typically multiple axes that can have discrete values, for turning on and off various features, or continuous values lying between two limits. The latter can be used, for example, for choosing the weight of the font along a continuum.

To test the new Typst feature, I downloaded two open-source variable fonts: Roboto Flex, a general-purpose font with 13 axes, and Zycon, a font containing no letters but 17 small decorative pictures. Zycon's six continuous axes smoothly alter various aspects of the pictures, making the font useful in animations.

Here is a Typst document that uses both of these fonts, varying one of the axes for each of them:

   #set text(font:"Roboto Flex")
   #for n in (-305, -200, -98) {
       set text(variations:("YTDE": n)) 
       [A penguin jumped quietly.
   
       ]
   }
   
   #set text(font:"Zycon")
   #for n in array.range(0, 10, inclusive:true) {
     set text(variations:("M1  ": n/10))
     str.from-unicode(127773)
   } 

In Typst, a "#" character puts the document in "code mode", where the rest of the line or block is interpreted as code in Typst's built-in language. Within code mode, material enclosed in square brackets is interpreted as "text mode", or text to be typeset.

The code above contains two commands to set the fonts by name, using one of the options of the text() function. Next we have for loops, which operate as might be expected. Inside the loops we call the text() function to set the variations variable. The text to be used with the Roboto Flex font is entered directly, but the output using the Zycon font is a single character specified with the str.from-unicode() function. It could have been entered directly, but readers may not have been able to see it, depending on the coverage in their browser's font.

The axis that we manipulate in Roboto Flex is called YTDE. As the figure below shows, this axis determines the length of the font's descenders, while leaving its other characteristics unchanged. This might be useful when typesetting tables, for example, to avoid collisions between the descenders and the table-cell boundaries. One of Zycon's six axes, called "M1 " (the two trailing spaces are part of the name; all axis names contain four characters), does different things to different characters. The effect on the Moon glyph is to change the lunar phase.

Compiling this document with typst compile vfont.typ produces a PDF in vfont.pdf, which is shown in the screen shot below:

MathML

HTML export is still an experimental feature of Typst, but this release shows significant progress. The main new HTML feature is the translation of mathematics into MathML. The previous article on Typst showed the markup for a certain definite integral and displayed its output when rendered into a PDF by Typst. The same Typst code, when rendered into HTML, produces a long string of MathML markup that almost all reasonably current web browsers know how to interpret. The result appears like this:

$${\int }_0^1{\left(arcsin𝑥\right)}^2\frac{d𝑥}{{𝑥}^2\sqrt{1-{𝑥}^2}}=𝜋ln2$$

If you don't see an equation above, your browser does not support MathML. If you do see it, a comparison with the typeset equation from the previous article shows that the PDF and HTML results are essentially identical. The ability to use Typst to produce TeX-quality mathematics in web browsers, without requiring a JavaScript library such as MathJax or having to resort to images, is a boon for scientific communication.

To enable the experimental HTML output, Typst requires a special flag:

   typst c --features html integral.typ integral.html  

That is the command used to typeset the equation markup in a file called integral.typ into HTML.

"Bundle" output

The typical use case for software such as Typst is the creation of single files, usually papers or books in the form of PDFs. The new "bundle" feature allows the author to specify a collection of output files, in any of the formats that Typst supports, in a single source file. These files can share data and contain both intra-document and inter-document links.

The feature is ideal for the creation of web sites, which consist of an interlinked network of HTML files. It should also be of interest to academics who might want to generate a slide deck for a conference talk along with the associated preprint from a single source file.

Here is a simple example of a Typst file that creates three interlinked documents, two HTML pages and a PDF, sharing a fragment of text:

   #let text = ['Twas brillig, and the slithy toves
         Did gyre and gimble in the wabe:]
   
   #document("poems.html", title: [Famous Poems])[
     #link(<jabberwocky>)[Here] is a famous nonsense poem.
   ]<home>
   
   #document("jabberwocky.html", title: [Jabberwocky by Lewis Carroll])[
   This famous poem begins like this:
   
   #text
   
   The #link(<jabberwockyPDF>)[rest] of the poem.
   
   Go #link(<home>)[home].
   ]<jabberwocky>
   
   #document("jabberwocky.pdf", title: [Jabberwocky: the Complete Poem])[
   #text ....
   
   Go #link(<home>)[home].
   ]<jabberwockyPDF>   

The command for processing this code, if it is saved in a file called bundle.typ, is:

   typst c --features html,bundle --format bundle bundle.typ   

In my tests, the bundle feature worked as advertised, but since it, along with HTML output, is still considered a work in progress, the compiler requires the --features flag.

The command above creates a new directory called "bundle" containing the three files defined in the #document() functions. They all contain the two initial lines of the poem saved in the text variable. There are hyperlinks between the HTML pages, from one of those to the PDF, and from the latter back to the two-page website. The links are targeted using the labels, contained within angle brackets, following each document function.

Multiple bibliographies

Typst now permits multiple bibliographies in a single document, which was an eagerly awaited feature. Its canonical application is for books that may need a separate reference section for each chapter. The feature is best introduced with a toy example:

   #show bibliography: set text(size: 8pt)
   
   = Chapter I
   
   According to @smith, Smith is uncommonly smart.
   
   #bibliography("works.bib",
   title: "References for Chapter I",
   group: none)
   
   = Chapter II
   
   Jones@jones has a different view. The issue was
   finally put to rest in the following year
   in @mergutroid.
   
   #bibliography("works.bib",
   title: "References for Chapter II",
   group: none)   

Here the first line specifies that the bibliographies should use a font size smaller than the default used in the main text. In that text, the "@" prefixes create a citation using the default number-in-brackets style. At the end of each chapter, the bibliography() function is called. Its first argument specifies which database should be used for the bibliographic information; each bibliography section can use a different database, or collection of databases, if desired (see our recent article on Pandoc for a description of these text-file databases). The group argument controls how the citations are numbered. The value of none causes the numbering to begin with one for each section; numbering can alternatively be continuous for the entire work, or be grouped arbitrarily.

The figure below shows the output of the listing as it appears in PDF form:

Multiple PDF standards

Avoiding the use of proprietary extensions is normally sufficient to ensure that the PDFs created with Typst, LaTeX, or any other competent software will fulfill the promise of the format: documents will be openable and appear identical in all readers, now and in the future. At a deeper level, however, a PDF is not just a PDF. There are various PDF versions and, on top of these, dozens of formal standards relating to archivability and accessibility. The standards for archivability are meant to ensure that the document really does work across a wide variety of reader software and that it will do so forever. The accessibility standards relate to the usability of a PDF for people with disabilities of various sorts.

Typst already had the ability to target various PDF versions and standards for archiving or accessibility. The new feature is the option to target more than one when compiling a document. This is useful, because one may want to generate a PDF that has both archival and accessibility attributes. The implementation of the feature helps the author to navigate the forest of PDF standards by issuing warnings or errors in the cases of incompatible combinations or failure to follow best practices for the standards targeted.

As an example, here is a command that attempts to compile the "book" document from the previous section, requesting both PDF version 1.7 and the A-1b archive standard:

   typst c --pdf-standard 1.7,a-1b book.typ  

The Typst compiler responds with this error message:

   error: PDF 1.7 is not compatible with PDF/A-1b
   hint: PDF/A-1b requires version PDF 1.4   

PDF version 1.7 is compatible with A-2b, so this will work:

   typst c --pdf-standard 1.7,a-2b book.typ   

If, however, we add the UA-1 accessibility standard, as in this command:

   typst c --pdf-standard 1.7,a-2b,ua-1 book.typ    

We again get an error:

   error: PDF/UA-1 error: missing document title
    = hint: set the title with `set document(title: [...])`   

Normally the Typst compiler doesn't insist on anything beyond correct syntax, but if we specify a particular accessibility standard, the document must conform to that standard. UA-1 requires, among other things, a document title.

The foregoing is not merely arcana, although it may seem to be of little relevance to the typical author of a scientific paper or textbook. These details are important to archivists and publishers; in addition to helping disabled users, producing accessible PDFs is a legal requirement applying to state and federal governments in the US and many other countries. Typst's advanced handling of multiple PDF standards makes it a useful tool in these contexts.

Conclusion

Typst 0.15 has other enhancements that are not described in detail in this article. Some of these are support for spot colors, detailed diagnostics that explain any failure of convergence during the compilation process, and new map and filter functions in the built-in scripting language. The documentation, which is updated to cover the new version, is now available in a 26MB PDF.

Typst is developed on GitHub, where it has 460 contributors. The creators of the project have written a guide for new contributors where they describe what a PR should look like and warn that any code or description generated by an LLM will be rejected. They are also forthright about the fact that Typst is a company as well as an open-source project, and that decisions about the direction of the project, as well as the suitability of individual contributions, will take the needs of the company into account. This is a factor that prospective contributors and users should keep in mind.

Progress in the development of the system is impressive. The community of users is enthusiastic; their participation has expanded the Typst ecosystem to over 1500 packages.

However, network effects in the publishing world are preventing Typst from fulfilling the potential that we saw for it in our previous article. Although users have devised templates to match the style specifications of several journals, those that require source submissions (rather than just PDFs) still insist on LaTeX, Word, or some other format. Very few accept manuscripts marked up in Typst. This is not likely to change while Typst remains in a pre-1.0 status (which is probably still a ways off) and is in danger of requiring significant, possibly breaking changes to documents.

Despite this, Typst is an immensely useful tool today. For example, I recently had to create an SVG logo and found that writing a textual description using Typst's built-in graphics commands was quicker and easier than reaching for a drawing program. It's impossible to say whether Typst's advantages will lead to it becoming a "LaTeX replacement", as many of its admirers describe it. But, if development continues at the current pace, it is a distinct possibility, although one that may take a decade or two to come to fruition.

Comments (13 posted)