Reports from OSPM 2026, day two

(See also: coverage from day 1 and day 3).

Devfreq for uncore DVFS — Jie Zhan

The talk covered work on building uncore dynamic voltage and frequency scaling (DVFS) — encompassing L3 cache, interconnects, and memory controllers, but excluding I/O and GPUs — on top of the kernel's existing devfreq framework. The motivation is concrete: on the chips studied, uncore accounts for around 41% of SoC power consumption at idle and around 17% under a CPU-bound workload. A SPECpower run with uncore DVFS enabled saved 7–35% SoC power in the 0–50% load range without measurable throughput loss. The goal is reducing power with minor performance impact, particularly on server SoCs where no generic upstream solution exists.

After reviewing devfreq's structure and its five existing governors, the discussion focused on two challenges. First, frequency-scaling strategy: the simple_ondemand governor produced severe ping-pong between the minimum and maximum frequencies in experiments because it scales down too aggressively and the reported "load" itself scales with frequency. The proposal is a tweaked proportional/integral (PI) governor — using a proportional term for fast upscaling and an integral term for smooth downscaling — which stabilized frequency around 900 MHz with no throughput loss. Second, governor-driver coupling: devfreq passes only one type of governor-specific data at device registration, so adding or switching to a new dynamic governor isn't really supported. The maintainer's guidance was simple: send patches and discuss them on-list.

The audience engaged actively. Several attendees argued for trying a full PID controller rather than PI, noting the derivative term could anticipate spikes, though tuning was acknowledged as notoriously hard and workload-dependent. One attendee challenged the "good result" claim: with frequency parked around 900 MHz while load spiked to 60–80%, the governor may be preventing the hardware from reaching peak throughput — SPECpower can adjust its own load, but other applications or benchmarks could suffer under the same setup.

A recurring suggestion was using the existing interconnect framework with bandwidth and quality-of-service hints tied to CPU frequency, since it can aggregate constraints across CPU, display, and image signal processor. The counterpoint was that this interconnect is largely static today and that reactive load detection always lags — a scheduler-level hint at task enqueue (and a forthcoming latency-based QoS proposal referenced for the next day) may suit L3/uncore better. Another question was about per-scheduling-entity tracking; the response noted that related-CPU topology is already attached to the uncore device, so scheduler utilization could be pulled from there.

Three open questions closed the talk: defining common uncore event descriptions so governors aren't per-driver, pulling hints from CPU or I/O devices (scheduler utilization, enqueue-time signals), and restructuring the governor-driver data interface to decouple tuning parameters from a single governor.

Video: Devfreq for uncore DVFS - Jie Zhan (OSPM26)

Using AMU/PMU to reduce useless power consumption in CPUFreq — Hongyan Xia

Hongyan Xia described the fact that real CPU capacity often does not scale linearly with frequency. Depending on the workload, the bottleneck might not be in CPU capacity but rather in other parts of the system, mostly cache and DRAM contention. However, schedutil is unaware of such factors and will only try to raise CPU frequency to boost performance.

Xia attempted to use counters in the Arm64 activity monitoring unit (AMU) and performance monitoring unit (PMU) to inform the CPU-frequency governor when the CPU power is not the bottleneck and when raising the CPU frequency is not useful. His presentation included how to identify the most helpful AMU/PMU counters in such scenarios and how to use a linear-regression model with the proper counters to limit CPU frequency, achieving almost the same performance level with improved energy efficiency.

Video: Using AMU/PMU to reduce useless power consumption in CPUFreq — Hongyan Xia

Using the slice duration of fair tasks in CPU selection — Vincent Guittot

Vincent Guittot presented some results about using the EEVDF slice duration to order task scheduling while not breaking fairness. The custom-slice feature can be used to theoretically run tasks with short slices first, but figures show that the current scheduler still suffers significant latency outliers.

For his tests, Guittot used the cyclictest benchmark to evaluate the scheduling latency of short-running tasks and hackbench or rt-app to overload the system. Cyclictest sets an 8ms slice whereas hackbench and rt-app use a 20ms slice in order to emulate short, interactive tasks versus background activities. While latency remains acceptable on idle systems, performance degrades under stressed conditions, such as those generated by hackbench. These workloads involve high frequencies of wakeups, sleeps, and migrations. In overloaded scenarios, the maximum latency for a short-running task, as simulated by the Cyclictest benchmark, can reach approximately 9ms (near its full slice duration) even when it should be prioritized and the 99.9th percentile of latency often exceeds 4ms.

This suggests that the current EEVDF implementation fails to preempt running tasks in a number of corner cases:

• Negative lag preservation: tasks often stay in a queued state to pay back their overconsumption of CPU time (measured as "negative lag"). However, if new tasks are enqueued during this period, an existing task's negative lag can actually increase. The proposed fix ensures that, when a task re-enqueues, its negative lag is at most what it was when it went to sleep, preventing it from being unfairly penalized for system activity that occurred while it was inactive.
• Next-buddy shortcut: the scheduler sometimes uses the buddy mechanism to favor a specific task or group, which bypasses the EEVDF's selection of the most eligible task. By clearing the buddy if EEVDF would have picked a different task, the scheduler ensures that preemption remains consistent for shorter-slice tasks.
• Delayed-dequeue bias: the delayed-dequeue tasks, which wait on the run queue to become eligible, often have short deadlines and can easily be the next to be picked to run. They can thus prevent shorter-slice tasks from preempting the current task. To address this problem, those delayed-dequeue tasks are actually dequeued when checking for wakeup preemption.

With these corner cases fixed, the 99.9th percentile of latency moves below 700µs, except for one case, but some cases still need further studies:

• Newly enqueued entities: between the preemption check and the pick of the next entity, new tasks can be enqueued, moving the average vruntime and making the preempting task not eligible anymore. Using the next buddy to force picking the task that triggered the preemption needs further study.
• Permanent positive lag: while negative lag can be decayed with delayed dequeue, the positive lag lasts forever and gives an unfair advantage. Decaying the positive lag according to the sleep duration is under consideration. Tracking the uncontended time of a CPU to reset all lags is another option to evaluate.
• Disabling run to parity improves the maximum scheduling latency because the scheduler can switch tasks more aggressively and keep minimum lag to all tasks, but at the cost of system throughput. Furthermore, any task that needs to run for more than 0.7ms (the default base time-slice value for the fair scheduler) will be preempted before finishing its work, delaying the time to finish a short, but longer than 0.7ms, computation.

In addition to all previous changes, the logic for CPU selection should follow the sequence:

1. Idle CPU Search: an idle CPU remains the best target, offering immediate execution and avoiding the overhead of preemption.
2. Minimum slice comparison: if no cores are idle, the scheduler compares the minimum slice duration of all enqueued tasks on the target CPU. The goal is to place the waking task on a core where its slice is shorter than the minimum, maximizing its chance of preempting the current task.
3. Energy-aware scheduling (EAS): On mobile platforms, the scheduler must weigh these latency gains against the energy cost of waking up a specific power domain or shifting to a more expensive core.
4. A push callback mechanism: since the window between selecting a CPU and the task actually being enqueued is not atomic, a new task might arrive and steal the eligibility of our waking task in the interim. The push callback acts as a fallback: if the task arrives on a CPU and finds it is no longer the first to run, it is immediately placed in a list to be pushed to a better core.

Video: Using the slice duration of fair tasks in CPU selection — Vincent Guittot

Rethinking multi-cluster scheduling domains for Arm64 servers — Dietmar Eggemann

One of the main concerns is that these Arm64 systems often expose only a single large scheduling domain. This limits the scheduler's ability to make effective wakeup and placement decisions. Even though the system is fully cache-coherent, latency across the mesh is not uniform, and this information is not currently visible to the scheduler. As a result, potential locality benefits are not being utilized.

Initial experiments suggest that scheduling-domain size can influence performance, with a tradeoff between the overhead of searching for idle CPUs and the quality of placement decisions. The impact is highly workload-dependent, and early exit conditions from the task-wakeup functions make these effects difficult to observe and reason about.

A number of possible directions were discussed, including splitting large domains into smaller clusters, making better use of firmware-provided topology information, and exploring ways to derive structure from latency measurements between CPUs or memory of the mesh. Each of these comes with its own challenges, particularly around complexity, stability, and avoiding regressions across different workloads.

Overall, there was agreement that the lack of exposed and usable topology in large Arm64 systems is a limitation for the scheduler today. Ongoing work on this topic will further explore the approaches mentioned above to try to find a robust and generally applicable solution that will require coordination between hardware, firmware, and OS teams.

Video: Rethinking multi-cluster scheduling domains for Arm64 servers — Dietmar Eggemann

Evolving sched_ext: resource control, topology awareness, and energy efficiency for modern systems — Changwoo Min and Gavin Guo

Changwoo Min and Gavin Guo gave a joint talk covering two improvements to scx_lavd — the Latency-criticality Aware Virtual Deadline scheduler built on top of sched_ext. When Min first presented scx_lavd at OSPM 2025, it was a gaming-focused scheduler aimed at improving Windows games running on Linux through SteamOS, with waker/wakee frequency as its primary hint for task urgency. A year later, the project is broader, expanding scx_lavd into a potential default fleet scheduler, and that expansion has highlighted two parts of the scheduling problem. The first is support for control-group-v2 CPU-bandwidth control — the cpu.max interface — which multi-tenant systems (containers, VMs, cloud workloads) need in order to enforce hard CPU quotas. The second is a load balancer that understands both task-size heterogeneity and CPU-capacity heterogeneity.

Sched_ext cpu.max — moving the work off the critical path

Min opened with a quick recap of what cpu.max is supposed to do: it allows administrators to specify a (quota, period, burst) tuple per control group, and tasks in an over-quota group are throttled, meaning that they are dequeued and parked until the next period boundary. He quickly reviewed the three design aspects of the kernel's existing implementation that motivated his approach toward improving its performance.

The first is task selection. The kernel mirrors the control-group hierarchy as a nested red-black tree, so picking the next task to run becomes a walk down that nested tree. As Min explained, the cost of task selection grows linearly with the depth of the hierarchy.

The second is throttle detection. The current kernel implementation uses what Min called a "synchronous pull model" — each CPU borrows a 5ms slice from a central, per-control-group quota pool, consumes it locally, and pulls more when it runs out. When the local budget is exhausted and the group itself has no more quota, the CPU must walk up the control-group hierarchy to find a source of budget. The result is that the throttle check on every dispatch is expensive, touches global memory, and again increases with hierarchy depth.

The third is replenishment. The kernel uses two timers per group — a period_timer that refills the quota every period, and a slack_timer that returns unused local budget to the global pool asynchronously. The total number of timers in the system therefore grows linearly with the number of groups. The sched_ext cpu.max library Min built reorganizes all three concerns around a single principle: get the expensive work off the dispatch path. The library is exposed as lib/cgroup_bw and can be linked into any sched_ext scheduler; scx_lavd is its first consumer.

For task selection, the library keeps all unthrottled tasks in the regular dispatch queue (DSQ). Only throttled tasks are moved aside, into a per-control-group backlog task queue (BTQ). Task selection on the unthrottled DSQ therefore stays O(log N); the nested red-black tree disappears entirely. The throttle check on the hot path becomes a single flag read — no locks, no atomics, no hierarchy walk.

That last simplification is only possible because the library trades the kernel's accurate and immediate single-period enforcement model for what Min called eventual bandwidth control across multiple periods. A control group is allowed to overspend for up to one accounting interval; overspend is captured as debt and exactly subtracted from the next period's budget. Min was careful to emphasize that long-run-average usage still converges exactly to the configured quota.

To keep the detection latency small, the library arms an adaptive accounting timer. Each control group maintains an exponentially-weighted moving average of its consumption rate; the timer predicts when the group will hit its limit and fires early enough before the group is throttled.

Finally, the library normalizes every control group's period to a fixed, 100ms window. With every group using the same time unit, a single replenishment timer can serve the entire system — total timer count drops from two per group to two for the whole machine.

Min showed results from a 2-socket, 96-core AMD EPYC machine (192 CPUs) running stress-ng --cpu. He explained the scheduler overhead, measured as kernel-time CPU cycles during a pure user-space workload. As control-group depth increased from one to  32, EEVDF's overhead grew from roughly two CPU-equivalents to roughly five, while scx_lavd stayed flat at about two. At a control-group depth of 32 under a load sweep, EEVDF overhead spiked above ten CPU-equivalents at 125% load while scx_lavd remained near two.

The first question came from Andrea Righi, who pointed out that the BTQ is currently built on top of arena task queues, which, in turn, are built on BPF arenas. On older kernels that lack arena support, this becomes a portability problem; Righi asked whether the same machinery could be built on DSQs. Min said it should be possible in principle, but that DSQs currently lack an API for moving a task directly from one DSQ to another — the only existing transfer path is from a BPF DSQ to a local DSQ for execution. Iterating across DSQs would work, he said, but is expensive. Adding a direct DSQ-to-DSQ task-move kfunc to sched_ext would let BTQ shed its arena dependency.

An audience member asked whether any of these ideas could be contributed back to the kernel. Min replied that he was open to that discussion, but had wanted first to confirm that the design worked as intended; with several months of evidence that the overhead and accuracy numbers behave as expected, he said collaborators interested in improving the kernel's cpu.max implementation were welcome.

Another audience member asked whether the comparison had been run against the post-rework EEVDF. Around September 2025, the kernel's cpu.max throttling was reworked so that throttling is deferred to the user-space return boundary, with debt carry-over already implemented on the kernel side. Min acknowledged that his baseline predated that rework, and agreed that the EEVDF overhead numbers would be lower against the current kernel. The depth-independence of throttle detection and the elimination of per-group timers should remain wins regardless, but the specific overhead delta will need to be remeasured.

Another audience member asked whether the hierarchical accounting was inherent, or whether a flat model could avoid the bottom-up and top-down tree walks altogether. Min's answer was that the control-group interface is hierarchical, so the accounting has to be; the hierarchical work, however, runs only in the background accounting timer, not on the dispatch hot path.

Task-size-aware load balancing in scx_lavd

Guo then took over to describe the second improvement: a new load balancer for scx_lavd. He began by reminding the audience that scx_lavd is a domain DSQ scheduler — each last-level-cache (LLC) domain has a single DSQ shared by all of its CPUs, and domains are assigned "stealer" or "stealee" roles every 10ms based on their queued load relative to the system average. When a CPU's local run queue drains, ops.dispatch() first tries to steal CPU time from a remote stealee, then takes a task from the local domain's DSQ, and finally falls back to a force-steal from any nearby DSQ.

The current balancer has three weaknesses. Its load metric is the sum of utilization and the scaled queue length, which Guo said was obviously problematic — a domain with ten short-lived tasks looks busier than one with two long-running tasks, even though the latter is doing more work. There is no migration budget, so many CPUs race to drain the same overloaded domain in one round, with only probabilistic gating to limit the resulting thundering herd. And migration is task-type-blind: large tasks can land on small cores.

The new design replaces the load metric with queued_load_invr + util_invr, where queued_load_invr is the sum of task sizes in the domain rather than the count of tasks, and util_invr captures how busy the domain's CPUs currently are. The "invariant" qualifier matters on heterogeneous systems: run time is scaled by CPU capacity and frequency, so loads on P-cores, E-cores, and LP-E-cores are directly comparable. Each domain is then given a capacity-proportional fair share. For example, a domain with 40% of the system's capacity is expected to carry 40% of the total queued load; big-core domains, having more compute capacity, naturally carry more.

Migration is bounded by a symmetric 50% budget. A stealee allows half of its excess load above fair share to be migrated out; a stealer accepts half of its deficit below fair share. Closing only half the gap in any single round avoids the thundering herd problem that would occur if the full imbalance were corrected at once — the stealee would simply become the next stealer.

Guo showed schbench-wakeup-latency results on two machines, with six runs each. On a heterogeneous, 14-CPU Meteor Lake system, p99 dropped from 5,867µs (scx_lavd main) to 5,613µs (−4.3%), and p99.9 from 9,899µs to 9,195µs (−7.1%). On a homogeneous, 192-CPU AMD EPYC 9R14, p99 dropped from 5,867µs to 5,741µs (−2.1%), and p99.9 from 7,297µs to 6,777µs (−7.1%). The capacity-aware aspect of the design helps more on heterogeneous platforms, as expected, but the migration budgeting and task-size metric — which are independent improvements — transfer cleanly to the homogeneous case.

An audience member followed up with a question about how capacity is computed on SMT systems. The Meteor Lake box, he pointed out, is not the pure capacity-asymmetric model that EAS targets — it is also an SMT system, and scx_lavd currently treats the four logical CPUs in the P-core domain as fully independent. He asked whether it is fair to assume they have the same capacity and are independent. The answer was that they are not independent; SMT siblings impact each other. Once both siblings of a physical core are busy, the effective capacity of each drops, and the relationship is workload-dependent (two compute-bound siblings interfere more than one compute-bound and one memory-bound). Guo agreed that the current model is an overcount and said he wanted to enhance it in the future. However, in a domain-based scheduler, the problem matters less than performance overhead, as the load balance is based on the domain capacity instead of the CPU capacity.

Ricardo Neri then explained how EAS-style schedulers handle the same problem. CPU capacity is not used for SMT systems, he said — busy/idle transitions are too fast and the overhead of tracking sibling state would be prohibitive. Instead, scheduling on SMT cores is done by priority; a CPU's priority is proportional to its capacity, but SMT siblings are assigned the lowest priority and are populated last. On Meteor Lake configurations where SMT is disabled, the question does not arise; where SMT is enabled, EAS relies on priority alone. Guo said this was exactly the kind of input he had hoped to get from the talk, and added that he would welcome more suggestions from Intel and Arm engineers on what else the model should capture.

The discussion closed with Guo previewing the next step: selective migration. Today, when a stealer pulls from a stealee, it always picks the head-of-DSQ task — which may not be the best candidate to migrate. The plan is to peek at the next two-to-eight tasks in the DSQ and choose based on task size (matched to domain capacity), latency criticality, cache locality, and waiting time (so long-waiting tasks can be rescued to bound tail latency), while skipping tasks that are still cache-hot using a task-hot guard, analogous to the completely fair scheduler's task_hot(), with a roughly 500µs threshold.

Video: Evolving sched_ext - Changwoo Min and Gavin Guo

Steam deck on large servers (continued) — David Dai and Ryan Newton

David Dai's portion of the presentation explored the performance of the scx_lavd scheduler on large servers, specifically analyzing the waker/wakee heuristics in production environments. In a user-facing web service involving a complex chain of wakeups, scx_lavd successfully identifies threads with frequent short wakeups as highly critical. Prioritizing these tasks reduces tail latencies during periods of burst traffic or CPU contention. However, a tradeoff exists: scx_lavd relies on shared dispatch queues per last-level cache, which increases task migrations, negatively impacts L1/L2 cache locality, and results in a small throughput penalty.

The heuristics yielded unintended results in a second case study involving a caching service. This service purposefully delays read tasks to batch them together, lowering their wake-up frequency compared to writer tasks. Consequently, scx_lavd assessed the writers as more latency-critical, creating a heuristic inversion. During the Q&A, audience members like Peter Zijlstra and Guittot suggested that developers could resolve latencies in EEVDF by manually setting the time slice for tasks based on their specific work cycle, a solution Dai acknowledged that he could try and test.

Another issue identified in the caching service was the impact of software interrupts. Cache workers could spend up to a quarter of their running time processing these interrupts instead of their primary tasks. To mitigate this, Dai proposed categorizing CPUs into "steady" CPUs and "turbulent" CPUs. When an audience member asked how to define a turbulent CPU, Dai explained that the current threshold categorizes any CPU spending 15% or more of its time processing interrupts as turbulent, which accounted for roughly a third of the cores in their tests. By calculating a "preemption vulnerability" score that combines a task's latency criticality and utilization, the system can steer highly vulnerable and critical tasks away from turbulent CPUs. This latency-aware placement improved overall tail latencies and better balanced the load across cores.

Ryan Newton's segment of the talk shifted the focus to creating testing environments that move scheduler development away from production systems. His goal is an "abstract, fix, test" loop where production traces are simplified into portable reproducer workloads. This pipeline involves taking production traces, creating a simplified Rust program, running it in a smaller virtual machine topology, and then translating it into a JSON-based workload specification using RT-app. Ultimately, these specifications are executed in SCX-SIM, a user-space simulator.

SCX-SIM does not use a live kernel; instead, it emulates kernel functions via C stubs. This allows SCX-SIM to be bitwise-deterministic and portable, operating three to ten times faster than real time while simulating multiple cores on a single core. Furthermore, SCX-SIM employs controlled concurrency testing. Although the simulator executes sequentially, it artificially interleaves chunks of task execution at controlled, randomized preemption points to expose race conditions and test for concurrency bugs. Addressing simulator accuracy during the Q&A, Ryan noted that, while SCX-SIM introduces randomized delays, to model context-switching overhead for example, , it lacks memory and cache modeling. An audience member suggested recording real instructions-per-cycle penalties during task migrations using hardware performance monitoring unit counters to feed back into the simulator, which Newton agreed would be a highly valuable addition.

The simulation workflow incorporates AI coding agents to help close the gap between the simulator and production. AI agents iteratively adjust the simulator's calibration parameters — such as wake frequency and time-slice distributions — to ensure its output traces closely match actual Perfetto traces from production. Additionally, agents are directed to write unit tests for specific branches of the scx_lavd scheduler, which has increased the test code coverage up to 77%. The audience showed interest in this AI-driven approach, prompting discussions about open-sourcing the prompts and configurations so the broader community could collaborate on simplifying production traces. Finally, when questioned about the human's role in this loop, Newton concluded that humans are primarily needed to establish external guardrails that prevent AI hallucinations and to correct the AI when it makes errors

Video: Steamdeck on large servers (cont.) - David Dai and Ryan Newton (OSPM26)

Hierarchical constant bandwidth server: current state and future challenges — Yuri Andriaccio

This talk presented the latest updates of the patch set that is aimed at replacing the realtime group scheduler with the hierarchical constant bandwidth server (HCBS) mechanism. The patch set was originally presented at OSPM 2025, and has since been sent to the kernel's mailing list to gather comments on its implementation. At the time of this talk, the latest proposed version was RFC v4.

The talk first focused on what the hierarchical constant bandwidth server is, and why it matters. HCBS reworks realtime group scheduling using deadline servers, introducing them to the control-group-v2 world (and dropping v1 support), significantly reducing code footprint, and reusing existing subsystems. Other practical improvements are focused on the realtime soundness of the scheduling algorithm, better control on bandwidth allocation, and the possibility of the execution of unprivileged FIFO or round-robin realtime tasks.

The implementation details are not different from what has already been discussed at OSPM 2025. The general idea consists of allocating a number of deadline servers and run queues for each control group, one for each physical CPU. The servers provide the bandwidth reservation for each CPU, and, whenever they are picked for execution, they recursively invoke the FIFO/RR scheduler on the new control-group-specific run queues.

HCBS is constantly being worked on, with its latest version (at the time of this talk) based on kernel version 6.18. It has been actively reviewed by the scheduler maintainers and other contributors. A growing set of HCBS-specific tests is also constantly being updated and executed to make sure that critical code sections are tested, and that temporal and isolation guarantees are provided.

The talk covered some of the issues that arose during the development of the patch set, like integration with CPU-hotplug mechanism and frequency scaling. While they were just a broad idea in 2025, multi-CPU control groups were further investigated and discussed in this talk. Given that the new HCBS mechanisms are going to be only usable with v2 control groups, it was suggested to integrate the cpuset and CPU controllers to implement partial reservations — to allow execution of FIFO/RR tasks on a subset of the CPUs, while also updating the admission tests for this setup. The original multi-CPU idea also features different budget and period reservations on a per-CPU basis, but integration with existing subsystems is still an open problem.

Another discussion arose about the current meaning of the sched_rt_{runtime/period}_us sysfs knobs. These originally were used to specify the maximum bandwidth allowed for realtime tasks and to implement the realtime throttling mechanism. That throttling was removed in kernel version 6.12, in favor of fair deadline servers. Since then, the sched_rt settings only limit the maximum allocatable bandwidth in the deadline scheduling class, but do not affect FIFO/RR tasks. The accepted solution was to just update the default bandwidth for deadline entities to be 100%, which has been recently merged in the sched/tip branch of the kernel.

A final discussion arose on the meaning and possible substitutes of the current deadline-scheduler admission test. That test in fact does not guarantee that every task will respect its deadlines, instead it only guarantees that the response time of deadline tasks is not unbounded.

Video: Hierarchical CBS: current state and future challenges - Yuri Andriaccio (OSPM26)