Reports from OSPM 2026, day three

(See also: coverage from day 1 and day 2).

GPU-aware auto-affinitization with sched_ext — Andrea Righi and Balbir Singh

On systems with multiple GPUs and multiple NUMA nodes, CPU-to-GPU locality often becomes the dominant performance factor for AI and accelerator-heavy workloads. A task whose CPU threads run on one NUMA node while it drives a GPU attached to another pays a real cost on every transfer between the CPU and the GPU. Today, users work around this manually with numactl pinning and by disabling NUMA balancing, but this amounts to ad-hoc static partitioning and depends on the user knowing the system's topology. This talk described an experimental sched_ext-based approach that tries to do this automatically.

A prototype has been implemented in the scx_cosmos scheduler. A small user-space component, written in Rust, queries NVIDIA's NVML library to track per-task GPU memory and compute utilization. When both metrics exceed a threshold, the task is considered to be actively using a specific GPU, and an entry is added to a BPF map mapping the task to its preferred NUMA node. The BPF scheduler then consults this map during task rescheduling, migrating the task toward CPUs that are local to the GPU it is actively using.

Results on a four-GPU NVIDIA GB200 system running a RegNet image-processing workload were encouraging. The default fair scheduler reached around 56 frames per second (fps), while manual numactl pinning to the correct node pushed that to about 77fps. Scx_cosmos with GPU auto-affinitization slightly exceeded even manual pinning, reaching up to 80fps. The small extra gain came from a useful side effect: only the tasks actually touching the GPU were migrated, leaving auxiliary tasks free to spread across the other nodes and avoid overloading the CPUs that are local to the GPUs.

Several caveats remain. Migrating a task closer to its GPU does not move the memory it has already allocated, so accesses can remain remote unless NUMA balancing or an explicit numactl call follows. Multi-threaded workloads with shared state may be hurt by migrating only the GPU-active thread. And aggressive packing onto a single NUMA node can conflict with the scheduler's load balancer. Some of these can be addressed in user space (signalling numactl from the BPF side once a migration is decided, for example). For the rest, Peter Zijlstra suggested a much simpler kernel-side option: a per-task knob that fixes the preferred NUMA node and disables NUMA-balancing scans, which would generalize cleanly beyond GPUs to network interfaces, storage, and other devices.

A recurring theme in the discussion was that statistical inference from utilization counters is fundamentally post-facto: by the time the scheduler concludes that a task is using a specific GPU, the misplacement has already cost performance. The clear next step is to obtain hints from the user-space framework managing the device, CUDA being the obvious example, so that placement decisions can be made before the workload starts. Such hints can also be two-way: the kernel side can validate them against observed behavior and report back when a user-space framework is asking for something nonsensical.

The talk closed with several speculative directions: device-aware fairness would extend vruntime-style accounting beyond CPU time to also capture GPU (and more generally device) utilization, so that a task burning GPU cycles without much CPU usage does not appear "light" to the scheduler. Periodic GPU workloads, such as graphics frames, look like natural candidates for deadline-based scheduling. Embedded and mobile use cases bring thermal coordination between CPU and GPU into scope. And virtualization, with pass-through devices and paravirtualized scheduling, opens an entirely separate problem space that the sched_ext prototype has not yet touched. The unifying observation is that as accelerators become the primary consumers of CPU work, the scheduler needs a richer view of where devices live and what tasks are doing with them; sched_ext is a convenient place to experiment with that view.

Video: GPU-Aware Auto-Affinitization with sched_ext - Andrea Righi, Balbir Singh (OSPM26)

Profile-guided CPU scheduling — Kumar Kartikeya Dwivedi

This talk was centered around the premise that CPU schedulers often make workload-sensitive decisions with limited workload-specific information, leading to poor system efficiency. Scheduler heuristics, such as cache-aware placement or migration-cost heuristics, can work well for one workload but fail to generalize. The core problem is that the scheduler usually does not know whether threads share data, whether request phases have distinct locality, how much scheduling latency a task can tolerate, or how quickly useful cache state decays after a task stops running.

The proposed direction in this talk was profile-guided scheduling: collect offline and online signals about workload behavior, then use those profiles to parameterize scheduling decisions. The talk focuses on cache locality for illustrative purposes.

The main prototype shared in the talk augments a sched_ext scheduler, scx_layered, with a profiling tool that samples memory accesses with perf, clusters threads by similarity of accessed physical addresses, and feeds the resulting groups into the scheduler. Run-time classification avoids unstable identifiers like process or thread IDs and, instead, uses thread names plus application-provided metadata such as request phase or request type.

A central mechanism used to ensure locality is "soft partitioning": dynamically sized, topology-aware CPU sets for groups of threads that appear to share data. Unlike hard partitioning, these partitions can adapt to load and preserve work conservation within and across the soft partition. The scheduler assigns cache domains, especially L3/LLC domains, to groups while resizing allocations based on observed demand.

The main case study is Meta's web-serving backend running HHVM. The workload is sensitive to scheduling behavior, including work conservation, involuntary context switches, and cache locality. On multi-CCX AMD systems, the approach separates CPU-heavy worker threads, I/O threads, and management jobs on the machine, then uses locality-derived grouping and request-phase metadata to guide placement.

The evaluation shows about 3-4% throughput improvement over the scx_layered baseline, 5-8% lower average latency, 6-7% fewer L3 misses, and 2-3% better IPC. It also provides roughly 7% better throughput than EEVDF. Dwivedi noted that a 1% gain is already considered significant for this service, so these results are practically relevant for the evaluated workload.

Several technical insights were presented. First, average utilization at 100ms-1s granularity is often the wrong signal for partition sizing; tail utilization at finer time granularity better captures bursts and avoids undersized partitions. Second, locality decisions are load-dependent. At low load, taking any idle CPU is usually better than waiting for cache affinity to achieve better results. Near saturation, it may be worth waiting briefly for a CPU in the previous cache domain if the expected locality benefit exceeds the scheduling delay. Third, request phases can have distinct locality and latency importance, so separating early latency-critical request work into its own soft partition can reduce latency. However, these phase partitions should collapse again into a single soft partition without distinction once the system exceeds its service-level objective and maximizing bandwidth is the primary goal.

The future-work section of the talk focused on highlighting unresolved questions around temporal locality and cache decay. The current approach identifies spatial sharing, but not how long cached data remains useful or how likely a task is to reuse it. Dwivedi raised questions about whether miss rate, hit rate, reuse distance, memory-access samples, or run-time perturbation experiments are the right signals. L1 and L2 locality appeared hard to exploit robustly, while L3 looked more promising because useful locality persists longer relative to scheduler decision time.

The audience discussion explored these open problems in more detail. Questions focused on whether hit rate is a better signal than miss rate for temporal reuse, whether isolation experiments translate to production, whether page faults are too expensive for run-time profiling (yes), whether L2 accesses might better characterize pressure on L3, and whether hard memory/cache partitioning would be preferable. Other suggestions included using reuse-distance analysis from richer offline traces, deliberately perturbing scheduling to measure workload sensitivity, and using stack traces or program phases to improve persistent thread identity across runs.

Overall, the talk presented profile-guided scheduling as a way to make scheduler heuristics depend on measured workload behavior rather than fixed assumptions. The key idea is to learn when locality matters, which threads should be grouped or separated, and when the latency/throughput tradeoff changes with load.

Video: Profile-Guided CPU Scheduling — Kumar Kartikeya Dwivedi

A scheduler scorecard — Steven Rostedt

The idea behind the scheduler scorecard is to rate the characteristics of the scheduler and not to add just another benchmark. For example, Rostedt brought up the characteristic of how often a scheduler migrates a task. On some architectures, migration can be expensive, but if a system has a large L3 cache that is shared among several CPUs, the cost of migration may be minimal. A benchmark for the same scheduler on two different system configurations can show drastically different results; one may show the scheduler with a high performance rate and on the other system, it would be very slow. He brought up other characteristics, such as how often a task is blocked, CPU-frequency changes, etc.

Rostedt stated that benchmarks may show how well a scheduler behaves on a particular setup but do not explain why it behaved that way. After running a benchmark on a workload, one can guess why one scheduler algorithm worked better than another but, without seeing what the scheduler actually does, that guess may be wrong. Having a scorecard that shows the characteristics of a scheduler can help determine exactly why the scheduler performed the way it did.

To support the scheduler scorecard concept, Rostedt has written has a small utility that measures various metrics. It is based on his library libtraceeval which, he stated, is not quite ready for prime time as he is not comfortable with the current API. This utility records how long each CPU is idle rather than running a task, For each process, it reports the maximum, minimum, average, and standard deviation of the time spent running on the CPU, time spent being preempted (another task is running when the task is on the run queue), wakeup latency (time spent between being woken up until it runs on the CPU), time a task was blocked (it is in the TASK_UNINTERRUPTIBLE state), and the time a task was sleeping (it is in the TASK_INTERRUPTIBLE state). It also shows the count of times the task was in each of those states. It then shows the same metrics for each of the process's specific threads. For the process, it also shows how many times the process migrated.

A real-world example of benchmark confusion came about when a member of the Pixel team tried out PREEMPT_RT and ran benchmarks on it. They used Geekbench 6 to do the benchmark; PREEMPT_RT performed 13% worse than a kernel without PREEMPT_RT enabled. A record of trace-cmd was executed on the Pixel device while running Geekbench 6 with and without PREEMPT_RT; Rostedt then used his utility on the trace.dat files produced by trace-cmd. It showed that the PREEMPT_RT run had the tasks preempted more often and for longer than without realtime, which is expected due to the way PREEMPT_RT works. This also caused tasks to be scheduled in and out much more often. The wakeup latency was pretty much the same.

One of the biggest differences was that, with PREEMPT_RT, the tasks were blocked for a much longer time than without it. This is expected, as in PREEMPT_RT, spinlocks are converted to mutexes; a task would block on contention on a realtime kernel but would not block in non-realtime as it would simply spin. The utility showed the impact of spinlocks being converted to mutexes for this particular benchmark. With further analysis, the impact of contention with the converted spinlocks was proven to be the culprit of the performance degradation in the benchmark.

Rostedt then brought up questions to the audience about what else could be recorded. He gave a list, including CPU-frequency changes, NUMA mappings, cache misses, and wakeup chains (recording what tasks wake up other tasks). There are changes to ftrace to record perf events like cache misses into the trace buffer so that the cache-miss count can be displayed at every scheduler switch. But the interface for this is not ready to be submitted upstream. Juri Lelli brought up the use of timerlat. Rostedt stated that it could also be incorporated and is not mutually exclusive to the tracing he is working on. The trace is still required to keep track of hundreds or thousands of threads, which would not be something to do inside the kernel. Rostedt also mentioned the ability to do the analysis offline and not depend on the kernel performing all of the calculations.

The session ended with various discussions about overhead of the utility, what events can be used, and other various enhancements that can be made to the tracing subsystem.

Video: Scheduler score card - Steven Rostedt (OSPM26)

Paravirtualized scheduling: a framework for better CPU usage — Shrikanth Hegde and lya Leoshkevich

The discussion started with a brief description of problem that occurs in virtualized environments with multiple virtual machines (VMs), where there is an overcommitment of CPU resources and high CPU utilization in many VMs simultaneously. Due to this overcommitment, the host cannot meet the CPU requirements, leading to vCPU preemptions, which can have a high performance cost.

A quick glance at the proposed solution was provided and discussed. The idea is simple: "steal time" — the amount of time a runnable vCPU waits for the physical CPU to become available — is a well-known metric in the VM world that indicates CPU contention. When the guest detects high steal time, it reduces its vCPU request by using fewer vCPUs. This means the workload is dynamically adjusted to use a limited set of CPUs instead of all available ones.

The lack of existing methods to fully fix the issue was discussed, followed by implementation details. A new CPU state called cpu_preferred was introduced, and its design constructs and advantages were reviewed. The talk also covered the s390 perspective of integrating this work with existing mechanisms like HiperDispatch and warning track interrupts.

Performance numbers across PowerPC, s390, and x86 systems were presented, showing significant gains in real-life workloads and microbenchmarks without major regressions. The challenges regarding upstreaming this work were then discussed. There was a general consensus among the audience that this approach is viable, and patches will be reviewed. Suggestions regarding implementation were provided, which the authors will consider for subsequent versions to be sent out soon.

Current version: https://lore.kernel.org/all/20260407191950.643549-1-sshegde@linux.ibm.com/

Video: Paravirtualized scheduling: a framework for better CPU usage — Shrikanth Hegde and lya Leoshkevich

Platform QoS — Ionela Voinescu

The talk presented an RFC for integrating platform-specific resource prioritization into the Linux quality-of-service (QoS) subsystem, focusing on ACPI CPPC resource-priority registers and SCMI QoS. The main problem described was that firmware already makes platform-specific decisions about how power or thermal headroom is distributed, but Linux has limited ability to indicate which CPUs or domains are most important for the active workload.

ACPI CPPC resource prioritization provides per-logical-CPU priority controls for resources such as processor boost, processor throttle, cache access, and memory bandwidth. These priorities allow the operating system to indicate, for example, which CPUs should receive boost preferentially, or which CPUs should be throttled later under constrained conditions. SCMI QoS provides a similar mechanism through SCMI performance domains, with support for boost and throttle prioritization using either relative priorities or weights. Unlike CPPC, SCMI QoS is domain-based and does not cover cache or memory-bandwidth prioritization, which are expected to be handled through mechanisms such as resctrl.

The discussion points were mainly around the shape of the Linux interface. One option is an administrator-oriented interface, similar in spirit to resctrl, where privileged software configures priorities explicitly. This is more suitable for server-oriented use cases with pinned workloads and relatively stable policies. Another option is a more dynamic scheduler or task-driven model, where priorities are inferred from existing kernel signals such as control groups, uclamp, or future task attributes. This may be more suitable for edge use cases, but only if firmware transport and reaction latency are sufficiently low.

A key point raised in the discussion was that the interface should be motivated by concrete use cases rather than by the existence of new firmware controls. The clearest example discussed was a power or thermally constrained mobile-gaming scenario, where the system may need to protect the CPU running the main game thread from throttling, or direct available boost budget toward it.

There was broad agreement that cache and memory-bandwidth prioritization should not duplicate existing mechanisms such as resctrl. The more relevant gap is boost and throttle prioritization, where cpufreq can request performance levels but cannot express which CPU should receive opportunistic boost or be throttled last.

The main outcome of the discussion was that there is not yet enough justification for a rich, scheduler-facing, per-task interface. A scheduler-integrated model may still be relevant in the future, but it would require concrete use cases, clear semantics, and evidence that firmware can react quickly enough for task-following behavior to be useful. The current viable direction is therefore a simpler privileged user-space interface. This could either be an independent ABI for boost and throttle prioritization, or an extension to resctrl.

Video Platform QoS — Ionela Voinescu

A latency-focused QoS framework backend for kernel devices — Lukasz Luba and Chris Redpath

This talk presented an early proposal for a latency-focused quality-of-service framework aimed at improving coordination between kernel-managed devices such as CPUs, caches, memory, and interconnects. The motivation stems from the observation that modern systems rely on multiple governors, drivers, and firmware components independently inferring workload requirements and making performance decisions, often based on incomplete information and without awareness of the actions taken elsewhere in the system.

The speakers argued that many workloads ultimately care about latency rather than specific resource settings such as frequencies or performance states. Today, mechanisms such as scheduler-utilization signals, utilization clamping, cpufreq governors, and firmware-managed policies attempt to translate workload behavior into performance decisions, but much of the original intent can be lost as requests propagate through the stack. As a result, different devices may react inconsistently or redundantly to the same workload.

The proposed framework would treat latency as a first-class concept and provide a common mechanism for propagating latency requirements to registered devices. Rather than each subsystem independently attempting to infer application needs, devices could receive a more explicit representation of workload requirements and coordinate their responses. This could help avoid situations where CPUs, memory controllers, caches, and interconnects make separate policy decisions that are individually reasonable but collectively suboptimal.

A significant part of the discussion focused on how such a framework could coexist with existing kernel infrastructure. Participants explored the relationship with utilization-based scheduling signals, uclamp, operating performance points (OPPs), cpufreq, and firmware-controlled performance management. Questions were raised about the proper level of abstraction, how latency requirements should be represented, and whether the effort should be limited to backend infrastructure or eventually provide a frontend interface through which applications and middleware could express QoS requirements directly.

The audience generally agreed that current systems suffer from fragmented policy decisions and that preserving workload intent across subsystem boundaries is an important challenge. At the same time, several open questions remain regarding the exact interfaces, ownership of policy decisions, and integration with existing kernel mechanisms. The session concluded as a design discussion rather than a concrete implementation proposal, with future work to be focused on refining the architecture, identifying practical use cases, and evaluating whether coordinated latency propagation can provide measurable improvements in responsiveness and energy efficiency.

Video: Latency-focused QoS framework backend for kernel devices - Lukasz Luba and Chris Redpath (OSPM26)

Improving the SCHED_DEADLINE wakeup rule in presence of micro-sleeps — Tommaso Cucinotta and Luca Abeni

The issue manifests during "micro-sleeps": situations where a deadline task suspends for a negligible duration while still possessing a substantial portion of its run-time budget. The current CBS logic may respond by aggressively truncating the task's remaining budget to maintain the bandwidth ratio. Although this behavior upholds the theoretical isolation properties of the scheduler, Cucinotta argued that it is often overly pessimistic for real-world applications.

A simple scenario was presented to illustrate the problem: two deadline tasks competing for a single CPU. If the second task performs a momentary sleep shortly after it begins its work, the scheduler might slash its budget even though, had the task remained active, it would have finished its execution without causing any additional system interference. Consequently, a minor suspension results in an unwarranted penalty that reduces the work the task can actually complete.

The proposed solution involves an adjustment to the wakeup logic. Rather than strictly following the budget reduction mandated by the revised CBS rule, the scheduler would evaluate two alternatives — including the run time the task would have held had it never suspended — and select the maximum safe value. This approach aims to uphold temporal isolation while preventing the unnecessary loss of budget during transient inactivity.

Attendees questioned whether this optimization remains mathematically sound within the CBS framework. Cucinotta maintained that the refined rule preserves existing isolation guarantees while aligning more closely with practical user expectations. Attendees explored various execution paths and edge cases, particularly focusing on configurations where task deadlines do not match their periods, as these are the most sensitive to wakeup-rule variations.

This led to a broader exploration of the interface between applications and the deadline-scheduler policy. The discussion covered the fundamental semantics of run time, deadline, and period, as well as the alignment of reservation windows. Several participants sought clarification on how software can better anticipate the scheduler's internal decisions and the nuances of task resumption compared to application-level timing requirements.

A specific question was aimed at clarifying whether the revised wakeup rule that keeps the deadline but reduces the run time would be always better than the traditional CBS wakeup rule that resets the parameters to maximum run time and deadline one period apart. The traditional behavior is still useful because it provides SCHED_DEADLINE with a self-synchronizing ability: waking up a tiny bit before the programmed deadline, which is a common situation if the application uses its own timer or wakes up in response to external stimuli, causes a SCHED_DEADLINE task to start afresh with reset parameters, and this is what the user mostly expects.

Using the revised rule in such a case would force the task to keep its absolute deadline, slashing the remaining budget and quite likely causing a forced throttling of the task until its former deadline, causing an unexpected delay and de-synchronization of the task due to the in-kernel time accounting not matching the one used in user-space. Eventually, Cucinotta argued that an end user might want to choose among the two behaviors, so this could be done properly by adding another flag to the sched_setattr() API to explicitly request the revised rule (in the current API, you can force use of the revised wakeup rule by setting "period = deadline + 1ns", which seems a weird way to request that).

Then, the thread of the conversation revisited the need for better user-space observability. The authors discussed an earlier proposal to expose a task's current absolute deadline and instantaneous budget through an enhanced sched_getattr() system call. Current procfs entries were described as inadequate because they lack realtime updates and use internal kernel timestamps, whereas the new interface would provide actionable data that developers can correlate with their own timing logic.

The session also reviewed several advanced features of the subsystem, such as bandwidth reclamation via the GRUB flag, notifications for run-time overruns, and the constraints surrounding the creation of new deadline processes. These points were raised by audience members interested in the practicalities of deploying and debugging deadline-scheduled workloads in production environments.

The consensus among the attendees was that the current wakeup rule is indeed too conservative for workloads characterized by micro-sleeps. The proposed modification was viewed as a targeted usability fix that improves the experience for developers without altering the fundamental semantics of the CBS algorithm.

Moving forward, the authors plan to validate the new rule against a wider array of workloads and edge cases while continuing the review process on the mailing lists. The evaluation of new user-space interfaces remains a priority, reflecting a sustained interest in making the deadline scheduler more transparent and easier to integrate into complex application stacks while keeping its strong temporal guarantees intact.

Video: Improving the SCHED_DEADLINE wake-up rule in presence of micro-sleeps - Tommaso Cucinotta (OSPM26)