Recent work in memory tiering

Hot-page promotion

"Demotion" refers to the act of moving a page of data from a faster tier to a slower one, while "promotion" is movement in the opposite direction. A page would be most likely to be demoted if the kernel concludes that it is not being actively used; promotion would happen when memory that is in use happens to be in slow memory. Demotion decisions are relatively easy (it is a similar decision to swapping a page out) and works reasonably well, but detecting hot pages in slow memory has proved to be more difficult. So either the kernel ends up leaving hot pages in slow memory, or it promotes pages that turn out not to be hot after all; in both cases, the result is worse performance.

Bharata B Rao has been working on an infrastructure for hot-page detection and promotion for some time now; LWN covered this work in 2025. This patch series is built around the idea that there may be multiple sources of data about which pages in slow memory are seeing high activity and should be promoted. The kernel might generate that data by marking pages as being inaccessible and tracking the resulting faults when a workload tries to use them, for example. CXL systems can also support access tracking and reporting, providing relatively fine-grained information without adding CPU overhead. Rao's proposed "pghot" mechanism integrates tracking data from the available sources and uses it to decide which pages should be promoted to faster memory. A new kernel thread, called kmigrated, uses this data to attempt to move the hottest pages.

The previous article covered the third version of the series; there have been five new versions since then. The core tracking machinery remains similar in concept, though various improvements have been made. This subsystem now can use profile data from AMD's Instruction-Based Sampling and has gained a new "precision mode". By default, pghot uses one byte of memory to track each lower-tier page; in precision mode, that overhead expands to four bytes and, among other things, enables tracking of which NUMA node is accessing each page. That information allows kmigrated to try to promote hot pages to the node where they are being used. That, in turn, enables pghot to (optionally) take over much of the work that is currently handled by the kernel's NUMA-balancing code.

Andrew Morton started the discussion of the current version by asking whether DAMON, which is already in the kernel, could perform this function. Rao answered with an LLM-generated comparison of the two mechanisms, which concluded that DAMON could perhaps do the job, but only after being tuned for the specific workload, and only after the workload had run for some time. DAMON author SJ Park was somewhat unimpressed by this response, asking Rao to actually learn about DAMON before going any further. In a separate response to Morton, Rao described some differences he saw between pghot and DAMON; Park disagreed with some of them, but also said that some pghot-specific features (such as integrating data from multiple sources) were currently being implemented for DAMON.

Matthew Wilcox had a different view of the entire patch set and the ideas that it is implementing:

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

Gregory Price quickly responded that Meta is "actively interested in this work and helping to drive the design of it". Yongting Lin also expressed support, albeit more lukewarmly, for this work. So there does appear to be some interest in this capability, even if it is not widespread.

David Hildenbrand pointed out that, even after eight revisions, there are no review tags for any of the patches; he described the series as "far away from being mergeable". This work, he said, has not been perceived as having even a medium level of priority, so the memory-management maintainers have not yet taken a hard look at it. The question of whether this use case can be adequately handled by DAMON also remains without a definitive answer. So it would appear that pghot is likely to be stuck on the ground floor for some time yet.

Tier-aware memory control groups

In the pghot discussion, Price mentioned a problem that comes up on tiered-memory systems: the first workloads to run fill the faster memory with their data; the workloads that come later then end up in slower memory because that is all that is left. Pghot, he said, would allow the system to rectify this problem over time. But there is also interest in preventing it from coming about in the first place.

Control groups with the memory controller (often referred to as "memcgs") would seem like the logical solution to this problem, but the memory controller is not currently designed to handle tiered memory. It can limit a group to a certain amount of memory overall, but cannot control where the memory is placed. So, naturally, the solution would be to add awareness of tiering to the memory controller. There are currently two separate patch sets that accomplish that goal.

This series from Joshua Hahn takes a relatively simple approach to configuration. If a given system has 25% of its memory as DRAM and 75% as slower CXL memory, then any given control group will be able to use 25% of its memory limit for DRAM, and must use the rest in CXL. That will prevent the first workload that shows up from filling the system's DRAM, leaving memory available for others. The hard and soft limits are both scaled in this way, allowing a workload to use more DRAM (if so configured) when contention for memory is low.

Implementation of the limits is handled in a number of places. Most obviously, perhaps, is at allocation time, when the system will allocate memory from tiers where there is still space available when possible. If that is not possible, and an allocation is placed in a tier with no space remaining, then targeted reclaim will be kicked off to remove some pages from that tier. Various other actions (such as page migration) that can exhaust a tier's limit will also start reclaim. Promotion of pages to faster tiers will be slowed when the limit is reached.

Meanwhile, Qiqi Liu has posted a separate solution to this problem. Like Hahn's series, Liu's sets limits for each tier that match the proportions of each memory type in the system. In this series, though, the administrator can adjust those limits if needed. Liu's solution does not implement the low limits (the minimum amount of memory that is guaranteed to the group), and is based heavily on reclaim to enforce the limits.

Either series seems capable of solving the core problem, but the maintainers will certainly not want to merge both of them. Michal Hocko asked about the differences between the two, while expressing doubt about the goal of adding tier-level control. Hahn, too, asked about the differences, noting that the ability to set per-tier limits in Liu's series seemed to be the biggest of those. Liu responded with a more detailed look at the two, and concluded by saying that Hahn's approach seemed like the best one to carry forward. So, while the question of whether the kernel should really gain tier-level control in the memory controller still needs to be answered, there does appear to be agreement among the interested developers on how to get there.

Note: you can avoid this step in the future by logging into your LWN account.