Management of volatile CXL devices

When people think about the management of CXL memory, they often have tiering in mind, he said. But there are more basic questions that need to be answered before it is possible to just put memory on the PCI bus and expose it to the page allocator. There needs to be a way, he said, to avoid putting some applications into some classes of memory. The mempolicy API is the obvious tool to use, but he is not sure if it is the right one.

A running system, he said, has three broad components: the platform, the kernel, and the workloads. The platform (or its firmware) dictates how the kernel objects representing the system, such as NUMA nodes, memory banks, and memory tiers, are created; these objects are then consumed by the mempolicy machinery, by DAMON, and the rest of the system. But question of how and when those objects are created is important. For example, can the contiguous memory allocator (CMA) deal properly with NUMA nodes that are not online at boot time? Getting the answers wrong, he said, will make it hard to get tiering right.

Adam Manzanares suggested that it would be possible to create QEMU configurations that would match the expected hardware and asked if that would be helpful. Price said it would, enabling developers to figure out some basic guidelines governing what works and what does not.

The system BIOS, Price said, tells the kernel where the available memory ends up; the kernel then uses that information to create NUMA nodes describing that memory. A suitably informed kernel could create multiple nodes with different use policies if needed to use the memory effectively. Jonathan Cameron interjected that NUMA nodes are "garbage", at least for this use case; Price agreed that NUMA might not be the right abstraction for the management of CXL memory.

At a minimum, he continued, somebody should create documents describing what the kernel expects. He has done some work in that direction; links can be found in his proposal for this session. Any vendors that go outside those guidelines would be expected, at least, to provide an example of how they think things should work. Dan Williams said that a wider set of tests would also be helpful here; the current tests are mostly there to prevent regressions rather than ensuring that the hardware is behaving as expected.

Price would like to isolate the kernel from CXL memory as much as possible — if kernel allocations land in CXL memory, the performance and reliability of the system as a whole could suffer. There are a lot of tests to ensure that specific tiering mechanisms work well, but not so many when it comes to landing kernel allocations in the right place. There is no easy way to say which kernel memory, if any, should be in CXL memory. Configuring that memory as being in ZONE_MOVABLE would solve that problem (kernel allocations are not movable, so cannot be made from that zone), but it creates other problems. For example, hugetlb memory cannot be in ZONE_MOVABLE either.

John Hubbard suggested creating a more general mechanism, perhaps there is a need for a ZONE_NO_KERNEL_ALLOC. He, too, asked whether NUMA is the right abstraction for managing this memory. Williams suggested just leaving the memory offline when it is not in use, but Hubbard said that would run into trouble when the memory is put back into the offline state after use. If any stray, non-movable allocations have ended up there, it will not be possible to make the memory go offline. Williams said that the moral of that story is to never put memory online if you want to get it back later; Hubbard answered that this is part of the problem. The rules around this memory do not match its real-world use.

Alistair Popple suggested that this memory could be set up as device-private; that, however, would prevent the use of the NUMA API entirely, so processes could not bind to it. Williams said that device-private memory is kept in a driver that would manage how it is presented to user space. Popple said that approach can work, but is a problem for developers who want to use standard memory-management system calls. Hubbard said that he would rather not be stuck with a "niche" approach like device-private memory.

Price continued, noting that the standard in-kernel allocation interface is kmalloc(). He wondered whether developers would be willing to change such a widely used interface to add a new memory abstraction. If they do, though, perhaps adding something for high-bandwidth memory at the same time would make sense. Michal Hocko suggested that it would be better to make a dedicated interface than to overload kmalloc() further.

User-space developers want something similar; there tend to be a number of different workloads running on a system, some of which should be isolated from CXL memory. Page demotion can ignore memory configurations now, with the result that it can force pages into slower memory even if the application has tried to prevent that. He has, he said, even seen stack pages demoted to slow memory, which is "an awful idea". That said, he added, there are some times when it might make sense to demote unused stack memory.

An important objective is "host/guest kernel parity", where both a host system and any guests it is running have full control over memory placement. The guests should be able to do their own tiering, and should be able to limit what ends up in slower memory, just like the host does. He would like to define some KVM interfaces to pass information about memory layout and needs back and forth.

A big problem is the current inability to place 1GB huge pages in ZONE_MOVABLE. Allocating those pages to hand to guests can improve performance, but this limitation keeps them from being allocated in CXL memory. He has heard suggestions to use CMA, but the CXL memory is not online when CMA initializes, so CMA cannot use it.

There are other problems as well, he said at the end. If the placement of a bank of CXL memory is not properly aligned, that memory will be trimmed to the page-block size, wasting some of that memory. There are a lot decisions that can ripple through the system and create problems later on. Hocko commented that the page-block size is arbitrary, and that the memory-management developers would like to get rid of it. The fact that it is exported via sysfs makes that change harder, though. A lot of these problems, he said, are self-inflicted and will be fixed eventually.

Price concluded with an observation that memory tiers, as currently implemented, do not make much sense. A common CXL use case involves interleaving pages across multiple devices; that forces those devices into a single NUMA node. That creates confusion on multi-socket systems and really needs to be rethought, he said.

Price has posted the slides for this session.