This session, led by Andi Kleen, was titled "NUMA APIs," but talked little
about APIs. The real question was how processes should be managed on NUMA
NUMA architectures are, by definition, multiprocessor systems which are
organized into nodes of one or more processors. Each node includes a
certain amount of local memory. That memory is accessible across the
entire NUMA system, but accesses from the local node are much faster than
accesses from farther away. The NUMA architecture creates some interesting
challenges for operating systems; those which allocate their memory
properly will perform better than the others.
There are, according to Andi, three basic allocation techniques which may
be used in NUMA systems:
- "Membind" locks process allocations to a specific node. Keeping all
process memory on the local node should yield better performance, as
long as other references to that memory remain local as well. The
membind scheme can lead to allocation failures, however, if the node
runs low on local memory.
- "Home node" is like membind, except that non-local memory can be
allocated if local memory is not available.
- "Interleaved" allocations rotate through the nodes, allocating memory
from each. In some situations (shared memory regions, perhaps),
interleaved allocation schemes can yield better performance.
Different policies work better in different situations; the real question
is how those policies should be specified and controlled. One could set up
a scheme where each process virtual memory area (VMA) has its own
allocation scheme. Linus, however, is opposed to the idea of applications
controlling how their own memory is allocated. Rather than having each
allocation try to optimize its performance on NUMA systems, this sort of
decision is best left to system administrators. There is also, it seems, a
shortage of publicly available numbers on just how much performance benefit
is to be had from complicated NUMA allocation schemes.
NUMA is an issue that will not go away, but more work must be done before
we'll see how Linux will support NUMA in the future.
to post comments)