Why a system call?

Why a system call?

I don't understand how userspace is expected to know when and why it
would be worthwhile to move pages from node to node. We don't have
explicit system calls requesting that pages be written out to the swap
device or faulted back -- the OS handles it automatically and it's the
job of the VM system to do it well. Recent optimisations like swap
readahead are progress in that direction.

I imagine the intermediate use of the swap device is merely a stepping
stone for the implementor. I'm sure Christoph is aware of the price of
disk latency!

Perhaps nonlocal memory should be treated as a fast swap device after a
process has been migrated -- pages can be faulted across to local memory
when they are used on the new node. As an optimisation this could be
'extra lazy', since all pages are actually accessible -- eg. only pages
written to by the new node (and, if there is a good heuristic for this,
the most frequently read pages) need be copied to local memory.

Do NUMA systems have memory-to-memory copy operations which don't trash
the processor caches? I can imagine a "DMA" between node memories could
take place while the source pages remain readable by the processor.