Hardware? [LWN.net]

Hardware?

Posted Mar 18, 2012 12:45 UTC (Sun) by dlang (guest, #313)
In reply to: Hardware? by slashdot
Parent article: Toward better NUMA scheduling

the hardware can't do this transparently because the memory addresses of the page would change.

NUMA is still a single-system image, so all the memory on all the nodes is in the same address space. move a page and you have to update all pointers to that page (or the mapping to that page if you go through some level of indirection)

to post comments

Hardware?

Posted Mar 18, 2012 14:34 UTC (Sun) by slashdot (guest, #22014) [Link] (5 responses)

The hardware already transparently maps physical addresses to locations in L1, L2, L3 caches or RAM, and moves data between them, without this being software visible, so it's surely possible to do the same among NUMA nodes.

I guess the issue is that the CPU caches include hardware to check all cache lines in a set in parallel for whether they contain a certain address, while DDR3 doesn't, so the CPU would need to do lookup operations manually via extra reads/writes, which might be so expensive that just using the interconnect is faster.

And probably the market for huge NUMA systems isn't big enough to make manufacturing dedicated memory sticks with included caching/migration logic cost-effective.

But this is just a guess, I'm not really sure.

Hardware?

Posted Mar 18, 2012 16:53 UTC (Sun) by khim (subscriber, #9252) [Link] (4 responses)

Why isn't page migration done automatically by the hardware, by basically applying a cache protocol to RAM contents as well?

Because it's pointless.

I guess the issue is that the CPU caches include hardware to check all cache lines in a set in parallel for whether they contain a certain address, while DDR3 doesn't, so the CPU would need to do lookup operations manually via extra reads/writes, which might be so expensive that just using the interconnect is faster.

DDR3 is not a problem. DMA is. Caches are only “transparent” for userspace programs. Kernel need to perform a complex dance to support system with caches and DMA. And this will be true for your “automatic page migration”, too! This makes the whole exercise totally insane: instead of adding migration logic to kernel (where it can be done transparently from the rest if the system because hardware already includes required logic: it's called page tables) we add all that complexity to the hardware and then introduce complex dance in the kernel anyway to support DMA. What will it give us? Additional complexity and restrictions? Do not want.

Hardware?

Posted Mar 18, 2012 18:15 UTC (Sun) by slashdot (guest, #22014) [Link] (3 responses)

AFAIK no kernel code is needed for operation of the CPU caches, since the BIOS does all the setup (with the exception of marking uncacheable memory ranges on some systems).

As for DMA, surely the system could manage that automatically as well?

That is, IOMMUs would map to the 64-bit automatically managed address space, and the system would move memory for DMA just like it does for CPU access, and just like PCIe DMA is cache coherent for L1/L2/L3, it can be cache-coherent for this hypotetical L4 cache.

To clarify, a simple way to do this is to just add a few gigabytes of per-node L4 cache (in standalone chips), and use the same cache-coherency mechanism for it used for the L3 level.

The advantage could be that memory movement would happen by specialized hardware in parallel with CPU operation.

Hardware?

Posted Mar 18, 2012 19:08 UTC (Sun) by khim (subscriber, #9252) [Link] (2 responses)

AFAIK no kernel code is needed for operation of the CPU caches, since the BIOS does all the setup (with the exception of marking uncacheable memory ranges on some systems).

This only true if you don't ever use DMA and don't play tricks with page tables. Since kernel does both it includes huge amount of code which is supposed to keep all the data in sync.

As for DMA, surely the system could manage that automatically as well?

To do that it basically needs to virtualize all memory accesses by all devices. Yes, it's doable but it'll slowdown everything and will either hog the VT-x/AMD-V or introduce yet another emulation level (which will require specialized CPU or separate emulation chips). Not a good idea: large NUMA systems are exactly where things like KVM are most valuable.

like PCIe DMA is cache coherent for L1/L2/L3

Fail. PCIe DMA is not cache coherent for L1/L2/L3. It's resposibility of kernel to make sure everything works correctly despite the fact that IOMMU may have different setup from MMU in CPU and despite the fact that DMA moves data to main memory without bothering to do anything with CPU caches.

To clarify, a simple way to do this is to just add a few gigabytes of per-node L4 cache (in standalone chips), and use the same cache-coherency mechanism for it used for the L3 level.

Success. “Cache-coherency mechanism used for the L3 level” is part of OS kernel and yes, it's possible to add transparent handling of memory from different NUMA nodes to it. You don't need anything on hardware level for that - this was my point.

The advantage could be that memory movement would happen by specialized hardware in parallel with CPU operation.

Impossible. Contemporary systems attach memory directly to CPU - this means that any such mechanism will slow down regular memory accesses which will probably make the whole schema quite pointless.

Basically this is nice idea which looks fine on paper but requires radical redesign of everything (kernel, CPU, chipset, etc) from the ground up which makes it pointless in practice.

Hardware?

Posted Mar 18, 2012 21:20 UTC (Sun) by slashdot (guest, #22014) [Link] (1 responses)

> To do that it basically needs to virtualize all memory accesses by all devices

Which the hardware already does where IOMMUs are present...

> Fail. PCIe DMA is not cache coherent for L1/L2/L3

Uh?

PCI and PCIe are definitely cache coherent (or more precisely, they support it, although you can tell devices to not snoop caches).

> Success. “Cache-coherency mechanism used for the L3 level” is part of OS kernel

What?!?

This is totally false, and is simply a ridiculous claim.

The cache coherence of L3 caches in x86 SMP systems certainly isn't managed by the kernel!

Hardware?

Posted Mar 19, 2012 4:21 UTC (Mon) by khim (subscriber, #9252) [Link]

PCI and PCIe are definitely cache coherent (or more precisely, they support it, although you can tell devices to not snoop caches).

PCI does not support it at all (initially it was optional part of the standard but since nobody bothered to implement it later versions just removed it completely) and PCIe does not recommend to use it on large and/or busy systems (and NUMA systems tends to be both large and busy).

Which the hardware already does where IOMMUs are present...

Nope. IOMMU only hides physical addresses from hardware devices. OS kernel is in charge and must keep everything in sync. IOMMU presence is quite visible for device drivers. Since you want to make something not visible in kernel at all you need yet another level of indirection.

> Success. “Cache-coherency mechanism used for the L3 level” is part of OS kernel

What?!?

This is totally false, and is simply a ridiculous claim.

The cache coherence of L3 caches in x86 SMP systems certainly isn't managed by the kernel!

See above. If your system includes some hardware which does not care about L3 cache coherence (contemporary system tend to include few PCI devices at least and on busy systems you don't want to use built-in PCIe cache snooping because it sucks significant amount of inter-CPU bandwidth which is scarce on such systems) then your kernel is charge of keeping L3 cache and main memory in sync.