| |||||||||||||
memory mirroring?memory mirroring?Posted Mar 17, 2012 8:40 UTC (Sat) by khim (subscriber, #9252)In reply to: memory mirroring? by dlang Parent article: Toward better NUMA scheduling
I think you may be mistaking how much of a win this is. There is an improvement in the speed of accessing memory, and even if we say that it's a 2x improvement, it still may not make much practical difference. Note that 2x improvement limit is recent improvement (when you have 8-16 cores in a single CPU you can build quite capable NUMA system with very few CPUs). It's not uncommon for older NUMA systems to have 10x or even 100x difference between access to local memory and remote memory. Not sure if anyone still builds such systems (SGI used to, but it's dead now).
(Log in to post comments)
memory mirroring? Posted Mar 17, 2012 12:24 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
True, current NUMA machines are almost all in the category of multi-socket AMD64 systems, which have fast enough interconnects that it's usable if you ignore NUMA ad just treat is as a SMP machine.
Historic NUMA machines had MUCH slower interconnects, the best comparison in moderns systems would be if you were connecting your CPU nodes together with high speed networks.
There are still some people building such machines (I think the current Cray systems are this category), but when you get to interconnects that are that expensive, you are frequently better segmenting the system and running it as if it was a cluster of systems, or (the more common case), just build a cluster of commodity systems instead of the monster NUMA system in the first place.
I think that if AMD hadn't introduced NUMA to the commodity desktop/server with the Opteron, NUMA would be something that's so rare that the overhead and complexity of it's logic wouldn't be acceptable in the kernel.
There are some applications that really are hard to split into a multi-machine cluster, and for those NUMA (including RDMA setups that tie multiple commodity machine together) are the right tool for the task, but they are pretty rare, it's almost always worth re-architecting the application to avoid this requirement.
memory mirroring? Posted Mar 17, 2012 23:07 UTC (Sat) by davecb (subscriber, #1574) [Link]
Quite large systems have large penalties for using distant nodes than small ones: bus backplane latency is not your friend (;-)) Smaller systems with bus lengths in the millimeters don't pay so great a penalty.
For one modern architecture there is a big hit after 32 sockets even when using a backplane derived from Cray's lowest-latency design. The speed of light needs improvement!
Ancient mainframes used a radial design to avoid having to be NUMA, at the expense of having an exceedingly complicated, multi-ported "system controller" where we'd put a bus.
--dave
memory mirroring? Posted Mar 18, 2012 0:56 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
> For one modern architecture there is a big hit after 32 sockets
32 sockets * 6 (true) cores/socket = 192 core system
at that sort of scale, I'll bet that locking overhead is at least as big a problem as the memory access times.
now, the 'commodity' NUMA keeps creeping up the scale, what is it now, 8 sockets * 6 cores = 48 core systems (*2 or more if you want to include hyperthread 'cores')?
you're a bit low.... Posted Mar 30, 2012 21:55 UTC (Fri) by cbf123 (guest, #74020) [Link]
Current high-end xeons have 8 "real" cores.
memory mirroring? Posted Mar 17, 2012 17:13 UTC (Sat) by arjan (subscriber, #36785) [Link]
also note that while latency might be 2x longer due to distance, there are cases where it's more beneficial to actually aggregate the memory bandwidth of local + remote rather than being local only....
single threaded, highly memory bandwidth bound apps come to mind in this regard.
memory mirroring? Posted Apr 16, 2012 19:34 UTC (Mon) by adavid (subscriber, #42044) [Link] SGI might be dead but sgi lives on after Rackable's 'switcheroo' . SGI NUMA is certainly alive with their Ultraviolet range.
|
|||||||||||||