| |||||||||||||
A potential NUMA scheduling solutionA potential NUMA scheduling solutionPosted Nov 15, 2012 18:24 UTC (Thu) by slamb (guest, #1070)In reply to: A potential NUMA scheduling solution by marcH Parent article: A potential NUMA scheduling solution
I don't know about need, but anyone who needs a lot of computing power might prefer N M-package machines to N*M 1-package machines for efficiency's sake. At Google, we certainly do. I won't reveal anything new, so take a look at this public talk (by Jeff Dean and Sanjay Ghemawat, always interesting people to follow). Multiple things per machine; their thing alone uses "144 machines, ~2300 cores", or 16 cores/machine. This sort of configuration makes sense if you have to bin-pack applications with different needs, like a memory-heavy memcached, a CPU-heavy application server, and a disk/SSD-heavy filesystem daemon. Or if you have processes with a lot of fixed overhead to run - better to get as much out of each one as possible. Or if the N M-package machines use less power than the N*M 1-package machines.
(Log in to post comments)
A potential NUMA scheduling solution Posted Nov 15, 2012 20:45 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
It depends how much communication you need between the different processes that you are using.
Using a bunch of 1-package machines imposes a much higher communication cost than multi-package machines.
There's also a question of the features provided on the motherboards. There are very few 1-package motherboards out there that have 'server' features like ECC memory, serial or network consoles, etc. If any of these features are desirable to you, you will probably be using 2-package motherboards at the smallest.
In terms of cost for computing power, 2-package machines seem to still be the sweet spot, you avoid having to pay for multiple power supplies, drives, and other per-system infrastructure, but you don't have the price premium that larger multi-package systems tend to have.
But even a 2-package machine can see advantages with proper NUMA handling (not as drastic as larger systems, but there is still an advantage)
In addition, a large amount of workload on large systems nowdays is virtual machine based with the hardware systems being oversubscribed. While a VM can be moved from one machine to another, it can be a fairly expensive thing to do, so larger systems work better in practice as they allow for the peaks and valleys of demand to average out better, so you can oversubscribe your hardware more.
A potential NUMA scheduling solution Posted Nov 15, 2012 23:12 UTC (Thu) by intgr (subscriber, #39733) [Link]
> There are very few 1-package motherboards out there that have 'server' features like ECC memory, serial or network consoles, etc
Even most desktop motherboards support ECC these days and have serial port pin headers at the very least. All Intel server motherboards (and probably HP and IBM too) come with BMC and an optional IP KVM addon.
A potential NUMA scheduling solution Posted Nov 15, 2012 23:39 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
serial port pin headers are not he same thing as having a serial console.
how many of these Intel server motherboards are single socket?
I've purchased single-package systems from HP, IBM, Sun and other tier-1 vendors. In every case the system that I received had a multi-package motherboard with only one package installed in it.
In that situation, the most cost-effective way to get compute capability is to go ahead and fully populate the motherboard. Even if that (usually second) package isn't as efficient as it would have been in a separate system (with no contention for RAM bandwidth, etc) it's still the best thing to do because the marginal cost of adding it to an existing system is so small.
A potential NUMA scheduling solution Posted Nov 16, 2012 1:26 UTC (Fri) by intgr (subscriber, #39733) [Link]
> how many of these Intel server motherboards are single socket?
See R1304BTLSHBN for example (S1200BTL), it has a single-socket ATX motherboard in a 1RU chassis, ECC memory and an optional IP KVM addon.
|
|||||||||||||