How will Linux be leveraged in next-gen supercomputers? (NewsForge) [LWN.net]

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 23:54 UTC (Mon) by dlang (subscriber, #313)
In reply to: How will Linux be leveraged in next-gen supercomputers? (NewsForge) by jsbarnes
Parent article: How will Linux be leveraged in next-gen supercomputers? (NewsForge)

the user workload definantly affects the scalability of a system, you could have a OS that scales perfectly to 512 CPU's and an application that only uses one CPU.

and on the other hand you could have a single-threaded OS, but an application that scales perfectly to 512 CPU's

however in the second case you will still be limited becouse things like scheduling that need to take all the CPU's into account may not work very well.

overall things will work best if both the OS and the App scale well.

the scaleing 'limits' I listed in the prior post are the points where the kernel starts tripping over it's own feet and will realisticly gain very little from any additional CPU's, even if your app would scale well beyond this point.

(Log in to post comments)

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 2:47 UTC (Tue) by hazelsct (subscriber, #3659) [Link]

No, the app and kernel in combination still dictate scalability. Though 2.0 doesn't run on machines with 512 CPUs, a well-written program with eight or sixteen processes or threads which doesn't make heavy use of kernel resources (e.g. networking, graphics, maybe pipes) will run quite nicely under 2.0 on an 8- or 16-processor machine.

1.2 was a single-threaded OS, which could not do SMP at all. 2.0 could be called a "single-threaded OS" only in that code *inside the kernel* ran under the BKL. But user space code could run in any number of threads and processes with reasonable efficiency.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 3:46 UTC (Tue) by dlang (subscriber, #313) [Link]

you would also need to make sure that it didn't make much use of the scheduler, memory allocator, disks, etc.

if you are talking about threads that load their entire working set into ram and then sit and spin on that set until they are done then yes you can exceed the kernel scalability by quite a bit (bit if you can do that then you can also fairly trivially run your job on a loosly coupled cluster, which would be far cheaper then a large N-way SMP box)

for more normal workloads you need both to be scalable, and in general your performance will start tailing off as soon as you pass either the kernel or app scaleability limits.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 4:37 UTC (Tue) by drag (subscriber, #31333) [Link]

I think the kernel itself (at least 2.6 series) scales a bit better then was given credit for when one of you said it could use up to 32 cpus.

HP with it's 'BigTux' project showed that a out-of-the-box kernel is able to scale perfectly well (with benchmarks and such) in a 64-way NUMA architecture. I beleive they used a Suse smp kernel for testing. And a unique thing it can do is do it dynamicly. That is when they plug in more cpus to a system the kernel is able to automaticly detect them and set up memory management so that it's all optimized for regular numa-style operation.

This is something special, I think. Propriatory unix kernels would have to be modified code-wise or have to be reconfigured at least to be able to run efficiently in a single cpu or dual cpu enviroment vs a 64 cpu enviroment. At least that's the impression that I get.

Also don't forget that SGI has it's Linux supercomputers that are able to use up to 2048 cpus in a single machine. Of course this isn't the vanilla kernel.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 5:11 UTC (Tue) by dlang (subscriber, #313) [Link]

I did say at least 32 CPU's :-)

the post a week or so ago about the 512 CPU machine was useing something pretty close to a stock kernel and they were still at ~50% scaleing per cpu.

I've also seen posts with IBM 64 CPU machines running very nicely with stock kernels.

in any case, even with multi-core CPU's the limit is still comfortably ahead of what's easy to get hold of.

and with the steady trickle of fixes for the bottlenecks that they do run into on the 512 CPU machines it's just getting better.

when you go from a 1 CPU machine to a SMP machine you do need to recompile the kernel, and when you move from a SMP setup to a NUMA setup (useually around 16-32 CPU's) there are a bunch of new things that have to be set right for the exact hardware (to have the memory management work optimily, although there is a lot of work being done to autodetect the nessasary settings), but there's a very broad range in there from 1.5 (HT) to ~64 cpu's where things just work

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 8:21 UTC (Tue) by drag (subscriber, #31333) [Link]

Cool.

Linux rocks. :P

Pretty soon we will be all running Numa kernels anyways.

Take a dual socket AMD64 board, which is nice enough to have a seperate memory bank for each memory controller, and then stick 2 dual core cpus on it. Instant complex numa machine. 4 cores.. 2 cores share one level of cache and main memory on one half, the other 2 on the other with the ability to access each other's stuff in a pinch.

Fun stuff.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 9:02 UTC (Tue) by joib (guest, #8541) [Link]

Actually, the dual core AMD64:s don't share cache. Each core has a separate 1 MB (or 512 on some of the lower end desktop models) L2 cache. Although AFAIK they are able to copy stuff directly between caches without having to go via main memory.