Sponsored link
Vyatta –
Linux & Open Source
Alternative to Cisco –
Advanced Routing,
Firewall, VPN, QoS..
Free Download ->
| |||||||||||
How will Linux be leveraged in next-gen supercomputers? (NewsForge)How will Linux be leveraged in next-gen supercomputers? (NewsForge)Posted Sep 26, 2005 19:56 UTC (Mon) by dlang (subscriber, #313)In reply to: How will Linux be leveraged in next-gen supercomputers? (NewsForge) by gallir Parent article: How will Linux be leveraged in next-gen supercomputers? (NewsForge)
in 2.0 the kernel itself was single threaded (all kernel execution was protected by a single lock), but it supported multiple CPU's and could run one or more userspace processes cuncurrently with the kernel on the additional CPU's.
ever since the big lock has been getting split up (or code moved outside of it's coverage) to allow the kernel itself to take advantage of more CPU's.
the 2.0 kernel could only efficiantly use 1-2 CPU's
(Log in to post comments)
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 26, 2005 20:17 UTC (Mon) by jsbarnes (subscriber, #4096) [Link] "the 2.6 kernel is comfortable up to at least 32 CPU's (the people running512 CPU's are working on the fact that by the time they get to that point their scaling factor has dropped to 50% so they really only take advantage of half of that 512th core, which is actually surprisingly good)" I'm not quite sure what you mean here. In my experience, "kernel scaling" is a near useless term. All you can say for certain is whether a _given workload_ scales well (i.e. efficiently utilizes available resources). This, in turn, depends on how the application is designed and coded, what kernel services it makes use of (e.g. disk I/O, filesystem access, etc.), and whether they scale well. So for some workloads the Linux kernel limits scaling to a handful of processors on a typical SMP system architecture. On other workloads, near perfect scaling can be achieved up to even 512p.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 26, 2005 23:54 UTC (Mon) by dlang (subscriber, #313) [Link] the user workload definantly affects the scalability of a system, you could have a OS that scales perfectly to 512 CPU's and an application that only uses one CPU.
and on the other hand you could have a single-threaded OS, but an application that scales perfectly to 512 CPU's
however in the second case you will still be limited becouse things like scheduling that need to take all the CPU's into account may not work very well.
overall things will work best if both the OS and the App scale well.
the scaleing 'limits' I listed in the prior post are the points where the kernel starts tripping over it's own feet and will realisticly gain very little from any additional CPU's, even if your app would scale well beyond this point.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 2:47 UTC (Tue) by hazelsct (subscriber, #3659) [Link] No, the app and kernel in combination still dictate scalability. Though 2.0 doesn't run on machines with 512 CPUs, a well-written program with eight or sixteen processes or threads which doesn't make heavy use of kernel resources (e.g. networking, graphics, maybe pipes) will run quite nicely under 2.0 on an 8- or 16-processor machine.
1.2 was a single-threaded OS, which could not do SMP at all. 2.0 could be called a "single-threaded OS" only in that code *inside the kernel* ran under the BKL. But user space code could run in any number of threads and processes with reasonable efficiency.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 3:46 UTC (Tue) by dlang (subscriber, #313) [Link] you would also need to make sure that it didn't make much use of the scheduler, memory allocator, disks, etc.
if you are talking about threads that load their entire working set into ram and then sit and spin on that set until they are done then yes you can exceed the kernel scalability by quite a bit (bit if you can do that then you can also fairly trivially run your job on a loosly coupled cluster, which would be far cheaper then a large N-way SMP box)
for more normal workloads you need both to be scalable, and in general your performance will start tailing off as soon as you pass either the kernel or app scaleability limits.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 4:37 UTC (Tue) by drag (subscriber, #31333) [Link]
I think the kernel itself (at least 2.6 series) scales a bit better then was given credit for when one of you said it could use up to 32 cpus.
HP with it's 'BigTux' project showed that a out-of-the-box kernel is able to scale perfectly well (with benchmarks and such) in a 64-way NUMA architecture. I beleive they used a Suse smp kernel for testing. And a unique thing it can do is do it dynamicly. That is when they plug in more cpus to a system the kernel is able to automaticly detect them and set up memory management so that it's all optimized for regular numa-style operation.
This is something special, I think. Propriatory unix kernels would have to be modified code-wise or have to be reconfigured at least to be able to run efficiently in a single cpu or dual cpu enviroment vs a 64 cpu enviroment. At least that's the impression that I get.
Also don't forget that SGI has it's Linux supercomputers that are able to use up to 2048 cpus in a single machine. Of course this isn't the vanilla kernel.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 5:11 UTC (Tue) by dlang (subscriber, #313) [Link] I did say at least 32 CPU's :-)
the post a week or so ago about the 512 CPU machine was useing something pretty close to a stock kernel and they were still at ~50% scaleing per cpu.
I've also seen posts with IBM 64 CPU machines running very nicely with stock kernels.
in any case, even with multi-core CPU's the limit is still comfortably ahead of what's easy to get hold of.
and with the steady trickle of fixes for the bottlenecks that they do run into on the 512 CPU machines it's just getting better.
when you go from a 1 CPU machine to a SMP machine you do need to recompile the kernel, and when you move from a SMP setup to a NUMA setup (useually around 16-32 CPU's) there are a bunch of new things that have to be set right for the exact hardware (to have the memory management work optimily, although there is a lot of work being done to autodetect the nessasary settings), but there's a very broad range in there from 1.5 (HT) to ~64 cpu's where things just work
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 8:21 UTC (Tue) by drag (subscriber, #31333) [Link] Cool.
Linux rocks. :P
Pretty soon we will be all running Numa kernels anyways.
Take a dual socket AMD64 board, which is nice enough to have a seperate memory bank for each memory controller, and then stick 2 dual core cpus on it. Instant complex numa machine. 4 cores.. 2 cores share one level of cache and main memory on one half, the other 2 on the other with the ability to access each other's stuff in a pinch.
Fun stuff.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 9:02 UTC (Tue) by joib (guest, #8541) [Link] Actually, the dual core AMD64:s don't share cache. Each core has a separate 1 MB (or 512 on some of the lower end desktop models) L2 cache. Although AFAIK they are able to copy stuff directly between caches without having to go via main memory.
How will Linux be leveraged in next-gen supercomputers? (NewsForge) Posted Sep 27, 2005 14:30 UTC (Tue) by dlapine (subscriber, #7358) [Link] The altix I'm adminning at this moment ( :p ) has 2 512 cpu nodes.
The kernel is SGI enhanced to support all 512 Cpus, but what's not common knowledge is that a set of 6 cpu's are reserved to support OS operations, including the parallel filesystem cxfs. The other 506 CPU's are scheduled with Moab, and the users run MPI code on them with better efficiency than they get on my other cluster, the 887 node Itanium 2 cluster with myrinet.
My point is this: the SGI-modified Linux kernel running on the RHEL OS requires 6 CPUS to handle all of the tasks the OS is asked to perform (multiple 10Gige connections, 200TB+ of parallel filesystems, SMP scheduler/batch job system, etc). What better example of the power of the linux kernel to utilize multiple processors (and potentially, multiple cores) would you like?
|
|
||||||||||