How will Linux be leveraged in next-gen supercomputers? (NewsForge) [LWN.net]

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 17:43 UTC (Mon) by clugstj (subscriber, #4020) [Link]

I think he is being clue-less.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 3:42 UTC (Tue) by csamuel (subscriber, #2624) [Link]

My suspicion is that he's talking about applications rather than the
kernel. Some applications (and problems) scale better than others.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 17:44 UTC (Mon) by dlang (subscriber, #313) [Link]

back in the 2.0 kernel it was single threaded, but with 2.6 kernels they are posting problems and fixes for 512 CPU systems with 1TB of ram.

he is obviously waaaay out of date.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 18:05 UTC (Mon) by gallir (subscriber, #5735) [Link]

I always thought it was multithreaded although it suffered of coarse
granularity locking (and the infamous big kernel lock).

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 19:00 UTC (Mon) by danielthaler (subscriber, #24764) [Link]

Not really...

In fact I found this: http://lkml.org/lkml/2004/8/20/191, which is definite evidence that that is not the case.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 19:03 UTC (Mon) by gallir (subscriber, #5735) [Link]

It seems i didn't explain very well. I was answering to "back in the 2.0
kernel it was single threaded". I didn't mean "current kernel".

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 19:34 UTC (Mon) by proski (subscriber, #104) [Link]

Windows 98 boots on my Dell Precision 370. This proves that Windows 98 support hyperthreading and 64-bit instructions. This also presents definite evidence that hyperthreading support in Windows 98 does not suffer of coarse granularity locking. Because you know, if granularity locking were coarse, Windows 98 wouldn't boot at all.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 20:11 UTC (Mon) by jsbarnes (subscriber, #4096) [Link]

Simply booting the kernel on such an extreme configuration required quite
a bit of scalability work in several areas to minimize cacheline
contention and prevent live locks at boot. I was most excited by the fact
that those changes had finally been accepted into Andrew's kernel and were
queued for inclusion in the mainline.

As you note, however, the fact that it boots doesn't mean that it will run
all codes with perfect scalability. Achieving that is a constant
struggle.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 0:37 UTC (Tue) by mmarq (guest, #2332) [Link]

" In fact I found this: http://lkml.org/lkml/2004/8/20/191, which is definite evidence that that is not the case. "

hmmm,... maybe, but i think that is a lot of missing the point here too.

How are those HPC Linux configurations mentioned in the article really ? Proprietary ?

The trend now, like in a "mania" is to put something under Linux and claim it is Linux! They call them hypervisors, nested OSes, Injection something... with extended command sets, consoles... and it's just getting started.

For instance; from my reduced expert capability i belive that Xen is not an application, but and OS in itself. XEN IS AN OPERATHING SYSTEM that borrows some fundamental principles from ExoKernels, upon which they run a VMM as the fundamental domain, and which function also like a HAL guaranteing better portability.

Thought Xen is more fit for the intended porpouse, things start to get weird when they bach Denali http://www.cl.cam.ac.uk/Research/SRG/netos/papers/2003-xe...
because of lack of capabilities, when Denali under the same principle of Exokernels, http://en.wikipedia.org/wiki/Exokernel#Exokernels could *EASLY* run several *MODIFIED* instances of Linux or WindowsXP(if possible) as LibOSes and achieve some of the functionality of Xen without the overhead of a VMM. The "big or small" problem would be that *MODIFYING* needed.

Exokernels, IMO, are the future even on desktops, because something in the direction of CELL http://www-128.ibm.com/developerworks/power/cell/ is also the future. So the problem is not another stupid and pueril academic discussions of whither monolithic, micro or no kernel, but taking advantage of the best possible of all of them. Small introduction here:
http://pdos.csail.mit.edu/exo.html and more in here: http://pdos.csail.mit.edu/PDOS-papers.html

And in here
http://www.pdos.lcs.mit.edu/papers/exokernel-sosp95.ps
he can see a pet reserch project, a MIT exokernel running ExOS as LibOS, 10 years ago(it could very well be Linux as LibOS now) **COMPLETELY BLOWING OUT OFF THE WATER**, in some simple tests, a mature tested and tried commercial monolithic OS like Ultrix.

Back to the topic, "IF" what ever is running under Linux on those HPC configurations is not Linux, then the **REAL** OS should not be called Linux. Dirty simple as that, for not letting more milking the caw and push the community in *MODIFYING* linux for running like some kind of LibOS in the direction of proprieatry standard HPC configurations.

IMO it should and could be a ExoLinux(?). It is a perfect marriage for *Real Secure & Persistence Safe* features like those in EROS http://www.eros-os.org/essays/00Essays.html, and Linux as it is could run without much modification(almost no waste) as a LibOS in top of ExoLinux(?), and the HPC guys will have a perfect HPC standard that would not be proprietary.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 19:56 UTC (Mon) by dlang (subscriber, #313) [Link]

in 2.0 the kernel itself was single threaded (all kernel execution was protected by a single lock), but it supported multiple CPU's and could run one or more userspace processes cuncurrently with the kernel on the additional CPU's.

ever since the big lock has been getting split up (or code moved outside of it's coverage) to allow the kernel itself to take advantage of more CPU's.

the 2.0 kernel could only efficiantly use 1-2 CPU's
the 2.2. kernel could efficiantly use 2-4 CPU's
the 2.4 kernel could efficiantly use 4-8 CPU's
the 2.6 kernel is comfortable up to at least 32 CPU's (the people running 512 CPU's are working on the fact that by the time they get to that point their scaling factor has dropped to 50% so they really only take advantage of half of that 512th core, which is actually surprisingly good)

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 20:17 UTC (Mon) by jsbarnes (subscriber, #4096) [Link]

"the 2.6 kernel is comfortable up to at least 32 CPU's (the people running
512 CPU's are working on the fact that by the time they get to that point
their scaling factor has dropped to 50% so they really only take advantage
of half of that 512th core, which is actually surprisingly good)"

I'm not quite sure what you mean here. In my experience, "kernel scaling"
is a near useless term. All you can say for certain is whether a _given
workload_ scales well (i.e. efficiently utilizes available resources).
This, in turn, depends on how the application is designed and coded, what
kernel services it makes use of (e.g. disk I/O, filesystem access, etc.),
and whether they scale well.

So for some workloads the Linux kernel limits scaling to a handful of
processors on a typical SMP system architecture. On other workloads, near
perfect scaling can be achieved up to even 512p.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 26, 2005 23:54 UTC (Mon) by dlang (subscriber, #313) [Link]

the user workload definantly affects the scalability of a system, you could have a OS that scales perfectly to 512 CPU's and an application that only uses one CPU.

and on the other hand you could have a single-threaded OS, but an application that scales perfectly to 512 CPU's

however in the second case you will still be limited becouse things like scheduling that need to take all the CPU's into account may not work very well.

overall things will work best if both the OS and the App scale well.

the scaleing 'limits' I listed in the prior post are the points where the kernel starts tripping over it's own feet and will realisticly gain very little from any additional CPU's, even if your app would scale well beyond this point.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 2:47 UTC (Tue) by hazelsct (subscriber, #3659) [Link]

No, the app and kernel in combination still dictate scalability. Though 2.0 doesn't run on machines with 512 CPUs, a well-written program with eight or sixteen processes or threads which doesn't make heavy use of kernel resources (e.g. networking, graphics, maybe pipes) will run quite nicely under 2.0 on an 8- or 16-processor machine.

1.2 was a single-threaded OS, which could not do SMP at all. 2.0 could be called a "single-threaded OS" only in that code *inside the kernel* ran under the BKL. But user space code could run in any number of threads and processes with reasonable efficiency.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 3:46 UTC (Tue) by dlang (subscriber, #313) [Link]

you would also need to make sure that it didn't make much use of the scheduler, memory allocator, disks, etc.

if you are talking about threads that load their entire working set into ram and then sit and spin on that set until they are done then yes you can exceed the kernel scalability by quite a bit (bit if you can do that then you can also fairly trivially run your job on a loosly coupled cluster, which would be far cheaper then a large N-way SMP box)

for more normal workloads you need both to be scalable, and in general your performance will start tailing off as soon as you pass either the kernel or app scaleability limits.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 4:37 UTC (Tue) by drag (subscriber, #31333) [Link]

I think the kernel itself (at least 2.6 series) scales a bit better then was given credit for when one of you said it could use up to 32 cpus.

HP with it's 'BigTux' project showed that a out-of-the-box kernel is able to scale perfectly well (with benchmarks and such) in a 64-way NUMA architecture. I beleive they used a Suse smp kernel for testing. And a unique thing it can do is do it dynamicly. That is when they plug in more cpus to a system the kernel is able to automaticly detect them and set up memory management so that it's all optimized for regular numa-style operation.

This is something special, I think. Propriatory unix kernels would have to be modified code-wise or have to be reconfigured at least to be able to run efficiently in a single cpu or dual cpu enviroment vs a 64 cpu enviroment. At least that's the impression that I get.

Also don't forget that SGI has it's Linux supercomputers that are able to use up to 2048 cpus in a single machine. Of course this isn't the vanilla kernel.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 5:11 UTC (Tue) by dlang (subscriber, #313) [Link]

I did say at least 32 CPU's :-)

the post a week or so ago about the 512 CPU machine was useing something pretty close to a stock kernel and they were still at ~50% scaleing per cpu.

I've also seen posts with IBM 64 CPU machines running very nicely with stock kernels.

in any case, even with multi-core CPU's the limit is still comfortably ahead of what's easy to get hold of.

and with the steady trickle of fixes for the bottlenecks that they do run into on the 512 CPU machines it's just getting better.

when you go from a 1 CPU machine to a SMP machine you do need to recompile the kernel, and when you move from a SMP setup to a NUMA setup (useually around 16-32 CPU's) there are a bunch of new things that have to be set right for the exact hardware (to have the memory management work optimily, although there is a lot of work being done to autodetect the nessasary settings), but there's a very broad range in there from 1.5 (HT) to ~64 cpu's where things just work

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 8:21 UTC (Tue) by drag (subscriber, #31333) [Link]

Cool.

Linux rocks. :P

Pretty soon we will be all running Numa kernels anyways.

Take a dual socket AMD64 board, which is nice enough to have a seperate memory bank for each memory controller, and then stick 2 dual core cpus on it. Instant complex numa machine. 4 cores.. 2 cores share one level of cache and main memory on one half, the other 2 on the other with the ability to access each other's stuff in a pinch.

Fun stuff.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 9:02 UTC (Tue) by joib (guest, #8541) [Link]

Actually, the dual core AMD64:s don't share cache. Each core has a separate 1 MB (or 512 on some of the lower end desktop models) L2 cache. Although AFAIK they are able to copy stuff directly between caches without having to go via main memory.

How will Linux be leveraged in next-gen supercomputers? (NewsForge)

Posted Sep 27, 2005 14:30 UTC (Tue) by dlapine (subscriber, #7358) [Link]

The altix I'm adminning at this moment ( :p ) has 2 512 cpu nodes.

The kernel is SGI enhanced to support all 512 Cpus, but what's not common knowledge is that a set of 6 cpu's are reserved to support OS operations, including the parallel filesystem cxfs. The other 506 CPU's are scheduled with Moab, and the users run MPI code on them with better efficiency than they get on my other cluster, the 887 node Itanium 2 cluster with myrinet.

My point is this: the SGI-modified Linux kernel running on the RHEL OS requires 6 CPUS to handle all of the tasks the OS is asked to perform (multiple 10Gige connections, 200TB+ of parallel filesystems, SMP scheduler/batch job system, etc). What better example of the power of the linux kernel to utilize multiple processors (and potentially, multiple cores) would you like?