Cray CTO: Linux clusters don't play in HPC (Search Enterprise Linux) [LWN.net]

Surprised? NOT!

Posted Apr 12, 2004 21:01 UTC (Mon) by AnswerGuy (guest, #1256) [Link] (10 responses)

So what did anyone expect the CTO of Cray Computing to say? "Yep, those commodity PC and Apple G5 xserver clusters are really eating us for lunch! We expect to completely vanish from the Top500 supercomputers list in the next year or so."

I would have like to see some comments with some technical meat in them. This was just PR flush.

For instance there are real differences between a truly high performance system (cluster or now) and a "loose collection of unmanaged, individual, microprocessor-based computers".

Management is a function of the software and all of the major Linux "cluster" offerings have a diverse variety of different management suites and tools.

"Loose" could mean anything. I would charactize TCP/IP running over ethernet as a "loose" interconnect; but Myrinet, Infiniband or even a raw ethernet fram e protocol over ethernet esp. over the gigabit ethernet provided by some blade backplanes; and various other interconnects are not as "loose" in terms of I/O and node interconnection.

His use of phrases "supercomputer performance" and "a high performance computer" are vague. Obviously there are an increasing number of real-world sites that consider Beowulf and similar clusters to be "high performance" enough for their "supercomputer" requirements. If Dr. Terry wants to compete with more than empty rhetoric he should differentiate what specific performance constraints are imposed by the Linux clusters (and overcome by Cray's offerings) and he should explain how management of a Cray is better than the various cluster management tools that are available for Linux.

He tosses out a specious "8%" (typical delivered performance vs. peak claims) without any substantiation. If customers were getting "8%" of what they thought their vendors promised them --- there'd be lynchings in the streets! There would be a real story there. Ergo he has to explain what he means by that or find that we ignore it as incredible posturing as he tows the company line.

Ironically the "talk" (not even as in-depth as an interview) goes on to suggest that Cray's new XD1 systems will be running AMD Opteron CPUs which sound like "microprocessor-based computers" to me. All the fluff about interconnect and "directly linked processor" technology and memory and bottlenecks will have to be tested against Myrinet and Infiniband to see if they really offer better performance.

Overall it's a disappointing piece and I wasted far too much time writing up this comment on it.

Surprised? NOT!

Posted Apr 12, 2004 22:19 UTC (Mon) by josh_stern (guest, #4868) [Link] (5 responses)

Code has to be modified in one way to take advantage of clusters and
it has to be modified in a different way to take advantage of SMP
style computers. That's it, they're different...and tool support
is probably better for doing CASE-based SMP...and the Cray guy wants
to equate HPC with SMP. Give him credit for not trying to hide the
fact that clusters are a much cheaper way to get a given quantity
of compute power.

Ehm...

Posted Apr 13, 2004 7:05 UTC (Tue) by Los__D (guest, #15263) [Link] (4 responses)

Not completely true... An app for a beowulf cluster needs specific teqhniques and libraries, but an openmosix cluster can speed up any SMP-capable program... Off course there's apps that can use the mosix cluster better than others, but as a base argument it uses the cluster without special programming.

Dennis

Ehm...

Posted Apr 13, 2004 16:13 UTC (Tue) by evgeny (subscriber, #774) [Link]

> an openmosix cluster can speed up any SMP-capable program

Don't take your dreams for reality. From http://howto.ipng.be/openMosixWiki/index.php/FAQ:

[...] openMosix can't (as of yet) migrate threaded programs. If you want a single task to run on multiple machines simultaneously, you'll have to use fork() to create multiple processes.

Ehm...

Posted Apr 13, 2004 16:20 UTC (Tue) by josh_stern (guest, #4868) [Link] (2 responses)

Mosix didn't previously support shared memory in any meaningful way:

http://sourceforge.net/docman/display_doc.php?docid=10390&group_id=46729

Has that changed now that Linux 2.6 has NUMA support?

What I really meant above is that one will optimize
an algorithm in a different way depending on whether data sharing
happens by a) address mapped to fast (real) memory, b) address mapped
to slow (disk) memory, or c) explict message update. For distributed
systems, a) and c) are more similar in performance
characteristics while a) and b) are similar in what the code looks
like. So each of three differ in at least one important way, and
Cray, historically emphasized a).

It's true that Beowulf has a lot more mindshare than Mosix for
no rational reason.

Ehm...

Posted Apr 14, 2004 4:03 UTC (Wed) by sitaram (guest, #5959) [Link] (1 responses)

> Mosix didn't previously support shared memory in any meaningful way:

http://mcaserta.com/maask/

I must admit I dont know what the official status of that patch is, though.

Ehm...

Posted Apr 14, 2004 7:40 UTC (Wed) by josh_stern (guest, #4868) [Link]

Very interesting! Include the fact that it was created
by five female Indian undegraduates as a senior project!

Surprised? NOT!

Posted Apr 13, 2004 14:48 UTC (Tue) by snitm (guest, #4031) [Link] (3 responses)

The XD1 uses infiniband for its embedded interconnect; so aside from hypertransport they are using infiniband chips from mellanox as the highspeed bus between shelves. Sounds like Opteron and Infiniband is just yet another "cluster" to me.

Suspected as much

Posted Apr 13, 2004 19:33 UTC (Tue) by AnswerGuy (guest, #1256) [Link] (2 responses)

I suspected that their XD1 might be just an Opteron cluster with some proprietary interconnect --- as I said in my first post it would remain to be seen if this interconnect would beat out Infiniband, Myrinet, or SCI (scalable coherent interface).

To hear that it *is* Infiniband based answers that question. It would have been nice if Dr. Terry had just said that rather than trying to FUD his way into the news.

As many peoople pointed out in the ./ thread on this story --- ultimately the performance of a cluster vs. SMP vs. mainframe vs. monolithic supercomputer (vector processor) is going to depend on the program design. Some algorithms are partitionable and some impelementations and software designs take advantage of different parallelization trade-offs better than others.

They don't say what OS they'd be running (Unicos?) but I'd be amazed if Linux isn't ported to the XD1 platform in short order. Cray might not do it or condone it, but the users almost certainly will.

JimD

Suspected as much

Posted Apr 13, 2004 19:44 UTC (Tue) by snitm (guest, #4031) [Link] (1 responses)

They are running a modified Linux on the XD1 (aka Octigabay 12K). The resource manager that they use at the core of their management software is a customized Sun Grid Engine (SGE). These insights come from speaking with OctigaBay at SC2004.

Suspected as much

Posted Apr 13, 2004 19:45 UTC (Tue) by snitm (guest, #4031) [Link]

err make that SC2003 ;)

Cray CTO: Linux clusters don't play in HPC (Search Enterprise Linux)

Posted Apr 12, 2004 21:23 UTC (Mon) by xorbe (guest, #3165) [Link]

Wow, Cray turns on the FUD ray! Just tell me they aren't porting UnixWare or MS Windows to their "exclusive" super-computer club... ;-)

don't kill the messenger

Posted Apr 13, 2004 1:35 UTC (Tue) by vblum (guest, #1151) [Link]

You know ... don't take him all that negatively there. In practice, he has a point. Linux
clusters are not quite as convenient as supercomputers - they are vastly more complex to
operate, scale less well, you have to be dramatically more careful how you split up your
code, and queuing can be a nightmare. Some things just don't work nearly as well. All this is
true, and the Linux community would do well to realize those limits.

However, Cray still has a problem. The above is not enough for viability. If that
supercomputer is expensive enough that four CPUs cost the price of forty in a Linux cluster,
they must lose, despite all their added quality. The ratio just isn't right. They must be blind
and deaf to not know that. Plus, they underestimate the dramatic value of a commoditized
operating system - you can plug in powerful free software out of the box, on any Linux
system - no such hassle as on one of those reportedly high-quality hpc systems. Their
compatibility with modern tools generally sucks.

And then a word on Scyld Linux. Sounds great in principle - but they are going down the
very road of deliberate or accidental incompatibility. With their specialized, unique tools
(how can you replace standard ssh with something as obscure as bpsh???), if soimething
doesnt work, you are faced with two choices ... either swallow hard and live with the
deficiency, or throw away the system in its entirety ...

Linux is making fast progress, but there's still room for improvement. I'd like to try
Mandrake Cluster though - experiences, anyone?

HPC or MPP

Posted Apr 13, 2004 11:30 UTC (Tue) by christos_gentsis (guest, #20871) [Link]

Why to use an MPP (supercomputer) then you can use an HPC? if you use MPP
then you need the company who make it for ever... with the clusters you
don't need them....

could you update your self the MPP?? NOP you need the company :)
but you could update the HPC, you could even build it yourself.

how much power need the MPP and who much an HPC with the same
performance? HPP need about 5-6 times less...

so why to use a MPP instead of an HPC?

Cray CTO: Linux clusters don't play in HPC (Search Enterprise Linux)

Posted Apr 13, 2004 19:48 UTC (Tue) by dcoutts (guest, #5387) [Link] (2 responses)

This isn't a Linux vs. the establishment thing, it's a shared memory vs. cluster argument.

Cray could easily run Linux on their new shared memory supercomputers, it's just that for historical reasons they are using a (heavily modified) version of Irix (there has been some rumour of them changing to Linux in the future due to contractual issues over Irix).

The CTO is just bad-mouthing the Linux clusters because that is the current competition. If they were old-unix clusters the argument would be the same.

The point is really about hardware. Cray's shared memory machines are much easier to program for certian problems (eg old Fortran progs). Everyone agrees that for problems that can be suffieiently split up to run on a cluster with reasonable performance, that clusters are definately the way to go (which is why Cray will sell you a linux cluster). However there are certianly problems that cannot be split up to work on distributed memory machines with reasonable performance or problems where the effort required to rewrite the software is far too great. For example there are 1M LOC Fortran progs that have evolved continuously over 30 years where it is cheaper to buy an extremly expensive new supercomputer than to rewrite the software for a cluster.

As far as I know, Cray's shared memory interconnect is still an order of magnitude or so better in terms of bandwidth/latency than infiniband/myrinet etc. The other major difference is that supercomputer CPU's have hundreds of general purpose registers and several kilobytes worth of vector registers.

Disclaimer: I was an intern with Cray for 3 months several years ago.

Cray CTO: Linux clusters don't play in HPC (Search Enterprise Linux)

Posted Apr 14, 2004 6:51 UTC (Wed) by arasila (guest, #20891) [Link] (1 responses)

Cray could easily run Linux on their new shared memory supercomputers, it's just that for historical reasons they are using a (heavily modified) version of Irix (there has been some rumour of them changing to Linux in the future due to contractual issues over Irix).

This computer (Cray XD1) is a Linux system. Accoring to their own datasheet, the system is running "CRAY HPC enhanced Linux, Kernel version 2.4.21".

The system seems to be some kind of NUMA system, quite similar to SGI Altix, but with AMD Opteron CPUs instead of Itaniums. So, the guy is just trying to differentiate their from self-made Linux clusters and not to make any Linux vs. UNIX argument.

Cray CTO: Linux clusters don't play in HPC (Search Enterprise Linux)

Posted Apr 14, 2004 11:59 UTC (Wed) by dcoutts (guest, #5387) [Link]

I meant they could run Linux on thier traditional style supercomputers as well as their new cluster style machines.

At the moment, Cray have basically got two products:

'cluster style' machines (eg T3E,XD1) comprising of lots of off-the-shelf processors linked together with a proprietary interconnect to create a NUMA shared memory system.

Their other type (eg SV1,SV2) are the more traditional machines with monster vector cpus in SMP nodes and linked together (NUMA) with Cray's proprietary interconnect. These are the ones that run Irix but could just as easily run Linux.