OK, so now we have Linux running on everything...
OK, so now we have Linux running on everything...
Posted Jun 26, 2005 11:48 UTC (Sun) by dgc (subscriber, #6611)In reply to: OK, so now we have Linux running on everything... by jwb
Parent article: Linux and the Top500
Linux ext3 tops out at about 400MB/s read or write on today's fastest machines. XFS doesn't seem to have a read limit, but writes don't exceed 500MB/s for some reason.
I assume you are referring to these results, right?
http://scalability.gelato.org/DiskScalability_2fResults
24 Sata disks doesn't seem like the sort of setup to be able to do multiple GiB/s of write throughput to me. You're comparing that to a result from a machine with 2500 disks attached!
Sure, they may have identified a bottleneck, but it's quite likely that it is a hardware bottleneck that is the issue problem here.
Posted Jun 26, 2005 14:53 UTC (Sun)
by whitemice (guest, #3748)
[Link] (3 responses)
Yep. Comparing 2500 fiber-channel attached spindles to 24 SATA spindles is absurd; that is like comparing the Grand Canyon to a road-side ditch. SATA is barely an enterprise-grade storage system, I wouldn't be suprised if an FC cage of 12 spindles out-ran an 24 spindle SATA cage.
Posted Jun 26, 2005 16:10 UTC (Sun)
by jwb (guest, #15467)
[Link] (2 responses)
Unlike the lot of you blind sycophants, I've actually tried to boost Linux filesystem I/O past the 500MB/s barrier. It just doesn't work. I have an SATA setup here capable of 2GB/s linear reads, but it only hits about 450MB/s when using ext3. And it doesn't even matter if I add or remove CPUs: still 450MB/s. That's what we call a scalability barrier.
Read the paper that gdt linked earlier in the thread. The folks at CERN improved disk bandwidth by a large factor just by switching off Linux to Windows.
Posted Jun 26, 2005 17:33 UTC (Sun)
by iabervon (subscriber, #722)
[Link]
Is this appending tons of data to a single file, or to a set of files, or writing a ton of small new files, or what? That's obviously going to matter in how much the filesystem affects the result. Have you tried just writing the data to a block device without a filesystem, to see if you're maxing out the SATA drivers or something?
Posted Jun 27, 2005 13:21 UTC (Mon)
by rakoch (guest, #4666)
[Link]
>24 Sata disks doesn't seem like the sort of setup to be able to do multiple OK, so now we have Linux running on everything...
>GiB/s of write throughput to me. You're comparing that to a result from a
>machine with 2500 disks attached!
I would be surprised, because I've tried it. SATA is faster than FC and more efficient. Using port multipliers you can quite easily saturate the 300MB/s nominal speed of current SATA channels.OK, so now we have Linux running on everything...
Have you done any profiling to figure out where it's blocking? I'd guess that it's something like your journal being too small (for this load, not in general) or some setting limiting the amount of I/O in progress. I think that having a lot of simultaneous outstanding SATA requests is a feature still under development, so it might be that.OK, so now we have Linux running on everything...
So where is the bottleneck? Inefficiency in handling some latency? CPU OK, so now we have Linux running on everything...
overuse because of Filesystems being single threaded?
Just for comparison: Did you try how sequencial raw i/o goes?
-Rudiger