The 2007 Linux Storage and File Systems Workshop [LWN.net]

The 2007 Linux Storage and File Systems Workshop

Posted Mar 21, 2007 13:47 UTC (Wed) by saffroy (subscriber, #43999)
In reply to: The 2007 Linux Storage and File Systems Workshop by drag
Parent article: The 2007 Linux Storage and File Systems Workshop

It's nice that you mention Lustre, actually I was kind of surprised that it would not be mentioned in this article. Lustre definitely has a great potential (great scalability, sequential I/O performance, client cache, excellent POSIX conformance), and I feel it could be a good general purpose global fs someday.

That is, if its creators (CFS) let it grow out of its niche HPC market: at the moment, I feel they're more concerned about implementing the features asked by their paying customers, which are big supercomputing centers. I'm certainly not blamining them for that, but for instance, they are more sensitive to large file throughput (tens of GB/s) than to file creation rates (Lustre is still damn slow here).

If the community or the customers push in the right direction, Lustre can become an excellent distributed fs for nearly everyone, but I feel it has yet to happen -- and I hope it will.

Oh, and don't take CFS roadmaps too seriously. ;-)

(Log in to post comments)

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 2:01 UTC (Thu) by drag (subscriber, #31333) [Link]

I don't take them too seriously. :)

But they aren't to far off. One thing worth noting is that Ext4 integration isn't mentioned anywere on them, but it's obvious that Ext4 is going to play a large part in it.

One thing about CFS, which I think is important to keep in mind, is that they are decendents of the failed Coda and then the Intermezzo projects. I don't know the exact relationships, but I think that they were developers in those projects.

The thing is is that they learned the hard way that distributed network file system protocols aren't a easy thing to make, even if you are good at it. It takes a lot of time and effort to get anything going and a long time of development to get to the point were you can actually release anything.

So it's not something that lends itself to the Linux-style development proccess of 'release early', 'release often'.

So they formed CFS to pursue the money nessicary to support themselves while they hacked on Lustre full time. The HPC market is the easiest and most profitable place to target for this sort of stuff, and they know that from Beowolf stuff that open source and distributed computing can lead to dramatic results.

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 11:56 UTC (Thu) by nix (subscriber, #2304) [Link]

Of course Coda itself was an enhancement of AFS (losing most of its scalability in the process, though)...

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 17:08 UTC (Thu) by jwb (subscriber, #15467) [Link]

What do you mean by Ext4 integration? Lustre already includes all Ext4 features and then some. In fact you might say that Ext4 is just rolling features into the mainline kernel that have long been used in Lustre. mballoc, delalloc, and extents have all been in Lustre for years.