Weekly edition	Kernel	Security	Distributions	Contact Us	Search
Archives	Calendar	Subscribe	Write for LWN	LWN.net FAQ	Sponsors

The 2007 Linux Storage and File Systems Workshop

Posted Mar 21, 2007 0:02 UTC (Wed) by saffroy (subscriber, #43999)
In reply to: The 2007 Linux Storage and File Systems Workshop by bronson
Parent article: The 2007 Linux Storage and File Systems Workshop

"There's no good solution today."

Of course there is one, it's called OpenAFS. Except installing and managing an AFS cell is not quite easy (although the Debian packager provides useful scripts to quickly start a simple server)...

But AFS has its share of good features:
- good Linux and Windows clients (and other *nixes as well)
- Kerberos authentication
- powerful ACLs
- excellent scalability: large sites (MIT, CMU...) run cells with thousands of clients (and several servers sharing the load of course)
- client-side caching using local storage
- the namespace is global, users can use the same paths on all clients
- data is organized in volumes (subtrees) that can be mounted anywhere in the global namespace, both by admins and users
- volumes can be snapshotted, and users can mount snapshots (no need for the admin to restore from tapes when user accidentally deletes a file and notices immediately)

...and more. :)

I've used OpenAFS on Linux for years, and never complained (it helps that I have a competent and knowledgeable admin).

(Log in to post comments)

The 2007 Linux Storage and File Systems Workshop

Posted Mar 21, 2007 2:54 UTC (Wed) by drag (subscriber, #31333) [Link]

OpenAFS is, indeed, very nice.

But it's Windows support is crap. Not because OpenAFS is not cool, but because Window uses SMB or Microsoft's DFS to do it's thing and nobody else's. OpenAFS has to use a sort of SMB emulation were it deals with AFS stuff then translates that to something that the system can use.

But if your just dealing with Linux clients then that's not a problem.

Also the file and directory permission model is bizzare and isn't realy compatable with just standard Unix-style ACL (user/group/world read/write/execute) model. So people used to Linux permissions have to relearn how to deal with AFS permissions.

It's not posix, and it's not compatable with special file types like named pipes.

Also there is no real way to access your data unless your AFS server stuff is actually running. OpenAFS tends to incure a higher amount of knowledge and administration stuff isn't very easy to deal with.

Then it's large file performance is realy bad. It's just plain slow and the volumes are very limited in size.

What it's VERY good for is if you have a large distributed network.

Say you have a wireless network or a WAN-wide thing were you have a entire campus of computers to take care off. It handles disconnection very well, it's caching support is very good for semi-offline work (ie you can still edit a file even if you temporarially lost contact with the servers.

It's security stuff is nice. The volume management is very nice, snapshotting and mirroring stuff. It's safe to use over the internet and unencrypted wireless networks.

And as a special bonus it's /afs/ directory tree is very handy. It allows people to move volumes around, change servers, setup mirrors, and all sorts of stuff without having to have the clients know of any of these changes. Were as with NFS or SAMBA if you change out file servers or whatnot then the clients all have to be reconfigured to know the new locations and names of the servers and directories. With OpenAFS this is not nessicary.

But considuring the lack of posix support and poor large file performance as well as permission issues it's not realy a replacement for NFS. It's a alternative that is usefull in places were NFS is not.

And it's poor Windows support means that it's not usefull as a replacement for Samba.

But it's nice.

The OpenAFS points out a huge problem for Linux in general though. AFS is ancient. It's old old old. It's like X Windows/Athena/Kerberos ancient. Still, even with it's age, it's still MUCH more sophisticated then NFS or SMB network protocols. Nobody has realy produced anything better.

Lustre, maybe. It certainly has a lot of cool features and is fast. But I don't think that it has any security.

Supports lots of stuff. TCP networking, ininaband, and all sorts of other bizzare interconnects.

Supports ACLs, extended ACLs, extended attributes. Lots of high aviability and high performance features. Failover, extra redudancy. You can use it as root FS. It supports Quotas.

It doesn't require special patches and kernel recompiles for Linux client support.

The only thing that it lacks is robust security. They plan on supporting GSSAPI and Kerberos with the 1.8.0 release. This is due out by the end of this year according to their roadmap...
For Unix and Windows comaptability it supports SMB and NFS v2/v3/v4 export.

The 2007 Linux Storage and File Systems Workshop

Posted Mar 21, 2007 13:47 UTC (Wed) by saffroy (subscriber, #43999) [Link]

It's nice that you mention Lustre, actually I was kind of surprised that it would not be mentioned in this article. Lustre definitely has a great potential (great scalability, sequential I/O performance, client cache, excellent POSIX conformance), and I feel it could be a good general purpose global fs someday.

That is, if its creators (CFS) let it grow out of its niche HPC market: at the moment, I feel they're more concerned about implementing the features asked by their paying customers, which are big supercomputing centers. I'm certainly not blamining them for that, but for instance, they are more sensitive to large file throughput (tens of GB/s) than to file creation rates (Lustre is still damn slow here).

If the community or the customers push in the right direction, Lustre can become an excellent distributed fs for nearly everyone, but I feel it has yet to happen -- and I hope it will.

Oh, and don't take CFS roadmaps too seriously. ;-)

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 2:01 UTC (Thu) by drag (subscriber, #31333) [Link]

I don't take them too seriously. :)

But they aren't to far off. One thing worth noting is that Ext4 integration isn't mentioned anywere on them, but it's obvious that Ext4 is going to play a large part in it.

One thing about CFS, which I think is important to keep in mind, is that they are decendents of the failed Coda and then the Intermezzo projects. I don't know the exact relationships, but I think that they were developers in those projects.

The thing is is that they learned the hard way that distributed network file system protocols aren't a easy thing to make, even if you are good at it. It takes a lot of time and effort to get anything going and a long time of development to get to the point were you can actually release anything.

So it's not something that lends itself to the Linux-style development proccess of 'release early', 'release often'.

So they formed CFS to pursue the money nessicary to support themselves while they hacked on Lustre full time. The HPC market is the easiest and most profitable place to target for this sort of stuff, and they know that from Beowolf stuff that open source and distributed computing can lead to dramatic results.

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 11:56 UTC (Thu) by nix (subscriber, #2304) [Link]

Of course Coda itself was an enhancement of AFS (losing most of its scalability in the process, though)...

The 2007 Linux Storage and File Systems Workshop

Posted Mar 22, 2007 17:08 UTC (Thu) by jwb (guest, #15467) [Link]

What do you mean by Ext4 integration? Lustre already includes all Ext4 features and then some. In fact you might say that Ext4 is just rolling features into the mainline kernel that have long been used in Lustre. mballoc, delalloc, and extents have all been in Lustre for years.