The 2010 Linux Storage and Filesystem Summit, day 2

Posted Aug 11, 2010 21:24 UTC (Wed) by sniper (subscriber, #13219) [Link]

Posted Aug 13, 2010 23:29 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

... I test 'cluster' and 'gfs2' quite often on a test cluster that comprises two VMs ...

As a hobby? The point was that hobbyists don't care about cluster filesystems because they don't have enterprise-level storage arrays and cluster filesystems aren't good for anything else.

Posted Aug 15, 2010 23:33 UTC (Sun) by tytso (subscriber, #9993) [Link]

Enterprise-level storage arrays aren't required to run clustered filesystems. In fact, a single machine (laptop in my case) is sufficient, assuming you can run either Xen or KVM.

Shared-block cluster file systems (which is what Red Hat's GFS and OCFS2 are) don't provide any redundancy in case of failure; they depend on the hardware being utterly reliable. Which generally means enterprise storage arrays which are FC connected, which can be equally accessed by all all of the nodes in the cluster file system. You don't _have_ to run them on an enterprise storage array, but then if any of the hard drives fail, you're really badly out of luck.

There are other possibilities, such as a network block device connected a server which uses multiple disks via an MD software RAID device, but your nbd server becomes a single point of failure. In theory at least an enterprise storage array has many more redundancies and can survive a controller card or hard drive failure, even though the enterprise storage array itself is still a single point of failure.

There are other types of cluster file systems, such as Google's GFS, Hadoopfs (which is basically a copy of GFS as described in the GFS paper), Lustre, Ceph, etc. These generally use an object-based storage paradigm with the objects replicated across multiple object stores, so you can survive a single disk or a single server biting the dust.

But what was being discussed in the original post was really focused on the two cluster file systems currently in the kernel which are supported by enterprise distributions: GFS2 and OCFS2, which are really very much identical in terms of feature set, scalability, and design at the 10,000 foot level. The fact that we have two of them is largely an accident of history, but it's splitting the amount of resources available hack on shared-block cluster file systems, which since they require rather specialized and expensive equipment in order to use them practically, tends to constrain the size of their developer communities.