Building a High-Performance Cluster with Gentoo

By using Gentoo, you can optimize compilation to your heart's content without being forced to leave the distribution's packaging system. The Portage package manager supports arbitrary setting of compilation flags and linker flags as well as non-GCC compilers. Fortran may seem like a dead language to many readers, but its use in scientific computing remains vast. Many HPC cluster administrators install multiple Fortran compilers, each with its own strengths and weaknesses, so supporting these compilers within a distribution's packaging system makes the admin's job significantly easier.

Creating a Gentoo-based cluster is not for the lighthearted, however. Less experienced Linux administrators who don't need to optimize their clusters for speed or size may wish to go with a prepackaged cluster distribution such as OSCAR, Rocks, or Warewulf. But if you need to get the most from your hardware, if you want to minimize your on-disk profile by leaving out useless features and packages, or if you enjoy the easy maintenance Portage provides, then use Gentoo. I founded the Gentoo Cluster Project four years ago to make Gentoo better for clustering by creating a community of cluster administrators and writing documentation to help those new to Gentoo or new to clusters. A major trade-off of using Gentoo rather than a prepackaged cluster distribution, in my mind, is increased initial set-up time but ongoing ease of administration. This is the same trade-off you will find in going with diskless rather than diskful clusters.

Gentoo's flexibility as a metadistribution means you can make whatever you want from it without hacking and slashing all over the place, as you may need to if starting from another distribution. Your changes to the base configuration are easy to find, document, and reproduce. You can even start out with something more minimal than a Gentoo base system by taking advantage of Portage's ROOT support to install only what you need to an arbitrary location (described in more detail in this LWN article). I find this most useful for diskless clusters. You can easily install to a location on an NFS server such as /opt/cluster/, which the diskless nodes use as their filesystem root. By using UnionFS to mount a read-only NFS root with tmpfs layered on top, all of the nodes can use the same filesystem without any concerns about multiple simultaneous writes. You can push only security fixes using `glsa-check`, and with a single invocation of `emerge`, you can manage full system updates to the server root or the diskless root.

Diskful clusters can also benefit from Gentoo. By now, you've probably wondered why anyone would use Gentoo on a diskful cluster, because they would need to compile every package on all of these hundreds of machines. But that isn't the case. Portage supports use of a binary package server, so you can compile packages just once per architecture rather than once per machine. For a serious cluster, you may wish to create more finely grained packages, however, based on the roles of machines within the cluster. File servers require a different set of features (USE flags, in Gentoo) than compute nodes, and they may even benefit from a different set of compilation flags, for example to produce smaller binaries and thus lower disk I/O.

Now you've learned a little about the basic idea behind a HPC cluster and how it works on Gentoo, but what about the applications and communications? A big stack of middleware makes it all possible. At the lowest level, all HPC programs have to talk to each other somehow. The dominant standard today is the Message Passing Interface (MPI). HPC programs must be specifically written to use MPI; it is not transparent to the application. MPI implementations are API-compatible, but regretfully, they are not ABI-compatible. Programs must be specially compiled for each MPI implementation they use. As with Fortran compilers, each MPI implementation has its strengths and weaknesses. One popular, "new" implementation is Open MPI. It's a merger of three existing implementations: FT-MPI, LA-MPI, and LAM/MPI. The other most popular, open-source implementation is MPICH2. Both projects are under active development, so testing them with your workloads is a requirement if you must choose one.

On the level above these custom-written applications sits a batching system such as Torque. This is where users send their computing jobs, and it takes care of the details of when and how to run these jobs. Submitted jobs sit in a queue until their turn, and the batching system can use a number of scheduling algorithms to decide when to run jobs. Sometimes, these simpler batching systems fall short of your needs. That's when you call in the big guns: something like Maui. It's an extremely flexible job scheduler that supports a vast array of scheduling policies, priorities, job reservations, and resource sharing.

At some point, a basic cluster like this will fall short of your needs. You may need to investigate specialized clustering filesystems such as LustreFS or PVFS2, migrate your network to something with better performance than basic Ethernet such as Myrinet or Infiniband, or find another solution to your problem. In clustering, the answer is almost always to benchmark and profile, because the problem is specific to your application rather than being generic to all clusters. Using Gentoo gives you the flexibility and power to make many of these changes while still staying within the Portage package management system.