Facebook helps establish Supercell testing infrastructure

Oregon State University's Open Source Lab (OSU-OSL) has gotten a hand up from Facebook to create a on-demand testing infrastructure for open source projects called Supercell. The idea behind Supercell is to provide limited duration hosting for open source projects that need to test on specific operating systems, as well as providing facilities for projects to test software in a large cluster with several VMs running concurrently. When finished, Supercell will provide test infrastructure for open source projects that don't have their own server farm and testing infrastructure.

The project was announced on Thursday, January 20 by Facebook's Scott MacVicar. MacVicar wrote that there's a disparity in development resources between many open source projects and companies doing in-house development of software: namely that many open source projects lack the kind of hardware and testing infrastructure that companies have at their disposal.

To help solve the problem, Facebook has decided to donate hardware and funding to OSU-OSL to develop Supercell, a service for projects to test on multiple operating systems and architectures. The Open Source Lab provides hosting to quite a few open source projects and communities, so it's not surprising that Facebook would look to OSL for assistance with Supercell. Why is Facebook interested? OSL's Leslie Hawthorn said that, while she didn't want to speak for Facebook, her conversations with the company indicated that Facebook's goal with open source is to "let people make useful stuff," and that OSL was a natural partner because "they know we're neutral, and we're here for the benefit of open source."

Currently the hardware is x86 and AMD64, with a number of guest OSes available. At present, Supercell supports Debian Lenny (5.0), CentOS 5.5, Gentoo, Gentoo Hardened, and Ubuntu Lucid (10.04), Karmic (9.10), and Maverick (10.10). According to MacVicar, some Mac OS X servers (two Apple Xserves) are also available "for those projects that explicitly need to test on Mac OS." The current hardware for Supercell is two Dell servers with 4 Opteron 2.1GHz 12-core CPUs and 128GB of RAM each, another with 2 Intel E5620 2.4GHz 4-core CPUs and 12GB of RAM, and 12TB of disk for NFS storage.

Plans are also on the table to support Fedora, FreeBSD, and OpenBSD in the near future, and OSU-OSL is evaluating feedback from the community in deciding on additional OSes and architectures. According to the FAQ, Supercell may support Alpha, ARM, ARMel, PowerPC, SPARC, and others as a longer term goal.

The hardware cluster is being managed by Ganeti and Ganeti Web Manager on top of Linux and KVM. The entire stack under Supercell is, of course, open source. Ganeti, which was originally developed by Google, is a tool for virtualization management that handles deploying and managing virtual machine instances on top of KVM or Xen. According to its documentation, Ganeti can deploy a new virtual machine running Ubuntu in under 30 seconds, including hostname, networking, and serial console set up. Images are a gzipped tarball or filesystem dump of an operating system, usually running between 200 and 400MB in size.

Ganeti has been in development for some time, with its initial public release in August 2007 under the GPLv2. Google started the project in its Zurich office for cluster management of virtual servers on commodity hardware — pretty much the same thing that OSU-OSL and Facebook were looking for.

The Ganeti Web Manager is a bit newer, a result of collaboration between OSL and Google Code-In students from the 2010 Google Code-In. Ganeti Web Manager is a Django-based application that provides Web-based management for Ganeti clusters. It's still maturing, but the 0.4 release from December 22, 2010 is considered "enough to get people to start using it in production" according to OSL's Lance Albertson. The 0.4 release implements basic VM management, VNC console, a permissions system for managing clusters and virtual machines, and SSH Key management.

Since the service is "on-demand," what happens when a project comes back for a second round of testing? According to OSL's operations manager Jeff Sheltren, OSL plans to tie into a configuration management framework such as Puppet so projects can save and reuse configurations. "This will allow OSL to provide a base set of standard configurations people can use (think: 'I need a LAMP stack') as well as giving projects the ability to fine tune their environment and re-use that configuration for future VMs."

Projects eager to get hands on with Supercell will have to wait a few more months, at least if they're hoping to use OSU's hardware and services. The service is considered early alpha at this point, with a projection that it will be ready by the third quarter of 2011. In the interim, OSU is looking to find additional sponsors for Supecell. Sheltren says that Facebook's donation amounts to about $50,000 in hardware and funds. This will support a fair amount of development, but there's plenty of work ahead.

Currently there is no discussion list for the Supercell service, but OSL is looking for feedback on what other operating systems that Supercell could support and other requests from the community. By submitting the feedback form, interested parties can sign up for the Supercell announcement list as well. Hawthorn did say that OSL may set up a discussion list, and will be providing regular updates about Supercell via the blog and Twitter.

But projects looking to implement their own "Supercell" can start today, just add hardware. Developers interested in helping with Ganeti and Ganeti Web Manager can find more info on the wiki including the mailing list and documentation.

Though Ganeti Web Manager is considered "production ready" by Albertson, it still has a lengthy roadmap of features that OSL plans to integrate. For instance, templates for virtual machines, ability to modify or reinstall VMs, implement support for noVNC instead of the Java VNC client, and serial console support.

With any luck, Facebook won't be the sole supporter of Supercell outside of OSU-OSL. The project has a lot of potential to provide a much-needed facility for short-term testing resources that many projects simply couldn't afford.

---

(Log in to post comments)