User: Password:
Subscribe / Log in / New account


Facebook helps establish Supercell testing infrastructure

February 2, 2011

This article was contributed by Joe 'Zonker' Brockmeier.

Oregon State University's Open Source Lab (OSU-OSL) has gotten a hand up from Facebook to create a on-demand testing infrastructure for open source projects called Supercell. The idea behind Supercell is to provide limited duration hosting for open source projects that need to test on specific operating systems, as well as providing facilities for projects to test software in a large cluster with several VMs running concurrently. When finished, Supercell will provide test infrastructure for open source projects that don't have their own server farm and testing infrastructure.

The project was announced on Thursday, January 20 by Facebook's Scott MacVicar. MacVicar wrote that there's a disparity in development resources between many open source projects and companies doing in-house development of software: namely that many open source projects lack the kind of hardware and testing infrastructure that companies have at their disposal.

To help solve the problem, Facebook has decided to donate hardware and funding to OSU-OSL to develop Supercell, a service for projects to test on multiple operating systems and architectures. The Open Source Lab provides hosting to quite a few open source projects and communities, so it's not surprising that Facebook would look to OSL for assistance with Supercell. Why is Facebook interested? OSL's Leslie Hawthorn said that, while she didn't want to speak for Facebook, her conversations with the company indicated that Facebook's goal with open source is to "let people make useful stuff," and that OSL was a natural partner because "they know we're neutral, and we're here for the benefit of open source."

Currently the hardware is x86 and AMD64, with a number of guest OSes available. At present, Supercell supports Debian Lenny (5.0), CentOS 5.5, Gentoo, Gentoo Hardened, and Ubuntu Lucid (10.04), Karmic (9.10), and Maverick (10.10). According to MacVicar, some Mac OS X servers (two Apple Xserves) are also available "for those projects that explicitly need to test on Mac OS." The current hardware for Supercell is two Dell servers with 4 Opteron 2.1GHz 12-core CPUs and 128GB of RAM each, another with 2 Intel E5620 2.4GHz 4-core CPUs and 12GB of RAM, and 12TB of disk for NFS storage.

Plans are also on the table to support Fedora, FreeBSD, and OpenBSD in the near future, and OSU-OSL is evaluating feedback from the community in deciding on additional OSes and architectures. According to the FAQ, Supercell may support Alpha, ARM, ARMel, PowerPC, SPARC, and others as a longer term goal.

The hardware cluster is being managed by Ganeti and Ganeti Web Manager on top of Linux and KVM. The entire stack under Supercell is, of course, open source. Ganeti, which was originally developed by Google, is a tool for virtualization management that handles deploying and managing virtual machine instances on top of KVM or Xen. According to its documentation, Ganeti can deploy a new virtual machine running Ubuntu in under 30 seconds, including hostname, networking, and serial console set up. Images are a gzipped tarball or filesystem dump of an operating system, usually running between 200 and 400MB in size.

Ganeti has been in development for some time, with its initial public release in August 2007 under the GPLv2. Google started the project in its Zurich office for cluster management of virtual servers on commodity hardware — pretty much the same thing that OSU-OSL and Facebook were looking for.

The Ganeti Web Manager is a bit newer, a result of collaboration between OSL and Google Code-In students from the 2010 Google Code-In. Ganeti Web Manager is a Django-based application that provides Web-based management for Ganeti clusters. It's still maturing, but the 0.4 release from December 22, 2010 is considered "enough to get people to start using it in production" according to OSL's Lance Albertson. The 0.4 release implements basic VM management, VNC console, a permissions system for managing clusters and virtual machines, and SSH Key management.

Since the service is "on-demand," what happens when a project comes back for a second round of testing? According to OSL's operations manager Jeff Sheltren, OSL plans to tie into a configuration management framework such as Puppet so projects can save and reuse configurations. "This will allow OSL to provide a base set of standard configurations people can use (think: 'I need a LAMP stack') as well as giving projects the ability to fine tune their environment and re-use that configuration for future VMs."

Projects eager to get hands on with Supercell will have to wait a few more months, at least if they're hoping to use OSU's hardware and services. The service is considered early alpha at this point, with a projection that it will be ready by the third quarter of 2011. In the interim, OSU is looking to find additional sponsors for Supecell. Sheltren says that Facebook's donation amounts to about $50,000 in hardware and funds. This will support a fair amount of development, but there's plenty of work ahead.

Currently there is no discussion list for the Supercell service, but OSL is looking for feedback on what other operating systems that Supercell could support and other requests from the community. By submitting the feedback form, interested parties can sign up for the Supercell announcement list as well. Hawthorn did say that OSL may set up a discussion list, and will be providing regular updates about Supercell via the blog and Twitter.

But projects looking to implement their own "Supercell" can start today, just add hardware. Developers interested in helping with Ganeti and Ganeti Web Manager can find more info on the wiki including the mailing list and documentation.

Though Ganeti Web Manager is considered "production ready" by Albertson, it still has a lengthy roadmap of features that OSL plans to integrate. For instance, templates for virtual machines, ability to modify or reinstall VMs, implement support for noVNC instead of the Java VNC client, and serial console support.

With any luck, Facebook won't be the sole supporter of Supercell outside of OSU-OSL. The project has a lot of potential to provide a much-needed facility for short-term testing resources that many projects simply couldn't afford.

Comments (none posted)

Brief items

Quotes of the week

To be honest, I don't really understand how Emacs development works -- it seems to be an anarchic free-for-all, but I have trouble accepting that this could possibly be the case.
-- Tom Tromey

I'm coming to think of POSIX compatibility more as a legacy matter than something genuinely useful in a Unicode world.
-- Tom Christiansen

Comments (none posted)

Celery 2.2 released

Celery is "an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well." The version 2.2 release adds support for eventlets and gevents, Jython support, built-in virtual transports, messaging with Kombu, and more.

Full Story (comments: 1)

crosstool-NG 1.10.0 released

crosstool-NG is a system for the easy building of cross-compiler toolchains. "crosstool-NG can build from generic, general purpose toolchains, to very specific and dedicated toolchains. Simply fill in specific values in the adequate options." The 1.10.0 release is out; it adds support for gcc 4.5.2, the gold linker, gcc plugins, and building statically-linked toolchains.

Comments (1 posted)

HTSQL 2.0-FINAL released

HTSQL is a gateway allowing the encoding of SQL queries in URL syntax. "The target audience for HTSQL is the accidental programmer -- one who is not a SQL expert, yet needs a usable, comprehensive query tool for data access and reporting." The idea is interesting but one hopes that the security aspects have been thought through; there is little mention of security on the project web site.

Full Story (comments: none)

KDevelop 4.2

KDevelop 4.2 has been released. There's a lot of improvements in this release, including search-and-replace functionality, better documentation, improved C++ support, a "rename assistant," and more.

Comments (1 posted)

MPlayer 1.0rc4 released

The MPlayer 1.0rc4 release is out. "Notable additions are VP8 decoding, H.264 bug fixes and speedups, unencrypted Blu-ray support. Network streams can now be played through FFmpeg, there has been quite a bit of subtitle work and Ogg and Matroska demuxer defaults were switched to libavformat. The window position is now decided by the window manager."

Full Story (comments: 1)

Pyramid 1.0 released

The developers of repoze.bfg have come to the credible conclusion that "Pyramid" is a better name for their web application framework project. The Pyramid 1.0 release is now available; new features (beyond the name change) include a number of template improvements, built-in session support, support for Mako templating, improved configuration, and more. See the "what's new" page for more information.

Comments (none posted)

Newsletters and articles

Development newsletters from the last week

Comments (none posted)

Openmoko Community Update

The Openmoko Community Update for February 1 has been released with news of new Openmoko hardware. "GTA04 is a project by the long time distributor and hw developer, German company Golden Delicious. The name is loaned from Openmoko project because of the spiritual continuation - GTA01 was the codename for Neo1973, GTA02 was the Neo FreeRunner, and GTA03 was the canceled successor product. Besides offering improved versions of Neo FreeRunner (better battery life, better audio output), they've a complete replacement board planned to fit an existing Neo FreeRunner case and use the existing display." The new board has an OMAP3530 ARMv7 CPU, UMTS/3G (HSPA), USB 2.0 OTG, WLAN, BT, and FM transceiver, Barometric Altimeter, Accelerometer, Compass, Gyroscope, and optionally a camera. The update also includes information on various distribution releases for Openmoko devices. (Thanks to Sam Tygier.)

Comments (5 posted)

Open source powers new Aussie space race (Computerworld)

Computerworld reports on a talk about an open hardware/software effort targeting a moon landing. The Lunar Numbat project aims to put a Linux-powered robotic marsupial on the moon. "The Australia defence force has donated a stockpile of rockets like the Zuni missile to ASRI for experiments. 'We are designing and implementing electronic controllers for the rockets using open software and open hardware,' [Luke] Weston said. Back in 2004 researchers at the University of NSW also worked on putting Linux in space for one of its satellite projects. The software is generally licenced under the GPL and the hardware under the TAPR Open Hardware licences."

Comments (none posted)

Phipps: The Open-By-Rule Governance Benchmark

Simon Phipps offers some suggestions on how to judge a project's governance. "Can you find everything about the community, including why things happened as well as what happened? Are all the governance conversations visible (apart from the bits where personal privacy is appropriate)? Can you track all the commits and find out why each was made? An open-by-rule community will have it all there somewhere, including the dirty laundry (arguments, trolls and the like)."

Comments (10 posted)

Zander: Views and a Conversion

Thomas Zander presents his vision of KOffice on the blog. The Calligra suite split off from KOffice back in December, and Zander is describing what he sees for KOffice going forward. "My idea for KOffice is to make a set of applications that help people get their work done by being fun and with those plugins there are unimagined possibilities to get users to have fun and to get more types of work done quicker. Specifically the idea of what an office suite is should be [challenged]. Its useless to try to compete in an already saturated market by just copying the competition. Just like with KDE4 its much more fun to build a platform that allows both the old and imaginative new usages."

Comments (7 posted)

Page editor: Jonathan Corbet
Next page: Announcements>>

Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds