Building custom appliance distributions with rBuilder

As it happens, rPath has been in the appliance business for a little while now. Recently, the company has made its appliance-building infrastructure available to free-software products in the form of rBuilder Online. In essence, rBuilder can be used to create and maintain a custom distribution oriented around the delivery of a specific application. The result is a "software appliance" which, in theory, makes the given application available in a self-contained, standalone distribution.

There are a number of example appliances available on the site. They include:

• Bongo, an attempt to revitalize work on the Hula mail client

• Gallery, a standalone photo album

• LochDNS, a DNS server

• Openfiler, a storage management system

There are several others oriented around content management systems, telephony applications, database servers, and more. All told, quite a few projects have shown interest in creating software appliances for their applications.

Your editor grabbed a copy of the Openfiler appliance and installed it onto a spare box which had been cluttering up the office. Appliances from rBuilder start out looking like a Fedora system; they use the same Anaconda installer. The installed system also shows a lot of Red Hat heritage, such as /etc/sysconfig, various system-config-* commands, an /etc/inittab file which credits Mark Ewing and Donnie Barnes, etc. But there is a crucial difference: there is no rpm command. Instead, these appliances are based on rPath's Conary package management system, which takes a very different approach to the software management problem. But there are still similarities with Fedora: your editor attempted a conary updateall operation on the LochDNS appliance, only to see it fail with a set of file conflict errors; it was almost like running Rawhide again.

Appliance users are not supposed to have to dirty their fingertips with command-line administrative operations, though. To help them avoid this fate, rBuilder-based appliances come with the rPath Appliance Platform Agent, otherwise known as a web-based administration interface. Once the user gets past the usual set of obnoxious Firefox dialogs ("this site has an SSL certificate which is not only unknown, but is almost certainly hostile and is ugly besides"), this interface provides a set of administrative screens for standard tasks (networking, updating the system, etc.) along with some specific to the Openfiler application.

In theory, it should be possible to manage one of the appliances without ever going to the command line - or even knowing that the command line exists. In practice, how well that works depends a lot on how the administration screens are designed. In the Openfiler case, quite a bit of clicking around in circles was required, but your editor did finally succeed in setting up a volume based on a USB key, perform a software update, and shut down the system at the end.

The creation of appliances would appear to be relatively straightforward; details can be found in this document. One creates an account in the rBuilder system, then puts together a file describing which components (packages) are necessary in the final system. Those components will presumably include at least one application provided by the appliance builder - that application being the reason for the creation of the software appliance in the first place. The "rMake" system will then pull all of the pieces together, bring in any needed dependencies, and wrap it all up inside a minimal distribution; the resulting system image seems to run at about 300MB.

There are several possible output formats, including the Anaconda-based installation CD image; the rPath folks would appear to have put a lot of effort into making appliances work on a number of virtualization platforms as well. Appliances can be built for VMWare, various forms of Xen, VirtualIron, and Microsoft VHD. Notably absent is anything based on Lguest or KVM. Even more notably absent is any kind of live CD appliance; anything not running in a virtual machine must be installed onto the host system's disks.

rPath's Conary servers seem to be set up to handle software updates. It is also possible to obtain source for the packages found in an appliance through the rBuilder site, though one must do a little digging first. Both of these features are important: anybody creating a distribution-based appliance has to arrange for updates and source availability somehow. One assumes that most appliance creators have no real desire to get into the broader distribution business, so it's nice for them to be able to offload these tasks. Anybody distributing these appliance images should note that rPath does not appear to have undertaken any obligation to continue to provide these services in the future. Should rPath decide to stop, some interesting questions on who is ultimately responsible for satisfying the source-availability provisions of the GPL could come up.

Naturally enough, rPath offers commercial services for those who would like stronger guarantees about long-term support, or who want to include proprietary software in their appliances.

For the time being, this approach to software distribution would seem to be most useful for companies which are in the business of building real, hardware-based appliances. Distributing software in virtual machines has the look of a new and truly impressive form of bloat; even "just enough operating system" is a lot of baggage for an application to drag around. For situations where one wants to try out a complex system, appliance distribution may be worth its cost, but one would probably not want to get every application this way.

There may be value, though, in software distributions which can run almost anywhere, and which can be nicely isolated from the outside world. Locking network-exposed applications - server processes or web browsers - into their own little world could help to avoid a lot of security problems in a way which seems more straightforward than SELinux or containers.

But, perhaps most interestingly, the appliance approach could eliminate a number of distribution-compatibility issues by putting many more people into the distribution business. Now anybody can throw together a special-purpose distribution without having to deal with all of the plumbing that makes the whole thing actually work. Something interesting will certainly come of this idea, even if it's hard to say just what that might be at the moment.

Maybe that's not that bad an idea. It's the natural evolution from running each and every
process in a "virtual machine" that we have had for the last 30 years.

And if you use vservers and vhashify, some of this bloatware may actually be shared with
special vserver COW, Copy On Write, hardlinks.  This means that if different appliances use
certain matching libraries they will actually be shared (yet isolated) on disk and in memory!


I use vservers even without vhashify (since I want separate partitions for my vservers so that
I can fail them over to other machines) on my systems at home, and it still seems worth the
penalty to have isolated systems for each server application.  This eliminates most software
upgrade nightmares.  Think about the natural version hell of using various php/database
applications, all eliminated.   ~300MB on debian per major application does not seem like a
severe penalty to pay in the days when a standard HD is probably ~500GB and 4GB of memory is
cheap even for the home user.  

The editor should not dismiss this as a unreasonable solution so easily.  In fact I am
beginning to believe that it is becoming the ONLY reasonable solution to managing server
applications. ;)

I tried to watch the demo you linked to on the rPath website, but the Javascript it uses to
guess whether I have the Flash plugin guesses wrong.  It refuses to show me the content and
links to Adobe's download page despite having the Flash plugin installed in Firefox and
working with other sites.

It seems to work fine on Windows/IE though.

Thanks for letting us know!  I've forwarded this information internally at rPath.

For the record, I've only ever seen this run on Linux machines running Firefox, as far as I
recall, so it's not the case that it's broken for all Linux and/or Firefox installations.
I've seen what you describe only when running the noscript plugin (as I always do...) when I
haven't yet marked the necessary sites as allowed by noscript.

For the record, this has nothing to do with the noscript extension (I don't use it).  Mine is
a fairly standard Ubuntu packaged firefox with the Adobe flash plugin installed.

Just a quick note: we think we know why this is happening for you, and expect our next refresh
of the tour site (I don't know exactly when that will be) to fix the problem.  There's better
flash detection javascript code out there than the older version we had deployed.  Thank you
very much for mentioning the problem!

Would/will it be possible to build an appliance for special hardware such as in my case a qnap
ts-209 Pro II ? Their current "firmware" leaves much to be desired so I can see an rBuilder
based firmware being a really nice option...

"Even more notably absent is any kind of live CD appliance"

It is absolutely possible to create live CD/DVD appliances. rBuilder calls it a "demo CD";
marketing folks found that the term "live CD" was relatively unknown outside the Linux
community.

When in the Create Image page on rBuilder, they are also referred to as "Demo CD/DVD (Live
CD/DVD)".

Also absent from rPath is the ability to build an appliance release for OpenVZ which is OS
Virtualization / container based.  A number of pre-create OS Templates for various Linux
distributions are available from the OpenVZ Project and creating containers running the
desired Linux distribution is a snap. Turning a generic distribution container into a
customized appliance container is very easy... and it is very easy to turn an existing
appliance container into a new OS Template to use a cookie cutter for additional containers.

While OS Virtualization (OpenVZ and Linux-VServer) haven't gotten as much press attention and
third-party adoption as I think they warrant, with the "container" features slowly finding
their way into the mainline Linux kernel I can see a future where distributions makers,
including appliance distro makers like rPath, start making OS Templates for containers
available.

For the person who mentioned that creating a separate virtual machine for each appliance setup
is somewhat wasteful of resources... I agree completely and it is one of the reasons I believe
that OpenVZ containers are well suited for the appliance application market... since
containers provide very close to native performance but require less system resources than
machine / hardware virtualization scenerios yielding much greater density and better
scalability.  MKJ, any comment?

> one image type rBuilder builds is a compressed tar archive. 

This must be buried somewhere deeper, but I don't see a link to tarball e.g. here:
http://www.rpath.com/rbuilder/project/rpath/release?id=6071

MKJ,

Thanks for replying.  I don't think there is much interest in rPath / rBuilder generating an
OS image for an OpenVZ host node... what we would like see added would be the ability for your
users to output OpenVZ compatible OS Templates for running inside of containers.  These "OS
Templates" of which I speak are nothing more than a .tar.gz file of an installed Linux
distribution with the desired applications installed and perhaps some application
configuration modifications.

How does the OpenVZ OS Template differ from simply .tar.gz'ing up the root filesystem of a
Linux distribution from a physical machine?  OpenvZ OS Templates are subsets of those mostly
with unneeded software and services removed.  For example, there is only one kernel running on
the system and it is on the host node and is not needed in the container so that is
stripped... as are unnecessary services like udev, usually anything related to a desktop
environment, etc.  That's probably getting into the details that you don't care about.

Most of the configuration settings that would make a container unique (hostname, ip address,
and DNS settings, desired resource settings) are stored are not stored within the OS Template
but in a container specific configuration file on the host node.  When a container is created
from an OS Template and started, values are read from the container's configuration file on
the host node and config files inside of the container are modified to give the container its
unique network settings and resources.

I believe you asked if there was any OpenVZ specific software installed inside of the
containers.  The answer to that would be no... BUT as I mentioned, a typical Linux
distribution's default set of packages has been trimmed down quite a bit.

Did I address the issues you needed additional information on?

BTW, kolyshkin is Kir Kolyshkin who is the OpenVZ Project manager.

As a Bongo developer, I can say that I've been pretty impressed with what rBuilder can do. One
of the things which has been quite important to us has been the ability to give people a test
playground where they can look at the software without having to mess with their existing
stuff, and using rBuilder means we don't have to think about the OS bits.

The crack about Fedora and install conflict errors isn't fair. Or rather, it's perfectly fair
to one who recognizes the typical dry humor of our editor and who knows that Rawhide is the
built-daily Fedora development tree not designed for end-user consumption. The casual reader,
however, should not leave with the assumption that the production version of Fedora is ridden
with package management errors -- it's neither true, nor what the article means to say.