User: Password:
Subscribe / Log in / New account


News and Editorials

NLUUG: Minimizing downtime on servers using NanoBSD, ZFS, and jails

May 12, 2010

This article was contributed by Koen Vervloesem

On May 6, NLUUG held its Spring Conference with the theme System Administration. There were a lot of talks about very specific tools or case studies, but one struck your author because it married conceptual simplicity with a useful goal: Minimizing service windows on servers using NanoBSD + ZFS + jails by Paul Schenkeveld. Over the last four years, Paul has searched for methods to upgrade applications on a server with minimal downtime. The system he implemented is in production now on various servers, which require only a few seconds downtime for an application upgrade and the same amount of time for a rollback if the upgrade fails.

Jail time

System administrators have to keep their servers up-to-date, but when they upgrade the operating system or applications, the server may be unavailable for the users for some time, varying from a few seconds to much longer. If the administrator is lucky, he can schedule the upgrade after office hours when nobody uses the server, but if that's not possible, there will be some noticeable downtime for the users. In the latter case, it's extremely important that the downtime remains minimal, or the system administrator will face some angry shouting.

And that's where Murphy comes around the corner: upgrades are notorious ways to introduce unexpected problems. If an upgrade breaks things, the system administrator has to roll back to the previous upgrade. But this introduces some additional downtime, which is often longer than the upgrade downtime because rollbacks are not as easy as upgrades.

To minimize the risk of upgrades, there has been a trend to use one (virtual) server per application. Doing this, a problematic upgrade will impact only one application at a time. As a FreeBSD user, Paul obviously chose to isolate applications in jails, a lightweight form of operating-system-level virtualization. Each jail can be upgraded separately, minimizing the risk that other applications on the same server have to be brought down to fix one application's failed upgrade.

ZFS snapshots to the rescue

The root filesystem of each jail in Paul's system is in fact a read-only snapshot of the filesystem of a template jail (Paul called it a "prototype jail"). This allows the system administrator to prepare upgrades offline: upgrade the operating system and applications (FreeBSD ports) in the template jail and create a new snapshot of its filesystem (with the zfs snapshot command). Upgrading a production jail then involves stopping the jail, changing its root filesystem to the new snapshot and then restarting it. This upgrade takes only a few seconds because it has been prepared offline. Using snapshots rather than just multiple root filesystem images has the advantage that it saves space because the filesystems of all the jails are mostly identical.

When unexpected problems crop up, rolling back is as easy as stopping the jail, changing its root filesystem to the previous snapshot and then restarting it. The system administrator can then investigate and fix the problem offline, without any unneeded downtime.

Obviously, a read-only filesystem is not enough: the applications that run inside the jail need to be able to write their data. Therefore, the directory tree for a jail is set up as a combination of the read-only snapshot of the template jail's root filesystem with some read-write filesystems for the applications, such as /var and /home.

ZFS, which is deemed production ready since FreeBSD 8, easily handles many filesystems, has a flexible quotas system, and offers fast snapshots (thanks to the copy-on-write nature of the filesystem). Therefore, it's a perfect match to create a robust way of upgrades and rollbacks, something that has also been used by Nexenta.

Inspired by embedded systems

Combining jails and ZFS is straightforward, but Paul went further and thought about the underlying operating system. He got inspired by the way embedded systems are operated, and that's why he looked at NanoBSD. This is a toolkit that comes with the FreeBSD base system and that creates a FreeBSD system image for embedded applications that is suitable for use on a Compact Flash card.

NanoBSD is an interesting server choice for a number of reasons. First, it has the same functionality as FreeBSD, unless specific features are explicitly removed from the NanoBSD image when it is created. Moreover, every application that exists as a FreeBSD port or package can be installed and used in NanoBSD. The main differences are that the complete operating system is built and upgraded offline and that the root filesystem is mounted read-only.

The drive where NanoBSD is installed is divided into three partitions by default: two image partitions and one configuration partition. All of them are mounted read-only. The /etc and /var directories are memory disks (i.e. RAM disks). After the system boot, the configuration partition is briefly mounted read-only under the /cfg directory and all files in it are copied to /etc. If the system administrator wants to make persistent changes to a file (say, /etc/resolv.conf), they have to mount the configuration partition under /cfg, copy the modified files from /etc to /cfg and unmount the configuration partition. The whole system is set up in this way to minimize the number of writes to the flash drive and prolong its life. And with this completely read-only system, there is no necessity to run fsck after a non-graceful shutdown, so the system reboots with minimal downtime.

The update process of NanoBSD is also well thought-out. While the running NanoBSD is installed on one of the image partitions, a newly built NanoBSD image is written to the other image partition. Then the system is rebooted and started from the newly installed partition. If anything goes wrong, the system can be rebooted back into the previous partition, which still contains the old, working image. The system administrator can then investigate and fix the problem offline.

Tying it all together

Combining these three technologies (NanoBSD, ZFS, and jails), Paul reached his goal of setting up a FreeBSD server that can be upgraded with minimal downtime. All user-visible applications run in jails. Underneath the jails, a minimal FreeBSD operating system runs, built using the NanoBSD script. This holds the kernel, some low-level services, and the tools for building a new system image for upgrading the operating system. The NanoBSD system image can be put on a partition of a regular disk drive, but Paul prefers to put it on a separate flash drive, because NanoBSD is specifically designed for it and using a separate drive for the operating system makes it easier for the system administrator when the hard drives with the jails fail.

System administration on this system is much the same as on a regular FreeBSD system, except for software maintenance. Of course the embedded roots of NanoBSD mean that the system administrator needs to be aware of the differences from a regular server operating system. The volatile nature of /etc is one example: it's easy to forget to copy all changed configuration files to /cfg to preserve the changes after a reboot.

The directory /var is also a memory disk, so by default NanoBSD doesn't keep log files, which is not helpful to the system administrator. One solution is to put /var on a hard disk instead of using a memory disk, but then the operating system depends on the hard disk, which Paul wanted to avoid. Therefore, he chose another solution: telling syslog to log to a syslog daemon on another host or to a syslog daemon inside a jail on the same system.

A general architecture

Each system administrator has their own way of configuring servers. In the beginning of his talk, Paul warned that he didn't mean to provide the best solution for every situation, he just wanted to describe the way that he builds and maintains FreeBSD servers. Although his talk was specifically about the combination of NanoBSD, ZFS, and jails, the architecture he described was general enough to be usable elsewhere. The same ideas can be implemented with other minimal operating systems, other filesystems with snapshot abilities, and other forms of operating-system-level virtualization.

While FreeBSD jails are probably the most well-known type of operating-system-level virtualization, other operating systems have it too. OpenSolaris for instance has Zones, which are even more flexible than FreeBSD jails. Linux has similar solutions, such as OpenVZ and LXC (Linux containers). In particular, OpenVZ (and its proprietary variant Virtuozzo Containers) is popular among providers of virtual private servers. So on the virtualization level, the same architecture that Paul uses is perfectly possible on Linux.

The second important component in Paul's scheme are the filesystem snapshots. Although ZFS is not available for Linux (at least not in kernel space), there are many other snapshot technologies. For example, LVM (Logical Volume Manager) has a snapshots facility, just as Btrfs (which we looked at a while back). So for example with OpenVZ and LVM, Linux should be perfectly capable of creating OpenVZ containers based on read-only snapshots of some template container. Proxmox already makes use of LVM snapshots to create a backup of a container without downtime.

The last step Paul took is maybe the most interesting one: in a time where operating systems, even for servers, get more and more bloated, it's refreshing to see him take some inspiration from the embedded world. While at first the limitations of an embedded operating system on read-only storage seem too cumbersome, it actually helps a lot in clearly separating the operating system from its configuration and applications, which can only be good. Moreover, the dual-image approach and fast reboots have the advantage of more robust and less intrusive upgrades to the host operating system.

Because Linux shines in the embedded world, it's not very far-fetched to try this approach in the Linux world too. When leaving Paul's talk, your author heard a couple of people in the audience thinking out loud that it would be interesting to have such a system using Linux. A quick search didn't turn up anything useful, but it's clear that it should be possible, most likely with LVM snapshots and OpenVZ containers.

Comments (6 posted)

New Releases

Gilmore: Fedora 12 SPARC beta

Dennis Gilmore has announced the availability of Fedora 12 SPARC beta. "I have pushed a fedora 12 beta sparc tree the master mirror, you can find one close to you Here under /releases/test/12-Beta/sparc/ the tree is pretty much complete. there are a few broken deps that need resolving. Partitioning is fragile. it mostly works however sometimes you will be best off to do manual partitioning in rescue mode or breaking out into a shell."

Comments (none posted)

Linux Mint LXDE Debian 5.0 PPC is a fact

Jeroen Diederen has announced that an installer for Linux Mint LXDE based on Debian Lenny is available. "After hours and hours of hard work I can now all tell you with a lot of pride that a first version of an installer for Linux Mint LXDE based on Debian Lenny is a fact. This means that this is the first official installer of Linux Mint for PowerPC. For the moment only a 32-bits version is available as I don't have a 64-bits machine to test. The advantage of LXDE over GNOME, KDE or XFCE is that it is much snappier on old G3/G4 machines. The Mint themes make the desktop look a lot sexier than standard Debian. Linux Mint LXDE PPC comes with a huge amount of preinstalled programs, aimed at the desktop user."

Comments (none posted)

Lavergne: Lubuntu 10.04

Julien Lavergne has announced the release of Lubuntu 10.04. "Lubuntu is a project to make a Ubuntu variant using the LXDE desktop. It's designed to be a lightweight and easy-to-use desktop environment. Lubuntu is currently not part of the Ubuntu family, and not build with the current Ubuntu infrastructure. This release is considered as a "stable beta", a result that could be a final and stable release if it was included in the Ubuntu family. Please note also that Lubuntu 10.04 is not a LTS version."

Comments (none posted)

Mandriva Directory Server 2.4.0 available

Mandriva Directory Server 2.4.0 is available for download. Click below for a look at the new features.

Full Story (comments: none)

Slackware 13.1 BETA1

Slackware has announced Slackware 13.1 BETA1, as seen in the May 6, 2010 entry in the slackware-current changelog. "Hi folks! We have some pretty big changes today, with an update to the latest KDE SC 4.4.3, and the addition of support for ConsoleKit and PolicyKit which have been enhanced to use shadow authentication. Thanks to Andrew Psaltis for doing some great work on polkit-1, and to Robby Workman for spending months following the sometimes random developments coming from the CK/PK camp. :-) Thanks to Eric Hameleers for leading the KDE 4.4.x Slackware development and handling the out-of-tree testing through And with that, we're calling this Slackware 13.1 BETA1. A stable release should be just around the corner..."

Comments (2 posted)

Distribution News

Debian GNU/Linux

Release update: transitions status and freeze, RC-bugs

The Debian release team has an update on the "squeeze" release. "Due to the rate of change in unstable, it's not easy at the moment to accurately estimate when we might be able to freeze. In order to help us keep a clearer picture of which changes still need to occur before we can freeze, we will be introducing a "transition freeze" before the end of this month."

Full Story (comments: none)

Bits from the NM process: advocacy messages

Enrico Zini presents a few bits about the New Maintainer process. "after some recent debian-newmaint and IRC conversation, it makes sense to post a quick note about advocacy messages."

Full Story (comments: none)

New DMUP version 1.1.2

The Debian System Debian Administration Team has announced an updated version of the Debian Machine Usage Policies (DMUP). "Please note that this version only fixes the most important issues, such as cleaning up DAM vs. DSA responsibilities. While we are working on a new and improved DMUP that will hopefully fix more warts and will concentrate on the really significant issues, this is a work in progress which might take a while yet."

Full Story (comments: none)

porter chroot updates

Peter Palfrader has announced some new volunteers to help DSA in maintaining the development chroots on the porting machines. Click below for their names and some guidelines for install requests.

Full Story (comments: none) implications for developers

Debian keeps a copy of all packages (sources + binaries) and accompanying metadata that get uploaded into any of our archives on "Short version: If you uploaded stuff to debian that is not redistributable you will have to let the snapshot people know to remove it."

Full Story (comments: none)


Fedora 13 release slips

The traditional slip of the Fedora release date has happened right on schedule. Due to a few remaining blocker bugs, Fedora 13 is now scheduled to be released on May 25.

Full Story (comments: 23)

Fedora election update

The Fedora Project is holding elections for 3 seats on the advisory board and 5 seats on the engineering steering committee. The F14 elections questionnaire is available, with questions from the communtity and answers from the candidates.

Comments (none posted)

Making 3D Free for Innovation: Fedora 13 Graphics Drivers (Red Hat News)

Red Hat News continues its blog series highlighting features slated for Fedora 13 with a look at video drivers. "Following on the capabilities for the drivers for Intel and ATI based graphics cards, in Fedora 13, the Nouveau driver provides 3D hardware acceleration and is designed to support a wide array of NVidia based graphics cards. Because these drivers are fully free (as in freedom), open source software developers can build additional software against the functions they provide, taking fuller advantage of the hardware that users have purchased."

Comments (none posted)

Fedora names recipient of 2010 Fedora Scholarship

The Fedora Project has announced that Ian Weller is the recipient of the 2010 Fedora Scholarship. "Now in its third year, the Fedora Scholarship recognizes college and university-bound students across the globe for their contributions to free software and the Fedora Project. Weller has contributed to Fedora for more than two years as a packager and designer, and has also played an integral role in Fedora's project-wide transition to a new wiki system in 2009."

Full Story (comments: none)

Fedora 14 release name

The Fedora 14 release name is ... Laughlin. From the wiki page: "Robert Goddard was a professor of physics, and so is Robert Laughlin. He was awarded the 1998 Nobel Prize in physics for his explanation of the fractional quantum Hall effect. Moreover he argues for emergence which is a concept that says "The whole is more than the sum of its parts". Fedora is more than the sum of its software."

Full Story (comments: none)

SUSE Linux and openSUSE

Wafaa: Getting openSUSE from A to Y

Andrew Wafaa has put out a call to action on accessibility for openSUSE on his blog. "So as a community do you think this is something we could get behind? I would love to see openSUSE 12.0 released as the most accessible distribution; Ubuntu currently makes the statement that they are the most accessible desktop system available and this is a statement I would love to show as false! Not through animosity but through sheer prowess of the Geeko! To do that we (yes that means you at the back, listening to your boom boom music and chatting to your friends on FaceSpace) have to roll our sleeves up and get educated and start educating!"

Comments (4 posted)

Ubuntu family

Shuttleworth: Unity, and Ubuntu Light

Mark Shuttleworth takes a look at an Ubuntu experience mainly designed for dual-boot netbooks. "A few months ago we took on the challenge of building a version of Ubuntu for the dual-boot, instant-on market. We wanted to be surfing the web in under 10 seconds, and give people a fantastic web experience. We also wanted it to be possible to upgrade from that limited usage model to a full desktop. The fruit of that R&D is both a new desktop experience codebase, called Unity, and a range of Light versions of Ubuntu, both netbook and desktop, that are optimised for dual-boot scenarios."

Comments (12 posted)

Shuttleworth: No default GNOME Shell in Ubuntu 10.10 (The H)

The H reports on Mark Shuttleworth's question and answer session that was part of Ubuntu Open Week. "Because he wants a singular user experience, Shuttleworth is also not considering the proposal to make application defaults configurable at installation. 'One of the really strong values we have is that two users of Ubuntu should, by default, either be having the same experience, or be expert enough to understand why they are not'. He sees benefits in the common user experience allowing users to help each other more easily; 'They can help each other, just talking about "the browser" ... 'for the beginning, out of the box experience, we benefit a lot from keeping it tight'."

Comments (41 posted)

Amber Graner: Call For Nominations for the Elected Ubuntu Women Project

Last January LWN covered the Ubuntu Women Project. At that time the project had asked the Ubuntu Community Council to appoint an interim leader and Amber Graner became the transitional project leader. Now Amber reports that the project has opened nominations for a new Ubuntu Women Project Leadership Committee. Nominations will be open until May 21, 2010 for a 3 person leadership committee.

Comments (none posted)

Maverick is open for development

Martin Pitt has announced that Maverick Meerkat (Ubuntu 10.10) is now open for development.

Full Story (comments: none)

Other distributions

CentOS 3 6-Month End Of Life Notice

The CentOS project has announced that CentOS 3 will reach its end-of-life in six months. "It is recommended that any system still running CentOS 3 should be upgraded to a more recent version of CentOS before this date to ensure continued security and bug fix support."

Full Story (comments: none)

New Distributions


Quirky is a "puplet", a distribution based on Puppy Linux using the Woof build system. It was created by Puppy founder Barry Kauler as an outlet for some of his quirkier ideas. The recently released Quirky 1.0 "is quite straight, not very "quirky". Some of the interesting ideas that I want to try are still to come. The focus for now is to test a lot of the new stuff in Woof, such as rerwin's analog and 3G modem detection/setup scripts. Also, I have attempted to build a "multimedia special" that can play just about anything, despite the approx. 100MB size of the live-CD."

Comments (none posted)

Distribution Newsletters

DistroWatch Weekly, Issue 353

The DistroWatch Weekly for May 10, 2010 is out. "Mandriva Linux, a distribution that was one of the first to understand the concept of user-friendliness on the desktop, is apparently for sale and in negotiations with two potential buyers. That's according to some unconfirmed reports that appeared on the Internet over the weekend. But the company itself remains mute on the issue, while the development of the upcoming version 2010.1 continues unabated. In other news, Red Hat explains the genealogy of its enterprise kernels, Debian and Slackware update KDE to version 4.4.3 in their respective development branches, Sabayon announces availability of daily, bleeding-edge DVD builds, and Astaro apologises for last week's updates that went terribly wrong. Also in this issue, The Economist magazine explains the reasons for setting up a Launchpad account, while The Times urges users to abandon Windows and to switch to Ubuntu. Finally, for the fans of lighter distributions we have a first-look review of CDlinux, Canonical's announcement about a new "Unity" desktop for netbooks, news about a special edition of Unity Linux with Enlightenment, and an introduction to an inaugural release of Quirky, a new mini-distribution from the developers of Puppy Linux. There is something for everyone - happy reading!"

Comments (none posted)

Fedora Weekly News 224

The Fedora Weekly News for May 5, 2010 is out. "This issue begins with Project announcements, including another Fedora Community Gaming session this weekend, details on an upcoming Fedora Board IRC meeting, and opening of voting for naming Fedora 14. In news from the Fedora Planet, thoughts on a Fedora Kiosk spin, a report on Sugar-on-a-Stick testing, and what you don't know about NetworkManager. In Marketing team news, an update on the Allegheny Team and their accomplishments, report on some work for the F13 spins website, and an upcoming blog posting on hardware enablements in F13. One post from Fedora In the News this week, from Linux Gazette, covering some of the exciting aspects of Fedora 13 beta. In Design Team news, a new owner for Hackergotchis request service, and Security Advisories rounds out this issue with security-related Fedora packages released this past week for F11, F12 and F13. Read on!"

Full Story (comments: none)

openSUSE Weekly News #122

The openSUSE Weekly News for May 8, 2010 is out. "Now the eighteenth Week goes to the End, and we are pleased to announce our new issue. In this week we are busy with the Milestone 6. After that we're interested in the last Milestone (7). We would like to invite all of our readers to test the new Milestones. And please file all found Bugs into our buzilla. So we're hoping, that you like the new Weekly News. We wish you many joy by reading it..."

Comments (none posted)

Ubuntu Weekly Newsletter #192

The Ubuntu Weekly Newsletter for May 8, 2010 is out. "In this issue we cover: Maverick is open for development, Call for Ubuntu User Days Instructors, Window indicators, New Ubuntu Regional Membership Boards, Maverick UDS Translations Sessions, Patch Day Success, Ubuntu Open Week en Español closes on high note, Ubuntu Open Week - Lucid: Community, Canonical, Collaboration, Call For Nominations: Ubuntu Women Leader Leadership Committee, Ubuntu Server and Apache Tomcat - supporting MuleSoft, Full Circle Podcast #6: Mark's Space Brain from the Future, and much, much more!"

Full Story (comments: none)

Distribution reviews

Peppermint OS: a review (Linux Critic)

The Linux Critic reviews Peppermint OS. "What do you get when you combine the flexibility, versatility and ease of maintenance of Ubuntu, the blinding speed and simplicity of LXDE, and a focus on social media and the cloud? You get Peppermint OS, that's what! Brought to you by the same developer responsible for Linux Mint 8 LXDE Community Edition, and for resurrecting Linux Mint Fluxbox CE as well, Peppermint OS is a lightweight, fast, stable implementation of what Kendall Weaver's vision of the perfect Linux distro might be for speed and the web."

Comments (none posted)

RHEL 6 - your sensible but lovable friend (Channel Register)

The Register has a review of Red Hat Enterprise Linux 6. "For RHEL 6, Red Hat is using a Fedora development release based on the Linux 2.6.32 kernel - technically, it's a hybrid of several recent kernels. Red Hat engineers have hardened the Fedora base and added quite a few features - with a strong emphasis on virtualization."

Comments (none posted)

Review: Ubuntu Enterprise Cloud (ComputerWorld)

ComputerWorld takes a look at creating a private cloud using Ubuntu Linux 10.04. "In keeping with its open source pedigree, Ubuntu Enterprise Cloud is integrated with the open source Eucalyptus private cloud platform, making it possible to create a private cloud with much less configuration than installing Linux first, then Eucalyptus. And for those thinking about eventually moving resources to the public cloud, or simply bursting to the public cloud when workloads spike, the Ubuntu/Eucalyptus internal cloud offering is designed to be compatible with Amazon's EC2 public cloud service."

Comments (none posted)

Ubuntu 10.04 LTS Server Is Getting There (eWeek)

Joe "Zonker" Brockmeier reviews Ubuntu 10.04 LTS Server. "Canonical and the Ubuntu Project have done great things to help bring Linux to the mainstream desktop. But what about the server edition? If Ubuntu can bring the same level of polish to its server offerings, it should be a formidable competitor to Microsoft and other Linux vendors. Looking at Ubuntu Server 10.04, aka "Lucid Lynx," there's a lot to like and also some disappointments."

Comments (none posted)

Lucid dream: Ars reviews Ubuntu 10.04 (ars technica)

Ryan Paul has written a lengthy review (split into nine pages) of Ubuntu 10.04. "Although Ubuntu still has a long road ahead before it will fulfill the aspirations of its creators, version 10.04 includes a number of impressive improvements that artfully advance the platform. The new theme and more cohesive branding contribute to a more compelling visual appearance, tighter integration with the cloud expands the boundaries of the desktop, and usability improvements enrich the user experience."

Comments (none posted)

Page editor: Rebecca Sobol
Next page: Development>>

Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds