Illumos: the successor to the OpenSolaris community

In the August 2010, storage vendor Nexenta Systems announced Illumos, as a project to free the OpenSolaris distribution. The ultimate goal of Illumos is twofold: to replace all proprietary code that is still in OpenSolaris by open source code, and to build an independent community around the former OpenSolaris code base. Now almost 10 months later, Nexenta Systems organized its Nexenta European User Conference in Amsterdam on May 20 and presented its plans for Illumos and the Nexenta products, so this is the perfect opportunity to take a look at where Illumos is heading.

Code

The Illumos code base forms or will form the foundation of a couple of distributions, including a storage appliance from Nexenta, a cloud platform from Joyent, and the general-purpose OpenIndiana (which is the spiritual successor of the OpenSolaris distribution). One of its initial goals was to replace the few remaining binary-only components of the OpenSolaris code base by open source counterparts. Today, Illumos still has some of these binary-only components, such as the NFS lock manager, the IKE daemon (Internet Key Exchange, part of IPsec), some support libraries, and a few drivers, but according to Garrett D'Amore (Director of Engineering at Nexenta Systems) the Illumos developers are busy addressing all those issues. Also, while last year compiling Illumos still depended on the proprietary Sun Studio compiler, Illumos now also builds using GCC, although it needs more testing.

Illumos has already received code contributions from some hardware vendors, such as Areca and 4Front, and according to D'Amore there are additional code contributions in the pipeline, especially from vendors of host bus adapters and network interface cards. Moreover, Intel and LSI are technology partners of Nexenta, the main sponsor of Illumos, which means that Nexenta has access to specifications and engineering support of these companies. According to D'Amore, code contributions from both vendors will be coming. The most notable absentees here are AMD and Nvidia, which apparently don't have a friendly relationship with Illumos/Nexenta. However, in the comments of one of his blog posts, D'Amore explained that these relationships or non-relationships are not carved in stone: "I'm happy to be friendly with either NVDA or AMD or any other vendor that wants to reciprocate. Its just that at this point, only Intel of those three has stepped up for illumos or Nexenta."

Growing pains

Recently, there was some criticism about the way Illumos is being managed. Stamos Tolias has called for a fork because he is dissatisfied with the "narrow-minded, ZFS centric dictatorship and cathedral development model". Apparently he has funding and wants to go public with the fork in November, with project leadership coming from the University of Athens, financial support from an unspecified EU research fund, and EADS as a commercial partner. Tolias didn't respond to your author's questions for clarification.

As some of the criticism by Tolias is directed personally at D'Amore, the latter responded to the allegations on his blog:

Further, D'Amore explains that Illumos is not under the absolute control of Nexenta:

Currently, D'Amore is working with the Software Freedom Law Center to create a legal entity for Illumos, which would be a foundation completely independent of Nexenta. So while it may be true to some degree that Illumos development is not happening as openly and independently as it could be, Nexenta is doing its best to lay the groundwork for a real community-made Illumos.

Nexenta

Nexenta is building on the Illumos base, and the result is twofold: an open source GNU/Solaris operating system, Nexenta Core Platform (NCP) on the one hand (we looked at version 2 in 2009), and a commercial storage solution, NexentaStor, on the other hand. The latter is a ZFS-based storage solution that is typically purchased from systems integrators and VARs as a part of a total solution with certified hardware from vendors such as LSI, Quanta, Dell, Supermicro, and Intel.

The key message from Nexenta is its so-called "Open Storage" approach: NexentaStor users don't face vendor lock-in for their data, because they can easily migrate away to any other operating system supporting (the same or a newer version of) ZFS, such as Solaris, FreeBSD, and so on. In contrast, the big storage vendors use a proprietary on-disk format, and their solutions are tightly coupled to their own hardware. With Nexenta, users can choose for themselves which hardware to buy.

While the user space of previous Nexenta releases was based on Ubuntu, Nexenta 4 (which will be released at the end of 2011) will be based on Debian 6 ("Squeeze") instead. The Nexenta engineers made this decision because Debian is more portable and Ubuntu has too many patches that make use of specific Linux features which makes porting the user space to the Illumos kernel more difficult. While Nexenta 3 was still based on the OpenSolaris kernel, Nexenta 4 will be the first Illumos-based Nexenta product. So with the change from the OpenSolaris kernel to the Illumos kernel and from an Ubuntu user space to a Debian user space, Nexenta 4 will be a big switch.

However, there's still a lot of "fear, uncertainty, and doubt" spread about ZFS and patents. Last year, storage vendor NetApp sent a threatening letter to its competitor Coraid, and as a result Coraid decided to stop reselling their ZFS based storage appliance (based on NexentaStor). There was also a mutual lawsuit between Oracle and NetApp, which had to do with NetApp's contention that ZFS violated certain of its patents on the one hand, and with NetApp's rights to use certain technologies that originated in Sun on the other hand. Oracle and NetApp settled their dispute, but the details of their agreement were never made public.

Nexenta and any other companies using ZFS in their operating system surely don't have to be afraid of lawsuits by Oracle: the CDDL license that the ZFS code was made available under explicitly allows the use of any patents Oracle might have on the code. However, NetApp or any other party can still sue Nexenta for violations of their patents in the ZFS code, although since the beginning of this year Nexenta is part of the Open Invention Network (OIN). In his keynote speech at the Nexenta European User Conference, CEO Evan Powell made it clear that he considered this membership as a protection against potential patent infringement litigation by NetApp: "If NetApp wants to sue us, we can sue them with the core UNIX patents from Novell that are shared through the OIN."

StormOS

In January 2011, Nexenta Systems also hired Andrew Stormont, the developer of StormOS, an Xfce-based derivative of Nexenta Core Platform. It's a desktop operating system for people who like the ZFS/apt combo that Nexenta offers, but don't want to install the desktop packages themselves (because Nexenta is not aimed at desktop users). Stormont is now working as a software engineer at Nexenta Systems, primarily packaging software for the upcoming Nexenta Core Platform 4, but he recently also submitted patches for Illumos.

As part of his job at Nexenta Systems, Stormont gets one day a week to work on StormOS, and most of his work at Nexenta ends up benefiting StormOS too, but StormOS is still pretty much a one-man show since it started. The next release, StormOS Flash, will be based on NCP4 with some components from Debian 7 (Wheezy), such as newer X.Org and Xfce packages. One of the new features will be a graphical interface for Xfce to integrate ZFS snapshots in the file manager, like the Time Slider functionality in GNOME on OpenSolaris.

A more experimental feature that Stormont is considering is FatELF to support 32 and 64-bit architectures in a single binary. The Illumos kernel is already capable of automatically booting up in either 32 or 64-bit mode based on the hardware it is running on, so it makes sense for the binaries sitting on top of the kernel to also automatically run in the correct mode.

In an email interview with your author, Stormont explains which benefits FatELF could offer to StormOS:

Other companies

Nexenta is not the only one backing Illumos. Joyent is another big contributor, and it hired some core Solaris people like Bryan Cantrill and Brendan Gregg. Joyent has published a Git repository with their patches to the Illumos tree. In his announcement of the tree, John Sonnenschein wrote:

Some of these patches are fairly Joyent-specific, but others are useful more generally. Most of the patches are enhancements for zones functionality (operating-system level virtualization), as well as DTrace (the Solaris dynamic tracing framework) enhancements for better analytics in the cloud. Many of these changes will probably find their way into Illumos in the near future, although there's the practical issue that Illumos is using Mercurial while Joyent uses Git.

Another code contributor is the database company Delphix, which hired Adam Leventhal who worked on RAID-Z (a redundancy scheme similar to RAID 5, using ZFS) and is also a co-inventor of DTrace. Delphix seems to prefer to keep its involvement more quiet, as they haven't announced anything with respect to their contributions, but a cursory look at their commits in the Illumos repository shows that they are definitely involved. So while Nexenta Systems is still the main contributor, there are others too.

Google Summer of Code

After less than a year Illumos is already active as a mentoring organization for Google Summer of Code (GSoC) 2011. Two Illumos GSoC projects have been accepted: Harshit Jain will update Illumos to be able to boot with GRUB 2 and to support booting from a disk with a GUID partition table (GPT), which will be mentored by D'Amore, and Shashank Karkare will rewrite some of the Perl system components (like intrd and kstat) in C, under the supervision of Joyent's John Sonnenschein. The goal of the latter project is to remove Perl as a runtime dependency for Illumos.

Furthermore, Nexenta also funded three students that didn't get into Google Summer of Code. For instance, Andrey Sokolov will work on supporting Linux binaries on Illumos without zones (branded zones are a way to run Linux binaries on Solaris in a virtual container), Bayard Bell will work on an IKEv2 daemon to replace IKE for Illumos, and Viktor Ikonnikov will write a GUI frontend for the service configuration tool svccfg. For future editions of GSoC or Illumos Students, Illumos has an extensive list of project ideas.

ZFS working group

Another interesting development under the Illumos umbrella, although it is somewhat under the radar, is the ZFS working group. This group was founded a few months ago by Adam Leventhal and Matthew Ahrens (who has worked on ZFS at Sun/Oracle) of Delphix and Garrett D'Amore. Ahrens is the leader of this group. The goal is to facilitate collaboration of the community around ZFS, by coordinating the work and exchanging ideas. In particular, the founders had concerns about fracturing ZFS into sub-communities of the various operating systems that are supporting the file system.

The ZFS working group contains developers from Illumos/OpenIndiana, Mac OS X (most likely Don Brady developing ZFS support with his company Ten's Complement), FreeBSD (probably Pawel Dawidek who ported ZFS), Linux, and Solaris. According to D'Amore, Oracle has also a seat on the group. The workgroup's activities are not done in the open, and some of the members don't want their membership to be known, but it's the intention to publish results of the discussions regularly.

The first result from the ZFS working group is the feature flags proposal. In the announcement, Matthew Ahrens wrote:

As Ahrens said, this feature flags functionality is needed because ZFS development outside Oracle is now happening in a much more distributed way, compared to when only Oracle/Sun developed ZFS. After ZFS v28, Oracle has stopped publishing its source code, and the developers of Illumos, FreeBSD, and other ZFS-supporting operating systems don't want to wait until Oracle's next code drop after the release of Solaris 11, so development goes on. Thanks to the feature flags, these developers outside Oracle could add new compression algorithms or new RAID variants without breaking interoperability: in most cases the operating system should still be able to access a ZFS pool or file system in read-only mode if it has an unknown feature. Feature flags will be implemented this summer and integrated into Illumos.

Conclusion

Illumos is clearly going forward, although the way it is doing this is frustrating some people. The Illumos developers have more of a cathedral-like development model, while some of the more vocal members of the Illumos community would like more of a completely open bazaar-like development model like Linux is using. The call for a fork and some criticism about the closed nature of the ZFS working group are consequences of these sentiments.

However, the critics forget that the OpenSolaris development model was much more closed. Although Sun released the bulk of the Solaris system code and called OpenSolaris "open source", there never was a real OpenSolaris community: it always depended too much on Sun and later Oracle, and both companies didn't make an effort to create an independent community around their "open source" project. Oracle's current development model for Solaris 11 is completely behind closed doors.

So while Illumos is not yet the completely open Solaris code base and does not yet have the vibrant community that many OpenSolaris people have dreamed of, things are going in the right direction. It's not a Nexenta-only project anymore, there will be an independent Illumos foundation, and the project has attracted support of some big hardware vendors. Now it's just waiting on the Illumos-based Nexenta release and the first stable Illumos-based OpenIndiana release to get the ball rolling and regain the interest of all the people who left the OpenSolaris community during Oracle's radio silence.