Illumos: new hope for the OpenSolaris community?

On August 3, storage vendor Nexenta Systems hosted a conference call [MP3] announcing Illumos, a project to free the OpenSolaris distribution. Garrett D'Amore, a former Sun and Oracle Solaris developer and now senior director of engineering at Nexenta, disclosed the goal of the project: Illumos will replace all proprietary code that is still in OpenSolaris by open source code. More information can be found in the announcement slides [PDF].

Although its name suggests something else, OpenSolaris has never been completely open source. Some critical components are only available as binaries because of copyright issues. The most critical part is the internationalization (i18n)) framework of libc, but also the NFS lock manager, portions of the cryptographic framework, and numerous critical drivers are proprietary. This presents a lot of challenges to downstream OpenSolaris distributions, such as Nexenta and Belenix. We only have to look at Darwin, Apple's open source part of Mac OS X, to see what the effect can be of some critical components that are closed: Darwin never really took off as an independent operating system.

So the first short-term goal of Illumos is open sourcing all of OpenSolaris, or more correctly: all of OS/Net (ON), which is the OpenSolaris kernel, C library, drivers, and some basic commands. Illumos will track OpenSolaris ON closely, but will replace all closed bits by open source alternatives, while still staying 100% ABI compatible. Right now, Illumos libc is completely open: Garrett developed libc_i18n by borrowing code from FreeBSD, picking up a little from NetBSD, and finishing it with his own code.

At the moment, Garrett's code is the only one that has been integrated in Illumos, but there are a number of other projects that are near to integration. For example, Anil Gulecha developed GRUB and boot graphics, Steve Stallion and David Gwynne have been working on some open source replacements for drivers, Roland Mainz, Olga Kryzhanovska and Rich Lowe have replaced some user-space utilities, and Jörg Schilling has been working on archivers.

But even when all these projects get integrated, there is still the issue that only Sun's proprietary compiler Sun Studio can currently build a bootable OpenSolaris kernel. The intention is to enable a fully open toolset, probably using GCC. So a fully open source self-hosting OpenSolaris derivative is not yet possible, but it should become a reality by the end of 2010. A preliminary version of Illumos using the closed bits and compiled with Sun Studio already boots.

A fork or not a fork?

So Illumos is not really a fork, it's more of a downstream project that is happy to contribute to upstream ON. Oracle employee Joerg Moellenkamp explains the situation like this:

However, the ability to fork is important for OpenSolaris, as Bryan Cantrill explains:

So although Illumos is not aimed at creating an OpenSolaris fork, according to Bryan the power to fork is essential for the vitality of the OpenSolaris community. Jörg Schilling looks at this in an historical context:

A lack of communication

Jörg is right in his description. When Oracle acquired Sun, they talked a lot about MySQL and Java, but said virtually nothing about (Open)Solaris. In February 2010, Peter Tribble complained that Oracle hadn't even mentioned the distribution in a five-hour webcast after the acquisition of Sun. And at the times they did mention OpenSolaris, it was in vague terms. For example, at the OpenSolaris Annual Meeting on IRC in February 2010, Oracle's Dan Roberts said:

Almost half a year later, there hasn't been an OpenSolaris 2010.03 (which was due in March 2010); there still isn't a newer release than OpenSolaris 2009.06. Oracle didn't respond to an open letter by Ben Rockwood, a respected OpenSolaris community member. In April 2010, Oracle stopped shipping free OpenSolaris CDs. In general, Oracle's silence continued, and a lot of community members were getting fed up with the insecurity. This led the OpenSolaris Governing Board to the decision to issue an ultimatum to Oracle: if Oracle doesn't appoint a liaison who has the authority to talk about the future of OpenSolaris by August 16, the OGB will disband itself. OGB chair John Plocher explained the issues:

Oracle didn't respond to the ultimatum, and the long radio silence about the future of OpenSolaris has already taken some victims. PulsarOS, a file server operating system, has switched from OpenSolaris to Linux. The high-integrity operating system AuroraUX was first an OpenSolaris distribution but has quietly shifted to become a DragonFlyBSD derivative. And the other OpenSolaris derivatives are stagnant or only reach a small niche, with the exception of Nexenta.

The community

The Illumos project (which Garrett started about two months ago, even before the OGB's ultimatum) wants to tackle these issues, not only by writing code but also by building an independent community. In this first phase, Garrett will serve as the benevolent dictator of Illumos, but this dictatorship is a temporary measure. He will appoint some initial members for an Administrative Council, which will work on non-technical matters such as defining a code of conduct. Garrett is not only head of this Administrative Council, but also the technical lead: head of the Developer Council. This will be made up of developers with commit rights, and it is a meritocracy: anyone can join in and will be judged by their code. Development will be consensus driven, but the technical lead is the final arbiter if consensus is not possible. A process to replace the technical lead will be discussed in the future.

Nexenta is currently the main sponsor of Illumos (Garrett's efforts have been funded by Nexenta), but the goal is to have a code base that is free from the control of any company. Corporate entities are welcome, though, to contribute. About one week before the launch of Illumos, Garrett invited Oracle to participate in Illumos. He contacted Bonnie Corwin, but she told him that she simply did not know who was responsible for decisions relating to OpenSolaris. She also said that she would endeavor to find out, but Garrett hasn't heard anything back. Right now Illumos has about a dozen developers, many of them well-known people in the OpenSolaris community, working for Nexenta Systems, hosting company Joyent, Greenviolet, Belenix, Schillix, and Everycity.

Some have argued that Illumos is purely driven by emotional feelings about OpenSolaris, by people who can't stand thinking about the death of the operating system, but this is far from true. For example, Nexenta and Joyent really depend on OpenSolaris for their business. They obviously don't like that they ultimately depend on what Oracle decides to do with OpenSolaris. So from a business point of view, it makes perfect sense that they want to free OpenSolaris and create an active developer community, independent from Oracle's good will. If they succeed, these companies can be much less concerned about their business continuity.

Of course, they have much less development manpower than Oracle (the vast majority of OpenSolaris development has always come from paid Sun/Oracle engineers), but they have some really good ex-Sun developers, like Garrett D'Amore (Nexenta's senior director of engineering), Richard Elling (Nexenta's senior director of solutions engineering), Bryan Cantrill (Joyent's vice president of engineering), and so on. Other notable backers of Illumos are Dennis Clarke, who is the founder of the Blastwave project, and Jörg Schilling, who built the first OpenSolaris distribution (Schillix) even before Sun did. By uniting manpower in an open source community, they can tackle their shared concerns together.

From source to distribution

This isn't the first time that an external code repository was created by the OpenSolaris community. For example, the Genunix web site that was created in 2006 at the request of the OpenSolaris Community Advisory Board (the precursor of the OpenSolaris Governing Board) maintained an independent SVN repository for some time, and they also have some Mercurial repositories, for instance for OpenSolaris OS/Net. The latter is an active clone that tracks Oracle's repository.

However, the repository of Illumos is the first one that will be backed by substantial development by non-Oracle people. This allows OpenSolaris distributions that adopt Illumos as their base to depend less on Oracle. Illumos is also insurance against Oracle pulling the plug on OpenSolaris: if Oracle stops supplying the source code to OpenSolaris, distributions can still build upon Illumos. Of course, the project would then become an OpenSolaris fork by necessity, and it's uncertain whether it has the critical mass to continue without Oracle.

For now, it's just a project of open sourcing the whole OS/Net code, but in time Illumos also wants to become a repository for experimental innovations and patches that are not accepted by Oracle. For instance, integration of other architectures like S390, PowerPC, or ARM could be possible in Illumos, because OpenSolaris has no interest in other architectures than x86/x86_64 and SPARC.

All source code that will be integrated into Illumos will first be restricted to BSD/MIT licenses and CDDL (Common Development and Distribution License) with a signed SCA (Sun Contributor Agreement), which gives Sun and the contributor joint copyrights on the code. Other possibilities for licenses are still under investigation, and a critical requirement is of course that the license must be acceptable for merging back into the upstream OpenSolaris code. Illumos is looking specifically at the Apache 2.0 license, because, in Garrett's words, "it offers the freedoms of BSD/MIT without the baggage of copyleft, while still providing a protection against submarine patenting." Moreover, Garrett pinpoints one property that he prefers in any license used in Illumos:

Illumos is focused only on ON, so it's not an OpenSolaris distribution. However, the project may expand in the future and host some affiliated projects, such as Illumos distributions. For example, there could be a binary ISO with a minimal Illumos system to bootstrap a real Illumos distribution. Although Garrett is a Nexenta employee, Illumos is not focused on Nexenta: for example it imposes no specific package manager, although it has a preference for IPS (Image Packaging System), the OpenSolaris package management system. IPS manifests are the canonical data format in Illumos, but there's a tool to generate .deb files from IPS, which Nexenta uses for its OpenSolaris/Ubuntu hybrid. This system can be extended to generate RPM, SVR4, or any other type of packages from the IPS packages. This is a choice that each derivative distribution can make.

Too little, too late?

In the Linux world, Illumos wasn't received with much enthusiasm. For example, according to Joe "Zonker" Brockmeier OpenSolaris should really die, and he calls Illumos "a misguided attempt to keep the Solaris legacy OS alive for another generation". While Zonker rightly questions Sun's strategy with respect to OpenSolaris, your author finds this death wish to be a bridge too far. But no one can deny that Illumos will face a lot of issues.

A project to liberate OpenSolaris probably should have happened five years ago when Sun released the bulk of the Solaris system code. If Illumos had started then, right from the start of OpenSolaris, the closed bits would have been replaced by now, with less hurdles for OpenSolaris derivatives and a more open community as its consequences. So one could ask the question: is Illumos coming too late? The OpenSolaris community has already lost a lot of users and derivatives because of the uncertain future of the distribution, and only time will tell whether Illumos can still change this downward trend.