Bringing OSTree to real-world desktops

At GUADEC 2016 in Karlsruhe, Germany, Owen Taylor presented a talk entitled "Reworking the desktop distribution" that outlined his vision for how next-generation packaging concepts like Flatpak applications and OSTree system images can be harmonized with the more flexible model long used by Linux distributions. As it turns out, while these newer packaging concepts work well for some basic systems, the story gets more complicated if the user wants to perform certain tasks on their machine—such as develop software.

The concept

He began with the rationale for moving toward OSTree and Flatpak. Traditionally, he said, a distribution is made of "lots and lots of packages." The downside is that this means every user's system is "custom" in a sense. Thus, when somebody says that they "tested Fedora 24," it means that they tested something quite different than you might on your own Fedora 24 machine. It also means distributions and application projects are unable to distinguish between corrupted and uncorrupted systems, modified and standard systems, and other divergent cases.

Users who install distribution updates repeatedly, Taylor added, will eventually end up with some configuration that varies from what would be found on a fresh install—whether it is some changed default setting, some different service running, or "some input method that we tried and dropped a long time ago." Consequently, no one is comfortable releasing packages and saying to users "here you go; it'll have no problems, so you don't need to call me," Taylor said.

Improving on this tangled status quo is one of the goals of the OSTree project. The basic idea, he said, is that the OS image is immutably fixed and all of the applications are self-contained (be they in Flatpak format or in some other flavor of bundle). Several end-user OSes have already taken this static-OS approach, he said, namely Android, ChromeOS, One Laptop Per Child, and now Endless OS. But bringing this model to a "classic Linux user" has remained a challenge.

The first implementation

There has, however, been recent progress via the Fedora Workstation project. Fedora ships a relatively unaltered version of upstream GNOME, Taylor said, with just a few modifications (such as replacing GNOME Web with Firefox). And Atomic Host has already been shipping immutable OS images, "although they do different things in many places" compared to the immutable desktop design. So the Fedora Atomic Workstation effort has been working to combine the Atomic Host approach with Flatpak's self-contained application bundles—and Taylor has been exploring how to add support for software development.

OSTree is "like Git but for binary file trees," Taylor said, and is already used in the GNOME Continuous project. OSTree can be used to build the immutable OS images, he noted, "but how do we make this binary OS image? Out of packages." After a chuckle from the audience, he explained. Traditional package formats (like RPM and Debian) leave a lot undefined, but that does not make them useless. A distribution's kernel package, for example, holds a lot of configuration information. Furthermore, traditional package formats make it easier to distribute security fixes.

Thus, the rpm-ostree tool attempts to build on existing packaging knowledge, and construct OSTree images out of existing packages. It supports a layering facility, so that sets of packages can be overlayed on a base image. That will be the approach used to deploy updates when rebuilding the entire system is not feasible. Though, Taylor confided, "especially here at the beginning, we'll probably get some things wrong." Getting the granularity of the layers right is tricky; "the more that you layer, the more you will experience 'package pain'," he said.

Currently, adding a new layer built with rpm-ostree onto a running system requires a reboot, although that may change. There is also a new "unlock" feature for administrators. It adds a writable layer on top of the root system so that administrators can make alterations; the layer disappears on reboot.

As for Flatpaks, he said, very few of the thousand or so applications packaged for Fedora Workstation have also been packaged as Flatpaks, so the plan is to develop a tool that will automatically create Flatpaks from RPM packages. Things appear to be on track for Fedora Atomic Workstation to be released with Fedora 25; the OSTree builds are currently unofficial but will move into the official Fedora infrastructure and GNOME Software recently added support for managing Flatpak packages in addition to other formats.

About developers

Fedora Workstation is designed to be a distribution for software developers, Taylor said, but their needs differ from those of other desktop users—in ways that are challenging for the OSTree model. To develop software on Linux today, one typically has to install hundreds of packages (which vary by project); if you try to implement all of those packages as OSTree layers, Taylor said, then you'd might as well not use OSTree at all. So some other solution is called for.

The basic requirements, he explained, are, first, that the development environment should not be the same as the workstation environment. "You shouldn't have to break your GTK+ to work on it." Second, the development environment should be reproducible. Third, the environment should function on a variety of different host OSes and for both production and test systems. In addition, he noted, developers should be able to install multiple self-contained development environments in parallel.

The solution that Taylor has been exploring starts with considering different "flavors" of software development separately. For starters, he looked at two specific scenarios: web development and native GNOME application development. The development environments used for those tasks are quite different.

Web developers tend to use a lot of terminal windows and command-line tools for the frameworks of interest (Node.js, Ruby on Rails, etc.), and perhaps a code editor like Atom. "We want to accommodate this, but we want to do it in a way that's better than saying 'here's a terminal; have fun'," he explained.

He then showed a demo of a tool he created called PurpleEgg (which is purely a pre-production code name, he said). It enables web developers to set up and use development environments on an OSTree-based system by creating a separate container for each development project. Using the PurpleEgg tool, he initialized a new Django project and described the results. The tool created an empty Docker container, installed Git, Python, and other system packages, installed and configured Django inside of a Python virtual environment, and installed a project template.

The container was created with one user (created to match his user on the host system) and included a copy of his existing Git configuration. A desktop launcher was also created in the host OS that will launch the container and open up a terminal window inside it. There are still other features yet to be added, he said, such as starting a web server in the container for testing and doing more extensive Git configuration.

But the host OS remains untouched, which is the ultimate goal. Still, the web-development case was the simplest. GNOME application developers have a less predictable setup: some use JHBuild, some might use an IDE like Builder. And, whatever the setup of today looks like, it may change in the future when Flatpak usage increases. Plus, there are numerous other development fields that might expect a different solution, he said, such as scientific development, kernel hacking, mobile development, and game development. So further work is clearly called for.

The session ended with a brief question-and-answer period. Bastien Nocera pointed out that "system development" (meaning work on daemons, systemd, low-level configuration, etc.) may be yet another category of software development that needs to be treated separately. "Sometimes we would need to install an overlay and restart everything that lives," he said. "How do we handle that?" Taylor replied that OSTree also supports a "hotfix" mode that lets users make persistent changes to the system; it could enable making changes that persist after reboot.

Christian Hergert expressed concern about OSTree's extensive use of OverlayFS for layering, noting that the filesystem is not POSIX-complaint and, in fact, is so different that fixing it would amount to rewriting OverlayFS from scratch. Taylor replied that something else may be better, but also said that the OSTree layers are ultimately used for hacking, while updates to the full image are meant to be used for long-term updates, so perhaps weaker stability guarantees are acceptable.

Taylor emphasized that the PurpleEgg tool is merely a proof of concept, and there are many issues remaining to be worked out. But it could provide the template necessary to bridge the gap between the purely-static OS environment of OSTree and the per-app isolation offered by Flatpak and other application-sandboxing formats.

[The author would like to thank the GNOME Foundation for travel assistance to attend GUADEC 2016.]