A Modularity rethink for Fedora

We have covered the Fedora Modularity initiative a time or two over the years but, just as the modular "product" started rolling out, Fedora went back to the drawing board. There were a number of fundamental problems with Modularity as it was to be delivered in the Fedora 27 server edition, so a classic version of the distribution was released instead. But Modularity is far from dead; there is a new plan afoot to deliver it for Fedora 28, which is due in May.

The problem that Modularity seeks to solve is that different users of the distribution have differing needs for stability versus tracking the bleeding edge. The pain is most often felt in the fast-moving web development world, where frameworks and applications move far more quickly than Fedora as a whole can—even if it could, moving that quickly would be problematic for other types of users. So Modularity was meant to be a way for Fedora users to pick and choose which "modules" (a cohesive set of packages supporting a particular version of, say, Node.js, Django, a web server, or a database management system) are included in their tailored instance of Fedora. The Tumbleweed snapshots feature of the openSUSE rolling distribution is targeted at solving much the same problem.

Modularity would also facilitate installing multiple different versions of modules so that different applications could each use the versions of the web framework, database, and web server that the application supports. It is, in some ways, an attempt to give users the best of both worlds: the stability of a Fedora release with the availability of modules of older and newer packages, some of which would be supported beyond the typical 13-month lifecycle of a Fedora release. The trick is in how to get there.

The main problem that arose with the modular server edition was, in effect, a lack of modules. It turned out to be far more painful for packagers to build modules than expected, so few did. That left it up to the Modularity team to build the modules that would ship with Fedora 27. As Stephen Gallagher, who has been one of the driving forces behind the initiative, put it:

In addition, the first mechanism chosen to build modules relied on a "bootstrap" module that, among other things, made it difficult for existing Fedora users to upgrade into a modular server release. Third-party software was also problematic in this first approach, since it would need to be built into a module—something that was difficult for anyone but the Modularity team to accomplish.

New approach

The original plan was to define a build environment (buildroot) specifically for the modular server, but that seems to have caused more problems than it solved. The new plan is to use the "everything" repository for the Fedora 28 release as the underlying "platform module", which makes things more straightforward. Importantly, it makes things easier for module packagers:

That change will also make it easy for users to simply upgrade into a modular release. Modules and traditional packages can coexist on a system as well. So far, the plan has been for only the server edition to support modules, but with an easier upgrade path and the ability to support both packages and modules, the idea could be adopted by other editions (e.g. workstation) of Fedora.

In fact, the module-creation process will become so straightforward that automated tools will be provided to create the configuration to build single-source-package modules. "Even for more complex multi-package modules, the automatically-created module definitions provide an easy and obvious starting point." This will make it easy to support multiple versions, as Gallagher notes:

For future Fedora releases, there will be two sets of repositories to support both the traditional RPM-based distribution and the modular approach. Those who have no interest in modules can disable the modular repositories and continue on as they always have. For others who are looking for the modular approach, though, it will be as easy as simply using the DNF package manager with some new target-specification syntax to pick up modules.

It is not surprising that a change of this nature might run into some turbulence as it gets integrated into a well-established distribution packaging ecosystem like Fedora's. It is a pretty fundamental change to the distribution, so problems are to be expected during the upheaval. As Fedora project leader Matthew Miller put it in the announcement of the rethinking: "Sometimes experiments produce negative results. That's okay — the project learns even when trying a path that doesn't work out, and it iterates to something better." For his part, Gallagher expressed optimism that the Modularity project is now on better track:

As Miller pointed out, progress is not made without some missteps along the way. It remains to be seen if Modularity represents progress, but the problem it addresses is certainly real—the approach Fedora is taking seemingly has the potential to solve it. One the major benefits of development in the open is that these kinds of missteps are not hidden behind delayed releases, vaporware, press releases, and other obfuscation techniques as they often are in the proprietary software world. In the free-software world, we get to see the sausage being made (and remade), so projects can learn from each other. With luck, that makes for better software throughout our ecosystem.