Mageia bootstraps its package update policy

Mageia, the community-driven fork of Mandriva Linux, has made its initial release, just eight months after the project's inception. It is a major accomplishment, but the development team has already turned its attention to the next pressing challenge: how to set up and manage package updates, including how to handle security fixes, how to integrate backports and missing packages, and how to bootstrap a QA process.

The release of Mageia 1 marks a turning point for the distribution, because end users are now encouraged to install and use it on a daily basis — and up until now, most of those users were still running Mandriva, which continued to receive regular package updates.

A user switching to Mageia is likely to encounter several distinct package update scenarios: security updates to close specific vulnerabilities, incremental bugfix updates, and new packages that for one reason or another were not ready for inclusion with the initial release of Mageia 1. Each scenario has its peculiarities, although for simplicity's sake, it is in the project's interest to provide as uniform a testing and update process as possible.

Mageia's effort to hash out an update process is not simply a start-up problem. Many on the team were unhappy with Mandriva's process over the years. As Michael Scherer explained:

The Mageia team, therefore, has set goals for its own process to be more streamlined, more scalable, and to take security vulnerabilities more seriously. Many of the core team members are former Mandriva contributors (and quite a few were Mandriva employees), so the process includes their opportunity to learn directly from Mandriva's mistakes.

Thus far, Mageia has not pushed out any updated packages to the updates repository, although a June 19 message indicates that the first package updates have at least reached testing. Meanwhile, the outstanding list of "missing package" backport requests, which includes security updates made by Mandriva, stands at over a dozen. So, since the release of Mageia 1, security updates have languished along with other updates, though it appears that will be changing soon.

Updates (security and otherwise)

The discussion over how to structure the update process began on the mageia-dev mailing list in early June. As is the case for Mandriva, Mageia plans to run separate RPM repositories for main releases and package updates, plus testing repositories for use by developers and packagers. The draft update policy draws the distinction between the "updates" repository (which is to be used to provide bug-fix or security-fix updates to existing packages), and the "backports" repository (which is to be used to provide new releases of packages).

Nevertheless, there are a handful of exceptions. First, new versions of packages that require an update in order to work with a remote online service are allowed in the updates repository (the examples include networked games and virus scanners). Second, new versions of applications are permitted when the old version is no longer supported by the upstream project itself (a rule that primarily encompasses applications with a short lifecycle like the Chromium and Firefox web browsers).

The update release process follows the same broad plan as used by many other distributions: anyone can file a bug against a package; once the bug is triaged and assigned, the package maintainer can apply the relevant patches and rebuild the package. The maintainer is then responsible for submitting the new build to the updates_testing repository and re-assigning the bug to the QA team. The QA team tests the package to ensure that it fixes the bug, that it both installs and successfully updates the previous version of the package, and (in the case of security fixes) that it closes the security vulnerability.

A few obstacles remain to implementing this process. The first is that, as a new distribution, very few packages have official maintainers. Mageia may assign bugs to package committers as an initial step, but the understanding is that maintainership of a package is a voluntary task. The second obstacle is setting up QA communication channels, to ensure that QA team members do not accidentally step on each other's toes. Whether that takes the form of a read-only mailing list or a web announcement mechanism has yet to be determined.

The third obstacle is that while the QA team is significant in size (and growing), the security team is still much smaller, and less formalized. Conventional wisdom states that the security team has the time-consuming task of monitoring security vulnerability lists and databases, as well as the difficult job of assessing that a rebuilt package fixes the security vulnerability, as opposed to simply verifying a specific "normal" bug fix.

That is not to diminish the amount of work that goes into the QA process. Mageia, like Mandriva before it, prides itself on providing well-tested packages that both integrate with free software applications and play well with third-party (potentially proprietary) offerings. It supports both the GNOME and KDE desktop environments when many distributions choose either one or the other, and is available for both 32-bit and 64-bit architectures.

Backports and support

That desire to offer well-tested packages comes to the forefront in the project's discussion over how to handle "missing packages" and backports. The root of the issue is that Mageia 1 lacks several packages that were available in previous Mandriva releases. Due to personnel constraints, it simply isn't possible to package and test every application and library — so the project did its best with the Mageia 1 release, and is attempting to craft a policy for providing the missing pieces to users between now and the release of Mageia 2.

A case could be made that missing packages are essentially no different than other bugs, and if a package can be tested and added to the repository, it should go in the updates repository. The alternate viewpoint is that updates is by definition reserved for fixes to existing packages, and that the backports repository exists for the specific purpose of allowing users early access to packages slated for the next release — early access at their own risk.

But Mageia 1 is a special circumstance; many of the missing packages have long existed for Mandriva users, and in some cases are even stable and "expected" in a standard distribution (examples from the mailing list include passwd-gen and dd_rescue). Forcing users in need of those packages to enable an unstable repository like backports (which could inadvertently update other packages to new, un-QA-validated versions) puts them at an unacceptable risk. Yet treating backports as a fully-supported repository just to enable the addition of missing packages has its own risks, namely far more work for the QA and security teams.

Scherer argued that if the root cause of the trouble was that backports has historically been synonymous with "bug-filled," then fixing the backports process was the solution. Others suggested that the real pain-point was the notion that backports is "unsupported" — a term casually tossed around in many distributions, but rarely if ever defined. After all, in most cases, a user reporting a problem with a backports package still receives help fixing it from the community. Complicating the issue further is that Mandriva's package management tool RPMDrake allowed users to access specific packages from Mandriva's backports repository, but without enabling the repository as a whole — a behavior that carries its own set of QA and security trade-offs.

Samuel Verschelde suggested a plan of attack for bugs filed against backports, that differentiates them from bugs in updates by establishing a "maintainer first, then upstream" process that relieves the regular QA team from the burden of integrating the fixes. Christiaan Welvaart recommended a way to tie bugs in the backports repository more closely in with the existing QA process by providing better step-by-step guidance to bug reporters.

Eventually, the team came to a consensus around a plan that makes a special exception for the transitional period between Mageia 1 and the release of Mageia 2. During the transition period, missing packages can be added to the updates repository provided that they follow the same QA process as every other update, and that there is a bug report filed formally requesting the package. After Mageia 2 is released, that process will be closed, and missing packages should go through the backports repository, or simply wait for the next release cycle.

Of course, Mageia is also in the process of working out the details of its release cycle, so the proposed backports plan has its small share of uncertainty. At the moment, the distribution has a bleeding-edge distribution named "Cauldron" that operates as a rolling release, but stable end-user releases are tentatively planned to follow the six-month cycle established by Mandriva.

Bubble bubble, hopefully with minimal toil and trouble

Bootstrapping a Linux distribution is always a tremendous amount of work, even for teams like Mageia's, with its healthy number of ex-Mandriva engineers and community contributors. But as many old Linux veterans will tell you, getting the first release out the door is not nearly as taxing as setting up a smooth development, testing, and release process that is sustainable over a multi-year period. Even established distributions struggle with QA and package testing issues from time to time.

The Mageia backports discussion touched on an intriguing issue: what precisely it means when an open source project declares something "supported" or "unsupported." It appears as if that thread has fallen by the wayside, and the team has moved on to the more practical issues of fleshing out the backports process and handling the corner-case of assisting Mandriva users as they transition to the first ever Mageia release. But perhaps Mageia's unusual position as a brand-new distribution staffed by team members that moved over collectively from Mandriva gives it a unique chance to address the supported/unsupported question. It has already allowed them to produce a full-featured distribution in well under a year's time.