Mageia, the community-driven fork of Mandriva Linux, has made its initial release, just eight months after the project's inception. It is a major accomplishment, but the development team has already turned its attention to the next pressing challenge: how to set up and manage package updates, including how to handle security fixes, how to integrate backports and missing packages, and how to bootstrap a QA process.
The release of Mageia 1 marks a turning point for the distribution, because end users are now encouraged to install and use it on a daily basis — and up until now, most of those users were still running Mandriva, which continued to receive regular package updates.
A user switching to Mageia is likely to encounter several distinct package update scenarios: security updates to close specific vulnerabilities, incremental bugfix updates, and new packages that for one reason or another were not ready for inclusion with the initial release of Mageia 1. Each scenario has its peculiarities, although for simplicity's sake, it is in the project's interest to provide as uniform a testing and update process as possible.
Mageia's effort to hash out an update process is not simply a start-up
problem. Many on the team were unhappy with Mandriva's process over the
years. As Michael Scherer explained:
Security and stable updates were quite disconnected from
maintainers, and so it didn't scale well. It didn't scale because people
didn't know security procedure (it was not part of the expected curriculum
of a packager, and often was done without them implied), it didn't scale
because security was only for a restricted set of [salaried employees]
taking care of everything on separate systems.
The Mageia team, therefore, has set goals for its own process to be more
streamlined, more scalable, and to take security vulnerabilities more
seriously. Many of the core team members are former Mandriva contributors
(and quite a few were Mandriva employees), so the process includes their
opportunity to learn directly from Mandriva's mistakes.
Thus far, Mageia has not pushed out any updated packages to the updates
repository, although a June 19 message
indicates that the first package updates have at least reached testing.
Meanwhile, the outstanding list of "missing package" backport
requests, which includes security updates made by Mandriva, stands at
over a dozen. So, since the release of Mageia 1, security updates have
languished along with other updates, though it
appears that will be changing soon.
Updates (security and otherwise)
The discussion over how to structure the update process began on the
mageia-dev mailing list in early
June. As is the case for Mandriva, Mageia plans to run separate RPM
repositories for main releases and package updates, plus testing
repositories for use by developers and packagers. The draft update policy
draws the distinction between the "updates" repository (which is to be used
to provide bug-fix or security-fix updates to existing packages), and the
"backports" repository (which is to be used to provide new releases of
Nevertheless, there are a handful of exceptions. First, new versions of
packages that require an update in order to work with a remote online
service are allowed in the updates repository (the examples
include networked games and virus scanners). Second, new versions of
applications are permitted when the old version is no longer supported by
the upstream project itself (a rule that primarily encompasses applications
with a short lifecycle like the Chromium and Firefox web browsers).
The update release process follows the same broad plan as used by many
other distributions: anyone can file a bug against a package; once the bug
is triaged and assigned, the package maintainer can apply the relevant
patches and rebuild the package. The maintainer is then responsible for
submitting the new build to the updates_testing repository and re-assigning
the bug to the QA
team. The QA team tests the package
to ensure that it fixes the bug, that it both installs and successfully
updates the previous version of the package, and (in the case of security
fixes) that it closes the security vulnerability.
A few obstacles remain to implementing this process. The first is that, as a new distribution, very few packages have official maintainers. Mageia may assign bugs to package committers as an initial step, but the understanding is that maintainership of a package is a voluntary task. The second obstacle is setting up QA communication channels, to ensure that QA team members do not accidentally step on each other's toes. Whether that takes the form of a read-only mailing list or a web announcement mechanism has yet to be determined.
The third obstacle is that while the QA team is significant in size (and
growing), the security team is
still much smaller, and less formalized. Conventional wisdom states that
the security team has the time-consuming task of monitoring security
vulnerability lists and databases, as well as the difficult job of
assessing that a rebuilt package fixes the security vulnerability, as
opposed to simply verifying a specific "normal" bug fix.
That is not to diminish the amount of work that goes into the QA process. Mageia, like Mandriva before it, prides itself on providing well-tested packages that both integrate with free software applications and play well with third-party (potentially proprietary) offerings. It supports both the GNOME and KDE desktop environments when many distributions choose either one or the other, and is available for both 32-bit and 64-bit architectures.
Backports and support
That desire to offer well-tested packages comes to the forefront in the project's discussion over how to handle "missing packages" and backports. The root of the issue is that Mageia 1 lacks several packages that were available in previous Mandriva releases. Due to personnel constraints, it simply isn't possible to package and test every application and library — so the project did its best with the Mageia 1 release, and is attempting to craft a policy for providing the missing pieces to users between now and the release of Mageia 2.
A case could be made that missing packages are essentially no different than other bugs, and if a package can be tested and added to the repository, it should go in the updates repository. The alternate viewpoint is that updates is by definition reserved for fixes to existing packages, and that the backports repository exists for the specific purpose of allowing users early access to packages slated for the next release — early access at their own risk.
But Mageia 1 is a special circumstance; many of the missing
packages have long existed for Mandriva users, and in some cases are even
stable and "expected" in a standard distribution (examples from the mailing
passwd-gen and dd_rescue). Forcing users in need of those packages to
enable an unstable repository like backports (which could inadvertently
update other packages to new, un-QA-validated versions) puts them at an
unacceptable risk. Yet treating backports as a fully-supported repository
just to enable the addition of missing packages has its own risks, namely
far more work for the QA and security teams.
that if the root cause of the trouble was that backports has historically
been synonymous with "bug-filled," then fixing the backports process was
the solution. Others suggested that the real pain-point was the notion
that backports is "unsupported" — a term casually tossed around in
many distributions, but rarely if ever defined. After all, in most cases,
a user reporting a problem with a backports package still receives help
fixing it from the community. Complicating the issue further is that
Mandriva's package management tool RPMDrake allowed users to access
specific packages from Mandriva's backports repository, but without
enabling the repository as a whole — a behavior that carries its own
set of QA and security trade-offs.
Samuel Verschelde suggested
a plan of attack for bugs filed against backports, that differentiates them
from bugs in updates
by establishing a "maintainer first, then upstream" process that
relieves the regular QA team from the burden of integrating the fixes.
Christiaan Welvaart recommended
a way to tie bugs in the backports repository more closely in with the
existing QA process by providing better step-by-step guidance to bug
Eventually, the team came to a consensus around a
plan that makes a special exception for the transitional period between
Mageia 1 and the release of Mageia 2. During the transition
period, missing packages can be added to the updates repository provided
that they follow the same QA process as every other update, and that there
is a bug report filed formally requesting the package. After Mageia 2
is released, that process will be closed, and missing packages should go
through the backports repository, or simply wait for the next release
Of course, Mageia is also in the process of working out the details of its release cycle, so the proposed backports plan has its small share of uncertainty. At the moment, the distribution has a bleeding-edge distribution named "Cauldron" that operates as a rolling release, but stable end-user releases are tentatively planned to follow the six-month cycle established by Mandriva.
Bubble bubble, hopefully with minimal toil and trouble
Bootstrapping a Linux distribution is always a tremendous amount of work, even for teams like Mageia's, with its healthy number of ex-Mandriva engineers and community contributors. But as many old Linux veterans will tell you, getting the first release out the door is not nearly as taxing as setting up a smooth development, testing, and release process that is sustainable over a multi-year period. Even established distributions struggle with QA and package testing issues from time to time.
The Mageia backports discussion touched
on an intriguing issue: what precisely it means when an open source project
declares something "supported" or "unsupported." It appears as if that
thread has fallen by the wayside, and the team has moved on to the more
practical issues of fleshing out the backports process and handling the
corner-case of assisting Mandriva users as they transition to the first
ever Mageia release. But perhaps Mageia's unusual position as a brand-new
distribution staffed by team members that moved over collectively from
Mandriva gives it a unique chance to address the supported/unsupported
question. It has already allowed them to produce a full-featured
distribution in well under a year's time.
to post comments)