Fedora looks to increase Rawhide testing

Development branches of a distribution are generally hard environments to use because they tend to frequently be in a broken state—so broken that it is impossible to get one's work done. Fedora Rawhide is such a branch, which, up until recently at least, came with the scary warning: "Rawhide eats babies". So it is a bit surprising to see an effort to increase the number of Rawhide users. The benefits for Fedora are obvious, but the number of headaches and complaints that could come from more users might offset the extra testing that it would get.

Rawhide horror stories abound, but, in general, its quality has been improving in recent times. As part of a report from his recent orientation at Red Hat headquarters, Adam Williamson posted some goals for Fedora QA to the fedora-testers mailing list. The first specific goal listed—and the one that attracted most of the comments on his post—was to "increase participation in Rawhide". Williamson was formerly a community liaison with Mandriva and recently took on a similar role in QA at Red Hat. He outlined some specific steps that the QA group wants to take with Rawhide:

Mark McLoughlin suggested coming up with some criteria for what a testable ("dogfoodable" in his words) Rawhide looks like. Changes that cause it to fall below that line—because it doesn't boot or some core functionality, like networking or graphics, doesn't work—should be added to bugzilla as a RawhideBlocker bug. Pressure could then be applied to get those bugs fixed quickly. Interested testers would also have an opportunity to see if Rawhide was in a testable state before installing or updating.

Concerns were expressed about just who should be considered a good candidate for testing Rawhide. McLoughlin thinks "we should keep trying out new things to get it to the stage that anyone involved in Fedora development should be able to run rawhide". Williamson agrees:

But Bruno Wolff is worried that the bar is being set too low: "you need to be able to rescue your system when booting fails. I think you pretty much need to be an amateur sysadm." Williamson, based at least partially on his Mandriva experiences, is not too worried about that problem:

It is clear that one can run into problems with Rawhide, but the author was able to write the bulk of this article—along with handling a few other normal tasks—on a laptop running Rawhide from February 24 with few problems. The display would not default to the 1280x800 resolution of the laptop—likely caused by bug 485913—but that could be worked around by use of the KDE display setting program. Wolff also reported some nasty boot problems and alluded to kernel modesetting issues both of which would be problematic for a regular user to overcome. Some grumpy guy from LWN, who often runs on the bleeding edge, pointed out a few other issues (with tomboy, cups, and others) that he has run into using Fedora 11 Rawhide.

But, the only kind of testing that is likely to find these kinds of problems is real-world day-to-day use of the distribution—a quick install test won't show them. It is the classic chicken-or-egg problem that distributions face. Most distributions opt for recommending that users stay away from their development branches, instead awaiting alphas, betas, or release candidates. Finding critical bugs at that point is much more painful, however. Fedora is trying to find a middle ground between getting buried in bug reports, while still finding bugs as early as possible in the process.

Each user has their pain threshold that they are willing to bear while helping to improve the free software they use. Some have a threshold near zero, while others have enough experience—or masochism—to be willing to deal with the kinds of messes that can result from tracking a development branch. It is best for all concerned to make sure that the right message is sent, so that the right people are using Rawhide. If expectations are not set correctly, it could well leave Fedora worse off than it was before. It is an interesting experiment, one worth keeping an eye on.