Fedora looks to increase Rawhide testing [LWN.net]

By Jake Edge
February 25, 2009

Development branches of a distribution are generally hard environments to use because they tend to frequently be in a broken state—so broken that it is impossible to get one's work done. Fedora Rawhide is such a branch, which, up until recently at least, came with the scary warning: "Rawhide eats babies". So it is a bit surprising to see an effort to increase the number of Rawhide users. The benefits for Fedora are obvious, but the number of headaches and complaints that could come from more users might offset the extra testing that it would get.

Rawhide horror stories abound, but, in general, its quality has been improving in recent times. As part of a report from his recent orientation at Red Hat headquarters, Adam Williamson posted some goals for Fedora QA to the fedora-testers mailing list. The first specific goal listed—and the one that attracted most of the comments on his post—was to "increase participation in Rawhide". Williamson was formerly a community liaison with Mandriva and recently took on a similar role in QA at Red Hat. He outlined some specific steps that the QA group wants to take with Rawhide:

I am going to work on communication and documentation issues around that, and Will [Woods] is going to work on producing a tool which simply tests, every day, whether you can a) install Rawhide fresh and b) update from latest stable+updates to Rawhide. This serves two purposes: it both lets you know whether it's worth actually attempting to install Rawhide that day if you wanted to know, and if we track the results over time, it provides an incentive to the developers to improve the reliability of Rawhide.

Mark McLoughlin suggested coming up with some criteria for what a testable ("dogfoodable" in his words) Rawhide looks like. Changes that cause it to fall below that line—because it doesn't boot or some core functionality, like networking or graphics, doesn't work—should be added to bugzilla as a RawhideBlocker bug. Pressure could then be applied to get those bugs fixed quickly. Interested testers would also have an opportunity to see if Rawhide was in a testable state before installing or updating.

Concerns were expressed about just who should be considered a good candidate for testing Rawhide. McLoughlin thinks "we should keep trying out new things to get it to the stage that anyone involved in Fedora development should be able to run rawhide". Williamson agrees:

The point is that this pool of people is in fact far larger than the number of people who currently run Rawhide. It should at least include the vast majority of packagers, yet from what I've seen, it seems that a lot of Fedora packagers only run stable releases, which is a pretty reliable indicator that we really could have more people running Rawhide.

But Bruno Wolff is worried that the bar is being set too low: "you need to be able to rescue your system when booting fails. I think you pretty much need to be an amateur sysadm." Williamson, based at least partially on his Mandriva experiences, is not too worried about that problem:

Usually, also, if the problem is one that affects more than a few people, someone will post a note about what's wrong and how to fix it to the discussion list. Or, they would, if enough people ran Rawhide. :)

It is clear that one can run into problems with Rawhide, but the author was able to write the bulk of this article—along with handling a few other normal tasks—on a laptop running Rawhide from February 24 with few problems. The display would not default to the 1280x800 resolution of the laptop—likely caused by bug 485913—but that could be worked around by use of the KDE display setting program. Wolff also reported some nasty boot problems and alluded to kernel modesetting issues both of which would be problematic for a regular user to overcome. Some grumpy guy from LWN, who often runs on the bleeding edge, pointed out a few other issues (with tomboy, cups, and others) that he has run into using Fedora 11 Rawhide.

But, the only kind of testing that is likely to find these kinds of problems is real-world day-to-day use of the distribution—a quick install test won't show them. It is the classic chicken-or-egg problem that distributions face. Most distributions opt for recommending that users stay away from their development branches, instead awaiting alphas, betas, or release candidates. Finding critical bugs at that point is much more painful, however. Fedora is trying to find a middle ground between getting buried in bug reports, while still finding bugs as early as possible in the process.

Each user has their pain threshold that they are willing to bear while helping to improve the free software they use. Some have a threshold near zero, while others have enough experience—or masochism—to be willing to deal with the kinds of messes that can result from tracking a development branch. It is best for all concerned to make sure that the right message is sent, so that the right people are using Rawhide. If expectations are not set correctly, it could well leave Fedora worse off than it was before. It is an interesting experiment, one worth keeping an eye on.

to post comments

Fedora looks to increase Rawhide testing

Posted Feb 26, 2009 11:07 UTC (Thu) by taggart (subscriber, #13801) [Link] (3 responses)

So Fedora is trying to get what Debian has had for >10 years? :)
Yeah I think that will help them immensely. Having a huge number of people updating daily really tightens up the development cycle and makes life easier for all developers. This only became possible for Fedora once they got yum, but already the same culture is starting to develop and this article is a sign of that.

Fedora looks to increase Rawhide testing

Posted Feb 26, 2009 13:48 UTC (Thu) by miguelzinho (guest, #40535) [Link] (2 responses)

I couldn't agree more.

Fedora looks to increase Rawhide testing

Posted Feb 26, 2009 18:44 UTC (Thu) by JoeBuck (subscriber, #2330) [Link] (1 responses)

Rawhide is generally a lot more unstable than Debian unstable, precisely because so many use Debian unstable, and also because Debian also has "experimental" which gets things likely to cause breakage.

Fedora looks to increase Rawhide testing

Posted Feb 27, 2009 14:14 UTC (Fri) by amit (subscriber, #1274) [Link]

experimental was introduced quite a long time after they had the 'stable', 'testing' and 'unstable' repositories. In fact I used to be on sid and updated once every three days on my desktop system. If anything ever broke, I could go back to the previous version or check back in a couple of hours and found a fixed package already available.

Fedora looks to increase Rawhide testing

Posted Feb 26, 2009 11:47 UTC (Thu) by Cato (guest, #7643) [Link] (1 responses)

Perhaps the solution here is to make instant and up to date backups really easy to configure in Rawhide - as long as you have another server somewhere, and can ensure you lose no more than an hour of work, it's not such a big deal to have problems. Just reboot into a stable distro on another partition, or even a live CD.

Internet-based backups would make recovery even easier and faster - something like Mozy would be good. Unfortunately Linux has very few (if any) available online backup services that don't require a lot of configuration and setup - what I'm thinking of is "yum/apt-get install foobackup", 2 minutes configuration, and you're done. Foxmarks makes this pretty easy for bookmarks, and Dropbox is good for 2-way syncing on Linux, but nobody has cracked this for remote backup.

Fedora looks to increase Rawhide testing

Posted Feb 26, 2009 14:53 UTC (Thu) by zlynx (guest, #2285) [Link]

Online backup is hideously slow. Even on a company network, it can take hours to pull in several gigabytes.

USB external drives is the way to go.

But I agree about the importance of system backup. I think Fedora should add a configure backup section to the system startup script that runs after first install. Backup is something that every system should have.

Fedora looks to increase Rawhide testing

Posted Feb 27, 2009 0:10 UTC (Fri) by giraffedata (guest, #1954) [Link]

Most distributions opt for recommending that users stay away from their development branches, instead awaiting alphas, betas, or release candidates.

If they're recommending that users use alpha releases, they don't know what an alpha release is. An alpha release is a release made for alpha testing. Alpha testing is simulating use of a product for the sole purpose of finding bugs in it. Beta testing is using a product for the dual purpose of getting work out of it and finding bugs. Production is using a product for the sole purpose of getting work out of it. So alpha releases are for testers, not users.

In open source practice, the development branch is typically used as a beta release (whether the project calls it that or not).

Fedora looks to increase Rawhide testing

Posted Mar 6, 2009 13:46 UTC (Fri) by Duncan (guest, #6647) [Link]

"you need to be able to rescue your system when booting fails. I think you pretty much need to be an amateur sysadm."

[mode=homer] Doh! [/mode]

By very definition, people who are making decisions about what gets installed and run-permissions on their system, etc, people touching system config files, that is, not limited to editing their own preferences in /home, are administering their system, that is, they are being system administrators, aka sysadm.

Part of the problem with MS in fact was that they tried to pretend this wasn't the case, that ordinary users, and more importantly, those with no INTEREST in being anything but ordinary users, could handle the responsibility of admining their systems. They tried to take shortcuts. We /know/ where /that/ got us!

The Linux community has historically shied away from that, making fun of and recommending against the distribs that didn't HIGHLY encourage users to keep separate user and root accounts and configs, and to run as ordinary users most of the time. There's absolutely nothing wrong with being just a user, nor is there nothing wrong with not even WANTING the responsibility of having to decide what's secure and safe to click on and what's not, but such users need to be under the safe care and administration of real system administrators, those who tho they may be inexperienced and make mistakes once in awhile (we all start somewhere, and surely, one of the marks of a good sysadmin is that they know they can fat-finger things on occasion regardless of experience, but are prepared and can recover when they do), at least CARE about security enough to spend TIME learning what's safe to click and what's not, and when adminning for others as well as one's self, how to setup permissions appropriately to protect those who don't care and don't WANT to care, about being more than users.

Now I'm reading that it's accepted that it's OK for people that aren't sysadmins to, despite the definition, "admin" systems, , as long as they are "stable"? Whiskey-Tango-Foxtrot!? Like it or not, by very definition, people who admin systems are -- surprise -- sysadms! And the sooner everyone quits pretending there's a way around it and starts learning to live with it, the sooner sysadms themselves realize it and either start paying attention to stuff like backups and security and recovery-boot scenarios, or run screaming back to the relative safety of normal userdom where someone /else/ takes that responsibility and makes those decisions, the better off everyone, the entire community, the entire Internet, the entire WORLD, is.

Isn't that something the Linux and larger FLOSS community has been telling the MSs of the world all along? Now it looks like we need to do a bit more telling our own, the Red Hats and Fedoras of the world, the same thing.

Duncan