Left by Rawhide

Rawhide, as it happens, is older than Fedora; it was originally launched in August, 1998—almost exactly 14 years ago. Its purpose was to give Red Hat Linux users a chance to test out releases ahead of time and report bugs; it could also have been seen as an attempt to attract users who would otherwise lean toward a distribution like Debian unstable. Rawhide was not a continuously updated distribution; it got occasional "releases" on a schedule determined by Red Hat. One could argue that Fedora itself now plays the role that Red Hat had originally envisioned for Rawhide. But Rawhide persists for those who find Fedora releases to be far too stable, boring, and predictable.

The Rawhide distribution does provide occasional surprises, to the point that any rational person should almost certainly not consider running it on a machine that is needed for any sort of real work. But, at its best, Rawhide is an ideal tool for LWN editors, a group that has not often been accused of being overly rational. Running Rawhide provides a front-seat view into what the development community is up to; fresh software shows up there almost every day. And it can be quite fresh; Fedora developers will often drop beta-level software into Rawhide with the idea of helping to stabilize it before it shows up in finished form as part of a future Fedora release. With Rawhide, you can experience future software releases while almost never having to figure out how to build some complex project from source.

Rawhide also helps one keep one's system problem diagnosis and repair skills up to date—usually at times when one would prefer not to need to exercise such skills. But that's just part of the game.

In the early days of Fedora, Rawhide operated in a manner similar to Debian unstable, but with a shorter release cycle. When a given Fedora release hit feature freeze, Rawhide would freeze and the flow of scary new packages into the distribution would stop. Except, of course, when somebody put something badly broken in anyway, just to make sure everybody was still awake. While the Fedora release stabilized, developers would accumulate lots of new stuff for the next release; it would all hit the Rawhide repository shortly after the stable release was made. One quickly learned to be conservative about Rawhide updates during the immediate post-release period; things would often be badly broken. So it seemed to many that Rawhide was a little too raw during parts of the cycle while being too frozen and boring at other times.

Sometime around 2009, the project came up with the "no frozen Rawhide" idea. The concept was simple: rather than stabilize Fedora releases in the Rawhide repository, each stable release would be branched off Rawhide around feature-freeze time. So Rawhide could continue forward in its full rawness while the upcoming release stabilized on a separate track. It was meant to be the best of both worlds: the development distribution could continue to advance at full speed without interfering with (or getting interference from) the upcoming release. It may be exactly that, but this decision has changed the nature of the Rawhide distribution in fundamental ways.

In May, 2011, Matthew Miller asked the fedora-devel list: "is Rawhide supposed to be useful?" He had been struggling with a problem that had bitten your editor as well: the X server would crash on startup, leaving the system without a graphical display. The fact that Rawhide broke in such a fundamental way was not particularly surprising; Rawhide is supposed to break in horrifying ways occasionally. The real problem is that Rawhide stayed broken for a number of weeks; the responsible developer, it seems, had simply forgotten about the problem. Said developer had clearly not been running Rawhide on his systems; this was the sort of problem that tended to make itself hard to forget for people actually trying to use the software.

So your editor asked: could it be that almost nobody is actually running Rawhide anymore? The fact that it could be unusably broken for weeks without an uproar suggested that the actual user community was quite small. One answer that came back read: "In the week before F15 change freeze, are you really surprised that nobody's running the F16 dumping ground?" At various times your editor has, in response to Rawhide bug reports, been told that running Rawhide is a bad idea (example, another example, yet another example). There seems to be a clear message that, not only are few people running Rawhide, but nobody is really even supposed to be running it.

The new scheme shows its effects in other ways as well. Bug fixes can be slow to make it into Rawhide, even after the bug has been fixed in the current release branch. Occasionally, the "stable" branch has significantly newer software than Rawhide does; Rawhide can become a sort of stale backwater at times. It is not surprising that Fedora developers are strongly focused on doing a proper job with the stable release; that bodes well for the project as a whole. But this focus has come at the expense of the Rawhide branch, which is now seen, by some developers at least, as a "dumping ground."

Recently, your editor applied an update that brought about the familiar "GNOME just forgot all your settings" pathology, combined with the apparent loss of the ability to fix those settings. It was necessary to return to xmodmap commands to put the control key where $DEITY (in the form of the DEC VT100 designers) meant it to be, for example. Some time had passed before this problem was discovered, so the obvious first step was to update again, get current, and see if the problem had gone away. Alas, that was just when Rawhide exploded in a fairly spectacular fashion, with an update leaving the system corrupted and unable to boot. Not exactly the fix that had been hoped for. Fortunately, many years of experience have taught the value of exceptionally good backups, but the episode as a whole was not fun.

But what was really not fun was the ensuing discussion. Chuck Forsberg made the reasonable-sounding suggestion that perhaps developers could be bothered to see if their packages actually work before putting them into Rawhide. Adam Williamson responded:

This, in your editor's eyes, is not the description of a distribution that is actually meant to be used by real people.

The interesting thing is that Fedora developers seem to be mostly happy with how Rawhide is working. It gives them a place to stage longer-term changes and see how they play with the rest of the system. Problems can often be found early in the process so that the next Fedora development cycle can start in at least a semi-stable condition. By looking at Rawhide occasionally, developers can get a sense for what their colleagues are up to and what they may have to cope with in the future.

In other words, Rawhide seems to have evolved into a sort of distribution-level equivalent to the kernel's linux-next tree. Developers put future stuff into it freely, stand back, and watch how the monster they have just created behaves for a little while. But it is a rare developer indeed who actually does real work with linux-next kernels or tries to develop against them. Producing kernels that people actually use is not the purpose of linux-next, and, it seems, producing a usable distribution is not Rawhide's purpose.

This article was meant to be a fierce rant on how the Fedora developers should never have had the temerity to produce a development distribution that fails to meet your editor's specific needs. But everybody who read it felt the need to point out that, actually, the Fedora project is not beholden to those needs. If the current form of Rawhide better suits the project's needs and leads to better releases, then changing Rawhide was the right thing for the project to do.

Your editor recognizes that, and would like to express his gratitude for years of fun Rawhide roller coaster rides. But it also seems like time to move on to something else that better suits current needs. What the next distribution will be has yet to be decided, though. One could just follow the Fedora release branches and get something similar to old-style Rawhide with less post-release mess, but perhaps it's time for a return to something Debian-like or to go a little further afield. However things turn out, it should be fun finding a new distribution to get grumpy about.