Testing the bleeding edge [LWN.net]

There is nothing like the joy of running a development distribution. Nowhere else can one find the same combination of huge updates (it's amazing how often the X bitmap fonts seem to change), unstable software, broken dependencies, and, for extra spice, the occasional blown-away configuration file. Whether it's called sid, Dapper, Rawhide, Cooker, or something else, a development distribution is a sure way to learn - usually at inopportune times - about what is happening at the leading edge of the development community.

Development distributions are also a good way to keep track of what developers and packagers are doing. A development distribution is alive, forever changing, forever interesting. It is a constant reminder that Linux and free software are a process, not a product. When compared to the vitality of a development distribution, stable releases seem flat and boring.

These distributions exist for a reason: having more people testing the system will help the creation of more stable releases. So developers want to have outsiders running the development version. But those developers might, perhaps, prefer to do without users who don't know what they are getting into. Consider, for example, this note sent to the fedora-testers list:

I think somewhere along the way netizens appear to think that Rawhide is stable (or at least for public consumption). I'd think we need to discuss how we can provide more constructive information for developers and send a clear message to non-testers that Rawhide (a.k.a FC5 ) is not for general use.

Another participant responded with this suggestion:

I can scream that the development tree will eat your children and destroy not only your data but your neighbor's data until I'm blue in the face... but for people who don't want to hear the warning.. they will choose not to hear the warning... and the only way for them to learn is to actually have rawhide eat their data. So i say.. every week there should be a deliberate package update in the development tree which destroys data. Thrown into the package pool at random, with an appropriate changelog entry so those of us who read the daily rawhide reports will know exactly which package to exclude.

One can safely assume that this idea was offered in a tongue in cheek mode. But the discussion as a whole does raise a question: who should be running development distributions, and for what purposes?

Development releases routinely come with warnings about their explosive nature and admonitions not to use them for any serious purpose. But the fact is that the only way to find the problems with these distributions is to use them for serious purposes. There is little to be learned by putting the distribution on a test box, noting that the installer works, and admiring the pretty desktop graphics. It's only through serious use that one discovers, say, that the web server does not handle load as well as before, that the compiler produces bogus code in certain situations, that emacs feels pretty today, or that the Wesnoth sound effects have stopped working. These are all things which are best discovered before the release is shipped; having to put together Wesnoth patches in a hurry to satisfy a service contract to a large corporation is just a real pain.

So it is important to have "real users" working with development distributions; those are the users who will come up with many of the important bug reports. Discouraging them can be a counterproductive thing to do. On the other hand, these users do need to know what they are getting into. A development distribution will bite back, sooner or later, and it's important to be able to put the pieces back together when that happens. Testers who are not prepared when disaster strikes will not, in the long run, be helpful to the development process.

This aspect of the free software development process is not often talked about. But, without widespread testing in real-world environments, software will not stabilize as well as it should. Proprietary companies run closed beta programs to obtain this testing; the free software world, for the most part, has moved away from that mode in favor of open development repositories. Open development systems are a good thing, they allow a wide variety of participants to try out the software. But these development releases are not for everybody; finding the right way to communicate that fact may be an ongoing challenge.

(Log in to post comments)

Just protect your data.

Posted Mar 2, 2006 14:47 UTC (Thu) by jabby (guest, #2648) [Link]

Absolutely. So, is there anything that the development version of the distro can do to facilitate or implement this for the unaware or lazy tester?

For instance, could Rawhide maintain a failsafe backup RPM database and a snapshot solution for /etc/, /boot/, /usr/local/, ... that allows one to successfully recover from a catastrophic failure? Obviously, this would not help if all of the filesystems are deleted, but that should be rare and we wouldn't want to take *all* the fun out if it... :-)

I know that I would be more likely to run a development distro if there were some sort of built-in recovery tool included. I don't want to have to learn and set up a snapshot/recovery system. (Yes, I'm the lazy tester.)

Oh, and could you make it a nice red button? ;-)

Make protecting your data REALLY easy

Posted Mar 3, 2006 2:19 UTC (Fri) by dwheeler (subscriber, #1216) [Link]

How's this -- before you can complete the boot, make sure that there's a remote backup system, with checkpointing, so that data is constantly backed up? Actually, having a really good backup system easily set up isn't a bad idea for the "stable" things... :-).

Testing the bleeding edge

Posted Mar 2, 2006 7:34 UTC (Thu) by danieldk (subscriber, #27876) [Link]

"It's only through serious use that one discovers, say, that the web server does not handle load as well as before, that the compiler produces bogus code in certain situations, that emacs feels pretty today, or that the Wesnoth sound effects have stopped working."

True, but serious testing also occurs on stable distributions. E.g. a lot of people run a stable distribution, and install the latest version of, say, OpenOffice.org.

Distributions are now much more feature-driven than they used to be. Beta releases of some distributions include beta releases of gcc, GNOME, et al. This speeds up software development a bit, but is not really good for the quality of the final product.

Testing the bleeding edge

Posted Mar 2, 2006 13:13 UTC (Thu) by PaulDickson (subscriber, #478) [Link]

True, but serious testing also occurs on stable distributions. E.g. a lot of people run a stable distribution, and install the latest version of, say, OpenOffice.org.

But this will not provide any testing of packages that OpenOffice.org depends on or is dependent on OppenOffice.org. Somewhere this has got to be tested or there will be no stable distribution. Testing with the stable distribution is far too limited.

Testing the bleeding edge

Posted Mar 2, 2006 17:31 UTC (Thu) by danieldk (subscriber, #27876) [Link]

Of course, I fully agree. But there is a difference between putting out a testing distribution with stable software versions, and putting out a testing distribution with beta software.

What ja where "stable" is?

Posted Mar 2, 2006 17:37 UTC (Thu) by ebirdie (subscriber, #512) [Link]

Could stable and development be put up differently than what it have been used to?

Could there be stable layers in a dist to form a "stable" distribution? For example be there layers: sysbase-stable, appbase-stable, app-stable and their *-unstable counterparts. You could form a stable dist by having all the layers from *-stable and putting an unstable layer into the mix makes the dist unstable aka development.

Further the downward layers are formed according to their dependacy-% in upper levels ie. a lib, considered to belong to appbase, needs to have 40% dependacy on it in upper layers. The layer percents could be anything by distribution. Possibly the layers could have versioning.

I think this scheme could draw more development over "stableness" and have bugs fixed onto downward (as in stableness and layers) as well.

Have this kind of scheme discussed or tried somewhere?

How about calling it a "beta"?

Posted Mar 2, 2006 18:20 UTC (Thu) by bjanz (subscriber, #1560) [Link]

Then we could convince corporate managers that it's the coming thing, that they'll have to dump their current hardware to use it, and that it has a better interface then the previous version.

We could tell users about all the neat features in the new release, and then come up with a list of features to remove at the last minute.

Of course, we'd have to start using build numbers. That way, we could release "Build xyzzy" as a beta to the "community"...

...hold on...

...was just told that it's been done already.

Sorry...

\burt
(tongue firmly implanted - heck, *embedded* - in cheek... :-))

Testing the bleeding edge

Posted Mar 2, 2006 19:56 UTC (Thu) by iabervon (subscriber, #722) [Link]

Perhaps development versions should give a much closer relationship to the bug reporting systems. I could imagine a system where you get development packages through the bug tracker, where you first see the known issues list and become acquainted with the mechanism needed to report bugs if you find them.

Of course, this wouldn't work too well with binary package distributions, where you generally can't test the development versions of only a limited number of packages without also switching to development versions of everything. Although maybe development versions should be always source-based, for this reason, and also for better ability to track issues to the responsible package (i.e., when a program crashes, is it that the new version of the program has a bug, or is it that the new glibc has a bug that this particular program exposes?).

Testing the bleeding edge

Posted Mar 4, 2006 18:07 UTC (Sat) by job (subscriber, #670) [Link]

One problem with the culture around the various Linux distributions is that new versions of packages are added to the 'unstable' tree. Now a distribution can be unstable, especially when changing the X subsystem, moving to UTF8, a new gcc or whatever it may be that distributions do from time to time.

But a new version of some pretty self-contained end user software, let's say Mozilla or something, already is rather stable when it is released. And most end users probably want the new version as soon as possible. Given the choice of compiling a lot of software themselves or running the unstable tree of their respective distro, many probably chooses the latter.

Many distributions have a backports tree, porting all new software back to the stable tree, but that's not perfect. It's not often an official tree, and may present a challenge when upgrading the distribution to the next release later.

I'm not saying having a stable tree is bad in any way, it's perfect for servers and large deployments. But there is definitively room for a "moving" tree, which tracks most software while still offering a stable base and security fixes. The only distribution I know of that does this is Gentoo, which sort-of hardly has a stable tree to begin with and doesn't fit all home users very good.

Testing the bleeding edge

Posted Mar 5, 2006 18:03 UTC (Sun) by alspnost (subscriber, #2763) [Link]

Indeed - this describes my setup too. I run Gentoo, and I have a list of packages tagged in my 'unstable' list, so I can get the bleeding edge versions. This way, I've *never* broken my system, as the core is solid, but I can still play with the latest apps etc. Mind you, even on Gentoo, architectures like AMD64 are still slightly second-tier, so crucial things like Firefox can take an age to be marked stable. I just run nightly binaries of 1.5, which suits me well. I'd be very happy to use a distro with a solid, stable core (kernel, toolchain, libraries, server components) and then to play about with bleeding edge stuff on top. But going for something like Fedora, which sometimes seems to have beta/CVS versions of the entire damn system, just doesn't inspire me.

Testing the bleeding edge

Posted Mar 25, 2006 11:23 UTC (Sat) by job (subscriber, #670) [Link]

I also run Gentoo on a number of systems, but my experience is different.
Even the stable tree broke my system a couple of times. ("Broke" in the
sense I had to fix things manually.) It doesn't really compare to Debian
stables, even if it has other features Debian doesn't.