LWN: Comments on "Why people don't test development distributions"

Second class bug reporting

pboddie — Thu, 23 Jul 2009 13:19:12 +0000

This was kind of mentioned in the article and mostly ignored by the comments, but I have to take issue with the continual persuasion to run development distributions. Just because I'm a software developer, it doesn't mean that I'm developing and testing the various distribution components from kernel through the user interface to the different applications - I write my own software, too, and want a reasonable level of stability to be able to do so, even though I may want to install cutting-edge releases of some packages. In other words, while I'm perfectly happy to report bugs, testing the fundamental components of the system is not exactly my job. And in some cases, it's clear that testing the basics was nobody's job.

What one sees with Ubuntu, for example, is that when reporting bugs, there's a resistance to do a great deal with them if one isn't running the latest pre-release version. The argument goes that only the upcoming release is fixable and the previous releases are mostly "done". Thus, "you can help with the new release" takes priority over improving something which is already doing its job. (It's also irritating to see that things like the packages Web site also defaults to the latest release, rather than "any", but I suppose they'd get complains either way.)

So, the choice is between running something that probably won't be fixed or something which is possibly broken. How about a better range of choices, distro makers?

Why people don't test development distributions

maco — Wed, 22 Jul 2009 03:09:30 +0000

Not enough developers who know what they're doing. Plenty of folks running around trying to triage bugs and getting stuck, not so many folks who know how to code, let alone code well. If left to hack, they're more efficient, but then folks like me who want to learn and have patches to fix those annoying issues keep interrupting them going "hey hey can you upload this patch?" and breaking their concentration. Patch acceptance procedures are being streamlined to try to fix that.

Why people don't test development distributions

DarthCthulhu — Tue, 21 Jul 2009 16:06:37 +0000

One of the best things about GoboLinux is the ability to have (and run!) multiple versions of software on the system. None of this: Upgrade Everything And Never Be Able to Go Back. If you upgrade a library or piece of software and it turns out to suck, you can always go back to using the one that doesn't suck. You get the best of both worlds: fast access to Shiny New Software, as well as Continuing Access to Stuff That Works.

I've even run KDE3 and KDE4 at the same time! Literally, both were running and operational in parallel. When KDE4 would inevitably crash hard, I could continue working with the KDE3 interface while KDE4 eventually raised itself from confused sloth to restart.

This does mean you have to occasionally go in and manually uninstall old libraries and software you're not using any more (by 'manually', I mean run the RemoveProgram script; you can also do it by rm-ing the files and symlinks if you want), but that is a small price to pay.

Why people don't test development distributions

Wol — Thu, 16 Jul 2009 15:34:02 +0000

The *most* *important* feature of 4.0 was the API freeze.

App developers weren't writing/testing apps because the underlying libraries were a moving target. 4.0 was the release that said "this target has stopped moving, please start porting your apps".

So OF COURSE there were no apps worth speaking of on 4.0.

Cheers,
Wol

Why people don't test development distributions

jcm — Tue, 14 Jul 2009 09:45:20 +0000

I enjoy having a laptop that actually works without surprises - you know, where you can plug in a projector and have a reasonable expectation that it works as well as it did the last time you tried a few weeks ago, and that an update didn't break it yesterday. And many other examples. I admit to being somewhat amused when people say "OMG, it broke horribly!" as a machine they rely upon randomly breaks at the worst moment (I'm not attacking Jon here!).

No. The only possible answer is to run development distributions on a dedicated development machine - I usually do it under KVM, and use full screen X forwarding and the like (to the same laptop) so it feels much like the real thing. Except I can reboot and generally things don't fall apart. Those who want to test on the bare metal - I'm sure there are plenty of second systems out there, kernel test boxes, and the like. Old laptops - you name it - but running any development distribution for daily productivity purposes is a giant time vampire and will bite eventually.

Jon.

Why people don't test beta releases and RCs

marcH — Tue, 14 Jul 2009 09:00:17 +0000

> However, the distros never get these bugs fixed in time for the release. They don't even tend to fix them within the lifespan of that release. Sometimes they are fixed in the dev version only. So what's the point?

Just lack of resources? You get what you paid for.

Bug reports like yours are still tremendously useful. I mean not just to the upstream developer(s): you have no idea how many end users are delighted to find and apply your fix or workaround.

Preling but not gdm

dirtyepic — Sun, 12 Jul 2009 07:56:35 +0000

> The real solution is shown by Gentoo: package is added to the system in
> "masked" state. It's possible to use it - but you need to specifically
> ask for it. Once enough "success stories" are obtained the mask is
> removed and all "unstable" users are upgraded.

We do? ;) There are only a couple situations I can think of where a new package/version would be added to the tree in a masked state:

- you know it's going to break things and you need guin- er, testers to smoke out the worst of the bugs (eg. gcc-4.4)
- you have a large number of packages with interdependencies between each other that need to all be made available at the same time (eg. gnome)

...unless by masking you're actually referring to keywording (~arch / arch). Then yes, packages need to be in ~arch (unstable) a minimum of one month before they can be stabilized. We don't need success stories, just a lack of bug reports. Stabilization is done by dedicated arch-tester teams, so you will always have at least one other set of eyes on a package before it hits stable.

IMO Gentoo generally manages to keep the unstable tree in working order and usable enough for everyday work. I think this can be attributed in part to the fact that we don't do releases and therefore don't have that period of churn that conventional distros do where everything is getting updated at once. Basically, we don't have a development cycle; we're always in development.

Why people don't test development distributions

nix — Sat, 11 Jul 2009 09:48:28 +0000

That was the war story I was referring to, as well. It's almost alone:
I've seen a single additional tale in which an IRIX kernel hacker did
something similar to get a dead IRIX box back to life, and that's all.
(The 'as one whose world has just come to an end' story doesn't approach
the same level of brilliance, sorry.)

People who can do things like that are *rare*.

Why people don't test development distributions

jengelh — Fri, 10 Jul 2009 22:25:54 +0000

Because it only attracts the deadweight from the Windows base - the real developers stay with what they had before ;-)

Why people don't test beta releases and RCs

Richard_J_Neill — Fri, 10 Jul 2009 21:46:38 +0000

In my experience (variously with Mandrake/Mandriva and Ubuntu), I'm usually willing to install ONE of the beta or RC releases before the final release, and put some effort into testing it. I usually find about a dozen bugs varying from outright breakage to smaller errors, and I report these. I usually track down the root cause as best I can, and include a fix or workaround.

However, the distros never get these bugs fixed in time for the release. They don't even tend to fix them within the lifespan of that release. Sometimes they are fixed in the dev version only. So what's the point?

I think there is an implied social contract here: if I (the user) test a distro alpha/beta and take the time to make a correct and detailed bug report, with all the information needed, and if I spend multiple hours of my time doing it, I *expect* the distributor to read the bug report and get the fix deployed in a timely manner, and pushed into the *stable* release, not just the next dev cycle. But most distros don't do this.

Why people don't test development distributions

Velmont — Fri, 10 Jul 2009 21:07:41 +0000

It's true that Ubuntu remains broken trough it's entire lifespan. Why is that? To not get new bugs? It's very troubling and unsatisfying.

twice a year hell, better than constant hell ?

kamil — Fri, 10 Jul 2009 14:54:10 +0000

There's some hassle with either approach, I'd just say it's a different sort of hassle.

If I update, say, the X server in Gentoo, I can expect and be prepared for some problems with, say, 3D, but I still expect the kernel to boot, sound to function, etc.

If I upgrade the whole distro, all bets are off. Will the printer still work afterwards? No idea.

I don't know which approach ultimately costs more time, I just know from experience that I personally found the distro upgrades more frustrating than individual package updates.

twice a year hell, better than constant hell ?

langagemachine — Fri, 10 Jul 2009 08:54:58 +0000

>I'm just pointing out that there are alternatives to the "reinstall >everything twice a year"-hell. There are costs involved, but to me at >least, they are worth it.

Not 100% convinced about this. After 2 years running Gentoo, I have just decided to drop it in favor of an other 'rolling' distro (Arch Linux) on the ground that I spent more time fixing upgrade conflicts than actually using the computer (well, maybe not more time, but too much time).

I am fairly pleased with Arch Linux, although this week, a system upgrade hosed bash-completion; I could get it back thanks to a gross hack, but this has left me wondering about the whole 'rolling' distro concept: do constant upgrades not mean that you are constantly unsettling your system ?

So, update every week or upgrade twice a year, the hassle is probably the same ?

Come to think of it, perpetual unstability is character of life, as my biology teacher would teach us ...

Why people don't test development distributions

johnflux — Thu, 09 Jul 2009 22:58:56 +0000

As a KDE developer - I agree. The whole KDE 4.0 fiasco was a mess.

But the developers who are embarrassed by it all just keep quiet and get to work trying to make the code better, leaving behind the noisy people who claim that KDE isn't to blame.

Why people don't test development distributions

nim-nim — Thu, 09 Jul 2009 20:59:20 +0000

I'm happy for you.

I'm sure people have the same stories for Fedora/RHEL/Centos derivatives (not recommended is different from you can make it work most of the time with little pain)

Though I doubt any distro ever shortlisted a package management system because it made easy to upgrade to a different distro.

Preling but not gdm

yokem_55 — Thu, 09 Jul 2009 19:40:21 +0000

Gentoo, while it is rare for a stable or even unstable update to completely hose a system, a couple of years ago they pushed through an upgrade to expat which bumped the .so version number, which in turn hosed everything depending on expat. On top of this, the usual tool for fixing missing libraries (revdep-rebuild) had some nasty weaknesses that made fixing one's system decidedly non-trivial. I personally had no idea what expat was or how much of a system depended on it (nearly everything that ran in X to start). There are still people running into this as the support thread for the issue in the Gentoo forums shows.

Why people don't test development distributions

mdz@debian.org — Thu, 09 Jul 2009 16:58:25 +0000

Thanks for this article. It will make interesting reading for all of the folk who ask "why don't distributions just ship pristine upstream source without any patches?" The answer, of course, is that the result doesn't actually work at all (even when updating versions from a stable state).

It turns out that making a working system out of the whole mess is actually a fair bit of work. :-)

Why people don't test development distributions

Tet — Thu, 09 Jul 2009 16:39:40 +0000

If you know beforehand the problem, with a lot of ingenuity and chance, you may recover

Sometimes, the level of ingenuity necessary is beyond that of mere mortals...

Another factor why people don't test development distributions

kamil — Thu, 09 Jul 2009 15:52:27 +0000

Such selective updates are possible in, e.g., Gentoo. You can also skip on various rapidly evolving components -- my completely up to date systems do not have pulseaudio on them, and they are still running KDE 3.5.

Mind you, I'm not advocating that every newbee should switch to Gentoo; I'm just pointing out that there are alternatives to the "reinstall everything twice a year"-hell. There are costs involved, but to me at least, they are worth it.

Another factor why people don't test development distributions

elanthis — Thu, 09 Jul 2009 14:55:44 +0000

Seconded. I run into far, far, far more bugs in every Linux stable distro release than I did with even the Beta of Windows 7. It's ridiculous. I get why, sure -- hobbyist developers have the right to only work on what they want (they're doing it for free and most of us are just freeloading off of their hard work), and features and rearchitecturing are far more fun than bug hunting. Free Software evolves at a very rapid pace but many (not all, of course) projects tend to have very low quality assurance standards.

Fedora 11 has a lot of bugs. I have no doubt that Fedora 12 will fix all of those big Fedora 11 bugs and then replace 3-4 core system components with some "new and improved" do-over project that introduces a whole new slew of bugs.

Upgrading a Linux distro is a gamble. You want to get all the stupid bugs the current distro has fixed and hope that the new set of bugs don't impede your work/fun more than the old set.

I think a large part of this has to do with the whole release strategy of Linux distributions. Install a specific set of software. Get no updates other than critical bug/security fixes. 6 months later, install a new version of EVERYTHING all at once. Get no updates other than critical bug/security fixes. 6 months later, install a new version of EVERYTHING all at once, again. Repeat ad naseum.

You can't get updated software without updating everything, so it's impossible to test the bits you care about without being forced to test all the crap you didn't see a need to change in the first place.

Why people don't test development distributions

nix — Thu, 09 Jul 2009 10:46:10 +0000

My understanding is that snapshotting the root filesystem, especially under memory pressure, is still deadlock-prone: and after the deadlock has hit everything accessing the root fs blocks, which basically means a reboot.

Why people don't test development distributions

PaulWay — Thu, 09 Jul 2009 01:47:05 +0000

The snapshotting idea is a good one, and easily implementable using LVM. Just make sure that your LVs don't use all of your disk space and you can create snapshots to your heart's content. Then you can simply tell GRUB to boot off that snapshot (AFAIK).

Having an automated system to do this in development versions of distros would be awesome though. I've done it manually on my work machine prior to trying the 'preupgrade' process to Fedora 11 and it would be nice to have that done automatically. LVM should make that a snap, if you'll forgive the pun.

Have fun,

Paul

Why people don't test development distributions

rsidd — Thu, 09 Jul 2009 01:15:43 +0000

Well, the main advantage is that upgrades <i>just work</i>. Even across distros. I've gone from Knoppix to Debian to Ubuntu via apt-get dist-upgrade. Ubuntu has a graphical update tool that takes care of some subtle things during the upgrade process, but using command-line apt-get dist-upgrade will not break your system. Fedora, last I checked, still recommends backup+reinstall for upgrading.

Why people don't test development distributions

nim-nim — Wed, 08 Jul 2009 13:31:02 +0000

dpkg+apt's main advantage is the existence of a large package pool (Debian) with little involvement by a commercial entity like Red Hat, and just static/problematic enough it's not succeeding as much as it could. So you can take the 80% done by Debian, add your 20% and get something dramatically better from the user point of view. You don't see any newer distro that chose dpkg+apt without forking the Debian Repository too.

If you try the same thing with the Fedora or Red Hat package pool you'll have an hard time proving your fork is better because there are so many public efforts pouring in the main branch (as Oracle, Mandriva, etc learnt). So it's not attractive for third-parties that want to make their own name. It takes behemmots like Intel or Oracle to try to outcompete Red Hat on its home turf nowadays.

If Ubuntu manages to raise the limit Debian-side the same way, you'll see third-parties looking elsewhere for a new base.

Why people don't test development distributions

nix — Wed, 08 Jul 2009 11:05:21 +0000

prelink *has* self-tests, but perhaps this is a problem that only shows up if ld-linux.so.2 itself is prelinked, in which case you wouldn't see it unless you prelinked the whole system.

(I'm running eglibc 2.10-head and prelink and see no trouble, though. Local RH patch breaking something? What to?)

Why people don't test development distributions

rsidd — Wed, 08 Jul 2009 04:20:13 +0000

Debian has a bad reputation because of the constant bickering on what "free" really means (but these discussions are useful in the long run, to everybody, not just to Debian!). And also because of the painful delays in its "stable" releases, but this is very misleading: Debian testing and unstable, as you say, are much more stable than the releases of many other distros.

I think the reason for Debian's success is the package management system: nothing is perfect but Debian's is as close as one can get (further improvements will require support from the OS/filesystem, such as snapshotting and rollbacks). The newer distros realise this and nearly all of them have chosen to use Debian's package management rather than RPM. If Fedora can make RPM+yum as robust and stable as dpkg+apt, then many problems may go away. But might it not be easier to just scrap RPM altogether and use dpkg+apt? Is it the NIH syndrome? Or do RPM/yum really offer something to Fedora that dpkg/apt don't?

Typo

Max.Hyre — Tue, 07 Jul 2009 19:39:17 +0000

`Predilections'. So much for level of testing.

Why people don't test development distributions

Max.Hyre — Tue, 07 Jul 2009 19:32:26 +0000

It makes a big difference to actually enforce testability. You get your nose rubbed in how much better things work.

In one wonderful (in the sense of getting to follow all those ``best practices'' you read about in books) job, we had automated testing, and were required to build in test points. Which were then evaluated in the code reviews.

Of course, we were building medical devices, and you tend to be a bit more careful when the FDA is looking over your shoulder. :-) I suspect we were within shouting distance of the 80–90 % number.

But that's only 80–90 % number. If that's what we could do under duress, it's hardly surprising we're lucky to get maybe 30 % in the real world. It's human nature (especially programmer nature) to believe both

I made no mistakes this time, and
I can't afford to take the time for real testing.

Until we figure how to overcome those predilictions, we'll have alpha-level ``releases''.

Preling but not gdm

cry_regarder — Tue, 07 Jul 2009 19:05:25 +0000

Fedora already has this. It is called bodhi. See:

http://bodhi.fedoraproject.org

But the discussion is about development __distributions__ not development __packages__.

Why people don't test development distributions

kragil — Tue, 07 Jul 2009 18:54:50 +0000

Yum does not support rollbacks in every situation. Snapshots are better.

Why people don't test development distributions

nim-nim — Tue, 07 Jul 2009 17:26:18 +0000

> Existing proccesses are unlikely to respawn more than 1 process each,
> which will fail to start, which does not qualify as a fork bomb, surely?

That depends on how many do at the same time, if some of them are configured to maintain a process worker pool, and how quickly they retry.

Why people don't test development distributions

joey — Tue, 07 Jul 2009 16:57:52 +0000

Reading the article, I can't help but think Jon is generalizing from one development distribution and reaching some not so general conclusions, although there's good stuff in there too. (I'm looking forward to btrfs rollbacks..)

My ancedote: I've run Debian unstable on all my servers and desktops for 10+ years, and only 3 times have I had breakage on the order of a broken libc or bad prelink. If I had chosen to run Debian testing, I'd have missed 2 of the breakages (testing was not available when the first happened).

Distributors would like to see wider testing of their development releases, but, as your editor's recent experience shows, there are limits to how wide this testing community can be expected to be.

According to popcon, currently 30% of the subset of Debian users who choose to enable reporting use unstable or testing, not stable. If anything, some in Debian may wish to see less wide use of its development branches, but stable is not useful for lots of users.

Anybody who has worked with development distributions for any period of time knows that the early part of the distribution development cycle is when things are most likely to go wrong.

This is certianly true, and a look at the debian-user mailing list will find similar warnings about using unstable after a release. Less so for the testing distribution, as the way testing's algorythm reacts to lots of churn and bugs in unstable is to stop updating many packages from unstable until the churn quiets down.

Provide an indication of the state of the distribution. Many beaches are equipped with red flags which are posted when dangerous currents are present. Wouldn't it be nice if an apt-get upgrade could respond with a message like "the current threat condition is orange, you may want to reconsider"?

As previously noted, apt-listbugs can do just that. It looks like about 1 in 10 users of unstable (or testing) uses apt-listbugs.

Why people don't test development distributions

iabervon — Tue, 07 Jul 2009 16:31:12 +0000

One think I really like about Gentoo is how the "developement version" works. The user gets to list all of the packages where they would like to get development versions (which can be "everything"), and they get development versions of those and not of other packages. (Of course, this depends on the distribution supporting being in a mixed state, which is much easier with a source-based distribution than with a binary distribution, where you'd have to make some hard choices about which versions of libraries everything links against; there's still obviously some cases where some combinations fail to meet version requirements for dependencies).

This is really handy for being able to test particular packages that you're interested in, and being able to adjust these as needed. For example, I could positively identify that X.org server 1.5 (and not anything else) broke my arrow keys. Also, I could then say that I didn't want to test it any more, because I'd determined that it had problems and didn't feel compelled to suffer needlessly until there was a possible fix. (Actually, it turned out that I was accidentally remapping the wrong keys in a config file because keycodes had been renumbered, so the fix was to my configuration, but I could use my arrow keys in 1.4 while I figured this out).

I suspect that it would be really helpful to the process in general to be able to tell your package manager: don't give me a system where bug #X is known to be present. Then you could do your update, find that sound doesn't work, report the bug (or find the existing bug report), say that bug's a show-stopper for you, update again (causing that package to be downgraded to an earlier version), and find out whether the new version of your music player is broken or not. When the sound bug might be fixed, the package manager would try upgrading again so you could test, and you could either say that it's still broken (and again revert to the known-good version) or find that it now works.

Why people don't test development distributions

joey — Tue, 07 Jul 2009 16:31:04 +0000

Existing proccesses are unlikely to respawn more than 1 process each, which will fail to start, which does not qualify as a fork bomb, surely?

I had excellent luck once recovering from a hosed ld.so using only zsh. zsh, you see, includes a built-in ftp client that doesn't need to fork at all in order to download a file. Only an open root shell was needed.

Why people don't test development distributions

salimma — Tue, 07 Jul 2009 15:33:18 +0000

yum (Fedora) and zypper (openSUSE) support delta RPMs, so only incremental changes need to be downloaded during an update. yum supports rollbacks too.

Why people don't test development distributions

salimma — Tue, 07 Jul 2009 15:30:03 +0000

From the bug posting, it looks more like a glibc/prelink mismatch after a *glibc* update, not a broken prelink -- though, granted, both ought to have been upgraded at the same point (and ideally maintained by the same person).

Preling but not gdm

mjthayer — Tue, 07 Jul 2009 14:49:04 +0000

> The real solution is shown by Gentoo: package is added to the system in "masked" state. It's possible to use it - but you need to specifically ask for it.

You echo my thoughts. I'm sure many users would be happy (even eager) to test certain unstable packages that they are interested in. Compare that to the number who would be happy to work with an unstable distribution (I for one would be very reluctant to). The only thing that Gentoo is lacking in this respect is a way to pull in the dependencies of the unstable packages you wish to test without having to remove the stable versions that your other packages are using.

Endless updates

southey — Tue, 07 Jul 2009 14:40:32 +0000

In at least the case of Fedora rawhide you often get updates which is a little tiresome because there can be many many packages to update at the same time. If you choose to update, then update fails because usually due to some fake dependency on other package (especially if the package is backwards compatible). It also becomes tedious to even select which updates to apply or find the package(s) that are blocking your update.

But on the positive side, it does allow you to get the major changes to the core packages like KDE and Gnome without having to wait until the next release in a years time. Also with Fedora you can stop at a release if you get a system you like!

Why people don't test development distributions

nim-nim — Tue, 07 Jul 2009 13:52:45 +0000

You keep reading this kind of article because Fedora is a very active and open project. Fedora makes headlines about its problems because it always tries to improve, so heated post-mortem public discussions are common and expected.

Users of other distributions seem to prefer tip-toeing around problems to avoid giving their choice a bad rep.

Why people don't test development distributions

mrshiny — Tue, 07 Jul 2009 13:38:44 +0000

I'd be willing to run a development distro except I have so many problems with the stable one, I can't imagine dealing with more problems. Maybe it's my choice of Distro, but sometimes Fedora's stable updates break random things for me. I already feel like I'm testing beta software.