Bug trackers and kernel development [LWN.net]

The Kernel Bug Tracker ("bugme") is a BugZilla system run by the Open Source Development Lab. It currently holds information on over 300 reported bugs in the 2.5 kernel. The Tracker is seen by many as a useful tool that brings some organization and discipline to the task of stabilizing the kernel. So it came as a surprise to many when David Miller, maintainer of the networking subsystem, requested that networking bugs not be entered into the Tracker. It is, he says, the wrong way of solving the problem.

The complaint with bug tracking systems is that they try to centralize what is otherwise an inherently distributed process. Bugs accumulate in the database, and a single person gets the job of managing all the bugs for a particular subsystem. If that person does not devote a significant amount of time to the task, the tracking system quickly clogs up with outdated reports, duplicated entries, and generally useless stuff. The time that goes into maintaining the bug tracker is, of course, time that is not available to actually fix the bugs.

The proper way of dealing with bugs, according to David, is to simply report them to the relevant mailing list. The report will be seen by the developers who can fix the bug, others who have been affected by the bug can contribute additional information, and fixes can be publicly discussed. And people who, for whatever reason, do not want to deal with a particular bug report can simply hit "delete" and the message goes away.

Of course, the "goes away" part is not always popular with those who report bugs; they would rather see the report hang around and annoy people until one of them deals with the problem. But anybody who has sent a few bug reports to a public list knows that those reports can simply vanish without a trace - a rather unsatisfying result. Why bother to report bugs if the reports can simply be ignored?

According to David (and others), the lossy nature of mailing list bug reporting is actually a feature. Bug reporting, it is said, is a process similar to patch submission. Users who do not get satisfaction from a bug report should resubmit it. If the bug is not important enough for the user to "maintain" the report, it's not worth a whole lot of effort to fix.

The "submit and retry" approach does have some advantages. Since it puts more of the responsibility for bug reports on the users submitting those reports, it scales more reliably as the number of users increases. Unimportant or "operator error" bugs vanish automatically without anybody having to shovel them out of a bug tracking system. Bugs which are fixed by (seemingly) unrelated patches also fade away automatically. The whole thing works in a scalable way without the need for central managers.

This approach is foreign and scary, however, to those who feel the need to track every bug and keep a firm hand on the development process. It provides FUD fodder for those who would portray free software development as immature and untrustworthy. It's also frustrating to those who want to retain bug report information for statistical or data mining purposes. It is, however, typical of how the kernel development process works in general. And that process, for all its faults, has produced excellent results over years as the kernel (and its development team) has grown.

(Log in to post comments)

Bug trackers and kernel development

Posted Jul 17, 2003 3:30 UTC (Thu) by hp (subscriber, #5220) [Link]

The GNOME tracker is hugely useful to GNOME, though, and the GNOME developers will tell you so if you ask. In fact there's an OLS talk about GNOME experiences with bugzilla that hopefully some of the kernel hackers will attend, Luis knows his stuff.

Some large percentage of bugs are simply never going to make it to the top of the core developers's priority queue; some other problem is always larger. From time to time new developers appear and fix a block of these, which makes tracking them worthwhile.

Often they just sit in the bug tracker, but this is a problem with software in general, not with the tracker; not enough developers to fix all the bugs. The bug tracker makes you face facts. ;-) Still, having the bugs there in case developers appear turns out to be handy sometimes.

There are a lot of bugs that do make it to the top of the core maintainers' queue, and they are very useful to track. Generally speaking, if a bug isn't closed within a month or two it's probably going to sit there for a long time, but many bugs do get closed quickly.

When releasing a large coordinated-among-many-people project you want to track release showstoppers, this is one of the most useful aspects of bug tracking. Otherwise you can't ever know what work remains before you are ready to release, and you can't focus a group of people on the remaining work.

Having used a bug tracker for years for large open source projects, I don't think it's accurate to say that if a bug is important it will be reported over and over. Often very serious bugs are only reported once or twice with no reporter followup, *especially* if the bug is in a library or the kernel where regular users will suffer from the bug but not really be able to track it down or report it. So I would say that very serious bugs do slip through the cracks if they aren't tracked.

GNOME has a process for making regular stable releases on schedule every 6 months, not a lot of other projects can say that, and the nice thing is that it seems to improve rate of innovation/change at the same time. Bugzilla is a big part of what makes the process work.

One thing to consider is that bug triagers can often separate the high priority bugs from the ones that would never make it to the top of the pile, so developers need not ever see the "bug tracker cruft" - there are a lot of nondeveloper bug triagers helping out with GNOME and Mozilla.

Bug trackers and kernel development

Posted Jul 17, 2003 9:10 UTC (Thu) by minichaz (subscriber, #630) [Link]

I would be willing to be a "bug triager" for a project. I've just graduated in Computer Science and, while I'm not a good C programmer yet (taught Java at university and learning C now), I think this could be a good way to get involved in a project and get started in FOSS development.

Any projects out there looking for a triager?

I'm pretty handy with GNU/Linux, I'm really interested in the Linux kernel and have done a bit of kernel hacking for my third year university project (designing and implementing a steganographically encrypted filesystem: I ran out of time so it is very much a work in progress).

Thanks

Bug trackers and kernel development

Posted Jul 17, 2003 12:00 UTC (Thu) by wookey (subscriber, #5501) [Link]

Yes please, Debian would love to have you on board.

Debian has been using a centralised bug-tracker for years and it's very
useful for keeping track of a great deal of otherwise-disparate
information; not just bugs, but info like the fact that someone is
working on packaging a bit of software but it's not uploaded yet (to
avoid accidental duplication of effort).

It also make us painfully aware of the number of release-critical bugs we
currently have and need to fix before we can release. We'd like to be
heading for a release right now but the RC bug count is relentlessly
climbing past 1000. More people working on fixing them would be very
welcome. Some bugs are really hard because you get tied up in package
dependencies and working out exactly what to fix where (and we have a
team of experts for the hard stuff which you can defer to), but many are
easy and simply require someone to recompile an app with a newer compiler
or fix some trivial thing and do a new upload.

It's hard to get people to concentrate on this work so we'd love to have
anyone on board who is interested. You don't need to become a debian
developer to help out, although it will help with some tasks.

Bug trackers and kernel development

Posted Jul 18, 2003 12:44 UTC (Fri) by gerv (subscriber, #3376) [Link]

I would be willing to be a "bug triager" for a project.

Not wanting to compete with the Debian guy :-), but The Mozilla project, in its new shiny independent form, would love to have your help also.

Gerv

Bug trackers and kernel development

Posted Jul 17, 2003 16:46 UTC (Thu) by cpeterso (guest, #305) [Link]

btw, I have reported a few bugs in the FreeBSD bug database. They were all fixed or proven to be "by design" behaviour within a couple weeks. My GNOME bugs have received no attention what so ever for the past six months.

JWZ describes his experience reporting GNOME bugs as follows (http://www.livejournal.com/users/jwz/154529.html):

Today a bunch of the outstanding bugs I'd reported to the Gnome propellerheads in the last couple of years were closed as follows:

Because of the release of GNOME 2.0 and 2.2, and the lack of interest in maintainership of GNOME 1.4, the gnome-core product is being closed. If y0u feel your bug is still of relevance to GNOME 2, please reopen it and refile it against a more appropriate component. Thanks... This is, I think, the most common way for my bug reports to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version.

I'm so totally impressed at this Way New Development Paradigm. Let's call it the "cascade of attention-deficit teenagers" model.

Bug trackers and kernel development

Posted Jul 17, 2003 16:59 UTC (Thu) by hp (subscriber, #5220) [Link]

jwz whines a lot. All his issues have been fixed, just not backported to GNOME 1.4. No volunteers or companies have shown interest in still supporting 1.4, so it isn't happening. Equivalent of expecting kernel 2.6 features in kernel 2.0.