User: Password:
|
|
Subscribe / Log in / New account

Fedora wrestles with the auto-closing of bugs

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

By Jake Edge
February 12, 2014

Bug tracking is a perennial source of pain for distributions and other free software projects. Bug reports vary widely in quality, there is never enough developer time to even verify all of the bugs, and some bugs may get fixed in the normal course of development, entirely separate from the bug report. But bug lists that just keep growing are not terribly useful, so there is typically some kind of automatic closing of bugs that is done. The criteria for auto-closing is often contentious; finding a balance between a ballooning bug database and not irritating reporters and other users can be difficult—as a recent fedora-devel mailing list thread shows.

In some ideal world, bug reporting would go something like this:

  1. Someone reports a bug, with lots of detailed information.
  2. The bug gets triaged and then assigned to the proper developer.
  3. The developer asks for any more information needed to help reproduce and/or fix the bug.
  4. The reporter responds promptly with the needed information.
  5. The bug gets fixed and closed.
All too often, however, the first four steps are mishandled (too little information, no real triage or assignment, no developer time to look into the problem, no answer to any queries) so that the last never happens. Sometime in the interim, a new version of the package—or a whole new version of the distribution—is released and the bug's validity for that version comes into question.

That's where auto-closing comes into play. For Fedora, a new release of the distribution triggers the "end of life" (EOL) of the N-2 release (i.e. the release of Fedora 20 meant that Fedora 18 went EOL shortly thereafter). The project has a policy that it won't carry bug entries that only apply to EOL releases, so four weeks before the release goes EOL, a message is posted to the bug to that effect (example). If the bug is not updated to change the release it applies to, it will be closed.

In the example bug, though, David Strauss does note that the bug still affects Fedora 19, but he didn't change the release version it applies to—because he couldn't. He did not report the bug, nor does he have the privileges needed to edit all bugs, so he just posted a comment to the bugzilla entry. Thus he was a bit irritated to find that the bug had been closed:

I specifically followed up to say the issue continues in Fedora 19, and nothing changed. The bug tracker should not expire bugs if there's been a comment after the EOL warning.

There were suggestions to change the version on the bug (he can't), to join the Fedora BugZappers in order to get privileges needed (that group is defunct), and to file an upstream bug with X.org (as it is a driver bug). Some showed a distinct lack of sympathy for Strauss's complaint but, as Colin Macdonald put it, bugs getting automatically closed does not show Fedora as "a welcoming community to newcomers".

However, as Adam Williamson pointed out, no one has come up with a better solution yet. Not closing bugs that get comments after the EOL warning risks leaving bugs open if someone simply reports that the problem has gone away in a newer release, he said. Several thought that reports of non-working systems would be much more likely than those reporting that the bug had gone away (since those whose systems are working tend to be less attentive to bugs that got fixed).

There was some confusion in the thread, but Williamson and others clarified that the reporter can change the version (and thus leave the bug open). In the case of Strauss's bug, the reporter had stopped posting on the bug, which leaves it to the developer responsible (typically package maintainers who tend to have a lot on their plates) to notice any comments and update the bug accordingly.

Though he called it "a radical plan", Williamson suggested one possible fix: allowing anyone to reopen closed bugs. Michael Catanzaro had a different fix: just allow anyone to change the version affected by the bug. That would eliminate the problem that Strauss ran into. Matthew Miller had a more elaborate plan that would leave bugs open for one more release cycle to give users more time to respond before they get the message that their bug has been closed.

Williamson is not sure there is a real problem, partly because there are too many open bugs already, so adding more (by not auto-closing them) isn't helpful. If the bug didn't get the attention it needed, leaving it open doesn't make it any more likely that it will . But Miller noted that it is a complaint he hears regularly at conferences. When people take the time to file a bug, they are clearly hoping for more than just an eventual "closed due to EOL" message.

On the other hand, Alexander Kurtakov described the problems that package maintainers have with open bugs:

[...] if there is request for information from the assignee of the bug and there is none given the bug should be auto-closed (time period to be decided - 3 months sounds more feasible). This would be more helpful to people that fix bugs and triage their bugzillas too. Due to triaging many such bugs it makes it impossible for me to keep the pace and the very few good bugs (with enough information) are lost sometimes in the huge number of bugs so good reporters get upset. It's unfortunate that this happen but my personal experience is that >90% of reporters are unwilling to spend time helping reproducing the problem which makes the reports useless most of the time.

That's a common problem with bugs. And one that's not likely to be solved anytime soon.

For the EOL issues, there is a Fedora wiki entry that covers the tasks, but it still assigns handling bugzilla entries to the disbanded BugZappers group. Miller filed a bug that eventually found its way to a Fedora Engineering Steering Committee ticket. The latter resulted in some changes to the wording of the EOL message, but there may be more changes coming.

The discussion makes it clear that few are completely satisfied with the current scheme. Fedora Program Manager Jaroslav Reznik has gathered some statistics to help quantify the number of bugs closed due to EOL releases and it is substantial: roughly 6000-8000 for each release Fedora 14-18 (with Fedora 15 omitted due to lack of data).

Ultimately, the problem is rooted in the fact that many bugs get little or no attention after they are filed. If there were enough package maintainer and/or upstream developer time to address the bugs, there would be few left to auto-close. The upstream projects are typically better equipped to handle bugs in their code, which has led some to call for filing all bugs upstream.

Bug handling is often tricky and contentious. Reporters generally believe their bugs are high quality and should get immediate attention, while maintainers often find a different picture—if they have time to find anything at all. Like other projects, Fedora is trying to feel its way through something of a minefield.


(Log in to post comments)

Fedora wrestles with the auto-closing of bugs

Posted Feb 13, 2014 14:53 UTC (Thu) by mattdm (subscriber, #18) [Link]

It looks like the proposal I can get to pass FESCo will be:

1. New "CLOSED:EOL" resolution added.
2. Only the EOL script will be able to put things into CLOSED:EOL
3. Script will be run immediately when a release hits end of life.
4. Any user will be able to reopen bugs which are CLOSED:EOL, but without
special privileges will be made to select a new, open Fedora version

This doesn't make me as happy as my proposal to extend the needinfo period significantly, but I do think that it's progress.

The arguments against needinfo seem to be that 1) people don't really pay attention until the bug is actually closed, 2) not closing sends the message that there's the possibility for a fix in the old release, and 3) bugs sitting there open but with the needinfo flag add complication and noise for maintainers.

I think we should also look at some ways to automatically flag certain bug reports for human triage (e.g. but not limited to: high number of incoming duplicates, lots of comments, already closed and reopened previously). There's really no reasonable way for anything but a small army to go through the number of bugs we're auto-closing now, and it would be nice to find a way to highlight a reasonable subset for attention. But that's not really a developed idea yet. (Help wanted, etc.)

Fedora wrestles with the auto-closing of bugs

Posted Feb 13, 2014 16:19 UTC (Thu) by epa (subscriber, #39769) [Link]

That's great, but there needs to be a way to select a new Fedora version *before* the bug is closed as EOL. Although ordinarily only the reporter can change the version, any user should be able to mark the bug as "this still happens in Fedora 20, please don't EOL it". Let the user enter that information today, don't make them wait until after some arbitrary expiry date to update the bug tracker.

Fedora wrestles with the auto-closing of bugs

Posted Feb 13, 2014 16:29 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

Anyone can duplicate bugs if it happens on more than one release.

Fedora wrestles with the auto-closing of bugs

Posted Feb 13, 2014 18:53 UTC (Thu) by mattdm (subscriber, #18) [Link]

Interesting point, although in that case, moving the bug report forward may mean that a bug is only fixed in the newer release whereas otherwise the maintainer would have backported a fix to the old one too.

It adds complication, but maybe that's something we could turn on in the last month of a release's life (that is, when the N+2 release comes out). (That's kinda ugly.)

The bug reporter, maintainer, and people with "editbugs" can change the release forward, and _hopefully_ would notice any comments to that effect. Of course, if that were 100% effective, we wouldn't be having this whole discussion.

I also learned that if you set a bug to "rawhide" and add the FutureFeature keyword, it won't be auto-rebased to a numbered release and so kept open. That's pretty much an easter egg, though.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 8:18 UTC (Fri) by epa (subscriber, #39769) [Link]

Obviously the Fedora version needs to be a set rather than a single field, since a bug can be present in more than one version.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 16:26 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

I tend to use RFE in the subject instead which accomplishes the same thing. Is there some special tracking of FutureFeature that RFE misses?

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 17:13 UTC (Fri) by mattdm (subscriber, #18) [Link]

Yes. Bugs filed against rawhide and not marked FutureFeature eventually get batch-reassigned to a versioned release and then eventually EOLed. If you want it to stay open in rawhide forever, FutureFeature is the ticket.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 18:15 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

No, RFE in the subject has the same effect for the EOL script[1] (as do Review Requests and Merge Requests, but I think those are detected by the component); I was wondering whether FutureFeature had *other* benefits. Personally, I like the RFE route since it's visible in the emails that get sent.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=803863

Fedora wrestles with the auto-closing of bugs

Posted Feb 15, 2014 1:06 UTC (Sat) by mattdm (subscriber, #18) [Link]

Oh, whoops. I missed that that was magic too. Never mind. :)

Fedora wrestles with the auto-closing of bugs

Posted Feb 15, 2014 4:12 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

Oh, FYI, the automatic update monitoring tool bugs are also pinned to Rawhide.

Fedora wrestles with the auto-closing of bugs

Posted Feb 13, 2014 19:12 UTC (Thu) by intgr (subscriber, #39733) [Link]

> It's unfortunate that this happen but my personal experience is that >90% of reporters are unwilling to spend time helping reproducing the problem

It works the other way around too: in many projects, 90% of bugs get no attention. If I spend hours and sometimes days on reproducing and tracking down an issue, to write up a detailed bug report, only to have the bug closed years later by a bot with no evidence of intelligent life, I'm not going to spend any more of my time on reporting bugs to that project. So it's a feedback loop issue: only people who don't invest much time in bug reports will keep reporting bugs.

It's very frustrating, but given limited resources, I don't know if there's much that can be done about it. Some projects manage; if I find a bug in PostgreSQL, I will get to it immediately because it will usually get fixed within days of reporting. If I find a bug in ALSA, however, I'm better off spending my time writing a novel to /dev/null.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 16:25 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

For ALSA bugs, maybe try notifying PulseAudio devs?

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 3:34 UTC (Fri) by SiliconSlick (subscriber, #39955) [Link]

Maybe it is wrong, but sometimes, when a bug has been marked closed, I open a new bug, with a supported version, give all the requested info, and then cite the previous, auto-closed, bug.

Works sometimes.

Red Hat can be slow (3+ years) at times, but they don't auto-close. For the most part, sooner or later they address the issue.

I know Fedora is the fast track, but sometimes valuable knowledge is lost in the rush onward. Sometimes that includes triage, and maybe solutions, provided by non-packagers simply due to an itch in the side ("why doesn't his work?"). All that info/research gets pushed aside/lost (if not searched for) when the next time the bug arises.

I'll confess, I'm an obvious Fedora/RHEL/CentOS user.

I'm not here for holy wars. But... I've browsed Canonical's bug tracking on occasion and found it quite offensive in its dismissal of bugs. I hope
Fedora doesn't go there.

I don't have a solution, but in the meantime, I'll be contrary and open a new entry on bugzilla, point out that the problem _still_ (*sigh*) exists, and then point it to a supported version.

e.g. RH can be slow but...

Posted Feb 14, 2014 4:52 UTC (Fri) by SiliconSlick (subscriber, #39955) [Link]

As an addendum, I'll offer up this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=500767

It took a while to address, but sooner or later it was.

I think I provided a decent triage of a very peculiar corner case. I provided a detailed (I thought) example. I provided a possible fix ('s/>/>=/').

Although it took awhile, the problem was finally addressed. But it was never closed in the meantime. As frustrating as it was (not much, given that I knew of the problem), it isn't as frustrating as having a similarly detailed bug report dismissed when reported against a current Fedora version that then reaches EOL.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 14:11 UTC (Fri) by michel (subscriber, #10186) [Link]

My experience has been:
  1. I report a bug, with (possibly lots of detailed) information.
  2. The bug gets automatically triaged and then assigned to the proper package maintainer.
  3. Not much happens here, but sometimes other folks post useful comments
  4. The bug gets fixed by upstream and closed (possibly by me if I remember to do so).
Given the amount of packages and churn, not sure there's much more one can do. But one thing that might be nice is a better triage/linkage/communication with upstream projects. I for one (and perhaps that makes me lazy, a bad citizen, whatever) would probably not file bugs upstream if that is what was required, since I'd have to register with yet another reporting system, remember where to go track down my report, etc.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 19:13 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

1. I report a bug, with (possibly lots of detailed) information.

I think you're wasting valuable time including detailed information in a report like this. There's a good chance 1) no one will be available to work on the bug; or 2) it's a well-known problem and the information is already known; or 3) it's widely reproducible so that additional information from you wouldn't help much.

If I bother to negotiate the bug reporting system at all, I just put in a brief description, and if I get a response saying, "I'm interested, but based on that information, it works for me," then I knuckle down and gather version numbers, screen shots, etc. I might even try to fix the bug, once I know there's someone who might use my fix.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 18:10 UTC (Fri) by ssam (guest, #46587) [Link]

One possible system.

At EOL post a message asking if it can be reproduced in the latest release, and set bug to NEEDINFO. If a bug stays NEEDINFO for N days then close it. I suggest N somewhere from 30 to 90.

Fedora wrestles with the auto-closing of bugs

Posted Feb 14, 2014 19:19 UTC (Fri) by mattdm (subscriber, #18) [Link]

That was basically my first proposal, although I suggested a longer N. But see my first comment to this post; that idea didn't pass.


Copyright © 2014, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds