A plan for the kernel Bugzilla

The kernel's Bugzilla instance is largely unloved and ignored, at least as a bug-reporting tool for the bulk of the upstream kernel. At the recent Maintainers Summit, Bugzilla was discussed during the regression-handling session led by Thorsten Leemhuis. In a followup to that discussion, Leemhuis posted some ideas for improving the state of bugzilla.kernel.org to the ksummit-discuss mailing list recently; the resulting discussion helped clarify a number of problem areas for it—and for the Bugzilla tool itself.

In his post, Leemhuis noted that those present at the summit expressed a fair amount of dissatisfaction with the kernel Bugzilla, so his goal was to propose a few different fixes to make things better. The main complaints during that session were effectively that bug reports via email work better for most kernel developers; there was also concern expressed that the Bugzilla project is unmaintained at this point. Part of his Kernel Summit session (YouTube video) on regression tracking was another place where many of the same problems with Bugzilla were raised. But there are kernel developers (and kernel subsystems) that use and rely on Bugzilla, so the ultimate "solution"—dropping the bug-reporting tool—is not really a viable option even though it is a popular sentiment.

Leemhuis's suggested path is to make it clear that most of the kernel does not use or pay attention to what gets submitted to Bugzilla. There are only 20 of around 2500 entries in the kernel MAINTAINERS file that specify Bugzilla as the place to post bugs; the rest either point to email addresses for mailing lists and maintainers, or to external bug trackers. Part of the problem from the user side is that there are "lots of bug and regression reports (even good ones!)" that never get a reply from a developer, as his analysis back in April showed. His goal is to redirect most of these reports to the proper places, or at least to make it clear to reporters that their bug may well be ignored.

He tries to shepherd bugs that look like regressions to the proper places and, he noted: "Artem S. Tashkinov also looks into some (all?) reports and tries to help reporters." But Tashkinov seems to have a rather different view of the path forward—one that did not sit well with many of the commenters in the thread. He suggested that instead of minimizing the use of Bugzilla, it should be expanded. Perhaps the most controversial piece was the idea that subsystem maintainers would be required to participate or, at least, that bugs reported for a subsystem be automatically sent to the appropriate mailing list(s). Furthermore:

Kernel bugzilla must be opt-out, not opt-in. To be honest I'd [automatically] add everyone who's [committed] to the kernel in the past 6 months and of course if new developers commit to the kernel, I'll add them as well. Only if they _hate_ getting bugzilla emails, they are free to unsubscribe.

Tashkinov noted that the Bugzilla instance is run by the Linux Foundation (LF), which is a well-funded organization; "The fact that no one is seriously working on it [the kernel Bugzilla] looks shameful and sad." But Konstantin Ryabitsev, who works for the LF on infrastructure for Bugzilla and the rest of kernel.org, pointed out that the Bugzilla software is old and unmaintained at this point; the infrastructure team is keeping it limping along, but that may not last forever. There is the possibility of the LF funding work in this area, he said, but there needs to be agreement within the kernel community on what it should look like, "otherwise we'll just be replacing an old and rusty dumpster on fire with a new and shiny dumpster on fire".

Herding bugs

The underlying problem, according to Ryabitsev, is the lack of someone to ride herd on the bug reports, but that is difficult to solve:

Being a bugmaster is a thankless job that leads to burnout, regardless of how well you are paid. Everyone is constantly irate at you from both ends -- the users are annoyed because their stuff doesn't work, and the maintainers are annoyed because you keep yanking them to work on dull problems that require a ton of back-and-forth with people who aren't capable of applying patches and booting custom kernels.

That led Tashkinov to recommend a wholesale shift to GitLab, which he sees as solving most of the problems that both he and Ryabitsev identified. But Greg Kroah-Hartman said that switch is not really an option either:

That's a non-starter for a wide range of reasons, not the least being you are trying to solve a "we have no one who wants to wrangle bugs in bugzilla" problem with "move all of our code hosting infrastructure to a totally different thing that can't even provide the basic things that we have today".

Sorry, not going to happen, gitlab is not the solution here.

Ryabitsev concurred: "you will still have all the exact same problems as long as nobody is in charge of handling incoming bugs". He also mentioned the danger of turning over a community resource to a for-profit company that might shift gears after "being swallowed by some $ENTITY_NOBODY_LIKES". Tashkinov replied with some cogent complaints about using email versus Bugzilla for tracking bugs; email is harder to search for existing bug reports, harder to generate reports of open bugs from, and not really suitable for large attachments (log files, configuration information, etc.) that are often needed as part of a bug report or discussion. Ryabitsev acknowledged those problems but said that the approach that is being taken is one that does not require a wholesale shift in workflows:

We've recognized this a while ago, which is why our efforts have been targeted at query-based message feeds. Hence, tools like lore.kernel.org and lei. It's a work in progress, for sure, but it doesn't require any "everyone must switch workflows today" kind of coordination, and avoids introducing single points of failure by making it easy to replicate everything to mirrored systems.

The current tooling is not adequate for tracking bugs, Laurent Pinchart said, though email is not terrible for bug reports themselves. But the tools are a side issue because it is the process that needs work: "it's quite pointless to improve the tooling if we don't first improve the process, and find a way to allocate people and time to handling bug reports". Once the process is established, the tooling will come, "be it through bugzilla or something else".

In another part of the thread, Pinchart noted that while the email standards might be sufficient to use for bug tracking, today's email clients are not up to the task. Meanwhile, a better bug tracker does not actually solve the underlying problems:

The huge elephant in the room is that most maintainers are overworked. Whether a bug report arrives in my mailbox as an e-mail straight from the reporter or from a bug tracker will make very little difference if I don't have time to look into it (I would even argue that bug trackers are even worse there: if I'm really short of time, I'm more likely to prioritize replying to e-mails instead of having to open a link in a web browser).

There certainly can be problems figuring out where to send bug reports, as Tashkinov complained about. Leemhuis acknowledged the problem, but pointed to the "Reporting issues" document as part of the solution; in the end, users should make their best guess following those guidelines and post the report there, which will hopefully lead to suggestions of where else it could be reported. Tashkinov said that users will not read the document, nor will they be able to follow its suggestions if they did. The current system is not perfect, clearly, but, as Ted Ts'o pointed out, it is volunteers who are trying to address these bugs; for those who need it, there are other options available:

Artem, it seems to me that you are hoping that volunteers will provide a commercial level of support --- and that's just never going to happen.

The users vastly outnumber us developers by orders of magnitude, and if someone needs a huge amount of hand-holding, maybe they should be paying for a support contract with Red Hat, or Suse or Canonical, or CIQ.

But Tashkinov renewed his call for adding developers to Bugzilla in order to turn that bug tracker into an opt-out tool, rather than its current opt-in nature. That was not a popular suggestion, as might be guessed. Kroah-Hartman called it "a sure way to get lots of people instantly mad at you and have them add the address to their filters". Ryabitsev said that asking "senior engineers to do first-line QA" is the wrong way to go about it; while Tashkinov maintains that it is easy for people to unsubscribe from Bugzilla, Ryabitsev said that there is an even easier path:

Please listen to the actual maintainers telling you that this won't work and will just lead to bugzilla being added to everyone's block list (that's even faster than logging in to bugzilla, finding where to change the default assignee], etc).

Tashkinov reported that there are around two dozen bugs reported in Bugzilla each week, but claimed that reporting bugs to the various mailing lists is far less effective. Geert Uytterhoeven pointed out that more than 40 bugs are getting fixed per day in the mainline (based on "Fixes:" tags), almost all of which followed the mailing-list path rather than being reported on Bugzilla.

At least twice in the thread, Tashkinov pointed to a specific kernel Bugzilla entry as an example of a bug that could not be tracked and fixed via the mailing lists. It took over two years and more than 250 messages to resolve it, which would be quite hard to manage via email. However, that bug report is hardly a model of collegial collaboration either. The tracking may well have worked better, but some of the participants certainly did not feel all that well-treated along the way.

A way forward

These are hard problems to solve, but some kind of bug-triage person (or better yet, team) would seem to be the biggest hole at the moment, at least for some commenters. Leemhuis expressed some interest in doing more of that (he already does it for regressions), though he, like many others, is lacking for time to do so; Slade Watkins has also volunteered to help. Leemhuis is not convinced that a bug liaison is necessarily highest on the priority list as he thinks that more should be done to help guide bug reporters through the process of creating good reports and getting them to the right places.

Setting the expectations of Bugzilla reporters, as Leemhuis suggested in his initial mail, would also help. That is only "some band-aids" being applied to the problems, though, he said. Ryabitsev posted a rather more ambitious plan that revolves around a triage team and a lot more automation. It was met with general agreement, though Leemhuis wondered about relying on Bugzilla going forward:

Your plan would afaics mean that we invest further into a software abandoned by its upstream and already becoming more and more of a maintenance burden. That investment would also further increase our dependency on that software by establishing workflows that rely on it. Is that really wise at this point? Wouldn't it be better to spend that time and effort to build something better that is more future proof?

But Ryabitsev said that keeping Bugzilla alive for kernel work might actually help to revive the project more generally since it is used by others: "Red Hat uses bugzilla, and so does OpenSuse, so there's a pretty good core of well-funded companies that would be in a position to help keep bugzilla going if it's looking like the platform is still alive". But meanwhile, he wants to try to ensure future-proofing by keeping it largely independent of the web application itself:

I'm hoping that by keeping the bulk of this exchange relying on established decentralized end-to-end messaging, we won't be painting ourselves into the corner quite as much as with a tool that requires all interaction to happen strictly via the web interface.

The alternative is to hire the folks behind patchwork to write "bugwork".

There are several steps to Ryabitsev's plan, but "the hard part will be keeping a team of people who are willing to do the bug triage work". There are a few people interested in helping out, so that is a starting point. No one has really complained about that path and Tashkinov said that it "looks fabulous". It will take some time to get there, but there does seem to be a plan to rework the kernel Bugzilla in a way that will hopefully improve it—perhaps to the point where some of the current skeptics will engage with it down the road.