Development process issues

Linus started off by saying that there are no serious problems that he can see. There are specific subsystems that are occasionally problematic; he shouts at them, and they generally do better. In recent times, he hasn't even had to shout all that often.

One thing that does bother him is developers who send him fixes in the -rc2 or -rc3 time frame for things that never worked in the first place. If something never worked, then the fact that it doesn't work now is not a regression, so the fixes should just wait for the next merge window. Those fixes are, after all, essentially development work. When they arrive he usually accepts them, but it's annoying and adds noise to the process. They add to his "are we getting ready for a release?" stress. In general, he said, if a fix applies to a feature that is not currently being used, it should wait for the next development cycle.

Overall, though, he said, things are working smoothly. The largest kernel release ever (measured by the number of commits) is currently in progress. This cycle has been a little painful, but size has nothing to do with it; instead, he ran into a ten-year-old bug that took a lot of work to track down. It is one of those things that happens occasionally, rather than any sort of process issue.

He suggested that one reason for the size of the 4.9 development cycle is the pre-announcement by Greg Kroah-Hartman that it would be a long-term-support release.

When asked if he should be more vocal about the above-mentioned mid-cycle fixes, Linus added that they are not really a huge issue for him. Additionally, some fixes are fine, especially for really new code. For certain areas, such as cellphones that have a short shelf life, it makes sense to push (and fix) drivers aggressively. Laptop support was also mentioned; he would like non-technical users to be able to install Linux on their new laptop that they just bought, so those kind of fixes are welcome almost anytime. But that is not what he's seeing; instead, he sees fixes for enterprise features that, due to the conservative nature of that sector, are not likely to be used for some time yet.

Bugged by Bugzilla

The bulk of the discussion, though, related to the kernel Bugzilla instance hosted on kernel.org. Laura Abbott said that bug reporting is a problem, in that users who report bugs never quite know what kind of response they are going to get. Some subsystem maintainers watch the Bugzilla and respond to issues there; others want nothing to do with it. As a result, users often have a poor experience, and are often subjected to shot-in-the-dark attempts to track down bugs from people who are not closely related to the subsystem in question.

James Bottomley said that the root of the problem is that the Bugzilla has no integration with email, which is the primary means by which kernel developers communicate. Perhaps it's time to look into using a different tool? Al Viro added that he couldn't imagine being paid enough to deal with Bugzilla. Linus said that there are groups that are more accepting and make use of it; they tend to be happy with it. But the rest of the community tends to hate it.

Darren Hart said that the Bugzilla is a good bridge between developers and users. That said, there is not much that he (as the x86 platform maintainer) can do with most of the bugs filed there. He only has five laptops, so he will be unable to reproduce most of the problems that have been reported. Some of the bug reporters can be convinced to move to email, but not all of them will do that. If the Bugzilla goes away, he said, we will lose some useful information.

There was some talk of modifying the Bugzilla to refer users to the appropriate mailing list for their problem. But kernel.org administrator Konstantin Ryabitsev said that he wants to avoid local modifications to Bugzilla if at all possible. Tweaks make the upgrade path much more complicated; he would rather remain "as vanilla as possible." He suggested, instead, that somebody should be hired to do bug triage.

Those upgrades, James said, are contributing to the current problems; the Bugzilla used to have better email integration, but that was lost in an upgrade. Konstantin said there was little alternative; kernel.org lacks the staff to backport security patches into a custom Bugzilla deployment.

Len Brown said that he likes the Bugzilla system well enough. It is not the best tool, but it can be made to work. Staying on top of things helps a lot; the power-management developers have managed to close over 3,000 bugs in recent years. The most important bugs, he said, are the most recent ones. If a developer responds immediately to a bug, there is a good chance of getting useful information back. A month later, it probably isn't going to work. Old bugs should be scrubbed aggressively; if a reporter doesn't reply to requests for information, the bug should just be closed. That keeps the list short and manageable.

The Bugzilla is a good place to collect ancillary information (such as screen shots) from bug reporters. Some developers said that email works well for that, but Linus said he would much rather deal with Bugzilla than try to fish through email archives for information. Len also said that it's generally not a good idea to put the best developers on Bugzilla duty. Instead, put a new developer there; it's a good opportunity for them to learn, and for everybody else to see how they work.

Linus said that he gets emails from the Bugzilla, and that he thinks it works OK much of the time. Many bug reports start in distribution bug trackers, though. So one ends up hopping through various links in different trackers; it is a painful process and he hates it. In the end, he would rather not see the Bugzilla be the primary bug tracking system for the kernel.

James asked if the kernel needed a dedicated person for bug management. Len replied that it would take somebody who is really good; such a person could also be a subsystem maintainer. What's really needed, he said, is a community to take on this task. Ted Ts'o said that a "bug ombudsperson team" would be a good thing to have, but it would need to be prepared to grow; as they get better at dealing with bugs, usage of the bug tracker will go up. He expressed doubts that this kind of work could be funded, it is hard to put together a business case for it. Konstantin suggested that perhaps Core Infrastructure Initiative (CII) funds could be found for kernel-bug management.

Len said that it would be nice to augment the Bugzilla to obtain other relevant information, such as the kernel version the user is running. Ben Hutchings noted that Debian's reportbug tool can run package-specific scripts to obtain the needed information. Laura added that Fedora's automatic bug reporting has useful information, but is very noisy. The best reports, she said, come directly from users who took the time to prepare them.

There was talk of eliminating the kernel's Bugzilla entirely. One approach would be to direct users to their distribution's trackers; that would not be helpful for users running mainline kernels, though. An alternative would be to replace it with a set of subsystem-specific trackers for the subsystems that are interested. The problem there, though, is that the relevant subsystem often changes as a bug is understood; moving an entry between separate trackers would be painful.

Konstantin said that kernel.org will soon be upgrading to Bugzilla 5; it is fairly different, he said, with a nicer user interface. He suggested doing that upgrade first, then perhaps seeking an intern from CII. Then, at least, we would get to a point where users who file bugs will see a response. And that was more or less how the session closed; the next step will be to see how the upgrade of the kernel's Bugzilla goes.