KS2012: The future of kernel regression tracking

For several years, Rafael Wysocki tracked regressions in the kernel, producing lists and statistical analysis of regressions in each kernel release. This task, which provided an (imperfect) measure of the increase or decrease in the quality of successive kernel releases, was considered so valuable by other kernel developers that Rafael was regularly invited to present his observations at the Kernel Summit (2008, 2009, 2010, and 2011). However, his presentation on this topic on the first day of the 2012 Kernel Summit had a rather different flavor, asking his peers what might be the future of regression tracking.

Over time, Rafael has steadily moved to working on other tasks in the kernel, and has had less time for regression tracking. Fortunately, for a time, a couple of other people stepped in to assist with the task of creating and maintaining the kernel Bugzilla reports that were used to track regressions. However, this work did not run all that smoothly. Rafael had already noted on previous occasions that the kernel Bugzilla was not well suited to the task of generating lists of kernel regressions. In addition, as Rafael stepped still further back from regression tracking, there seemed to be some differences of understanding between his successors and various kernel developers about the how the Bugzilla should be used to track regressions. (Of note is the fact that Rafael was using Bugzilla bugs merely as a tool to track and measure regressions; whether kernel maintainers made use of those bugs as part of their work in fixing those regressions was a matter left to the maintainers.) These differences in understanding appear to be one of the reasons that Rafael's successors also stepped back from the task of regression tracking.

Which brings us to where we are today: for nearly half a year, there has been no tracking of kernel regressions. Furthermore, Rafael noted that his other commitments meant that he would not have time to return to this task in the future. This led him to ask a simple question: do we want track kernel regressions?

At this point, many kernel developers spoke up to emphasize how valuable they had found Rafael's work. H. Peter Anvin, for example, noted that he is not a fan of Bugzilla, "But, I found the lists of regressions useful. It made me do things I didn't want to do." Linus Torvalds also noted that he loved the kind of overview that Rafael's work provided to him.

The session digressed into a variety of other topics. Rafael wondered whether the Bugzilla is even very useful as a tool for tracking and resolving kernel bugs. Responses from various developers showed that use of Bugzilla varies greatly across subsystems, with some subsystems relying on it heavily, while others avoid it in preference to mechanisms such as email. James Bottomley made the point that Bugzilla allows people unfamiliar with mailing lists to fire a bug onto Bugzilla, which then (automatically) appears on a mailing list. Bugzilla thus provides those users with a means to interact with the kernel developer workflow. Later in the session, the topic of Bugzilla versus mailing lists led Rafael to raise another concern: when some subsystems use Bugzilla while others use mailing lists or other mechanisms, what should the kernel developer community tell bug reporters about how to report bugs? That question often forms a difficult first hurdle for bug reporters. Unfortunately, there was little time to delve into that topic.

There was some general discussion about how Bugzilla should be used to track regressions, and whether there might be better tools than Bugzilla for this task, but no concrete alternatives were proposed. In the end, it was agreed that the question of tooling is secondary, and the tool choice might best be left to whomever takes on the task of regression tracking. The main point was that there was widespread consensus in the room that developers would like to see the regression tracking list return, and that the top priority was to find a person (or, possibly, several, so as to avoid overloading one individual and to ensure continuity when people are absent for vacations and so on) who would be willing to take on this task.

At this point then, there's a vacancy for one or more kernel regression trackers. Although the work is unpaid, regression tracking is clearly a task that is highly valued by many kernel developers, and as Rafael's experience shows, when the work is done in a way that matches with the development community's needs, the role has a high profile. (Interested volunteers should contact Rafael.)

Note: you can avoid this step in the future by logging into your LWN account.