The stable kernel process

In particular, Sasha said, the stable trees often end up pulling in commits that, in truth, are not good enough for the mainline kernel in general (but which land in mainline anyway). Then distributors pick up the stable release and ship new bugs to users. It is not clear what can be done about this problem; the stable maintainers cannot, in general, delay fixes for months to see how well they work. He suggested that, maybe, developers could rank commits by both urgency and "scariness." But Greg Kroah-Hartman, who maintains several stable kernels at any given time, said this would not work; it's the "obviously correct" patches that turn out to be broken in the end.

Beyond that, Greg said, what often happens is that developers will revert a patch that turns out to be broken but forget to tell the stable maintainers about it. James Bottomley asked if it might help to avoid shipping stable updates during the merge window when, presumably, most of the buggy patches are merged. Greg replied that he had tried that and didn't see any difference in the results. Buggy patches can show up at any time.

Sasha complained that there is almost no testing of stable updates at all. He would like to work more closely with the distributors and get them to ship "proposed" updates for wider testing. When asked how many regressions he is talking about, he said there were one or two per release. Greg suggested that number was actually pretty good given the number of patches being shipped, but Sasha disagreed. When asked how many complaints he gets, he said one comes in every few weeks, usually for a problem that has already been fixed in the mainline.

Olof Johannson asked whether there should be -rc releases for the stable updates. Greg replied that, with a stable update coming out about once per week, there is simply no time for -rc releases.

Sasha then turned to a recent complaint of Greg's: there are too many stable kernels and too many stable maintainers out there. Greg said that we now have a situation where people don't know which stable kernel to use. Almost nobody uses more than one, and the benefit to the community as a whole is rather small. He said that Debian does benefit from the maintenance of the 3.2 kernel by Ben Hutchings, but that is the exception rather than the rule.

There was a question about whether the next stable release could be announced ahead of time, since that would allow distributors and other users to plan accordingly. Greg said that, in the past, such announcements have led to everybody trying to push crap into the mainline to get it into the stable release. It was suggested that there might be less pressure to do so now, given that most distributions do not use the long-term releases supported by Greg. The system-on-chip trees do use those releases, though.

How about making 4.4 the next stable kernel? It is too late at this point to queue up a bunch of half-baked code to go in, so that could be a relatively safe announcement to make. Announcing the stable release ahead of time could improve the predictability of the whole process. In the end Greg agreed; he has since announced that the next long-term stable kernel will, indeed, be 4.4.

That led to concerns about Greg maintaining too many kernels. It is about time to drop support for 3.10, though; evidently Willy Tarreau will be picking up extra-long-term maintenance of that kernel for those who want it. Some users, it was said, would really like to have 20-year support for their kernels, but volunteers for that task were scarce in the room.