The stable kernel process

Sasha Levin maintains the 3.18 stable kernel series for Oracle. At the 2015 Kernel Summit, he led a session discussing the process of creating stable kernels in general and how it can be made more robust. The session ended with a surprise decision on the next long-term support kernel, but, first, Sasha wanted to talk about the problem of too many bugs getting through to kernels that are supposed to not acquire new problems.

In particular, Sasha said, the stable trees often end up pulling in commits that, in truth, are not good enough for the mainline kernel in general (but which land in mainline anyway). Then distributors pick up the stable release and ship new bugs to users. It is not clear what can be done about this problem; the stable maintainers cannot, in general, delay fixes for months to see how well they work. He suggested that, maybe, developers could rank commits by both urgency and "scariness." But Greg Kroah-Hartman, who maintains several stable kernels at any given time, said this would not work; it's the "obviously correct" patches that turn out to be broken in the end.

Beyond that, Greg said, what often happens is that developers will revert a patch that turns out to be broken but forget to tell the stable maintainers about it. James Bottomley asked if it might help to avoid shipping stable updates during the merge window when, presumably, most of the buggy patches are merged. Greg replied that he had tried that and didn't see any difference in the results. Buggy patches can show up at any time.

Sasha complained that there is almost no testing of stable updates at all. He would like to work more closely with the distributors and get them to ship "proposed" updates for wider testing. When asked how many regressions he is talking about, he said there were one or two per release. Greg suggested that number was actually pretty good given the number of patches being shipped, but Sasha disagreed. When asked how many complaints he gets, he said one comes in every few weeks, usually for a problem that has already been fixed in the mainline.

Olof Johannson asked whether there should be -rc releases for the stable updates. Greg replied that, with a stable update coming out about once per week, there is simply no time for -rc releases.

Sasha then turned to a recent complaint of Greg's: there are too many stable kernels and too many stable maintainers out there. Greg said that we now have a situation where people don't know which stable kernel to use. Almost nobody uses more than one, and the benefit to the community as a whole is rather small. He said that Debian does benefit from the maintenance of the 3.2 kernel by Ben Hutchings, but that is the exception rather than the rule.

There was a question about whether the next stable release could be announced ahead of time, since that would allow distributors and other users to plan accordingly. Greg said that, in the past, such announcements have led to everybody trying to push crap into the mainline to get it into the stable release. It was suggested that there might be less pressure to do so now, given that most distributions do not use the long-term releases supported by Greg. The system-on-chip trees do use those releases, though.

How about making 4.4 the next stable kernel? It is too late at this point to queue up a bunch of half-baked code to go in, so that could be a relatively safe announcement to make. Announcing the stable release ahead of time could improve the predictability of the whole process. In the end Greg agreed; he has since announced that the next long-term stable kernel will, indeed, be 4.4.

That led to concerns about Greg maintaining too many kernels. It is about time to drop support for 3.10, though; evidently Willy Tarreau will be picking up extra-long-term maintenance of that kernel for those who want it. Some users, it was said, would really like to have 20-year support for their kernels, but volunteers for that task were scarce in the room.

Index entries for this article
Conference	Kernel Summit/2015

---

to post comments