A new stable security tree

The history of the stable kernel tree goes all the way back to some mailing list discussions in 2005. It has clearly been a great success that now spans multiple supported kernel versions, some of which are supported for two years as part of the long-term support initiative and some that are supported for far longer than that. But there have always been worries in some quarters that following the stable kernels—with fairly frequent updates including many patches—might introduce more bugs than would be fixed (at least bugs relevant to a particular user). So, a "security fix only" stable kernel has some appeal, which is exactly what Sasha Levin announced on April 11.

Levin described his "linux-stable security tree" (which is available as a Git tree) as "a derivative tree from the regular stable tree that would contain only commits that fix security vulnerabilities". The repository contains branches and -security tags for all stable series starting with 3.0 (though some of those are now unsupported), which gives an idea of what kinds of fixes he is targeting. Levin plans to support all of the maintained stable kernels with a -security variant. He went on to outline the reasons behind his plan:

One major problem, of course, is defining what constitutes a "security fix". Greg Kroah-Hartman, who is the founder of the stable trees and still maintains several of them, wondered how Levin would "define 'security vulnerabilities'". He noted that everyone's needs are different, so they should either do the cherry-picking themselves or simply rely on one of the stable trees: "It's not that much churn at all, given the % of the mainline tree". He also pointed out that there are fixes that have gone into the stable trees that weren't recognized as security flaws until long after those trees were released. Anyone who was cherry-picking fixes might well have missed one or more of those and been exposed to vulnerabilities for far longer than necessary.

Levin replied that the security tree would carry patches for "anything that qualified for a CVE plus anything exploitable by a local unprivileged user". He noted that the security trees would simply be "a subset of [its] corresponding -stable tree". But Kroah-Hartman expressed further concern about the goals of the project:

But there are plenty of patches that end up in the stable trees that actually introduce bugs, Levin said. He pointed to the number of "Fixes:" tags that refer to commits in the stable tree as an indicator of that. Furthermore:

There are also fixes for CVEs that have not made it into the stable tree, he said, typically because they were not sent to the stable maintainers when they were merged and a CVE was requested later once the security implications were known. While Levin did not say so directly, it would seem that the idea is that the maintainers of this new tree could help ensure those fixes get into the stable tree, since that provides the path for them to reach the stable security tree as well. But, he acknowledged that the tree would not be able to achieve "100% success rate, but this is the same story as the stable tree and the patch selection there".

Beyond that, though, Levin said that reviewing a few cherry-picked fixes, rather than the 100+ fixes all over the tree that often come with a stable release will help certain projects:

The stable kernels do tend to have a lot of patches. 4.5.1 had 238, the 4.4 series has had 1004 patches through 4.4.7, and the 3.14 series contains 4580 patches through 3.14.66. The trees in Levin's repository show markedly fewer patches.

Willy Tarreau, who had been maintaining the 2.6.32 stable kernel until its recent end of life, was worried that the use of "security" in the name of the tree will lead users astray:

Tarreau suggested an alternate strategy to create an "easily searchable" list of mainline commits that were security fixes. Those could perhaps be put into a tree: "We could even amend the commit messages to mention the CVE id once it's assigned, and sometimes 'relies on this preliminary patch'." But it would be "tough work" and difficult to find someone interested in doing it. In the end, though, he believes the underlying problem boils down to "more of an issue educating our users than an issue with the code we distribute".

Kroah-Hartman agreed with Tarreau about the education problem, but also complained that there are "users that just don't want to upgrade as it's 'hard' for them". There is no real difference in taking 200 patches in an update versus 100, he said, so the "small subset" argument doesn't make sense to him. But Levin had previously suggested that stable-security tree releases might carry on the order of five patches, so it's clear that the two have clashing visions of how it will all play out.

The one participant in the discussion who is strictly a user of the stable kernels (rather than a maintainer and user) was Eddie Chapman. He said that the quality of the stable kernels has always been excellent in his experience; he has run into regressions only rarely and uses the stable tree for "virtually all my own projects, as well as many clients' projects". He agreed with much of what Kroah-Hartman and Tarreau said in the thread and warned: "IMO you deserve everything you get if you only applied the fixes in Sasha's new tree and ignored the stable releases completely."

With that said, though, Chapman is in favor of the new tree. He recognizes that the line between a bug fix and a security fix is blurry, but any assistance in narrowing down which fixes are applied is welcome.

But Tarreau is skeptical that the tree will generally be used in the right way. The security fixes will only appear in it if they are already in the stable tree, which means that fixes that are missed by the stable maintainers (which is common for him, he said) won't be present.

Chapman agreed that there is a risk, but "I think it is better than no repository". Users who are blindly applying fixes are already in trouble, "so things can't really effectively get any worse", he said. Blindly applying a small set of fixes from the stable-security tree will perhaps help in a relatively small way: "Their situation might then be upgraded from 100% screwed to maybe only 70% screwed".

That's where the conversation trailed off, at least for now. In the end, the proof will be in the pudding. If Levin can provide a tree that is useful to some projects—and perhaps help get some additional fixes into the stable tree itself—it will be a success. If, on the other hand, it ends up causing more headaches than it solves, it seems likely that it will just go away at some point. Only time will tell.