|
|
Log in / Subscribe / Register

Cramming features into LTS kernel releases

By Jonathan Corbet
October 10, 2017
While the 4.14 development cycle has not been the busiest ever (12,500 changesets merged as of this writing, slightly more than 4.13 at this stage of the cycle), it has been seen as a rougher experience than its predecessors. There are all kinds of reasons why one cycle might be smoother than another, but it is not unreasonable to wonder whether the fact that 4.14 is a long-term support (LTS) release has affected how this cycle has gone. Indeed, when he released 4.14-rc3, Linus Torvalds complained that this cycle was more painful than most, and suggested that the long-term support status may be a part of the problem. A couple of recent pulls into the mainline highlight the pressures that, increasingly, apply to LTS releases.

As was discussed in this article, the 4.14 kernel will include some changes to the kernel timer API aimed at making it more efficient, more like contemporary in-kernel APIs, and easier to harden. While API changes are normally confined to the merge window, this change was pulled into the mainline for the 4.14-rc3 release. The late merge has led to a small amount of grumbling in the community.

The problem isn't necessarily the addition of timer_setup() which, on its own, cannot really break anything. But that addition has been followed by a series of conversions to the new interfaces, which are being sent to the relevant maintainers. Accepting a timer_setup() conversion into a maintainer tree will only work if that tree has timer_setup() itself; that implies that the maintainer tree must be current with the mainline as recently as 4.14-rc3. Many subsystem maintainers branch from the mainline around -rc1 or -rc2, so they won't be able to apply the conversion patches unless they perform a separate merge first. The merge is not usually hard, but subsystem trees containing "back merges" with the mainline can run into trouble during the merge window, so maintainers have understandably become leery of them.

In this case, the grumbling is already done, and the conversion to the new timer API can be expected to be completed on schedule in 4.15. And, perhaps more to the point, those who want to backport a bunch of conversions to 4.14 (so as to have them in a long-term supported kernel that is likely to be shipped in many mobile devices) will have a much easier task of it. It was never explicitly said that 4.14, in particular, was an important target for this work, but it seems unlikely that it wasn't in developers' minds.

In another case, things were more explicit. Thomas Gleixner recently sent a pull request for a significant refactoring of the watchdog timer subsystem; it was a reworked version of a patch set that had been refused by Torvalds during the merge window. Part of the reasoning for requesting a pull this late in the development cycle was a desire to get the work into this release in particular:

As 4.14 is a long term stable kernel, I prefer to have working watchdog code in that and the lockdep issues resolved. I wouldn't ask you to pull if 4.14 wouldn't be a LTS kernel or if the solution would be easy to backport.

Stable kernel maintainer Greg Kroah-Hartman complained about that request: "This is exactly what I did _NOT_ want to ever see happen when I did the 'let's announce the LTS kernels ahead of time'". He suggested that perhaps future long-term support kernels will return to post-release announcements. That notwithstanding, Torvalds pulled the changes for the 4.14-rc4 without comment.

This work almost certainly will not break the 4.14 kernel; it was essentially ready during the merge window. But it does show that the LTS release is motivating pull requests that might have otherwise waited another cycle. That is not how things were supposed to work; part of the idea behind a nine-week release cycle was that, since the cost of missing one cycle was minimal, there would no longer be any great incentive to hurry code into any particular release. It is clear, though, that this incentive has not entirely gone away; indeed, it may be getting stronger.

For those who are inside the kernel community, one development cycle looks much like the next. But, for those making use of the kernel, all kernel releases are decidedly not equal. The release that they actually plan to ship is the one that they care about. There has been a determined effort to encourage the industry to ship the LTS kernels in the hope of improving the support for deployed kernels in general. This effort has seen some success, which is a positive change, but it does tend to focus even more attention on the LTS releases. That can only result in more pressure to get features into those releases.

In a sense, the situation vaguely resembles how things worked before 2.6 came out: major kernel releases were separated by years, so there was immense pressure to get features in before the deadline. As the LTS kernels become more widely used, they start to look like the major releases of old. The LTS releases are the ones that everybody wants to get their features into, and they only happen once each year. Missing an LTS release means waiting a year for a feature to make it into the next LTS release and, probably, maintaining it out-of-tree for products shipped in the meantime. It's not surprising that the idea of getting code into the mainline sooner, even if it requires fixing later, has some appeal, but there is a cost to doing things that way. As Kroah-Hartman said: "We've been down this path before, and it was not good".

That said, the kernel development community has changed considerably since the adoption of the short release cycle. Code is generally of a much higher quality at the time it is merged into the mainline. So if a bit more of it is jostling to get into 4.14, the result may be a more turbulent development cycle. It should not, however, replicate the situation of 15-20 years ago, where a "stable" kernel release would require another year to truly stabilize. We are probably not at risk of repeating the misery of the early 2.x years.

In the cases described here, the quality of the code being merged is not in question. It is really just a matter of the timing, and the discussion wound down quickly. But this topic can be expected to return. Neither the pressure to get changes into LTS releases nor human desire to game the system will go away, even if the pre-announcement of LTS releases comes to an end.

Index entries for this article
KernelDevelopment model/Stable tree


to post comments

Cramming features into LTS kernel releases

Posted Oct 10, 2017 22:25 UTC (Tue) by zblaxell (subscriber, #26385) [Link] (2 responses)

Pre-announcing LTS releases is definitely doing things in the wrong order. Part of the problem is that the result Greg KH was afraid of is not just an outlying possible outcome--it's the obvious move for rational self-interested actors. The other part of the problem is that conflating fitness-for-purpose with "LTS" status is a doomed attempt to predict the future of undesirable outcomes that cannot be avoided by engineering (e.g. bad stuff from latent security bugs to regrettable API decisions).

The best possible result of applying the LTS tag to a baseline kernel up to the moment of its release should be roughly the same as choosing randomly. If mainline contributors behave differently for LTS releases then the result can be worse than random selection. This is what lead to the demise of the "2.x | x is even == stable" pattern--the incentives aren't aligned the right way, so the LTS and mainline kernels effectively forked, benefiting neither. It might seem OK now, but we are probing the envelope of failure here.

The main downstream benefit of LTS kernels starts after they're released (usually 2-4 mainline kernel releases after). The extra developer attention focused exclusively on fixing bugs instead of rewriting APIs, merging features, or non-bug-fixing activity of any kind eventually makes LTS kernels more fit than any others for long-term use. It doesn't matter which baseline kernel that attention is focused on--the initial conditions are similar enough that the extra effort and isolation from destabilizing changes are more important than any other factors.

We maintain and test all of the mainline kernel versions between LTS releases. To us the LTS kernels are curated collections of patches for the non-LTS kernels. We keep data on which kernel versions are implicated in test failures, and use that as our fitness metric. The LTS kernels initially do quite poorly, then pull ahead starting from somewhere between release .20 and .40, and do well thereafter except for occasional one-off regressions; however, it's not always enough. The best kernel in any given year is usually the kernel that happened to have the fewest bugs visible to our test cases at the time of its release, and it's more often not LTS simply because most mainline kernels are not LTS.

Ideally the LTS release announcements would be post-announced--start with the most fit kernel release of the year, and retroactively declare it "LTS", possibly maintaining stable patches for all of the previous 5 mainline non-LTS kernel releases until a clear winner emerges. That would bias the randomness toward whatever the fitness test was without creating disruptive behavior incentives among kernel contributors.

Cramming features into LTS kernel releases

Posted Oct 11, 2017 3:07 UTC (Wed) by rahvin (guest, #16953) [Link]

Personally I think LTS releases should be based on two things. Kernels the distributions are planning on shipping first and which kernels are showing the fewest amount of bugs second. Most distributions will ship the kernel long before it gets the LTS tags, some won't but it shouldn't be that hard to consult the distributions and find out which kernels they are targeting. In fact I'd argue it would be advisable for the distribution decisions makers should be meeting (maybe call it the Kernel LTS advisory board with 1 member per distribution at least covering the main distributions), that would be voting and providing recommendations to Greg on which kernels they'd like to see get the LTS support, though it should be his decision alone. This would help the distributions align their LTS releases with the kernels.

This is particularly important to distributions that move slower than the rest and have fewer iterations like Debian. And though discussions and recommendations should all be public, the actual decision shouldn't be public until the decision has already been made so you don't get the self serving behavior such decisions would elicit if people and developers are aware of the decision before the kernel is released.

I'm not a developer so my idea is probably dumb but it's out there so do with it what you will.

Cramming features into LTS kernel releases

Posted Oct 16, 2017 22:19 UTC (Mon) by tbird20d (subscriber, #1901) [Link]

Part of the reason Greg pushed up the announcement was that doing it post-release was not soon enough to get some major ecosystem players onto it. By then, various parties (including Google, with Android), had already selected their next major kernel, and had begun working on it. Whether the version they were using then got selected as an LTS kernel was then a crapshoot. I think the feature stuffing is something to watch out for, but that we shouldn't judge the results by one release. As you have pointed out, there is a real value to getting as many ecosystem players as possible using the same kernel (within a given time period), in terms of long-term stability and quality of that kernel.

In Google's case, this isn't a case of wanting to stuff the release, but of having enough time for their downstream semiconductor vendors to port their monstrous patch loads onto the next kernel (which I'm not defending, by the way - just saying what the current reality is).

Cramming features into LTS kernel releases

Posted Oct 10, 2017 23:20 UTC (Tue) by xnox (subscriber, #63320) [Link] (3 responses)

Should linux kernel have a two step process for LTS? E.g. all new features, subsystems, syscalls, APIs must land in v4.13. And then v4.14 is only drivers update and bugfixes. And once v4.14 is released, it becomes LTS? Or is this what greg did this time around, and actually v4.15 will be LTS one? =)))

Cramming features into LTS kernel releases

Posted Oct 11, 2017 14:57 UTC (Wed) by tlamp (subscriber, #108540) [Link] (2 responses)

> Should linux kernel have a two step process for LTS? E.g. all new features, subsystems, syscalls, APIs must land in v4.13. And then v4.14 is only drivers update and bugfixes.

As the article itself says, that's resembles a lot how the "before 2.6" tactics were, and they were abandoned for good:

> In a sense, the situation vaguely resembles how things worked before 2.6 came out: major kernel releases were separated by years, so there was immense pressure to get features in before the deadline. As the LTS kernels become more widely used, they start to look like the major releases of old. The LTS releases are the ones that everybody wants to get their features into, and they only happen once each year.

With your proposal 4.13 would be the "cram all into that" and 4.14 the make it somewhat work release.

Its much better to not do that and make each release the same in steady intervals so that's not a big deal if one release got missed as the next is really soon.
That encourages that things only get in once really (seemingly) ready.

As Greg KH already mentioned in his mail[1], he will probably just abandon the pre-release LTS announcements .
But as most people can probably pin down the next LTS release to two future releases as, looking at the near past, they happen roughly every 5 to 6 releases even that won't completely abandon "LTS feature cramming".
That maybe can be solved by maintainers and Linus just rejecting pulls that do to much to late.

[1] https://lwn.net/Articles/735900/

Cramming features into LTS kernel releases

Posted Oct 11, 2017 20:53 UTC (Wed) by zblaxell (subscriber, #26385) [Link] (1 responses)

> But as most people can probably pin down the next LTS release to two future releases as, looking at the near past, they happen roughly every 5 to 6
releases even that won't completely abandon "LTS feature cramming".

The uncertainty changes the incentives around cramming. Cramming provides an advantage only with a high degree of certainty that a specific future release is special; otherwise, features would just be merged normally, with no incentive to avoid delay.

Labelling with LTS after the release gives the LTS maintainer the additional option of acting directly against anticipatory cramming behavior. If contributors anticipate the LTS release then they can be defeated by choosing an LTS release on either side of the anticipated releases, especially if there is evidence of cramming in the most likely candidates. This is simply the scaled-up version of -rc8 mainline releases for LTS.

The LTS maintainer can also perturb anticipatory crammers by occasionally retroactively blessing a past kernel release as LTS. "OK, because of all the crap that got crammed in v4.14-rc6, I'm making v4.13 the next LTS kernel instead of v4.14" would eliminate the incentives in this case (and 4.13 still gets -stable patches, so it wouldn't break -stable patch continuity). One release look-behind would likely be enough to deter cramming--turning the clock back too far has problems of its own.

If the uncertainty window is three releases wide, then the only kernels that wouldn't be LTS candidates would be the two immediately following an LTS release--and in those cases there is plenty of time to prepare for the next LTS.

Cramming features into LTS kernel releases

Posted Oct 12, 2017 8:47 UTC (Thu) by tlamp (subscriber, #108540) [Link]

> Labelling with LTS after the release gives the LTS maintainer the additional option of acting directly against anticipatory cramming behavior. If contributors anticipate the LTS release then they can be defeated by choosing an LTS release on either side of the anticipated releases, especially if there is evidence of cramming in the most likely candidates. This is simply the scaled-up version of -rc8 mainline releases for LTS.

But then you still get the cramming and one (or two) kernel versions which are biased to be less stable, i.e. more crammed.
That goes in the direction of the old days, which has proven to be simply not good - although it would be surely better than what happened to 4.14 ...

> The LTS maintainer can also perturb anticipatory crammers by occasionally retroactively blessing a past kernel release as LTS. "OK, because of all the crap that got crammed in v4.14-rc6, I'm making v4.13 the next LTS kernel instead of v4.14" would eliminate the incentives in this case (and 4.13 still gets -stable patches, so it wouldn't break -stable patch continuity). One release look-behind would likely be enough to deter cramming--turning the clock back too far has problems of its own.

This sound like a good solution to me.

Cramming features into LTS kernel releases

Posted Oct 11, 2017 8:23 UTC (Wed) by mjthayer (guest, #39183) [Link] (1 responses)

Perhaps it would make sense, rather than officially declaring LTS releases, to work on persuading people or (especially) companies to do their maintenance of whatever kernel version they settle for upstream, in collaboration with anyone else who is using that kernel version. Any kernel version would then be as long term as anyone needed it to be, and people like Greg could put their energy into helping others do the maintenance rather than doing it themselves.

Cramming features into LTS kernel releases

Posted Oct 11, 2017 10:46 UTC (Wed) by dunlapg (guest, #57764) [Link]

Perhaps it would make sense, rather than officially declaring LTS releases, to work on persuading people or (especially) companies to do their maintenance of whatever kernel version they settle for upstream, in collaboration with anyone else who is using that kernel version.

Right, but in the absence of LTS tags, which version would you "settle for"? Probably one which the most other distributions are using or planning to use. The more other people are using a given release, the higher the chance it is that someone else will find bugs and fix them before you have to. The LTS tag is just a sort of way of coordinating that effort up front.

Code development isn't the only activity where the more people contribute the better the aggregate result is. Maintenance (consisting of bug-fixing and security support) has the same property that the more people contribute to it by "scratching their own itch", the lower each person's cost will be.

Cramming features into LTS kernel releases

Posted Oct 11, 2017 13:45 UTC (Wed) by epa (subscriber, #39769) [Link]

Isn't Greg K-H giving a hostage to fortune by announcing, ahead of time, "I will maintain release x.yz for long term support, no matter what junk it contains"?

He might do better to say "I am planning to maintain 4.14 as a long-term release, but if it gets lots of new disruptive stuff crammed in at the last minute, that may alter my decision; in that case I might pick one of the 4.14-rc releases and base a long term series off that."

Cramming features into LTS kernel releases

Posted Oct 11, 2017 13:51 UTC (Wed) by ballombe (subscriber, #9523) [Link]

The same thing happens in Debian each time that freeze date is announced in advance. Debian maintainers scramble to update to the latest upstream just before the freeze. (not that they were slacking before, but they had to wait for new upstream to be released). This lead to more up-to-date but less tested packages.

Cramming features into LTS kernel releases

Posted Oct 12, 2017 16:57 UTC (Thu) by flussence (guest, #85566) [Link]

LTS sounds like it's turning out to be the new -rc6.

Cramming features into LTS kernel releases

Posted Oct 21, 2017 8:59 UTC (Sat) by roblucid (guest, #48964) [Link]

The article states that most people are not affected in practice by any pressures to include new features, as by the time an LTS tagged kernel reaches the masses it has been stabilised somewhere between .20 & .40.

Yet there is a considerable long term benefit, in the features making the cut, in past distros with the LTS idea, have had to exclude whole desktop environments because they need to avoid shipping code planned to be obsolete within say 2 years.

Is this quality problem, not then an annoyance for developers and those living on the bleeding edge? Is it actually harming their productivity significantly? It looks a little like a storm in a teacup to me, some releases will always need more work than others before they're really solid.


Copyright © 2017, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds