Civil Infrastructure Platform to maintain 6.1 for 10 years

[Posted October 12, 2023 by corbet]

The Civil Infrastructure Platform project has announced that it will be maintaining the 6.1 kernel for a minimum of ten years past its initial release (and, thus, through 2032).

CIP kernels are maintained like regular long-term-stable (LTS) kernels, and developers of the CIP kernel are also involved in LTS kernel review and testing. While regular LTS kernels are moving back to 2 years maintenance, CIP kernels are set up for 10 years. In order to enable this extended lifetime, CIP kernels are scoped-down in actively supported kernel features and target architecture. At the same time, CIP kernels accept non-invasive backports from newer mainline kernels that enable new hardware.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 12, 2023 18:43 UTC (Thu) by florianfainelli (subscriber, #61952) [Link]

This is very welcome, thanks to everyone doing this!

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 12, 2023 19:43 UTC (Thu) by faramir (subscriber, #2327) [Link] (8 responses)

Ever since the Kernel LTS 2 year support limit announcement was made, I've been wondering what companies like Google, Redhat, Suse, and Canonical (Ubuntu) were going to do about their kernel support. Between Android and ChromeOS, Google is apparently planning on supporting Linux devices up to 10 years. I think Redhat is at least 5 years. Not sure about Suse. From what I can tell, Ubuntu is at least 5 years for their LTS releases. In the case of Ubuntu, I guess they could mandate using their HWE (Hardware Enablement) kernel updates which bump up the kernel version. Are the other companies already maintaining non-LTS kernel version for extended periods. If not, have any of them said what they are going to do?

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 12, 2023 20:28 UTC (Thu) by smoogen (subscriber, #97) [Link] (1 responses)

For at least 15 years, Red Hat has been maintaining kernels for various releases for at least 10 years. I believe SuSE and Canonical also do similar work for their kernels. Basically a kernel is chosen to be worked from and then over say 5 years, code is backported from various newer kernels to that older kernel. That code may be enablement of enterprise hardware or kernel features. After that 5 years, the work is limited to mainly security and major bug fixes found. This work ends up with what various kernel maintainers call at their nicest a 'frankenkernel'.

While the EL7 kernel says it is 3.10.0, the EL8 says its 4.18.0, and the EL9 says its 5.14.0, what the actual code is may be very different in places. I expect that something similar is done for the other distributions but on different timelines.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 11:08 UTC (Fri) by joib (subscriber, #8541) [Link]

I don't know about SuSE, but AFAIU Canonical doesn't do backports at nearly the scale that RedHat does. Though Canonical has at least for the past couple of LTS releases managed to use an LTS kernel (5.4 for 20.04 and 5.15 for 22.04), and AFAIU most of the patches come from the upstream kernels. No idea what their plan wrt to the upstream kernel support period dropping to 2 years is, will they continue backporting fixes themselves, or will they upgrade everyone to the latest LTS, or will they switch to these CIP SLTS kernels at some point.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 11:59 UTC (Fri) by DOT (subscriber, #58786) [Link] (5 responses)

What is the use case for these backport-heavy 'frankenkernel' forks? I assume that QA processes are in place to check for regressions on these kernels. Wouldn't the same QA processes work for checking mainline updates for regressions?

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 12:19 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (4 responses)

There are many customers that simply don't trust anything that changes the version number. They trust Red Hat to evaluate which subsystems to update and when.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 14:56 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (3 responses)

There are many customers that pushed the limits of their software stack and have complex piles of code deployed over a Linux system that work thanks to a lot of luck, gluetape, praying and individual IT specialist heroics.

Those customers admit they need some form of security update process but live in deep fear any behaviour change will cause the gluetaped software skyscraper to crumble. They are ready to pay good money to someone like Red Hat to gate changes and postpone as long as possible the need to deal with most kernel changes.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 15:02 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

And the frankenkernels serve both to gate the changes in a given kernel version (but LTS also does that), but also to delay the need to move to a newer kernel version by backporting the “good” bits while changing nothing else (typically the parts that have huge performance implications or are required to use new hardware once the original one is toast).

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 15:03 UTC (Fri) by smoogen (subscriber, #97) [Link] (1 responses)

And there is a heap of OEM software which only give support for specific versions of 'tools': kernel, glibc, compilers, etc. Updating any part of them with something not on the list will end up with the OEM saying 'so sorry your payroll is broken Go back to what we said we support.. and call again'. (Been on that call twice in my lifetime.. not a good place when you may have 10,000+ paychecks maybe not going because someone decided to move from OEM approved version to version+1) This tends to have a large knockon effect where maybe only 2-3 servers are running such OEM software, but as you said, they then start gluing more together and together not knowing when the structure will fail.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 16, 2023 10:56 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

That exactly.

Networked IT is very powerful, but it also means any problem in any point of the network of inter-linked apps will have huge effects (and it’s not a linear progression, the cost of mistakes grows much faster than the size of IT systems as a whole).

But the cost savings when everything is done right is also big enough there is a huge pressure to interlink everything.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 7:31 UTC (Fri) by Lionel_Debroux (subscriber, #30014) [Link] (5 responses)

I can't wait for civil infrastructure running a severely outdated kernel base to be taken down by an exploitation chain which uses at least one of the hundreds (*), if not several thousands after 10 years, of more or less severe vulnerabilities whose fixes have never been backported :)
AFAICS in https://gitlab.com/cip-project/cip-kernel/cip-kernel-config , the 3 known sources of streams of vulnerabilities I looked at are either enabled in modern versions (BPF_JIT, USER_NS though the real problem is with _unprivileged_ user namespaces) or not explicitly disabled (IO_URING).

*: the grsecurity patch for 4.4.162, which was published at the end of 2018, was in the hundreds of missing backports ballpark, and that was for a period much shorter than 10 years. Not all of the vulnerabilities were severe, but still. On 2019/01/01, I selected and sent 5 commit IDs to Greg KH, one of which had a message containing "oops"; I received a reply on 2019/11/21.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 9:13 UTC (Fri) by wtarreau (subscriber, #51152) [Link]

There's no "one-size-fits-all" unfortunately for options. Some don't need them and want them disabled, others absolutely need them. Regarding the delay you experienced in getting patches backported, it just highlights what has been said for many years now, that kernel maintainers need more manpower to help them do their work. I think your participation would be very welcome if you have some skills to help on a regular basis.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 12:16 UTC (Fri) by mcatanzaro (subscriber, #93033) [Link] (3 responses)

The dark secret behind LTS distributions is that only about 15% of security bugs fixed upstream receive CVEs, and only those security bugs can be tracked and fixed. My 15% number is fake because it's impossible to know how many security bugs receive CVEs, but whatever the real percentage is, it's sure to be very low. My personal rule of thumb is to request a CVE only for problems that seem particularly noteworthy.

The best way to stay safe is to regularly update to the latest upstream version every few years, i.e. don't use LTS distros for too long. I'd say 2-3 years is the longest time that I would be reasonably comfortable with personally. LTS distros do a good job of tracking and fixing CVEs, but they cannot save you from the unknown. Maintain reasonable expectations when you see outrageously long support commitments.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 12:45 UTC (Fri) by danielthompson (subscriber, #97243) [Link]

> only about 15% of security bugs fixed upstream receive CVEs, and only
> those security bugs can be tracked and fixed.

I'm dubious about "only [] security bugs [with a CVE] can be tracked and fixed". Sure, that will be true for some projects but it's certainly not the case for the LTS kernel releases (and is likewise unlikely to be only way to maintain CIP kernels either). For LTS kernels, it is git metadata that drives the backporting of bug fixes, not CVEs.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 15:11 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

> The best way to stay safe is to regularly update to the latest upstream version every few years, i.e. don't use LTS distros for too long. I'd say 2-3 years is the longest time that I would be reasonably comfortable with personally.

However that does not work when you live in software version support matrix hell, the people paying want to wait at least a year to shake out bugs before committing to any distro version, integrating all the bits that will be deployed over this distro version once the matrix is resolved is itself a long many-months process, and you can add some software procurement negotiations delays to the mix.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 16, 2023 9:04 UTC (Mon) by taladar (subscriber, #68407) [Link]

The other dark secret of LTS distributions is that an old version with backports with an unchanged version number is still a new, untested version that can have new bugs, break things or introduce new security holes.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 9:47 UTC (Fri) by arekm (guest, #4846) [Link] (1 responses)

Do they release publicly available tarballs or git only?

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 12:38 UTC (Fri) by wtarreau (subscriber, #51152) [Link]

It's on kernel.org so you have both:

https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux...

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 16:07 UTC (Fri) by mfuzzey (subscriber, #57966) [Link] (1 responses)

I can't seem to find (but maybe I'm not just looking hard enough or in the right place) what the rules and process are / will be for the CIP kernels.

For the normal LTS kernels it's described in Documentation/process/stable-kernel-rules.rst

In particular
* What are the criteria for an upstream patch to be back ported?
* What is the process for requesting a backport?
* Will things tagged "stable" in mainline commits get merged into both normal LTS and CIP?

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 13, 2023 17:37 UTC (Fri) by geert (subscriber, #98403) [Link]

I guess the process is the same as for v4.4
https://sched.co/1LcQW

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 14, 2023 16:59 UTC (Sat) by fratti (guest, #105722) [Link] (3 responses)

Looking forward to seeing this kernel used in 2037.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 14, 2023 19:51 UTC (Sat) by willy (subscriber, #9762) [Link] (2 responses)

That'd be a shame. Here's a benchmark which regresses from 3.1s to 3.9s with v6.1. It then improves to 0.9s with v6.6. https://lists.infradead.org/pipermail/maple-tree/2023-Oct...

The hardware of 2037 is likely to be even more multicore and multithreaded than the hardware of today. So these improvements are only going to be more significant in the future.

No, they're not in any meaningful sense backportable fixes. The fetish for maintaining ancient software for new hardware is harmful. The kernel doesn't (intentionally) break userspace compatibility. Upgrade your kernel every couple of years and be happy.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 14, 2023 21:06 UTC (Sat) by fratti (guest, #105722) [Link] (1 responses)

My reply was sarcastically pointing out how a product aimed at customers who don't want to upgrade will probably end up not being upgraded once it's past its "support" phase.

Civil Infrastructure Platform to maintain 6.1 for 10 years

Posted Oct 23, 2023 4:21 UTC (Mon) by nilsmeyer (guest, #122604) [Link]

That's my observation as well, organizations like that also don't really have the institutional muscle memory to perform upgrades, those in charge of the last upgrade often left the company (or even retired) and it's done so infrequently that there is no documentation and no process in place.