|
|
Log in / Subscribe / Register

White paper: Vendor Kernels, Bugs and Stability

[Posted May 17, 2024 by corbet]

Ronnie Sahlberg, Jonathan Maple, and Jeremy Allison of CiQ have published a white paper looking at the security-relevant bug fixes applied (or not applied) to the RHEL 8.x kernel over time.

This means that over time, the security of the RHEL kernels get worse and worse as more issues are discovered in the upstream code and are potentially exploitable but fewer and fewer of the fixes for these known bugs are back-ported into RHEL kernels.

After reaching RHEL 8.7, the theory is that the kernel has been stabilized, with a corresponding improvement in security. However we still have an influx of newly discovered bugs in the upstream kernel affecting RHEL 8.7 that are not addressed. Each minor version of upstream is released on an approximately quarterly basis and we can see that the influx of new bugs that are unaddressed in RHEL is growing. The number of known issues in these kernels increases by approximately 250 new bugs per quarter or more.


to post comments

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 13:37 UTC (Fri) by bluca (subscriber, #118303) [Link] (69 responses)

Literally this morning Daan found out yet another occasion of a new kernel version intentionally breaking userspace compatibility:

https://fosstodon.org/@bluca/112455500079789967

Any OS booting with any released version of systemd and expecting it to mount a btrfs filesystem is now broken with kernel 6.8+.

But why, oh why do people use distro kernels and do not want to upgrade? Guess we'll never know.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 13:59 UTC (Fri) by snajpa (subscriber, #73467) [Link]

one reason I see is that there are ecosystem components which don't always keep up with upstream kernel development; I always get my a** kicked by kpatch-build, for example :D

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 14:16 UTC (Fri) by Bigos (subscriber, #96807) [Link] (45 responses)

Why is systemd using deprecated mount options? I can understand "an old systemd will no longer work with a new kernel", but it sounds like *no* systemd will actually work from your message.

Has the information about the deprecation not been advertised enough? It has been 2 years apparently.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 14:22 UTC (Fri) by bluca (subscriber, #118303) [Link] (1 responses)

It hasn't at all. Nobody apart from kernel developers follow the LKML. At the very minimum, searching for affected code on Github or so and filing bugs would be expected.

If it doesn't happen, and userspace compatibility is broken without regards, then it's really not surprising that people do not want to upgrade their kernels. And the "we do not break userspace" mantra that we always hear some kernel maintainers repeat is, at the very least, inaccurate.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 16:57 UTC (Fri) by Tobu (subscriber, #24111) [Link]

Looking at the commit, the removed options had been showing kernel warnings for the deprecation period. In the rest of the file, usebackuproot and nologreplay are on the way out as well.

It looks like these were intended as rescue options, whereas what systemd wants is something that will make sure images and block devices stay immutable. Wasn't there work to enforce that at the block layer recently?

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 17:00 UTC (Fri) by mezcalero (subscriber, #45103) [Link] (42 responses)

A deprecation that you only know about if you follow lkml or one of its auxiliary mls is not a useful deprecation.

Also why in heaven even deprecate this at all? The functionality is still available and supported after all, they just renamed the option. And what's worse, the other big relevant file systems have an option of the same name. Doing the same stuff. Only btrfs is the sole outlier now. Wtf.

Sorry, but you don't get to change kernel interfaces like this and then claim you are a compatibility extremist like the kernel folks "we dont break userspace" mantra suggests.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 18:07 UTC (Fri) by hmh (subscriber, #3838) [Link] (41 responses)

> A deprecation that you only know about if you follow lkml or one of its auxiliary mls is not a useful deprecation.

Agreed, and the patch authors agreed too, apparently. The kernel has been [trying to?] warn everyone about it since 5.11. The patch that removed the functionality, added a call to btrfs_warn() in its place.

So, what was removed in 6.8(?) was the btrfs_warn() and option handling. Which is obviously a bug, since it caused an userspace regression, and I fully expect it will be reverted everywhere now that it has been reported as such...

But "announced only though LKML" (which I extend to "source-code-change-flow", i.e. other MLs as well as the git commit log) is not correct. The code clearly attempted to get out a warning to the kernel log, every time anyone tried to mount a btrfs partition with said flag, and done that since 5.11 until the change that caused the current issue.

I wonder if btrs_warn() is being "silenced" by default somewhere? But to an outsider, it looks like whatever the reason, a better way of doing this is needed.

Documentation/ABI/ comes to mind, it is used for sysfs and configfs a lot, maybe it could be used for mount options and filesystem functionality, too...

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 18:48 UTC (Fri) by bluca (subscriber, #118303) [Link] (33 responses)

Warnings that _only_ users actually running BTRFS, and only those who actually look for them, are not really enough. I mean we do have a CI running BTRFS, but it logs twenty million things per run, so who would notice such a warning?

And besides, the main point is a different one: if there are users of a userspace API, do not remove it. Or do, but then stop claiming "we do not break userspace", and don't publish papers wondering why users don't trust new upstream kernel releases and just stay on enterprise stable kernels.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 3:59 UTC (Sun) by wtarreau (subscriber, #51152) [Link] (32 responses)

> I mean we do have a CI running BTRFS, but it logs twenty million things per run, so who would notice such a warning?

So that may be what needs fixing in the first place. If new kernel warnings are not detected by the CI, it's for sure aiming at unnoticeable breakage upon upgrades.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 10:56 UTC (Sun) by bluca (subscriber, #118303) [Link] (20 responses)

Why would it? It's not a kernel CI, we barely have time and resources to test our own stuff, we are certainly not going to do QA for the kernel

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 11:17 UTC (Sun) by wtarreau (subscriber, #51152) [Link] (19 responses)

It's not about testing the kernel, it's about testing that what you're relying on still works. That's the same for any project, we all depend on lower layers and the tests are expected to catch failures. If you know you're relying only on abstraction layers, you probably don't care, but when you start dealing with syscalls or mount options yourself, you know that the risk becomes higher to face some changes over time. I know it's difficult and such changes are rare enough that even having tests doesn't warrant that this or that change will be caught. But probably that kernel warnings for an init system are worth being at least logged for occasional inspection.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 13:18 UTC (Sun) by bluca (subscriber, #118303) [Link] (18 responses)

That is only the case if "we do not break userspace" is false advertisement, and every userspace interface provided by the kernel is subject to breakages and incompatible changes. Is it?

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 13:59 UTC (Sun) by mb (subscriber, #50428) [Link] (16 responses)

There are different kinds of interfaces.
There are interfaces used by normal programs and there are special interfaces used by special programs like systemd and udev.

Normal interfaces are changed extremely rarely and these obviously are the ones meant by the "do not break userspace" rule.

Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
Every other decades old application will continue to work. That is what counts.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 14:11 UTC (Sun) by bluca (subscriber, #118303) [Link] (15 responses)

Sorry, but this is nonsense. Apart from the fact that a mount option, that can be used by anything or anyone with a simple mount command, seems hardly any "special", either the userspace interface is stable and breaking it is bad, or not. Kernel maintainers say "we do not break userspace", they don't say "we do not break userspace, apart from programs X, Y and X - because fuck those people". This is something that you have just made up post-facto to retroactivly justify an obvious and clear regression.

Which by the way, neatly explains why vendor kernels are needed and are in fact the only sane choice, despite what the paper cited in this article says. Nobody should run production payloads on upstream kernels at this point, given basic stuff like mount options just breaks left and right.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 15:20 UTC (Sun) by mb (subscriber, #50428) [Link] (14 responses)

> Apart from the fact that a mount option, that can be used by anything or anyone with a simple mount
> command, seems hardly any "special"

I didn't say that it is. I did not talk about this specific thing, because I don't know anything about it. I was talking about "do not break userspace" in the general form, not in this specific case.

Whether this mount change is a sane change is up to somebody else to judge.

> they don't say "we do not break userspace, apart from

Well, they pretty much do exactly that.
Sometimes they actually spell out what "apart from" means.
For example trace points are an exception. There are more exceptions.

If you want 100% full "don't break userspace" without exceptions, we must basically stop all kernel development now.
Every change is user visible eventually. Even simple changes like adding a new syscall can break programs, if the program was using the new syscall number and depended on it returning ENOSYS.

Having a "don't break userspace without exceptions" is impossible.

> This is something that you have just made up

No. See my example of trace points.

> this is nonsense
> because fuck those people

I think it would be good to calm down before continuing the discussion.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 17:32 UTC (Sun) by bluca (subscriber, #118303) [Link] (13 responses)

> I didn't say that it is. I did not talk about this specific thing, because I don't know anything about it. I was talking about "do not break userspace" in the general form, not in this specific case.

This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.

> Well, they pretty much do exactly that.

No, they very much do not. Look at all the enthusiastic comments from kernel people pointing to the paper in the article and saying "See? Vendor kernels are BAD, just upgrade to upstream kernels, it's fine really", and when told that new kernel version break applications and that's the real reason why vendor kernels are used, they shrug it away as "impossible, we do not break userspace"

> No. See my example of trace points.

Yes, it is exactly what you did, and there was no mention anywhere of trace points:

> Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
> Every other decades old application will continue to work. That is what counts.

You have made up a new rule according to which it's fine to break systemd or udev (if it's not made up, then just point to where on https://kernel.org/doc/ it is defined), but *unspecified other applications* must continue to work. That is very convenient of course, it's always unspecified other applications that are supported, and the ones that break are never actually supported. That's a very easy way of guaranteeing compatibility - every time something goes wrong just say that case was never actually supposed to continue working and move on.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:10 UTC (Sun) by mb (subscriber, #50428) [Link] (11 responses)

>You have made up a new rule

There have always been exceptions and I didn't make that up. That's just silly. I even gave you an example (tracepoints).
It's up to you to ignore that. But please stop saying that I made it up.

I respect you for what you do for Linux, Systemd and so on. But you're acting like a child right now.

>it's always unspecified other applications that are supported

Yes. That is exactly like it is.

I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.
Yes, that is unfortunate and could certainly be improved.
But please don't generalize to other applications.

>every time something goes wrong just say that case was never actually supposed to continue working and move on.

That's not how things are done, though.
There have been reverts of ABI changes due to application breakages in the past.
It's done on a case by case basis.

Now you will reply: You have made up yet another rule!

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:32 UTC (Sun) by bluca (subscriber, #118303) [Link] (10 responses)

> There have always been exceptions and I didn't make that up. That's just silly. I even gave you an example (tracepoints).
It's up to you to ignore that. But please stop saying that I made it up.

Literally nobody has mentioned tracepoints. I mean I'm not even sure that really qualifies as a userspace interface - maybe it does, it would seem strange, but I am not a tracing experts. But it is completely unrelated to mount options being removed.

> I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.

Says who? That is very much not true. Every interface that I can think of is used by multiple unrelated applications. I have no idea where you get this from. Cgroups and namespaces? Throw a rock in the general direction of a container runtime and you'll hit either or both. Netlink? There are as many network and interface managers as there are Linux vendors. Process management? That's been around since literally forever, and see the point about container management again. Mounting filesystems? fstab is older than me, I am quite sure.

'We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init' doesn't sound as catchy, does it now?

> It's done on a case by case basis.

I am well aware. And the triaging of that case by case goes like this: did it affect the machine that Linus happened to boot on that week? If so, it gets reverted and unpleasant emails are shot left and right. Else, nothing to see, move along.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:48 UTC (Sun) by mb (subscriber, #50428) [Link] (9 responses)

> Literally nobody has mentioned tracepoints.

What the? I did. I mentioned them as an example for a non-stable interface. After you have asked.

> I mean I'm not even sure that really qualifies as a userspace interface

Oh. I get it. *You* want to define what a userspace interface is and what not.
And everybody who disagrees is "making it up" or talking "nonsense".

That is silly.

> Says who?

Me. But I'm not sure why that matters.

> We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init

Well. I have never experienced a breakage due to a kernel interface change.
I run a two decades old binary and it still works fine.

That is my experience.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 19:31 UTC (Sun) by bluca (subscriber, #118303) [Link] (8 responses)

> I mentioned them as an example for a non-stable interface. After you have asked.

Again, I do not know the first thing about tracepoints and have zero interest in that. Maybe it's a supported interface, maybe it's not, I really cannot say, nor care, and can't see what it has to do with mount options.

> *You* want to define what a userspace interface is and what not.

No, userspace defines what is a userspace interface, as per Hyrum's Law.

> Me. But I'm not sure why that matters.

Because it's just wrong, as explained, there are no "special custom interfaces" being used anywhere, just bog standard stuff used by most components of an operating system.

> I run a two decades old binary and it still works fine.

'We do not break userspace, as long as userspace is mb's statically linked printf("hello world\n") /sbin/init' still not quite as catchy I'm afraid

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 20:23 UTC (Sun) by mb (subscriber, #50428) [Link] (7 responses)

Ok. Everybody is wrong, except you. I got it.
I'll stop here.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:30 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (2 responses)

I mean. The not breaking userspace only applies to a subset of things userspace can do.

I've had to fix software because of a kernel update, because some files in /sys were moved. But for some reason that doesn't count.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:41 UTC (Mon) by mb (subscriber, #50428) [Link]

> I mean. The not breaking userspace only applies to a subset of things userspace can do.

That is exactly what I was saying. Yet, I'm apparently wrong.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:45 UTC (Mon) by bluca (subscriber, #118303) [Link]

> I mean. The not breaking userspace only applies to a subset of things userspace can do.

Where is that subset defined?

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:09 UTC (Mon) by wtarreau (subscriber, #51152) [Link]

> Ok. Everybody is wrong, except you. I got it.
> I'll stop here.

Welcome to discussions with bluca. Agressivity, half-reading of arguments, and accusations often arrive in the second or third message when he disagrees with you. There are such people who constantly criticize Linux and who would probably do good to the community by switching to another OS of choice :-/

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:26 UTC (Mon) by bluca (subscriber, #118303) [Link] (2 responses)

You are literally trying to gaslight me into believing that dropping a mount option (not just making a noop, literally deleting it so that a hard error is returned where it wasn't previously), that is currently in use by userspace, is not a compatibility breakage. Make of that what you will.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:54 UTC (Mon) by mb (subscriber, #50428) [Link] (1 responses)

>You are literally trying to gaslight me

Wow. This is a new level.

Stop here please

Posted May 20, 2024 12:57 UTC (Mon) by corbet (editor, #1) [Link]

This clearly is not going anywhere useful, can we all let it go at this point, please?

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 15:48 UTC (Thu) by anton (subscriber, #25547) [Link]

This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.
Is it? When the breakage of existing code is reported as a bug, do the kernel developers declare the bug report as invalid, or do they fix the bug? If it's the latter, they live up to the principle. Sure, one might wish that such bugs would never happen, but apparently they feel that that going for that would be too constricting for kernel development.

Whether that means that vendor kernels are needed, or that one can use upstream kernels if one is selective about them is up to the vendors and their customers to decide.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:28 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

Well the stuff in /sys moves around a lot.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 10:12 UTC (Thu) by tlamp (subscriber, #108540) [Link] (10 responses)

IMO what actually needs fixing is how deprecated options, maybe even drivers, are communicated and tracked.

A major improvement here could consist of adding a common infrastructure in the kernel to track deprecation.
It should allow the kernel build system to generating a declarative list (or something more structured like JSON) that includes info like "driver/module", "option" name, "kernel release it got deprecated", and "kernel release where removal is planned".

This data should be assembled on kernel build, possibly even made available on runtime in one of the virtual FS, would allow distros and projects with a lot of kernel interaction like systemd to actually track those and notice those for sure, as scanning for arbitrary warnings that can change wording every point release is just an ugly mess with lots of false-positives/negatives waiting to happen.

If it was available on runtime then checks could be added to the pre- / post-installation scripts/hooks of the kernel distro packages so that users can get a much more noticeable warning printed out on upgrade if their system is affected by such an option removal.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 12:35 UTC (Thu) by mb (subscriber, #50428) [Link] (6 responses)

> A major improvement here could consist of adding a common infrastructure in the kernel to track deprecation.

We had such a deprecation list under Documentation, but I think it got removed a couple of years ago.
It was not very useful and suffered from major bitrot.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 13:20 UTC (Thu) by Wol (subscriber, #4433) [Link] (5 responses)

> We had such a deprecation list under Documentation, but I think it got removed a couple of years ago.
It was not very useful and suffered from major bitrot.

Far better to do it in the kernel itself. Probably not easy, but move all deprecated stuff into a (or several) modules behind an option "deprecated-6.8" or whatever. Bleeding edge sets all these to "no", and either someone steps up and supports it (removing the deprecated option), or it bitrots until someone says "oh, this broke ages ago, let's delete it".

And then, if there's stuff you really want to get rid off but people need it, every year or so it gets upgraded to "deprecated latest kernel", so hopefully people stop using it and it finally drops out of sight ...

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 14:45 UTC (Thu) by mb (subscriber, #50428) [Link] (4 responses)

>but move all deprecated stuff into a (or several) modules behind an option "deprecated-6.8" or whatever.

It would change nothing.
Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
Such options are useless.

>let's delete it

And that is exactly when people will first start to notice.
And there is not much anybody can do about that *except* to not break/deprecate stuff.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 20:56 UTC (Thu) by Wol (subscriber, #4433) [Link] (2 responses)

> It would change nothing.

Except it changes everything

> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.

You just said it!

The distributions are enabling something that is disabled by default? They're accepting responsibility for keeping it working.

Developers are enabling something that is disabled by default? They're accepting the associated risks.

People are enabling something that is marked "deprecated"? They're being placed on notice that it's being left to bit-rot.

The fact that people have to actively enable something that developers clearly don't want activated means that anybody using it will have three choices - migrate their code away, take over maintenance, or do an ostrich and bury their heads in the sand. Users will still be able to be complain "I didn't know", but their upstream won't have that excuse.

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 21:01 UTC (Thu) by mb (subscriber, #50428) [Link] (1 responses)

>Developers are enabling something that is disabled by default? They're accepting the associated risks.

Do you realize, that most kernel options are disabled by default?

>The fact that people have to actively enable something that developers clearly don't want activated means

It means that developers don't have a clue what people (users!) actually want and need.

Closing your eyes won't make the demand go away, unless you are less than three years old.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 22:41 UTC (Thu) by Wol (subscriber, #4433) [Link]

> Do you realize, that most kernel options are disabled by default?

And how many of those options have "deprecated" in their name? Surely that's a massive red flag.

> It means that developers don't have a clue what people (users!) actually want and need.

And how many developers are employed by (therefore are) users? I believe Alphabet employs loads. Meta employs loads. Most of the kernel developers I have contact with are employed by large end users. It's a little difficult to be oblivious of your own needs. (Some people manage, I'm sure ...)

How difficult is it - to set a "not enabled" flag that cannot be accessed without some sort of warning that this flag will enable deprecated functionality Surely it's not beyond the wit of your typical kernel developer? That's ALL that's required.

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 24, 2024 8:15 UTC (Fri) by tlamp (subscriber, #108540) [Link]

> It would change nothing.
> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
> Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
> Such options are useless.

I don't think so, mostly because my spitballed proposal was not targeted at solving the "distros get never hurt by deprecation", as IMO that cannot be solved, besides not doing any deprecation at all anymore which hardly is a good solution. Rather, I wanted to target the "how things get communicated and noticed" part and having ab extra compile option with something like "deprecated-6.8-removal-6.12" in the name could actually be quite good for that. The build configs are often tracked and even diffed, and as simple single file can be easily grep'ed against _DEPRECATION_ and then diffed for ones that would trigger soon or new ones, probably even by a CI like systemd uses.

I.e., the status quo is having warnings for deprecation, which can be brittle and are not easy to digest/parse, having that info in an easier to digest manner would help a lot as tools/distros that depend on such options can easily find out when a used one will vanish soon(ish), then they also have no excuse about being unprepared.

> And that is exactly when people will first start to notice.

If their distro or tooling did not do their work then yes, but it wouldn't be the fault of the kernel having a messy deprecation process. IME most bigger distros or big projects like systemd want to avoid that, so if they'd have the definitive information required to do so in just somewhat digestible way, then I really think that most would actually act on that.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 14:57 UTC (Thu) by smurf (subscriber, #17840) [Link] (2 responses)

I don't need a static declarative list of all deprecations that might or might not exist in my kernel.

I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).

A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.

No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

White paper: Vendor Kernels, Bugs and Stability

Posted May 24, 2024 8:01 UTC (Fri) by tlamp (subscriber, #108540) [Link]

> I don't need a static declarative list of all deprecations that might or might not exist in my kernel.

As said, I'd assemble on build so those options that are not relevant for a kernel build config would not be in there (or if still wanted could be tracked differently, i.e., with an extra flag or separate list)

> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).

With a declarative list this is trivial to create, as a tool can just scan all modules, mount, ... options and compare if anything explicitly set is in the static list. So if you want this then a static list is IMO really the best way to achieve it, one first needs the definitive list of information before being actually able to do something with it, keeping it dumb on the kernel side and include as much as possible allows (user space or build) tooling to actually do the smart checks.

> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.

Not sure how this minus bikeshedding is any different what I proposed, but I like that we agree in general.

> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

I'd named JSON simply as an option, explicitly also stated that a simple list could do. But, I named JSON as 1. generating it is trivial (compared to parsing, which isn't hard either, but not trivial anymore) 2. Allows more flexible extension for whatever data or use case gets relevant in the future without having to do a /proc/deprecation2 3. in my projects I try to avoid another not-invented-here format with it subtleties to be added, but sure if it's a simple CSV list that gets generated by the common infra (i.e., not under the control of each kernel dev with their own opinions of the day) then fine by me (not that my acknowledgment would matter anything :)

White paper: Vendor Kernels, Bugs and Stability

Posted May 24, 2024 13:45 UTC (Fri) by donald.buczek (subscriber, #112892) [Link]

> I don't need a static declarative list of all deprecations that might or might not exist in my kernel.
>
> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).
>
> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.
>
> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

This would be perfect! We would see, what we need to address in our fleet (we are not using a distribution). But distributions would have something to build on, too. They might create a feedback path of this information from the systems of their users to the distribution. The basis for everything is that the information "you are using a mechanism, which will go away" is made available in a structured way.

It is important that the information cannot only be found by digging through masses of unstructured text in mailing lists, documentation, NEWS files, dmesg or other sources and then having to analyze in each individual case whether it is is relevant to you at all.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 14:01 UTC (Thu) by eru (subscriber, #2753) [Link] (6 responses)

The kernel has been [trying to?] warn everyone about it since 5.11.

These days just about every distribution "helpfully" hides boot messages behind a splash screen, so few users will ever see such warnings. Might as well not be there.

It is a mystery to me why this is done (and after I found out how, I disabled it). The stream of messages is actually a nice progress indicator, and if they get stuck at some point, one gets some idea about what might be wrong.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 14:49 UTC (Thu) by mb (subscriber, #50428) [Link] (2 responses)

>It is a mystery to me why this is done

Because nobody can read over 9000 messages in 0.5 seconds.

>The stream of messages is actually a nice progress indicator

No, it isn't. Today's boot process is so fast (unless you're stuck with a legacy init system), that console output is pretty much useless and just slows things down, at best.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 15:19 UTC (Thu) by eru (subscriber, #2753) [Link] (1 responses)

Isn't that way on my system. There are enough operations that take humanly noticeable amount of time, and as noted, if something is stuck, the scrolling pauses there. The OS is Ubuntu 22.04 which uses systemd; the laptop is 10 years old, but but yet dust-bin-ready, as it was rather upscale when obtained.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 15:26 UTC (Thu) by mb (subscriber, #50428) [Link]

We're talking about kernel messages here, right? Not the systemd start-stop messages.

I'm not saying that messages are not useful for debugging. And for that exact reason it's always possible to enable a verbose boot.
But in the vast majority of cases nothing is "stuck" (didn't happen in a decade for me) and there would just be a blast of messages during a couple of seconds boot time that nobody reads.
Therefore, the default should be no kernel messages and also no systemd messages.

A kernel deprecation message would certainly go by unnoticed during the burst of messages.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 15:14 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (2 responses)

But the users would be helpless even if they saw the deprecation messages!

Developers, on the other hand, are the target of such messages. I cannot imagine developing a low-level component – like systemd – and not running `dmesg` from time to time. If you use kernel features more than anyone else, you should pay more attention than anyone else.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 21:14 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (1 responses)

So first of all "dmesg" doesn't work anymore on Fedora (you need perms now).

But even beyond that: there's *so* *much* *stuff* in dmesg right now. It's a wall, a deluge of text. Don't expect me to read all that. I only look there when I am looking for something, and then I usually do "journalctl -ke", and it never showed up there.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 21:48 UTC (Thu) by pizza (subscriber, #46) [Link]

> So first of all "dmesg" doesn't work anymore on Fedora (you need perms now).

Fedora just flipped the default, starting with their 6.8 kernels IIRC.

Reverting it is just a matter of:

sysctl kernel.dmesg_restrict=0

> It's a wall, a deluge of text. Don't expect me to read all that.

Fortunately, it's rare when anything beyond the final dozen or so lines matters. (eg the messages that show up when you plug something in, or if something goes wrong...)

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 14:28 UTC (Fri) by willy (subscriber, #9762) [Link] (14 responses)

I don't understand why they felt the need to remove these mount options. I made NFS "intr" a no-op in 2007 and have no intention of ever removing it. Just laughing smugly to myself every time I see a new article written that extols the benefits of using "intr" as a mount option.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 14:55 UTC (Fri) by bluca (subscriber, #118303) [Link] (4 responses)

That sounds indeed like the appropriate way to do these kind of changes - an option is not needed anymore? Make it a no-op, maybe log a message, but just leave it there, so nothing is broken.

I mean we change public interfaces like this in systemd too every now and then - but we just keep the old configuration option around too, undocumented, and either map it to the new one if any, or make it a no-op.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 15:06 UTC (Fri) by mb (subscriber, #50428) [Link] (3 responses)

Making things a no-op also is an interface change. You're just hiding the fact.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 15:17 UTC (Fri) by bluca (subscriber, #118303) [Link] (2 responses)

It is, the point is that if an interface change is needed, then it's better to make it transparent than to hard break stuff, in case it's something that it's no longer necessary and doesn't make any difference anyway.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 16:14 UTC (Fri) by mb (subscriber, #50428) [Link] (1 responses)

>then it's better to make it transparent than to hard break stuff

No. It depends on the change itself whether a silent ignore is better or not. It might be worse to silently ignore something the user has requested.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 16:22 UTC (Fri) by bluca (subscriber, #118303) [Link]

> in case it's something that it's no longer necessary and doesn't make any difference anyway.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 16:20 UTC (Fri) by joib (subscriber, #8541) [Link] (6 responses)

Ha ha. At a $PREVIOUS_JOB I removed intr from our mount options precisely because it was a no-op by then. Lo and behold, a few months later someone else helpfully added them back "in order to make NFS more robust against hangs that otherwise would require a reboot". ;-)

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 16:53 UTC (Fri) by adobriyan (subscriber, #30858) [Link] (5 responses)

In theory, kernel could do equivalent of "sed -i '/intr//g' /etc/fstab via call_usermodehelper()...

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 19:46 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (4 responses)

There are a couple of problems with this idea:

0. /etc/fstab is a configuration file. It's not the kernel's place to modify config files (software like Puppet or Ansible will change it back).
1. Assuming you meant /etc/mtab, that's (usually) managed in userspace. It's also none of the kernel's business.
2. Assuming you meant /proc/mounts, there might be some userspace software that parses it and compares it against what it "should" look like, and misbehaves if a random mount option is missing (e.g. gets stuck in a hot loop of repeatedly remounting the device with the "correct" mount options, fires an alert and tells the sysadmin to come running, etc.).

(No, paging the sysadmin at 3 AM is not a reasonable response to this situation. Nothing is actually broken!)

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 20:29 UTC (Sat) by adobriyan (subscriber, #30858) [Link] (1 responses)

> It's not the kernel's place to modify config files

I know, relax :-) it was a joke.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 4:01 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

Even as a joke, please do not throw ideas like this, there's always a risk someone finds them great and starts to try to implement them and advocate them around.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 9:07 UTC (Sun) by smurf (subscriber, #17840) [Link] (1 responses)

Umm, /etc/mtab isn't managed in userspace. Or rather, it hasn't been for ages. These days it's a symlink to /proc/self/mounts.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 17:35 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

It was... like 10-15 years ago when I cut my teeth on this stuff. But nowadays, we have software for everything, so I haven't actually played with low-level mounting etc. in a long time.

Which is also, sort of, the point I'm trying to make here. It is no longer accurate to divide userspace into "applications" and "stuff the sysadmin manually fiddles with." Sysadmins are not manually fiddling with mount options etc. these days. That's all managed by some other piece of software that isn't "the application," but can still break and cause problems all the same. E.g. k8s, Puppet, Docker, etc.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 17:11 UTC (Sat) by shemminger (subscriber, #5739) [Link]

There are various networking statistics in /proc that have been always 0 since 2.4 kernel days since the code changed and those statistics were no longer relevant.
Easier to just silently sweep legacy under the rug, and get on with fixing real issues.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 4:00 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

I still have it on commented out lines in my fstab ;-)

btrfs in 6.8.x

Posted May 18, 2024 8:05 UTC (Sat) by zdzichu (subscriber, #17118) [Link]

This is not true. I have a number of systems running Fedora 39, 40 and rawhide, with btrfs rootfs and other mountpoints. They all run 6.8.x kernels released by Fedora and none has problem with mounting filesystems.

And as pointed out, the patch you mention removes 3 deprecation warning that were printed to dmesg for the past couple years. If one doesn't pay attention to the dmesg, kernel developers have no other ways of contacting him.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 9:33 UTC (Sat) by niner (guest, #26151) [Link]

I'm running Linux 6.8.6 and my root fs is btrfs and of course OpenSUSE uses systemd. File system mounts just fine. I also don't see why systemd would even pass those flags. You have never needed them for normal operations.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 12:03 UTC (Mon) by georgyo (subscriber, #121727) [Link]

I'm confused, what broke here? I run 6.8+ with btrfs/systems on multiple machines and never had any trouble booting.

I see a lot of people participating in this discussion, but not a lot of people actually affected.

Btrfs regression

Posted May 21, 2024 13:29 UTC (Tue) by corbet (editor, #1) [Link]

Just for anybody who hasn't long since tuned out this thread...today the Btrfs regression was reported to the Btrfs developers. Less than one hour later, a Btrfs developer acknowledged the problem and agreed to add the norecovery option back.

White paper: Vendor Kernels, Bugs and Stability

Posted May 22, 2024 0:03 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

and now a revert has been sent and merged in btrfs-next: https://lore.kernel.org/all/44c367eab0f3fbac9567f40da7b27...
So looks like it was a regression after all ;-) All's well what ends well

White paper: Vendor Kernels, Bugs and Stability

Posted May 22, 2024 5:16 UTC (Wed) by smurf (subscriber, #17840) [Link] (1 responses)

Seems that it's somewhat helpful to file such bugs on the channel(s) actually designated for them, instead of, like, bitching on LWN about them. ;-)

White paper: Vendor Kernels, Bugs and Stability

Posted May 22, 2024 11:52 UTC (Wed) by bluca (subscriber, #118303) [Link]

but where's the fun in that :-P

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 15:22 UTC (Fri) by prarit (subscriber, #27126) [Link] (8 responses)

"In addition, some of these bugs may be in code paths that are disabled via kernel config file settings. No analysis has been done on which bugs may be enabled or disabled for a specific vendor kernel config."

and

"No analysis of [bug] severity has been undertaken but it is probable that at least some of them are serious and important."

Are there missing security fixes? Certainly, yes, but the above quotes makes me question the analysis.

A better, and more reasonable approach would be to look at the areas of the kernel that RHEL8 enables, and consider the subset of CVE/Security issues reported as critical and above. #IMO of course. I'm not a security expert.

Unfixed bugs in not-shipped components...

Posted May 17, 2024 16:26 UTC (Fri) by sandeen (guest, #42852) [Link] (2 responses)

Exactly.

Just looking at filesystems, there are about 700 "unfixed bugs" in RHEL8 for filesystems which are /not shipped in RHEL8/ based on Fixes: tags in commits to those filesystems since v5.14.

Failing to filter out bugfixes for subsystems which are configured off in the RHEL kernel makes this "analysis" lazy at best, disingenuous or malicious at worst.

Unfixed bugs in not-shipped components...

Posted May 17, 2024 17:30 UTC (Fri) by sandeen (guest, #42852) [Link] (1 responses)

Oops, I didn't count everything - RHEL8 is v4.18 not v5.14. (time flies!)

There are over 1500 "not fixed bugs" since v4.18 in not-shipped filesystems alone.

That is fully 1/3 of the "unfixed bug" count in their whitepaper without even looking at other unshipped drivers, subsystems and architectures.

There may be an interesting analysis to be made, but this whitepaper doesn't seem to be it.

Unfixed bugs in not-shipped components...

Posted May 19, 2024 4:06 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

And how can you be certain that no other fixes rely on these ones being present, if the patch authors themselves don't know ? That's why stable maintainers constantly say "do not cherry-pick, take them all".

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 17:38 UTC (Fri) by jra (subscriber, #55261) [Link] (3 responses)

"Are there missing security fixes? Certainly, yes, but the above quotes makes me question the analysis."

We did do some analysis. Look at the keyword list at the end of the paper:

Appendix 1. List of missing fixes in RHEL 8 by keyword search
“uaf|use.after.free” : 117
“crash|panic|oops” : 289
”deadlock”: 97
”corruption”: 58
“out.of.bounds”: 53
”double.free”: 27
"null.pointer|null.ptr": 156
"null.deref": 22

These *are* security problems.

FYI. We considered these lists serious enough that I've sent gpg encrypted emails to the security contacts at Red Hat, SuSE, Oracle, Rocky and Alma letting them know exactly what commits these are.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 19:26 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (2 responses)

Did you also set a cutoff at the time the release was made? Checking the state of 8.8 vs commits up to mid 2024 makes no sense, RHEL users are not using the kernel you compared against.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:06 UTC (Fri) by ronniesahlberg (guest, #171541) [Link] (1 responses)

The reason for 8.8 was that was the random kernel at hand at the moment. But that is less relevant.
This is not about a particular version of a kernel but rather the model itself.

Making this model even more difficult to maintain is because it is sometimes difficult to even identify what needs to be backported or not.
Until recently the vast majority of important bugs were not even tagged as a CVE. With kernel.org becoming a CNA this is beginning to rectify with assigning ~80-100 CVEs per week (and ramping up), compared to ~200 CVEs for all of 2022.
That they are flagged now as CVEs helps.

But even looking for things that are flagged as "Fixes:" is flawed. I compared a region of commits in upstream an searched for commits that fixed use-after-free bugs and kernel-panics. Only about half of them even contained a Fixes: tag and indicated they were a bug-fix.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 14:51 UTC (Mon) by sandeen (guest, #42852) [Link]

You know what would be fascinating, and a useful control case to test thewhitepaper's theory?
Run your same analysis on the 4.19 LTS kernel.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:23 UTC (Fri) by ronniesahlberg (guest, #171541) [Link]

The reason for that is because there are just too many to analyze.
A small subset of random commits in this list were analyzed and verified manually.

Some of them even contain description in the commit message how to trigger the bug and crash the kernel from unprivileged userspace.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 19:05 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (37 responses)

> due to Red Hat not publishing the complete and corresponding source code in the preferred form of the work for making modifications to it.

Ok I guess this clarifies how serious the white paper is.

The full CVS, then svn, then git tree has never been considered complete and corresponding source code.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 19:56 UTC (Fri) by jra (subscriber, #55261) [Link] (36 responses)

That phrasing is deliberate (I wrote it). Red Hat does *not* publish the complete and corresponding source code in the preferred form of the work for making modifications to it for the kernel. A tarball in an rpm build is not the preferred form of the work for making modifications to it. That would be a clonable git tree of the RHEL kernel source code.

This was covered by lwn back in 2011.

https://lwn.net/Articles/430098/

"Red Hat is making things harder by shipping its RHEL 6 kernel source as one big tarball, without breaking out the patches."

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:01 UTC (Fri) by jra (subscriber, #55261) [Link] (16 responses)

To clarify further, this comment from that previous story lays it out precisely:

https://lwn.net/Articles/430192/

"- The kernel tarball inside the srpm is created from a git tree that is only accessible to Red Hat engineers.

- The orders to do this, to make it harder to rebuild the kernel with and without patches, and to make it harder to extract specific patches from the Red Hat kernel came from the top. This is with the knowledge of, and by the order of, the CEO: Jim Whitehurst."

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:15 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (15 responses)

> "- The kernel tarball inside the srpm is created from a git tree that is only accessible to Red Hat engineers.

By the same token, Bash is not free software. Its releases are made as giant git commit dumps: https://git.savannah.gnu.org/cgit/bash.git/commit/?id=886...

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 21:27 UTC (Sat) by rschroev (subscriber, #4164) [Link] (13 responses)

But AFAICS the complete Bash tree is publicly available on Savannah, in conflict with the Red Hat kernel case where the full git tree is only available to Red Hat engineers.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 23:06 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (12 responses)

> But AFAICS the complete Bash tree is publicly available on Savannah

Yes, and that tree (I linked to it) only has giant dumps. Just like RedHat.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 23:23 UTC (Sat) by mb (subscriber, #50428) [Link] (11 responses)

> Just like RedHat.

No.
RedHat is not the author of the majority of the code.

The authors can do whatever they like to their code.
Downstream distributors can't.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 23:25 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

The maintainers of Bash don't have exclusive copyright to it, they still need to obey the GPL.

If giant tree dumps are not the preferred form, then Bash is arguably violating the GPL.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 23:35 UTC (Sat) by mb (subscriber, #50428) [Link] (9 responses)

>The maintainers of Bash don't have exclusive copyright to it

I think they pretty much do, due to FSF Copyright assignment rules.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 23:53 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

So... FSF is still violating its own license?

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 8:58 UTC (Mon) by paulj (subscriber, #341) [Link] (6 responses)

The licensor is not bound by a licence that places obligations only on licensees.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:34 UTC (Mon) by Wol (subscriber, #4433) [Link] (5 responses)

It's called "hypocrisy".

You know, that thing that totally destroys the credibility of our "moral guardians".

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 10:51 UTC (Mon) by paulj (subscriber, #341) [Link] (4 responses)

It is also quite possible that the "preferred form for modification" differs across different contexts. For one set of licensors and licensees it may be one form, for another set another.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 12:38 UTC (Mon) by pbonzini (subscriber, #60935) [Link] (3 responses)

Even in the one case that is precisely defined in the license itself?

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 12:46 UTC (Mon) by paulj (subscriber, #341) [Link] (2 responses)

What do you mean exactly?

If you mean "For an executable work, complete source code means ..." - that's defining the complete source code, but not the preferred form for modifications.

You can have the complete source code, but not have it in the preferred form for modification.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 13:54 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

The "preferred form of modification" was, and is, a text file. Or a directory tree with a large heap of 'em.

True, the preferred form of keeping track of your modifications, distributing them, etc.etc.etc., once was patches and tar files; now it's git and pulling commits. The problem is that the GPL doesn't talk about any of that. It merely requires "a" medium commonly used for software exchange. Not "the medium everybody else who's contributing to this particular piece of source code is using".

So for better or worse, you can appeal to common sense and interoperability and the equivalent of sportsmanship and whatnot and I'll be the first person to be argue right along with you. But the GPL? sorry you pour cold water on your fire but you can't use it to demand git trees from anybody.

Disclaimer: IMHO and IANAL and all that.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 14:36 UTC (Mon) by paulj (subscriber, #341) [Link]

You can say what you want, but the GPL deliberately does not define what "preferred form" means. And the "For an executable work, ..." part doesn't change that, because it it self refers to "source code" - it clarifies that the executable work includes the "source code" for X, Y and Z. And "source code" means "the preferred form for modifications".

You may indeed argue that the preferred form is a text file.

However, if you are a company that has deliberately stripped away context that you yourself use for hosting and modifying said source code as part of your commercial operation, and you have made that deliberate act in order to frustrate others who also wish to modify said source code, then I - if it mattered to me and I were a copyright holder - could well argue to a court that the form you have provided is clearly not the form you prefer to use yourself for making modifications, and that you have therefore violated the licence I gave you.

E.g., I - as your upstream - could well be trying to follow along with what you're doing to my code, so that I can cherry-pick the good bits to integrate back into the code. Your deliberate obfuscation, done to keep for yourself the preferred form of modifications and hence frustrate others from picking out what you changed, I would argue as being clearly against the wording of the licence.

The form you prefer to use internally, versus the form you chose to release, would be prima facie evidence of your failure to honour the licence.

Who would win, who knows. But I don't think your position would be 100% safe.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 0:01 UTC (Sun) by bluca (subscriber, #118303) [Link]

I don't think bash does copyright assignments, I've sent patches in the past and they were merged, and I do not remember signing anything

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 7:27 UTC (Sun) by gioele (subscriber, #61675) [Link]

> By the same token, Bash is not free software. Its releases are made as giant git commit dumps: https://git.savannah.gnu.org/cgit/bash.git/commit/?id=886...

Bash's repository has two main branches:

* `master` (<https://git.savannah.gnu.org/cgit/bash.git/log/>), that follows the historical 1-mega-patch-per-release paradigm, and
* `devel` (<https://git.savannah.gnu.org/cgit/bash.git/log/?h=devel>), where development is carried out using more modern piece-wise commits.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:36 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (16 responses)

< That would be a clonable git tree of the RHEL kernel source code.

Based on what novel reasoning?

If the "complete and corresponding source code in the preferred form for making modifications" included the full source code history, that history would have to be distributed to whoever receives a binary. Not just for Linux: for all packages. And it would have to include *the whole upstream history* if you are performing commercial distribution and therefore must follow sections 3a or 3b of the GPLv2.

Now, it's not just Red Hat that does not do that. There is simply no commercial distribution that does that, not even counting hardware manufacturers that use GPL'd software in the gadgets they sell. If you disagree, sue them all including your employer. Or more likely, any serious lawyer would have told you that your assertion is ridiculous.

More technically: the preferred form for making modifications is not the preferred form for analysis. So for example, when I backport a fix to a Fedora package, I extract the tarball and apply any Fedora-specific patches with "fedpkg prep", then typically follow with 

   cd package-1.2.3
   git init
   git add .
   git commit -mimport
   git remote add upstream /path/to/upstream/repo
   git cherry-pick UPSTREAM-COMMIT

I use the git tree from upstream (the tree that I analyzed to find the bug and the fix) and not the git history from Fedora (which "fedpkg prep" has thrown away). The same is true for the RHEL kernel: I _have_ the git history of course, but in practice I use the upstream git tree (which I am analyzing but not modifying) and I don't need to look at the history of the RHEL kernel (which I'm modifying).

> "Red Hat is making things harder by shipping its RHEL 6 kernel source as one big tarball, without breaking out the patches."

"Distribution in this form should satisfy the GPL". Literally three lines below.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:06 UTC (Fri) by ronniesahlberg (guest, #171541) [Link] (15 responses)

I think this might be splitting hairs. The main argument about lack of full git history is not about compliance but that without the commit history it makes it harder to identify what is and what is not in the kernel.

It is harder for an end user to easily analyze a kernel to see what exactly it contains. Making it harder to answer questions like "does this kernel contain this particular fix".

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:25 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (14 responses)

> The main argument about lack of full git history is not about compliance but that without the commit history it makes it harder to identify what is and what is not in the kernel.

No, I am sorry but what Jeremy said has serious legal implications. The wording he used (especially with the confirmation that he chose it deliberately) is a clear accusation of violating the GPL. Don't defend the indefensible, please.

I agree that the lack of granularity in CentOS 8 is bad. Fortunately, in CentOS Stream 9 the development is done again entirely in the open. Of course that is too early in the release stream to show the slowdown that you point out, but let's see where we will be three years from now. :)

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:44 UTC (Fri) by jra (subscriber, #55261) [Link] (13 responses)

Whether this is GPL compatible is subject to serious debate. Here for example:

https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analy...

People of good conscience (and I certainly include you in this) can have different opinions on this matter.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:54 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (5 responses)

The Conservancy blog post is about the RHEL business model, and not at all about whether the kernel patches are split or not. As should be obvious from it not containing any of the words "commit", "split", "patch", "kernel", "2011", "tree", "git".

Also, the Conservancy blog post does not really debate the fact that the RHEL business model complies with the GPL. Rather, it complains about the difficulty of *verifying* from the outside that Red Hat does indeed comply in practice and not only on paper. I think it's a reasonable complaint, especially from the point of view of an entity like Conservancy—and considering that the nuances of free software licensing are not obvious to all of the employees on Red Hat, on neither the engineering side nor the sales side.

That said, it's also not a standard to which many companies are held, and perhaps that's a point of honor for Red Hat. :)

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 22:16 UTC (Fri) by jra (subscriber, #55261) [Link] (4 responses)

> That said, it's also not a standard to which many companies are held, and perhaps that's a point of honor for Red Hat. :)

That's a really good point, and one to which I wholeheartedly agree. I *do* expect better of Red Hat. As I have often said, I have many Red Hat engineering friends and a great respect for the engineering talent and prowess of Red Hat.

It disappoints me greatly to see Red Hat moving away from the ethos of improving the availability of Free Software, hiding git trees from other engineers, and generally seeming to move away from open collaboration.

Still, I believed in Google Open Source too, and look where that got me :-) :-).

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 0:43 UTC (Sat) by Wol (subscriber, #4433) [Link] (3 responses)

Thing is, if you look at WHY Red Hat did it, it was in direct response to an attack on their financial stability. It's all very well saying they should abide by a higher standard, but if that puts them on a suicide course, what on earth do you expect them to do?

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 4:51 UTC (Sat) by jra (subscriber, #55261) [Link] (1 responses)

Well they did it because of Oracle. Do you think ruining their source releases in that way really had much impact on what Oracle did or how successful Oracle was with Oracle Linux ? Personally I don't. It did, however, break their reputation with the Open Source community in other ways.

Reacting due to fear often leads to the wrong response.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 8:53 UTC (Sat) by Wol (subscriber, #4433) [Link]

> Reacting due to fear often leads to the wrong response.

And investigating before reacting (I don't know whether RH did, but they took their time reacting) often leads to surprising conclusions and actions.

(I'm currently working as a business analyst. It's my job to react to events like this. And I'm expected to have a good justification for my reactions.)

People often confuse cause and effect. Or blame something totally irrelevant. If RH determined that Oracle's actions were causing serious damage to their bottom line, then they had to do something about it - regardless of the cost. If you really are facing an existential threat, you're not going to care about your reputation.

(And as others have said, I totally agree that it's nice that RH are held to a higher standard than others, but I also think that's unfair! :-)

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:00 UTC (Mon) by paulj (subscriber, #341) [Link]

One may have sympathy for RedHat in that, however that still does not give them permission to deliberately withhold the source code in the form they prefer for modifications.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 22:00 UTC (Fri) by Wol (subscriber, #4433) [Link] (6 responses)

Given that there is no guarantee that stuff even USES some sort of version control, by demanding a full version control you are placing obligations on people releasing free software.

Given that "dumping the source over the wall" is perfectly okay for projects without source control, except not popular with recipients, you are now getting into the legally extremely hairy situation where what is perfectly okay for one project, but allegedly illegal for another ...

Do we really want to go down that rabbit hole?

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 22:20 UTC (Fri) by jra (subscriber, #55261) [Link] (4 responses)

Times change. And the meaning of "complete and corresponding source code in the preferred form of the work" has changed with it.

I remember the days of throwing tarballs on an ftp site (and indeed did so myself :-). But these days that really isn't how collaboration is done.

Git is king, for better or worse (better, IMHO). A git tree really is the preferred form of the work these days (or insert distributed source code control system of your choice for more obscure projects).

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 22:32 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (1 responses)

But the license says "preferred form for modification". The line is very clearly placed by the license which even has an example for executables. That example does not cover understanding past development, collaborating on further development, or receiving continued support.

If the meaning has changed, then no one is distributing GPLd software commercially and complying with the supposed new meaning. Which to me is a pretty good sign that the meaning has not changed, and that's for good reasons.

White paper: Vendor Kernels, Bugs and Stability

Posted May 18, 2024 9:31 UTC (Sat) by smurf (subscriber, #17840) [Link]

Problem is, "source code" is a bunch of files. The preferred form of modification is still those files which one edits/compiles/debugs, git or no git.

Yes, you cannot use the kernel source in any meaningful way without git these days. Unfortunately, that's not covered by the GPLv2, just like Tivofication isn't (and that one is a far worse problem). You need the GPLv3 for that – which the kernel cannot and will not transition to.

Thus while yes there's a moral / good practices / be-nice-to-the-community obligation to provide the git archive, legally? almost certainly not.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 16:46 UTC (Sun) by Wol (subscriber, #4433) [Link]

> Times change. And the meaning of "complete and corresponding source code in the preferred form of the work" has changed with it.

Why?

> I remember the days of throwing tarballs on an ftp site (and indeed did so myself :-). But these days that really isn't how collaboration is done.

And this is called pushing your views down other peoples throats. (Personally, I think even doing "on your own" development benefits from version control, BUT IT'S NOT MY PLACE TO TELL OTHER PEOPLE WHAT TO DO!)

What's that advice I've come across before? "You think you're a sensible, rational human being. When you disagree with someone, you should assume they are sensible and rational too. So any disagreement must be down to you having a different set of facts from them. At which point, how do you know it's not YOU who've got the facts wrong."

Face it, even today there are people who don't/won't use version control. Even today there are people (especially sole copyright holders) who will throw a tarball over the fence and say "This will compile and work, I'm done". If that was acceptable in the past, it has still to be acceptable today. And if it's acceptable today, why is it acceptable for some and not others?

You can say it's not being a good citizen, not being fair (I don't know if I'd agree with you), but if it's in reaction to someone else playing dirty and not being a good citizen, then I'd much rather you went after the *real* culprit, the cuckoo in the nest, not the victim trying to make the best of a bad job. It's called "blame the victim", don't cha know...

>Git is king, for better or worse (better, IMHO). A git tree really is the preferred form of the work these days (or insert distributed source code control system of your choice for more obscure projects).

And for those people who don't use version control (beyond datestamped backups) ???

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 21, 2024 9:18 UTC (Tue) by jamesh (guest, #1159) [Link]

You are essentially arguing that it isn't enough to distribute the source code corresponding to the binaries, but that the source code to all previous versions is also required. That seems like it'd be a problem for far more than just RHEL kernel packages. Off the top of my head:

  • Many pieces of software have changed license at some point in their history. A distro might be happy with the current license but not the old one. Do they need to distribute all the versions released under the bad license now?
  • A number of packages in Debian modify the upstream source tarballs to remove files with licenses that don't meet their guidelines. If they were required to distribute a git tree with history linking back to upstream, that'd include the problem files.
  • Some projects have been accused of plagiarism and removed the offending code, but it remains in the version control history.

There are open source licenses that require that you split out patches from the upstream release (e.g. the QPL), but the GPL is not one of them.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:05 UTC (Mon) by paulj (subscriber, #341) [Link]

That there may exist an upstream GPL project whose upstream releases consist of tarballs with many changes aggregated, does not release RedHat from their obligations to release the code in the form preferred for modification. And if RH internally are tracking GPL source code by distinct changesets against base versions than that is _at least_ RedHat's preferred form for modification, and for RH to deliberately withhold that form because they know /others/ would also prefer the code _in that form_ would then be an *obvious* GPL violation.

Good they made the kernel code available in the preferred form again. It still stinks they ever thought playing those tricks was acceptable.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:39 UTC (Fri) by pizza (subscriber, #46) [Link] (1 responses)

>A tarball in an rpm build is not the preferred form of the work for making modifications to it.

You're wrong, according to the plain text of the GPL:

"The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable."

I'll also add a brief snippet of what was so well written in https://lwn.net/Articles/431651/ :

"The language in the license was added to prevent someone from shipping a printed version of the source or other forms of distributing the source that amount to useless for an end user. Sure it's the source all right, but not machine readable nor editable or usable to make a modified distribution from. The language of the license protects us, the end users, from these kinds of actions by distributors of GPL applications."

RH's kernel SRPMs are just as GPL compliant as the tarballs provided on the front page of kernel.org. And most FSF-provided software too, I might add.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 16:53 UTC (Sun) by Wol (subscriber, #4433) [Link]

And to spell out what I hadn't noticed before ...

> For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable."

Where is ANY mention of the history? Where is ANY mention of commit messages? Where is ANY mention of any documentation, user or developer?

As I read that, if I fed the source through a pre-processor that stripped all comments, anything like that, that left just pure source, that would still comply with the licence.

Yes it's nice to have all the extra stuff, but that term seems quite clear - as long as what you're given compiles, then it is COMPLETE. In other words, a git tree is not the "complete source" because if you fed that into a compiler, it would rather seriously choke !!!

I'm always amazed how well the GPL has been crafted - this is just one more example.

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:12 UTC (Fri) by carlosrodfern (subscriber, #166486) [Link]

"Note that when these CVEs are announced, they have already been fixed in the stable Linux kernel git branch."

As far as I know Debian only ship LTS kernels. Those that mean Debian is doing better keeping up with these CVEs?

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 20:58 UTC (Fri) by jd (guest, #26381) [Link] (13 responses)

I understand the value in keeping old kernels - anything that's dependent on a feature whose semantics have changed (even subtly) isn't good news for corporations, and new codex risks new bugs.

However, I'm pretty sure we've reached the point where old vulnerabilities are costing businesses and there's also the cost of hiring developers to backport features.

Here's a what-if to consider. What if the distros and the corps spent exactly the same amount of money on a mix of QA engineers and developers for much more recent stable kernels?

In other words, actively find and fix the new bugs in kernels where many of the old vulnerabilities have already been corrected, rather than passively backport a selection of those fixes?

Yes, you would need much more QA, because you're having to find bugs that haven't yet been fixed by anyone. But my suspicion is that this would be a far more productive approach.

However, suspicion and reality don't usually agree. Security issues can be evaded, to a degree, with firewalls, and there may be an economic penalty from corps preferring known defects to unknown ones. You can mitigate known risks, after all.

Furthermore, such an approach depends on competent QA and my experience is that really good QA engineers are rare because if they're that good, they're unlikely to be attracted to QA.

But if long-term stable Linux distros aren't to acquire the same security taint as... certain other vendors I could mention, the status quo won't work and this is the most obvious alternative.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:05 UTC (Fri) by bluca (subscriber, #118303) [Link] (9 responses)

The problem is not bugs. The problem is when userspace backward compatibility is intentionally broken, which happens on every kernel release. That means you have to go and make all affected userspace compatible with both new and old kernels, and that's expensive, and you cannot do that every 3 months.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 22:37 UTC (Fri) by jra (subscriber, #55261) [Link] (8 responses)

> The problem is when userspace backward compatibility is intentionally broken, which happens on every kernel release.

I don't believe that is true. Can you point to any lkml discussions about intentionally breaking userspace (that don't end up with Linus being very cross :-) ?

The kernel community IMHO is extraordinarily careful not to break userspace compatibility.

Compatibility

Posted May 17, 2024 22:42 UTC (Fri) by corbet (editor, #1) [Link] (7 responses)

Unfortunately, Luca and company are not entirely wrong on this front. I still need to find some time to figure out what ultimately happened with this recent episode, for example. We're not as good at this as we like to think we are.

Compatibility

Posted May 17, 2024 22:48 UTC (Fri) by jra (subscriber, #55261) [Link] (6 responses)

Testing testing testing :-). Every time we do this in Samba we add a new regression test to make sure we don't at least break that specific network feature ever again.

I think by the time upstream makes it into Greg's kernel trees any userspace breakage has already been found and fixed. I'm not aware of any userspace breakages there (although please correct me if I'm wrong, I'm still pretty new to this kernel stuff).

"You can never be too thin, too rich, have too much disk space or too many regression tests" :-).

Compatibility

Posted May 17, 2024 22:52 UTC (Fri) by bluca (subscriber, #118303) [Link]

See first comment on this article for another even more recent example

Compatibility

Posted May 18, 2024 6:47 UTC (Sat) by marcH (subscriber, #57642) [Link]

Promises without tests are just empty words; if it's not tested then it does not work. Every piece of software is only as good as the tests it's passing.

Etc.

Compatibility

Posted May 19, 2024 4:40 UTC (Sun) by wtarreau (subscriber, #51152) [Link] (3 responses)

One difficulty that makes this a reality is present in all software: some features happen by accident and were not intentional by developers but are more convenient side effects. They also have the interesting property of not being tested since they're unknown. When users start to rely on these and the code changes, things can break.

This is commonly visible in various setup scripts that rely on contents of /proc, /sys, output from lsmod etc. In other software sometimes a major upgrade will add support for new options to a keyword and reveal that a previous version would silently ignore any such options and that a typo in a previous version now raises an issue.

There's never a perfect solution to this. If users would only use perfectly documented and intended features, systems and programs would be very poor and boring, and nothing would evolve. They would rightfully complain even more about the occasionally needed breakage due to architectural changes. If they use any single possibility, they spark the addition of new features but face unintended changes more often, and developers have a more difficult time making their code evolve.

A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.

Overall I find that Linux is not bad at all on this front. You can often run very old software on a recent kernel and it still works fine. And at the same time I'm one of those being extremely careful about major upgrades because if I don't see the breakage, I suspect it will happen in my back at the moment I would have preferred not to face it. Nobody wants to discover their microphone is no longer working when joining a visio-conference, or that their 4G adapter fails to initialize when waiting for a train or plane at the station for example. On a laptop I've very rarely experienced issues, except with broken or poorly supported hardware whose behavior would change in various ways along versions. On servers I'm much more cautious because of subtle changes experienced overa long time in bonding/bridging/routing setups, and iptables also showing some more config granularity that reveals in field that you're missing some new config options from time to time. And this is a perfect reason for sticking to a stable kernel when you have other things to do than to retest everything.

Compatibility

Posted May 19, 2024 17:00 UTC (Sun) by Wol (subscriber, #4433) [Link] (1 responses)

> There's never a perfect solution to this. If users would only use perfectly documented and intended features, systems and programs would be very poor and boring, and nothing would evolve. They would rightfully complain even more about the occasionally needed breakage due to architectural changes. If they use any single possibility, they spark the addition of new features but face unintended changes more often, and developers have a more difficult time making their code evolve.

I don't think you're right here. We'll never find out, of course, because developers (and open source collaborative projects are amongst the worst) are very bad at writing documentation.

> A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.

That sweet spot, as an absolute minimum, needs to include DESIGN documentation. This is where Linus' management style really fails (although given that he's cat-herd in chief I don't know any style that would succeed), because so much of Linux is written by people scratching itches, fixing bugs, and everyone is fixing their immediate problem.

Nobody steps back, and asks "what is the purpose of the module I'm working? What is sensible or stupid? How can I design this to be as *small*, *complete*, and *self-contained* as possible?".

Cheers,
Wol

Compatibility

Posted May 20, 2024 11:15 UTC (Mon) by wtarreau (subscriber, #51152) [Link]

> needs to include DESIGN documentation

You're absolutely right on these points. I have asked myself in the past the reasons for this situation in a lot of such software documentation and came to the conclusion that it's based on the fact that aside a few exception, there's mostly no design but an improvement on top of something that already exists. As such, lack of design phase implies lack of design documentation. Sometimes someone spends so much time reversing something they work on that they take a lot of notes and end up creating a doc of the existing design. It does not necessarily justify choices but can be a good point to encourage new participants to update the doc. But that's not *that* common.

I'm not seeing any good solution to this , though :-/

Compatibility

Posted May 23, 2024 16:49 UTC (Thu) by anton (subscriber, #25547) [Link]

A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.
That's what C compiler advocates use to defend miscompilation: "works as documented". And while this bad principle is used for declaring gcc bug reports invalid, the actual practice of gcc development seems to be better: They use a lot of tests to avoid breaking "relevant" code (including for cases outside the documented behaviour), and the irrelevant code rides in the slipstream of relevant code.

Fortunately, the kernel aspires to a better principle, the often-cited "we don't break user space". And given that we have no way to automatically check whether a program conforms to the documentation, that principle is much more practical (and the practice of gcc is also in that direction, but only for "relevant" code).

Concerning accidental features, a careful design of interfaces avoids those. E.g., one can catch unintended combinations of inputs and report them as errors. Or define and implement useful behaviour in such a case. Exposing some arbitrary accident of the implementation in such cases leads to the difficulties you point out. Given that Linux' interface to user space is also a security boundary, carefully defining interfaces and avoiding exposing implementation artifacts is a good idea anyway.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:23 UTC (Fri) by ronniesahlberg (guest, #171541) [Link] (2 responses)

"Here's a what-if to consider. What if the distros and the corps spent exactly the same amount of money on a mix of QA engineers and developers for much more recent stable kernels?"

I think this is where we need to go, and in a sense it would be a continuation on a collaborative process that has been happening for a while.
For example, in the past it was very common that different vendors all had their own private trees with their own private drivers and it was always very difficult and expensive for them to move from one kernel release to another. Today most of those vendors all collaborate and work in upstream. Making life much easier for those vendors themselves as they now can focus on a common codebase instead of spending time maintaining their own incompatible code-bases.

I think would be good if we extended this also to the packaging of stable vendor kernels. Instead of everyone maintaining their own private special kernels and all backporting their own selected small subset of important fixes, everyone could instead collaborate on a set of upstream stable kernels and ensure that they all get all the fixes that are needed.

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 21:26 UTC (Fri) by ronniesahlberg (guest, #171541) [Link]

To clarify, my post above that talks about vendors in the past all having their own private trees is referencing embedded systems vendors and arm, not distribution vendors.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 12:20 UTC (Mon) by Wol (subscriber, #4433) [Link]

> "Here's a what-if to consider. What if the distros and the corps spent exactly the same amount of money on a mix of QA engineers and developers for much more recent stable kernels?"

"You can't QA quality into a product".

"10 minutes spent on design knocks 1hr off development".

"Every problem that slips through one phase takes 10 times longer to correct in the next".

As I moaned above, design, Design! DESIGN. If that money was spent on *documentation* and *design* it would probably find far more bugs, far more quickly, and what's more important get them fixed (probably by other people :-)

The current setup encourages a kernel held together by baling wire, duck tape, and sealing wax ...

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

Posted May 17, 2024 23:00 UTC (Fri) by ronniesahlberg (guest, #171541) [Link] (2 responses)

A couple of things we should have been clearer about in the whitepaper is that this is not a unique flaw of rhel but rather the model itself and we would likely found the exact same thing if we looked at a suse kernel, or any other vendor maintained kernel.
The main difference might just be that a different vendor might just end up with a different incomplete subset of important fixes.

I think one of the underlying assumptions is that "all bugs are found and fixed relatively soon-ish, probably".
And that if you freeze a kernel and maintain it for very many years, eventually most of the bugs will have been found, fixed and selected for backporting. (This is assuming the selection process can even identify most of the bugs).

I certainly thought that "almost all bugs are found and fixed relaltively quickly" before I started looking at data.
But the case is that very many bugs take a very long time before they are found.

Fun with numbers:
In upstream, in the last 3 months there are 34 commits that reference 1da177e4. That is a really old commit. That is the start of git history 19 years ago. :-)

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 17:49 UTC (Sun) by willy (subscriber, #9762) [Link]

FWIW, Oracle bases on LTS (for UEK) and the git history is public on GitHub. I am paid by Oracle but I work upstream, not on UEK. If I want to get a fix into UEK, I make sure GregKH has it. I think it would be an interesting comparison because it would tell us how successful the two models are at getting fixes into customer hands.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 15:00 UTC (Mon) by sandeen (guest, #42852) [Link]

I commented in another thread, but maybe it's more relevant here:

I think it would be fascinating run your same analysis on the 4.19 LTS kernel, which is supported until the end of this year, IIRC.

More breaking of userspace

Posted May 17, 2024 23:32 UTC (Fri) by roc (subscriber, #30627) [Link] (6 responses)

Just yesterday I discovered a fairly recent upstream kernel change that broke an rr regression test:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...

This change was released in 6.7.0. It significantly changes the behavior of mmap(): many cases of large anonymous mappings are now much more aligned than they used to be. The rr test was broken because if you specify an address hint without MAP_FIXED and there is nothing already mapped in the desired range, mmap() used to always create the mapping at the hinted address, and now it doesn't. Sure, none of this is specified behavior, but it is a change that has serious effects on userspace.

In fact it led to a number of regression fixes appearing in 6.7.6:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... 32-bit processes that use most of their address space were just broken
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... some VMAs that used to be allocated contiguously now aren't, which means less VMA merging, leading to a kernel test using far too many VMAs
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... alignment changes revealed an existing bug in a kernel test
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... alignment changes caused a regression in a thread stress test because stacks started getting transparent huge pages. The fix here assigns semantics to MAP_STACK which was previously a no-op, so anyone creating threads and not using MAP_STACK is still in trouble.

The timeline is interesting. The original change was committed on December 14 and appeared in the 6.7 release on January 7. The regression fixes were released on February 23. So apparently it's considered OK to make a change like this just three weeks before a kernel major release. And apparently quite serious regressions can take several weeks to be fixed after a major release.

More importantly, I think someone should have realized that, given kernel tests were breaking, other userspace code is probably breaking and the original change should at least have been reverted from the stable kernel branches.

So yeah, it's completely rational for users to keep their distance from upstream kernels.

More breaking of userspace

Posted May 17, 2024 23:40 UTC (Fri) by roc (subscriber, #30627) [Link]

Oh, the original change wasn't committed until December 29. So it had about one week of testing in mainline before being released in 6.7... and week when a lot of people are not at work.

Something is wrong.

More breaking of userspace

Posted May 18, 2024 13:54 UTC (Sat) by vbabka (subscriber, #91706) [Link] (2 responses)

The timing of the breaking commit close to 6.7 release was definitely a mistake, that wasn't a hotfix that would be normally eligible for mainline outside of merge window.

Is the rr regression still happening or was rr changed to use MAP_FIXED? I can imagine we could interpret the address hint (without MAP_FIXED) as a stronger signal than the new huge alignment, if that's what missing. If the regression is due to a different mmap() getting aligned differently and thus influencing the following mmap() with address hint, it's more complicated.

More breaking of userspace

Posted May 18, 2024 21:05 UTC (Sat) by roc (subscriber, #30627) [Link] (1 responses)

I changed rr to use MAP_FIXED. I don't think it's worth making a kernel change just for old versions of rr, the impact on rr users is actually minimal in practice.

But I do think it would make sense to honour the hint if possible. If that's what the user wants, why not give it to them?

More breaking of userspace

Posted May 23, 2024 7:46 UTC (Thu) by taladar (subscriber, #68407) [Link]

I would suggest the opposite. The only implementation I would never want to see is one treating a hint as a guarantee. If you guarantee the behavior specify it as a guarantee, don't make it implicit like that.

More breaking of userspace

Posted May 18, 2024 19:51 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (1 responses)

This is also a fantastic example of Hyrum's law:

> With a sufficient number of users of an API,
> it does not matter what you promise in the contract:
> all observable behaviors of your system
> will be depended on by somebody.

From https://www.hyrumslaw.com

More breaking of userspace

Posted May 19, 2024 4:43 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

I didn't know this one and I totally agree.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 5:04 UTC (Sun) by wtarreau (subscriber, #51152) [Link] (1 responses)

There's a difference that is not always perceived by users between stability and security.

To make it simple, any security fix is likely to break your system's stability or compatibility. We all know machines at various places running totally outdated, unbreakable kernels that have never experienced a single crash in multiple years and that nobody dares to update by fear of starting to experience stability issues.

Stability is only measured by user feedback and complaints (and a little bit by testing, to eliminate the most visible problems upfront). Security on the other hand indicates how hard it is to abuse privileges on a given system, and is addressed by restricting what users were allowed to do in subtle ways, which is why, if workloads were relying on a side effect, a security fix can break them.

My vision on this is that mainline is the most secure because it's the place where all known bugs are fixed, and it's also the least stable since it's where all yet-to-discover regressions are present. Distro kernels are the most stable due to missing (purposely or not) some changes, and are likely as well the least secure since any mistaken analysis of a fix that leads to it being skipped (or included without an important dependency) may leave a hole open.

Stable/LTS kernels are between the two and are *expected* to contain all known fixes in reasonable time while *trying hard* not to break anything. Stable kernels receive lots of automated testing but probably not as much as distro kernels. And they can occasionally miss some security fixes present in mainline that were not analyzed as such for example (e.g. accidental fixes). Their real strength (sometimes debated here) is that patch authors are offered the option to comment on or reject patches to be included, something that does not happen with distro kernels (nor any other private patch collection). But as they move faster, they're naturally more prone to regressions than distro kernels, though these regressions are quickly detected, reported, and addressed (often by simply dropping the offending patch).

My practice on this is to deploy a few versions old LTS kernels for which no breakage was apparently reported, and to stick as long as possible to the same branch. For now this has been working remarkably well.

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 7:50 UTC (Thu) by taladar (subscriber, #68407) [Link]

What you describe is the traditional understanding. However I doubt that is actually true.

Every backport is also a new, untested version, one with changes by someone with significantly lower understanding of either the original code base or the fix than the person working on that part of the code on the latest version. Distro kernels see significantly less testing than the latest version does because of its limited audience which is largely comprised of people who only want it once it has been tested.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 15:15 UTC (Mon) by smoogen (subscriber, #97) [Link] (2 responses)

[Note: I am currently employed by Red Hat. I have worked for Red Hat for 19 years as such my comments should be seen as biased. ]

In looking at the white paper, I was wondering why the comparison was done between Red Hat's which was 'based' off of 4.18 and the 5.x series. I would have thought a better analysis would have been to see what had been backported to the long term 4.19 kernel to see if there were apples to apples between the two. Then do a comparison between 4.18 and the 5.x series as various things were moved back from that kernel in places.

The items I have run into with customers either in Red Hat or outside is that generally they want with a kernel series is:
1. Stability. The majority of them will put up with various security problems over any stability issues. This stability goes beyond 'crashes', but also interface and other changes. Stability covers ability to use the same configs for related software and workloads versus special casing because core applications are different.
2. New hardware support. Generally they want to be able to run their operating system over a general set of hardware purchased over a 6 year period. [They would love to have that be hardware over a 14 year period.. but there are limits]
3. Security fixes. [Make sure that they can pass various audits to maintain usage of certain systems].
...
4. New software support on the same OS.

What the customer will say is usually a different order but what they complain about after they have a contract is usually in the first order.

Personally I would love the order to be different but all PCI and other audit requirements has done is move 3 up from probably 4 or 5 (where 3 was New Software Support). [This is also a spectrum. There are also probably a billion Linux systems out there so that even if 80% used that order, the number having completely different priorities would be hundreds of millions of systems.]

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 16:08 UTC (Mon) by jra (subscriber, #55261) [Link] (1 responses)

> In looking at the white paper, I was wondering why the comparison was done between Red Hat's which was 'based' off of 4.18 and the 5.x series.

I think Ronnie answered this question above in this comment.

https://lwn.net/Articles/974151/

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 16:33 UTC (Mon) by smoogen (subscriber, #97) [Link]

I didn't actually think it covered my concerns but I also did not clearly state them in my first email.

The reason to compare against 4.19 first would be to see if X was backported to that also or if it was avoided because of reasons (the problem came in after 4.19, the problem only affects hardware not supported by 4.19, etc). And it would not have mattered if the kernel being compared was the 8.8 kernel, the 9.0 kernel or similar.

I understand that doesn't cover the Red Hat Enterprise Linux kernel very well since it is not 4.18 straight. It is a chimera with different functionality from later kernels backported which does bring in other changes. I also realize my main confusion was I am not sure what the purpose of the whitepaper was for. I thought it would be on selling CIQ's knowledge of how a competitors kernel internals were and to show the limitations in a clear fashion to better sell their services. [AKA you can trust us because we can show you clearly and concisely how this was done.] That may not have been the purpose and so my suggestions would not work.


Copyright © 2024, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds