White paper: Vendor Kernels, Bugs and Stability
This means that over time, the security of the RHEL kernels get worse and worse as more issues are discovered in the upstream code and are potentially exploitable but fewer and fewer of the fixes for these known bugs are back-ported into RHEL kernels.After reaching RHEL 8.7, the theory is that the kernel has been stabilized, with a corresponding improvement in security. However we still have an influx of newly discovered bugs in the upstream kernel affecting RHEL 8.7 that are not addressed. Each minor version of upstream is released on an approximately quarterly basis and we can see that the influx of new bugs that are unaddressed in RHEL is growing. The number of known issues in these kernels increases by approximately 250 new bugs per quarter or more.
Posted May 17, 2024 13:37 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (69 responses)
https://fosstodon.org/@bluca/112455500079789967
Any OS booting with any released version of systemd and expecting it to mount a btrfs filesystem is now broken with kernel 6.8+.
But why, oh why do people use distro kernels and do not want to upgrade? Guess we'll never know.
Posted May 17, 2024 13:59 UTC (Fri)
by snajpa (subscriber, #73467)
[Link]
Posted May 17, 2024 14:16 UTC (Fri)
by Bigos (subscriber, #96807)
[Link] (45 responses)
Has the information about the deprecation not been advertised enough? It has been 2 years apparently.
Posted May 17, 2024 14:22 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (1 responses)
If it doesn't happen, and userspace compatibility is broken without regards, then it's really not surprising that people do not want to upgrade their kernels. And the "we do not break userspace" mantra that we always hear some kernel maintainers repeat is, at the very least, inaccurate.
Posted May 17, 2024 16:57 UTC (Fri)
by Tobu (subscriber, #24111)
[Link]
Looking at the commit, the removed options had been showing kernel warnings for the deprecation period.
In the rest of the file, usebackuproot and nologreplay are on the way out as well. It looks like these were intended as rescue options, whereas what systemd wants is something that will make sure images and block devices stay immutable. Wasn't there work to enforce that at the block layer recently?
Posted May 17, 2024 17:00 UTC (Fri)
by mezcalero (subscriber, #45103)
[Link] (42 responses)
Also why in heaven even deprecate this at all? The functionality is still available and supported after all, they just renamed the option. And what's worse, the other big relevant file systems have an option of the same name. Doing the same stuff. Only btrfs is the sole outlier now. Wtf.
Sorry, but you don't get to change kernel interfaces like this and then claim you are a compatibility extremist like the kernel folks "we dont break userspace" mantra suggests.
Posted May 17, 2024 18:07 UTC (Fri)
by hmh (subscriber, #3838)
[Link] (41 responses)
Agreed, and the patch authors agreed too, apparently. The kernel has been [trying to?] warn everyone about it since 5.11. The patch that removed the functionality, added a call to btrfs_warn() in its place.
So, what was removed in 6.8(?) was the btrfs_warn() and option handling. Which is obviously a bug, since it caused an userspace regression, and I fully expect it will be reverted everywhere now that it has been reported as such...
But "announced only though LKML" (which I extend to "source-code-change-flow", i.e. other MLs as well as the git commit log) is not correct. The code clearly attempted to get out a warning to the kernel log, every time anyone tried to mount a btrfs partition with said flag, and done that since 5.11 until the change that caused the current issue.
I wonder if btrs_warn() is being "silenced" by default somewhere? But to an outsider, it looks like whatever the reason, a better way of doing this is needed.
Documentation/ABI/ comes to mind, it is used for sysfs and configfs a lot, maybe it could be used for mount options and filesystem functionality, too...
Posted May 17, 2024 18:48 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (33 responses)
And besides, the main point is a different one: if there are users of a userspace API, do not remove it. Or do, but then stop claiming "we do not break userspace", and don't publish papers wondering why users don't trust new upstream kernel releases and just stay on enterprise stable kernels.
Posted May 19, 2024 3:59 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link] (32 responses)
So that may be what needs fixing in the first place. If new kernel warnings are not detected by the CI, it's for sure aiming at unnoticeable breakage upon upgrades.
Posted May 19, 2024 10:56 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (20 responses)
Posted May 19, 2024 11:17 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link] (19 responses)
Posted May 19, 2024 13:18 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (18 responses)
Posted May 19, 2024 13:59 UTC (Sun)
by mb (subscriber, #50428)
[Link] (16 responses)
Normal interfaces are changed extremely rarely and these obviously are the ones meant by the "do not break userspace" rule.
Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
Posted May 19, 2024 14:11 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (15 responses)
Which by the way, neatly explains why vendor kernels are needed and are in fact the only sane choice, despite what the paper cited in this article says. Nobody should run production payloads on upstream kernels at this point, given basic stuff like mount options just breaks left and right.
Posted May 19, 2024 15:20 UTC (Sun)
by mb (subscriber, #50428)
[Link] (14 responses)
I didn't say that it is. I did not talk about this specific thing, because I don't know anything about it. I was talking about "do not break userspace" in the general form, not in this specific case.
Whether this mount change is a sane change is up to somebody else to judge.
> they don't say "we do not break userspace, apart from
Well, they pretty much do exactly that.
If you want 100% full "don't break userspace" without exceptions, we must basically stop all kernel development now.
Having a "don't break userspace without exceptions" is impossible.
> This is something that you have just made up
No. See my example of trace points.
> this is nonsense
I think it would be good to calm down before continuing the discussion.
Posted May 19, 2024 17:32 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (13 responses)
This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.
> Well, they pretty much do exactly that.
No, they very much do not. Look at all the enthusiastic comments from kernel people pointing to the paper in the article and saying "See? Vendor kernels are BAD, just upgrade to upstream kernels, it's fine really", and when told that new kernel version break applications and that's the real reason why vendor kernels are used, they shrug it away as "impossible, we do not break userspace"
> No. See my example of trace points.
Yes, it is exactly what you did, and there was no mention anywhere of trace points:
> Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
You have made up a new rule according to which it's fine to break systemd or udev (if it's not made up, then just point to where on https://kernel.org/doc/ it is defined), but *unspecified other applications* must continue to work. That is very convenient of course, it's always unspecified other applications that are supported, and the ones that break are never actually supported. That's a very easy way of guaranteeing compatibility - every time something goes wrong just say that case was never actually supposed to continue working and move on.
Posted May 19, 2024 18:10 UTC (Sun)
by mb (subscriber, #50428)
[Link] (11 responses)
There have always been exceptions and I didn't make that up. That's just silly. I even gave you an example (tracepoints).
I respect you for what you do for Linux, Systemd and so on. But you're acting like a child right now.
>it's always unspecified other applications that are supported
Yes. That is exactly like it is.
I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.
>every time something goes wrong just say that case was never actually supposed to continue working and move on.
That's not how things are done, though.
Now you will reply: You have made up yet another rule!
Posted May 19, 2024 18:32 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (10 responses)
Literally nobody has mentioned tracepoints. I mean I'm not even sure that really qualifies as a userspace interface - maybe it does, it would seem strange, but I am not a tracing experts. But it is completely unrelated to mount options being removed.
> I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.
Says who? That is very much not true. Every interface that I can think of is used by multiple unrelated applications. I have no idea where you get this from. Cgroups and namespaces? Throw a rock in the general direction of a container runtime and you'll hit either or both. Netlink? There are as many network and interface managers as there are Linux vendors. Process management? That's been around since literally forever, and see the point about container management again. Mounting filesystems? fstab is older than me, I am quite sure.
'We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init' doesn't sound as catchy, does it now?
> It's done on a case by case basis.
I am well aware. And the triaging of that case by case goes like this: did it affect the machine that Linus happened to boot on that week? If so, it gets reverted and unpleasant emails are shot left and right. Else, nothing to see, move along.
Posted May 19, 2024 18:48 UTC (Sun)
by mb (subscriber, #50428)
[Link] (9 responses)
What the? I did. I mentioned them as an example for a non-stable interface. After you have asked.
> I mean I'm not even sure that really qualifies as a userspace interface
Oh. I get it. *You* want to define what a userspace interface is and what not.
That is silly.
> Says who?
Me. But I'm not sure why that matters.
> We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init
Well. I have never experienced a breakage due to a kernel interface change.
That is my experience.
Posted May 19, 2024 19:31 UTC (Sun)
by bluca (subscriber, #118303)
[Link] (8 responses)
Again, I do not know the first thing about tracepoints and have zero interest in that. Maybe it's a supported interface, maybe it's not, I really cannot say, nor care, and can't see what it has to do with mount options.
> *You* want to define what a userspace interface is and what not.
No, userspace defines what is a userspace interface, as per Hyrum's Law.
> Me. But I'm not sure why that matters.
Because it's just wrong, as explained, there are no "special custom interfaces" being used anywhere, just bog standard stuff used by most components of an operating system.
> I run a two decades old binary and it still works fine.
'We do not break userspace, as long as userspace is mb's statically linked printf("hello world\n") /sbin/init' still not quite as catchy I'm afraid
Posted May 19, 2024 20:23 UTC (Sun)
by mb (subscriber, #50428)
[Link] (7 responses)
Posted May 20, 2024 9:30 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link] (2 responses)
I've had to fix software because of a kernel update, because some files in /sys were moved. But for some reason that doesn't count.
Posted May 20, 2024 9:41 UTC (Mon)
by mb (subscriber, #50428)
[Link]
That is exactly what I was saying. Yet, I'm apparently wrong.
Posted May 20, 2024 9:45 UTC (Mon)
by bluca (subscriber, #118303)
[Link]
Where is that subset defined?
Posted May 20, 2024 11:09 UTC (Mon)
by wtarreau (subscriber, #51152)
[Link]
Welcome to discussions with bluca. Agressivity, half-reading of arguments, and accusations often arrive in the second or third message when he disagrees with you. There are such people who constantly criticize Linux and who would probably do good to the community by switching to another OS of choice :-/
Posted May 20, 2024 11:26 UTC (Mon)
by bluca (subscriber, #118303)
[Link] (2 responses)
Posted May 20, 2024 11:54 UTC (Mon)
by mb (subscriber, #50428)
[Link] (1 responses)
Wow. This is a new level.
Posted May 20, 2024 12:57 UTC (Mon)
by corbet (editor, #1)
[Link]
Posted May 23, 2024 15:48 UTC (Thu)
by anton (subscriber, #25547)
[Link]
Whether that means that vendor kernels are needed, or that one can use upstream kernels if one is selective about them is up to the vendors and their customers to decide.
Posted May 20, 2024 9:28 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link]
Posted May 23, 2024 10:12 UTC (Thu)
by tlamp (subscriber, #108540)
[Link] (10 responses)
A major improvement here could consist of adding a common infrastructure in the kernel to track deprecation.
This data should be assembled on kernel build, possibly even made available on runtime in one of the virtual FS, would allow distros and projects with a lot of kernel interaction like systemd to actually track those and notice those for sure, as scanning for arbitrary warnings that can change wording every point release is just an ugly mess with lots of false-positives/negatives waiting to happen.
If it was available on runtime then checks could be added to the pre- / post-installation scripts/hooks of the kernel distro packages so that users can get a much more noticeable warning printed out on upgrade if their system is affected by such an option removal.
Posted May 23, 2024 12:35 UTC (Thu)
by mb (subscriber, #50428)
[Link] (6 responses)
We had such a deprecation list under Documentation, but I think it got removed a couple of years ago.
Posted May 23, 2024 13:20 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (5 responses)
Far better to do it in the kernel itself. Probably not easy, but move all deprecated stuff into a (or several) modules behind an option "deprecated-6.8" or whatever. Bleeding edge sets all these to "no", and either someone steps up and supports it (removing the deprecated option), or it bitrots until someone says "oh, this broke ages ago, let's delete it".
And then, if there's stuff you really want to get rid off but people need it, every year or so it gets upgraded to "deprecated latest kernel", so hopefully people stop using it and it finally drops out of sight ...
Cheers,
Posted May 23, 2024 14:45 UTC (Thu)
by mb (subscriber, #50428)
[Link] (4 responses)
It would change nothing.
>let's delete it
And that is exactly when people will first start to notice.
Posted May 23, 2024 20:56 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (2 responses)
Except it changes everything
> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
You just said it!
The distributions are enabling something that is disabled by default? They're accepting responsibility for keeping it working.
Developers are enabling something that is disabled by default? They're accepting the associated risks.
People are enabling something that is marked "deprecated"? They're being placed on notice that it's being left to bit-rot.
The fact that people have to actively enable something that developers clearly don't want activated means that anybody using it will have three choices - migrate their code away, take over maintenance, or do an ostrich and bury their heads in the sand. Users will still be able to be complain "I didn't know", but their upstream won't have that excuse.
Cheers,
Posted May 23, 2024 21:01 UTC (Thu)
by mb (subscriber, #50428)
[Link] (1 responses)
Do you realize, that most kernel options are disabled by default?
>The fact that people have to actively enable something that developers clearly don't want activated means
It means that developers don't have a clue what people (users!) actually want and need.
Closing your eyes won't make the demand go away, unless you are less than three years old.
Posted May 23, 2024 22:41 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
And how many of those options have "deprecated" in their name? Surely that's a massive red flag.
> It means that developers don't have a clue what people (users!) actually want and need.
And how many developers are employed by (therefore are) users? I believe Alphabet employs loads. Meta employs loads. Most of the kernel developers I have contact with are employed by large end users. It's a little difficult to be oblivious of your own needs. (Some people manage, I'm sure ...)
How difficult is it - to set a "not enabled" flag that cannot be accessed without some sort of warning that this flag will enable deprecated functionality Surely it's not beyond the wit of your typical kernel developer? That's ALL that's required.
Cheers,
Posted May 24, 2024 8:15 UTC (Fri)
by tlamp (subscriber, #108540)
[Link]
I don't think so, mostly because my spitballed proposal was not targeted at solving the "distros get never hurt by deprecation", as IMO that cannot be solved, besides not doing any deprecation at all anymore which hardly is a good solution. Rather, I wanted to target the "how things get communicated and noticed" part and having ab extra compile option with something like "deprecated-6.8-removal-6.12" in the name could actually be quite good for that. The build configs are often tracked and even diffed, and as simple single file can be easily grep'ed against _DEPRECATION_ and then diffed for ones that would trigger soon or new ones, probably even by a CI like systemd uses.
I.e., the status quo is having warnings for deprecation, which can be brittle and are not easy to digest/parse, having that info in an easier to digest manner would help a lot as tools/distros that depend on such options can easily find out when a used one will vanish soon(ish), then they also have no excuse about being unprepared.
> And that is exactly when people will first start to notice.
If their distro or tooling did not do their work then yes, but it wouldn't be the fault of the kernel having a messy deprecation process. IME most bigger distros or big projects like systemd want to avoid that, so if they'd have the definitive information required to do so in just somewhat digestible way, then I really think that most would actually act on that.
Posted May 23, 2024 14:57 UTC (Thu)
by smurf (subscriber, #17840)
[Link] (2 responses)
I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).
A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.
No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.
Posted May 24, 2024 8:01 UTC (Fri)
by tlamp (subscriber, #108540)
[Link]
As said, I'd assemble on build so those options that are not relevant for a kernel build config would not be in there (or if still wanted could be tracked differently, i.e., with an extra flag or separate list)
> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).
With a declarative list this is trivial to create, as a tool can just scan all modules, mount, ... options and compare if anything explicitly set is in the static list. So if you want this then a static list is IMO really the best way to achieve it, one first needs the definitive list of information before being actually able to do something with it, keeping it dumb on the kernel side and include as much as possible allows (user space or build) tooling to actually do the smart checks.
> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.
Not sure how this minus bikeshedding is any different what I proposed, but I like that we agree in general.
> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.
I'd named JSON simply as an option, explicitly also stated that a simple list could do. But, I named JSON as 1. generating it is trivial (compared to parsing, which isn't hard either, but not trivial anymore) 2. Allows more flexible extension for whatever data or use case gets relevant in the future without having to do a /proc/deprecation2 3. in my projects I try to avoid another not-invented-here format with it subtleties to be added, but sure if it's a simple CSV list that gets generated by the common infra (i.e., not under the control of each kernel dev with their own opinions of the day) then fine by me (not that my acknowledgment would matter anything :)
Posted May 24, 2024 13:45 UTC (Fri)
by donald.buczek (subscriber, #112892)
[Link]
This would be perfect! We would see, what we need to address in our fleet (we are not using a distribution). But distributions would have something to build on, too. They might create a feedback path of this information from the systems of their users to the distribution. The basis for everything is that the information "you are using a mechanism, which will go away" is made available in a structured way.
It is important that the information cannot only be found by digging through masses of unstructured text in mailing lists, documentation, NEWS files, dmesg or other sources and then having to analyze in each individual case whether it is is relevant to you at all.
Posted May 23, 2024 14:01 UTC (Thu)
by eru (subscriber, #2753)
[Link] (6 responses)
These days just about every distribution "helpfully" hides boot messages behind a splash screen, so few users will ever see such warnings. Might as well not be there.
It is a mystery to me why this is done (and after I found out how, I disabled it). The stream of messages is actually a nice progress indicator, and if they get stuck at some point, one gets some idea about what might be wrong.
Posted May 23, 2024 14:49 UTC (Thu)
by mb (subscriber, #50428)
[Link] (2 responses)
Because nobody can read over 9000 messages in 0.5 seconds.
>The stream of messages is actually a nice progress indicator
No, it isn't. Today's boot process is so fast (unless you're stuck with a legacy init system), that console output is pretty much useless and just slows things down, at best.
Posted May 23, 2024 15:19 UTC (Thu)
by eru (subscriber, #2753)
[Link] (1 responses)
Posted May 23, 2024 15:26 UTC (Thu)
by mb (subscriber, #50428)
[Link]
I'm not saying that messages are not useful for debugging. And for that exact reason it's always possible to enable a verbose boot.
A kernel deprecation message would certainly go by unnoticed during the burst of messages.
Posted May 23, 2024 15:14 UTC (Thu)
by zdzichu (subscriber, #17118)
[Link] (2 responses)
Developers, on the other hand, are the target of such messages. I cannot imagine developing a low-level component – like systemd – and not running `dmesg` from time to time. If you use kernel features more than anyone else, you should pay more attention than anyone else.
Posted May 23, 2024 21:14 UTC (Thu)
by mezcalero (subscriber, #45103)
[Link] (1 responses)
But even beyond that: there's *so* *much* *stuff* in dmesg right now. It's a wall, a deluge of text. Don't expect me to read all that. I only look there when I am looking for something, and then I usually do "journalctl -ke", and it never showed up there.
Posted May 23, 2024 21:48 UTC (Thu)
by pizza (subscriber, #46)
[Link]
Fedora just flipped the default, starting with their 6.8 kernels IIRC.
Reverting it is just a matter of:
sysctl kernel.dmesg_restrict=0
> It's a wall, a deluge of text. Don't expect me to read all that.
Fortunately, it's rare when anything beyond the final dozen or so lines matters. (eg the messages that show up when you plug something in, or if something goes wrong...)
Posted May 17, 2024 14:28 UTC (Fri)
by willy (subscriber, #9762)
[Link] (14 responses)
Posted May 17, 2024 14:55 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (4 responses)
I mean we change public interfaces like this in systemd too every now and then - but we just keep the old configuration option around too, undocumented, and either map it to the new one if any, or make it a no-op.
Posted May 17, 2024 15:06 UTC (Fri)
by mb (subscriber, #50428)
[Link] (3 responses)
Posted May 17, 2024 15:17 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (2 responses)
Posted May 17, 2024 16:14 UTC (Fri)
by mb (subscriber, #50428)
[Link] (1 responses)
No. It depends on the change itself whether a silent ignore is better or not. It might be worse to silently ignore something the user has requested.
Posted May 17, 2024 16:22 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted May 17, 2024 16:20 UTC (Fri)
by joib (subscriber, #8541)
[Link] (6 responses)
Posted May 17, 2024 16:53 UTC (Fri)
by adobriyan (subscriber, #30858)
[Link] (5 responses)
Posted May 18, 2024 19:46 UTC (Sat)
by NYKevin (subscriber, #129325)
[Link] (4 responses)
0. /etc/fstab is a configuration file. It's not the kernel's place to modify config files (software like Puppet or Ansible will change it back).
(No, paging the sysadmin at 3 AM is not a reasonable response to this situation. Nothing is actually broken!)
Posted May 18, 2024 20:29 UTC (Sat)
by adobriyan (subscriber, #30858)
[Link] (1 responses)
I know, relax :-) it was a joke.
Posted May 19, 2024 4:01 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link]
Posted May 19, 2024 9:07 UTC (Sun)
by smurf (subscriber, #17840)
[Link] (1 responses)
Posted May 20, 2024 17:35 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link]
Which is also, sort of, the point I'm trying to make here. It is no longer accurate to divide userspace into "applications" and "stuff the sysadmin manually fiddles with." Sysadmins are not manually fiddling with mount options etc. these days. That's all managed by some other piece of software that isn't "the application," but can still break and cause problems all the same. E.g. k8s, Puppet, Docker, etc.
Posted May 18, 2024 17:11 UTC (Sat)
by shemminger (subscriber, #5739)
[Link]
Posted May 19, 2024 4:00 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link]
Posted May 18, 2024 8:05 UTC (Sat)
by zdzichu (subscriber, #17118)
[Link]
This is not true. I have a number of systems running Fedora 39, 40 and rawhide, with btrfs rootfs and other mountpoints. They all run 6.8.x kernels released by Fedora and none has problem with mounting filesystems. And as pointed out, the patch you mention removes 3 deprecation warning that were printed to dmesg for the past couple years. If one doesn't pay attention to the dmesg, kernel developers have no other ways of contacting him.
Posted May 18, 2024 9:33 UTC (Sat)
by niner (subscriber, #26151)
[Link]
Posted May 20, 2024 12:03 UTC (Mon)
by georgyo (subscriber, #121727)
[Link]
I see a lot of people participating in this discussion, but not a lot of people actually affected.
Posted May 21, 2024 13:29 UTC (Tue)
by corbet (editor, #1)
[Link]
Posted May 22, 2024 0:03 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (2 responses)
Posted May 22, 2024 5:16 UTC (Wed)
by smurf (subscriber, #17840)
[Link] (1 responses)
Posted May 22, 2024 11:52 UTC (Wed)
by bluca (subscriber, #118303)
[Link]
Posted May 17, 2024 15:22 UTC (Fri)
by prarit (subscriber, #27126)
[Link] (8 responses)
and
"No analysis of [bug] severity has been undertaken but it is probable that at least some of them are serious and important."
Are there missing security fixes? Certainly, yes, but the above quotes makes me question the analysis.
A better, and more reasonable approach would be to look at the areas of the kernel that RHEL8 enables, and consider the subset of CVE/Security issues reported as critical and above. #IMO of course. I'm not a security expert.
Posted May 17, 2024 16:26 UTC (Fri)
by sandeen (guest, #42852)
[Link] (2 responses)
Just looking at filesystems, there are about 700 "unfixed bugs" in RHEL8 for filesystems which are /not shipped in RHEL8/ based on Fixes: tags in commits to those filesystems since v5.14.
Failing to filter out bugfixes for subsystems which are configured off in the RHEL kernel makes this "analysis" lazy at best, disingenuous or malicious at worst.
Posted May 17, 2024 17:30 UTC (Fri)
by sandeen (guest, #42852)
[Link] (1 responses)
There are over 1500 "not fixed bugs" since v4.18 in not-shipped filesystems alone.
That is fully 1/3 of the "unfixed bug" count in their whitepaper without even looking at other unshipped drivers, subsystems and architectures.
There may be an interesting analysis to be made, but this whitepaper doesn't seem to be it.
Posted May 19, 2024 4:06 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link]
Posted May 17, 2024 17:38 UTC (Fri)
by jra (subscriber, #55261)
[Link] (3 responses)
We did do some analysis. Look at the keyword list at the end of the paper:
Appendix 1. List of missing fixes in RHEL 8 by keyword search
These *are* security problems.
FYI. We considered these lists serious enough that I've sent gpg encrypted emails to the security contacts at Red Hat, SuSE, Oracle, Rocky and Alma letting them know exactly what commits these are.
Posted May 17, 2024 19:26 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (2 responses)
Posted May 17, 2024 21:06 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link] (1 responses)
Making this model even more difficult to maintain is because it is sometimes difficult to even identify what needs to be backported or not.
But even looking for things that are flagged as "Fixes:" is flawed. I compared a region of commits in upstream an searched for commits that fixed use-after-free bugs and kernel-panics. Only about half of them even contained a Fixes: tag and indicated they were a bug-fix.
Posted May 20, 2024 14:51 UTC (Mon)
by sandeen (guest, #42852)
[Link]
Posted May 17, 2024 20:23 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link]
Some of them even contain description in the commit message how to trigger the bug and crash the kernel from unprivileged userspace.
Posted May 17, 2024 19:05 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (37 responses)
Ok I guess this clarifies how serious the white paper is.
The full CVS, then svn, then git tree has never been considered complete and corresponding source code.
Posted May 17, 2024 19:56 UTC (Fri)
by jra (subscriber, #55261)
[Link] (36 responses)
This was covered by lwn back in 2011.
https://lwn.net/Articles/430098/
"Red Hat is making things harder by shipping its RHEL 6 kernel source as one big tarball, without breaking out the patches."
Posted May 17, 2024 20:01 UTC (Fri)
by jra (subscriber, #55261)
[Link] (16 responses)
https://lwn.net/Articles/430192/
"- The kernel tarball inside the srpm is created from a git tree that is only accessible to Red Hat engineers.
- The orders to do this, to make it harder to rebuild the kernel with and without patches, and to make it harder to extract specific patches from the Red Hat kernel came from the top. This is with the knowledge of, and by the order of, the CEO: Jim Whitehurst."
Posted May 17, 2024 21:15 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (15 responses)
By the same token, Bash is not free software. Its releases are made as giant git commit dumps: https://git.savannah.gnu.org/cgit/bash.git/commit/?id=886...
Posted May 18, 2024 21:27 UTC (Sat)
by rschroev (subscriber, #4164)
[Link] (13 responses)
Posted May 18, 2024 23:06 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (12 responses)
Yes, and that tree (I linked to it) only has giant dumps. Just like RedHat.
Posted May 18, 2024 23:23 UTC (Sat)
by mb (subscriber, #50428)
[Link] (11 responses)
No.
The authors can do whatever they like to their code.
Posted May 18, 2024 23:25 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (10 responses)
If giant tree dumps are not the preferred form, then Bash is arguably violating the GPL.
Posted May 18, 2024 23:35 UTC (Sat)
by mb (subscriber, #50428)
[Link] (9 responses)
I think they pretty much do, due to FSF Copyright assignment rules.
Posted May 18, 2024 23:53 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (7 responses)
Posted May 20, 2024 8:58 UTC (Mon)
by paulj (subscriber, #341)
[Link] (6 responses)
Posted May 20, 2024 9:34 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (5 responses)
You know, that thing that totally destroys the credibility of our "moral guardians".
Cheers,
Posted May 20, 2024 10:51 UTC (Mon)
by paulj (subscriber, #341)
[Link] (4 responses)
Posted May 20, 2024 12:38 UTC (Mon)
by pbonzini (subscriber, #60935)
[Link] (3 responses)
Posted May 20, 2024 12:46 UTC (Mon)
by paulj (subscriber, #341)
[Link] (2 responses)
If you mean "For an executable work, complete source code means ..." - that's defining the complete source code, but not the preferred form for modifications.
You can have the complete source code, but not have it in the preferred form for modification.
Posted May 20, 2024 13:54 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (1 responses)
True, the preferred form of keeping track of your modifications, distributing them, etc.etc.etc., once was patches and tar files; now it's git and pulling commits. The problem is that the GPL doesn't talk about any of that. It merely requires "a" medium commonly used for software exchange. Not "the medium everybody else who's contributing to this particular piece of source code is using".
So for better or worse, you can appeal to common sense and interoperability and the equivalent of sportsmanship and whatnot and I'll be the first person to be argue right along with you. But the GPL? sorry you pour cold water on your fire but you can't use it to demand git trees from anybody.
Disclaimer: IMHO and IANAL and all that.
Posted May 20, 2024 14:36 UTC (Mon)
by paulj (subscriber, #341)
[Link]
You may indeed argue that the preferred form is a text file.
However, if you are a company that has deliberately stripped away context that you yourself use for hosting and modifying said source code as part of your commercial operation, and you have made that deliberate act in order to frustrate others who also wish to modify said source code, then I - if it mattered to me and I were a copyright holder - could well argue to a court that the form you have provided is clearly not the form you prefer to use yourself for making modifications, and that you have therefore violated the licence I gave you.
E.g., I - as your upstream - could well be trying to follow along with what you're doing to my code, so that I can cherry-pick the good bits to integrate back into the code. Your deliberate obfuscation, done to keep for yourself the preferred form of modifications and hence frustrate others from picking out what you changed, I would argue as being clearly against the wording of the licence.
The form you prefer to use internally, versus the form you chose to release, would be prima facie evidence of your failure to honour the licence.
Who would win, who knows. But I don't think your position would be 100% safe.
Posted May 19, 2024 0:01 UTC (Sun)
by bluca (subscriber, #118303)
[Link]
Posted May 19, 2024 7:27 UTC (Sun)
by gioele (subscriber, #61675)
[Link]
Bash's repository has two main branches:
* `master` (<https://git.savannah.gnu.org/cgit/bash.git/log/>), that follows the historical 1-mega-patch-per-release paradigm, and
Posted May 17, 2024 20:36 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (16 responses)
< That would be a clonable git tree of the RHEL kernel source code.
Based on what novel reasoning?
If the "complete and corresponding source code in the preferred form for making modifications" included the full source code history, that history would have to be distributed to whoever receives a binary. Not just for Linux: for all packages. And it would have to include *the whole upstream history* if you are performing commercial distribution and therefore must follow sections 3a or 3b of the GPLv2.
Now, it's not just Red Hat that does not do that. There is simply no commercial distribution that does that, not even counting hardware manufacturers that use GPL'd software in the gadgets they sell. If you disagree, sue them all including your employer. Or more likely, any serious lawyer would have told you that your assertion is ridiculous.
More technically: the preferred form for making modifications is not the preferred form for analysis. So for example, when I backport a fix to a Fedora package, I extract the tarball and apply any Fedora-specific patches with "fedpkg prep", then typically follow with
I use the git tree from upstream (the tree that I analyzed to find the bug and the fix) and not the git history from Fedora (which "fedpkg prep" has thrown away). The same is true for the RHEL kernel: I _have_ the git history of course, but in practice I use the upstream git tree (which I am analyzing but not modifying) and I don't need to look at the history of the RHEL kernel (which I'm modifying).
> "Red Hat is making things harder by shipping its RHEL 6 kernel source as one big tarball, without breaking out the patches."
"Distribution in this form should satisfy the GPL". Literally three lines below.
Posted May 17, 2024 21:06 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link] (15 responses)
It is harder for an end user to easily analyze a kernel to see what exactly it contains. Making it harder to answer questions like "does this kernel contain this particular fix".
Posted May 17, 2024 21:25 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (14 responses)
No, I am sorry but what Jeremy said has serious legal implications. The wording he used (especially with the confirmation that he chose it deliberately) is a clear accusation of violating the GPL. Don't defend the indefensible, please.
I agree that the lack of granularity in CentOS 8 is bad. Fortunately, in CentOS Stream 9 the development is done again entirely in the open. Of course that is too early in the release stream to show the slowdown that you point out, but let's see where we will be three years from now. :)
Posted May 17, 2024 21:44 UTC (Fri)
by jra (subscriber, #55261)
[Link] (13 responses)
https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analy...
People of good conscience (and I certainly include you in this) can have different opinions on this matter.
Posted May 17, 2024 21:54 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (5 responses)
Also, the Conservancy blog post does not really debate the fact that the RHEL business model complies with the GPL. Rather, it complains about the difficulty of *verifying* from the outside that Red Hat does indeed comply in practice and not only on paper. I think it's a reasonable complaint, especially from the point of view of an entity like Conservancy—and considering that the nuances of free software licensing are not obvious to all of the employees on Red Hat, on neither the engineering side nor the sales side.
That said, it's also not a standard to which many companies are held, and perhaps that's a point of honor for Red Hat. :)
Posted May 17, 2024 22:16 UTC (Fri)
by jra (subscriber, #55261)
[Link] (4 responses)
That's a really good point, and one to which I wholeheartedly agree. I *do* expect better of Red Hat. As I have often said, I have many Red Hat engineering friends and a great respect for the engineering talent and prowess of Red Hat.
It disappoints me greatly to see Red Hat moving away from the ethos of improving the availability of Free Software, hiding git trees from other engineers, and generally seeming to move away from open collaboration.
Still, I believed in Google Open Source too, and look where that got me :-) :-).
Posted May 18, 2024 0:43 UTC (Sat)
by Wol (subscriber, #4433)
[Link] (3 responses)
Cheers,
Posted May 18, 2024 4:51 UTC (Sat)
by jra (subscriber, #55261)
[Link] (1 responses)
Reacting due to fear often leads to the wrong response.
Posted May 18, 2024 8:53 UTC (Sat)
by Wol (subscriber, #4433)
[Link]
And investigating before reacting (I don't know whether RH did, but they took their time reacting) often leads to surprising conclusions and actions.
(I'm currently working as a business analyst. It's my job to react to events like this. And I'm expected to have a good justification for my reactions.)
People often confuse cause and effect. Or blame something totally irrelevant. If RH determined that Oracle's actions were causing serious damage to their bottom line, then they had to do something about it - regardless of the cost. If you really are facing an existential threat, you're not going to care about your reputation.
(And as others have said, I totally agree that it's nice that RH are held to a higher standard than others, but I also think that's unfair! :-)
Cheers,
Posted May 20, 2024 9:00 UTC (Mon)
by paulj (subscriber, #341)
[Link]
Posted May 17, 2024 22:00 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (6 responses)
Given that "dumping the source over the wall" is perfectly okay for projects without source control, except not popular with recipients, you are now getting into the legally extremely hairy situation where what is perfectly okay for one project, but allegedly illegal for another ...
Do we really want to go down that rabbit hole?
Cheers,
Posted May 17, 2024 22:20 UTC (Fri)
by jra (subscriber, #55261)
[Link] (4 responses)
I remember the days of throwing tarballs on an ftp site (and indeed did so myself :-). But these days that really isn't how collaboration is done.
Git is king, for better or worse (better, IMHO). A git tree really is the preferred form of the work these days (or insert distributed source code control system of your choice for more obscure projects).
Posted May 17, 2024 22:32 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (1 responses)
If the meaning has changed, then no one is distributing GPLd software commercially and complying with the supposed new meaning. Which to me is a pretty good sign that the meaning has not changed, and that's for good reasons.
Posted May 18, 2024 9:31 UTC (Sat)
by smurf (subscriber, #17840)
[Link]
Yes, you cannot use the kernel source in any meaningful way without git these days. Unfortunately, that's not covered by the GPLv2, just like Tivofication isn't (and that one is a far worse problem). You need the GPLv3 for that – which the kernel cannot and will not transition to.
Thus while yes there's a moral / good practices / be-nice-to-the-community obligation to provide the git archive, legally? almost certainly not.
Posted May 19, 2024 16:46 UTC (Sun)
by Wol (subscriber, #4433)
[Link]
Why?
> I remember the days of throwing tarballs on an ftp site (and indeed did so myself :-). But these days that really isn't how collaboration is done.
And this is called pushing your views down other peoples throats. (Personally, I think even doing "on your own" development benefits from version control, BUT IT'S NOT MY PLACE TO TELL OTHER PEOPLE WHAT TO DO!)
What's that advice I've come across before? "You think you're a sensible, rational human being. When you disagree with someone, you should assume they are sensible and rational too. So any disagreement must be down to you having a different set of facts from them. At which point, how do you know it's not YOU who've got the facts wrong."
Face it, even today there are people who don't/won't use version control. Even today there are people (especially sole copyright holders) who will throw a tarball over the fence and say "This will compile and work, I'm done". If that was acceptable in the past, it has still to be acceptable today. And if it's acceptable today, why is it acceptable for some and not others?
You can say it's not being a good citizen, not being fair (I don't know if I'd agree with you), but if it's in reaction to someone else playing dirty and not being a good citizen, then I'd much rather you went after the *real* culprit, the cuckoo in the nest, not the victim trying to make the best of a bad job. It's called "blame the victim", don't cha know...
>Git is king, for better or worse (better, IMHO). A git tree really is the preferred form of the work these days (or insert distributed source code control system of your choice for more obscure projects).
And for those people who don't use version control (beyond datestamped backups) ???
Cheers,
Posted May 21, 2024 9:18 UTC (Tue)
by jamesh (guest, #1159)
[Link]
You are essentially arguing that it isn't enough to distribute the source code corresponding to the binaries, but that the source code to all previous versions is also required. That seems like it'd be a problem for far more than just RHEL kernel packages. Off the top of my head: There are open source licenses that require that you split out patches from the upstream release (e.g. the QPL), but the GPL is not one of them.
Posted May 20, 2024 9:05 UTC (Mon)
by paulj (subscriber, #341)
[Link]
Good they made the kernel code available in the preferred form again. It still stinks they ever thought playing those tricks was acceptable.
Posted May 17, 2024 20:39 UTC (Fri)
by pizza (subscriber, #46)
[Link] (1 responses)
You're wrong, according to the plain text of the GPL:
"The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable."
I'll also add a brief snippet of what was so well written in https://lwn.net/Articles/431651/ :
"The language in the license was added to prevent someone from shipping a printed version of the source or other forms of distributing the source that amount to useless for an end user. Sure it's the source all right, but not machine readable nor editable or usable to make a modified distribution from. The language of the license protects us, the end users, from these kinds of actions by distributors of GPL applications."
RH's kernel SRPMs are just as GPL compliant as the tarballs provided on the front page of kernel.org. And most FSF-provided software too, I might add.
Posted May 19, 2024 16:53 UTC (Sun)
by Wol (subscriber, #4433)
[Link]
> For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable."
Where is ANY mention of the history? Where is ANY mention of commit messages? Where is ANY mention of any documentation, user or developer?
As I read that, if I fed the source through a pre-processor that stripped all comments, anything like that, that left just pure source, that would still comply with the licence.
Yes it's nice to have all the extra stuff, but that term seems quite clear - as long as what you're given compiles, then it is COMPLETE. In other words, a git tree is not the "complete source" because if you fed that into a compiler, it would rather seriously choke !!!
I'm always amazed how well the GPL has been crafted - this is just one more example.
Cheers,
Posted May 17, 2024 20:12 UTC (Fri)
by carlosrodfern (subscriber, #166486)
[Link]
As far as I know Debian only ship LTS kernels. Those that mean Debian is doing better keeping up with these CVEs?
Posted May 17, 2024 20:58 UTC (Fri)
by jd (guest, #26381)
[Link] (13 responses)
However, I'm pretty sure we've reached the point where old vulnerabilities are costing businesses and there's also the cost of hiring developers to backport features.
Here's a what-if to consider. What if the distros and the corps spent exactly the same amount of money on a mix of QA engineers and developers for much more recent stable kernels?
In other words, actively find and fix the new bugs in kernels where many of the old vulnerabilities have already been corrected, rather than passively backport a selection of those fixes?
Yes, you would need much more QA, because you're having to find bugs that haven't yet been fixed by anyone. But my suspicion is that this would be a far more productive approach.
However, suspicion and reality don't usually agree. Security issues can be evaded, to a degree, with firewalls, and there may be an economic penalty from corps preferring known defects to unknown ones. You can mitigate known risks, after all.
Furthermore, such an approach depends on competent QA and my experience is that really good QA engineers are rare because if they're that good, they're unlikely to be attracted to QA.
But if long-term stable Linux distros aren't to acquire the same security taint as... certain other vendors I could mention, the status quo won't work and this is the most obvious alternative.
Posted May 17, 2024 21:05 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (9 responses)
Posted May 17, 2024 22:37 UTC (Fri)
by jra (subscriber, #55261)
[Link] (8 responses)
I don't believe that is true. Can you point to any lkml discussions about intentionally breaking userspace (that don't end up with Linus being very cross :-) ?
The kernel community IMHO is extraordinarily careful not to break userspace compatibility.
Posted May 17, 2024 22:42 UTC (Fri)
by corbet (editor, #1)
[Link] (7 responses)
Posted May 17, 2024 22:48 UTC (Fri)
by jra (subscriber, #55261)
[Link] (6 responses)
I think by the time upstream makes it into Greg's kernel trees any userspace breakage has already been found and fixed. I'm not aware of any userspace breakages there (although please correct me if I'm wrong, I'm still pretty new to this kernel stuff).
"You can never be too thin, too rich, have too much disk space or too many regression tests" :-).
Posted May 17, 2024 22:52 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted May 18, 2024 6:47 UTC (Sat)
by marcH (subscriber, #57642)
[Link]
Etc.
Posted May 19, 2024 4:40 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link] (3 responses)
This is commonly visible in various setup scripts that rely on contents of /proc, /sys, output from lsmod etc. In other software sometimes a major upgrade will add support for new options to a keyword and reveal that a previous version would silently ignore any such options and that a typo in a previous version now raises an issue.
There's never a perfect solution to this. If users would only use perfectly documented and intended features, systems and programs would be very poor and boring, and nothing would evolve. They would rightfully complain even more about the occasionally needed breakage due to architectural changes. If they use any single possibility, they spark the addition of new features but face unintended changes more often, and developers have a more difficult time making their code evolve.
A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.
Overall I find that Linux is not bad at all on this front. You can often run very old software on a recent kernel and it still works fine. And at the same time I'm one of those being extremely careful about major upgrades because if I don't see the breakage, I suspect it will happen in my back at the moment I would have preferred not to face it. Nobody wants to discover their microphone is no longer working when joining a visio-conference, or that their 4G adapter fails to initialize when waiting for a train or plane at the station for example. On a laptop I've very rarely experienced issues, except with broken or poorly supported hardware whose behavior would change in various ways along versions. On servers I'm much more cautious because of subtle changes experienced overa long time in bonding/bridging/routing setups, and iptables also showing some more config granularity that reveals in field that you're missing some new config options from time to time. And this is a perfect reason for sticking to a stable kernel when you have other things to do than to retest everything.
Posted May 19, 2024 17:00 UTC (Sun)
by Wol (subscriber, #4433)
[Link] (1 responses)
I don't think you're right here. We'll never find out, of course, because developers (and open source collaborative projects are amongst the worst) are very bad at writing documentation.
> A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.
That sweet spot, as an absolute minimum, needs to include DESIGN documentation. This is where Linus' management style really fails (although given that he's cat-herd in chief I don't know any style that would succeed), because so much of Linux is written by people scratching itches, fixing bugs, and everyone is fixing their immediate problem.
Nobody steps back, and asks "what is the purpose of the module I'm working? What is sensible or stupid? How can I design this to be as *small*, *complete*, and *self-contained* as possible?".
Cheers,
Posted May 20, 2024 11:15 UTC (Mon)
by wtarreau (subscriber, #51152)
[Link]
You're absolutely right on these points. I have asked myself in the past the reasons for this situation in a lot of such software documentation and came to the conclusion that it's based on the fact that aside a few exception, there's mostly no design but an improvement on top of something that already exists. As such, lack of design phase implies lack of design documentation. Sometimes someone spends so much time reversing something they work on that they take a lot of notes and end up creating a doc of the existing design. It does not necessarily justify choices but can be a good point to encourage new participants to update the doc. But that's not *that* common.
I'm not seeing any good solution to this , though :-/
Posted May 23, 2024 16:49 UTC (Thu)
by anton (subscriber, #25547)
[Link]
Fortunately, the kernel aspires to a better principle, the often-cited "we don't break user space". And given that we have no way to automatically check whether a program conforms to the documentation, that principle is much more practical (and the practice of gcc is also in that direction, but only for "relevant" code).
Concerning accidental features, a careful design of interfaces avoids those. E.g., one can catch unintended combinations of inputs and report them as errors. Or define and implement useful behaviour in such a case. Exposing some arbitrary accident of the implementation in such cases leads to the difficulties you point out. Given that Linux' interface to user space is also a security boundary, carefully defining interfaces and avoiding exposing implementation artifacts is a good idea anyway.
Posted May 17, 2024 21:23 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link] (2 responses)
I think this is where we need to go, and in a sense it would be a continuation on a collaborative process that has been happening for a while.
I think would be good if we extended this also to the packaging of stable vendor kernels. Instead of everyone maintaining their own private special kernels and all backporting their own selected small subset of important fixes, everyone could instead collaborate on a set of upstream stable kernels and ensure that they all get all the fixes that are needed.
Posted May 17, 2024 21:26 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link]
Posted May 20, 2024 12:20 UTC (Mon)
by Wol (subscriber, #4433)
[Link]
"You can't QA quality into a product".
"10 minutes spent on design knocks 1hr off development".
"Every problem that slips through one phase takes 10 times longer to correct in the next".
As I moaned above, design, Design! DESIGN. If that money was spent on *documentation* and *design* it would probably find far more bugs, far more quickly, and what's more important get them fixed (probably by other people :-)
The current setup encourages a kernel held together by baling wire, duck tape, and sealing wax ...
Cheers,
Posted May 17, 2024 23:00 UTC (Fri)
by ronniesahlberg (guest, #171541)
[Link] (2 responses)
I think one of the underlying assumptions is that "all bugs are found and fixed relatively soon-ish, probably".
I certainly thought that "almost all bugs are found and fixed relaltively quickly" before I started looking at data.
Fun with numbers:
Posted May 19, 2024 17:49 UTC (Sun)
by willy (subscriber, #9762)
[Link]
Posted May 20, 2024 15:00 UTC (Mon)
by sandeen (guest, #42852)
[Link]
I think it would be fascinating run your same analysis on the 4.19 LTS kernel, which is supported until the end of this year, IIRC.
Posted May 17, 2024 23:32 UTC (Fri)
by roc (subscriber, #30627)
[Link] (6 responses)
This change was released in 6.7.0. It significantly changes the behavior of mmap(): many cases of large anonymous mappings are now much more aligned than they used to be. The rr test was broken because if you specify an address hint without MAP_FIXED and there is nothing already mapped in the desired range, mmap() used to always create the mapping at the hinted address, and now it doesn't. Sure, none of this is specified behavior, but it is a change that has serious effects on userspace.
In fact it led to a number of regression fixes appearing in 6.7.6:
The timeline is interesting. The original change was committed on December 14 and appeared in the 6.7 release on January 7. The regression fixes were released on February 23. So apparently it's considered OK to make a change like this just three weeks before a kernel major release. And apparently quite serious regressions can take several weeks to be fixed after a major release.
More importantly, I think someone should have realized that, given kernel tests were breaking, other userspace code is probably breaking and the original change should at least have been reverted from the stable kernel branches.
So yeah, it's completely rational for users to keep their distance from upstream kernels.
Posted May 17, 2024 23:40 UTC (Fri)
by roc (subscriber, #30627)
[Link]
Something is wrong.
Posted May 18, 2024 13:54 UTC (Sat)
by vbabka (subscriber, #91706)
[Link] (2 responses)
Is the rr regression still happening or was rr changed to use MAP_FIXED? I can imagine we could interpret the address hint (without MAP_FIXED) as a stronger signal than the new huge alignment, if that's what missing. If the regression is due to a different mmap() getting aligned differently and thus influencing the following mmap() with address hint, it's more complicated.
Posted May 18, 2024 21:05 UTC (Sat)
by roc (subscriber, #30627)
[Link] (1 responses)
But I do think it would make sense to honour the hint if possible. If that's what the user wants, why not give it to them?
Posted May 23, 2024 7:46 UTC (Thu)
by taladar (subscriber, #68407)
[Link]
Posted May 18, 2024 19:51 UTC (Sat)
by NYKevin (subscriber, #129325)
[Link] (1 responses)
> With a sufficient number of users of an API,
Posted May 19, 2024 4:43 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link]
Posted May 19, 2024 5:04 UTC (Sun)
by wtarreau (subscriber, #51152)
[Link] (1 responses)
To make it simple, any security fix is likely to break your system's stability or compatibility. We all know machines at various places running totally outdated, unbreakable kernels that have never experienced a single crash in multiple years and that nobody dares to update by fear of starting to experience stability issues.
Stability is only measured by user feedback and complaints (and a little bit by testing, to eliminate the most visible problems upfront). Security on the other hand indicates how hard it is to abuse privileges on a given system, and is addressed by restricting what users were allowed to do in subtle ways, which is why, if workloads were relying on a side effect, a security fix can break them.
My vision on this is that mainline is the most secure because it's the place where all known bugs are fixed, and it's also the least stable since it's where all yet-to-discover regressions are present. Distro kernels are the most stable due to missing (purposely or not) some changes, and are likely as well the least secure since any mistaken analysis of a fix that leads to it being skipped (or included without an important dependency) may leave a hole open.
Stable/LTS kernels are between the two and are *expected* to contain all known fixes in reasonable time while *trying hard* not to break anything. Stable kernels receive lots of automated testing but probably not as much as distro kernels. And they can occasionally miss some security fixes present in mainline that were not analyzed as such for example (e.g. accidental fixes). Their real strength (sometimes debated here) is that patch authors are offered the option to comment on or reject patches to be included, something that does not happen with distro kernels (nor any other private patch collection). But as they move faster, they're naturally more prone to regressions than distro kernels, though these regressions are quickly detected, reported, and addressed (often by simply dropping the offending patch).
My practice on this is to deploy a few versions old LTS kernels for which no breakage was apparently reported, and to stick as long as possible to the same branch. For now this has been working remarkably well.
Posted May 23, 2024 7:50 UTC (Thu)
by taladar (subscriber, #68407)
[Link]
Every backport is also a new, untested version, one with changes by someone with significantly lower understanding of either the original code base or the fix than the person working on that part of the code on the latest version. Distro kernels see significantly less testing than the latest version does because of its limited audience which is largely comprised of people who only want it once it has been tested.
Posted May 20, 2024 15:15 UTC (Mon)
by smoogen (subscriber, #97)
[Link] (2 responses)
In looking at the white paper, I was wondering why the comparison was done between Red Hat's which was 'based' off of 4.18 and the 5.x series. I would have thought a better analysis would have been to see what had been backported to the long term 4.19 kernel to see if there were apples to apples between the two. Then do a comparison between 4.18 and the 5.x series as various things were moved back from that kernel in places.
The items I have run into with customers either in Red Hat or outside is that generally they want with a kernel series is:
What the customer will say is usually a different order but what they complain about after they have a contract is usually in the first order.
Personally I would love the order to be different but all PCI and other audit requirements has done is move 3 up from probably 4 or 5 (where 3 was New Software Support). [This is also a spectrum. There are also probably a billion Linux systems out there so that even if 80% used that order, the number having completely different priorities would be hundreds of millions of systems.]
Posted May 20, 2024 16:08 UTC (Mon)
by jra (subscriber, #55261)
[Link] (1 responses)
I think Ronnie answered this question above in this comment.
Posted May 20, 2024 16:33 UTC (Mon)
by smoogen (subscriber, #97)
[Link]
The reason to compare against 4.19 first would be to see if X was backported to that also or if it was avoided because of reasons (the problem came in after 4.19, the problem only affects hardware not supported by 4.19, etc). And it would not have mattered if the kernel being compared was the 8.8 kernel, the 9.0 kernel or similar.
I understand that doesn't cover the Red Hat Enterprise Linux kernel very well since it is not 4.18 straight. It is a chimera with different functionality from later kernels backported which does bring in other changes. I also realize my main confusion was I am not sure what the purpose of the whitepaper was for. I thought it would be on selling CIQ's knowledge of how a competitors kernel internals were and to show the limitations in a clear fashion to better sell their services. [AKA you can trust us because we can show you clearly and concisely how this was done.] That may not have been the purpose and so my suggestions would not work.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
There are interfaces used by normal programs and there are special interfaces used by special programs like systemd and udev.
Every other decades old application will continue to work. That is what counts.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
> command, seems hardly any "special"
Sometimes they actually spell out what "apart from" means.
For example trace points are an exception. There are more exceptions.
Every change is user visible eventually. Even simple changes like adding a new syscall can break programs, if the program was using the new syscall number and depended on it returning ENOSYS.
> because fuck those people
White paper: Vendor Kernels, Bugs and Stability
> Every other decades old application will continue to work. That is what counts.
White paper: Vendor Kernels, Bugs and Stability
It's up to you to ignore that. But please stop saying that I made it up.
Yes, that is unfortunate and could certainly be improved.
But please don't generalize to other applications.
There have been reverts of ABI changes due to application breakages in the past.
It's done on a case by case basis.
White paper: Vendor Kernels, Bugs and Stability
It's up to you to ignore that. But please stop saying that I made it up.
White paper: Vendor Kernels, Bugs and Stability
And everybody who disagrees is "making it up" or talking "nonsense".
I run a two decades old binary and it still works fine.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
I'll stop here.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
> I'll stop here.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
This clearly is not going anywhere useful, can we all let it go at this point, please?
Stop here please
White paper: Vendor Kernels, Bugs and Stability
This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.
Is it? When the breakage of existing code is reported as a bug, do the kernel developers declare the bug report as invalid, or do they fix the bug? If it's the latter, they live up to the principle. Sure, one might wish that such bugs would never happen, but apparently they feel that that going for that would be too constricting for kernel development.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
It should allow the kernel build system to generating a declarative list (or something more structured like JSON) that includes info like "driver/module", "option" name, "kernel release it got deprecated", and "kernel release where removal is planned".
White paper: Vendor Kernels, Bugs and Stability
It was not very useful and suffered from major bitrot.
White paper: Vendor Kernels, Bugs and Stability
It was not very useful and suffered from major bitrot.
Wol
White paper: Vendor Kernels, Bugs and Stability
Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
Such options are useless.
And there is not much anybody can do about that *except* to not break/deprecate stuff.
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
> Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
> Such options are useless.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
>
> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).
>
> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.
>
> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.
White paper: Vendor Kernels, Bugs and Stability
The kernel has been [trying to?] warn everyone about it since 5.11.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
But in the vast majority of cases nothing is "stuck" (didn't happen in a decade for me) and there would just be a blast of messages during a couple of seconds boot time that nobody reads.
Therefore, the default should be no kernel messages and also no systemd messages.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
1. Assuming you meant /etc/mtab, that's (usually) managed in userspace. It's also none of the kernel's business.
2. Assuming you meant /proc/mounts, there might be some userspace software that parses it and compares it against what it "should" look like, and misbehaves if a random mount option is missing (e.g. gets stuck in a hot loop of repeatedly remounting the device with the "correct" mount options, fires an alert and tells the sysadmin to come running, etc.).
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Easier to just silently sweep legacy under the rug, and get on with fixing real issues.
White paper: Vendor Kernels, Bugs and Stability
btrfs in 6.8.x
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Just for anybody who hasn't long since tuned out this thread...today the Btrfs regression was reported to the Btrfs developers. Less than one hour later, a Btrfs developer acknowledged the problem and agreed to add the norecovery option back.
Btrfs regression
White paper: Vendor Kernels, Bugs and Stability
So looks like it was a regression after all ;-) All's well what ends well
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Unfixed bugs in not-shipped components...
Unfixed bugs in not-shipped components...
Unfixed bugs in not-shipped components...
White paper: Vendor Kernels, Bugs and Stability
“uaf|use.after.free” : 117
“crash|panic|oops” : 289
”deadlock”: 97
”corruption”: 58
“out.of.bounds”: 53
”double.free”: 27
"null.pointer|null.ptr": 156
"null.deref": 22
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
This is not about a particular version of a kernel but rather the model itself.
Until recently the vast majority of important bugs were not even tagged as a CVE. With kernel.org becoming a CNA this is beginning to rectify with assigning ~80-100 CVEs per week (and ramping up), compared to ~200 CVEs for all of 2022.
That they are flagged now as CVEs helps.
White paper: Vendor Kernels, Bugs and Stability
Run your same analysis on the 4.19 LTS kernel.
White paper: Vendor Kernels, Bugs and Stability
A small subset of random commits in this list were analyzed and verified manually.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
RedHat is not the author of the majority of the code.
Downstream distributors can't.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
* `devel` (<https://git.savannah.gnu.org/cgit/bash.git/log/?h=devel>), where development is carried out using more modern piece-wise commits.
White paper: Vendor Kernels, Bugs and Stability
cd package-1.2.3
git init
git add .
git commit -mimport
git remote add upstream /path/to/upstream/repo
git cherry-pick UPSTREAM-COMMIT
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Unfortunately, Luca and company are not entirely wrong on this front. I still need to find some time to figure out what ultimately happened with this recent episode, for example. We're not as good at this as we like to think we are.
Compatibility
Compatibility
Compatibility
Compatibility
Compatibility
Compatibility
Wol
Compatibility
Compatibility
A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.
That's what C compiler advocates use to defend miscompilation: "works as documented". And while this bad principle is used for declaring gcc bug reports invalid, the actual practice of gcc development seems to be better: They use a lot of tests to avoid breaking "relevant" code (including for cases outside the documented behaviour), and the irrelevant code rides in the slipstream of relevant code.
White paper: Vendor Kernels, Bugs and Stability
For example, in the past it was very common that different vendors all had their own private trees with their own private drivers and it was always very difficult and expensive for them to move from one kernel release to another. Today most of those vendors all collaborate and work in upstream. Making life much easier for those vendors themselves as they now can focus on a common codebase instead of spending time maintaining their own incompatible code-bases.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
Wol
White paper: Vendor Kernels, Bugs and Stability
The main difference might just be that a different vendor might just end up with a different incomplete subset of important fixes.
And that if you freeze a kernel and maintain it for very many years, eventually most of the bugs will have been found, fixed and selected for backporting. (This is assuming the selection process can even identify most of the bugs).
But the case is that very many bugs take a very long time before they are found.
In upstream, in the last 3 months there are 34 commits that reference 1da177e4. That is a really old commit. That is the start of git history 19 years ago. :-)
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
More breaking of userspace
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... 32-bit processes that use most of their address space were just broken
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... some VMAs that used to be allocated contiguously now aren't, which means less VMA merging, leading to a kernel test using far too many VMAs
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... alignment changes revealed an existing bug in a kernel test
https://git.kernel.org/pub/scm/linux/kernel/git/stable/li... alignment changes caused a regression in a thread stress test because stacks started getting transparent huge pages. The fix here assigns semantics to MAP_STACK which was previously a no-op, so anyone creating threads and not using MAP_STACK is still in trouble.
More breaking of userspace
More breaking of userspace
More breaking of userspace
More breaking of userspace
More breaking of userspace
> it does not matter what you promise in the contract:
> all observable behaviors of your system
> will be depended on by somebody.
More breaking of userspace
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability
1. Stability. The majority of them will put up with various security problems over any stability issues. This stability goes beyond 'crashes', but also interface and other changes. Stability covers ability to use the same configs for related software and workloads versus special casing because core applications are different.
2. New hardware support. Generally they want to be able to run their operating system over a general set of hardware purchased over a 6 year period. [They would love to have that be hardware over a 14 year period.. but there are limits]
3. Security fixes. [Make sure that they can pass various audits to maintain usage of certain systems].
...
4. New software support on the same OS.
White paper: Vendor Kernels, Bugs and Stability
White paper: Vendor Kernels, Bugs and Stability