LWN: Comments on "White paper: Vendor Kernels, Bugs and Stability"

White paper: Vendor Kernels, Bugs and Stability

donald.buczek — Fri, 24 May 2024 13:45:31 +0000

> I don't need a static declarative list of all deprecations that might or might not exist in my kernel.
>
> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).
>
> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.
>
> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

This would be perfect! We would see, what we need to address in our fleet (we are not using a distribution). But distributions would have something to build on, too. They might create a feedback path of this information from the systems of their users to the distribution. The basis for everything is that the information "you are using a mechanism, which will go away" is made available in a structured way.

It is important that the information cannot only be found by digging through masses of unstructured text in mailing lists, documentation, NEWS files, dmesg or other sources and then having to analyze in each individual case whether it is is relevant to you at all.

White paper: Vendor Kernels, Bugs and Stability

tlamp — Fri, 24 May 2024 08:15:24 +0000

> It would change nothing.
> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
> Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
> Such options are useless.

I don't think so, mostly because my spitballed proposal was not targeted at solving the "distros get never hurt by deprecation", as IMO that cannot be solved, besides not doing any deprecation at all anymore which hardly is a good solution. Rather, I wanted to target the "how things get communicated and noticed" part and having ab extra compile option with something like "deprecated-6.8-removal-6.12" in the name could actually be quite good for that. The build configs are often tracked and even diffed, and as simple single file can be easily grep'ed against _DEPRECATION_ and then diffed for ones that would trigger soon or new ones, probably even by a CI like systemd uses.

I.e., the status quo is having warnings for deprecation, which can be brittle and are not easy to digest/parse, having that info in an easier to digest manner would help a lot as tools/distros that depend on such options can easily find out when a used one will vanish soon(ish), then they also have no excuse about being unprepared.

> And that is exactly when people will first start to notice.

If their distro or tooling did not do their work then yes, but it wouldn't be the fault of the kernel having a messy deprecation process. IME most bigger distros or big projects like systemd want to avoid that, so if they'd have the definitive information required to do so in just somewhat digestible way, then I really think that most would actually act on that.

White paper: Vendor Kernels, Bugs and Stability

tlamp — Fri, 24 May 2024 08:01:21 +0000

> I don't need a static declarative list of all deprecations that might or might not exist in my kernel.

As said, I'd assemble on build so those options that are not relevant for a kernel build config would not be in there (or if still wanted could be tracked differently, i.e., with an extra flag or separate list)

> I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).

With a declarative list this is trivial to create, as a tool can just scan all modules, mount, ... options and compare if anything explicitly set is in the static list. So if you want this then a static list is IMO really the best way to achieve it, one first needs the definitive list of information before being actually able to do something with it, keeping it dumb on the kernel side and include as much as possible allows (user space or build) tooling to actually do the smart checks.

> A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.

Not sure how this minus bikeshedding is any different what I proposed, but I like that we agree in general.

> No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

I'd named JSON simply as an option, explicitly also stated that a simple list could do. But, I named JSON as 1. generating it is trivial (compared to parsing, which isn't hard either, but not trivial anymore) 2. Allows more flexible extension for whatever data or use case gets relevant in the future without having to do a /proc/deprecation2 3. in my projects I try to avoid another not-invented-here format with it subtleties to be added, but sure if it's a simple CSV list that gets generated by the common infra (i.e., not under the control of each kernel dev with their own opinions of the day) then fine by me (not that my acknowledgment would matter anything :)

White paper: Vendor Kernels, Bugs and Stability

Wol — Thu, 23 May 2024 22:41:07 +0000

> Do you realize, that most kernel options are disabled by default?

And how many of those options have "deprecated" in their name? Surely that's a massive red flag.

> It means that developers don't have a clue what people (users!) actually want and need.

And how many developers are employed by (therefore are) users? I believe Alphabet employs loads. Meta employs loads. Most of the kernel developers I have contact with are employed by large end users. It's a little difficult to be oblivious of your own needs. (Some people manage, I'm sure ...)

How difficult is it - to set a "not enabled" flag that cannot be accessed without some sort of warning that this flag will enable deprecated functionality Surely it's not beyond the wit of your typical kernel developer? That's ALL that's required.

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

pizza — Thu, 23 May 2024 21:48:36 +0000

> So first of all "dmesg" doesn't work anymore on Fedora (you need perms now).

Fedora just flipped the default, starting with their 6.8 kernels IIRC.

Reverting it is just a matter of:

sysctl kernel.dmesg_restrict=0

> It's a wall, a deluge of text. Don't expect me to read all that.

Fortunately, it's rare when anything beyond the final dozen or so lines matters. (eg the messages that show up when you plug something in, or if something goes wrong...)

White paper: Vendor Kernels, Bugs and Stability

mezcalero — Thu, 23 May 2024 21:14:36 +0000

So first of all "dmesg" doesn't work anymore on Fedora (you need perms now).

But even beyond that: there's *so* *much* *stuff* in dmesg right now. It's a wall, a deluge of text. Don't expect me to read all that. I only look there when I am looking for something, and then I usually do "journalctl -ke", and it never showed up there.

White paper: Vendor Kernels, Bugs and Stability

mb — Thu, 23 May 2024 21:01:15 +0000

>Developers are enabling something that is disabled by default? They're accepting the associated risks.

Do you realize, that most kernel options are disabled by default?

>The fact that people have to actively enable something that developers clearly don't want activated means

It means that developers don't have a clue what people (users!) actually want and need.

Closing your eyes won't make the demand go away, unless you are less than three years old.

White paper: Vendor Kernels, Bugs and Stability

Wol — Thu, 23 May 2024 20:56:19 +0000

> It would change nothing.

Except it changes everything

> Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.

You just said it!

The distributions are enabling something that is disabled by default? They're accepting responsibility for keeping it working.

Developers are enabling something that is disabled by default? They're accepting the associated risks.

People are enabling something that is marked "deprecated"? They're being placed on notice that it's being left to bit-rot.

The fact that people have to actively enable something that developers clearly don't want activated means that anybody using it will have three choices - migrate their code away, take over maintenance, or do an ostrich and bury their heads in the sand. Users will still be able to be complain "I didn't know", but their upstream won't have that excuse.

Cheers,
Wol

Compatibility

anton — Thu, 23 May 2024 16:49:57 +0000

A sweet spot always has to be found, where developers document the intended use and users scratch a bit around what is officially supported, keeping the intended use in mind so as to limit the surprise in case of changes.

That's what C compiler advocates use to defend miscompilation: "works as documented". And while this bad principle is used for declaring gcc bug reports invalid, the actual practice of gcc development seems to be better: They use a lot of tests to avoid breaking "relevant" code (including for cases outside the documented behaviour), and the irrelevant code rides in the slipstream of relevant code.

Fortunately, the kernel aspires to a better principle, the often-cited "we don't break user space". And given that we have no way to automatically check whether a program conforms to the documentation, that principle is much more practical (and the practice of gcc is also in that direction, but only for "relevant" code).

Concerning accidental features, a careful design of interfaces avoids those. E.g., one can catch unintended combinations of inputs and report them as errors. Or define and implement useful behaviour in such a case. Exposing some arbitrary accident of the implementation in such cases leads to the difficulties you point out. Given that Linux' interface to user space is also a security boundary, carefully defining interfaces and avoiding exposing implementation artifacts is a good idea anyway.

White paper: Vendor Kernels, Bugs and Stability

anton — Thu, 23 May 2024 15:48:35 +0000

This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.

Is it? When the breakage of existing code is reported as a bug, do the kernel developers declare the bug report as invalid, or do they fix the bug? If it's the latter, they live up to the principle. Sure, one might wish that such bugs would never happen, but apparently they feel that that going for that would be too constricting for kernel development.

Whether that means that vendor kernels are needed, or that one can use upstream kernels if one is selective about them is up to the vendors and their customers to decide.

White paper: Vendor Kernels, Bugs and Stability

mb — Thu, 23 May 2024 15:26:35 +0000

We're talking about kernel messages here, right? Not the systemd start-stop messages.

I'm not saying that messages are not useful for debugging. And for that exact reason it's always possible to enable a verbose boot.
But in the vast majority of cases nothing is "stuck" (didn't happen in a decade for me) and there would just be a blast of messages during a couple of seconds boot time that nobody reads.
Therefore, the default should be no kernel messages and also no systemd messages.

A kernel deprecation message would certainly go by unnoticed during the burst of messages.

White paper: Vendor Kernels, Bugs and Stability

eru — Thu, 23 May 2024 15:19:16 +0000

Isn't that way on my system. There are enough operations that take humanly noticeable amount of time, and as noted, if something is stuck, the scrolling pauses there. The OS is Ubuntu 22.04 which uses systemd; the laptop is 10 years old, but but yet dust-bin-ready, as it was rather upscale when obtained.

White paper: Vendor Kernels, Bugs and Stability

zdzichu — Thu, 23 May 2024 15:14:48 +0000

But the users would be helpless even if they saw the deprecation messages!

Developers, on the other hand, are the target of such messages. I cannot imagine developing a low-level component – like systemd – and not running `dmesg` from time to time. If you use kernel features more than anyone else, you should pay more attention than anyone else.

White paper: Vendor Kernels, Bugs and Stability

smurf — Thu, 23 May 2024 14:57:04 +0000

I don't need a static declarative list of all deprecations that might or might not exist in my kernel.

I need a list of those deprecated calls/options/whatever that the current system is actually using (or rather has been using since booting).

A data structure that gets added to a list which you can check via /proc/deprecated would be quite sufficient for this.

No JSON fanciness required; a textual table identifying the subsystem or module, source file, first and last use timestamps, and its identifier in linux/Documentation/deprecations.yml [yes I know that file doesn't exist yet] would be quite sufficient.

White paper: Vendor Kernels, Bugs and Stability

mb — Thu, 23 May 2024 14:49:46 +0000

>It is a mystery to me why this is done

Because nobody can read over 9000 messages in 0.5 seconds.

>The stream of messages is actually a nice progress indicator

No, it isn't. Today's boot process is so fast (unless you're stuck with a legacy init system), that console output is pretty much useless and just slows things down, at best.

White paper: Vendor Kernels, Bugs and Stability

mb — Thu, 23 May 2024 14:45:29 +0000

>but move all deprecated stuff into a (or several) modules behind an option "deprecated-6.8" or whatever.

It would change nothing.
Every distribution and everyone building their kernel will just enable this option, because stuff will break without enabling it.
Just like everybody enabled the - how was it called? - EXPERIMENTAL option.
Such options are useless.

>let's delete it

And that is exactly when people will first start to notice.
And there is not much anybody can do about that *except* to not break/deprecate stuff.

White paper: Vendor Kernels, Bugs and Stability

eru — Thu, 23 May 2024 14:01:08 +0000

The kernel has been [trying to?] warn everyone about it since 5.11.

These days just about every distribution "helpfully" hides boot messages behind a splash screen, so few users will ever see such warnings. Might as well not be there.

It is a mystery to me why this is done (and after I found out how, I disabled it). The stream of messages is actually a nice progress indicator, and if they get stuck at some point, one gets some idea about what might be wrong.

White paper: Vendor Kernels, Bugs and Stability

Wol — Thu, 23 May 2024 13:20:35 +0000

> We had such a deprecation list under Documentation, but I think it got removed a couple of years ago.
It was not very useful and suffered from major bitrot.

Far better to do it in the kernel itself. Probably not easy, but move all deprecated stuff into a (or several) modules behind an option "deprecated-6.8" or whatever. Bleeding edge sets all these to "no", and either someone steps up and supports it (removing the deprecated option), or it bitrots until someone says "oh, this broke ages ago, let's delete it".

And then, if there's stuff you really want to get rid off but people need it, every year or so it gets upgraded to "deprecated latest kernel", so hopefully people stop using it and it finally drops out of sight ...

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

mb — Thu, 23 May 2024 12:35:19 +0000

> A major improvement here could consist of adding a common infrastructure in the kernel to track deprecation.

We had such a deprecation list under Documentation, but I think it got removed a couple of years ago.
It was not very useful and suffered from major bitrot.

White paper: Vendor Kernels, Bugs and Stability

tlamp — Thu, 23 May 2024 10:12:05 +0000

IMO what actually needs fixing is how deprecated options, maybe even drivers, are communicated and tracked.

A major improvement here could consist of adding a common infrastructure in the kernel to track deprecation.
It should allow the kernel build system to generating a declarative list (or something more structured like JSON) that includes info like "driver/module", "option" name, "kernel release it got deprecated", and "kernel release where removal is planned".

This data should be assembled on kernel build, possibly even made available on runtime in one of the virtual FS, would allow distros and projects with a lot of kernel interaction like systemd to actually track those and notice those for sure, as scanning for arbitrary warnings that can change wording every point release is just an ugly mess with lots of false-positives/negatives waiting to happen.

If it was available on runtime then checks could be added to the pre- / post-installation scripts/hooks of the kernel distro packages so that users can get a much more noticeable warning printed out on upgrade if their system is affected by such an option removal.

White paper: Vendor Kernels, Bugs and Stability

taladar — Thu, 23 May 2024 07:50:54 +0000

What you describe is the traditional understanding. However I doubt that is actually true.

Every backport is also a new, untested version, one with changes by someone with significantly lower understanding of either the original code base or the fix than the person working on that part of the code on the latest version. Distro kernels see significantly less testing than the latest version does because of its limited audience which is largely comprised of people who only want it once it has been tested.

More breaking of userspace

taladar — Thu, 23 May 2024 07:46:51 +0000

I would suggest the opposite. The only implementation I would never want to see is one treating a hint as a guarantee. If you guarantee the behavior specify it as a guarantee, don't make it implicit like that.

White paper: Vendor Kernels, Bugs and Stability

bluca — Wed, 22 May 2024 11:52:15 +0000

but where's the fun in that :-P

White paper: Vendor Kernels, Bugs and Stability

smurf — Wed, 22 May 2024 05:16:32 +0000

Seems that it's somewhat helpful to file such bugs on the channel(s) actually designated for them, instead of, like, bitching on LWN about them. ;-)

White paper: Vendor Kernels, Bugs and Stability

bluca — Wed, 22 May 2024 00:03:49 +0000

and now a revert has been sent and merged in btrfs-next: https://lore.kernel.org/all/44c367eab0f3fbac9567f40da7b27...
So looks like it was a regression after all ;-) All's well what ends well

Btrfs regression

corbet — Tue, 21 May 2024 13:29:54 +0000

Just for anybody who hasn't long since tuned out this thread...today the Btrfs regression was reported to the Btrfs developers. Less than one hour later, a Btrfs developer acknowledged the problem and agreed to add the norecovery option back.

White paper: Vendor Kernels, Bugs and Stability

jamesh — Tue, 21 May 2024 09:18:10 +0000

You are essentially arguing that it isn't enough to distribute the source code corresponding to the binaries, but that the source code to all previous versions is also required. That seems like it'd be a problem for far more than just RHEL kernel packages. Off the top of my head:

Many pieces of software have changed license at some point in their history. A distro might be happy with the current license but not the old one. Do they need to distribute all the versions released under the bad license now?
A number of packages in Debian modify the upstream source tarballs to remove files with licenses that don't meet their guidelines. If they were required to distribute a git tree with history linking back to upstream, that'd include the problem files.
Some projects have been accused of plagiarism and removed the offending code, but it remains in the version control history.

There are open source licenses that require that you split out patches from the upstream release (e.g. the QPL), but the GPL is not one of them.

White paper: Vendor Kernels, Bugs and Stability

NYKevin — Mon, 20 May 2024 17:35:49 +0000

It was... like 10-15 years ago when I cut my teeth on this stuff. But nowadays, we have software for everything, so I haven't actually played with low-level mounting etc. in a long time.

Which is also, sort of, the point I'm trying to make here. It is no longer accurate to divide userspace into "applications" and "stuff the sysadmin manually fiddles with." Sysadmins are not manually fiddling with mount options etc. these days. That's all managed by some other piece of software that isn't "the application," but can still break and cause problems all the same. E.g. k8s, Puppet, Docker, etc.

White paper: Vendor Kernels, Bugs and Stability

smoogen — Mon, 20 May 2024 16:33:38 +0000

I didn't actually think it covered my concerns but I also did not clearly state them in my first email.

The reason to compare against 4.19 first would be to see if X was backported to that also or if it was avoided because of reasons (the problem came in after 4.19, the problem only affects hardware not supported by 4.19, etc). And it would not have mattered if the kernel being compared was the 8.8 kernel, the 9.0 kernel or similar.

I understand that doesn't cover the Red Hat Enterprise Linux kernel very well since it is not 4.18 straight. It is a chimera with different functionality from later kernels backported which does bring in other changes. I also realize my main confusion was I am not sure what the purpose of the whitepaper was for. I thought it would be on selling CIQ's knowledge of how a competitors kernel internals were and to show the limitations in a clear fashion to better sell their services. [AKA you can trust us because we can show you clearly and concisely how this was done.] That may not have been the purpose and so my suggestions would not work.

White paper: Vendor Kernels, Bugs and Stability

jra — Mon, 20 May 2024 16:08:12 +0000

> In looking at the white paper, I was wondering why the comparison was done between Red Hat's which was 'based' off of 4.18 and the 5.x series.

I think Ronnie answered this question above in this comment.

https://lwn.net/Articles/974151/

White paper: Vendor Kernels, Bugs and Stability

smoogen — Mon, 20 May 2024 15:15:59 +0000

[Note: I am currently employed by Red Hat. I have worked for Red Hat for 19 years as such my comments should be seen as biased. ]

In looking at the white paper, I was wondering why the comparison was done between Red Hat's which was 'based' off of 4.18 and the 5.x series. I would have thought a better analysis would have been to see what had been backported to the long term 4.19 kernel to see if there were apples to apples between the two. Then do a comparison between 4.18 and the 5.x series as various things were moved back from that kernel in places.

The items I have run into with customers either in Red Hat or outside is that generally they want with a kernel series is:
1. Stability. The majority of them will put up with various security problems over any stability issues. This stability goes beyond 'crashes', but also interface and other changes. Stability covers ability to use the same configs for related software and workloads versus special casing because core applications are different.
2. New hardware support. Generally they want to be able to run their operating system over a general set of hardware purchased over a 6 year period. [They would love to have that be hardware over a 14 year period.. but there are limits]
3. Security fixes. [Make sure that they can pass various audits to maintain usage of certain systems].
...
4. New software support on the same OS.

What the customer will say is usually a different order but what they complain about after they have a contract is usually in the first order.

Personally I would love the order to be different but all PCI and other audit requirements has done is move 3 up from probably 4 or 5 (where 3 was New Software Support). [This is also a spectrum. There are also probably a billion Linux systems out there so that even if 80% used that order, the number having completely different priorities would be hundreds of millions of systems.]

White paper: Vendor Kernels, Bugs and Stability

sandeen — Mon, 20 May 2024 15:00:52 +0000

I commented in another thread, but maybe it's more relevant here:

I think it would be fascinating run your same analysis on the 4.19 LTS kernel, which is supported until the end of this year, IIRC.

White paper: Vendor Kernels, Bugs and Stability

sandeen — Mon, 20 May 2024 14:51:11 +0000

You know what would be fascinating, and a useful control case to test thewhitepaper's theory?
Run your same analysis on the 4.19 LTS kernel.

White paper: Vendor Kernels, Bugs and Stability

paulj — Mon, 20 May 2024 14:36:22 +0000

You can say what you want, but the GPL deliberately does not define what "preferred form" means. And the "For an executable work, ..." part doesn't change that, because it it self refers to "source code" - it clarifies that the executable work includes the "source code" for X, Y and Z. And "source code" means "the preferred form for modifications".

You may indeed argue that the preferred form is a text file.

However, if you are a company that has deliberately stripped away context that you yourself use for hosting and modifying said source code as part of your commercial operation, and you have made that deliberate act in order to frustrate others who also wish to modify said source code, then I - if it mattered to me and I were a copyright holder - could well argue to a court that the form you have provided is clearly not the form you prefer to use yourself for making modifications, and that you have therefore violated the licence I gave you.

E.g., I - as your upstream - could well be trying to follow along with what you're doing to my code, so that I can cherry-pick the good bits to integrate back into the code. Your deliberate obfuscation, done to keep for yourself the preferred form of modifications and hence frustrate others from picking out what you changed, I would argue as being clearly against the wording of the licence.

The form you prefer to use internally, versus the form you chose to release, would be prima facie evidence of your failure to honour the licence.

Who would win, who knows. But I don't think your position would be 100% safe.

White paper: Vendor Kernels, Bugs and Stability

smurf — Mon, 20 May 2024 13:54:09 +0000

The "preferred form of modification" was, and is, a text file. Or a directory tree with a large heap of 'em.

True, the preferred form of keeping track of your modifications, distributing them, etc.etc.etc., once was patches and tar files; now it's git and pulling commits. The problem is that the GPL doesn't talk about any of that. It merely requires "a" medium commonly used for software exchange. Not "the medium everybody else who's contributing to this particular piece of source code is using".

So for better or worse, you can appeal to common sense and interoperability and the equivalent of sportsmanship and whatnot and I'll be the first person to be argue right along with you. But the GPL? sorry you pour cold water on your fire but you can't use it to demand git trees from anybody.

Disclaimer: IMHO and IANAL and all that.

Stop here please

corbet — Mon, 20 May 2024 12:57:35 +0000

This clearly is not going anywhere useful, can we all let it go at this point, please?

White paper: Vendor Kernels, Bugs and Stability

paulj — Mon, 20 May 2024 12:46:29 +0000

What do you mean exactly?

If you mean "For an executable work, complete source code means ..." - that's defining the complete source code, but not the preferred form for modifications.

You can have the complete source code, but not have it in the preferred form for modification.

White paper: Vendor Kernels, Bugs and Stability

pbonzini — Mon, 20 May 2024 12:38:25 +0000

Even in the one case that is precisely defined in the license itself?

White paper: Vendor Kernels, Bugs and Stability

Wol — Mon, 20 May 2024 12:20:56 +0000

> "Here's a what-if to consider. What if the distros and the corps spent exactly the same amount of money on a mix of QA engineers and developers for much more recent stable kernels?"

"You can't QA quality into a product".

"10 minutes spent on design knocks 1hr off development".

"Every problem that slips through one phase takes 10 times longer to correct in the next".

As I moaned above, design, Design! DESIGN. If that money was spent on *documentation* and *design* it would probably find far more bugs, far more quickly, and what's more important get them fixed (probably by other people :-)

The current setup encourages a kernel held together by baling wire, duck tape, and sealing wax ...

Cheers,
Wol

White paper: Vendor Kernels, Bugs and Stability

georgyo — Mon, 20 May 2024 12:03:43 +0000

I'm confused, what broke here? I run 6.8+ with btrfs/systems on multiple machines and never had any trouble booting.

I see a lot of people participating in this discussion, but not a lot of people actually affected.