White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 11:17 UTC (Sun) by wtarreau (subscriber, #51152)
In reply to: White paper: Vendor Kernels, Bugs and Stability by bluca
Parent article: White paper: Vendor Kernels, Bugs and Stability

It's not about testing the kernel, it's about testing that what you're relying on still works. That's the same for any project, we all depend on lower layers and the tests are expected to catch failures. If you know you're relying only on abstraction layers, you probably don't care, but when you start dealing with syscalls or mount options yourself, you know that the risk becomes higher to face some changes over time. I know it's difficult and such changes are rare enough that even having tests doesn't warrant that this or that change will be caught. But probably that kernel warnings for an init system are worth being at least logged for occasional inspection.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 13:18 UTC (Sun) by bluca (subscriber, #118303) [Link] (18 responses)

That is only the case if "we do not break userspace" is false advertisement, and every userspace interface provided by the kernel is subject to breakages and incompatible changes. Is it?

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 13:59 UTC (Sun) by mb (subscriber, #50428) [Link] (16 responses)

There are different kinds of interfaces.
There are interfaces used by normal programs and there are special interfaces used by special programs like systemd and udev.

Normal interfaces are changed extremely rarely and these obviously are the ones meant by the "do not break userspace" rule.

Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
Every other decades old application will continue to work. That is what counts.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 14:11 UTC (Sun) by bluca (subscriber, #118303) [Link] (15 responses)

Sorry, but this is nonsense. Apart from the fact that a mount option, that can be used by anything or anyone with a simple mount command, seems hardly any "special", either the userspace interface is stable and breaking it is bad, or not. Kernel maintainers say "we do not break userspace", they don't say "we do not break userspace, apart from programs X, Y and X - because fuck those people". This is something that you have just made up post-facto to retroactivly justify an obvious and clear regression.

Which by the way, neatly explains why vendor kernels are needed and are in fact the only sane choice, despite what the paper cited in this article says. Nobody should run production payloads on upstream kernels at this point, given basic stuff like mount options just breaks left and right.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 15:20 UTC (Sun) by mb (subscriber, #50428) [Link] (14 responses)

> Apart from the fact that a mount option, that can be used by anything or anyone with a simple mount
> command, seems hardly any "special"

I didn't say that it is. I did not talk about this specific thing, because I don't know anything about it. I was talking about "do not break userspace" in the general form, not in this specific case.

Whether this mount change is a sane change is up to somebody else to judge.

> they don't say "we do not break userspace, apart from

Well, they pretty much do exactly that.
Sometimes they actually spell out what "apart from" means.
For example trace points are an exception. There are more exceptions.

If you want 100% full "don't break userspace" without exceptions, we must basically stop all kernel development now.
Every change is user visible eventually. Even simple changes like adding a new syscall can break programs, if the program was using the new syscall number and depended on it returning ENOSYS.

Having a "don't break userspace without exceptions" is impossible.

> This is something that you have just made up

No. See my example of trace points.

> this is nonsense
> because fuck those people

I think it would be good to calm down before continuing the discussion.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 17:32 UTC (Sun) by bluca (subscriber, #118303) [Link] (13 responses)

> I didn't say that it is. I did not talk about this specific thing, because I don't know anything about it. I was talking about "do not break userspace" in the general form, not in this specific case.

This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.

> Well, they pretty much do exactly that.

No, they very much do not. Look at all the enthusiastic comments from kernel people pointing to the paper in the article and saying "See? Vendor kernels are BAD, just upgrade to upstream kernels, it's fine really", and when told that new kernel version break applications and that's the real reason why vendor kernels are used, they shrug it away as "impossible, we do not break userspace"

> No. See my example of trace points.

Yes, it is exactly what you did, and there was no mention anywhere of trace points:

> Yes, it is annoying, if systemd/udev are affected by an interface change. Especially, if this interface change could have been avoided. But it's not the end of the world.
> Every other decades old application will continue to work. That is what counts.

You have made up a new rule according to which it's fine to break systemd or udev (if it's not made up, then just point to where on https://kernel.org/doc/ it is defined), but *unspecified other applications* must continue to work. That is very convenient of course, it's always unspecified other applications that are supported, and the ones that break are never actually supported. That's a very easy way of guaranteeing compatibility - every time something goes wrong just say that case was never actually supposed to continue working and move on.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:10 UTC (Sun) by mb (subscriber, #50428) [Link] (11 responses)

>You have made up a new rule

There have always been exceptions and I didn't make that up. That's just silly. I even gave you an example (tracepoints).
It's up to you to ignore that. But please stop saying that I made it up.

I respect you for what you do for Linux, Systemd and so on. But you're acting like a child right now.

>it's always unspecified other applications that are supported

Yes. That is exactly like it is.

I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.
Yes, that is unfortunate and could certainly be improved.
But please don't generalize to other applications.

>every time something goes wrong just say that case was never actually supposed to continue working and move on.

That's not how things are done, though.
There have been reverts of ABI changes due to application breakages in the past.
It's done on a case by case basis.

Now you will reply: You have made up yet another rule!

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:32 UTC (Sun) by bluca (subscriber, #118303) [Link] (10 responses)

> There have always been exceptions and I didn't make that up. That's just silly. I even gave you an example (tracepoints).
It's up to you to ignore that. But please stop saying that I made it up.

Literally nobody has mentioned tracepoints. I mean I'm not even sure that really qualifies as a userspace interface - maybe it does, it would seem strange, but I am not a tracing experts. But it is completely unrelated to mount options being removed.

> I understand that you are upset that the kernel apparently frequently breaks systemd/udev. But keep in mind that these applications are tightly coupled to the kernel. It's natural that these see more breakage than other average applications.

Says who? That is very much not true. Every interface that I can think of is used by multiple unrelated applications. I have no idea where you get this from. Cgroups and namespaces? Throw a rock in the general direction of a container runtime and you'll hit either or both. Netlink? There are as many network and interface managers as there are Linux vendors. Process management? That's been around since literally forever, and see the point about container management again. Mounting filesystems? fstab is older than me, I am quite sure.

'We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init' doesn't sound as catchy, does it now?

> It's done on a case by case basis.

I am well aware. And the triaging of that case by case goes like this: did it affect the machine that Linus happened to boot on that week? If so, it gets reverted and unpleasant emails are shot left and right. Else, nothing to see, move along.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 18:48 UTC (Sun) by mb (subscriber, #50428) [Link] (9 responses)

> Literally nobody has mentioned tracepoints.

What the? I did. I mentioned them as an example for a non-stable interface. After you have asked.

> I mean I'm not even sure that really qualifies as a userspace interface

Oh. I get it. *You* want to define what a userspace interface is and what not.
And everybody who disagrees is "making it up" or talking "nonsense".

That is silly.

> Says who?

Me. But I'm not sure why that matters.

> We do not break userspace, as long as userspace is a statically linked printf("hello world\n") /sbin/init

Well. I have never experienced a breakage due to a kernel interface change.
I run a two decades old binary and it still works fine.

That is my experience.

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 19:31 UTC (Sun) by bluca (subscriber, #118303) [Link] (8 responses)

> I mentioned them as an example for a non-stable interface. After you have asked.

Again, I do not know the first thing about tracepoints and have zero interest in that. Maybe it's a supported interface, maybe it's not, I really cannot say, nor care, and can't see what it has to do with mount options.

> *You* want to define what a userspace interface is and what not.

No, userspace defines what is a userspace interface, as per Hyrum's Law.

> Me. But I'm not sure why that matters.

Because it's just wrong, as explained, there are no "special custom interfaces" being used anywhere, just bog standard stuff used by most components of an operating system.

> I run a two decades old binary and it still works fine.

'We do not break userspace, as long as userspace is mb's statically linked printf("hello world\n") /sbin/init' still not quite as catchy I'm afraid

White paper: Vendor Kernels, Bugs and Stability

Posted May 19, 2024 20:23 UTC (Sun) by mb (subscriber, #50428) [Link] (7 responses)

Ok. Everybody is wrong, except you. I got it.
I'll stop here.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:30 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (2 responses)

I mean. The not breaking userspace only applies to a subset of things userspace can do.

I've had to fix software because of a kernel update, because some files in /sys were moved. But for some reason that doesn't count.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:41 UTC (Mon) by mb (subscriber, #50428) [Link]

> I mean. The not breaking userspace only applies to a subset of things userspace can do.

That is exactly what I was saying. Yet, I'm apparently wrong.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:45 UTC (Mon) by bluca (subscriber, #118303) [Link]

> I mean. The not breaking userspace only applies to a subset of things userspace can do.

Where is that subset defined?

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:09 UTC (Mon) by wtarreau (subscriber, #51152) [Link]

> Ok. Everybody is wrong, except you. I got it.
> I'll stop here.

Welcome to discussions with bluca. Agressivity, half-reading of arguments, and accusations often arrive in the second or third message when he disagrees with you. There are such people who constantly criticize Linux and who would probably do good to the community by switching to another OS of choice :-/

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:26 UTC (Mon) by bluca (subscriber, #118303) [Link] (2 responses)

You are literally trying to gaslight me into believing that dropping a mount option (not just making a noop, literally deleting it so that a hard error is returned where it wasn't previously), that is currently in use by userspace, is not a compatibility breakage. Make of that what you will.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 11:54 UTC (Mon) by mb (subscriber, #50428) [Link] (1 responses)

>You are literally trying to gaslight me

Wow. This is a new level.

Stop here please

Posted May 20, 2024 12:57 UTC (Mon) by corbet (editor, #1) [Link]

This clearly is not going anywhere useful, can we all let it go at this point, please?

White paper: Vendor Kernels, Bugs and Stability

Posted May 23, 2024 15:48 UTC (Thu) by anton (subscriber, #25547) [Link]

This is very much about "do not break userspace" in the general form. It's the perfect example of why that mantra needs to be put to bed, once and for all, as it's completely disconnected from reality.

Is it? When the breakage of existing code is reported as a bug, do the kernel developers declare the bug report as invalid, or do they fix the bug? If it's the latter, they live up to the principle. Sure, one might wish that such bugs would never happen, but apparently they feel that that going for that would be too constricting for kernel development.

Whether that means that vendor kernels are needed, or that one can use upstream kernels if one is selective about them is up to the vendors and their customers to decide.

White paper: Vendor Kernels, Bugs and Stability

Posted May 20, 2024 9:28 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

Well the stuff in /sys moves around a lot.