Never break userspace [LWN.net]

Never break userspace

Posted Feb 16, 2024 15:36 UTC (Fri) by bluca (subscriber, #118303)
In reply to: Never break userspace by corbet
Parent article: A turning point for CVE numbers

Sure, here's one:

https://lists.freedesktop.org/archives/systemd-devel/2022...
https://lists.freedesktop.org/archives/systemd-devel/2022...

The other one that hit me personally was when overlayfs was made incompatible with selinux. And then there's all the times that netlink changed. And all the time uevents changed. And all the times sysfs changed. And so on and so forth. The reality is that "we don't break userspace" is a nice story that kernel developers like to go around tell anybody who's willing to listen, but it's just that, a story. They barely care about syscall ABI stability, and even that gets broken from time to time as already pointed out by another comment.

to post comments

Never break userspace

Posted Feb 16, 2024 16:01 UTC (Fri) by mb (subscriber, #50428) [Link] (2 responses)

Well. Technically you are right. The kernel frequently breaks interfaces.
But in reality, systemd and udev are parts of the operating system.
So I don't care that much, if the OS breaks itself. That will get fixed eventually.

What I and most users care about is whether actual user applications break.
And that extremely rarely happens.
I run decades old binaries that work just fine.

It doesn't affect my application, because the OS as a whole still works as before, after porting systemd/udev to the new interfaces. A combination of updated kernel and incompatible systemd/udev would never hit stable distributions.

Therefore, do you have examples of real user applications breaking, that are not part of the OS?

Never break userspace

Posted Feb 16, 2024 16:08 UTC (Fri) by bluca (subscriber, #118303) [Link] (1 responses)

> Therefore, do you have examples of real user applications breaking, that are not part of the OS?

Need some help with all that goal post moving? Must be exhausting, going that far

Never break userspace

Posted Feb 16, 2024 16:54 UTC (Fri) by mb (subscriber, #50428) [Link]

I have said that you were right. You have said nothing wrong.
There is no such thing as a general interface stability guarantee.

I've just set things into perspective. That is no goal post moving.

Never break userspace

Posted Feb 16, 2024 16:06 UTC (Fri) by corbet (editor, #1) [Link] (12 responses)

As yes, the BIND/UNBIND thing was a big enough deal that I wrote about it at the time. What I suggested there might still seem to make sense: rather than sniping at the kernel community from the sideline, work with them to improve the situation. Let Thorsten know about regressions, preferably early enough to keep them from making it into a release. Things can be improved.

I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.

Never break userspace

Posted Feb 16, 2024 16:49 UTC (Fri) by bluca (subscriber, #118303) [Link] (10 responses)

> As yes, the BIND/UNBIND thing was a big enough deal that I wrote about it at the time. What I suggested there might still seem to make sense: rather than sniping at the kernel community from the sideline, work with them to improve the situation. Let Thorsten know about regressions, preferably early enough to keep them from making it into a release. Things can be improved.
>
> I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.

These things do get reported, and they get ignored/shrugged away if you are lucky, and if not you get taken for a long ride. For this case it's explained in the link above as well. For the overlayfs case Google even went as far as sending 20 revisions of a patchset to try and restore backward compatibility, albeit optionally, and it was stonewalled: https://lore.kernel.org/lkml/20211117015806.2192263-1-dva...
Apparently breaking userspace can just be waved through, while fixing it requires "building a security model" and other extremely high-bars to be met. All in the meanwhile anybody using selinux needs to completely open up the security policy to make it work at all, of course, which I guess makes for a very interesting "security model". I could go on, but can't be bothered to look up yet more references.

So yeah, trying and dispelling this myth that "the kernel doesn't break userspace" is pretty much all that's left. Reading blatantly false statements being made irks me really badly, especially when used to justify some potentially damaging process changes as it happened here.

Never break userspace

Posted Feb 16, 2024 17:01 UTC (Fri) by mb (subscriber, #50428) [Link] (9 responses)

> These things do get reported, and they get ignored/shrugged away if you are lucky

Quite honestly, systemd and udev also broke lots and lots of things about how the Linux operating system works.
Administrators had to change tons of scripts, because some things suddenly worked differently after the distribution updated from classic init to systemd.

But the correct answer to users complaining often enough is to "ignore it" or "shrug it away".
Things work differently now. Get used to it. That's the correct answer surprisingly often.
That's true for systemd and it's also true for parts of the kernel.

What should be avoided is breaking changes that don't have positive sides. Changes just for the sake of changing and breaking things. That is bad and must be avoided. And it should always be considered, if a non-breaking change is possible.

But if a change breaks things and at the same time brings big benefits (relative to the breakage)?
*shrug*

Never break userspace

Posted Feb 16, 2024 18:05 UTC (Fri) by bluca (subscriber, #118303) [Link] (8 responses)

> Quite honestly, systemd and udev also broke lots and lots of things about how the Linux operating system works.

Or in other words, different software work differently. Or, yet again, you are moving the goal posts. Because nobody ever said "systemd works in exactly the same way as your 1980s garden variety collection of shell scripts", in fact the idea was very much the opposite. Some compat layers for the main interfaces were provided, which were always clearly documented as sub-optimal and wonky and intended for transition purposes, and after 20 years or so we'll remove them too, with ample advance notice. But nobody ever claimed that every single workflow in existence would continue unchanged after switching.

In fact, we don't even make absolute claims such as "we never break compatibility, period". From time to time we do breaking changes, and we try to announce them in advance, and for really impactful ones we try to get consensus on the mailing list first, and in rare cases we even try to help distributions migrate ahead of time to ensure the impact is nominal only - see when we dropped support for unmerged-usr last year for example - this happened in v255, and nobody noticed.
Sometimes things break accidentally, and sometimes they get fixed and sometimes they don't.

But what we most certainly don't do, is going around claiming "we never break compatibility", and I certainly don't use such a claim to start firing a bogus CVE for each commit that I backport to every stable branch I maintain.

See where the difference is?

Never break userspace

Posted Feb 16, 2024 18:59 UTC (Fri) by mb (subscriber, #50428) [Link] (7 responses)

>Sometimes things break accidentally, and sometimes they get fixed and sometimes they don't.

>See where the difference is?

Nope.
It's exactly the same thing. Things change. Deal with it like everybody has to deal with systemd.

Never break userspace

Posted Feb 16, 2024 19:03 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (6 responses)

> It's exactly the same thing. Things change. Deal with it like everybody has to deal with systemd.

The difference is that kernel developers have publicly committed to never breaking userspace. Systemd developers haven't. It is the disconnect between the public messaging and reality that's causing the contention. Not the changes themselves necessarily.

Never break userspace

Posted Feb 16, 2024 19:10 UTC (Fri) by mb (subscriber, #50428) [Link] (3 responses)

>The difference is that kernel developers have publicly committed to never breaking userspace

Things like uevents, tracepoints, sysfs files, etc... were pretty much never part of that claim.
Devs try hard to not make unnecessary breakages, but if a sysfs file disappears/changes or an uevent changes, programs have to deal with it.
Has always been like that.

> It is the disconnect between the public messaging and reality that's causing the contention.

The disconnect between the expectation and the reality is causing the contention.

Never break userspace

Posted Feb 16, 2024 19:21 UTC (Fri) by bluca (subscriber, #118303) [Link] (2 responses)

> Things like uevents, tracepoints, sysfs files, etc... were pretty much never part of that claim.

Citation needed. That is very much not evident from any claim anybody has ever made that I have seen.

Never break userspace

Posted Feb 16, 2024 19:41 UTC (Fri) by mb (subscriber, #50428) [Link] (1 responses)

>Citation needed

Even syscalls have been removed in the past, breaking applications.
BUT these applications always were very limited in count and usually part of the OS itself.

Citation: Look at the sources.

There has never been a thing like a general stability guarantee.
It always has been a matter of common sense.
If a change only breaks udev or systemd and nothing else, it might make sense to do it.

Never break userspace

Posted Feb 16, 2024 20:07 UTC (Fri) by bluca (subscriber, #118303) [Link]

Excellent, you have now demonstrated that my original post, claiming that there is no such thing as 'kernel never breaks userspace', was indeed correct, and thus 'just update to the latest' cannot be suggested as a solution to security issues since public interfaces will be broken left and right with no regard for backward compatibility. Congrats!

> If a change only breaks udev or systemd and nothing else, it might make sense to do it.

I beg to differ

Never break userspace

Posted Feb 16, 2024 19:25 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> The difference is that kernel developers have publicly committed to never breaking userspace.

Where?

Okay, I know Linus says "never break user-space", and he is very strict about it. But at the end of the day, shit happens.

And there's plenty of kernel developers who *haven't* signed up to it. They just know that trying to get it past Linus is not a battle worth fighting most of the time.

There's one big example I can think of, that had a rather nasty fall-out, in the raid world. So bad, in fact, that kernels were modified to have an explicit "fail to boot" config, iirc!

Something to do with the fact that raid layout was accidentally changed. So you have pre-change kernels that will trash post-change arrays, pre-discovery kernels that will trash pre-change arrays, and post-discovery kernels that will refuse to access arrays without a "this is a pre/post-layout flag".

Sometimes that's all you can do :-(

Cheers,
Wol

Never break userspace

Posted Feb 16, 2024 20:04 UTC (Fri) by bluca (subscriber, #118303) [Link]

In this very thread you can see such statements

Never break userspace

Posted Feb 18, 2024 5:41 UTC (Sun) by ras (subscriber, #33059) [Link]

I smiled at the popcorn reference, thinking it was a joke. My bad. It was advice. Good advice, as it turned out.