Never break userspace
Never break userspace
Posted Feb 16, 2024 16:06 UTC (Fri) by corbet (editor, #1)In reply to: Never break userspace by bluca
Parent article: A turning point for CVE numbers
As yes, the BIND/UNBIND thing was a big enough deal that I wrote about it at the time. What I suggested there might still seem to make sense: rather than sniping at the kernel community from the sideline, work with them to improve the situation. Let Thorsten know about regressions, preferably early enough to keep them from making it into a release. Things can be improved.
I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.
Posted Feb 16, 2024 16:49 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (10 responses)
These things do get reported, and they get ignored/shrugged away if you are lucky, and if not you get taken for a long ride. For this case it's explained in the link above as well. For the overlayfs case Google even went as far as sending 20 revisions of a patchset to try and restore backward compatibility, albeit optionally, and it was stonewalled: https://lore.kernel.org/lkml/20211117015806.2192263-1-dva...
So yeah, trying and dispelling this myth that "the kernel doesn't break userspace" is pretty much all that's left. Reading blatantly false statements being made irks me really badly, especially when used to justify some potentially damaging process changes as it happened here.
Posted Feb 16, 2024 17:01 UTC (Fri)
by mb (subscriber, #50428)
[Link] (9 responses)
Quite honestly, systemd and udev also broke lots and lots of things about how the Linux operating system works.
But the correct answer to users complaining often enough is to "ignore it" or "shrug it away".
What should be avoided is breaking changes that don't have positive sides. Changes just for the sake of changing and breaking things. That is bad and must be avoided. And it should always be considered, if a non-breaking change is possible.
But if a change breaks things and at the same time brings big benefits (relative to the breakage)?
Posted Feb 16, 2024 18:05 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (8 responses)
Or in other words, different software work differently. Or, yet again, you are moving the goal posts. Because nobody ever said "systemd works in exactly the same way as your 1980s garden variety collection of shell scripts", in fact the idea was very much the opposite. Some compat layers for the main interfaces were provided, which were always clearly documented as sub-optimal and wonky and intended for transition purposes, and after 20 years or so we'll remove them too, with ample advance notice. But nobody ever claimed that every single workflow in existence would continue unchanged after switching.
In fact, we don't even make absolute claims such as "we never break compatibility, period". From time to time we do breaking changes, and we try to announce them in advance, and for really impactful ones we try to get consensus on the mailing list first, and in rare cases we even try to help distributions migrate ahead of time to ensure the impact is nominal only - see when we dropped support for unmerged-usr last year for example - this happened in v255, and nobody noticed.
But what we most certainly don't do, is going around claiming "we never break compatibility", and I certainly don't use such a claim to start firing a bogus CVE for each commit that I backport to every stable branch I maintain.
See where the difference is?
Posted Feb 16, 2024 18:59 UTC (Fri)
by mb (subscriber, #50428)
[Link] (7 responses)
>See where the difference is?
Nope.
Posted Feb 16, 2024 19:03 UTC (Fri)
by rahulsundaram (subscriber, #21946)
[Link] (6 responses)
The difference is that kernel developers have publicly committed to never breaking userspace. Systemd developers haven't. It is the disconnect between the public messaging and reality that's causing the contention. Not the changes themselves necessarily.
Posted Feb 16, 2024 19:10 UTC (Fri)
by mb (subscriber, #50428)
[Link] (3 responses)
Things like uevents, tracepoints, sysfs files, etc... were pretty much never part of that claim.
> It is the disconnect between the public messaging and reality that's causing the contention.
The disconnect between the expectation and the reality is causing the contention.
Posted Feb 16, 2024 19:21 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (2 responses)
Citation needed. That is very much not evident from any claim anybody has ever made that I have seen.
Posted Feb 16, 2024 19:41 UTC (Fri)
by mb (subscriber, #50428)
[Link] (1 responses)
Even syscalls have been removed in the past, breaking applications.
Citation: Look at the sources.
There has never been a thing like a general stability guarantee.
Posted Feb 16, 2024 20:07 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
> If a change only breaks udev or systemd and nothing else, it might make sense to do it.
I beg to differ
Posted Feb 16, 2024 19:25 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (1 responses)
Where?
Okay, I know Linus says "never break user-space", and he is very strict about it. But at the end of the day, shit happens.
And there's plenty of kernel developers who *haven't* signed up to it. They just know that trying to get it past Linus is not a battle worth fighting most of the time.
There's one big example I can think of, that had a rather nasty fall-out, in the raid world. So bad, in fact, that kernels were modified to have an explicit "fail to boot" config, iirc!
Something to do with the fact that raid layout was accidentally changed. So you have pre-change kernels that will trash post-change arrays, pre-discovery kernels that will trash pre-change arrays, and post-discovery kernels that will refuse to access arrays without a "this is a pre/post-layout flag".
Sometimes that's all you can do :-(
Cheers,
Posted Feb 16, 2024 20:04 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted Feb 18, 2024 5:41 UTC (Sun)
by ras (subscriber, #33059)
[Link]
Never break userspace
>
> I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.
Apparently breaking userspace can just be waved through, while fixing it requires "building a security model" and other extremely high-bars to be met. All in the meanwhile anybody using selinux needs to completely open up the security policy to make it work at all, of course, which I guess makes for a very interesting "security model". I could go on, but can't be bothered to look up yet more references.
Never break userspace
Administrators had to change tons of scripts, because some things suddenly worked differently after the distribution updated from classic init to systemd.
Things work differently now. Get used to it. That's the correct answer surprisingly often.
That's true for systemd and it's also true for parts of the kernel.
*shrug*
Never break userspace
Sometimes things break accidentally, and sometimes they get fixed and sometimes they don't.
Never break userspace
It's exactly the same thing. Things change. Deal with it like everybody has to deal with systemd.
Never break userspace
Never break userspace
Devs try hard to not make unnecessary breakages, but if a sysfs file disappears/changes or an uevent changes, programs have to deal with it.
Has always been like that.
Never break userspace
Never break userspace
BUT these applications always were very limited in count and usually part of the OS itself.
It always has been a matter of common sense.
If a change only breaks udev or systemd and nothing else, it might make sense to do it.
Never break userspace
Never break userspace
Wol
Never break userspace
Never break userspace