LWN: Comments on "A turning point for CVE numbers"

A turning point for CVE numbers

ju3Ceemi — Wed, 13 Mar 2024 12:45:23 +0000

You are correct : kernel upgrades on a sane system is not disruptive

When I need it, I just reboot the boxes (one by one, unless monitoring cries)

"If you cannot handle maintenances, you cannot handle incidents"

A turning point for CVE numbers

pizza — Sat, 09 Mar 2024 01:21:49 +0000

> This is a pretty large blanket statement that definitely isn't always true.

I didn't claim it was true in a general sense; I only claimed it was true for the Linux kernel.

A turning point for CVE numbers

DanilaBerezin — Sat, 09 Mar 2024 00:50:37 +0000

> It's _vastly_ cheaper to keep using it than to replace it with something else. By multiple orders of magnitude.

This is a pretty large blanket statement that definitely isn't always true. If it were as true as you claim it was, things wouldn't fade into obscurity or ever be replaced. There are plenty of conditions where replacing something is cheaper than continuing to use it. X is a great recent example.

A turning point for CVE numbers

DemiMarie — Thu, 07 Mar 2024 05:58:12 +0000

Why is a kernel upgrade disruptive? That seems like an indication that the application doesn’t have proper clustering support. Obviously, that’s quite possible, but my understanding is that best practice nowadays is to ensure that rebooting a server is just not a bug deal.

There are obviously many exceptions to this (HPC and embedded systems come to mind), but this seems to be the general rule. As far as stable kernel releases breaking stuff, is that something that should be caught in testing, preferable before the new stable tree is released?

A turning point for CVE numbers

DemiMarie — Thu, 07 Mar 2024 05:50:41 +0000

What about testing stable kernels, to ensure that they are actually stable?

An additional quote

Aissen — Thu, 22 Feb 2024 08:36:01 +0000

Anyone who cares about that should run production canaries and do actual regression testing. Yes, the Linux kernel testing strategy is lacking, but why put the onus on the project to do what *you* care about ? Anyone can build and test for regressions. And even do that in advance for the -rc versions (stable or upstream), so that when the release comes, you already know if it's breaking anything.

A turning point for CVE numbers

bluca — Tue, 20 Feb 2024 19:41:36 +0000

> But at any rate, the time of the go-lucky approach of installing Debian stable and believing the system will never come down after an update is over.

Debian stable regularly ships upstream kernel stable releases. Which is a problem, as we found out a couple of months ago, as "stable" kernels are not stable at all and can corrupt your disks and require a reinstall and restore from backups.

A turning point for CVE numbers

pizza — Tue, 20 Feb 2024 15:34:05 +0000

> But time has moved on, and we expect more for less money, and to some extent, we get it

Note that "for less money" in practice, means an increasing unwillingness to pay _anything at all_, because "something else is paying/subsidizing the cost of service"

(And one of those "something elses" is our service provider snooping on everything we do, including our at-rest data, finding "interesting" things to monetize. But hey, it's not "money", so that's fine!)

A turning point for CVE numbers

farnz — Tue, 20 Feb 2024 14:48:13 +0000

Your anecdote links to a known change we're seeing in the software world: failure is less and less of an option over time. Back In The Day™ (for various values of back in the day), it was fine to depend on user complaints to tell you if a service was running or not. It was fine for anyone who could telnet to a host to be able to log in as root with just a plaintext password to authenticate them. It was fine for a system to have a few days downtime while broken hardware got replaced. It was fine for sysadmins to go digging in people's files just to see if there was something interesting in there.

None of this is OK any more; arguably, much of it was never OK, it was just accepted because doing better cost more than people were willing to pay. But time has moved on, and we expect more for less money, and to some extent, we get it - I can pay someone like Fastmail for better e-mail service than I used to be able to get from an in-house server, backed up by improvements to connectivity (where my LAN might have shared a single dial-up link 30 years ago, I've now got high speed Internet that's faster than the LAN speeds I got 30 years ago, and mail protocols designed to cope with the latency added by going to an outside datacentre instead of to a machine on the 10BASE2 network).

A turning point for CVE numbers

pizza — Tue, 20 Feb 2024 13:42:16 +0000

> But at any rate, the time of the go-lucky approach of installing Debian stable and believing the system will never come down after an update is over.

What's the standard financial disclaimer... "past performance is no guarantee of future success"?

I once publicly called out someone (who _definitely_ should have known better) for professional incompetence after they went on a "systemd is responsble for everything wrong with society!!!111" rant after something went wrong on a Debian 9 (I think) upgrade on a critical system. A remote, (completely) headless critical system.

...Because you don't do _any_ updates on critical systems without some measure of testing first. Or, at minimum, some sort of reversion/recovery procedure. While even basic (end-user) smoke tests would have caught this particular failure [1] the fact that there wasn't any thought given to recovering from an update failure (not even "remote hands" capable of hooking up and looking at the local console) was inexcusable.

[1] Due to non-Debian-supplied software failing to start properly and systemd actually catching the failure instead of ignoring it.

A turning point for CVE numbers

gmgod — Tue, 20 Feb 2024 08:50:16 +0000

Amen to that

A turning point for CVE numbers

gmgod — Tue, 20 Feb 2024 08:48:39 +0000

Yeah yeah, in the real world people refuse to go see their doctor because they are afraid they might tell them they have cancer or any other grave illness...

The message is clear: you want security, you use the latest kernel (LTS is probably fine because fixes are backported there too).

I do agree and I do think that if the plan goes through, the kernel devs will have to up their testing game.

But at any rate, the time of the go-lucky approach of installing Debian stable and believing the system will never come down after an update is over. You talk about "real problems people have"... If you can afford a hardware failure or a borked package after such an upgrade, you can afford a kernel "failure". If you can't, there are already measures in place to handle those, "bad" kernel update included.

And again, this will go both ways. I bet you we'll see a lot of investment in testing in the next couple years.

A turning point for CVE numbers

error27 — Mon, 19 Feb 2024 13:59:34 +0000

Sasha's example is from 2024 but people have been complaining about the "file for a CVE" life hack for years... I'm not in a position to know how common it is to do it in real life though.

A turning point for CVE numbers

sashal — Mon, 19 Feb 2024 13:53:34 +0000

What authority? Anyone can request a CVE against a CNA.

An additional quote

gmgod — Mon, 19 Feb 2024 09:10:34 +0000

I know that's tongue-in-cheek kind of thing but backported kernels are not the panacea stability-wise (compared to LTS or even normal kernels).

Plus, and that's the whole point, pulling a google and updating the kernel once in a bluemoon, using "critical" CVEs as an indicator for backports and most ARM boards not even doing that and giving you the finger are not practices that should be condoned. This kind of backporting gives you an illusion of stability and security (it does improve ABI stability a fair bit but at the expense of other things).

I know the term has lost quite a fair bit of meaning but I'd argue the whole point of devop-ing is precisely developing ways to perform updates (and rollbacks) that are the most seamless so that your systems are always up and if something had to be rolled-back, gives you and your team the time to sort it out.

If you do so often, it's only little bits and bobs you need to fix here and there. If you don't and use Ubuntu LTS, it's a full migration, with many arguments, tears and pain every time you switch to a new LTS.

The debian model is catered towards sysadmins who install everything by hand once and generally would be in it very deep if any of their servers actually fell because of hardware or because config became invalid after an update.

If kernel updates are all that is possible to really get a secure one (because backports require too much work), the Linux ecosystem will evolve. Debian, for instance, will probably track LTS kernels instead but will also ask the kerbel dev to be much more methodic with how changes are made and documented.

The kernel devs' point is that CVEs are rubbish. A lot of them being redundant or not real and a lot of them downplaying the impact they have on the system. My current understanding is that backporting doesn't scale well. But we'll see how it pans out. In the meantime, if an "unfortunate" kernel upgrade has large business impact, try to think and implement ways to negate that impact. We are mostly using on-prem resources here and yet I can redeploy bare-metal servers as I wish and most our OSes provided automatic rollback should anything go wrong. That's the way to go, instead of complaining an update, any update, broke something in prod.

A turning point for CVE numbers

apollock — Mon, 19 Feb 2024 05:46:35 +0000

I sincerely hope they will.

The CVE 5 schema has a variety of ways of expressing machine-readable data about the vulnerability

https://cveproject.github.io/cve-schema/schema/v5.0/docs/...

I spent a lot of energy dealing with not-so-machine-readable data to convert non-CVE 5 schema CVE records to OSV for https://osv.dev/blog/posts/introducing-broad-c-c++-support/

A turning point for CVE numbers

mdolan — Sun, 18 Feb 2024 16:00:59 +0000

Users/vendors can pretend security bugs are not there today because there's no CVE published... but the bugs are there and they were fixed. The bad actors know this and track what changed. We know they're watching. Users/vendors not paying attention are big targets. Ivanti VPNs are an extreme, but real time example.

If you want to keep pretending and hoping, just filter the kernel CNA out. Greg has been telling everyone for a decade+ the solution is to keep updating with the LTS kernel. If you've done the engineering required to implement that process, nothing changes. If you ignored him and others and are still pretending... government regulation is here+more_coming. Like it or not, I'm sorry to say this isn't optional anymore, and it's not just going to be the kernel changing. I remember the days of happily running my own FTP+SMTP server. Things change.

Never break userspace

ras — Sun, 18 Feb 2024 05:41:43 +0000

I smiled at the popcorn reference, thinking it was a joke. My bad. It was advice. Good advice, as it turned out.

A turning point for CVE numbers

rwmj — Sat, 17 Feb 2024 07:34:12 +0000

I had to check the date while reading this article to make sure it wasn't April Fools Day. The only outcome of this is that kernel-issued CVEs will be ignored. I hope there's some automated way to look up which CNA issued a CVE number.

A turning point for CVE numbers

Darakian — Fri, 16 Feb 2024 23:30:46 +0000

Ya basically, but ideally part of the kernel org and making security claims about commits rather than compiled objects

Never break userspace

bluca — Fri, 16 Feb 2024 20:07:22 +0000

Excellent, you have now demonstrated that my original post, claiming that there is no such thing as 'kernel never breaks userspace', was indeed correct, and thus 'just update to the latest' cannot be suggested as a solution to security issues since public interfaces will be broken left and right with no regard for backward compatibility. Congrats!

> If a change only breaks udev or systemd and nothing else, it might make sense to do it.

I beg to differ

Never break userspace

bluca — Fri, 16 Feb 2024 20:04:24 +0000

In this very thread you can see such statements

Never break userspace

mb — Fri, 16 Feb 2024 19:41:51 +0000

>Citation needed

Even syscalls have been removed in the past, breaking applications.
BUT these applications always were very limited in count and usually part of the OS itself.

Citation: Look at the sources.

There has never been a thing like a general stability guarantee.
It always has been a matter of common sense.
If a change only breaks udev or systemd and nothing else, it might make sense to do it.

Never break userspace

Wol — Fri, 16 Feb 2024 19:25:40 +0000

> The difference is that kernel developers have publicly committed to never breaking userspace.

Where?

Okay, I know Linus says "never break user-space", and he is very strict about it. But at the end of the day, shit happens.

And there's plenty of kernel developers who *haven't* signed up to it. They just know that trying to get it past Linus is not a battle worth fighting most of the time.

There's one big example I can think of, that had a rather nasty fall-out, in the raid world. So bad, in fact, that kernels were modified to have an explicit "fail to boot" config, iirc!

Something to do with the fact that raid layout was accidentally changed. So you have pre-change kernels that will trash post-change arrays, pre-discovery kernels that will trash pre-change arrays, and post-discovery kernels that will refuse to access arrays without a "this is a pre/post-layout flag".

Sometimes that's all you can do :-(

Cheers,
Wol

Never break userspace

bluca — Fri, 16 Feb 2024 19:21:47 +0000

> Things like uevents, tracepoints, sysfs files, etc... were pretty much never part of that claim.

Citation needed. That is very much not evident from any claim anybody has ever made that I have seen.

Never break userspace

mb — Fri, 16 Feb 2024 19:10:18 +0000

>The difference is that kernel developers have publicly committed to never breaking userspace

Things like uevents, tracepoints, sysfs files, etc... were pretty much never part of that claim.
Devs try hard to not make unnecessary breakages, but if a sysfs file disappears/changes or an uevent changes, programs have to deal with it.
Has always been like that.

> It is the disconnect between the public messaging and reality that's causing the contention.

The disconnect between the expectation and the reality is causing the contention.

Never break userspace

rahulsundaram — Fri, 16 Feb 2024 19:03:02 +0000

> It's exactly the same thing. Things change. Deal with it like everybody has to deal with systemd.

The difference is that kernel developers have publicly committed to never breaking userspace. Systemd developers haven't. It is the disconnect between the public messaging and reality that's causing the contention. Not the changes themselves necessarily.

Never break userspace

mb — Fri, 16 Feb 2024 18:59:34 +0000

>Sometimes things break accidentally, and sometimes they get fixed and sometimes they don't.

>See where the difference is?

Nope.
It's exactly the same thing. Things change. Deal with it like everybody has to deal with systemd.

Never break userspace

bluca — Fri, 16 Feb 2024 18:05:50 +0000

> Quite honestly, systemd and udev also broke lots and lots of things about how the Linux operating system works.

Or in other words, different software work differently. Or, yet again, you are moving the goal posts. Because nobody ever said "systemd works in exactly the same way as your 1980s garden variety collection of shell scripts", in fact the idea was very much the opposite. Some compat layers for the main interfaces were provided, which were always clearly documented as sub-optimal and wonky and intended for transition purposes, and after 20 years or so we'll remove them too, with ample advance notice. But nobody ever claimed that every single workflow in existence would continue unchanged after switching.

In fact, we don't even make absolute claims such as "we never break compatibility, period". From time to time we do breaking changes, and we try to announce them in advance, and for really impactful ones we try to get consensus on the mailing list first, and in rare cases we even try to help distributions migrate ahead of time to ensure the impact is nominal only - see when we dropped support for unmerged-usr last year for example - this happened in v255, and nobody noticed.
Sometimes things break accidentally, and sometimes they get fixed and sometimes they don't.

But what we most certainly don't do, is going around claiming "we never break compatibility", and I certainly don't use such a claim to start firing a bogus CVE for each commit that I backport to every stable branch I maintain.

See where the difference is?

Never break userspace

mb — Fri, 16 Feb 2024 17:01:54 +0000

> These things do get reported, and they get ignored/shrugged away if you are lucky

Quite honestly, systemd and udev also broke lots and lots of things about how the Linux operating system works.
Administrators had to change tons of scripts, because some things suddenly worked differently after the distribution updated from classic init to systemd.

But the correct answer to users complaining often enough is to "ignore it" or "shrug it away".
Things work differently now. Get used to it. That's the correct answer surprisingly often.
That's true for systemd and it's also true for parts of the kernel.

What should be avoided is breaking changes that don't have positive sides. Changes just for the sake of changing and breaking things. That is bad and must be avoided. And it should always be considered, if a non-breaking change is possible.

But if a change breaks things and at the same time brings big benefits (relative to the breakage)?
*shrug*

Never break userspace

mb — Fri, 16 Feb 2024 16:54:13 +0000

I have said that you were right. You have said nothing wrong.
There is no such thing as a general interface stability guarantee.

I've just set things into perspective. That is no goal post moving.

Never break userspace

bluca — Fri, 16 Feb 2024 16:49:44 +0000

> As yes, the BIND/UNBIND thing was a big enough deal that I wrote about it at the time. What I suggested there might still seem to make sense: rather than sniping at the kernel community from the sideline, work with them to improve the situation. Let Thorsten know about regressions, preferably early enough to keep them from making it into a release. Things can be improved.
>
> I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.

These things do get reported, and they get ignored/shrugged away if you are lucky, and if not you get taken for a long ride. For this case it's explained in the link above as well. For the overlayfs case Google even went as far as sending 20 revisions of a patchset to try and restore backward compatibility, albeit optionally, and it was stonewalled: https://lore.kernel.org/lkml/20211117015806.2192263-1-dva...
Apparently breaking userspace can just be waved through, while fixing it requires "building a security model" and other extremely high-bars to be met. All in the meanwhile anybody using selinux needs to completely open up the security policy to make it work at all, of course, which I guess makes for a very interesting "security model". I could go on, but can't be bothered to look up yet more references.

So yeah, trying and dispelling this myth that "the kernel doesn't break userspace" is pretty much all that's left. Reading blatantly false statements being made irks me really badly, especially when used to justify some potentially damaging process changes as it happened here.

Never break userspace

bluca — Fri, 16 Feb 2024 16:08:20 +0000

> Therefore, do you have examples of real user applications breaking, that are not part of the OS?

Need some help with all that goal post moving? Must be exhausting, going that far

Never break userspace

corbet — Fri, 16 Feb 2024 16:06:37 +0000

As yes, the BIND/UNBIND thing was a big enough deal that I wrote about it at the time. What I suggested there might still seem to make sense: rather than sniping at the kernel community from the sideline, work with them to improve the situation. Let Thorsten know about regressions, preferably early enough to keep them from making it into a release. Things can be improved.

I have to say, Luca, that I would expect a systemd developer to understand how this kind of constant badmouthing from outside can make an environment toxic; systemd has certainly suffered its share of that. Why continue with that pattern? A more constructive approach might work wonders.

Never break userspace

mb — Fri, 16 Feb 2024 16:01:58 +0000

Well. Technically you are right. The kernel frequently breaks interfaces.
But in reality, systemd and udev are parts of the operating system.
So I don't care that much, if the OS breaks itself. That will get fixed eventually.

What I and most users care about is whether actual user applications break.
And that extremely rarely happens.
I run decades old binaries that work just fine.

It doesn't affect my application, because the OS as a whole still works as before, after porting systemd/udev to the new interfaces. A combination of updated kernel and incompatible systemd/udev would never hit stable distributions.

Therefore, do you have examples of real user applications breaking, that are not part of the OS?

A turning point for CVE numbers

pizza — Fri, 16 Feb 2024 15:49:06 +0000

> Sounds like a problem in that company then

Incidently, that company was that way because *EU regulations required them to be*.

(They laid off my research team on the tail end of a major process/policy revamp brought about by new regulations soon to come into effect. I was made to endure many training sessions about how those new/updated regulations affected every part of the overall product lifecycle, from early design to manufacturing to label placement/content to post-sales support to how end-of-life would be handled)

So it's not "that company's problem" so much as "the problem of any company operating in a regulated space"

Never break userspace

bluca — Fri, 16 Feb 2024 15:36:42 +0000

Sure, here's one:

https://lists.freedesktop.org/archives/systemd-devel/2022...
https://lists.freedesktop.org/archives/systemd-devel/2022...

The other one that hit me personally was when overlayfs was made incompatible with selinux. And then there's all the times that netlink changed. And all the time uevents changed. And all the times sysfs changed. And so on and so forth. The reality is that "we don't break userspace" is a nice story that kernel developers like to go around tell anybody who's willing to listen, but it's just that, a story. They barely care about syscall ABI stability, and even that gets broken from time to time as already pointed out by another comment.

A turning point for CVE numbers

pizza — Fri, 16 Feb 2024 15:32:59 +0000

> Sounds like a problem in that company then, why should that justify breaking everything for everybody else?

*shrug* You made an assertion such organizations do not exist (because you didn't know any) and used that to accuse others of making things up or otherwise speaking in bad faith. You were incorrect on both fronts.

You're free to argue that the current status quo has problems (or not). You're free to talk about *your* experiences, and how proposed actions by others will have ill effects on you or third parties.

But you don't get to claim that other people's direct experiences are wrong, incorrect, or irrelevant, and accuse them of bad faith for taking steps to improve the messes they are dealing with, "because you have no idea about either".

Never break userspace

corbet — Fri, 16 Feb 2024 14:28:05 +0000

Examples? Preferably with pointers to the discussion around the issue? I don't doubt there are places where the kernel community is failing to live up to its goals, but it's hard to make things better without some clarity about where the problem exists.

A turning point for CVE numbers

bluca — Fri, 16 Feb 2024 14:09:16 +0000

> I worked for a company that had such a policy.

Sounds like a problem in that company then, why should that justify breaking everything for everybody else?

> Respectfully, you need to STFU about stuff that is outside your realm of expertise and experience.

Respectfully, you need to STFU about my expertise and experience, because you have no idea about either (just like I don't about yours)