LWN: Comments on "An Ubuntu kernel bug causes container crashes"

An Ubuntu kernel bug causes container crashes

daenzer — Thu, 28 Jul 2022 07:45:32 +0000

Any non-trivial CI requires constant development & maintenance effort. I'd argue it's effort well spent though.

An Ubuntu kernel bug causes container crashes

wtarreau — Tue, 12 Jul 2022 04:30:08 +0000

Thanks for this, it's very important to have as many -rc users as possible, precisely for the reasons you explained well.

An Ubuntu kernel bug causes container crashes

farnz — Mon, 11 Jul 2022 18:16:55 +0000

The trouble is that stable kernels do contain bugs all over the shop, some of which are exploitable. So the question becomes not "are there bugs in my EOL kernel?", to which the answer is definitely "yes", but "are the bugs in my EOL kernel of concern to me, given that I do not know the scope and impact of the bugs in my kernel?", which is a much harder question to answer.

And it's made exponentially harder by regressions in newer kernels which means that there's no good answer - do you take a newer kernel that fails to boot one time in 10 because your PCIe GPU is left in a bad state by firmware, or stick to the older kernel that has a remotely exploitable bug that you don't know about that lets an intruder escalate privileges on your machine.

Ideally, there would simply not be regressions in the kernel, so updating would always be the right thing to do. But that's not the world we live in; my experience is that I'm better off taking Linus's recent release, finding regressions and reporting them ASAP (so that the bug reports go to people who've been working in the right bits of the kernel recently, and bisect is often possible in reasonable time) than putting off updates for as long as possible and then reporting a huge number of regressions in one go, but other people will have had other experiences.

An Ubuntu kernel bug causes container crashes

wtarreau — Mon, 11 Jul 2022 16:54:04 +0000

> EOLing a kernel does not magically cause bugs to appear.

No but one thing is certain, it will not magically fix all those that are discovered daily and that affect it.
For sure the best way not to know about bugs is to use an EOL version that doesn't receive fixes.

> In particular, the current stable kernel needs to contain 2000 bugs so that when it will be EOLed, it will miss 2000 fixes.

Maybe more maybe less, who knows.

> > In particular, "some will corrupt data, cause random hangs, disconnect your WiFi during an audio conf, make your screen disappear after resume, leave phantom USB devices after some errors, let an intruder escalate privileges on your machine, etc."
> This is not reassuring.

But that's why there are LTS kernels for those who want to stick as long as possible to what works best for them. Some people only deploy a kernel on sensitve systems after one year, so that most of the recent regressions are out of the way. I personally deploy new LTS kernels on my laptop so that I can spot changes or bugs early, and have time to get them fixed before these kernels need to reach servers. That's reasonable.

An Ubuntu kernel bug causes container crashes

wtarreau — Mon, 11 Jul 2022 16:47:21 +0000

Oh I've known such people and even had to combat them because they were refusing to apply the mandatory fixes for a bug that was causing their firewall to leak memory like crazy and crash every two weeks or so. Instead they wanted to profit from a downtime caused by the crash to install more RAM and postpone the next crash! When insisting for applying fixes, to the question "but if it bugs?" I had to respond "in the worst case it will continue not to work".

But what you seem to be ignoring here is that the older the kernel, the harder it is to backport fixes, and the most likely they are to be wrong, particularly when taken out of the context of all other fixes that were surrounding the original patch. When I was a stable maintainer, I used to receive many messages like "do not take this patch without this one" or "I'll provide you a different one for this version as it's not sufficient" etc. The risk of getting a fix wrong when applying it yourself to a tree without the author's approval is quite high. Thus in addition to missing tons of fixes, the few you get (the so called "security fixes" that make vendors sell) are often bogus and are the ones that will take your system down.

Really, do not use EOL kernels.

An Ubuntu kernel bug causes container crashes

ballombe — Mon, 11 Jul 2022 09:38:35 +0000

EOLing a kernel does not magically cause bugs to appear.
In particular, the current stable kernel needs to contain 2000 bugs so that when it will be EOLed, it will miss 2000 fixes. In particular, "some will corrupt data, cause random hangs, disconnect your WiFi during an audio conf, make your screen disappear after resume, leave phantom USB devices after some errors, let an intruder escalate privileges on your machine, etc."
This is not reassuring.

An Ubuntu kernel bug causes container crashes

jordan — Sun, 10 Jul 2022 19:33:50 +0000

Apologies; I came to those numbers by comparing release dates and EOL dates, and I'm already bad enough at math without getting calendars involved.

An Ubuntu kernel bug causes container crashes

mgedmin — Sun, 10 Jul 2022 14:02:24 +0000

> In contrast to the LTS releases, these releases are only supported for about six months, and are declared end-of-life (EOL) a month after a newer release is made available.

Ubuntu's non-LTS releases are supported for 9 months, and are declared EOL three months after a newer release is made available.

(This doesn't affect the rest of the article in any way, it's just that my inner pedant cannot leave inaccurate information alone.)

An Ubuntu kernel bug causes container crashes

smurf — Sun, 10 Jul 2022 09:55:01 +0000

> so much work to ensure it won't crap on something we absolutely need to work...

So up a reasonable CI system. Surprise: you probably need that anyway.

Yes, that's somewhat more effort … but you only need to spend it once, not with every release.

An Ubuntu kernel bug causes container crashes

jafd — Sat, 09 Jul 2022 19:32:54 +0000

Look at this from another side.

> a non-LTS EOL kernel probably misses 2000 fixes, for as many bugs that are fixed in all maintained versions around it, but not that one.

What if on the systems running that kernel, none of the fixes touched modules actually used in them?

> Some will corrupt data, cause random hangs

Not experienced once for a year, let's say

> disconnect your WiFi during an audio conf, make your screen disappear after resume, leave fantom USB devices after some errors

Not happened once in the drivers actually used and on that specific hardware.

But what's more likely to happen is that a newer version, while bringing a minor fix to a module or a subsystem you need, will also bring a mighty regression in a driver or a subsystem your workflow absolutely depends upon. A couple articles ago someone commented about precisely this situation here on LWN [0].

That's why there exist users (think companies) which find a kernel that doesn't crap on their hardware 99.999% of the time, and pin it, and swear to never upgrade it ever. Have you thought they may have had enough of the Russian roulette?

Jumping from LTS to LTS can also be akin to jumping centuries in a time travel vehicle. So many changes, so many surprises, so much work to ensure it won't crap on something we absolutely need to work...

[0] https://lwn.net/Articles/889787/, you were in that thread too.

An Ubuntu kernel bug causes container crashes

roc — Thu, 07 Jul 2022 22:48:58 +0000

Changing the names of things when their meaning changes helps reduce these issues. But you're right, there's no way to fully avoid them.

Red Hat stable kernels

hkario — Thu, 07 Jul 2022 12:41:52 +0000

The package version indicates the oldest piece of code as-shipped-upstream in a particular package, it doesn't tell what's the newest piece is. And in particular, the kernel commonly pulls in new versions of whole subsytems, not just individual patches.

An Ubuntu kernel bug causes container crashes

bartoc — Thu, 07 Jul 2022 08:52:29 +0000

Heh I play a game (X4: foundations, they have a pretty good linux version) that uses an extremely cursed XML based scripting language (<if condition="..."></if><else></else> etc) with the explicit motivation that it lets them have mods automatically merge with each-other using xpath/xinclude or apply xslt stylesheets to each-other.

This ends up going a _little_ better than you would expect, but not much.

You can do cocci style semantic diff and merge, where the merge is based on the AST, not the text. But this means you need to parse everything perfectly and it's not that much better than a good text-based diff/merge algorithm. I suppose in a way this is what __attribute__((flatten)) or always_inline do.

Actually it would be kinda neat to have a compiler plugin or extension that let you inject code "somewhere else". You could have a compiler plugin that let you write functions to be injected at specific locations. This would be merely "neat" for these kinds of functionality patches but I think it could actually be a useful feature for injecting static tracepoints without cluttering up the code too much (or forgetting to add one someplace).

Come to think of it nim-lang has an experimental feature for this called "term rewriting macros", you can write something like (from the manual):
template optMul{`*`(a, 2)}(a: int): int = a+a

once this is brought into scope the compiler will rewrite any further expressions like some_ident * 2 as some_ident * some_ident. You can see how this is basically a megaton yield footgun. The matching language is kinda neat, but it's very sensitive to the exact structure of the AST, so it can miss stuff (nim has macros too, but these operate on a kind of alternate "stable" ast that converts too and from the "real" AST. That's harder for TR macros because any change in the "real" ast is usually something TR macros might be interested in, or might be able to break if they don't understand.

Because of these issues, and because TR macros are not widely used, and because they have some performance problems they may be removed.

Oh Mathamatica's "Wolfram Language" is famously based on this concept, and there is a lot of literature out there about how amazing it is. I remain unconvinced, but it is unconventional and interesting for sure.

Oh, you can imagine that once you have such a facility and multiple users pop up trying to use it at once things get quite interesting quite fast.

An Ubuntu kernel bug causes container crashes

bartoc — Wed, 06 Jul 2022 22:05:36 +0000

well, there are cocci checks for a ton of ownership issues in the kernel, if the internal change had come with such a cocci check and the ubuntu folks had run it then maybe this could have been caught, but why should it have been? It was an internal refactoring after all.

I have my doubts about rust’s ability to prevent this kind of thing, but stuff like that does help (thats why the kernel has lifetime related cocci checks)

An Ubuntu kernel bug causes container crashes

atnot — Wed, 06 Jul 2022 21:40:30 +0000

I think the only real way of solving this is by reducing the amount of similar programs that also compile but have undesirable behavior. Proving a patch can't introduce say a use-after-free is probably about as hard as proving it doesn't have a use-after-free in the first place. C is somewhat of a worst case there because it has very few mechanisms for constraining the callers or surrounding code.

An Ubuntu kernel bug causes container crashes

wtarreau — Wed, 06 Jul 2022 21:32:15 +0000

Quite frankly, bugs happen, and one can always contest how shameful it is that a given bug was not caught earlier, but bugs are just this, something that evades humans' logic. And even if it was one of their own in-house patches that caused the breakage, I won't necessarily blame their developers for that. All of us caused a mess at least once with a shameful patch.

What's irritating me however (and has for a while) is their insistence on using EOL kernels. They made a good progress on their LTS branches but honestly, "upgrading" from a maintained kernel to an unmaintained one to get new drivers is really not acceptable. For having produced stable kernels myself, I can say it, **EOL kernels are completely bogus**, because the flow of patches that need to be backported is steady, but what happens when the kernel reaches EOL ? They're not merged anymore. After one year a non-LTS EOL kernel probably misses 2000 fixes, for as many bugs that are fixed in all maintained versions around it, but not that one. Some will corrupt data, cause random hangs, disconnect your WiFi during an audio conf, make your screen disappear after resume, leave fantom USB devices after some errors, let an intruder escalate privileges on your machine, etc. There are now huge efforts from the kernel community to provide a wide choice of high-quality LTS kernels, and there is absolutely zero (**ZERO**) excuse nowadays for any distro to ship a kernel that reaches EOL before the end of support of the distro (and even worse before the release here). This bad practice is irresponsible and must stop!

An Ubuntu kernel bug causes container crashes

NYKevin — Wed, 06 Jul 2022 17:39:45 +0000

You can't do this without either using an incredibly strict language or an incredibly broad definition of "breakage." Any time code interacts with state, the state is subject to some set of invariants (e.g. "when foo() returns -1, errno can take on these values..."). Most languages explicitly encode only a small subset of those invariants in their type systems, and leave everything else for comments and documentation. Even languages like SQL, where most invariants are encoded as constraints, still have business logic invariants (e.g. "the value in the price column is the real-world price of the item, excluding tax"), which usually cannot be expressed or checked by a type checker (sure, you can check for negative or zero price, but you can't check for "the price is too expensive for this item, because somebody changed the price column to include tax"). So you have to use a broad definition of "breakage" - X "breaks" Y if X touches any state which Y later reads. But that means that everything breaks everything, in practice.

The other problem is that your graph is probably not acyclic, because most nontrivial programs contain loops. So not only does X break Y, Y also probably breaks X, meaning that you can't automatically order X and Y with respect to one another.

An Ubuntu kernel bug causes container crashes

iabervon — Wed, 06 Jul 2022 15:16:41 +0000

I think you might be able to get useful results if you're not looking for whether the patch is correct (since people already try that), but instead whether the patch works differently in the new kernel as compared to the one it was written for.

I could see there being relatively few patches flagged for "between 5.x.y and 5.x.y+1, lines your patch affects are part of data flow analysis that doesn't match". It seems like it would give a definite warning for the reference count of an object the patch passes to fput() necessarily being different in 5.13 as compared to 5.8. It wouldn't be able to tell which one is correct, or whether the code had two references and is just releasing them in the opposite order now, but it would be clear that there's some sort of conflict resolution needed.

Red Hat stable kernels

pbonzini — Wed, 06 Jul 2022 15:00:34 +0000

Yes that's pretty accurate. :)

Red Hat stable kernels

nim-nim — Wed, 06 Jul 2022 10:14:43 +0000

Well it’s more like doing the work before others, what you see in LTS kernels will have often been exercised as a rh patch in their own products.

Red Hat stable kernels

taladar — Wed, 06 Jul 2022 07:43:56 +0000

I think of the Red Hat way of doing things less as "shipping older kernels" and more as "lying about version numbers".

An Ubuntu kernel bug causes container crashes

smurf — Wed, 06 Jul 2022 07:41:16 +0000

Well, there's some call flow analysis tools out there that tries to discover ownership issues like the one that caused this problem.

However, C isn't a particularly nice language to do that with, particularly when you start using these tools after the fact you get a heap of false positives.

The real solution is to switch to a language with built-in object lifecycle guarantees. In this case, Rust.

An Ubuntu kernel bug causes container crashes

developer122 — Wed, 06 Jul 2022 05:53:33 +0000

I suppose that you might be able to generate metadata for a piece of software in the form of a directed acyclic graph representing a kind of causality or dependency. Something that touches one branch isn't blocked by something on another branch, but something upstream blocks downstream patches due to changed downstream-visible behavior. With language fine-tuning you could manage what does and does not cause a downstream-visible change of behavior, and what behavior precisely a patch depends on. Dunno whether most software lends itself to a good directed acyclic graph of behavior causality though.

An Ubuntu kernel bug causes container crashes

derobert — Wed, 06 Jul 2022 04:10:01 +0000

I think you're right about it being in general impossible: after all, your patch could add a line after some function call f(). If f() doesn't return (which is the halting problem), then your patch is fine (it is never executed). So in general, it's not doable.

You can surely do a bunch of control & data flow analysis, but I suspect you'll get far too many false positives. After all, if it tells you to investigate half the patches, it's not that useful.

If this is really any "docker run", then that's an embarrassing testing failure. (Especially if it hits other container engines, like k8s).

Cloud images probably shouldn't be getting HWE kernels, at least not until a few months after desktop.

An Ubuntu kernel bug causes container crashes

DSMan195276 — Wed, 06 Jul 2022 02:38:37 +0000

I think the issue there is that in this case the patch applied correctly to the right locations, but the internals had changed in some way causing it to be wrong. It's entirely possible the relevant changes were in others source files, so unless you're hashing an entire directory tree or the entire source you still might not catch it. And at that point you're really just saying you should review every patch you're applying every time you apply it. Which might very well be true :D but I'm not sure you need the hashing at that point.

I think the more straightforward solution here is just testing after the patch is applied to verify the functionality still works. That's what we really care about anyway, staring at the code only gets you so far if you never actually try it. And also, don't maintain your own patches if you can avoid it :D

An Ubuntu kernel bug causes container crashes

NightMonkey — Wed, 06 Jul 2022 01:38:29 +0000

Naive sysadmin here. Couldn't a hash of the section to patch be included in the metadata of the patch? Then, if the hash doesn't match, the patch cleanly fails if the section has changed? Or is this already done and there is another angle I'm missing? Cheers.

An Ubuntu kernel bug causes container crashes

developer122 — Tue, 05 Jul 2022 23:41:06 +0000

Kind of a meta observation: Nobody seems to have really solved the theoretical problem of ensuring that a patch "actually" applies cleanly, ie. that what it touches hasn't changed in some way. May well be impossible.

Best case scenario your tooling is tracing subsystem interactions and noting dependencies in case there's any change at all, worst case it's trying to parse exactly what some code is doing/trying to do which is halting-problem territory.

Red Hat stable kernels

pbonzini — Tue, 05 Jul 2022 22:21:52 +0000

Depends on the directory. Large parts of the RHEL9 kernel are probably closer to 5.16 even.

Red Hat stable kernels

jake — Tue, 05 Jul 2022 21:33:32 +0000

> RHEL-8 is 4.18 based and RHEL-9 is 5.14 based, neither of which are upstream LTS.

ah, yes, thanks for the correction ... i have adjusted the article text accordingly ...

jake

Red Hat stable kernels

willy — Tue, 05 Jul 2022 21:32:32 +0000

If you diff the RHEL-9 kernel against 5.14 and 5.15, which diff is smaller?

An Ubuntu kernel bug causes container crashes

snajpa — Tue, 05 Jul 2022 21:11:29 +0000

uh, now I feel sorry for ever doubting you guys, I thought that initiative of shipping complete fscks at times was driven by the actual dev culture at Canonical, not the management... hasn't even occurred to me you guys might have been pressured to ship that sh*t, I just thought the management doesn't care...

so, please accept my apologies, I'm gonna crawl into that corner over there and think long and hard about this :D

An Ubuntu kernel bug causes container crashes

Paf — Tue, 05 Jul 2022 21:07:50 +0000

This seems a decently understandable oversight, what I’m a little surprised by is that it wasn’t caught by some form of CI…

Red Hat stable kernels

ribbo — Tue, 05 Jul 2022 21:02:21 +0000

Red Hat doesn't choose LTS kernels for their LTS distro (RHEL) releases, as stated in the article, RHEL-8 is 4.18 based and RHEL-9 is 5.14 based, neither of which are upstream LTS.

An Ubuntu kernel bug causes container crashes

brauner — Tue, 05 Jul 2022 20:50:39 +0000

It should be noted that when I was still working there we were forced to update and carry this fscking patchset. I hate it with a passion. It's nickname has always been sh*tfs and we refused any upstreaming. Thankfully we were able to solve this upstream and not as a separate fs.