LWN: Comments on "Shedding old architectures and compilers in the kernel"

It also depends on your security model

brooksmoses — Thu, 01 Nov 2018 05:23:20 +0000

Also, of course, there's the "which compiler?" question. Once you start getting into esoteric "extra" warnings, it's pretty common to end up with situations where avoiding a warning in one compiler will cause a warning in another.

Old compilers, new kernels

dps — Sun, 03 Jun 2018 14:11:07 +0000

You are clearly out of touch with far too many production systems. The idea of upgrading from long unsupported old releases of things, like python 2.3, very old sshd, etc with known unfixed security issues is anathema. I would advocate using current stable versions but that was not my pigeon and it was feared hat some bugs might be revealed. Actually fixing those bugs would obviously be the wrong thing.

I have extensive experience of not using things which are clearly the right thing because the production environment is sufficiently ancient that it is not supported. Let it suffice to say that this experience was obtained at multiple very large software companies.

I would not worry about the lack of spectre mitigation on those boxes because there are so many easier ways of owning them.

Shedding old architectures and compilers in the kernel

flussence — Sat, 28 Apr 2018 19:34:07 +0000

We have a unified arch/x86/ for i386 and amd64 nowadays, which wasn't always the case (see commit 208652d6b2c).

Shedding old architectures and compilers in the kernel

willy — Fri, 27 Apr 2018 21:08:21 +0000

Do arm and arm64 have enough in common that it's worth unifying them? It's not like we ever tried to unify i386 and ia64

It also depends on your hardware

excors — Thu, 08 Mar 2018 21:09:54 +0000

I believe nommu systems still need Spectre mitigation, as long as they do speculative execution. Something like a JavaScript interpreter that is designed to run untrusted code in a sandbox ought to be safe on a nommu system, but Spectre still makes it vulnerable exactly like with-mmu.

It also depends on your hardware

landley — Thu, 08 Mar 2018 20:20:52 +0000

Linux on nommu still exists. If a system without spectre/meltdown mitigation can't possibly be viable to run, you're expressing a pretty strong opinion about those too. (There are more nommu processors than with-mmu processors on the planet about the same way there are more bacterial cells than mammalian cells. This is the dark matter of the hardware world.)

Half the embedded developers already tend to stay several versions behind on everything (kernel, compiler, libc, userspace), as in it took them several years to notice upstream uClibc was dead and move to musl-libc or the uclibc necromancy fork. That's why I had a http://landley.net/toybox/faq.html#support_horizon in busybox (and now toybox): clear policy with leeway for old systems but also a defined "ok, that's enough, you can move now" cutoff.

If you remove support for all but the newest toolchains, they'll just stop upgrading the kernel in their sdk, as in they'll develop _new_ devices against old stuff. (It's already enough of a pain to convince embedded developers to stop developing new devices based on a 2.6 kernel they have lying around. Make your policies too tight, you lose traction.)

Rob

It also depends on your security model

excors — Thu, 08 Mar 2018 15:00:25 +0000

> Not often, but some things are clearly best practice. Like what I documented :-) at a previous employer ...
>
> "Switch compiler warnings to max, and if you can't fix them you need to explain them".

But that isn't clearly best. In some compilers, the maximum 'warning' level (e.g. Clang's -Weverything) includes various diagnostic messages that are merely pointing out something the compiler noticed about your code, without implying there's really any problem with it. Perhaps those messages are useful in niche situations, or they were requested by one customer with a bizarre coding style, or a bored compiler developer added them because they could. It's not worth your time to modify your code to avoid those warnings, and it's probably not even worth your time going through the list of the hundreds of available messages and selectively disabling them. The compiler developers have already produced a list of reasonably useful warnings and called it -Wall or something, so that's usually a better place to start.

Maybe "set everything to max" is good advice when -Wall *is* the max. But if you just state that advice without any rationale, it will become poor advice when people switch to a compiler with many less-useful warnings.

It also depends on your security model

Wol — Thu, 08 Mar 2018 14:17:35 +0000

> I agree with the rest of your comment but...

> > If you find yourself promoting terms like "best practice" (you didn't but you came close) then you are doomed, I tell you, doomed.

Not often, but some things are clearly best practice. Like what I documented :-) at a previous employer ...

"Switch compiler warnings to max, and if you can't fix them you need to explain them".

:-) :-)

Cheers,
Wol

Shedding old architectures and compilers in the kernel

BenHutchings — Wed, 07 Mar 2018 16:06:26 +0000

Debian 7 "wheezy" (released 2013) has gcc 4.7. But we will probably add gcc 4.9 shortly, specifically so we can build the kernel with retpolines.

However, RHEL 5 (2007) is also still supported and apparently still has gcc 4.1 as the default compiler (with 4.4 available as an alternative).

Old compilers, new kernels

mathstuf — Mon, 05 Mar 2018 14:44:36 +0000

I see this all the time. Not with kernels, but folks want to compile 10+ million SLOC projects changed yesterday using Ubuntu 12.04, so they require everyone support CMake 2.8.9 or whatever. Nevermind that just adding CMake to the pile of code they want to build would save everyone a lot of headaches.

Old compilers, new kernels

fratti — Mon, 05 Mar 2018 06:37:28 +0000

Are there any users who would conceivably build brand-spanking new kernels with decade-old compilers? Aside from appeasing archaeologists 1000 years from now trying to piece together what lead to humanity's downfall and running into issues dating fragments of our past, I can't really think of a user that would use an old compiler without also using an old kernel. Perhaps I'm not familiar enough with the embedded vendor landscape to know the restrictions they have to live by though.

Shedding old architectures and compilers in the kernel

nix — Sun, 04 Mar 2018 14:06:35 +0000

My premise is from, uh, kernel developers working for a major Linux distributor (unnamed to save their blushes). Maybe the problem is that nobody ever *rereads* Documentation/Changes after they read it once, since the assumption is that after your latest pull the deps are unchanged. (I am guilty here too, of course. So is everyone who builds a lot of kernels.)

But if kernel devs don't read the things, I see no reason to assume that anyone else will.

It also depends on your security model

nix — Sun, 04 Mar 2018 14:04:53 +0000

Yeah, the old guarantees that language purity could enforce security appear pretty much dead now. The implementation always needs to do extra work to keep people from reaching out beyond the bounds of the implementation via side channels like this -- and we don't know what new side channels may exist in the future, or may exist now yet be unknown to us :(

Shedding old architectures and compilers in the kernel

moltonel — Sat, 03 Mar 2018 22:14:00 +0000

The logical conclusion to "nobody ever reads the doc" is that the docs should be deleted, to reduce the maintenance burden. Maybe your premise is true for users. I guess it is true for me, a desktop user getting his build tools and kernel sources from a distro.

But it's clearly not true for (some) kernel devs, since they are discussing raising the minimum version (which is only defined in the docs). Since the goal is to get rid of old workarounds and to be sure that certain features are available, wouldn't it make sense for devs to document what those features and workarounds are ? So that devs know what can be removed from the kernel after raising requirements to this or that ? Wouldn't a short "version X brings you Y" table speed things up for the next reqs update discussion, compared to a single "version X required" statement ?

It also depends on your security model

excors — Sat, 03 Mar 2018 20:56:45 +0000

Unless you're running e.g. untrusted scripts in a sandboxed interpreter. Even if everything is running as root, you expect the sandbox to prevent the scripts accessing arbitrary memory. With Spectre you lose that guarantee. ("Script" could mean JS or eBPF or pretty much any kind of data that can cause the program to do a bounds-checked array access, which might include data that doesn't really look anything like a script at all.)

Shedding old architectures and compilers in the kernel

nix — Sat, 03 Mar 2018 19:19:31 +0000

That's... not really useful, because nobody ever reads the documentation. The kernel's build dependencies have been documented in the same place for decades (they moved from Documentation/Changes to Documentation/process/changes.rst recently, but left a symlink), yet when I've added build dependencies there I see many questions from people who encounter build failures because of the lack of those dependencies but never consider even looking to see if the kernel's build-deps are documented anywhere, let alone looking to see if they changed.

And that's something as obvious as a build failure when turning on a new feature: i.e. doing something active, intentionally. Nobody will think to look in documentation if a *lack* of upgraded compiler causes a failure (or, worse, a silent change in semantics). Nobody.

It also depends on your security model

nix — Sat, 03 Mar 2018 19:13:34 +0000

What about calling it 'best current practice' like the RFCs do, then? That doesn't carry the ridiculous implication that this practice is best for all time, the best ever, rah rah!

It also depends on your security model

nix — Sat, 03 Mar 2018 19:12:01 +0000

Quite. If everything important is running as one uid (as is true of most single-user setups!) both Spectre and Meltdown are also unimportant, because the attack doesn't need to use either when it could just ptrace() far more cheaply.

It also depends on your hardware

anton — Sat, 03 Mar 2018 13:53:33 +0000

keeping support for the old compilers might still be useful to some people even for x86-64.

In addition to those where Meltdown/Spectre are irrelevant because there is no attack scenario where they worsen the situation, there are also x86-64 CPUs that are not vulnerable to Meltdown/Spectre (in particular, those based on Bonnell), and hopefully we will see many more CPUs without these vulnerabilities in a few years.

It also depends on your security model

cladisch — Sat, 03 Mar 2018 06:21:23 +0000

User Rosinante on the Meta Stack Exchange expressed it best:

While you may be an exemplary, clear-thinking individual, who uses the term best practice in a constructive manner, you have been preceded by a giant procession of zombies who use it as the antithesis of thought. Instead of understanding the important specifics of their situation and looking for an appropriate solution, all they want is to spot a herd in the distance and go trotting off after it.

Thus, the term best practice has been rendered an extremely strong signal of an empty resonant cavity in the place where a brain should be, and questions that mention the phrase get closed.

It also depends on your security model

himi — Sat, 03 Mar 2018 02:47:41 +0000

I think the point was that if all your code is running as root, /any/ security vulnerability will open up access to any and all memory on the system - meltdown and spectre don't add any additional problems.

It also depends on your security model

gerdesj — Sat, 03 Mar 2018 00:13:19 +0000

"What's wrong with "best practice"?"

There be bad practice and there be good practice. As soon as I see anyone claiming that they know what is "best practice" I briefly lose the will to live. You said it yourself in a way: "The way I see it, a best practice documents general conventions that are a good idea to follow" "good idea to follow" - here you dilute "best" to "good idea".

I am being very pedantic with this point, I know. I doubt that many people here have ever written documentation that claimed to be *shudder* "a best practice". There are a lot of opinionated folk here (me included) and quite a lot of "I'm right and you are talking twaddle" style bike shedding err ... discussion, but rarely, if ever, do I see an invocation of "best practice". I do often see that phrase elsewhere rather a lot and I generally use it as a litmus test for possible but unlikely guide to solution of the problem at hand, which is a bit sad.

I suppose that in the end I'm a bit of a grammar fiend: good, better, best - adjective, comparative, superlative. Soz!

It also depends on your security model

abatters — Fri, 02 Mar 2018 17:13:34 +0000

In a discussion of dropping support for old compilers simply because they lack retpoline support, the point, of course, was that since retpoline support may not be essential to everyone, keeping support for the old compilers might still be useful to some people even for x86-64. Upgrading development environments is a PITA.

I have found the responses to my comment very insightful. I am glad I posted my comment, if only to get other people to share their perspectives. Let me summarize:

1) If a system does not allow 3rd-party code to be installed or run, and is not network-attached, then Meltdown/Spectre might not matter.

2) If a system does not allow 3rd-party code to be installed or run, and is network-attached, then Spectre might be remotely exploitable by network packet timings, although this has yet to be demonstrated and is only theoretical.

3) If a system does not allow 3rd-party code to be installed or run, and is network-attached, and an attacker manages remote code injection into a network process, then Meltdown/Spectre can be used for further compromise. In some cases, compromising an important network process is already "game over" so Meltdown/Spectre would not be of any use, but for other cases, Meltdown/Spectre could be used to gain complete control after compromising a low-privilege network process.

It also depends on your security model

farnz — Fri, 02 Mar 2018 13:34:43 +0000

The problem with "best practice" is that it comes without a rationale, and thus documents things that you "must" do, but not why you must do them. Thus, when advice in a best practices document becomes wrong (e.g. due to external changes), there's no way to even notice that it is wrong, let alone fix it.

It also depends on your security model

intgr — Fri, 02 Mar 2018 13:01:27 +0000

I agree with the rest of your comment but...

> If you find yourself promoting terms like "best practice" (you didn't but you came close) then you are doomed, I tell you, doomed.

What's wrong with "best practice"? The way I see it, a best practice documents general conventions that are a good idea to follow "stupidly" as you describe. Applying all security updates periodically is one such best practice, even if you think they don't apply to your use case. As opposed to cherry-picking patches that you think apply -- because you're bound to miss some.

It also depends on your security model

gerdesj — Fri, 02 Mar 2018 01:34:39 +0000

"My company deploys Linux in embedded x86-64 appliances that never run any untrusted 3rd-party code, so the whole spectre/meltdown thing is irrelevant to those systems even though the underlying arch is vulnerable. So for our use case we don't care about retpolines even on x86-64."

What exactly are you contributing to the discussion here? At best you are correct about all of your assertions and at worst some clever bugger will prove you very, very wrong on one or two of them.

Personally I always attack problems assuming I'm stupid. If you find yourself promoting terms like "best practice" (you didn't but you came close) then you are doomed, I tell you, doomed. You may be a clever person but there is always someone else that is better than you and our colleagues here will include some of them - they may point out the flaw in your reasoning. The baddies won't.

That's the way of the sys 'n' network admin as I see it. No tin foil, just simple scepticism. Perhaps you are the dog's bollocks when it comes to IT security but I note that your firm were not involved in this disclosure (or were you?), so were just as surprised as the rest of us (sort of, see papers from ~1993 onwards.)

Me? I'm stupid, always have been and always will be.

It also depends on your security model

matthias — Thu, 01 Mar 2018 12:49:29 +0000

You should keep in mind that at least in theory, Spectre can be exploited remotely: Send crafted network packets to train the speculation engine and afterwards extract information based on timing of future packets.

I do not expect this to be a problem right now, as it will be even harder than a local exploit and the bandwidth of the hidden channel will be much lower than for a local attacker. The SNR will be much worse, due to increased noise. But it is not as easy as saying "If $networkprocess is broken, then we have lost anyway". Spectre works even without breaking any process.

Shedding old architectures and compilers in the kernel

xtifr — Thu, 01 Mar 2018 04:16:25 +0000

Sounds like a great idea. How soon can you have a first draft for us to check out?

It also depends on your security model

rahvin — Thu, 01 Mar 2018 01:28:23 +0000

I disagree, the capability of Meltdown/Spectre to read out the entire memory stack makes them far and away some of the most dangerous bugs. We need to keep in mind that only the very first exploits have rolled on these and because Spectre is fundamental to the design of almost all modern processors it's going to be the bug that continues to give for a very long time.

IMO It's simply a matter of time with Spectre, and the timing attacks that it's based on, before we get the remote exploit that makes it the single biggest threat to computing. Hopefully I'm wrong.

It also depends on your security model

clump — Wed, 28 Feb 2018 18:54:00 +0000

We can only speculate(!) about abatters use case. You need to think about risks you can't forsee. If you don't use a network, can a network be created? Did you still include a network stack in your embedded OS? Your use case might not include peripherals but can peripherals be attached? Are you carrying software that's not used in your product?

You can't protect against everything but you can take precautions. Some performance impacting mitigations on later kernels can be disabled, but having them opt-out by default is reasonable.

Shedding old architectures and compilers in the kernel

geert — Tue, 27 Feb 2018 20:54:01 +0000

Can we get rid of one more arch/ directory by (finally) merging arm64 and arm? Or is it still too early?

Shedding old architectures and compilers in the kernel

arnd — Tue, 27 Feb 2018 20:35:03 +0000

Thanks, that's what I had been looking for. I should have a look there if I want to rebuild the cross toolchains for kernel.org. Not sure if we want to keep both version (with and without retpoline support), or if I should just replace the binaries I uploaded.

If I understand it right, the changes in that git tree are only relevant for x86; other architecture might need similar work but nothing else has it, right?

Shedding old architectures and compilers in the kernel

willy — Tue, 27 Feb 2018 19:12:14 +0000

Because we want to allow someone running a currently supported RH release to be able to build a current kernel.

Shedding old architectures and compilers in the kernel

moltonel — Tue, 27 Feb 2018 19:11:00 +0000

Yes, I know that's the whole point of droping support.

But my comment was about documenting the more nuanced support: the LKML thread shows that documenting a single minimum version hides a lot of useful info ("gcc foo is supported but you'll miss out on bar"). Detailed support docs should help both users of old tools and support-droping efforts by developers.

It also depends on your security model

pbonzini — Tue, 27 Feb 2018 18:40:03 +0000

There are many cases in which breaking $networkprocess is just as bad as getting root. (There are also many cases in which everything that you worry about is running as root, though these are less likely to run on a core with speculative execution). In both cases, Meltdown and Spectre are not really much to worry about.

It also depends on your security model

rahvin — Tue, 27 Feb 2018 17:56:57 +0000

If the appliance accepts a network connection any bugs in your software could make spectre/meltdown a real risk for a determined hacker, particularly if they are exposed to the broader internet. All you need is a buffer overflow or other bug that would allow a network service to insert a command that is then executed. Spectre and Meltdown are the bugs that are going to continue to give for a very long time.

Shedding old architectures and compilers in the kernel

kiko — Tue, 27 Feb 2018 16:22:19 +0000

For Ubuntu, I think we're fine.

While 12.04 and 14.04 LTS are still formally supported, they won't receive modern kernels via HWE; in reality only 4 kernel versions are ever available to be installable on any given Ubuntu release. See the "Kernel release end of life" for details at https://www.ubuntu.com/info/release-end-of-life -- but the summary is that neither of them would be affected by a GCC requirement uplift.

For 16.04 LTS, similarly, the last HWE kernel received will be 4.15, which is the kernel release targeted for the next LTS, 18.04 LTS, due in April. So it too would be unaffected by an uplift in GCC version.

Bionic itself comes with GCC 7.3 (and 6.4) which should be acceptable for any kernel version it ends up receiving.

Shedding old architectures and compilers in the kernel

mageta — Tue, 27 Feb 2018 15:47:09 +0000

AFAIK:

Redhat does. Suse does upgrades with service-packs. Canonical sticks to the original kernel, but for a shorter time and provides no hardware-enablement at all via that old version, but with new kernels (hwe kernels) instead. So this could hit the two later.

Shedding old architectures and compilers in the kernel

doublez13 — Tue, 27 Feb 2018 15:21:59 +0000

Because by completely dropping support for older compilers, we're able to get rid of chucks of code needed only to support them.

It also depends on your security model

abatters — Tue, 27 Feb 2018 14:41:59 +0000

My company deploys Linux in embedded x86-64 appliances that never run any untrusted 3rd-party code, so the whole spectre/meltdown thing is irrelevant to those systems even though the underlying arch is vulnerable. So for our use case we don't care about retpolines even on x86-64.