LWN: Comments on "Meltdown/Spectre mitigation for 4.15 and beyond"

Meltdown/Spectre mitigation for 4.15 and beyond

anton — Fri, 19 Jan 2018 12:43:40 +0000

Unfortunately, the Intel documentation is quite abstract. It does not tell us what these things actually do (probably because that's different for different generations); instead it tries to specify how to use them and/or what guarantees these things give (but even that is not very clear).

Meltdown/Spectre mitigation for 4.15 and beyond

roc — Wed, 17 Jan 2018 21:28:31 +0000

I don't think those are equivalent at all, since they require manual changes and the Microsoft approach is automatic (with pros and cons accordingly, as I wrote here: http://robert.ocallahan.org/2018/01/long-term-consequence...).

Meltdown/Spectre mitigation for 4.15 and beyond

foom — Wed, 17 Jan 2018 16:43:15 +0000

The currently-proposed equivalent to this on the Linux side is the GCC and LLVM patches adding a new "__builtin_speculation_safe_load" intrinsic (e.g. https://reviews.llvm.org/D41760, https://reviews.llvm.org/D41761, https://gcc.gnu.org/ml/gcc-patches/2018-01/msg01546.html).

Meltdown/Spectre mitigation for 4.15 and beyond

cesarb — Wed, 17 Jan 2018 11:13:39 +0000

That might mean Microsoft believes that the microcode and kernel changes are enough to protect against variant 2 (they probably consider systems where the microcode won't be upgraded a lost cause).

If Microsoft isn't going use retpolines in user space, won't that lead to a performance disadvantage for Linux distributions which decide to recompile everything with retpolines? And what about that variant 1 mitigation they mentioned, will gcc and llvm implement something like it?

Meltdown/Spectre mitigation for 4.15 and beyond

dwmw2 — Wed, 17 Jan 2018 11:08:06 +0000

Intel documentation is here. I haven't seen public AMD documentation yet (they have IBPB but not IBRS).

No, it isn't just disabling branch prediction completely. I think that what they could achieve in the microcode hacks was fairly limited. So in some ways setting IBRS is a partial barrier, and flushes certain predictions from the store. But leaving IBRS set also makes things go slow, which implies that it's doing some checking at all times. The details are opaque and will vary from generation to generation.

Thankfully we don't really need IBRS except on Skylake (where it doesn't suck quite so much anyway).

Meltdown/Spectre mitigation for 4.15 and beyond

roc — Wed, 17 Jan 2018 08:59:58 +0000

For mitigations of variant 2, the post links to another post: https://cloudblogs.microsoft.com/microsoftsecure/2018/01/... which is all about system updates, from which we can conclude that user-space code changes for variant 2 are not forthcoming for MSVC users. At least for now.

CET

dwmw2 — Wed, 17 Jan 2018 07:33:25 +0000

Userspace? What's that? Er... yes, I confess I hadn't got much past getting the kernel to do the right thing.

I suspect we'll find JITs and handwritten assembly that are going to need fixing for CET anyway. But this one is *conditional*. Maybe a flag in the auxvec to say we have CET or IBRS_ALL.

Meltdown/Spectre mitigation for 4.15 and beyond

mjthayer — Wed, 17 Jan 2018 07:12:20 +0000

Though it would probably break PCIDs on systems which have them. And on second thought, I don't see much benefit over the duplicate page tables.

Meltdown/Spectre mitigation for 4.15 and beyond

foom — Wed, 17 Jan 2018 05:36:58 +0000

Given the discussion about retpoline being somewhat ineffective on Skylake, it seems like it may not actually even be sufficient to mass-rebuild the entire userspace with retpoline?

Reading https://software.intel.com/sites/default/files/managed/c5... it looks like there's not really a "turn off the indirect predictor in all of userspace" option, either. (Although I could imagine IBRS might have that effect on some CPUs.)

And even though IBRS ("Indirect Branch Restricted Speculation") is initially described like a mode switch, the intel doc also says it actually needs to be used as a command at every privilege transition for some of the CPUs (those without the "Enhanced IBRS" feature), and even then it doesn't protect code running at a given privilege level from other code on the same core at the same privilege level but in a different process. So you gotta also use IBPB.

And I have to imagine that IBRS and IBPB must be pretty slow given all the work being put into retpoline.

But if we need retpoline on old processors, and can't depend on retpoline on new processors...does userspace code running on skylake need to take the hit of IBRS-always, and IBPB, *and* retpoline (because presumably there won't be multiple variants of all the distros, built with and without retpoline)? How slow is that gonna be?

(Too bad user code can't use the kernel boot-time alternatives patching trick to eliminate the retpoline manipulation where it's not useful. Well, I guess it could via the vsyscall stuff, but you'd need to do an indirect jump through PLT to get there...which kinda ruins the point.)

Meltdown/Spectre mitigation for 4.15 and beyond

paulj — Wed, 17 Jan 2018 05:29:13 +0000

So what does IBRS do exactly (as was also the question in the linux-kernel thread)?

Is it /disabling/ IBP CPU logic? In which case, Andrea Arcangeli's belief that setting it once is sufficient surely must be true?

Or have Intel, with the microcode update, managed to add some bits of context (privilege level, address space?) to the branch-prediction table, and setting this IBRS pseudo-MSR is needed to get the CPU to update its view of the context in some way, so that IBRS must be set on every security relevant context change? Which would be more in-line with David and Arjan's views in that thread?

The lack of documentation and explanation is less than ideal. The security issues are now public. It doesn't make sense to try 'manage' what information is made public about any mitigation features - it can only hamper the speed at which any flaws/issues with those mitigations are uncovered.

Meltdown/Spectre mitigation for 4.15 and beyond

corbet — Wed, 17 Jan 2018 01:55:56 +0000

That's not too surprising, since retpolines are a defense against variant 2...Am I missing something here?

Meltdown/Spectre mitigation for 4.15 and beyond

roc — Wed, 17 Jan 2018 01:23:10 +0000

Microsoft has released a post about MSVC mitigation for Spectre variant 1: https://blogs.msdn.microsoft.com/vcblog/2018/01/15/spectr...
Apparently they are not going to recommend or support retpolines, in user space at least. I wonder why they've diverged from the Linux community here.

CET

roc — Tue, 16 Jan 2018 22:47:31 +0000

Thanks! That explanation is very helpful. This has some interesting implications.

This implies that a kernel must only enable CET in userspace if it also enables IBRS_ALL successfully.

ld.so, or something else, must reliably detect IBRS_ALL or CET and use the correct thunk.

More worrying, all JITs and handwritten assembly must be modified to detect IBRS_ALL or CET and dynamically switch retpolines on/off. Hardcoded retpolines will not work with CET.

How is userspace going to detect whether to use retpolines? Are there going to be syscalls to detect IBRS_ALL and/or CET? Or some other technique?

Meltdown/Spectre mitigation for 4.15 and beyond

mjthayer — Tue, 16 Jan 2018 20:23:01 +0000

Regarding the fix for Meltdown, I was wondering whether emptying the TLB on context switches and rewriting the page tables in memory so that the kernel parts were not accessible to user processes would also have worked. Seems potentially slightly less invasive than the duplicate page table thing, assuming the page table structure makes it doable.

Meltdown/Spectre mitigation for 4.15 and beyond

mjthayer — Tue, 16 Jan 2018 20:18:20 +0000

And yes, I know that is roughly what IBPB does where it is available.

Meltdown/Spectre mitigation for 4.15 and beyond

mjthayer — Tue, 16 Jan 2018 20:14:27 +0000

> Ah, I see. So the only way to prevent that by changing the address space would be to put each user process at its own set of virtual addresses that do not overlap at all with virtual addresses used by other processes.

Or could one empty the branch predictor between process switches, possibly using a similar training mechanism?

Meltdown/Spectre mitigation for 4.15 and beyond

excors — Tue, 16 Jan 2018 15:44:29 +0000

Haswell reportedly only uses the lower ~31 bits of addresses for branch prediction, so you'd have to make sure processes' address spaces don't overlap modulo 2^31, which is impractical. (And other CPUs could use even fewer bits.)

Meltdown/Spectre mitigation for 4.15 and beyond

matthias — Tue, 16 Jan 2018 15:26:37 +0000

Would it be enough to put the text segments into non overlapping regions? Or is it possible to train the branch predictor to jump to virtual addresses that are marked non executable in the attackers context? After all such jumps can never really succeed.

If the attacker can only train the branch prediction to jump to addresses that are not mapped by the victim, there should be no information leak.

Meltdown/Spectre mitigation for 4.15 and beyond

epa — Tue, 16 Jan 2018 14:59:02 +0000

Ah, I see. So the only way to prevent that by changing the address space would be to put each user process at its own set of virtual addresses that do not overlap at all with virtual addresses used by other processes. Even on a 64-bit system this would limit the number of user processes.

Meltdown/Spectre mitigation for 4.15 and beyond

excors — Tue, 16 Jan 2018 14:52:14 +0000

As I understand it, the Spectre paper's "Example Implementation on Windows" has the victim process and attacker process *not* sharing an address space. The attacker process puts its own code at virtual addresses that coincide with interesting code in the victim, and uses that to train the branch predictor. The branch predictor only cares about virtual addresses (and maybe not even all their bits), and doesn't care about address spaces or physical addresses. The CPU switches to the victim's address space when running the victim process, but it keeps the attacker's branch predictor state, so the attacker can control what code is run (speculatively) by the victim.

Then the attacker chooses to make the victim run code that transmits the victim's memory through some covert channel - mainly timing of reads to cache lines that are shared between the two processes, but there are other variations of covert channels that don't rely on shared cache lines at all (e.g. a read will evict unrelated data at different addresses that happen to map onto the same cache set).

Meltdown/Spectre mitigation for 4.15 and beyond

epa — Tue, 16 Jan 2018 13:31:50 +0000

I was thinking of the "Attacks using native code" section of the Spectre paper. As I understand it, that variant of Spectre relies on the victim process being in the same address space as the attacker. You can speculatively execute arbitrary instruction sequences from the victim process and so read its memory space. This attack is prevented if different userspace programs don't share an address space.

There are other Spectre variants but I didn't think they were quite as powerful; they could not usually "dumping memory belonging to other user's processes" as the first post in this thread talks about.

As User I would like to have a cgroup wide way to disable KPTI

giggls — Tue, 16 Jan 2018 12:44:06 +0000

This way one could e.g. disable it for a psql Server using its systemd service file.

Thunk names

dwmw2 — Tue, 16 Jan 2018 12:28:01 +0000

"The GCC developers did force a name change for the retpoline thunk itself, though, breaking the existing kernel patches and making the compiler (when released) incompatible with the version that distributors have been using to create fixed kernels thus far. If that change sticks, it will require more 4.15 patches in the immediate future."

Yeah, that was a fun game on Sunday night. We did manage to avoid that though, in the end: https://gcc.gnu.org/ml/gcc-patches/2018-01/msg01300.html

Meltdown/Spectre mitigation for 4.15 and beyond

Sesse — Tue, 16 Jan 2018 11:48:41 +0000

Ah, to answer my own question, it's in the link: cmp rax, rbx ; sbb rax, rax. CMP sets the carry flag if (rax - rbx) < 0 (CMP and SUB are generally highly related instructions on x86), and SBB subtracts the carry flag.

Skylake

dwmw2 — Tue, 16 Jan 2018 11:44:19 +0000

"The remaining concern on Skylake processors would appear to be system-management interrupts (SMIs), which can cause unprotected code to be run in kernel context. There does not appear to be a consensus that SMIs are exploitable in the real world, though, and no known proofs of the concept."

The SMI thing is just one of a litany of conditions which on Skylake+ may cause the RSB to underflow, causing dangerous predictions to be taken from the branch predictor instead. There are many of those conditions which, individually, may have caused my to throw my toys out of the pram and say "Skylake+ gets IBRS". Including the one about deep call stacks of ≥16 in depth. How long did we spend playing whack-a-mole trying to make 4KiB stacks work? And that was when we got a nice clean crash with a stack overflow, not a silent vulnerability...

Meltdown/Spectre mitigation for 4.15 and beyond

Sesse — Tue, 16 Jan 2018 11:41:02 +0000

What are those two x86 instructions? I can't figure it out offhand, and neither can GCC or Clang.

CET

dwmw2 — Tue, 16 Jan 2018 11:36:43 +0000

Any processor with CET will always have the new IBRS_ALL feature too, where you just set the IBRS bit once and forget about it, and it's a "do the right thing" option that is faster than any of the options right now, even retpoline.

(I don't really understand why Intel want the MSR write, in fact, and why they don't just advertise it with a single CPUID bit and leave it at that.)

So, given that with IBRS_ALL we'll ALTERNATIVE away the retpoline thunks to a simple 'jmp *%reg', that means that CET will always be OK.

FWIW this is precisely why we changed the retpoline thunks from being ret-equivalent (with the target address on the stack), to taking their argument in a register. When it was on the stack, there was no way to make it CET-compatible.

Meltdown/Spectre mitigation for 4.15 and beyond

marcH — Tue, 16 Jan 2018 09:02:34 +0000

Good thing you weren't aware of this issue then and didn't leak it with random texts.

CET

roc — Tue, 16 Jan 2018 03:42:42 +0000

I read that on LKML, but I wish someone would explain it in detail, since it seems like retpolines are pretty much the exact thing CET should be protecting against.

Meltdown/Spectre mitigation for 4.15 and beyond

jcm — Tue, 16 Jan 2018 02:41:43 +0000

That's funny. Whenever I've got a problem to discuss on arm servers, I just text the architects involved at the companies building them because we're one big happy family. There's no need to go to a dungeon or through layers of corporate abstraction.

Meltdown/Spectre mitigation for 4.15 and beyond

roc — Tue, 16 Jan 2018 02:20:22 +0000

Sorry! Amidst all the confusion, overall you're doing a great job. Thanks!

Meltdown/Spectre mitigation for 4.15 and beyond

corbet — Tue, 16 Jan 2018 00:18:15 +0000

Yeah, OK, that probably wasn't the best thing I ever wrote. In my poor defense I'll say that I was awfully tired by the time I got to the end of all that stuff...

Meltdown/Spectre mitigation for 4.15 and beyond

andresfreund — Mon, 15 Jan 2018 23:47:16 +0000

That's the fix for meltdown, not spectre, which is the danger talked about in the subthread.

Meltdown/Spectre mitigation for 4.15 and beyond

pbonzini — Mon, 15 Jan 2018 23:39:56 +0000

There is work on using static analysis to place the new macro. But yes, at this point it's a whack-a-mole game.

Meltdown/Spectre mitigation for 4.15 and beyond

roc — Mon, 15 Jan 2018 22:55:24 +0000

> After all of this work, it would appear that the 4.15 kernel will be released with fairly complete Meltdown and Spectre protection

Isn't this an overstatement considering that thoroughly protecting the kernel against Spectre variant 1 requires using the new array-index macro everywhere it's needed, and no-one actually knows yet how to determine where it's needed?

Another source of confusion is that some people will interpret the above statement to mean that userspace is protected when running on the right kernel, when that is definitely not the case. This is similar to how the cloud providers quickly announced "we've fixed everything in our cloud!" when in fact they only fixed specific hypervisor-related issues and customers still have a ton of work to do.

Meltdown/Spectre mitigation for 4.15 and beyond

smoogen — Mon, 15 Jan 2018 21:22:30 +0000

The website for that would be https://hardenedbsd.org/ . I was not aware of this version of BSD until this post so good luck and good going.

Meltdown/Spectre mitigation for 4.15 and beyond

sthibaul — Mon, 15 Jan 2018 21:21:14 +0000

> speculation is almost entirely limited to control decisions, not data values.

Well, there *are* some people working on value speculation, with very interesting speedups, e.g. http://ieeexplore.ieee.org/abstract/document/6835952/ (pdf available on https://hal-univ-rennes1.archives-ouvertes.fr/docs/00/90/... ) but I do hope that the speculated read would be coherent with the speculated value, and thus the mask approach remains correct.

Meltdown/Spectre mitigation for 4.15 and beyond

epa — Mon, 15 Jan 2018 21:20:49 +0000

Can't it be fixed in the same way the attack on the kernel was fixed: by unmapping the memory? So each user space process would get its own memory mappings.

Meltdown/Spectre mitigation for 4.15 and beyond

shawn.webb — Mon, 15 Jan 2018 21:02:45 +0000

> One other issue with retpolines remains somewhat unresolved, though: using them requires support from the compiler, and almost nobody has a compiler with that support available.

FreeBSD's decision from years back to migrate to the llvm toolchain for its compiler (and eventually linker on amd64) gives it this support. In HardenedBSD, we've already switched amd64 to use ld.lld as the default linker. We're testing a full OS (world + kernel + packages) with retpoline right now. We'll likely debut retpoline in HardenedBSD 12-CURRENT/amd64 late this week or early next.

Meltdown/Spectre mitigation for 4.15 and beyond

Otus — Mon, 15 Jan 2018 20:49:10 +0000

Ah, so prediction is not disabled, just prevented from interacting between processes?