LWN: Comments on "Meltdown strikes back: the L1 terminal fault vulnerability"

Meltdown strikes back: the L1 terminal fault vulnerability

alejluther — Wed, 24 Oct 2018 06:48:58 +0000

Yes, it does exist. You are thinking in a VM with a server role serving client requests, so clients specifically access the system, so vulneravilities can be exploited. But VMs could have other services not directly connected or accessible to clients. For example, telcos have VMs working with packets where those packets do not have the VM as the endpoint.

Non-present, not in swap

corbet — Tue, 28 Aug 2018 21:10:37 +0000

File-backed pages are the most common example of pages that can be non-present but not in swap.

Meltdown strikes back: the L1 terminal fault vulnerability

loch — Tue, 28 Aug 2018 20:16:29 +0000

"...for pages that have been swapped out, the location in the swap area is stored in the PTE. In other cases, the data left in non-present PTEs is essentially random."

I'm a bit confused, what are the other cases? In what situation would a page be non-present, but also not in swap?

Meltdown strikes back: the L1 terminal fault vulnerability

ssmith32 — Thu, 23 Aug 2018 22:56:57 +0000

You are correct. Reading the article makes that clear. The default in server 2016 is the classic scheduler, which does not bind LP to VP 1:1. The core scheduler was introduced as an option in Server 2016, both for security & performance SLA reasons.

Meltdown strikes back: the L1 terminal fault vulnerability

Wol — Thu, 23 Aug 2018 21:08:00 +0000

Just don't fall foul of the wi-fi bug, where the secret was flushed immediately after the first negotiation, and if an attacker intercepted this and retried the negotiation, they knew the secret had been reset to 0 ...

WHOOPS!

Cheers,
Wol

Meltdown strikes back: the L1 terminal fault vulnerability

davidgerard — Thu, 23 Aug 2018 10:20:25 +0000

You can run AWS instances on separate hardware, or even dedicated hardware. It just costs more. (And some fancy AWS functions aren't available at the higher levels of separation, though I'm not sure on specifics.)

At this stage I think it'd be remarkable if IT in general goes back to in-house hosting from the cloud providers. Renting compute just makes IT management so ridiculously easier. Particularly when you get into Terraform etc, where you can literally program what infrastructure you have. I haven't been in a machine room for over five years now, and have no plans to go back.

Meltdown strikes back: the L1 terminal fault vulnerability

abatters — Mon, 20 Aug 2018 17:45:32 +0000

The Cascade Lake server platform, shipping later this year, should contain the first round of Intel's hardware mitigations.

Anandtech: Intel at Hot Chips 2018: Showing the Ankle of Cascade Lake

Anandtech: An Interview with Lisa Spelman, VP of Intel’s DCG: Discussing Cooper Lake and Smeltdown

Meltdown strikes back: the L1 terminal fault vulnerability

zlynx — Sun, 19 Aug 2018 21:36:32 +0000

A friend of mine had his identity stolen and it actually improved his credit. It was probably an illegal immigrant who just wanted an SSN to get a job, etc.

Meltdown strikes back: the L1 terminal fault vulnerability

gus3 — Sat, 18 Aug 2018 22:55:36 +0000

Or you could keep your credit rating as poor as possible. Then if someone steals your ID & tries to get a loan using it, the joke's on them. (I would love to see the bank manager's attempts to stifle the laughter...)

Meltdown strikes back: the L1 terminal fault vulnerability

k8to — Sat, 18 Aug 2018 17:31:23 +0000

Nice joke, but I meant "this revision of these cpus have closed the door to this category of attack in this way".

Meltdown strikes back: the L1 terminal fault vulnerability

willy — Fri, 17 Aug 2018 21:47:34 +0000

One of the bits that is inverted is the "Uncached" bit. The CPU will not attempt to speculatively bring a cache line in from an uncached page.

Meltdown strikes back: the L1 terminal fault vulnerability

marcH — Fri, 17 Aug 2018 08:50:48 +0000

Afraid we're not quite yet in the age of "Open Hardware"

Meltdown strikes back: the L1 terminal fault vulnerability

Rearden — Fri, 17 Aug 2018 00:01:03 +0000

I think that argument is pretty reductive, and goes well past individual mitigation for this particular threat, and the reason why it's not strictly required to take extra steps in the case where both the Host and Guest OS's are "trusted".

Of course some futher privilege escalation vulnerability could expose the VM host OS to this, but a further privilege escalation vulnerability would likely also expose all sorts of other things as well, this vulnerability being just one of many.

Big picture security comes down to risk mitigation through a layered approach, depending on the resources available and the risk associated with a particular breach. Some future, possible "privilege escalation" vulnerability must be planned for outside of the rememdy for this specific vulnerability. What I mean is, if your workflow and risk for a system that you own both the VM and Host OS is high enough that a compromise of one could impact imporant data, you probably need to be taking the steps associated with "untrusted" guest VMs anyway.

Meltdown strikes back: the L1 terminal fault vulnerability

jcm — Thu, 16 Aug 2018 18:42:09 +0000

A "trusted" VM doesn't exist in reality. There are always going to be new bugs discovered that a determined attacker can use to compromise and perform privilege escalation. Then that "trusted" kernel becomes whatever they want very quickly. This is a nuance that isn't getting the necessary attention because it's a boring detail...

Meltdown strikes back: the L1 terminal fault vulnerability

k8to — Thu, 16 Aug 2018 18:03:49 +0000

Not quite on topic, but are there good resources describing any mitigation that is coming or has come for any of these problems? In other words, are there generations of CPU cores which won't be suffering from some types of problems like this in the future?

Meltdown strikes back: the L1 terminal fault vulnerability

k8to — Thu, 16 Aug 2018 15:51:56 +0000

Keep in mind that "private clouds" often have real security concerns between their various workloads as well. When you're operating at significant scale you have a variety of problems. You have no idea what those various teams are doing, or what software they're running. They run at lot of third party software too, and no one has really analyzed it in every detail to know what it does.

This is mostly sane to do if you set policies that control what software can access what data, but this type of exploit is about circumventing that.

So I don't really see going private cloud as a solution for this type of problem.

You could go "non-converged" and isolate workloads, but I don't think that's on the cards.

Meltdown strikes back: the L1 terminal fault vulnerability

nix — Thu, 16 Aug 2018 09:28:47 +0000

This attack also allows people to bypass Intel's licensing restrictions and launch arbitrary production enclaves on non-patched processors.

Can we keep this valuable feature while blocking the rest, I wonder? (No doubt we can't.)

Meltdown strikes back: the L1 terminal fault vulnerability

jcm — Thu, 16 Aug 2018 05:18:56 +0000

There is a third alternative path to mitgation, and that is to completely refactor the virt stack on the host so that we either don't load secrets or scrub them when we do. The technology MS announced called HyperClean is essentially doing this in Hyper-V. To do it in Linux with a full host OS stack would be extremely tough, but if folks want to use HT with untrusted workloads, then it's one of the options worthy of considering in the longer term. The thing with flushing secrets and limiting them is it's probably going to come in useful any time there's some kind of information disclosure vulnerability in the future.

Meltdown strikes back: the L1 terminal fault vulnerability

Rearden — Thu, 16 Aug 2018 03:08:38 +0000

But, there would be no reason for a "friendly" VM to run the unpatched kernel in the virtualized environment. The issue for hostile hosts only manifests when the "attacker" can run an un-patched kernel in the VM. As long as the VM's kernel is patched to invert the PFN, then it doesn't matter if someone attempts the attack against the VM kernel, it won't be affected.

Meltdown strikes back: the L1 terminal fault vulnerability

ms-tg — Wed, 15 Aug 2018 23:38:38 +0000

Applause - I see what you did there

Meltdown strikes back: the L1 terminal fault vulnerability

rahvin — Wed, 15 Aug 2018 23:32:45 +0000

Simple, move to AMD, their processors are significantly different and have suffered from very few of the Spectre attacks to which Intel has been vulnerable. From what I've seen it looks like Intel took shortcuts for performance reasons where AMD appears to have done it as securely as possible for the most part. A move to AMD also keeps you on x86 which is significantly cheaper than moving to something like Power.

Meltdown strikes back: the L1 terminal fault vulnerability

rahvin — Wed, 15 Aug 2018 23:25:39 +0000

The developer of the hack only evaluated Intel, they think AMD does it different so it's probably not vulnerable but I would like to see someone come out and say they tested this hack and it's variants of this technique on AMD before we completely clear it.

Meltdown strikes back: the L1 terminal fault vulnerability

marcH — Wed, 15 Aug 2018 22:14:47 +0000

Yeah, because we should be very careful to keep "top-secret" the credit card numbers, addresses, date of births and social security numbers that we... keep handing out left and right and that can all be bought for next to nothing on the dark web.

I get your actual point, it's just that you could have chosen a real example as opposed to propagating the American security myth that confuses login and password.

Meltdown strikes back: the L1 terminal fault vulnerability

NAR — Wed, 15 Aug 2018 21:52:45 +0000

The existence of bugs doesn't cause problems on their own. The actual exploitation of bugs would cause problems. So unless a worm comes around that steals credit card information from all AWS instances, people won't leave the cloud.

Meltdown strikes back: the L1 terminal fault vulnerability

roc — Wed, 15 Aug 2018 20:09:29 +0000

Thanks for that!

Meltdown strikes back: the L1 terminal fault vulnerability

jcm — Wed, 15 Aug 2018 18:46:19 +0000

We should give kudos to Microsoft here. They thought about the potential risks from tight resource sharing some time ago and obviously were able to invest significantly ahead of any one vulnerability such as this one in secret scrubbing/address space isolation/core scheduling/etc. This is something that we'll need to look at in Linux and other OSes over time if we want to make HT totally safe as well. It's a great example of what's possible, but a big lift. We do need to also get over the mindset that SMT threads are "cores". Especially in projects like OpenStack. An SMT thread is not a core, and we should never treat it as such, but it's easy to think in those terms when looking at /proc/cpuinfo output and just counting "cpus".

Meltdown strikes back: the L1 terminal fault vulnerability

jcm — Wed, 15 Aug 2018 18:36:19 +0000

There is a limit depending upon configured MAXPHYADDR and the number of bits/translation levels supported. Effectively, even with things like Superdome, there are no boxes shipping today where it's a problem. In theory, it /could/ become a problem prior to future platforms with 5-level paging (extra PA bits) but I raised this very corner case a few months ago to keep an eye on it. Now this is public, I'll ping the vendors I can think of who might be impacted and ask them to further consider for future platforms.

Meltdown strikes back: the L1 terminal fault vulnerability

Sesse — Wed, 15 Aug 2018 18:34:08 +0000

FWIW, my request was for just “not at the same time” (to make timing attacks much harder), not flushing L1 on every context switch. That's too heavy-handed for most userspace.

Meltdown strikes back: the L1 terminal fault vulnerability

jcm — Wed, 15 Aug 2018 18:32:02 +0000

It would be nice if we could guarantee no secrets were loaded, track, and flush them, but this isn't something most general purpose OS stacks running a full environment on a host do today. It is something some hypervisors are doing to mitigate against L1TF, and obviously is going to be investigated over time to improve the state of available mitigations on Linux, but it's very non-trivial.

Meltdown strikes back: the L1 terminal fault vulnerability

zdzichu — Wed, 15 Aug 2018 16:29:29 +0000

If I've been stocking private DC, I would be buying AMD.

Meltdown strikes back: the L1 terminal fault vulnerability

Curan — Wed, 15 Aug 2018 15:31:26 +0000

Yes, only Intel is affected, according to the kernel documentation.

Meltdown strikes back: the L1 terminal fault vulnerability

JoelWilliamson — Wed, 15 Aug 2018 14:17:22 +0000

It looks like the Core Scheduler for Hyper-V was introduced in 2016. I'm sure Microsoft had some advance knowledge, but I doubt they knew about this before Project Zero discovered Meltdown/Spectre. More likely, Microsoft introduced this as a general mitigation against any leaking between threads sharing a core, or simply to give guests a bit more control of their own scheduling.

Meltdown strikes back: the L1 terminal fault vulnerability

nilsmeyer — Wed, 15 Aug 2018 14:02:59 +0000

Which also goes to show that they had advance knowledge of the vulnerability.

Meltdown strikes back: the L1 terminal fault vulnerability

adam820 — Wed, 15 Aug 2018 14:02:49 +0000

They probably make more from the massive horizontal growth of cloud providers. They win either way, really; how many bugs are needed to make a shift to a different architecture (POWER, maybe?), or to stop using Intel-based arch's altogether?

Even better would be to spend more time with more researchers looking for this kind of stuff before these devices ever get released. Can't catch 'em all, though.

Meltdown strikes back: the L1 terminal fault vulnerability

fuhchee — Wed, 15 Aug 2018 13:26:57 +0000

How many more such bugs are needed to undermine confidence in shared cloud servers, and have companies retrench to on-premises computing? It would be ironic if Intel were to benefit from their bugs by virtue of motivating a purchasing stampede for private data center hardware.

Meltdown strikes back: the L1 terminal fault vulnerability

MarcB — Wed, 15 Aug 2018 12:13:16 +0000

I previously linked to https://www.heise.de/ct/artikel/Exclusive-Spectre-NG-Mult...

So far, it has proven correct about dates - May and August - as well as impact:"Specifically, an attacker could launch exploit code in a virtual machine (VM) and attack the host system from there...Intel's Software Guard Extensions (SGX), which are designed to protect sensitive data on cloud servers, are also not Spectre-safe".

So, this provides a lower boundaty.

Meltdown strikes back: the L1 terminal fault vulnerability

danpb — Wed, 15 Aug 2018 11:48:02 +0000

Even if the same person/organization owns the guest VM and host, the "trusted" guest VM can become "untrustworthy" if some software component in it gets compromised (through one of the countless bugs all complex software has). So if they're relying on use of VMs to isolate their applications from each other, the risk is still notable even in the single owner case.

Meltdown strikes back: the L1 terminal fault vulnerability

amw — Wed, 15 Aug 2018 11:15:56 +0000

No, but it might be possible to work it out by observing their behaviour from the outside :-)

Meltdown strikes back: the L1 terminal fault vulnerability

roc — Wed, 15 Aug 2018 11:06:04 +0000

Is it "allowed" for someone in the limited-disclosure club to tell us just how many speculative-execution vulnerabilities are currently in the pipeline?

Meltdown strikes back: the L1 terminal fault vulnerability

roc — Wed, 15 Aug 2018 10:51:41 +0000

One very important issue that the article didn't discuss (maybe it deserves its own article?) --- by extracting secrets from Intel's architectural enclaves, the existence of these attacks has severely damaged the SGX remote attestation ecosystem.

> Was the remote attestation protocol affected by Foreshadow?
> Yes. Using Foreshadow we have successfully extracted the attestation keys, used by the Intel Quoting Enclave to vouch for the authenticity of enclaves. As a result, we were able to generate "valid" attestation quotes. Using these counterfeit quotes, successfully "proved" to a remote party that a "genuine" enclave was running while, in fact, the code was running outside of SGX, under our complete control.
> Is SGX long-term storage affected by Foreshadow?
> Yes. As Foreshadow enables an attacker to extract SGX sealing keys, previously sealed data can be modified and re-sealed. With the extracted sealing key, an attacker can trivially calculate a valid Message Authentication Code (MAC), thus depriving the data owner from the ability to detect the modification.

The ecosystem has to be effectively rebooted by distrusting all attestations from enclaves running on non-patched processors, and all sealed data produced by those enclaves.

This attack also allows people to bypass Intel's licensing restrictions and launch arbitrary production enclaves on non-patched processors.