Debian to vote on its firmware path

Posted Sep 7, 2022 1:03 UTC (Wed) by ras (subscriber, #33059)
In reply to: Debian to vote on its firmware path by farnz
Parent article: Debian to vote on its firmware path

> IIUC, AMD-SEV is later in the process; the "binary blob" khim refers to is the firmware blob that runs on a CPU that's just booted up, and first enables "cache-as-RAM" then sets up the DRAM controllers. Without running this firmware blob on your CPU, you have no DRAM, and no cache either.

That probably doesn't matter. This "trust relationship" really means setting up a set of keys the the host doesn't know about. It's the same in principle as the DRM running in the browser. Provided no one can crack the DRM module, the content being sent is safe - despite it being handled by a huge pile of untrusted code. The browser DRM is just software of course, so anyone believing it can't be cracked is in a state of sin. And the "analogue hole" still remains wide open, although it's no so analogue because it's passing through the kernel.

Since the AES keys are in principle a secret tightly held between on CPU TPM, the owner of the VM, and the memory channel doing the encrypting they should be safe from anything that can't see into those things. But this is all provided by AMD. If you don't trust AMD to do it right then it's all over, as they could sabotage the whole thing in a zillion different ways you have no hope of discovering. So 100% trust is a given. The point is you can get away with just trusting AMD, but not the Russian's, despite the Russian's having physical custody of the CPU, and total control of the host OS running the VM - and total access to the memory.

That also applies in reverse. If the Russian's trust AMD 100%, they should be able to run a VM on a AMD-SEV machine controlled by the NSA. But the Russian's trusting a US company who is presumably subservient to the NSA would be quite a stretch, as you say.

Debian to vote on its firmware path

Posted Sep 7, 2022 9:11 UTC (Wed) by farnz (subscriber, #17727) [Link] (7 responses)

The challenging part is that the binary blob is what brings up the memory channel, which is in the trust path. It's entirely plausible that there's debug modes in the memory channel that break your assumptions, and that the binary blob is responsible for locking out into the "production" config. That makes the binary blob running on the CPU during early boot (before UEFI starts up in a traditional firmware) a part of the trusted base for AMD-SEV, and yet it's software-modifiable (stored in the machine's reflashable firmware store).

Further, you have to take updates to this code from your motherboard vendor on trust - you have no easy way to tell whether a given update is direct from AMD, or has been altered by the motherboard vendor on the request of a dodgy agency. This means that your trusted base is not just AMD, but also your motherboard vendor (since they are in a position to install a signing key into the system as a whole that results in the early firmware being trusted) - and potentially everyone who had access to the motherboard before it arrives in your datacentre.

This is why open-source firmware is interesting: you can confirm that the firmware you're running is the firmware you expect and not backdoored by the motherboard vendor (or in transit) since you can replace the flash storage (wiping out old on-CPU TPM keys in the process, since those live in the flash) on arrival, and you know what the firmware you're putting on there does. Your trust base is thus reduced from "is the binary blob the motherboard vendor supplied actually AMD's blob, and does it do what AMD promise" to "does the hardware AMD sold me contain hidden storage, and does it work as promised".

Debian to vote on its firmware path

Posted Sep 7, 2022 10:28 UTC (Wed) by ras (subscriber, #33059) [Link] (6 responses)

> The challenging part is that the binary blob is what brings up the memory channel,

The encryption is done on the CPU die, so I presume the binary blob comes from AMD. As I said, trusting AMD is assumed - and that includes trusting all binary blobs that come from them.

> Further, you have to take updates to this code from your motherboard vendor on trust - you have no easy way to tell whether a given update is direct from AMD, or has been altered by the motherboard vendor on the request of a dodgy agency.

I haven't checked, but surely AMD digital signs their CPU firmware, and the chip only accepts something signed appropriately?

> It's entirely plausible that there's debug modes in the memory channel that break your assumptions,

It doesn't sound too plausible to me. There has been some research into attacks against AMD-SEV already, which have found weaknesses. (I don't know whether they were fatal.) IIRC, they involve modifying the encrypted memory (which of course then yields junk when de-encrypted), and observing what happens when it is used. Repeat that until you see a pattern. It just doesn't sound reasonable they would miss something as obvious as using debug mode to spy on a memory channel.

> This is why open-source firmware is interesting

It is why a totally open source solution is interesting. If the chip is RISC-V, and the VHDL is open source, and you have a "reproducible build" (I have no idea what that would look like for hardware), then there is no need to trust AMD. You can verify it yourself. Of course the firmware must be open source too, because you have to trust the entire stack, just as you say.

But if you don't trust he hardware, all bets are off. It doesn't matter if the firmware is open source or not. There will be non upgradeable ROM firmware on the CPU - even if it's just to load the uploadable RAM replacement. It could be doing anything, including loading your open source firmware then throwing it away. This is why open source firmware alone is not so interesting if you are looking at it from just a security standpoint. It has to be turtles all the way down.

Debian to vote on its firmware path

Posted Sep 7, 2022 12:23 UTC (Wed) by farnz (subscriber, #17727) [Link] (5 responses)

The binary blob that sets up the memory channel is supplied to you by the motherboard manufacturer - and at least the last time I had this level of access, the motherboard manufacturer has the ability to rebuild that blob with changes, because you do want to be able to do things like rule out specific settings that, while legitimate for the CPU, will cause EMC issues because of the specific length of motherboard traces, and to rearrange the order in which different options are tested to increase the chances of finding the right training settings first attempt. That includes the ability for the motherboard manufacturer to sign the blob that goes onto their motherboards, so that the CPU will run it.

And debug modes that are controlled by a one-way disable are not uncommon in hardware design - once the debug mode has been disabled (and the AMD-supplied code will do this for you), it can never be re-enabled at runtime, which makes it useless for researchers (since the normal AMD blob will turn off debug modes before training DRAM, so you have no ability to turn them back on). As there's no visibility into this process, you can't tell if this is going on - you're taking it on trust that AMD either has no ability to debug AMD-SEV silicon bugs (sounds unlikely to me) or has correctly written the binary blob such that it disables debug options before training DRAM, and that no-one who is in a position to tamper with the blob has done so.

With open-source firmware, you would not have to trust that the blob you are running is unmodified from AMD's version and could look for mistakes in the very early bring-up code. It seems unlikely given the research into breaking AMD-SEV that they have badly implemented debug options (it's trivial in ASIC design to design a write-only register that starts up in the "enabled" state, and switches to a hard disabled state on any write, and I would expect AMD to be competent enough to not have debug features turned on when the debug control register is set to a hard disabled state), but that doesn't protect you if the motherboard manufacturer can change the code you run.

Debian to vote on its firmware path

Posted Sep 8, 2022 5:52 UTC (Thu) by ras (subscriber, #33059) [Link] (4 responses)

> The binary blob that sets up the memory channel is supplied to you by the motherboard manufacturer - and at least the last time I had this level of access, the motherboard manufacturer has the ability to rebuild that blob with changes, because you do want to be able to do things like rule out specific settings that, while legitimate for the CPU, will cause EMC issues because of the specific length of motherboard traces, and to rearrange the order in which different options are tested to increase the chances of finding the right training settings first attempt.

Yes, of course. But none of that will allow the firmware blob to see what is in memory. If it did, AMD-SEV would be a lost cause.

The point I was trying to make is this statement by khim is too broad:

> If you want to ensure that your CPU wouldn't be under control of NSA then you first need to design something which doesn't depend on binary blobs to, you know, access the RAM.

Untrusted binary blobs running in any card with DMA may well have access to the RAM. Hell, the untrusted drivers in the host OS will probably have access to all of RAM. But we don't need to "first design something" to fix that. AMD-SEV fixes it, and it's available now.

The point khim is making about it being easier to run Windows than it is to run Linux on consumer hardware still stands, of course. I wasn't try to argue against it - it's always been true. And yes that means most people, even college IT students, if forced to use Linux will take the easiest path and just run it under Windows. It's a new(ish) phenomenon. It sounds ominous but it's actually a huge win, because being forced to become familiar with Linux is new. I doubt it was altruistic motives that caused Microsoft to release WSL - it was fear of losing those users, users who will be making corporate IT decisions in the future. And in the meantime the true believers will continue fight tooth and nail to run only open source on bare metal just as they've been doing for the last 25 years. I suspect khim loses sight of the fact that while as a proportion of the "IT industry" the number of true believers is being diluted, I strongly suspect in absolute terms their numbers they have always been growing, and are consequently a much larger force to be reckoned with than 25 years ago. They may never win their fight completely, but I doubt they will ever lose it either.

Debian to vote on its firmware path

Posted Sep 8, 2022 12:57 UTC (Thu) by farnz (subscriber, #17727) [Link] (3 responses)

You are, however, taking it on trust that the version of the blob supplied to AMD's partners only lets them change link training parameters, and can't be modified to run before AMD's part of the blob locks out low-level debug registers (or that AMD's registers for debugging AMD-SEV cannot be misused to compromise AMD-SEV).

While a fully open end-to-end verified system would definitely be nicer, it's still not possible to be 100% confident that AMD-SEV only requires you to trust AMD, and not your motherboard manufacturer with the current setup; in theory, AMD could separate out the part of the blob that does cache-as-RAM setup and also takes responsibility for locking out debug registers from the part that does DRAM link training, and allow you to confirm that the AMD blob is signed separately to the motherboard vendor blob, but that's not how it works today - instead, the early boot code is a single blob, and you can't tell if it's AMD's blob or not.

As it is, though, it's not implausible that AMD has debug registers that can be locked out with a single MSR write, and that cause AMD-SEV to make the encryption key visible to a debugging system (either via software, or to external hardware attached to the CPU under test). In normal operation, those registers would be locked out before DRAM training happens (hence before AMD-SEV is relevant), but a modified blob would be able to leave those registers exposed.

And this is the deep-seated trouble with completely closed firmware like the DRAM training firmware for Intel + AMD CPUs; we don't know what it actually does, and therefore whether there's a trust boundary between AMD/Intel and the motherboard vendor, or whether the actual design merges them together for the purposes of trust.

Debian to vote on its firmware path

Posted Sep 9, 2022 5:41 UTC (Fri) by ras (subscriber, #33059) [Link] (2 responses)

> While a fully open end-to-end verified system would definitely be nicer, it's still not possible to be 100% confident that AMD-SEV only requires you to trust AMD

You are saying because there may be bugs / mistakes in AMD's implementation, it's not possible to be 100% confident AMD-SEV works as advertised. That is totally correct. In fact if someone offered me a bet that someone will find a way past it in the next decade years, I'd take it. (If it was Intel SGX, I'd be happy to reduce the period to months.)

But it's an impossible standard. Flip the argument. Lets say it was all open source. Would it be possible to be 100% confident and say there are no bugs and thus 100% trustworthy? Of course not. I'd be happy to take the same bet.

I'll let you ruminate on where that leaves us, but clearly it isn't: Security is complex. All complex technology has bugs. Ergo we can't 100% trust the security of our systems. (I hold those statements as self-evident.) Therefore we should give up on security.

And no, open source does not solve that problem.

Debian to vote on its firmware path

Posted Sep 9, 2022 8:44 UTC (Fri) by farnz (subscriber, #17727) [Link] (1 responses)

No - I'm making a much more subtle argument than that, relating to the size of my trust base.

I have to trust AMD's designs, and AMD's manufacturing partner for silicon, when using AMD CPUs. That's just a fact of life as it is at the moment; if I used Intel CPUs, I'd just change the word AMD for Intel in that sentence, and it'd still hold true (albeit that Intel current manufacture their own chips). Same applies for any CPU vendor - I trust that vendor's designs, and I trust that their manufacturing will implement the design as created by the vendor, not a modified version to target me.

That is the core trust base I have when using any CPU - I can't avoid trusting the designer and the manufacturer of the silicon. Beyond that, my trust base can be extended by design decisions - for example, in a traditional cloud system (without AMD-SEV), I must also trust the motherboard manufacturer, the BIOS vendor, the DIMM manufacturer, the DRAM chip maker, the hypervisor vendor, the hypervisor operator, and the datacenter owner.

The promise of AMD-SEV is that I no longer need to trust as large a trust base, because AMD-SEV uses cryptography to remove all but AMD from the trust base. I still have to trust AMD and whoever made the CPU, but I no longer have to trust all the other parties involved, since they can only see encrypted data, and cannot get the key. The value of this promise is, however, weakened by the fact that we can't see into the memory controller startup blob - because it is entirely opaque, there is a reasonable possibility that AMD used the startup blob to disable AMD-SEV debug features that break the promises of AMD-SEV (and I would consider having these features that can be disabled in a one-way operation to be entirely reasonable on AMD's part - they need to be able to debug things if their chips misbehave in their labs). And because there is one combined blob that contains both AMD-supplied code and motherboard vendor code, the effect of this is to widen my trust base for AMD-SEV from "just AMD" to "AMD and the motherboard vendor".

There's at least two routes AMD could take to reduce the size of the trust base back down to that promised by AMD-SEV:

Split the blob. Have two blobs, one from AMD that does cache-as-RAM setup and anything needed to make AMD-SEV safe, and a second from the motherboard vendor that does the DRAM training. This would allow me to verify that the AMD blob is indeed from AMD, without needing to trust the motherboard vendor too.
Open up the AMD parts of the blob - fully describe the blob and its system accesses, so that I can validate that the blob from my motherboard vendor only touches the registers that are involved in DRAM training (and thus can be confident that I only need to trust AMD, since my motherboard vendor has done nothing that AMD say is suspicious).

Both work - you will notice that one of them doesn't even involve opening up the blob. But unless AMD do one of the two options, I have no way of telling whether or not the blob expands my trust domain to include the motherboard vendor, or whether it in fact limits me to just trusting AMD because the motherboard vendor modifications aren't able to affect AMD-SEV.

Debian to vote on its firmware path

Posted Sep 9, 2022 9:41 UTC (Fri) by paulj (subscriber, #341) [Link]

"albeit that Intel current manufacture their own chips"

Not sure if that will be true for much longer, least universally. E.g., the information on the new Ponte Vecchio HPC chips is that some of the tiles in the package will be produced by TSMC. The same sources that broke this (now confirmed) also say Intel has plans to have TSMC produce some CPUs.