LWN: Comments on "Replacing x86 firmware with Linux and Go"

Replacing x86 firmware with Linux and Go

l0k1verloren — Wed, 20 Nov 2019 13:35:48 +0000

The least mature part of Go is the standard library. There is no doubt that much of its under-the-hood stuff and syntax helps avoid most of the bugs that lead to exploits (buffer overflows, code injection, and so on) but it's stated up front that this is aimed equally towards productivity and stability as it is to security.

Go's use in quick-and-dirty and small applications usually written in an interpreted languages is a natural fit for a language that favours automation over manual hand coding, but its automation integration into the build system is poor and the modules are also very immature.

A vector of attack that I suspect will be found in Go (standard library, to be precise) will come from mishandling of file handles or its lack of (easy) control of mutability and flaws in the work stealing scheduler. Go is an immature language system, to be sure, but ticks a lot of boxes.

If you work with Go you would be aware that recently a lot of fixtures in the standard library are being replaced and extended, notably errors and several others.

Go's main advantage is it focuses on the weaknesses of programmers and development management's tendencies towards over-elaboration, this is in itself in one whack wiping out a whole swathe of potential seecurity issues. But that is today secure, tomorrow there will always have to be more changes.

I know in my code my most frequent bug type comes from scope shadowing, a problem solveable with static analysis tools though few exist for this problem yet.

It is easy to criticise the rapid mutation of this young language but it is very accessible to especially the type of impatient, creative type of programmer who is obstructed from working with more formalised and complex language syntaxes like most other languages, without sacrificing readability. Problems will emerge quickly and be solved quickly in this stage of development.

There will always be bugs but bug-blind programming practices are worse.

Replacing x86 firmware with Linux and Go

erpalmer — Fri, 01 Dec 2017 18:01:56 +0000

The source for OpenPOWER boot firmware is indeed available in GitHub. See an overview of the components here https://www.ibm.com/developerworks/library/l-protect-syst...

Replacing x86 firmware with Linux and Go

rahvin — Thu, 30 Nov 2017 23:55:37 +0000

Given the patents were sold off first and aren't contained in the sale to the VC firm I'm not sure. Paying that much money for a dying architecture with no significant intellectual property without plans to get more licenses out there would be very un-VC. Imagination was always pretty half-hearted with MIPS and didn't seem to be willing to advance the platform by spending money fixing it's problems. Properly executing and spending money on design and development and creating a model more like ARM with good tools, available off the shelf designs and multiple license types with a particular focus on designs that are already done and tested would do much to revitalize MIPS and make it competitive again.

Replacing x86 firmware with Linux and Go

arnd — Thu, 30 Nov 2017 10:37:05 +0000

My impression is that they would have a harder time turning it around than Imagination, which seemed to actually try but failed to get any significant market wins. While MIPS cores still make up a significant portion of the the router market, I'm not aware of any recent product announcements, the manufacturers have either replaced the CPU cores with ARMv7 a few years ago and moved on to ARMv8 in 2017 (Broadcom, Qualcomm/Atheros, Mediatek/Ralink, Cavium), replaced them with x86 (Intel/Lantiq), or discontinued the product line (Realtek).

The situation seems similar for architecture licensees (Cavium, Raza/Netlogic/Broadcom, Lexra/Realtek, Loongson, Ingenic, SiCortex), all of which seem to have stopped coming out with new MIPS based designs in the last few years.

Replacing x86 firmware with Linux and Go

klossner — Wed, 29 Nov 2017 20:05:11 +0000

That patent was filed in 1986, so it is no longer of concern.

Replacing x86 firmware with Linux and Go

rahvin — Wed, 29 Nov 2017 18:23:26 +0000

MIPS ownership changed hands in 2017, it was sold to a VC firm. This may mean wider deployment and licensing, right now MIPS is primarily contained to routers and other embedded systems.

>MIPS business is sold by Imagination Technologies to Tallwood Venture Capital as Tallwood MIPS Inc. for $65 million. [52]

https://en.wikipedia.org/wiki/MIPS_Technologies

Replacing x86 firmware with Linux and Go

joib — Fri, 24 Nov 2017 19:58:35 +0000

> From my talks with people from IBM and Google I have learned that Google is actually using large numbers of POWER servers in their infrastructure.

Oh? That's cool. Obviously google have been a public member of the entire openpower thing, but I wasn't aware that they are actually running it in production in significant quantitites.

Replacing x86 firmware with Linux and Go

glaubitz — Fri, 24 Nov 2017 12:28:11 +0000

> Imagine if Google demanded Intel to produce binary-blob-free versions of their processors and Intel refused. What is Google going to do about it? Go to AMD and demand binary-blob-free versions of their processors and hope they comply? Migrate their server farms over to ARM processors?

From my talks with people from IBM and Google I have learned that Google is actually using large numbers of POWER servers in their infrastructure. According to them, one of the main reasons why Debian has a POWER porr (ppc64el) is Google.

Replacing x86 firmware with Linux and Go

pityashian_sweatshopworker — Fri, 24 Nov 2017 11:13:15 +0000

Some rambling talking:

The final SMM base for each core is not always is constant. The SMM space is usually nearby other *stolen* memory spaces (like integrated GFX or ME) to make more available memory space to be continuous. The 0x30000 (hardware reset default) is used as a trampoline or temporary space to craft SMM context only. (Some old guy SMM base called as A-segment 0xA0000 which is nearby the VGA ROM, on the recent machine which is called as T segment, the T is referred as the 'T' in TOLUD possibly).

About flash based SMM handler which is robuster than memory based specially with RAS slang, but the footprint size vs cost problem emerged. The more functionality make the footprint size to grow up. Even recent Intel ME (for serve product segment) also reside in the stolen memory instead of SRAM! Once server ME stop working, a lot of function of the *isolated* BMC will stop working too.

I guessed bravely that prior to CPU/SMM context is initialized w/o any examination(maybe i am wrong), it seems unlikely SMM entry will be conducted for a CPU. Even in the more restrictive environment, STM maybe used to monitor SMM activities.

A-bit off topic,a notorious and common problem of SMM security is that in some laptop implementation, it will disable the execution out of SMM space exception and then call video BIOS or GOP in SMM directly and roughly.

Replacing x86 firmware with Linux and Go

glaubitz — Fri, 24 Nov 2017 10:16:10 +0000

> Sparc is toast. Theoretically it could be resurrected.

Meh, Oracle just recently released M8 and Fujitsu is still selling new servers.

SPARC is also used by ESA in the form of Leon. Oracle is also still paying kernel developers to submit kernel patches for sparc64, so it's not really sure what the current plans are.

Either way, Debian is still supporting sparc64 (I'm one of the porters).

Replacing x86 firmware with Linux and Go

mjg59 — Thu, 23 Nov 2017 21:03:00 +0000

That doesn't write to the secure part of the variable store.

Replacing x86 firmware with Linux and Go

flussence — Thu, 23 Nov 2017 19:31:07 +0000

> I see no reason that the secure part of flash ever needs to be written after boot.
How about the efivars pstore driver? Saved me a lot of grey hair the last time my home server hit a kernel panic in the middle of the night, and I could actually diagnose it afterwards.

Replacing x86 firmware with Linux and Go

renox — Thu, 23 Nov 2017 09:16:06 +0000

Not so open: I remember a lawsuit about a patent on a MIPS instruction..
That said the original MIPS ISA is very old so any patent on it should have expired..

Replacing x86 firmware with Linux and Go

joib — Thu, 23 Nov 2017 07:26:48 +0000

Wikipedia says that Longsoon/ICT have bought a MIPS architecture license: https://en.wikipedia.org/wiki/Loongson#MIPS_patent_issues

Replacing x86 firmware with Linux and Go

eru — Thu, 23 Nov 2017 07:17:40 +0000

MIPS instruction set is free and open, and you can actually get completely free MIPS chips. However, it's not easy to find them.

There is the Chinese MIPS-compatible Loongson. I recall at one point years ago RMS was said to use a Loongson-based laptop, because it was totally open. But seems to be discontinued: http://lemote.kd85.com/

Replacing x86 firmware with Linux and Go

Cyberax — Thu, 23 Nov 2017 05:28:16 +0000

MIPS instruction set is free and open, and you can actually get completely free MIPS chips. However, it's not easy to find them.

Replacing x86 firmware with Linux and Go

drag — Thu, 23 Nov 2017 02:17:43 +0000

Sparc is toast. Theoretically it could be resurrected.

POWER is going strong, but it's very expensive. Technically it's a lot more open then X86_64. https://raptorcs.com/TALOSII/

ARM itself is technically closed, but at least companies have the option to purchase a license. I like ARM because it's free and you can still get binary blob-free ARM computers. The 'bootloader' in most ARM systems provide the same functionality that UEFI and such things do on X86_64. You can have complete control over the cpu from voltage-on up. Graphic drivers remain problematic, but are not strictly needed for basic function. Small and slow systems though. I don't know about the attempts to create server versions.

I don't know about MIPS. Probably in the same boat as ARM, but ARM is generally more competitive in the market place.

Replacing x86 firmware with Linux and Go

khim — Thu, 23 Nov 2017 01:50:01 +0000

Migrate their server farms over to ARM processors?

Well, ARM, POWER, or MIPS. I don't think SPARC is viable anymore. Although I'm not sure ARM is better from "blob-free" POV. How are POWER and MIPS in that regard? I have no idea, actually.

Replacing x86 firmware with Linux and Go

pbonzini — Wed, 22 Nov 2017 23:52:10 +0000

Even if you have a chain of trust to the kexec-ed Linux, it isn't necessarily fine for it to do arbitrary accesess the key store (or at least, in the UEFI trust model, it's not).

If it's not, you need some "trusted" code to survive from the pre-kexec environment, in order to validate accesses to flash.

> And a little anecdote is default UEFI SMM handler always bright up other core APs into SMM synchronously (the CPU troop is spining and waiting).

Yup, in fact port 0xB2 gets all cores into SMM. We tried avoiding that for KVM's implementation of SMM (which is only used for UEFI secure boot), but we kept finding bugs in Tiano Core and finally decided to get the thundering herd of CPUs into SMM just like on bare metal. :( But we still do not support hotplug with secure boot because Tiano Core lacks the required SMM magic (Intel hasn't open sourced it yet).

Replacing x86 firmware with Linux and Go

drag — Wed, 22 Nov 2017 16:41:29 +0000

> Google should have the market power to force Intel to sell processors with the ME physically disabled.

Google's 'market power' is being expressed in these sorts of projects that help get Chromebooks running with Coreboot and such things.

> partner with Amazon, Facebook, Microsoft and the other cloud providers to demand CPU's with the ME disabled.

Believe it or not features like ME and vPro are actually _selling points_. Intel didn't add these features out of eviliness, having lights off management and back doors into computers is valuable features for people running large numbers of desktops and other systems. They can use them to help enforce corporate policy and such things.

This is what you get when you deal with closed source software.

These features are valuable, but unless they are open source then they are evil.

> It bothers me greatly that Google and other cloud providers don't use this market purchasing power if they value the ME being fully disabled.

Being a big customer does not give you godlike powers over another person's business by itself. If you want to have clout in purchasing decisions you have to have the ability to go and use competator's products.

Imagine if Google demanded Intel to produce binary-blob-free versions of their processors and Intel refused. What is Google going to do about it? Go to AMD and demand binary-blob-free versions of their processors and hope they comply? Migrate their server farms over to ARM processors?

Markets are self-regulating, but only if there is substantial competition. IP laws and such things limit competition and thus limit the ability for people to demand large corporations to obey. Production of unencumbered processors like RISC-V is the ultimate 'out' for this thing and if they are successful will solve these sorts of regulatory issues, but in the meantime Intel is only going to care if it costs them customers.

Hopefully AMD takes the hint.

Replacing x86 firmware with Linux and Go

cortana — Wed, 22 Nov 2017 14:46:10 +0000

If you run the installer in advanced mode, you can choose to not install any boot loader. Otherwise you can simply the grub-pc or grub-efi-amd53 packages.

Replacing x86 firmware with Linux and Go

nim-nim — Wed, 22 Nov 2017 08:26:15 +0000

It's not extensible but it ships with a compiler… WTF.

The Go ecosystem produces wonderful apps, but they need to fix their ship to production story. Piling up massive amounts of intertwined Go modules with no release discipline, massive forking and wheel reinvention, last-mile compilation of this mass in a single static binary, only goes so far. Security researchers are going to have a field day if Go software escapes the world of GitHub-plugged devs.

Go is only escaping black marks because there are few public Go deployments where everything is not continuously rebuilt (Not that the result is secure, it's just too mutating to be easily audited, so researchers have chosen to look elsewhere so far. Remember when they decided to look at Java and uncovered pervasive years-old vulnerabilities?).

Replacing x86 firmware with Linux and Go

MakeHinduGreateAgain — Wed, 22 Nov 2017 06:45:01 +0000

I thought the better description is that install my customer SMM handler in Linux payload (As https://github.com/rminnich/linux/commits/monitor). The RAS related function code is resident in SMM and some of the registers are SMM access only if my memory is correct.

And a little anecdote is default UEFI SMM handler always bright up other core APs into SMM synchronously (the CPU troop is spining and waiting). If you have more AP cores, the more latency you will get. (Although there are some security implication here). A modern server machine has at least twelve cores. There will be some performance and latency issue.

Replacing x86 firmware with Linux and Go

ThinkRob — Wed, 22 Nov 2017 00:32:10 +0000

AFAIK this doesn't work for laptops where Boot Guard is enabled and enforcing using the keys burned in during board manufacture.

I have an X230 running coreboot w/ a stripped + HAPd ME, but with subsequent ThinkPads actually modifying the ME seems to cause Boot Guard to brick the whole system. And since the key is supposedly OEM-specific and burned into the chipset (CPU?) itself, it's not like you can replace it yourself.

What I *don't* know is if setting the HAP bit but leaving the rest of the ME image the same will trigger Boot Guard's hissy fit.

(The above is based on my understanding of Boot Guard, which may be comically wrong. Please correct if it is!)

Replacing x86 firmware with Linux and Go

pbonzini — Wed, 22 Nov 2017 00:04:10 +0000

I forgot a sentence: 0x30000 doesn't point into flash, so you need to bring everyone in SMM when the newly-hotplugged CPU does SMBASE relocation.

Replacing x86 firmware with Linux and Go

pbonzini — Wed, 22 Nov 2017 00:02:54 +0000

It's certainly possible, the problem is that 0x30000 doesn't point into flash. :(

Defaulting SMBASE to 0xA0000 would have made a lot more sense.

Replacing x86 firmware with Linux and Go

rahvin — Tue, 21 Nov 2017 23:22:34 +0000

I commend your work but the question I've been asking all along is why is this necessary? Google should have the market power to force Intel to sell processors with the ME physically disabled. And even if Google alone doesn't have the market power they should be able to partner with Amazon, Facebook, Microsoft and the other cloud providers to demand CPU's with the ME disabled.

It bothers me greatly that Google and other cloud providers don't use this market purchasing power if they value the ME being fully disabled. Google purchase millions of servers a year and combined with the other cloud providers you constitute a major portion of the server market, I'd guess near 50% or higher. The cloud providers have the power to effect Intel directly and have not acted when the ME is a security threat of unknown proportion.

There are already blackhat exploits to provision the ME with user access to the system. Right now these attacks require user access but as well all know user access can quickly become remote access as the vulnerability is probed and more information comes to light. The ME is a massive security threat and if the cloud providers care about security they should be doing everything in their power to see processors without the ME made available.

Replacing x86 firmware with Linux and Go

luto — Tue, 21 Nov 2017 22:08:55 +0000

True, although adding a whole microcontroller would cost more than just an I2C flash chip. The only service that I think is really needed is some NVRAM that can be written outside SMM to queue up variable writes. I don't know whether current Intel chips can lock down part of boot flash while leaving some unlocked.

> The OS could overwrite 0x30000 and gain access to SMRAM. "Why 0x30000?" is near the top of my list of x86 unanswered questions, even higher than "What the heck was CR1?".

I've never understood why the reset/SIPI vectors and the default SMBASE don't point to addresses that are unconditionally in the boot flash. I guess that all that's really needed is the ability to securely run something from flash before any OS code can possibly execute on a newly hot plugged CPU. Is this not actually possible without kicking all CPUs into SMM before powering up a new CPU?

Replacing x86 firmware with Linux and Go

pbonzini — Tue, 21 Nov 2017 21:06:41 +0000

Well, if you can throw additional hardware at the task that's certainly a possibility. :-) Instead of a second I2C flash you could also delegate UEFI variable services and APEI to a microcontroller (also on SMBus).

Oh, and there's actually another thing that SMM does, which is orchestrating CPU hotplug. And that's an ugly one because, due to the asinine default choice of SMBASE made in the 386SL(*), it's the only case that really, really needs to bring *all* processors at the same time on SMM.

(*) The OS could overwrite 0x30000 and gain access to SMRAM. "Why 0x30000?" is near the top of my list of x86 unanswered questions, even higher than "What the heck was CR1?".

Replacing x86 firmware with Linux and Go

cyphar — Tue, 21 Nov 2017 18:25:42 +0000

You can use me_cleaner in both the "remove as many modules as possible" and "enable HAP / MeDisable bit" modes -- which should reduce your concerns. This is what I've done on my X220, and I believe it is what Purism is doing for their new laptops.

Replacing x86 firmware with Linux and Go

luto — Tue, 21 Nov 2017 16:47:16 +0000

I don't really buy this. I see no reason that the secure part of flash ever needs to be written after boot. A high quality implementation of UEFI should be able to queue up variable writes such that the next boot will find them, validate them, and apply them before ever executing code off the disk.

I'm not at all sure that Intel's flash hardware can do this, but it's surely doable with a second plain I2C flash chip on the system SMBUS.

Replacing x86 firmware with Linux and Go

flussence — Tue, 21 Nov 2017 16:36:08 +0000

That's true, though there is a really good static linter for bash/sh, shellcheck. I try to write good shell scripts and yet it still catches an awful lot of mistakes.

If someone gets better results by using Go (or dmd-run, or perl, or whatever), more power to them.

Replacing x86 firmware with Linux and Go

jond — Tue, 21 Nov 2017 15:21:29 +0000

> That's true, but the point was, I think, that it's not hard or cumbersome to write reliable bash scripts.

I've been writing (what I hope are, at this point, reliable) bash scripts for an embarrassingly long amount of my professional career, as well as my hobby stuff, and I couldn't disagree with this more. It's fiendishly hard to do it well.

Replacing x86 firmware with Linux and Go

merge — Tue, 21 Nov 2017 14:42:01 +0000

sorry, my comment was off-topic. I assumed you run coreboot.

Replacing x86 firmware with Linux and Go

merge — Tue, 21 Nov 2017 14:38:51 +0000

That's no problem. I always configure coreboot to use the latest SeaBIOS as payload, so far.

In my case, it'd be even better use GRUB instead, and don't install GRUB via Debian on disk. I simply haven't had the time to dig in. Are there any nice posts about coreboot+GRUB and how to install and configure Debian without GRUB, and how to maintain grub's config files on disk?

I guess there are even less GRUB releases as there are SeaBIOS releases that would make me want to re-flash :)

Replacing x86 firmware with Linux and Go

mirabilos — Tue, 21 Nov 2017 14:09:51 +0000

Would be happy to replace the ME on my IBM Thinkpad… but only with something that then provides seabios to the to-be-booted operating system on the main CPU, so I can boot DOS, OS/2, old-style BSD, etc. as well.

Replacing x86 firmware with Linux and Go

abo — Tue, 21 Nov 2017 12:46:35 +0000

That's true, but the point was, I think, that it's not hard or cumbersome to write reliable bash scripts. bash is a domain specific language for calling open(), dup(), fork(), exec() etc., so naturally it's not very useful without something to exec() into.

For further reading

danpb — Tue, 21 Nov 2017 12:33:12 +0000

But don't expect this will be the end of it. With the level of complexity in the firmware, and its increased profile amongst security researchers, there's no reason to expect it will fair better than any other comparative software out there. IOW, it is reasonable to expect a steady stream of security vulnerabilities being reported for years to come... :-(

Replacing x86 firmware with Linux and Go

joib — Tue, 21 Nov 2017 12:10:08 +0000

Yes, ARM servers will feature UEFI and ACPI. See https://lwn.net/Articles/584123/

The relevant ARM standards are called SBSA (Server Base System Architecture) and SBBR (Server Base Boot Requirements).

Interestingly, it seems OpenPOWER uses something pretty similar to NERF called 'petitboot'; a simple (well, compared to UEFI at least) firmware that loads a Linux kernel + userspace from the flash rom, that system then kexec()'s the final distro kernel.

Replacing x86 firmware with Linux and Go

hugelgupf — Tue, 21 Nov 2017 06:33:54 +0000

I think it's worth mentioning here that Go is absolutely not mandatory to this project. The Linux-in-firmware component (which we have started calling LinuxBoot) can go with whatever userspace you like -- we just happen to use a small Go-based busybox-like initramfs.