5.8 Merge window, part 1

Posted Jun 7, 2020 1:59 UTC (Sun) by flussence (guest, #85566)
In reply to: 5.8 Merge window, part 1 by darwi
Parent article: 5.8 Merge window, part 1

This would be a massive improvement over the current state of affairs; I've tried using the EFI backend before on a headless server and it worked great for a while, until one day it didn't and then fixing it required a screwdriver.

5.8 Merge window, part 1

Posted Jun 7, 2020 13:37 UTC (Sun) by josh (subscriber, #17465) [Link] (4 responses)

I would be interested in hearing the details on what went wrong.

5.8 Merge window, part 1

Posted Jun 8, 2020 5:46 UTC (Mon) by flussence (guest, #85566) [Link] (3 responses)

Extremely badly behaved AMI firmware; deleting EFI vars only hid them from view without deallocating space in nvram. After a few panics happened on one buggy kernel version it started returning -ENOSPC for efivarfs operations.

At the time I also had `installkernel` set up to update the efibootmgr entries during make install, which made everything worse - couldn't change the default boot to a known-good entry, it was stuck on the bad one.

5.8 Merge window, part 1

Posted Jun 8, 2020 5:55 UTC (Mon) by amacater (subscriber, #790) [Link] (1 responses)

That one bricked my Thinkpad - my main machine. Luckily, I was at a tech conference among friends and a couple of EFI experts where somebody suggested a complete reflash of the BIOS, someone else had an external USB that could fake a floppy - after one of the most panic stricken 3/4 hours of my life, it all worked. Many years later, it's beside me on the table, still working. Thanks MiniDebConf Cambridge

5.8 Merge window, part 1

Posted Jun 21, 2020 22:18 UTC (Sun) by nix (subscriber, #2304) [Link]

I heard about that incident. I've never dared use pstore on non-disposable hardware since.

I suspect this is too paranoid of me -- but it still feels nicer to be able to use a blockdev. (Or it would if all my disks weren't fully partitioned already -- I don't imagine it would work on USB mass storage... expecting *that* to work in panic state seems like asking a lot.)

5.8 Merge window, part 1

Posted Jun 8, 2020 8:04 UTC (Mon) by mjg59 (subscriber, #23239) [Link]

The "delete from view without reallocating" is actually a reasonable approach to not rewriting an entire flash block every time someone touches a variable. The problem was that the platform wouldn't trigger garbage collection until it detected it was out of space, and if you were *very* close to being out of space the firmware would run out of space before the garbage collection code ran.

There's not actually a good answer here. EFI firmware updates need to be in SMM because that's the only mechanism Intel provide to allow authenticated writes to flash, and SMM can't run without all cores being in SMM, so if you do a variable update and need to rewrite an entire block of flash you're going to halt the OS for long enough that things will be unhappy. Getting into a situation where you allow the OS to make the machine unbootable obviously isn't a great answer to that (https://lore.kernel.org/patchwork/patch/300747/ is arguably more egregious in this respect), but the singular bug that actually led us to this point is an understandable one.

(The -ENOSPC behaviour is accompanied by the kernel then attempting to create a variable it knows is too big in order to force the firmware to do a garbage collection run on next boot. If you boot via the UEFI boot stub then the kernel will do this while still in UEFI boot services, which should trigger the garbage collection before the kernel starts. As a result, this *should* now be largely invisible to users without putting systems at risk, but obviously we have no way whatsoever kf knowing how firmware actually works before we try it)