LWN: Comments on "Grub2 updates for Red Hat systems are making some unbootable"

Grub2 updates for Red Hat systems are making some unbootable

nix — Mon, 14 Sep 2020 17:04:43 +0000

Plus you get the ability to use an EFI shell to do all sorts of repair work before the OS even gets started. (Sure, the stuff in an EFI shell is mostly useless *until you need it*, but then it is crucial). There is no such thing as a lilo shell!

Grub2 updates for Red Hat systems are making some unbootable

flussence — Fri, 07 Aug 2020 09:50:34 +0000

There's no reason to use Lilo on a UEFI system; you get exactly the same user interface by using an EFI-stub kernel (with a built-in command line to set rootfs) and the BIOS' boot menu, except there's one less part to forget to update.

Grub2 updates for Red Hat systems are making some unbootable

cesarb — Mon, 03 Aug 2020 19:16:36 +0000

Now that this has been fixed (https://access.redhat.com/errata/RHBA-2020:3265 and https://access.redhat.com/errata/RHBA-2020:3262), does anyone know what was the bug? I haven't seen any official announcement or blog post explaining what went wrong.

The only thing I have found so far is https://git.centos.org/rpms/shim-unsigned-x64/c/834e5a0c4... ("[PATCH] hexdump.h: fix arithmetic error") which mentions fixing rhbz#1861977 (https://bugzilla.redhat.com/show_bug.cgi?id=1861977).

Also on Debian based systems

kreijack — Sun, 02 Aug 2020 06:05:26 +0000

> Theoretically, GPT-partitioned disks have a fake MBR with one huge untouchable “partition” that covers all the space within the GPT
> partitions. (This only gets you so far if your disk is bigger than MBR will support.)

Correct.

To complete the answer, I have to point out that below of the "MBR" label there are two kind if information:
- the partition table
- the boot loader

Both are stored in the first sector. Grub-install change only the latter. So in any case grub-installer can't change nor damage the GPT table. However installing a MBR boot loader, could start the bios in legacy mode (and not uefi one).

Grub2 updates for Red Hat systems are making some unbootable

xnox — Sat, 01 Aug 2020 18:04:01 +0000

Does it provide TPM2 measurements and signature verification of boot artefacts?

Grub2 updates for Red Hat systems are making some unbootable

xnox — Sat, 01 Aug 2020 15:08:12 +0000

That is correct that everyone needs to push out installation & recovery media signed with new keys before pushing out dbx update.

Ubuntu is making 20.04 LTS, 18.04 LTS, 16.04 LTS point releases for that reason. Once they are out dbx update will be pushed out.

Also on Debian based systems

gfernandes — Sat, 01 Aug 2020 05:36:48 +0000

I did that at first, the first time I installed to a UEFI laptop. And ended up using the UEFI boot menu to switch.

I later realised that all OSs should _add_ to the same UEFI partition, that should already be there if you got an off the shelf laptop with WhineDoze preinstalled.

So I now first increase the size of the UEFI partition, and then install Fedora on another disk with /boot/efi pointing to the same UEFI partition. And since then, dual boot works fine and can be driven from the grub menu.

Also on Debian based systems

anselm — Fri, 31 Jul 2020 21:23:36 +0000

In fact both GPT and MBR partition can co-exist together (and say the same thing or different thing ! more often the latter).

Theoretically, GPT-partitioned disks have a fake MBR with one huge untouchable “partition” that covers all the space within the GPT partitions. (This only gets you so far if your disk is bigger than MBR will support.)

Grub2 updates for Red Hat systems are making some unbootable

amacater — Fri, 31 Jul 2020 21:06:56 +0000

https://www.debian.org/security/2020-GRUB-UEFI-SecureBoot... is a well written page by someone I trust - debian-cd lead, someone who understand UEFI inside out from his time at ARM and someone who has been instrumental in the last couple of weeks in getting this stuff sorted.

Also on Debian based systems

kreijack — Fri, 31 Jul 2020 20:58:35 +0000

> > I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

> What would it do? Treat the GPT partition table as MBR and corrupt it? While definitely not pretty, that should be recoverable as GPT has a backup table hopefully not located in the first few sectors.

Apart the fact that boot device could not be /dev/sda, the GPT table would be not affected. In fact both GPT and MBR partition can co-exist together (and say the same thing or different thing ! more often the latter).

I think that the real risk is that some bios doesn't start in UEFI mode if a MBR partition table is available. My BIOS (which is quite old) didn't show any UEFI related option until I removed all MBR partition table.

Grub2 updates for Red Hat systems are making some unbootable

ajmacleod — Fri, 31 Jul 2020 19:13:14 +0000

I generally stay as far as possible from UEFI and where I have a choice of bootloader I like syslinux. Grub seemed like (was) a great advance on LILO in many ways but Grub2 is a huge confusing mess.

I wish I'd read this article before updating practically all my grub booting servers (CentOS 7 and Debian 10)! Here's hoping there's a fix that can be applied before any reboots are required...

Grub2 updates for Red Hat systems are making some unbootable

jhhaller — Fri, 31 Jul 2020 15:52:23 +0000

Does installation media need to up updated to include newly signed components? Once dbxtool for a distribution has been updated, and the UEFI revocation list rejects older grub2 or shims, one needs an installation media with the newer grub2/shims, or a re-installed system won't boot without disabling secure boot, unless I'm missing something. Are there signs of installation media/ISO being reissued with these patches? It looks like RedHat OpenShift/CoreOS is only being distributed with new boot images, but haven't seen evidence of updated installation media for the rest of RedHat so far. System rescue images are also likely to need updating. It looks like Debian 10.5 will be issued with these patches, so at least it's installation and live media will be available on August 1.

Also check your BIOS vendor, at least HP (not to be confused with HPE) has reported that a BIOS update is required for some of their models before installing the new revocation list.

Also on Debian based systems

nivedita76 — Fri, 31 Jul 2020 15:38:25 +0000

Right, I was just explaining how you can tell it which ESP if there's more than one. It could be doing that if you just supply the device name as well, just unsure if it does.

Grub2 updates for Red Hat systems are making some unbootable

nescafe — Fri, 31 Jul 2020 14:14:44 +0000

LILO has never supported UEFI, ELILO is abandonware and generally isn't signed to be secure boot enabled (ditto for gummiboot/systemd-boot). If you want UEFI secure boot, shim + grub2 is the only game in down that works out of the box for the major distros.

Grub2 updates for Red Hat systems are making some unbootable

anselm — Fri, 31 Jul 2020 12:38:53 +0000

How does LILO cope with things like encrypted disks?

Also on Debian based systems

cortana — Fri, 31 Jul 2020 12:29:28 +0000

Someone should update the GRUB manual then!

23 Invoking grub-install
************************

The program 'grub-install' generates a GRUB core image using
'grub-mkimage' and installs it on your system.  You must specify the
device name on which you want to install GRUB, like this:

     grub-install INSTALL_DEVICE

   The device name INSTALL_DEVICE is an OS device name or a GRUB device
name.

Grub2 updates for Red Hat systems are making some unbootable

purslow — Fri, 31 Jul 2020 10:47:55 +0000

Why not use Lilo instead ? It's simple, reliable & adequate for most installations.
I've never used anything else in 20 years of Linux, including 17 years of Gentoo.

Also on Debian based systems

Wol — Fri, 31 Jul 2020 08:55:43 +0000

Well, I don't understand UEFI and all that stuff, but ...

When I tried to install SUSE on this laptop, it quite happily installed - UEFI - pointing to the UEFI partition it created on sdb. Of course, the laptop boots Windows off sda, so SUSE-install (presumably grub) put it in the wrong place and the laptop doesn't even realise SUSE is there :-(

Cheers,
Wol

Grub2 updates for Red Hat systems are making some unbootable

madhatter — Fri, 31 Jul 2020 06:56:46 +0000

Having spent the past 24 hours trying very hard to glue my mail server back together (from the pile of tiny shards left behind by the update) I can confirm this affects CentOS 7 as well.

Also on Debian based systems

dmoulding — Fri, 31 Jul 2020 03:16:54 +0000

Oh, for sure. But if you tell it which disk you want to install it to, it can easily find the ESP on that disk (by looking for the one with the ESP type UUID) and mount it itself. Then you don't need to bother mounting it in advance, nor telling it where you've mounted it.

I suppose it's still got to know which file system has the contents of /boot so that it knows where to put its modules, and the other things it wants to put in /boot/grub. That's a little harder since there isn't a designated type UUID for the partition containing /boot (and it could be on an LVM volume or something else that's not a GPT partition). So you might have to in some cases give it the --boot-directory option, as well (or mount the desired file system at /boot in advance).

Nevertheless, I still don't see how specifying a disk (as long as it's not wrong), could be more harmful than not specifying a disk.

Also on Debian based systems

nivedita76 — Fri, 31 Jul 2020 02:20:44 +0000

grub-install takes an --efi-directory argument to specify where the ESP has been mounted.

Grub2 updates for Red Hat systems are making some unbootable

geuder — Fri, 31 Jul 2020 01:53:06 +0000

I updated Ubuntu 16.04 (no secureboot) and 20.04 (secureboot) yesterday. No problems.

Grub2 updates for Red Hat systems are making some unbootable

cesarb — Thu, 30 Jul 2020 23:38:01 +0000

This is the official page for RedHat, with the full troubleshooting and recovery steps: https://access.redhat.com/solutions/5272311 (found in the links in the bugzilla page).

Excerpt:

"Red Hat is aware of this bug and is working on a fix.

DO NOT apply the affected Errata RHSA-2020:3217 for RHEL 7.
DO NOT apply the affected Errata RHSA-2020:3216 for RHEL 8.

For Red Hat Satellite exclude this Errata from your Content Views"

Also on Debian based systems

dmoulding — Thu, 30 Jul 2020 23:19:35 +0000

This is giving me a mental segfault.

There should be nothing wrong with doing "grub-install /dev/sda", assuming /dev/sda is the disk that contains the ESP. And, in fact, if you're running grub-install from a rescue system (which is UEFI) against another UEFI system's disk (such that the system currently has *two* disks attached to it, each one containing an ESP), then specifying the disk should be *mandatory* because otherwise grub-install has to choose which of the two it should install to, and it's almost certainly not going to guess that correctly every time. I would think the same would apply if you've got a rescue USB stick plugged into the system, which also contains an ESP.

Also on Debian based systems

leromarinvit — Thu, 30 Jul 2020 22:56:52 +0000

> The correct command on Debian/Ubuntu, for EFI systems, is just "grub-install", and let it auto-detect what it should do.

Good to know, thanks.

> I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

What would it do? Treat the GPT partition table as MBR and corrupt it? While definitely not pretty, that should be recoverable as GPT has a backup table hopefully not located in the first few sectors.

> Heck, you don't tell anyone to grub-install /dev/sda *even* for grub-pc (the "PC BIOS" edition), you say something like /dev/<boot device>...

That's definitely a useful clarification. While it's probably obvious to the average LWN reader that a bootloader needs to be installed on whatever drive the system wants to boot from, people blindly following instructions might run into trouble here.

Grub2 updates for Red Hat systems are making some unbootable

hmh — Thu, 30 Jul 2020 21:54:05 +0000

The Ubuntu issue seems to be an already broken setup that is not properly detected, and thus grub-install fails. You end up with the old grub in EFI, and the new grub modules in /boot. If the two don't like each other, the system fails to boot.

Debian had a regression related to dual-booting Microsoft Windows. It has already been fixed, the regression-fixing packages are ready, and the Debian security team tells me the regression-fixing update is already going through the EFI-signature-and-final-upload pipeline (no idea how long that takes, though).

Hopefully all distros will converge on the regression-fixes very fast...

I am not rebooting my EFI systems for a few days if I can help it, though :-P

Also on Debian based systems

zlynx — Thu, 30 Jul 2020 21:53:51 +0000

Well that's good.

As I remember things it did not always do that check for UEFI. Or perhaps it was someone booting an OS that didn't understand UEFI and it did it.

That kind of thing is what leads people to get upset with Windows because it will repeatedly repair the corrupted boot records. Which is not Windows' fault when it is the Linux users doing it wrong.

Also on Debian based systems

hmh — Thu, 30 Jul 2020 21:46:38 +0000

Fixed the Debian wiki page.

Also on Debian based systems

hmh — Thu, 30 Jul 2020 21:43:43 +0000

The correct command on Debian/Ubuntu, for EFI systems, is just "grub-install", and let it auto-detect what it should do. Unless you unmounted /boot/efi and removed its definition in /etc/fstab, in which case you're on your own. Do yourself a favor and ensure /boot and boot/efi are mounted read-write first. Feel free to change them back to a much safer read-only afterwards, though.

I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

Heck, you don't tell anyone to grub-install /dev/sda *even* for grub-pc (the "PC BIOS" edition), you say something like /dev/<boot device>...

Also on Debian based systems

cjwatson — Thu, 30 Jul 2020 21:28:49 +0000

No it won't - on UEFI, grub-install (for better or worse, but in this case better) just ignores any device name you give it and installs to the EFI System Partition anyway.

Also on Debian based systems

leromarinvit — Thu, 30 Jul 2020 20:02:45 +0000

Interesting. Both the Debian wiki and the Arch wiki suggest using exactly this command. What does it break, and what should be used instead to (re-)install GRUB on UEFI systems?

Also on Debian based systems

zlynx — Thu, 30 Jul 2020 17:34:57 +0000

For anyone reading that, never do that grub install command on a UEFI boot system. It will ruin it forever in hard to fix ways.

Also on Debian based systems

Smon — Thu, 30 Jul 2020 17:28:31 +0000

This also happened on debian 10 (buster). Only my bios based systems had this problem, efi works fine. (I also read online, that these are the observations from other people)
For me, chrooting into the broken system (or doing it while the system is still running), `grub-install /dev/sda` solved it for me.

Grub2 updates for Red Hat systems are making some unbootable

cesarb — Thu, 30 Jul 2020 16:55:52 +0000

According to comments on the HN post (https://news.ycombinator.com/item?id=23999212), this is also affecting at least Ubuntu and Debian: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889509