|
|
Subscribe / Log in / New account

Grub2 updates for Red Hat systems are making some unbootable

As reported in the comments on the Grub2 secure-boot vulnerabilities report, the updates for grub2 for RHEL 8 and CentOS 8 are making some systems unbootable. The boot problems are seemingly unrelated to whether the system has secure boot enabled. It may be worth waiting a bit for that to shake out.

to post comments

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 30, 2020 16:55 UTC (Thu) by cesarb (subscriber, #6266) [Link] (2 responses)

According to comments on the HN post (https://news.ycombinator.com/item?id=23999212), this is also affecting at least Ubuntu and Debian: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889509

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 30, 2020 21:54 UTC (Thu) by hmh (subscriber, #3838) [Link] (1 responses)

The Ubuntu issue seems to be an already broken setup that is not properly detected, and thus grub-install fails. You end up with the old grub in EFI, and the new grub modules in /boot. If the two don't like each other, the system fails to boot.

Debian had a regression related to dual-booting Microsoft Windows. It has already been fixed, the regression-fixing packages are ready, and the Debian security team tells me the regression-fixing update is already going through the EFI-signature-and-final-upload pipeline (no idea how long that takes, though).

Hopefully all distros will converge on the regression-fixes very fast...

I am not rebooting my EFI systems for a few days if I can help it, though :-P

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 1:53 UTC (Fri) by geuder (subscriber, #62854) [Link]

I updated Ubuntu 16.04 (no secureboot) and 20.04 (secureboot) yesterday. No problems.

Also on Debian based systems

Posted Jul 30, 2020 17:28 UTC (Thu) by Smon (guest, #104795) [Link] (17 responses)

This also happened on debian 10 (buster). Only my bios based systems had this problem, efi works fine. (I also read online, that these are the observations from other people)
For me, chrooting into the broken system (or doing it while the system is still running), `grub-install /dev/sda` solved it for me.

Also on Debian based systems

Posted Jul 30, 2020 17:34 UTC (Thu) by zlynx (guest, #2285) [Link] (16 responses)

For anyone reading that, never do that grub install command on a UEFI boot system. It will ruin it forever in hard to fix ways.

Also on Debian based systems

Posted Jul 30, 2020 20:02 UTC (Thu) by leromarinvit (subscriber, #56850) [Link] (6 responses)

Interesting. Both the Debian wiki and the Arch wiki suggest using exactly this command. What does it break, and what should be used instead to (re-)install GRUB on UEFI systems?

Also on Debian based systems

Posted Jul 30, 2020 21:43 UTC (Thu) by hmh (subscriber, #3838) [Link] (5 responses)

The correct command on Debian/Ubuntu, for EFI systems, is just "grub-install", and let it auto-detect what it should do. Unless you unmounted /boot/efi and removed its definition in /etc/fstab, in which case you're on your own. Do yourself a favor and ensure /boot and boot/efi are mounted read-write first. Feel free to change them back to a much safer read-only afterwards, though.

I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

Heck, you don't tell anyone to grub-install /dev/sda *even* for grub-pc (the "PC BIOS" edition), you say something like /dev/<boot device>...

Also on Debian based systems

Posted Jul 30, 2020 21:46 UTC (Thu) by hmh (subscriber, #3838) [Link]

Fixed the Debian wiki page.

Also on Debian based systems

Posted Jul 30, 2020 22:56 UTC (Thu) by leromarinvit (subscriber, #56850) [Link] (3 responses)

> The correct command on Debian/Ubuntu, for EFI systems, is just "grub-install", and let it auto-detect what it should do.

Good to know, thanks.

> I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

What would it do? Treat the GPT partition table as MBR and corrupt it? While definitely not pretty, that should be recoverable as GPT has a backup table hopefully not located in the first few sectors.

> Heck, you don't tell anyone to grub-install /dev/sda *even* for grub-pc (the "PC BIOS" edition), you say something like /dev/<boot device>...

That's definitely a useful clarification. While it's probably obvious to the average LWN reader that a bootloader needs to be installed on whatever drive the system wants to boot from, people blindly following instructions might run into trouble here.

Also on Debian based systems

Posted Jul 31, 2020 20:58 UTC (Fri) by kreijack (guest, #43513) [Link] (2 responses)

> > I have no idea where that /dev/sda in the wiki came from. It is never right for an EFI install, hopefully grub-install will ignore it, because if it doesn't, well, the result won't be pretty.

> What would it do? Treat the GPT partition table as MBR and corrupt it? While definitely not pretty, that should be recoverable as GPT has a backup table hopefully not located in the first few sectors.

Apart the fact that boot device could not be /dev/sda, the GPT table would be not affected. In fact both GPT and MBR partition can co-exist together (and say the same thing or different thing ! more often the latter).

I think that the real risk is that some bios doesn't start in UEFI mode if a MBR partition table is available. My BIOS (which is quite old) didn't show any UEFI related option until I removed all MBR partition table.

Also on Debian based systems

Posted Jul 31, 2020 21:23 UTC (Fri) by anselm (subscriber, #2796) [Link] (1 responses)

In fact both GPT and MBR partition can co-exist together (and say the same thing or different thing ! more often the latter).

Theoretically, GPT-partitioned disks have a fake MBR with one huge untouchable “partition” that covers all the space within the GPT partitions. (This only gets you so far if your disk is bigger than MBR will support.)

Also on Debian based systems

Posted Aug 2, 2020 6:05 UTC (Sun) by kreijack (guest, #43513) [Link]

> Theoretically, GPT-partitioned disks have a fake MBR with one huge untouchable “partition” that covers all the space within the GPT
> partitions. (This only gets you so far if your disk is bigger than MBR will support.)

Correct.

To complete the answer, I have to point out that below of the "MBR" label there are two kind if information:
- the partition table
- the boot loader

Both are stored in the first sector. Grub-install change only the latter. So in any case grub-installer can't change nor damage the GPT table. However installing a MBR boot loader, could start the bios in legacy mode (and not uefi one).

Also on Debian based systems

Posted Jul 30, 2020 21:28 UTC (Thu) by cjwatson (subscriber, #7322) [Link] (3 responses)

No it won't - on UEFI, grub-install (for better or worse, but in this case better) just ignores any device name you give it and installs to the EFI System Partition anyway.

Also on Debian based systems

Posted Jul 30, 2020 21:53 UTC (Thu) by zlynx (guest, #2285) [Link]

Well that's good.

As I remember things it did not always do that check for UEFI. Or perhaps it was someone booting an OS that didn't understand UEFI and it did it.

That kind of thing is what leads people to get upset with Windows because it will repeatedly repair the corrupted boot records. Which is not Windows' fault when it is the Linux users doing it wrong.

Also on Debian based systems

Posted Jul 31, 2020 8:55 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

Well, I don't understand UEFI and all that stuff, but ...

When I tried to install SUSE on this laptop, it quite happily installed - UEFI - pointing to the UEFI partition it created on sdb. Of course, the laptop boots Windows off sda, so SUSE-install (presumably grub) put it in the wrong place and the laptop doesn't even realise SUSE is there :-(

Cheers,
Wol

Also on Debian based systems

Posted Aug 1, 2020 5:36 UTC (Sat) by gfernandes (subscriber, #119910) [Link]

I did that at first, the first time I installed to a UEFI laptop. And ended up using the UEFI boot menu to switch.

I later realised that all OSs should _add_ to the same UEFI partition, that should already be there if you got an off the shelf laptop with WhineDoze preinstalled.

So I now first increase the size of the UEFI partition, and then install Fedora on another disk with /boot/efi pointing to the same UEFI partition. And since then, dual boot works fine and can be driven from the grub menu.

Also on Debian based systems

Posted Jul 30, 2020 23:19 UTC (Thu) by dmoulding (subscriber, #95171) [Link] (3 responses)

This is giving me a mental segfault.

There should be nothing wrong with doing "grub-install /dev/sda", assuming /dev/sda is the disk that contains the ESP. And, in fact, if you're running grub-install from a rescue system (which is UEFI) against another UEFI system's disk (such that the system currently has *two* disks attached to it, each one containing an ESP), then specifying the disk should be *mandatory* because otherwise grub-install has to choose which of the two it should install to, and it's almost certainly not going to guess that correctly every time. I would think the same would apply if you've got a rescue USB stick plugged into the system, which also contains an ESP.

Also on Debian based systems

Posted Jul 31, 2020 2:20 UTC (Fri) by nivedita76 (subscriber, #121790) [Link] (2 responses)

grub-install takes an --efi-directory argument to specify where the ESP has been mounted.

Also on Debian based systems

Posted Jul 31, 2020 3:16 UTC (Fri) by dmoulding (subscriber, #95171) [Link] (1 responses)

Oh, for sure. But if you tell it which disk you want to install it to, it can easily find the ESP on that disk (by looking for the one with the ESP type UUID) and mount it itself. Then you don't need to bother mounting it in advance, nor telling it where you've mounted it.

I suppose it's still got to know which file system has the contents of /boot so that it knows where to put its modules, and the other things it wants to put in /boot/grub. That's a little harder since there isn't a designated type UUID for the partition containing /boot (and it could be on an LVM volume or something else that's not a GPT partition). So you might have to in some cases give it the --boot-directory option, as well (or mount the desired file system at /boot in advance).

Nevertheless, I still don't see how specifying a disk (as long as it's not wrong), could be more harmful than not specifying a disk.

Also on Debian based systems

Posted Jul 31, 2020 15:38 UTC (Fri) by nivedita76 (subscriber, #121790) [Link]

Right, I was just explaining how you can tell it which ESP if there's more than one. It could be doing that if you just supply the device name as well, just unsure if it does.

Also on Debian based systems

Posted Jul 31, 2020 12:29 UTC (Fri) by cortana (subscriber, #24596) [Link]

Someone should update the GRUB manual then!

23 Invoking grub-install
************************

The program 'grub-install' generates a GRUB core image using
'grub-mkimage' and installs it on your system.  You must specify the
device name on which you want to install GRUB, like this:

     grub-install INSTALL_DEVICE

   The device name INSTALL_DEVICE is an OS device name or a GRUB device
name.

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 30, 2020 23:38 UTC (Thu) by cesarb (subscriber, #6266) [Link]

This is the official page for RedHat, with the full troubleshooting and recovery steps: https://access.redhat.com/solutions/5272311 (found in the links in the bugzilla page).

Excerpt:

"Red Hat is aware of this bug and is working on a fix.

DO NOT apply the affected Errata RHSA-2020:3217 for RHEL 7.
DO NOT apply the affected Errata RHSA-2020:3216 for RHEL 8.

For Red Hat Satellite exclude this Errata from your Content Views"

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 6:56 UTC (Fri) by madhatter (subscriber, #4665) [Link]

Having spent the past 24 hours trying very hard to glue my mail server back together (from the pile of tiny shards left behind by the update) I can confirm this affects CentOS 7 as well.

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 10:47 UTC (Fri) by purslow (guest, #8716) [Link] (6 responses)

Why not use Lilo instead ? It's simple, reliable & adequate for most installations.
I've never used anything else in 20 years of Linux, including 17 years of Gentoo.

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 12:38 UTC (Fri) by anselm (subscriber, #2796) [Link]

How does LILO cope with things like encrypted disks?

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 14:14 UTC (Fri) by nescafe (subscriber, #45063) [Link]

LILO has never supported UEFI, ELILO is abandonware and generally isn't signed to be secure boot enabled (ditto for gummiboot/systemd-boot). If you want UEFI secure boot, shim + grub2 is the only game in down that works out of the box for the major distros.

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 19:13 UTC (Fri) by ajmacleod (guest, #1729) [Link]

I generally stay as far as possible from UEFI and where I have a choice of bootloader I like syslinux. Grub seemed like (was) a great advance on LILO in many ways but Grub2 is a huge confusing mess.

I wish I'd read this article before updating practically all my grub booting servers (CentOS 7 and Debian 10)! Here's hoping there's a fix that can be applied before any reboots are required...

Grub2 updates for Red Hat systems are making some unbootable

Posted Aug 1, 2020 18:04 UTC (Sat) by xnox (guest, #63320) [Link]

Does it provide TPM2 measurements and signature verification of boot artefacts?

Grub2 updates for Red Hat systems are making some unbootable

Posted Aug 7, 2020 9:50 UTC (Fri) by flussence (guest, #85566) [Link] (1 responses)

There's no reason to use Lilo on a UEFI system; you get exactly the same user interface by using an EFI-stub kernel (with a built-in command line to set rootfs) and the BIOS' boot menu, except there's one less part to forget to update.

Grub2 updates for Red Hat systems are making some unbootable

Posted Sep 14, 2020 17:04 UTC (Mon) by nix (subscriber, #2304) [Link]

Plus you get the ability to use an EFI shell to do all sorts of repair work before the OS even gets started. (Sure, the stuff in an EFI shell is mostly useless *until you need it*, but then it is crucial). There is no such thing as a lilo shell!

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 15:52 UTC (Fri) by jhhaller (guest, #56103) [Link] (2 responses)

Does installation media need to up updated to include newly signed components? Once dbxtool for a distribution has been updated, and the UEFI revocation list rejects older grub2 or shims, one needs an installation media with the newer grub2/shims, or a re-installed system won't boot without disabling secure boot, unless I'm missing something. Are there signs of installation media/ISO being reissued with these patches? It looks like RedHat OpenShift/CoreOS is only being distributed with new boot images, but haven't seen evidence of updated installation media for the rest of RedHat so far. System rescue images are also likely to need updating. It looks like Debian 10.5 will be issued with these patches, so at least it's installation and live media will be available on August 1.

Also check your BIOS vendor, at least HP (not to be confused with HPE) has reported that a BIOS update is required for some of their models before installing the new revocation list.

Grub2 updates for Red Hat systems are making some unbootable

Posted Jul 31, 2020 21:06 UTC (Fri) by amacater (subscriber, #790) [Link]

https://www.debian.org/security/2020-GRUB-UEFI-SecureBoot... is a well written page by someone I trust - debian-cd lead, someone who understand UEFI inside out from his time at ARM and someone who has been instrumental in the last couple of weeks in getting this stuff sorted.

Grub2 updates for Red Hat systems are making some unbootable

Posted Aug 1, 2020 15:08 UTC (Sat) by xnox (guest, #63320) [Link]

That is correct that everyone needs to push out installation & recovery media signed with new keys before pushing out dbx update.

Ubuntu is making 20.04 LTS, 18.04 LTS, 16.04 LTS point releases for that reason. Once they are out dbx update will be pushed out.

Grub2 updates for Red Hat systems are making some unbootable

Posted Aug 3, 2020 19:16 UTC (Mon) by cesarb (subscriber, #6266) [Link]

Now that this has been fixed (https://access.redhat.com/errata/RHBA-2020:3265 and https://access.redhat.com/errata/RHBA-2020:3262), does anyone know what was the bug? I haven't seen any official announcement or blog post explaining what went wrong.

The only thing I have found so far is https://git.centos.org/rpms/shim-unsigned-x64/c/834e5a0c4... ("[PATCH] hexdump.h: fix arithmetic error") which mentions fixing rhbz#1861977 (https://bugzilla.redhat.com/show_bug.cgi?id=1861977).


Copyright © 2020, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds