Fedora considers deprecating legacy BIOS

A proposal to "deprecate" support for BIOS-only systems for Fedora, by no longer supporting new installations on those systems, led to a predictably long discussion on the Fedora devel mailing list. There are, it seems, quite a few users who still have BIOS-based systems; many do not want to have to switch away from Fedora simply to keep their systems up to date. But, sometime in the future, getting rid of BIOS support seems inevitable since the burden on those maintaining the tools for installing and booting those systems is non-trivial and likely to grow over time. To head that off, a special interest group (SIG) may form to help keep BIOS support alive until it really is no longer needed.

Proposal

The proposal to "Deprecate Legacy BIOS" was, as usual, posted on behalf of its owners, Robbie Harwood, Jiří Konečný, and Brian C. Lane, by Fedora program manager Ben Cotton. It currently targets Fedora 37, which is due in October, though there is reason to believe the change will not happen quite that soon. The reasons for removing the support are described in the proposal:

UEFI is defined by a versioned standard that can be tested and certified against. By contrast, every legacy BIOS is unique. Legacy BIOS is widely considered deprecated (Intel, AMD, Microsoft, Apple) and on its way out. As it ages, maintainability has decreased, and the status quo of maintaining both stacks in perpetuity is not viable for those currently doing that work.

[...] While this will eventually reduce workload for boot/installation components (grub2 reduces surface area, syslinux goes away entirely, anaconda reduces surface area), the reduction in support burden extends much further into the stack - for instance, VESA support can be removed from the distro.

The proposal says that, eventually, BIOS support will have to be removed, but that maintaining the ability to boot existing Fedora systems with BIOS is meant to help smooth the process. Fedora already has requirements that effectively restrict it to systems made after 2006, so this change would extend those restrictions somewhat further.

Intel stopped shipping the last vestiges of BIOS support in 2020 (as have other vendors, and Apple and Microsoft), so this is clearly the way things are heading - and therefore aligns with Fedora's "First" objective.

The subject was raised back in 2020, which also led to a (predictably) long thread, though it was not a change proposal at that time. Some of the "relevant points from that thread" are listed in the Feedback section of the proposal. There are, of course, machines that are BIOS-only and any kind of hardware deprecation for the distribution is impossible "without causing some amount of friction". In addition, there is no way to migrate from a BIOS installation to a UEFI-based system, since "repartitioning effectively mandates a reinstall". In particular, a UEFI partition would need to be added to the system.

The Contingency Plan section of the proposal describes an ugly future for the status quo, but also a possible way forward:

Leave things as they are. Code continues to rot. Community assistance is required to continue the status quo. Current owners plan to orphan some packages regardless of whether the proposal is accepted.

Another fallback option could be, if a Legacy BIOS SIG organizes, to donate the relevant packages there and provide some initial mentoring. Longer term, packages that cannot be wholly donated could be split, though it is unclear whether the synchronization thereby required would reduce the work for anyone.

Affected systems

As might be guessed, multiple replies from users of affected systems were seen in the thread, starting with Neal Gompa. He said that he is sympathetic to the change, but thinks that it is "way too early to do across the board". He has a system that struggles to boot Linux with UEFI and his workarounds are beyond the abilities of most users.

David Airlie said that he recently worked on a project to rewrite the Mesa driver for a wide range of Intel GPUs. Many of the systems he used to develop and validate those drivers are pre-UEFI systems, but it was "of great benefit to me and the community" that he could use Fedora for that work. For future projects, he would have to consider moving away from Fedora, which would be a sad state of affairs.

Hans de Goede said that he had several systems at home that were BIOS-only and happily running Fedora now. Obsoleting them just contributes to the e-waste problem; "Looking specifically at fixed PCs and not laptops this proposal would (eventually) turn 2/5 PCs in my home unusable, which really is unacceptable IMHO." He also pointed to Airlie's work on Intel GPUs, saying that "it seems rather silly to drop support for this hw after just investing a significant chunk of time to breathe new [life] into their GPU support"

But it is not just desktops and laptops that are affected by a change of this sort. Fedora is also installed on cloud servers and virtual machines of various sorts, some of which do not support anything other than booting via BIOS. The proposal noted that the time of the 2020 discussion, Amazon's AWS did not support UEFI, but that has changed. Marc Pervaz Boocha pointed out that many virtual private server (VPS) providers do not support UEFI, giving Linode and Vultr as examples. Dominik "Rathann" Mierzejewski reported that OVH is also affected:

OVH is another big provider and they don't offer UEFI boot with their VPS range. I've just confirmed it with their support.

Since one of my servers is hosted by OVH, I'd have to either migrate to another hosting provider or migrate off Fedora. Which is ironic, considering my involvement in Fedora.

Stewart Smith added some "thoughts both from an EC2 [Elastic Compute Cloud] perspective, and an Amazon Linux as a downstream of Fedora perspective". Most EC2 instance types boot using BIOS by default, though many can use UEFI and new types are likely to get UEFI support, but there are "a *lot* of instance types, a whole bunch of which are less likely to support UEFI". There is no installer for cloud images, instead they use the Amazon Machine Image (AMI) format, but "AMIs that don't run on all instance types tend to cause confusion, no matter how [clearly] you document the limitations". So Amazon Linux has an interest in ensuring that BIOS booting still works well, "likely for a decent number of years to come (however much I wish this wasn't the case)".

Gompa complained that the change is not really a deprecation, but is instead a removal, because the lack of packages and tooling that support BIOS "makes several scenarios (including recovery) harder". It also puts the burden on users to determine if their hardware can boot and install Fedora, but UEFI has not yet reached a critical mass where it can be assumed to work. Alberto Abrao agreed, noting that this change would "leave behind a LOT of serviceable hardware", especially in the server space:

Ironically, Fedora is one of the distributions out there that allows me to extract the most out of older hardware. It would be a terrible loss to have to move to a different one, but it's hard to reason purchasing new hardware - especially right now, with pandemic-related supply issues still ongoing - to keep up with this change.

In that message, Gompa also said that he is a "a fan of using UEFI instead of BIOS" and that he has done work to add UEFI support to Fedora cloud images. He just does not believe that it is time, yet, to make that switch. Part of the reason is that he believes the UEFI experience on Fedora is not all that good.

A wander into secure boot

In his original reply to the proposal, Gompa also asked about Fedora support for NVIDIA drivers under UEFI secure boot. Peter Robinson said that was out of the scope of the proposal, since users can disable secure boot if they need support for drivers, like NVIDIA's, that are not signed by the Fedora kernel-module keys. But Gompa replied that it is sometimes easier to have users fall back to booting from BIOS; furthermore:

You're right that these are different problems, but I've also seen very little appetite for reducing the suffering of Fedora Linux users on UEFI Secure Boot with the *most common issue* we have: an NVIDIA driver that doesn't do anything because of the lockdown feature. If you're planning to say that UEFI is the only way to boot, then that means you need to be prepared to accept that our UEFI experience is *worse* than our BIOS one right now, and someone needs to take ownership to improve it.

Harwood, who is one of the feature owners, said that NVIDIA users can either use the open-source nouveau driver (which is signed), sign their own copy of the driver "(involves messing with certificates, so not appropriate for all users)", or disable secure boot. But several people pointed out that nouveau does not really solve the problem, especially for more recent NVIDIA hardware; it does not provide access to accelerated graphics operations for recent GPUs and tends to have quite a few bugs even for the older models. Michael Catanzaro pointed out that the options Harwood described are problematic from a user-experience perspective:

The user experience requirement is: user searches for NVIDIA in GNOME Software and clicks Install. No further action should be necessary. We didn't make the NVIDIA driver available from the graphical installer with the intention that arcane workarounds would be required to use it.

But Adam Jackson thought that the NVIDIA problem was not Fedora's to solve. There are technical means to make it work and "NVIDIA are the ones with the private source code". But Chris Murphy sees things differently, noting that Fedora is sending mixed messages:

When users have a suboptimal experience by default, it makes Fedora look bad. We can't have security concerns overriding all other concerns. But it's really pernicious to simultaneously say security is important, but we're also not going to sign proprietary drivers. This highly incentivizes the user to disable Secure Boot because that's so much easier than users signing kernel modules and enrolling keys with the firmware, and therefore makes the user *less safe*.

On the other hand, Fedora does not actually have a full secure boot implementation, Lennart Poettering said, so by disabling it "you effectively lose exactly nothing in terms of security right now". The reason is that the initrd is not signed, so it can be subverted:

What good is a trusted boot loader or kernel if it then goes on loading an initrd that is not authenticated, super easy to modify (I mean, seriously, any idiot script kiddie can unpack a cpio, add some shell script and pack it up again, replacing the original one) – and it's the component that actually reads your FDE LUKS password.

SIG

In his message linked above, De Goede volunteered to form a SIG along the lines suggested in the proposal. He noted that previous attempts to keep 32-bit x86 alive when Fedora removed support for i686 had run aground, but felt that BIOS is different:

Legacy BIOS boot support is basically only about the image-creation tools + the bootloader. As various people have mentioned in the thread BIOS support is still very much a thing in data-centers, so I expect the upstream kernel community to keep the kernel working with this for at least a couple of years. Where as both the kernel + many userspace apps were breaking on i686.

After Fedora project leader Matthew Miller started a new thread about the possibility of having a video call to discuss the BIOS issue, which was generally seen as not being something that would do much more than rehash what had already been aired, De Goede renewed his call for a Legacy BIOS SIG. He envisioned it as a lightweight organization that focused on testing Fedora on BIOS systems, particularly the next version of Fedora early in its development, in order to file and fix bugs.

The new thread largely went over the same ground as the first, at some length, naturally, though there was also some concrete planning of what a new SIG would do—and how. Harwood said that what is needed is a place to assign bugs and a way to get those problems addressed. "The overall goal of the SIG needs to be to reduce load on existing bootloader contributors." But Harwood also wondered whether Fedora should redefine its support for BIOS:

Given there is consensus that legacy BIOS is on its way out, we think Fedora release criteria in this area should be re-evaluated. Not only does support change from "fully supported" to "best effort", but we should re-evaluate what is/isn't release blocking, and probably clarify who owns what parts.

Several people disagreed with that characterization of the status of BIOS. Chris Adams said: "I don't think this statement is true, unless Fedora doesn't want to be considered for a bunch of popular VM hosts (e.g. Linode and such) that have no stated plans to support UEFI." Perhaps BIOS support for physical hardware could be phased out, though. De Goede agreed that BIOS support is still needed in some contexts:

Given what the server product folks have indicated that BIOS boot support is quite important for them I'm not sure if changing the release criteria is in order. I do agree that any blocker bugs related to legacy BIOS booting should be assigned to; and taken care of by the legacy BIOS boot SIG.

The change proposal will be discussed and decided by the Fedora engineering steering committee (FESCo), but its prospects do not look good. The FESCo ticket for the proposal has already gotten five "-1" votes from members and there are nine on the committee. Those votes are not binding, but it still looks pretty unlikely this change will go through—at least for Fedora 37.

But the change will, seemingly, happen eventually. One of the complaints seen in the ticket (and threads) is that the proposal calls it a deprecation, but that is not really what it entails; new versions of Fedora will not be able to be installed on the older hardware, which could be problematic if an in-place upgrade goes awry. In addition, Fedora versions are only supported for about a year, so at some point upgrading may not really be an option if BIOS support is removed. That means users of those systems would have to migrate to a different distribution or keep running an unsupported version as it slowly bitrots. In a blog post about the change, Cotton pointed out that while he did not think the distribution "should abandon old hardware willy-nilly", there is a balance to be struck:

I think some distros should strive to provide indefinite support for older hardware. I don't think all distros need to. In particular, Fedora does not need to. That's not what Fedora is. "First" is one of our Four Foundations for a reason. Other distros focus on long-term support and less on integrating the latest from upstreams. That's good. We want different distros to focus on different benefits.

With luck, the SIG will take over any packages needed to keep BIOS functioning, since their owners plan to orphan them regardless of the outcome of the proposal. Keeping BIOS alive on Fedora for another few years—how many is difficult to guess—would seem to have a fair number of benefits at this point, though there is obviously a maintenance burden associated with it as well. If the change does not get approved this time around, we will likely see the proposal recur for Fedora 38 (or later), but if the SIG takes off, that may be postponed for some time. When it does come up again, however, we can probably expect another lengthy discussion in Fedora-land.