Reducing the bandwidth needs for fwupd

The Linux Vendor Firmware Service (LVFS) provides a repository where vendors can upload firmware updates that can be accessed by the fwupd firmware update daemon on Linux systems. That mechanism allows users to keep the hardware components of their systems up to date with the latest firmware releases, but it has gotten so popular that the daily metadata queries are starting to swamp the LVFS content delivery network (CDN) server. So Richard Hughes, who developed fwupd and LVFS, suggested that it would make sense to start looking at ways to reduce that burden; the idea was discussed in a recent thread on the Fedora devel mailing list.

Hughes had a specific suggestion for an approach to spread the load by adding passim to Fedora 40 as an fwupd dependency and, potentially controversially, enable it by default. Passim will effectively allow systems on the same local network to share a single copy of the LVFS metadata, so that it is only downloaded once for a particular LAN. He wanted to discuss the idea before creating "lots of unnecessary drama". He described passim as follows:

The tl;dr: is I want to add a mDNS [multicast DNS] server that reshares the public firmware update metadata from the LVFS on your LAN. The idea is that rather than 25 users in an office downloading the same ~2MB file from the CDN every day, the first downloads from the CDN and the other 24 download from the first machine. All machines still download the [tiny] jcat file from the CDN still so we know the SHA256 to search for and verify.

It turns out that fwupd has been adopted by other operating systems, "ChromeOS, FreeBSD, Windows and macOS", which means that there is a "need to scale things up a couple of orders of magnitude". He thinks that passim could make a good test case for rolling out similar functionality for the DNF package manager and OSTree/libostree content. One important thing to note is that passim is not handling the actual firmware-update files, at least by default; it is only concerned with the metadata of the LVFS repository, which is roughly 2MB in size and is often downloaded daily.

Some of the concerns that were raised in the thread are, unsurprisingly, security-related. Getting some kind of malicious firmware metadata blob is the last thing anyone wants. But, as Stephen Smoogen put it, passim works on the assumption that "your local network (aka LAN) is a nice and friendly place, without evil people trying to overwhelm your system or feed you fake files", which may not actually be the case. He listed a few possible problem areas, including denial-of-service attacks or stolen signature keys that allow malicious versions of the metadata package to be created and continue to be distributed in the absence of some kind of key-revocation scheme. Hughes acknowledged those possibilities but noted that passim chooses a random source server if there are multiple offerings on the LAN, which may help reduce the scope of the problem if there is a single malicious provider.

The passim web page states that the metadata files are signature protected, and that clients will fall back to retrieving them from the CDN if the signature check fails. Files that are shared from a passim system are configured with a maximum age and a limit on the number of times they will be shared; once those are exceeded, the files are deleted and will no longer be advertised to others on the LAN. Clients will request the metadata file using HTTPS; local passim servers will use a self-signed TLS certificate for use in encrypting the connection.

As might be guessed, the use of a self-signed certificate was questioned; Leon Fauster asked what the benefit was since "it does not provide any 'security'". Hughes replied that it provides encryption so that other systems on the LAN cannot snoop on the traffic ("so client A can provide the file to client B without client C being aware what's being sent"). But, as Chris Adams pointed out, a self-signed certificate does not protect against a "man in the middle":

[...] there's no way for client B to know it is really talking to client A - it could be talking to client C with a man-in-the-middle attack and a different self-signed cert pretending to be client A.

It is not entirely clear what would be gained by snooping on the traffic, beyond knowing that a client is requesting the metadata, which might be inferred without actually snooping. Doing meaningful identification of LAN systems may be difficult as well. But the TLS protection is also meant to placate corporate security teams that would be unhappy with regular HTTP, "even if it's got two other layers of protection", Hughes said.

Simo Sorce had several suggestions, agreeing with Adams that the self-signed-certificate approach was only useful against passive, rather than active, attacks. "Trust on first use" (or TOFU) could be used on the clients to minimize the "window of impersonation", but there are still problems as it "requires clients to cache an association and then has weird failure modes to be dealt with if one of the actors get re-imaged or changes the cert for any reason". Overall, he did not think that using TLS added any real value; security teams that require it could provide their own certificates. Another option might be to use a pre-shared key, though Sorce did not really recommend it.

Using TOFU had some appeal, Hughes said, and asked for opinions on that. Sorce wondered how passim would deal with the new-certificate problem and asked about certificate expiration. Hughes's reply that currently the certificates effectively never expire (9999-year expiration) was not particularly well-received ("Ugh."). Meanwhile, certificate updates would cause that system to be ignored, Hughes said; that would eventually lead to all devices ignoring each other, Sorce pointed out.

Having a single host that was responsible for gathering the metadata file periodically and then serving it to the rest of the network was suggested by Peter Robinson. Hughes seemed amenable to that idea and asked Robinson to file a GitHub issue in order to track it. As the issue notes, some system administrators want to control where services run; there may also be a bastion host that is the only externally connected server, so it would be useful to be able to pin passim downloads to a particular host.

Enabling passim by default was questioned; Benson Muite said that it would be better to make passim opt-in, especially "for computers in an institutional setting where the LAN is well controlled". But Hughes replied that requiring people to enable it will simply lead to almost total lack of adoption and, thus, no real reduction in the network traffic. Marcus Müller agreed that opt-in was not really plausible, especially since it would need to enabled on a host-by-host basis.

He also asked about the plans for sharing firmware files in addition to the metadata; there are privacy concerns with advertising the firmware files that a system has downloaded. Hughes agreed, but firmware files are not included in phase one of his plan; if they are to be added at some point, there are some things to consider:

It's non-free software (which you have permission to redistribute, but still unpalatable for many) -- the compromise I've done for people changing the default to "metadata,firmware" is that you need to reboot into the new firmware before the published firmware gets shared; on the logic that you don't want to advertise to the world that you're currently running insecure firmware.

Privacy also came up in the context of Vit Ondruch's questions about the need to download all of the metadata daily. A given system will only have a handful of devices that get updated firmware, which should require far less than 2MB of metadata, he said. But Hughes replied that the whole database is always delivered from LVFS:

It is the metadata for every device -- every fwupd client deliberately gets the entire catalog rather than making a bespoke request like Windows update. This ensures that the LVFS doesn't know what hardware you have on your computer, and couldn't provide that kind of data even if compelled to by law enforcement. The entire architecture is privacy centric, and also allows it to scale to millions of devices without having thousands of servers.

There are, obviously, some problems that need to be addressed, but passim seems like a reasonable solution, at least for the metadata. The problem of scaling LVFS is real, so something will likely need to be done before too long. Those measures (whether passim-based or not) will need to be adopted widely, though, in order to make a serious reduction in the traffic burden. Fedora 40 is scheduled for April 2024, so even if the feature gets approved—Hughes is planning to proceed in making a change proposal—it will likely not make much of a dent until after that. A real dent in the traffic levels will require adoption in most or all of those other distributions or OSes that are now using fwupd.