LVFS tames firmware updates

Keeping device firmware up-to-date can be a challenge for end users. Firmware updates are often important for correct behavior, and they can have security implications as well. The Linux Vendor Firmware Service (LVFS) project is playing an increasing role in making firmware updates more straightforward for both end users and vendors; LVFS just announced its 20-millionth firmware download. Since even a wireless mouse dongle can pose a security threat, the importance of simple, reliable, and easily applied firmware updates is hard to overstate.

Red Hat's Richard Hughes started LVFS in 2016, and in 2019 the Linux Foundation took the project under its wing. Since its inception, LVFS has grown to provide firmware updates for over 2,000 devices from approximately 100 different vendors.

In a 2019 presentation [YouTube] (slides [PDF]), Hughes discussed firmware updates in terms of human, organizational, and technical complexity. As he explained, end users generally don't know what exact hardware they have in their machines, whether its firmware can be updated, if that firmware needs to be updated, where to get that update, or how to apply it. Additionally, users often do not understand the importance of these updates; as Hughes pointed out, "updating your mouse firmware when your mouse is working fine seems ridiculous."

Further, one vendor's hardware often contains updatable components from another vendor, and each only provides firmware for the hardware that it is directly responsible for. Taken together, as Hughes said, "users have no chance of getting this right." LVFS addresses this complexity by providing a centralized repository of firmware and the associated metadata, bringing vendors and end users together. LVFS, however, is more than a centralized firmware distribution site; in the words of Hughes, the LVFS provides "a pipeline right from the firmware author, all the way to the end user."

The project consists of two open-source components: the LVFS web site, which allows vendors to manage their respective firmware (GitHub repository), and the fwupd project (GitHub repository), which provides the command-line fwupdmgr tool and the fwupd daemon to interact with the LVFS web site. A separate project, GNOME Software, provides a user-friendly graphical interface to perform updates; it and fwupd are available by default in most distributions that support GNOME.

At the core of the LVFS project are the binary firmware blobs, provided by vendors as Cabinet files. It is an odd choice for a Linux-focused project, but Hughes explained that it "allows vendors to use the same deliverable for Windows Update as they do for LVFS." Inside the Cabinet file is a cryptographically signed binary blob containing the firmware and metadata in formats consumable by both LVFS and Windows Update. After processing, these Cabinet files are re-signed by LVFS to ensure fwupd clients can verify the integrity of the firmware. LVFS public keys that are used for this verification are distributed as part of fwupd.

Hughes described the LVFS web site, which provides the vendor interface, as "just a Python Flask project." It serves as the central firmware repository and management portal for vendors. Cabinet files uploaded by vendors to the web site are processed and, once validated, added to a master XML document called AppStream, signed by the site. This master list provides the data fwupd needs to notify users of any firmware updates available for their system. For security, Cabinet files are served over SSL by the site directly; the AppStream document, however, is distributed via a content distribution network.

The web site can also be hosted locally, giving organizations privacy and complete control over the update process. For a local instance, the same workflow described above is applied using an organization's own keys, signatures, and firmware packages; the fwupd installations for the organization's machines would then be configured to use the local instance. Hughes said there are a number of undisclosed organizations that use LVFS behind their firewalls for various reasons.

Firmware processing

The processing that is done on firmware provided to LVFS is extensive. In a private email, Hughes explained that the site can process various firmware payload formats into a collection of individual executables and resources. Two types, Pre-EFI Initialization modules (PEIM) and Driver Execution Environments (DXE) are extracted, hashed, and stored in the LVFS database; the project calls them "shards." These shards can be common across firmware when, as Hughes explained, a hardware vendor gives two OEMs the same driver. By collecting and hashing them, they can later be queried against. For example, a query to determine which hardware is the most popular across OEMs by looking for the same hash value in firmware blobs.

That's not the end of the project's firmware analysis, either; Hughes says that DXE shards are analyzed as well. He explained that the project looks for expired certificates, private keys, and other indicators that shouldn't exist in a production firmware like the strings "DO NOT SHIP" or "To Be Filled by O.E.M."

The process of extracting shards is something Hughes says helps "keep vendors honest." Sometimes, vendors will upload firmware described in the release notes as a "low-priority enhancement-only update" that turns out to also include security updates. While there is a limit to the information one can glean from binaries in firmware blobs, enhancement-only updates don't typically need to interact with (for example) the UEFI Secure Boot, explained Hughes; when a discovery like that is made, it allows him to reach out to the vendor to discuss it.

The ability to break firmware blobs down into base, traceable binaries is a significant benefit of the project; especially when it comes to security. For example, LVFS provides a means to block a shard with known security issues, and in his presentation, Hughes showed an example of a rule to block an object known to contain "the Dual EC backdoor for the NSA [National Security Agency]". Hughes said in my correspondence with him that it was "trivial" to query the shard database for firmware that includes the compromised Computrace module. Blocklist rules are defined in the blocklist plugin for the LVFS website. It is also worth noting that security researchers can get a free account on LVFS to issue YARA queries against the available (public) shard data.

As LVFS has gained more support from the industry, the project is imposing increasingly strict restrictions and expectations on vendors in order to participate. In a discussion during Hughes's presentation of firmware signatures, he told the audience that the project had a requirement that "the signature couldn't be more than three years out of date, and half the firmware failed. Over the last few months, we have been increasing the number of checks and making them more and more strict." While the checks can be ignored by vendors, Hughes points out "it's way less work" to do things correctly.

There is information that is only available to vendors on the LVFS web site, including firmware statistics such as installation success and failure rates. These statistics are enabled by another feature Hughes described as "very interesting," a bidirectional feedback mechanism to communicate with users applying updates. According to Hughes, "traditionally when you have a firmware update on an FTP site, you don't get any information back from the end user themselves." fwupdmgr, however, can provide the history of firmware updates made on a system to LVFS. This feedback mechanism can be used by users via the report-history sub-command of fwupdmgr; it is not accessible from GNOME Software. Limiting access to the feature is intentional; Hughes argues that users upgrading via the command line "understand the ramifications of uploading things like your kernel version" to a third party. Hughes has a blog post that details what is transmitted as part of the feedback mechanism.

The feedback not only helps vendors identify who is experiencing a problem with their firmware so they can respond accordingly, but is also valuable to the LVFS. For example, firmware updates that have a high failure rate can be removed from distribution automatically until the issue is fixed.

Working with vendors

While the most common firmware-update protocols are already available in fwupd (39 are currently supported), some vendors use proprietary protocols with implementations that they are unwilling to release under the LGPLv2+, as required by the project. Hughes described vendor secrecy concerns as "an order of magnitude more paranoid" than he experienced working for national defense contractors. Working with those vendors can be a challenge, one that the project is still trying to overcome. In his presentation, Hughes described how Logitech would not share the code for its flashing protocol. Still, the developers were able to answer any questions he had about it. To create the protocol implementation, Hughes engaged in what amounts to a game of Twenty Questions with them. Hughes says that most vendors can be convinced to release their protocols without such games, once it is explained to them that the intellectual property at question has little value; fundamentally, all flashing protocols operate identically.

Wrapping up

LVFS looks to be a useful project and service which is working to push hardware vendors in the right direction. GitHub reports around 120 contributors for the two projects. Of the two, fwupd is the more active project — it has 85 releases and near-daily commits. The LVFS web site only has nine releases with bi-weekly commits. Hughes made it clear that more contributors are welcome; he doesn't have enough time to do all of the things he'd love the project to take on.

As for what is next, Hughes hopes to further build the automated firmware analysis tooling, add more features to the LVFS web site to support vendors, and get more industry players like ASUS and Microsoft to participate. For readers who would like to keep informed on the project, the announcement mailing list is a good way to do so. The project also provides a developer page with information on the process of contributing or opening issues. With more vendors, support for additional update protocols, and so on, LVFS seems poised to strengthen the firmware-update landscape for years to come.