Fedora and secure release upgrades

As Fedora 18 nears its final release, some project members are concerned that churn in the update tools has placed users in a risky position by eliminating secure paths to update an existing installation from Fedora 17. Although it looks like workarounds will be available, the resulting choice will pit convenience against security, which is rarely a trade-off that ends up in security's favor. But as the whole debate nicely illustrates, when it comes to verifying the authenticity of software acquired over the network, no matter how many steps are involved, eventually the chain of trust must start somewhere — because users always make a leap of trust when first clicking on the download option.

The root of the issue is FedUp, the brand-new updater currently being developed for deployment with F18, which does not check the cryptographic signatures of the RPM packages it downloads over the network (although FedUp also downloads initramfs and vmlinuz files at the start of an upgrade; it does verify the integrity of these by checking their SHA256 checksums). A bug highlighting that deficiency was filed in November, noting that fetching and installing unverified binaries over the network leaves the system vulnerable to man-in-the-middle attacks and to trojaned packages on compromised mirrors. Fedora's RPM packages are signed, and various package installer tools do verify the signatures; it is only the FedUp release-upgrading tool that does not check the signatures before installing them.

The Fedora Engineering Steering Committee (FESCo) discussed the subject in its December 5 meeting and decided that the absence of network verification would not block the release of F18. Rahul Sundaram raised the importance of the issue on the fedora-devel list a week and a half later, arguing that the lack of a secure method to update from F17 to F18 was irresponsible. Part of Sundaram's initial concern was that the still-unfinished FedUp did not yet support updating from an optical disc, which would provide some measure of protection against attack because Fedora's downloadable ISO images are provided with checksums. That concern turned out to be partly unfounded; as Red Hat's Tim Flink explained in a different thread, FedUp does support extracting packages from optical media — it just does not support booting from an optical disc to upgrade an existing installation, or pulling packages from an ISO file.

Upgrades and release blockers

But the optical disc upgrade option is not the default behavior for FedUp; the user must know to employ it with the --device switch. FedUp's default is still to fetch packages from a remote repository. At the moment, FedUp is also fetching these pre-release packages over an unencrypted HTTP connection, although that could change when the final F18 is released. One can always argue that users in the know will seek out the secure means to update their machines, but it is far less risky to make the default behavior secure. After all, "users in the know" are not the ones most in need of protection. On that point, Sundaram noted in his original email that FedUp's lack of package verification is not a regression, since the update tool that it replaces, PreUpgrade, did not check package signatures either. Neither does Anaconda, the Fedora installer.

The only release-upgrade path that ever has verified remote repository package signatures was that of upgrading via the yum package manager, which is an unsupported option that non-advanced users are steered away from — and with good reason. Yum is a lower-level package installation utility; it can be used to manually upgrade the entire system to a new release, but the procedure is not recommended because a release upgrade involves more than simple package installation. Dedicated upgrade tools like PreUpgrade and FedUp are designed to handle large or complicated system changes between releases, such as the merging of /bin and other directories into the /usr hierarchy. Some of these changes require a reboot, which PreUpgrade and FedUp are built to manage, but yum is not.

Sundaram opened a FESCo ticket asking FESCo to classify the FedUp signature-verification issue as a blocker — although he later closed it after determining that the local-media option does offer a secure upgrade path for those that seek it out. But the issue is also tangled up with the question of whether or not FedUp is ready for release on other grounds. It still lacks a GUI front-end, logging, and progress indication. The GUI in particular is a limitation that some see as critical in its own right (Sundaram called it "a severe regression").

The fedora of trust

FESCo decided to defer the FedUp GUI feature until F19, and the question of providing any secure way to upgrade an F17 machine to F18 appears, for the moment, to be settled. The deeper question of making such a secure upgrade the default, however, remains unanswered.

Fedora's distribution upgrade tools have never baked signature verification into the upgrade process, and the project knows it. The discussion dates all the way back to the infamous bug 998, first opened in 1999 and filled with mysterious references to forgotten technologies like floppy disks (hint for younger readers: imagine a really thin SSD). But securely checking package signatures requires a secure method for the user to acquire the right public key (with which to check said signatures) in the first place. Even yum, which was raised during the FedUp discussion as the secure upgrade option, needs the user to manually find and import the correct GPG key for the new release. That is a weak link even in the manual case, as Fedora QA team member Adam Williamson observed:

Finding a way for the Fedora N release upgrade tool to securely fetch and install the correct key for release N+1 automatically is a tricky problem indeed. For example, Will Woods proposed a plan to add GPG signatures for the initramfs and vmlinuz files FedUp fetches at the beginning of the upgrade. As mentioned above, currently only a SHA256 checksum is used to verify the integrity of these files — courtesy of checksums found in the repository's .treeinfo file. Woods proposed adding GPG signatures to the repository at .treeinfo.signed, but noted that there still needs to be a way to get the F18 key onto F17 systems, and some way for the user to decide if the F18 key is trustworthy.

Establishing a chain of trust can be a bit of a chicken-and-egg problem: eventually one has to place one's trust in something on which the subsequent links build. Till Maas suggested to Woods that the F18 key could be securely fetched from the Fedora project from a well-known location over HTTPS. Of course, that relies on the security of HTTPS and the DNS system, which relies on the security of the certificate authorities, and so on.

For most mortals, however, some security — however limited — is still better than no security, which is what FedUp's default usage provides now. As recently as the end of November, users were still writing to the list confused about how best to upgrade their machines. There are clearly many more discussions ahead; even if Fedora establishes a workable plan for its main releases, also problematic is how downstream distributions (including Fedora "spins") and customized ISOs are handled. Whatever solution develops, one thing is for certain: it will not arrive until the F19 release cycle at the earliest.