Removing SHA-1 for signatures in Fedora

Disruptive changes are not much fun for anyone involved, though they may be necessary at times. Moving away from the SHA-1 hash function, at least for cryptographic purposes, is probably one of those necessary disruptive changes. There are better alternatives to SHA-1, which has been "broken" from a cryptographic perspective for quite some time now, and most of the software components that make up a distribution can be convinced to use other hash functions. But there are still numerous hurdles to overcome in making that kind of a switch as a recent discussion on the Fedora devel mailing list shows.

How to?

On March 8, Alexander Sosedkin posted a lengthy message asking for help in "planning a disruptive change" to switch Fedora away from SHA-1. Currently, SHA-1 is disabled for TLS, he said, but SHA-1 use is more widespread than that; cryptographic signatures for packages may use SHA-1 and other libraries, such as OpenSSL, are still using the older hash algorithm in some cases. But Fedora does not need to be first on doing the switch:

Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.

He does not think this kind of change can be done in a single Fedora release cycle, in part because finding and fixing all of the uses of SHA-1 is going to require breaking things: "the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration." But because Fedora has fairly short release cycles (six months or so between releases), even doing that will not give developers enough time to fix the problems—if they are even brought to light.

Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate.

He suggested several possible approaches, but did not think that simply announcing the change would be enough to cause all of the changes needed, likening it to the bypass announcements in the The Hitchhiker's Guide to the Galaxy. Breaking the creation and use of SHA-1 signatures in, say Fedora 37 Rawhide, backing it out for the release, then doing so again for Fedora 38 (without the back-out), might be one way. Another might be to break it in Rawhide and leave it that way, but unbreak it for the first release. "But these are all rather... crude?" He would rather find a smoother process, preferably one that had been used before.

Breaking things

Zbigniew Jędrzejewski-Szmek was not in favor of breaking things in order to flush out the problem areas. "We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose." He also noted that SHA-1 is perfectly reasonable to use in non-cryptographic settings. Beyond that, existing signatures that use SHA-1 need to be verifiable "essentially forever" and signatures using it will be provided going forward—Fedora cannot dictate signature policies to others—so being able to check them will always be important. The situation is similar to self-signed TLS certificates, where disallowing them completely "will only result in users jumping to different tools".

The focus of the effort is only on blocking cryptographic signatures using SHA-1, Simo Sorce said; that means "Certificates, TLS session setup, DNSSEC (although a lot of signatures are still SHA-1 based there ...), VPNs session establishment, PGP, etc..." While he did recognize that verifying SHA-1 signatures would still need to be supported by Fedora, he said that by default it should be blocked. Verifying signatures on older content should be reserved for local data, not things coming over the network:

This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.

Neal Gompa was concerned that "breaking" SHA-1 verification might also "break Fedora's *own* GPG keys and the GPG keys of preferred third party repositories". He encountered a problem of that sort when a similar change was made for CentOS, but Demi Marie Obenour said that all of the official RPM packages back to at least Fedora 25 have SHA-256 signatures. Gompa listed several third-party repositories, RPM Fusion, Google, Copr, and Microsoft, that might be affected; it turns out that Fedora's Copr repository was affected by the CentOS 9 change that Gompa mentioned, as reported more fully by Gary Buhrmaster. There is a bug report tracking the switch to use SHA-256 signatures for Copr going forward.

Sosedkin disagreed with Jędrzejewski-Szmek's idea that warnings would suffice; "Cryptographic libraries aren't in position to do any meaningful logging, and even if they did, it'd be gleefully ignored until we break things." He was skeptical that there was any other way to make this kind of change, though backward compatibility is important: "Yes, you can still make a modern Fedora do RC4 or DES, with just a little bit of extra configuration. No, we must phase things out of defaults, there's no way around this."

The configuration he is referring to is to change the setting in /etc/crypto-policies/config and then run update-crypto-policies as described in a feature added for Fedora 21. There are three pre-defined settings available: LEGACY, DEFAULT, and FUTURE. Those will reconfigure various libraries to different levels of cryptographic strength. Since the policy option was added, support for more libraries has been added, and the policies themselves updated for Fedora 28 and for Fedora 33.

RHEL 9

Daniel P. Berrangé asked what has been learned from the under-development RHEL 9, which will help lead the way for Fedora. He asked about the number and percentage of RHEL 9 packages that were affected by the switch in order to get a better idea what the impact will be for Fedora. "Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora." He suggested that perhaps there might only be "a few important remaining packages" that need to be evaluated, leaving the others to potentially fail, given that there is the LEGACY fallback.

Sosedkin seemed skeptical of that approach. He noted that RHEL 9 is not yet widely available, so the number of broken packages for it is still unknown. Furthermore, RHEL has roughly 10% of the packages that Fedora carries, so "leaving the 90% of the packages you've labelled 'less important' to be 'fixed after the fact' is gonna be a disaster." He reiterated that he did not think a single cycle would be sufficient time to handle the problems.

While the 90% of Fedora packages will not have been tested as part of RHEL 9, Berrangé said, that does not mean they are likely to be affected by this change. "Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library." He thought it possible that only a few of those packages will be using cryptographic signatures with SHA-1, and even less "will be considered important enough to be blockers for this change".

An explicit example of the kinds of problems that will be encountered was pointed out by Alexander Bokovoy. The Kerberos network-authentication protocol requires the use of SHA-1 "due to interoperability issues and also due to compliance with RFCs". He noted a March 4 bugzilla entry that reported the problem; the "fix" for RHEL 9 will be to override any restrictions on SHA-1 in OpenSSL for Kerberos, while other options "will be discussed upstream". Fedora will pick up that fix, of course, but it does show that these kinds of problems will crop up unexpectedly.

Warnings

Berrangé also brought up the idea of using "a scary warning message" on stderr or in the system logs. That could be turned on for a release cycle to see which bugs get reported. Sosedkin said that warnings to the log files would result in zero bugs being filed, while stderr reports would simply result in bugs about "'$CRYPTOLIB has no business messing up my stderr/stdout', which we'll promptly close by reverting the changes".

Fedora project leader Matthew Miller agreed with the thought, though it was "a little more cynically-phrased than I'd put it". He suggested logging the problem and having a tool that specifically looked for those log messages, "we could do something like a Test Day where people could send in reports". While Sosedkin still thought that logging from cryptographic libraries was "risky territory", it was a better approach than logging to stderr; "Especially if it accompanies a drawn-out multicycle change, it could be a noticeable impact dampener."

The problem with logging, however, is that it may not be available to the library, depending on the execution environment of the program, Berrangé said:

Security policies applied to [processes] (SELinux, seccomp, namespaces or containerization in general) can easily prevent ability to use journald, syslog, or opening of log files, leading to messages not being visible. At worst, with seccomp an attempt to use the blocked feature may lead to an abort of the application.

Sosedkin was concerned that applications might be reopening the stderr file descriptor in strange ways, resulting in "writing our scary messages who knows where", but Berrangé said that is fairly unlikely precisely because various libraries already print warnings and errors to stderr when needed. "Even glibc will print to stderr when it sees malloc corruption, and stuff in stderr will end up in the systemd journal for any system services."

SSH?

Older SSH protocols use SHA-1, so Richard W.M. Jones wondered if the change would cause problems using SSH. "This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything)." He also noted that he has some "ancient network equipment" that requires the LEGACY setting on Fedora in order to be able to connect to it. He asked if the changes Sosedkin wants to make will further impact SSH.

SSH uses SHA-1 as a hash-based message authentication code (HMAC), Sosedkin said, "which doesn't rely on collision resistance", so no change needs to be made. Obenour agreed that using SHA-1 as an HMAC is reasonable, though there are alternatives that perform better; "There are no known attacks on HMAC-SHA-1".

Goals

Along the way, Sosedkin listed his objectives in raising the issue:

I want to know, ultimately, how can I break it for devs but not the users so that

1. the need for it is communicated early,
2. all the relevant places are timely identified,
3. maintainers have a plenty of time to resolve it for good or opt out,
4. and the users are affected as late and as smoothly as possible.

Hope these are all reasonable things to wish for.

As yet, it does not seem that there is a clear plan that would meet those goals. It is a difficult problem, but one that seems likely to recur over time. Cryptographic algorithms and protocols change over time, generally because weaknesses are found in them; once they become entrenched in a distribution, and the internet itself, they are hard to excise. Finding a way to remove SHA-1 smoothly—as smoothly as possible, at least—will be a useful thing to figure out. It seems likely that more discussion will be required to get there.