Removing SHA-1 for signatures in Fedora
Disruptive changes are not much fun for anyone involved, though they may be necessary at times. Moving away from the SHA-1 hash function, at least for cryptographic purposes, is probably one of those necessary disruptive changes. There are better alternatives to SHA-1, which has been "broken" from a cryptographic perspective for quite some time now, and most of the software components that make up a distribution can be convinced to use other hash functions. But there are still numerous hurdles to overcome in making that kind of a switch as a recent discussion on the Fedora devel mailing list shows.
How to?
On March 8, Alexander Sosedkin posted
a lengthy message asking for help in "planning a disruptive
change
" to switch Fedora away from SHA-1. Currently, SHA-1 is
disabled for TLS, he said, but SHA-1 use is more widespread than that;
cryptographic signatures for packages may use SHA-1 and other libraries,
such as OpenSSL, are still using the older hash algorithm in some cases. But Fedora does
not need to be first on doing the switch:
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
He does not think this kind of change can be done in a single Fedora
release cycle, in part because finding and fixing all of the uses of SHA-1
is going to require breaking things: "the only realistic way to weed
out its reliance on SHA-1 signatures
from all of its numerous dark corners is to break them.
Make creation and verification fail in default configuration.
" But
because Fedora has fairly short release cycles (six months or so between
releases), even doing that will not give developers enough time to fix the
problems—if they are even brought to light.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate.
He suggested several possible approaches, but did not think that simply
announcing the change would be enough to cause all of the changes needed,
likening it to the bypass announcements
in the The Hitchhiker's Guide to the Galaxy. Breaking the creation
and use of SHA-1 signatures in, say Fedora 37 Rawhide, backing it out for
the release, then doing so again for Fedora 38 (without the back-out),
might be one way. Another might be to break it in Rawhide and leave it
that way, but unbreak it for the first release. "But these are all
rather... crude?
" He would rather find a smoother process,
preferably one that had been used before.
Breaking things
Zbigniew Jędrzejewski-Szmek was not in favor of
breaking things in order to flush out the problem areas. "We should make newer hashes
the default, and we can make tools print a warning if sha1 is used
where it shouldn't, but please don't break things on purpose.
" He
also noted that SHA-1 is perfectly reasonable to use in non-cryptographic
settings. Beyond that, existing signatures that use SHA-1 need to be
verifiable "essentially forever
" and signatures using it will be
provided going forward—Fedora cannot dictate signature policies to others—so being
able to check them will always be important. The situation is similar to
self-signed TLS certificates, where disallowing them completely "will only result in users
jumping to different tools
".
The focus of the effort is only on blocking cryptographic signatures using SHA-1,
Simo Sorce said;
that means "Certificates, TLS session setup,
DNSSEC (although a lot of signatures are still SHA-1 based there ...),
VPNs session establishment, PGP, etc...
" While he did recognize
that verifying SHA-1 signatures would still need to be supported by Fedora,
he said that by default it should be blocked. Verifying signatures on
older content should be reserved for local data, not things coming over the
network:
This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.
Neal Gompa was concerned
that "breaking" SHA-1 verification might also "break Fedora's *own* GPG
keys and the GPG keys of preferred third party repositories
". He
encountered a problem of that sort when a similar change was made for
CentOS, but Demi Marie Obenour said
that all of the official RPM packages back to at least Fedora 25 have
SHA-256 signatures.
Gompa listed
several third-party repositories,
RPM Fusion, Google, Copr, and Microsoft, that might be affected; it turns
out that Fedora's Copr repository was affected by the CentOS 9 change
that Gompa mentioned, as reported
more fully by Gary Buhrmaster. There is a bug report tracking the
switch to use SHA-256 signatures for Copr going forward.
Sosedkin disagreed
with Jędrzejewski-Szmek's idea that warnings would suffice; "Cryptographic libraries
aren't in position to do any meaningful logging,
and even if they did, it'd be gleefully ignored until we break
things.
" He was skeptical that there was any other way to make this
kind of change, though backward compatibility is important: "Yes, you
can still make a modern Fedora do RC4 or DES,
with just a little bit of extra configuration.
No, we must phase things out of defaults, there's no way around this.
"
The configuration he is referring to is to change the setting in /etc/crypto-policies/config and then run update-crypto-policies as described in a feature added for Fedora 21. There are three pre-defined settings available: LEGACY, DEFAULT, and FUTURE. Those will reconfigure various libraries to different levels of cryptographic strength. Since the policy option was added, support for more libraries has been added, and the policies themselves updated for Fedora 28 and for Fedora 33.
RHEL 9
Daniel P. Berrangé asked what has
been learned from the under-development RHEL 9, which will help lead the
way for Fedora. He asked about the number and percentage of RHEL 9 packages that were
affected by the switch in order to get a better idea what the impact will
be for Fedora. "Assuming RHEL-9 has dealt with the problems, Fedora should
inherit those fixes, which gives us a good base for the most
commonly used / important packages in Fedora.
" He suggested that
perhaps there might only be "a few important remaining
packages
" that need to be evaluated, leaving the others to
potentially fail, given that there is the LEGACY fallback.
Sosedkin seemed
skeptical of that approach. He noted that RHEL 9 is not yet
widely available, so the number of broken packages for it is still unknown.
Furthermore, RHEL has roughly 10% of the packages that Fedora carries, so
"leaving the 90% of the packages you've labelled 'less important'
to be 'fixed after the fact' is gonna be a disaster.
" He reiterated
that he did not think a single cycle would be sufficient time to handle the problems.
While the 90% of Fedora packages will not have been tested as part of
RHEL 9, Berrangé said, that does
not mean they are likely to be affected by this change. "Only a
relatively small subset will do anything
crypto related, and most of that just be HTTPS and completely
outsourced to a crypto library.
" He thought it possible that only a
few of those packages will be using cryptographic signatures with SHA-1,
and even less "will be considered
important enough to be blockers for this change
".
An explicit example of the kinds of problems that will be encountered was
pointed out by
Alexander Bokovoy. The Kerberos
network-authentication protocol requires the use of SHA-1 "due to
interoperability issues and also due to compliance with RFCs
". He
noted a March 4
bugzilla entry that reported the problem; the "fix" for RHEL 9 will
be to override any restrictions on SHA-1 in OpenSSL for Kerberos, while
other options "will be discussed upstream
". Fedora will pick
up that fix, of course, but it does show that these kinds of problems will
crop up unexpectedly.
Warnings
Berrangé also brought up the idea of using "a scary warning message
"
on stderr or in the system logs. That could be turned on for a
release cycle to see which bugs get reported. Sosedkin said
that warnings to the log files would result in zero bugs being filed, while
stderr reports would simply result in bugs about
"'$CRYPTOLIB has no business messing up my stderr/stdout',
which we'll promptly close by reverting the changes
".
Fedora project leader Matthew Miller agreed with
the thought, though it was "a little more cynically-phrased than I'd
put it
". He suggested logging the problem and having a tool
that specifically looked for those log messages, "we could do something like a Test
Day where people could send in reports
". While Sosedkin still thought
that logging from cryptographic libraries was "risky
territory
", it was a better approach than logging to
stderr; "Especially if it accompanies a drawn-out multicycle
change,
it could be a noticeable impact dampener.
"
The problem with logging, however, is that it may not be available to the library, depending on the execution environment of the program, Berrangé said:
Security policies applied to [processes] (SELinux, seccomp, namespaces or containerization in general) can easily prevent ability to use journald, syslog, or opening of log files, leading to messages not being visible. At worst, with seccomp an attempt to use the blocked feature may lead to an abort of the application.
Sosedkin was concerned
that applications might be reopening the stderr file descriptor in
strange ways, resulting in "writing our scary messages who knows
where
", but Berrangé
said that is
fairly unlikely precisely because various libraries already print warnings and
errors to stderr when needed. "Even glibc will print to
stderr when it sees malloc corruption,
and stuff in stderr will end up in the systemd journal for any
system services.
"
SSH?
Older SSH protocols use SHA-1, so Richard W.M. Jones wondered if
the change would cause problems using SSH. "This broke before, requiring
us to advise users to set the global policy for the machine to LEGACY
(thus ironically weakening crypto for everything).
" He also noted
that he has some "ancient network equipment
" that requires the
LEGACY setting on Fedora in order to be able to connect to it. He asked if
the changes Sosedkin wants to make will further impact SSH.
SSH uses SHA-1 as a hash-based
message authentication code (HMAC), Sosedkin said,
"which doesn't rely on collision resistance
", so no change
needs to be made. Obenour agreed
that using SHA-1 as an HMAC is reasonable, though there are alternatives
that perform better; "There are no
known attacks on HMAC-SHA-1
".
Goals
Along the way, Sosedkin listed his objectives in raising the issue:
I want to know, ultimately, how can I break it for devs but not the users so thatHope these are all reasonable things to wish for.
- the need for it is communicated early,
- all the relevant places are timely identified,
- maintainers have a plenty of time to resolve it for good or opt out,
- and the users are affected as late and as smoothly as possible.
As yet, it does not seem that there is a clear plan that would meet those goals. It is a difficult problem, but one that seems likely to recur over time. Cryptographic algorithms and protocols change over time, generally because weaknesses are found in them; once they become entrenched in a distribution, and the internet itself, they are hard to excise. Finding a way to remove SHA-1 smoothly—as smoothly as possible, at least—will be a useful thing to figure out. It seems likely that more discussion will be required to get there.