Linux's missing CRL infrastructure

In July 2024, Let's Encrypt, the nonprofit TLS certificate authority (CA), announced that it would be ending support for the online certificate status protocol (OCSP), which is used to determine when a server's signing certificate has been revoked. This prevents a compromised key from being used to impersonate a web server. The organization cited privacy concerns, and recommended that people rely on certificate revocation lists (CRLs) instead. On August 6, Let's Encrypt followed through and disabled its OCSP service. This poses a problem for Linux systems that must now rely on CRLs because, unlike on other operating systems, there is no standardized way for Linux programs to share a CRL cache.

CRLs are, as the name might suggest, another solution to the problem of certificate revocation. If a web server loses control of its signing certificate, the administrator is supposed to report this fact to the web site's certificate authority, which will publish a revocation for it. Clients periodically download lists of these revocations from the certificate authorities that they know about; if an attacker tries to use a key that has been revoked in order to impersonate a web site, the client can notice that the key is present in the list and refuse to connect. If the certificate revocation system were ever to stop working, an attacker with stolen credentials could perform a man-in-the-middle attack in order to impersonate a web site or eavesdrop on the user's communications with it.

This system worked well enough in 1999, but as the internet grew, the number of certificate revocations grew along with it. In 2002, RFC 6960 was standardized, creating the online certificate status protocol. Clients using OCSP send a request directly to the certificate authority to validate the certificate of each site they wish to contact. This had the benefit of freeing every client from having to store the CRL of every certificate authority they trusted, but it had a number of drawbacks.

Firstly, using OCSP means waiting to establish an entire second TCP connection to the certificate authority's potentially busy server before completing the TLS handshake with a web site. Perhaps more seriously, it also exposes which web sites a client is visiting to one or more certificate authorities. The original OCSP protocol did not even mandate that requests and responses should be encrypted, so the information about which web sites a client was visiting were potentially visible to everyone who could eavesdrop on data on the wire somewhere along the path.

Those problems were partially solved by the introduction of "OCSP stapling", an extension whereby the server makes its own OCSP request to the certificate authority, and then includes the response (which is signed by the certificate authority, and therefore can't be tampered with) in its opening TLS handshake. The response can be cached by the server for a short time, reducing the load on the certificate authority.

This led to OCSP recapitulating the way certificates work in the first place. A server makes a request to a certificate authority for timestamped cryptographic proof that it is who it claims, with the proof expiring after a certain time, and then presents this proof to clients that connect to it. But since OCSP is not universally supported, the absence of a stapled OCSP response could be as the server intends, or it could be a sign that the connection is being tampered with. Similarly, if a certificate authority doesn't respond to an OCSP request, it could be that its server is down, or it could be that an attacker is blocking the response. Because of the spotty support and extra complexity, in practice some clients began to ignore OCSP timeouts from certificate authorities — which, in the face of an attacker who can interfere with packets on the network, is equivalent to giving up on OCSP entirely.

This is part of what made browser vendors and other interested parties start pushing for certificates with shorter expiration dates: if servers could make requests to OCSP servers automatically, why not do the same for certificates themselves? Short-lived certificates, with lifetimes measured in days, don't really gain anything from OCSP, and don't spend as much time taking up space in a CRL if they are compromised.

In 2023, the CA/browser forum decided to make OCSP optional, and CRLs required once more. The major browsers now use CRLs preferentially, although Firefox will fall back to OCSP if a certificate is not included in its CRL and an OCSP server is available. In 2015, Mozilla began publishing a consolidated CRL that combines the CRLs from various certificate authorities, so that each browser would only need to download and process a single file. It also includes a list of known-good certificates, to avoid the overhead of OCSP for many web sites. Chrome does the same with its own list. Both macOS and Windows also have an OS-wide shared CRL downloading mechanism.

On Linux, outside the browser, there is no such shared mechanism. In practice, many programs just don't support certificate revocation. OpenSSL and other cryptographic libraries include the tools needed to verify certificates against a CRL or an OCSP response, but many programs don't use them. Curl has support for enabling OCSP as well, but it's not enabled by default. Individual programs probably shouldn't be responsible for deciding which CRLs to fetch and keeping them up to date in any case. The Let's Encrypt community forum hosted some discussion about potential solutions, but didn't come to a consensus. Jesper Kristensen noted that certificate validation on Linux is "a mess" in other ways too, with applications often using insecure root certificates:

Most [applications] fill the content of these root stores by copying Mozilla's root store and removing Mozilla-specific attributes from the roots because applications can't read these attributes. But Mozilla has repeatedly said that using their root store while ignoring these attributes is insecure.

Ellie Kanning, who started the discussion, filed follow-up bug reports with OpenSSL, NetworkManager, and Freedesktop's desktop specifications, but none of the projects thought that maintaining a system-wide CRL was in-scope for them. I emailed Kanning to request a comment, and she was clear that she isn't an expert in this area, just a Linux user who is worried about the security of her desktop. She called the situation "an incident waiting to happen", and seemed somewhat resigned to it, given the reception of the bugs she tried to file:

It seems to me like this can only be solved if some of the enterprise users, e.g. rallied behind the Linux Foundation, can be convinced to show some interest in maintaining an up-to-date cert store that has some sort of CRL subscription mechanism like Mozilla's cert push that is available for all Linux systems. Perhaps BSDs could then benefit from this as well.

Some people have suggested that the trend of shortening certificate lifetimes — with the maximum lifetime acceptable to browsers planned to be 47 days by 2029 — means that both CRLs and OCSP are becoming less important, and not worth the difficulty of supporting. Let's Encrypt does offer six-day certificates to a limited number of domains on a trial basis; that validity period is shorter than the lifetime of most OCSP responses. Andrew Hutchings, a developer of WolfSSL, said: "With OCSP being flawed in many implementations and CRLs just not scaling, there isn't really an alternative [to shortening certificate lifetimes] right now."

Kanning doesn't agree, pointing out that even the shortest lifetimes being discussed are still a fairly long time:

The problem is in my opinion that many services will simply not move to short-lived [certificates], especially big enterprises that don't have an IT focus I don't see doing that. The Chrome plan seems to be to limit it to around a month, but that's very long time if a cert were to leak right after issuing.

It's clearly not impossible to have secure TLS connections on a modern Linux desktop. Firefox has the code to support this, and Mozilla already maintains the CRL infrastructure to keep it secure. But with OCSP going away, most applications simply don't have access to the frequently updated CRL that they need in order to correctly establish a modern TLS connection. Linux systems would benefit from a standard way to update and maintain a CRL for all programs to use by default, if anyone decides to tackle the problem.