Changes in the TLS certificate ecosystem, part 2

The internet is moving toward more encrypted traffic via TLS, but the certificate authorities that issue certificates for encrypted connections have gotten a bad reputation. In various instances in the past, certificate authorities have issued rogue certificates, for example. In part one we covered some of these incidents and discussed how Certificate Transparency can help to improve this system.

While there are plenty of calls to ditch the certificate authority system altogether, none of the proposals for an alternative have gained any significant support in the past. Instead, the discussion has shifted to ways of improving the existing system. Here we will look at some of those improvements.

Key pinning

One method to strengthen the security of TLS certificates is key pinning. In principle, this is nothing new. The Secure Shell protocol (SSH) has used a form of key pinning for a long time. When one connects to an SSH host for the first time, a fingerprint of the server's key is stored and compared on future connections. This method is called Trust On First Use (TOFU). The obvious disadvantage of a TOFU system is that the first connection attempt is still vulnerable to man-in-the-middle attacks. In SSH, this is mitigated by the fact that users are shown the fingerprint, which they can manually check.

A first attempt at implementing key pinning for TLS was developed by Moxie Marlinspike and Trevor Perrin under the name TACK (Trust Assertion for Certificate Keys). It was implemented as a TLS extension. But TACK never achieved widespread adoption.

HTTP Public Key Pinning

Engineers from Google came up with a similar approach that was developed under the name HTTP Public Key Pinning (HPKP), which is now specified in RFC 7469. It is implemented as an HTTP header. This has the advantage that it doesn't require any software changes on the server side; it is merely a configuration issue. On the other hand, it is bound to HTTP and thus cannot be used to secure other protocols like IMAP or SMTP. It was first implemented in Chrome, followed by Firefox earlier this year. Alongside HPKP, the browser vendors ship with some pins for the keys of popular sites.

An example of an HPKP header looks like this:

    Public-Key-Pins: max-age=5184000; \
        pin-sha256="HD3EpAqgxJWKGiSuuXPyipmL33IwYlwhLUgF1gKYOuc="; \
        pin-sha256="dwUkkREEnv6pEtNJoRzlBHJm3IlUvPhgy0mdYFOM6V8="; \
        includeSubdomains

The pins are a Base64-encoded hash of a public key. I have created a simple bash script that can be used to generate hashes for pins. A site operator can choose whether they want to pin their site to a certificate authority or to the leaf certificate itself. The server is required to always set at least two key hashes in the header to facilitate key changes. To be valid, one of the key hashes needs to match either the key of the current certificate from the site or the key from one of the certificate authorities in that certificate's chain. The optional includeSubdomains keyword signals the browser that the key pinning should also apply to all subdomains of the current domain.

The browser will store these hashes; on future connections they will be checked to see that they match. The max-age value is the number of seconds a browser is requested to store a key pin for. It is usually set to a value representing several weeks or even months.

The challenging part of HPKP is the key management. That is the reason why there always must be at least two key hashes in the HPKP header. TLS certificates all have an expiration date and need to be changed on a regular basis. So, when a site gets a new certificate, it has to use a key that is already part of the existing pins.

Ideally, a site operator has a backup key that gets used as the key for the next certificate. Then the pin is changed to refer to this key and a new key. This key management procedure carries a risk, though. If all keys get lost, the site operator may end up in a situation where they are unable to create a certificate that the browsers that already have a pin for the site will consider as valid. Therefore, managing keys intelligently along with having safe backup copies of the keys are crucial if a site operator decides to implement HTTP Public Key Pinning.

It can be argued that key pinning is weak because it still needs a first connection to succeed with the correct certificate. A powerful adversary might be able to intercept all connections. However, HPKP is built on top of the already existing protections. A successful attack would require a compromised certificate and a man-in-the-middle-attack on the first connection attempt to a server. The attacker could be a malicious worker at an ISP or someone that controls a wireless access point, for example. It is likely that such an attempt would be detected, especially if HPKP is combined with using Certificate Transparency. Uncovering the attack would endanger the business of a certificate authority who had issued the bogus certificate, so such attacks seem relatively unlikely. HPKP is not perfect, but it is a big improvement.

Rules for operating certificate authorities

HTTP Public Key Pinning and Certificate Transparency are technical measures to improve the security of the certificate authority ecosystem. But, on an administrative level, things are also changing. In 2011 the CA/Browser forum published the first Baseline Requirements. They contain a set of rules for the issuance of TLS certificates, which browser vendors and certificate authorities have agreed upon. They are updated regularly and there are often fierce debates about which new security guidelines should be implemented. But the fact that clear rules exist often helps in deciding whether actions by a certificate authority are legitimate or not.

For example, earlier this year the news web page Tivi.fi reported that a user was able to receive a certificate from Comodo for the domain live.fi, which is the Finnish version of Microsoft's Live services. He was able to register the email address "hostmaster@live.fi" through Microsoft's email service. Comodo allowed verifying the ownership of a domain via an email to this address. So, was this incident Microsoft's or Comodo's fault?

The Baseline Requirements allow answering that question. The process of verifying domain ownership via email to a defined set of aliases (admin, administrator, webmaster, hostmaster, postmaster) is documented there. Microsoft can hardly argue that it didn't know that—the company is part of the CA/Browser forum that wrote the Baseline Requirements. But this incident serves as a reminder that all operators of public email services need to block the creation of these aliases for users.

Free certificates for everyone

Users have long been unhappy with the amount of money required to get a certificate—or even that any payment was required at all. But TLS certificates are much more affordable these days; some certificate authorities now issue certificates for free. StartSSL has issued free certificates for several years, however it restricts them to non-commercial use. The company received criticism after the Heartbleed bug when it charged a fee to users who wanted to revoke and renew their free certificates. Recently, the Chinese certificate authority Wosign also started offering free certificates, thus giving users dissatisfied with StartSSL another option.

A new certificate authority called Let's Encrypt is currently beta testing its system and will soon offer free certificates to everyone. Let's Encrypt was started with the intent of making HTTPS easier to deploy. It is hosted by the Linux Foundation and sponsored by the Electronic Frontier Foundation, Mozilla, Cisco, and Akamai. Let's Encrypt also has a cooperation agreement with the certificate authority Identrust, which will sign the Let's Encrypt intermediate certificate. This cross-signature from an existing certificate authority will guarantee that the Let's Encrypt certificates will be accepted by all major web browsers.

One aspect of Let's Encrypt that's worth noting is that the validity of its certificates is limited to 90 days. As discussed below, this short lifetime is intended to help minimize the consequences of a compromise, but it is likely to make these certificates harder to manage in the short term.

Remaining problems

With stricter rules, browsers less willing to accept misbehavior by certificate authorities, and new technologies providing additional protection, the ecosystem around TLS certificates is much safer these days than ever before. And with free certificates available for everyone, the costs of certificates are no longer an issue. However, a number of challenges remain.

The revocation of certificates is not working in practice. There exist two protocols for certificate revocation: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP). CRLs traditionally were just a signed list of revoked certificates provided by a certificate authority. It is obvious that with millions of certificates issued and sometimes many of them revoked at once, this doesn't scale in practice. With OCSP, the browser asks a dedicated OCSP server whether a certificate is still valid.

The problem with OCSP is that all browsers implement it in a soft-fail mode. That means that if a browser is unable to connect to an OCSP server, and thus cannot verify the certificate's validity, it will still consider it as valid. However, this soft-fail makes the whole protocol mostly useless. In the case where an attacker has a certificate that was revoked—maybe because the original owner knew the private key was stolen—and uses it for a man-in-the-middle attack, the attacker can just block all connections to the OCSP server.

Changing OCSP to hard-fail mode is not practical, though, because that would require almost 100% uptime and connectivity for the OCSP servers. Also, in certain situations the network configuration blocks the connection to an OCSP server. A typical case where that happens is captive portals from WiFi services.

The Chrome project has decided to pull the plug on OCSP and removed support for it from Chrome a while ago. Google uses an alternative called CRLset where the company pushes a list of revoked certificates to the browser. However, CRLset does not cover all revoked certificates; it is a reduced set of certificates that Google considers important enough to block.

The revocation problem could be solved with a technology called OCSP stapling. With OCSP stapling, the OCSP answer is no longer requested by the browser directly, instead that is the job of the server; it gets shipped within the TLS handshake. This is more robust than live requests to OCSP servers and also has privacy advantages. However, in order to provide a secure method of revocation, there needs to be a mechanism to enforce OCSP stapling. Such a mechanism was proposed in 2012 as a draft, but it never became an RFC.

An alternative to certificate revocations is short-lived certificates. Let's Encrypt decided to issue certificates that are only valid for a maximum period of 90 days. With this move, Let's Encrypt wants to encourage automation in the certificate-issuance process. If certificates are only valid for a shorter amount of time, revocations matter less. In case of a breach, an attacker has only a limited time to use stolen certificates and keys.

However this approach opens up other challenges. It is crucial to always generate new certificates with new keys, especially if a breach is suspected. Statistics of certificate revocations after the Heartbleed bug showed that many administrators are not aware of that. In combination with HPKP, the management of short-lived certificates gets even more complicated.

Relying on certificate expiration dates also makes the correctness of the system clock more important. An attacker could use an expired certificate that has a key that was exposed, then manipulate the victim's system time with forged NTP replies so that the certificate appeared to be valid. Security researcher Jose Selvi [YouTube] and a recent research paper from Boston University [PDF] have looked at attacks against NTP. An authenticated version of NTP is currently in development.

Getting certificates using BGP routing attacks

An upcoming challenge may be attacks on the certificate-issuing process itself. At the Black Hat conference this year, researchers presented a method to attack the certificate-issuing process [YouTube] through manipulations of the BGP routing protocol. The challenge is that all of the methods to verify the ownership of a domain are unauthenticated. Usually the proof of ownership is either done by sending a verification token by email or by uploading a verification token to a web site in the domain, both of which are susceptible to routing attacks.

Many of the improvements in the security of TLS certificates only target HTTPS. This leaves other protocols that also use TLS for encryption unprotected. In particular, SMTP is used for server-to-server-communication between email servers, and is open to attacks. SMTP can be secured with a mechanism called STARTTLS, but the protection provided by STARTTLS is limited. It prevents passive observers from reading the transmitted data, but an active attacker can easily strip the STARTTLS command and thus prevent an encrypted connection from being made. Also SMTP servers often use self-signed certificates, so it is common not to verify the validity of such certificates.

A recent study [PDF] by researchers from the University of Michigan in collaboration with Google has highlighted that these attacks aren't entirely theoretical. They found widespread evidence for STARTTLS stripping in practice. A mechanism similar to HTTP Strict Transport Security (HSTS) and HTTP Public Key Pinning (HPKP) could help, but nothing like that has yet been widely deployed in SMTP.

Regaining trust

Without doubt, technologies like key pinning and Certificate Transparency, stricter rules, and better enforcement of those rules by web browser vendors have made TLS certificates safer. But the adoption of the new technologies is only happening slowly. On the client side, HTTP Public Key Pinning is currently supported by Chrome and its free variant Chromium, Firefox, and Opera. Microsoft's Internet Explorer and Apple's Safari haven't picked up the feature yet. The number of web sites serving the HPKP header is still very small. Meanwhile, Certificate Transparency is currently only supported by the Chrome/Chromium browser.

The many failures of certificate authorities in the past have undermined the trust in one of the cornerstones of internet security. The certificate authorities themselves are largely to blame for that. In addition, the improvements that made the system more robust didn't come from the industry itself; instead the browser vendors and Google in particular pushed for improvements. It will still take some time to regain much trust in TLS certificates.