Ever-growing CRL? [LWN.net]

Ever-growing CRL?

Posted Dec 5, 2024 20:09 UTC (Thu) by NightMonkey (subscriber, #23051)
Parent article: Let's Encrypt sets date for ending OCSP support

From the CRL Wikipedia page linked to in the LWN post:

"CRL files may grow quite large over time e.g. in US government, for certain institution multiple megabytes. Therefore, incremental CRLs have been designed[14] sometimes referred to as "delta CRLs". However, only a few clients implement them.[15]"

So, am I understanding that this file (or shards of this file) will grow without bounds over time? Is there some part of the mechanism that "garbage-collects" or otherwise purges items from the CRL?

to post comments

Ever-growing CRL?

Posted Dec 5, 2024 20:17 UTC (Thu) by dskoll (subscriber, #1630) [Link] (15 responses)

Presumably, once a certificate has expired, it no longer needs to be in the CRL.

Ever-growing CRL?

Posted Dec 5, 2024 20:58 UTC (Thu) by hansl (subscriber, #5086) [Link]

We had to implement OCSP support in Wine not too long ago because the CRL that Steam downloads grew over 50MB. That exceeded the timeout on slow connections.

Ever-growing CRL?

Posted Dec 5, 2024 21:25 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (13 responses)

Yet another reason that many security folks have been pushing to shorten the maximum certificate lifespan.

But that's overly reductive. It's probably better to look at this argument holistically. The traditional CA approach looked roughly like this:

* You pay a CA some money for a certificate (and a useless HTML badge that does nothing other than "look professional").
* They issue a certificate, valid for 3-ish years.
* In 3-ish years, your site has a brief outage because the webmaster forgot to set a calendar reminder (or was on vacation, or quit, or whatever).
* You trust the CA, because their entire business model is dependent on them not screwing it up.
* Revocation is largely an afterthought.

The modern approach looks like this:

* You don't pay anyone any money at all.
* Let's Encrypt (or, hypothetically, someone else) issues a certificate, valid for 90 days.
* 90 days is short enough that you basically have to automate it, because it's too painful otherwise.
* You trust the CA, because all of their certificates are in the CT log (on pain of not working in modern browsers). If they issue a bad certificate, somebody will notice and scream bloody murder about it.
* Revocation is still problematic, but less of a dire problem because all certificates expire rapidly, and all certificates are publicly logged, so it is not unreasonably difficult to go and find all of the certificates that might need to be revoked.

In the traditional model, OCSP looks like a somewhat plausible means of making CRLs more scalable. But in the modern approach, this is unnecessary because your CRLs only have to cover 90 days of lookback in the first place. OCSP also has multiple problems:

* Most browsers fail-open if the OCSP service is unreachable, so an active MitM can simply block it.
* OCSP has privacy implications, since it sends all domain names that you visit to the CA for that domain.
* OCSP Must-Staple solves both of the above problems, but is more complicated to set up correctly, and an extended outage of the OCSP service can (eventually) cause an outage of your website. It's not quite as risky as HPKP, but I wouldn't want it on a service I have to sysadmin if I could avoid it.

Ever-growing CRL?

Posted Dec 6, 2024 11:18 UTC (Fri) by paulj (subscriber, #341) [Link] (7 responses)

If only we had a system that allowed blobs of data to be associated with names and easily looked up in a highly distributed manner, that came with meta-data that gave expiry times to that data so that said data could be cached and refreshed as needed by said distributed system.

If only.

Ever-growing CRL?

Posted Dec 6, 2024 12:59 UTC (Fri) by dskoll (subscriber, #1630) [Link] (6 responses)

That has the same information-leaking downside as OCSP, though.

Ever-growing CRL?

Posted Dec 6, 2024 13:52 UTC (Fri) by paulj (subscriber, #341) [Link] (5 responses)

You're almost certainly already doing a lookup for the name to find an address mapping anyway, if you're interested in whether a certificate has expired or not. "What's the status of cert for X?" doesn't leak any information of note over the "What's the address of X?" query likely already made.

So, you almost certainly already have that information leak. And if you care about information leaks, then you already have taken steps that fix that leak for /any query/. (E.g., DoH to a varied set of DoH servers, over ToR or other anonymising overlay - my personal machines are setup for dnscrypt for DoH over Tor).

Ever-growing CRL?

Posted Dec 6, 2024 15:51 UTC (Fri) by dskoll (subscriber, #1630) [Link] (4 responses)

You're almost certainly already doing a lookup for the name to find an address mapping anyway

Oh sure. You're leaking that info to your name servers, your ISP's name servers maybe, and all the servers along the chain up to the root of the domain you're looking up.

You're not leaking the info to some third-party that maintains the list of revoked certificates.

Ever-growing CRL?

Posted Dec 6, 2024 16:59 UTC (Fri) by paulj (subscriber, #341) [Link] (3 responses)

> You're leaking that info to your name servers, your ISP's name servers maybe, and all the servers along the chain up to the root of the domain you're looking up.

No, no I'm not. Whether it's address lookups or cert lookups, that's hugely sensitive information, and I am amazed that in this day and age we're still by default sending that information in the clear to some random local device (AP, router, whatever) managed by someone we may not trust much if at all with our sensitive data (cafe, airport, school, employer, some giant incumbent former-state-monopoly national telco, etc.), using IPs that are very likely tied to us.

I am especially amazed that technical people, such as those here, are still doing that. ;)

We should be using anonymised name lookup systems *BY DEFAULT* here in the 21st Century.

My machines are setup to resolve using a local dnscrypto, making queries over Tor, to public DoH servers. The DoH servers might see the queries, but they can't relate them back to me. My local and access networks handle packets, but they have no clue about what queries I'm making. If you don't like Tor, do something similar with a VPN provider. Not quite as good, but probably sufficient for most privacy needs.

(And yes, the Five Sisters and some other SIGINTs might have enough nodes to do correlation analysis to some degree, but that's not a concern to me).

Ever-growing CRL?

Posted Dec 6, 2024 18:10 UTC (Fri) by dskoll (subscriber, #1630) [Link]

Good for you. But you are clearly not in the target audience that Lets Encrypt believes it's protecting by deprecating OCSP. I'd say no more than 0.2% of all the people on the Internet have your level of privacy WRT DNS, and that's unlikely to change in the forseeable future.

Ever-growing CRL?

Posted Dec 19, 2024 12:17 UTC (Thu) by Lennie (subscriber, #49641) [Link] (1 responses)

I had found using Tor to connect to public DoH servers was pretty darn slow, what is your experience ?

Ever-growing CRL?

Posted Jan 2, 2025 15:28 UTC (Thu) by paulj (subscriber, #341) [Link]

dnscrypt-proxy can setup connections to a wide-range of public DoH servers (it can download lists of them) and then sort them by RTT. Chances are good that over all those different Tor circuits, to different DoH servers, you'll get at least a few that are fast.

Tor is pretty fast these days. A good few years ago Tor was indeed often noticeably slower than going direct. Today I browse the web via Tor by default, and I don't ever notice a downside when it comes to speed. You can pretty much reliably watch YouTube (or whatever) over Tor these days.

The one thing you do notice with Tor is that you regularly get the "Are you a human" interstitial checks. Cloudflare seem to be clever enough to just want you to click a check-box with your mouse and not check you again for quite a while, but some others will put you through a Captcha type puzzle. Thankfully, the majority of cases are Cloudflare, so it's not really a bother. Though, Google make you do a Captcha thing. There are a small handful of sites that outright block you when they detect Tor.

At some point, I think Tor (or similar onion routing) should become the default way for end-users to interact with the Web. Privacy should be a fundamental component of the Internet.

Ever-growing CRL?

Posted Dec 6, 2024 12:16 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (4 responses)

In theory the old model was checking that you were who you say you are, while the new model only checks that you have control over the domain.

Of course, the "check" step was not done properly before.

Ever-growing CRL?

Posted Dec 6, 2024 18:20 UTC (Fri) by tialaramex (subscriber, #21167) [Link] (3 responses)

The technology to have the CA warrant that "were who you say you are" still exists, and is exactly as functional as it was ten or twenty years ago. It's also exactly as useless.

The problems are manyfold, but most particularly the users do not know what they're supposed to be checking - in the US every single state (50 of them) gets to have their own business rules, and their own registrations, so you need to guess the "right" state for any particular business. For very local outfits it's probably the state where you are now, but for anywhere big enough to have an actual business structure they're likely registered somewhere financially & legally beneficial, such as Delaware. So who knows

Then, brands are what you care about as a customer, but the business name isn't the brand. Maybe you know them as "Big Kahuna Burger" but if their legal business name is "Edison Restaurant Group" too bad, the paperwork does not give the burger brand name and neither does the certificate.

Also, even assuming you do care that this site is owned by the "real" Edison Restaurant Group registered in Louisiana and not the one from Texas or California, the checking was always rather flimsy, it could hardly be otherwise. Somebody is going to get a business directory, they're about as reliable as a phone book was back when those existed. They're going to check that the person who asked for a cert can (say), read a Fax they sent to the right Fax number, something like that. How the check is performed is likely to be decided by the attacker, from a menu of options. Considering how relatively worthless this is, it's not exactly a huge security hole, but keep in mind that's what we're talking about - can you trick the person on some corporate front desk into letting you steal one fax, which you know when it's coming ? Or maybe make one phone call from a corporate number ? Or maybe make a decent reproduction of their corporate headed notepaper and wear a smart suit to a lawyer's office ? Weakest link applies, often it's "forge email from the right person" these days because modern business directories list email addresses.

Ever-growing CRL?

Posted Dec 7, 2024 3:22 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (1 responses)

I was thinking more of europe where a .se domain (generally) means the website is swedish, and so on. Unless of course they are playing clever wordgames with the toplevel domain, like do.it or something like that.

Ever-growing CRL?

Posted Dec 9, 2024 21:27 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

Physical locality is required by some ccTLDs, but it is far from universal, not always consistently enforced by the registries, and probably should not be relied on for security purposes. The same goes for geographic gTLDs,[1] at least in my estimation.

[1] : https://en.wikipedia.org/wiki/List_of_Internet_top-level_...

Ever-growing CRL?

Posted Dec 9, 2024 21:48 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

IIRC there was also someone who created a business called "Identity Verified" and then got an EV certificate for it... which caused the words "Identity Verified" to appear next to (or in the case of mobile Safari, *in place of*) the URL bar. Now, if you're familiar with how certificates work, you can probably figure out that the blanket phrase "Identity Verified" is not a promise the browser can plausibly make, even if you don't know the details of what's supposed to be displayed there. But the average end user should not be expected to understand how the web's PKI works (or fails to work). From their perspective, if the browser is confidently displaying a green UI that says "Identity Verified," well, then it must be OK, right?

And this leads us to the broader problem with EV: You are allowing third parties to display (effectively) arbitrary text in the browser's trusted UI, in green and right next to a lock symbol, with the intent that users accept that text at face value. There are probably a thousand different kinds of mischief that attackers can make with that. The only reason they mostly didn't bother is because, back when EV was still a thing, you didn't even need HTTPS for phishing in the first place since so much of the web was still on plain HTTP.

Ever-growing CRL?

Posted Dec 6, 2024 13:28 UTC (Fri) by mcatanzaro (subscriber, #93033) [Link] (3 responses)

Usage of actual CRLs is very limited. They are mostly supported by command line tools. No web browsers respect normal CRLs. I maintain the TLS integration in GNOME (glib-networking), which is based on GnuTLS. I refuse to support CRLs in GNOME for the reason you mention: they're too big and will tank performance. The security benefit is definitely not worth the cost.

Let's Encrypt is using the term "CRL" a little generously to additionally refer to proprietary solutions (proprietary as in nonstandard, not nonfree), specifically OneCRL from Mozilla and CRLSets from Chrome. The proprietary "delta CRLs" don't have any of these problems, but they are also not normal CRLs. I'm not interested in supporting them in GNOME because they are not Internet standards. To the best of my knowledge, they're really only used by Firefox and Chrome. I have ignored them. If one or the other could be standardized, then maybe it would be worth implementing in GnuTLS and OpenSSL? But as long as they remain nonstandard and not supported by TLS libraries, I don't care about them and users do not benefit.

In GNOME, the only supported form of certificate revocation is *stapled* OCSP. This doesn't have any privacy concerns that I'm aware of. We don't support TLS clients talking directly to OCSP servers (which is a terrible idea); instead, the TLS server is expected to talk to the OCSP server and "staple" the OCSP response by sending it to the client. It's only useful if you use the Must-Staple extension, though; otherwise, an attacker would simply not staple any OCSP response to guarantee no revocation. But it works. The Internet has collectively decided not to deploy it for unknown reasons; I'm really not sure why, because it's the only solution that really works. (Well, some sites deploy it. We've occasionally noticed certificate validation failures that don't occur in major web browsers when a website staples an invalid OCSP response.)

Ever-growing CRL?

Posted Dec 7, 2024 9:14 UTC (Sat) by tialaramex (subscriber, #21167) [Link] (2 responses)

The nonstandard solutions like OneCRL are just generated from the CRL that Let's Encrypt and the like will deliver.

The browser vendors consume the various CRLs and produce their preferred data, since they own it they get to make the rules, which is beneficial to them and, arguably their users. Suppose a CA says we don't consider that what we did was mis-issuance for 8000 certs, so, we're just not marking them revoked. The vendor can say too bad, we're marking them revoked immediately anyway. Which is a smaller stick than "Since you were uncooperative we're distrusting you"

As with the Web PKI itself, there's a trade off to be made about what you hope to get out versus available resources to do the work. Consuming OneCRL means now you're stuck with Mozilla's choices on everything, but you need only barely enough volunteer effort to write and maintain the OneCRL consumption code, nothing more. A hypothetical GNOME-CRL built the same way would be able to do whatever GNOME wants, regardless of Mozilla, but incurs a considerable maintenance overhead forever.

The biggest thing I expect that a OneCRL is handling is bugs. My guess is that over a year probably half a dozen cases arise where oops, this CA turned off the machine that makes a CRL and so now it's just a 404 or a timeout, or doh, that other CA accidentally uploaded their CRL to the wrong URL every single day for the last six weeks, somebody fat-fingered the destination filename, fetch some-ca.example/CRM.dat instead of CRL.dat. Building a pipeline that's robust against the first sort of nonsense yet flexible enough to cope when the second sort happens will make these worth their time.

As to OCSP stapling, my understanding is that the HTTPS server software is just garbage - especially Apache - and this software has mostly rusted in place, so if you're in the browser client business you can't expect anything which requires the server software to be properly engineered to ever work, you have to base your assumptions on it's garbage, it will never be fixed and nobody who could do anything about that is interested in doing so. It's very much a case of if you don't Do It Yourself (as things like OneCRL do) then it won't get done. This is not a problem for the top 1% sites, your Facebooks, Youtubes, Wordle - somebody is getting paid to keep those working, and they probably read documentation and apply patches _they_ could have OCSP stapling (but many do not). But 99% of sites aren't like that, maybe they were thrown together by somebody's nephew six years ago and live "in the cloud" on a web server which is never maintained and will be gradually rusting until they're taken over by crooks or turned off. They're never getting OCSP-stapling.

Ever-growing CRL?

Posted Dec 7, 2024 14:09 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link] (1 responses)

I don't think that's a great argument in support of OneCRL, because the end result is zero certificate revocation support for anybody who's not using the one or two applications that support the weird proprietary CRLs. Almost everything on your desktop is using GnuTLS or maybe OpenSSL and they simply do not implement nonstandard stuff.

I also don't think that's a even valid argument against OCSP stapling, since stapling is not mandatory, and theoretically only servers that support it properly would ever opt in.

Ever-growing CRL?

Posted Dec 16, 2024 5:14 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

> I don't think that's a great argument in support of OneCRL, because the end result is zero certificate revocation support for anybody who's not using the one or two applications that support the weird proprietary CRLs.

Speaking as a Google employee who has no direct involvement with Chrome or Chromium, I would not describe OneCRL as "proprietary" when it has two independent FOSS client implementations. If other FOSS applications do not want to take code from Chromium or Firefox, that's a valid choice for them to make, but it does not mean that OneCRL is proprietary.