Fallout from upcoming Let's Encrypt certificate changes

[Posted November 6, 2020 by corbet]

As described in this Let's Encrypt blog entry, certificates issued by Let's Encrypt will soon be signed solely by that organization's own root certificate, which is accepted by all modern browsers. There is one little catch, though: versions of Android prior to 7.1.1 (released in late 2016) do not recognize that certificate and will start throwing errors. "Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites." There appears to be little to be done about this problem other than to encourage owners of older Android devices to install Firefox.

to post comments

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 6, 2020 18:40 UTC (Fri) by smurf (subscriber, #17840) [Link] (9 responses)

Isn't it possible to install a root cert on these things? Granted, you get an annoying "your device may be monitored" message all the time, but that's still way better than "your web site isn't trusted".

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 2:19 UTC (Sat) by flussence (guest, #85566) [Link] (6 responses)

You can inject one into the system partition if you have a steady hand and a rooted phone (and with one this old, there's no warranty left to void). In my experience it's no harder than updating the preinstalled emoji fonts.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 8:49 UTC (Sat) by smurf (subscriber, #17840) [Link] (5 responses)

Rooting old phones requires erasing them. I'd hazard that the users of those phones would be cautious about that (data loss), as opposed to current phones (loss of access to baking and game apps).

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 11:05 UTC (Sat) by smurf (subscriber, #17840) [Link] (2 responses)

Ba*n*king of course. Sheesh.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 19:09 UTC (Sat) by ejr (subscriber, #51652) [Link] (1 responses)

I have notes on recipes stored on my phone, so it's fine. ;)

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 19:12 UTC (Sat) by bobsol (subscriber, #54641) [Link]

so I should update my baking apps right away?

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 8, 2020 10:16 UTC (Sun) by Moarc (guest, #137864) [Link] (1 responses)

Rooting new phones (more precisely, unlocking the bootloader in order to root them) also requires erasing user data, AFAIK.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 8, 2020 13:42 UTC (Sun) by smurf (subscriber, #17840) [Link]

I know. My point is that on a new phone you're unlikely to have old apps which are no longer in the app store, data you can't get back up in a sane way, or whatever.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 12, 2020 5:43 UTC (Thu) by Comet (guest, #11646) [Link] (1 responses)

(Note: I haven't looked at the situation here in some years; it may have changed, but even if so, it won't be fixed for the older devices which are the ones where this remedy is being proposed, so this old understanding is still relevant.)

Unfortunately, a very underdocumented "feature" here is that there's some kind of corporate CA mode where the corporates with TLS interception boxes want to be able to have their certs verify for anything, and rather than doing things "right", the approach taken is that any extra CAs you add manually yourself disable hostname verification for the certificates issued by them.

I remember disbelieving it could be this broken, confirming it and being horrified. The fact that some organizations might have a need for this should not make this gaping security problem the default for all CAs added.

So only add a root cert where you tightly control the root and know that it won't be issuing other certs. Doing this for the ISRG roots is ill-advised.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 13, 2020 10:09 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

This is deliberate breakage on the part of the browser people that control the HTTPs standard process: write specs as if interception boxes did not exist (even though everyone knows they do exist, for many practical and legal reasons), and then hack thing up browser side so Google and Mofo can still peddle their stuff corporate side.

The more broken this mode is the more browser people are happy.

This is cheer hypocrisy and makes things painful for users for no good technical reasons. It just shows that software is a social, not technical process.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 6, 2020 22:56 UTC (Fri) by ibukanov (subscriber, #3942) [Link] (6 responses)

They are not so clear on timing. On 2021-09-01 the root certificate that was cross-signed for Let's Encrypt expires, but from 2021-01-11 they switch for new certificates to their own root. In the mean time as a workaround one can use an alternative configuration allowing to get certificates with older cross-signed root.

But if one just gets a new certificates on 2021-01-10, it will last until 2021-04-10, so the workaround gets less then 5 months of an extra time. I do not see why they bother with that at all. They could just continue to issue the current setup for few more months and then switch to the new root.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 6, 2020 23:45 UTC (Fri) by dtlin (subscriber, #36537) [Link] (3 responses)

Before 2021-01-11: As a Let's Encrypt client, you can make an educated guess, but you might not be able to accurately measure the impact of the root change on your infrastructure or userbase without actually making the change.

Before 2021-09-01: You can get certificates with either the old or the new root. This gives you nine months to incrementally identify and fix any issues that arise, instead of everything breaking at once.

Issued certificates won't be different

Posted Nov 8, 2020 2:24 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (2 responses)

The leaf certificates aren't different. Nothing about the Let's Encrypt certificates you get per se will change in January next year.

In both cases the leaf certificate will say issuer was "Let's Encrypt Authority X3" and be signed by the corresponding key.

You can take such a certificate, issued by Let's Encrypt today, or last year, and you can present it with a "chain" containing the certificate for Let's Encrypt Authority X3 signed by IdenTrust's "DST Root CA X3" _or_ the one signed by ISRG (Let's Encrypt)'s own ISRG Root X1. These certificates are both for the same CA key (a key used only for this Let's Encrypt intermediate) but they're signed differently and so they will be trusted by different clients.

In practice you could even present both. Most modern clients will cheerfully consume two, three, maybe even fifteen extra certificates you say might be relevant, chuck them all into a stew and try to figure out if any combination of them results in trusting your leaf certificate. But the TLS 1.2 (and earlier) RFCs say not to do this, and there are certainly some clients (probably not web browsers) that would freak out. It's conceivable that even when updated to TLS 1.3 (which endorses this already widespread practice) those clients explode if given anything that isn't strictly a join-the-dots exercise. Computing trust paths correctly is hard, ask teams at Mozilla, Chromium, Apple... What people often think of as a "tree" is actually a directed graph, and not even a DAG, it's full of weird cycles, and almost cycles and strange dangling bits and special cases.

Anyway, obviously assembling that "chain" file by hand is not what you want. So ACME clients, such as Certbot and acme.sh and doubtless dozens of more or less capably hand-rolled scripts at medium sized hosting companies around the world, are supposed to use the protocol to go fetch themselves a pre-baked chain, and what changes in January is that Let's Encrypt clients that don't offer any opinion about which chain they need will be given the ISRG Root X1 chain, not the DST Root CA X3 chain.

On top of setups that break because they didn't make a choice when they should have (some of which need to migrate off Let's Encrypt to survive the expiry of DST Root CA X3) there are also going to be setups that actually ignored ACME and just pasted in a certificate that seems to work. Those won't break... until DST Root CA X3 expires. Then, they still won't break in popular web browsers, because all the popular web browsers have smart(ish) trust path computation and will discard their proposed path to DST Root CA X3 because that's expired and useless, inserting instead a working path to ISRG Root X1. Remember the leaf certificate is the same, so it's not a security problem it's just a graph problem. But a lot of crappy "I built this with OpenSSL back in 2004 and it seems like it works?" tooling will blow up on that day and so will default browser on millions of obsolete Android phones.

Anybody who knew which they wanted can (should) specify before then and get what they want. The 11 January change is for the vast majority of people who have no idea this is a thing, to give as many of them as possible an opportunity to discover they have a problem and need to make plans to fix it properly, while a workaround is still possible.

Issued certificates won't be different

Posted Nov 9, 2020 16:48 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (1 responses)

> On top of setups that break because they didn't make a choice when they should have (some of which need to migrate off Let's Encrypt to survive the expiry of DST Root CA X3) there are also going to be setups that actually ignored ACME and just pasted in a certificate that seems to work.

My understanding is that:

1. While it is possible to do this, LE certificates expire really fast (90 days), so you either need to keep renewing by hand or build some sort of automation for it.
2. Certbot requires less effort than either of those options; it is the path of least resistance.
3. Certbot uses ACME and will pull some kind of trust chain for you, whether you expected that to happen or not.

> But a lot of crappy "I built this with OpenSSL back in 2004 and it seems like it works?" tooling will blow up on that day and so will default browser on millions of obsolete Android phones.

I agree that lots of crappy *client*-side tooling will break, but I'm not 100% convinced of the server-side problem. Sure, there will be a few NIH setups that try to do some weird curl/OpenSSL nonsense rather than just using Certbot or another ACME client, but I find it hard to believe that this will be widespread.

Issued certificates won't be different

Posted Nov 10, 2020 4:22 UTC (Tue) by tialaramex (subscriber, #21167) [Link]

It's not so much about humans pasting stuff in blindly as machines (under instruction of humans) doing so.

There are a *lot* of other ACME clients besides Certbot. Some of them implement the entire ACME spec. correctly so far as we know - but if the programmer who built your client only cared about getting a working configuration out then "Just paste in this boilerplate that seems to work" is easier than "Read and understand Link alternate headers in HTTP responses" even though the latter was required by the specification. I believe ACMEv2 (the API documented in the RFC and deployed today, but not the first one launched with Let's Encrypt) makes it pretty much easier to just Do The Right Thing™ here but I am not as certain as I'd like to be that everyone does so.

One thing I got wrong which might help is that apparently ISRG intends to default to Let's Encrypt R3 (from their new hierarchy) sooner rather than later. If anybody is just pasting in either version of Let's Encrypt Authority X3 blindly that will break when the certificate was actually issued from Let's Encrypt R3. A compliant ACME implementation will be given the appropriate version of the Let's Encrypt R3 certificate instead of course.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 0:58 UTC (Sat) by fountain (guest, #142949) [Link] (1 responses)

They talk about this in their forum post: https://community.letsencrypt.org/t/transition-to-isrgs-r...

They don't expect the Android situation to improve significantly over the next year, which means sites that need to support old Android will have to move to a paid CA. So they're deliberately triggering the breakage early, while there's still an easy workaround, to make sure those sites have time to transition.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 8, 2020 9:00 UTC (Sun) by patrakov (subscriber, #97174) [Link]

What's the situation with the Buypass CA on old android clients?

https://www.buypass.com/ssl/products/acme

Smart move!

Posted Nov 7, 2020 9:08 UTC (Sat) by oldtomas (guest, #72579) [Link] (5 responses)

Plausibly deniable way to send users up the upgrade treadmill.

C'mon, Android users! Throw away your devices, again!

Smart move!

Posted Nov 7, 2020 11:19 UTC (Sat) by smurf (subscriber, #17840) [Link] (4 responses)

Yeah, annoying and all that, but the idea that the LE people are in any way responsible for that effect is disingenious IMHO.

Smart move!

Posted Nov 7, 2020 14:03 UTC (Sat) by oldtomas (guest, #72579) [Link] (3 responses)

That's the plausibly deniable part of it.

Smart move!

Posted Nov 7, 2020 15:02 UTC (Sat) by clump (subscriber, #27801) [Link]

It's ridiculous to suggest Let's Encrypt of all efforts would be party to any planned obsolescence initiatives on the part of Android, hardware manufacturers, or carriers. There's much to be improved in the mobile phone ecosystem, but Let's Encrypt wouldn't be where I'd start.

Smart move!

Posted Nov 7, 2020 17:35 UTC (Sat) by smurf (subscriber, #17840) [Link]

The idea that LE actively supports such a move is not plausible.

Smart move!

Posted Nov 9, 2020 8:45 UTC (Mon) by tim_small (guest, #35401) [Link]

Never ascribe to malice, that which can be reasonably explained by incompetence.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 13:52 UTC (Sat) by rsidd (subscriber, #2582) [Link] (6 responses)

"Use Firefox" is excellent advice actually. I have disabled Chrome on my android phone and some strange issues (unwanted popup ads even when not web browsing) completely went away. But why can't "use an updated Chrome" be the answer, ie why can't Google push the cert update via a Chrome update?

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 18:44 UTC (Sat) by josh (subscriber, #17465) [Link]

I wouldn't be surprised if many of the un-updatable devices also don't have any kind of decent web browser. Some of them will still have a strange vendor browser, or the stock Android browser from older Android, or some other weird and likely insecure thing.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 7, 2020 22:27 UTC (Sat) by ibukanov (subscriber, #3942) [Link] (2 responses)

Chrome on Android uses certificates from the OS.

Chrome Root trust

Posted Nov 8, 2020 1:25 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (1 responses)

Chrome has announced plans to change that. https://www.chromium.org/Home/chromium-security/root-ca-p...

Obviously they can't do anything about it on iOS where everything is actually just Apple's Safari anyway, but on other platforms they apparently have decided Google should operate a real root trust programme like Mozilla's.

Chrome Root trust

Posted Nov 8, 2020 16:42 UTC (Sun) by nilsmeyer (guest, #122604) [Link]

This usually isn't as much of an issue on iOS because Updates are supplied for a longer time (over 5 years now for some models).

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 15, 2020 20:22 UTC (Sun) by jhhaller (guest, #56103) [Link] (1 responses)

My sister-in-law had the problem of unwanted popup ads, and it was the result of accepting some page's request for allowing notifications. Turned off those notifications in Chrome, and the phone worked properly after that. I've used Firefox on Android, and not all mobile web sites don't seem to test it well (or at all). Quora's commenting/answering textbox doesn't work properly on Firefox. At first, I thought that was a less-than-subtle hint that they really want people to use their App, but the textbox works fine in Chrome. I'm not sure if the problem is in Quora's code, or in Firefox, but not interested enough to get the two of them to work out where the problem is.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 15, 2020 22:31 UTC (Sun) by mpr22 (subscriber, #60784) [Link]

I think there is a general problem with terribly clever javascript text editors and Firefox for Android, since on my phone using FF for Android I have weird glitchiness with both AO3 and two web forums I frequent that I do not encounter on my desktop PC running plain old desktop Firefox.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 9, 2020 10:51 UTC (Mon) by Lennie (subscriber, #49641) [Link] (2 responses)

These older devices/browser already were having issues, they throw errors because they only support older TLS protocols which aren't supported anymore by a bunch of servers.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 10, 2020 4:31 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (1 responses)

Lollipop (5.0) onwards does TLS 1.2 and so do many (but not all) 4.4 builds. So that's a large population and it's also in that sweet spot of devices that probably don't seem that old to their users, where this won't be "one last straw" after lots of other things broke already.

Fallout from upcoming Let's Encrypt certificate changes

Posted Nov 10, 2020 15:40 UTC (Tue) by Lennie (subscriber, #49641) [Link]

Ahh, OK, so their is an other cut off point.

Not surprised of course, but good to know.

Thanks!

A reason not to force https on clients

Posted Nov 12, 2020 11:23 UTC (Thu) by mildred593 (guest, #107325) [Link] (2 responses)

I already had this problem on a previous phone that I bought around 2010 and stopped working a few years ago. it was running Android Gingerbread 2.3 and could not update. Some sites stopped working because the TLS protocols were out of date.

This is where I don't agree with websites that forces https over http. I agree this is a security in case you don't pay attention, but it forces planned obsolescence on old devices. Whenever you want it or not, TLS is part of that and constant security upgrades blocks old clients. When all you want is to read a README on a particular GitHub repository, being told that the page cannot load, is not very nice when you know that with plain http it would have worked.

So https everywhere, yes but only enforced on the client, not on the server.

A reason not to force https on clients

Posted Nov 12, 2020 12:09 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

"Unable to update your apps, you need to update Play Store first"

"Insufficient space to update Play Store, please delete your apps to make room"

That was my old Samsung Galaxy Ace.

Cheers,
Wol

A reason not to force https on clients

Posted Nov 12, 2020 14:54 UTC (Thu) by geert (subscriber, #98403) [Link]

It's hard to believe, and amazing, that my first Android phone (ZTE Racer running Android 2.1) had a separate 12 MB partition as a temporary space to download APKs to. Of course it didn't take long for that to become a severe limitation, not only for installing new apps (IIRC, the Skype APK was the first one to surpass 12 MB), but also for keeping apps up-to-date.