Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 3:58 UTC (Mon) by NYKevin (subscriber, #129325)
In reply to: Garrett: We need better support for SSH host certificates by epa
Parent article: Garrett: We need better support for SSH host certificates

In a corporate environment, you can have IT deploy known_hosts with known-good key values to everyone's workstations, and in that scenario you might be reasonably safe. This is still bad, because it requires significant effort on the part of your organization, and is wildly impractical for people who don't work for a corporation in the first place. But if you let end users override that with a prompt, then you're just dumping the corporate users into the "nothing is safe by default" TOFU bucket with everybody else.

When you have dug a hole of this magnitude, the correct solution is to put down the shovel. No more TOFU. No more ad-hoc SSH keying (except perhaps for toy systems). HTTP transitioned to free, automated, and centralized PKI many years ago. There is no logical reason that SSH is unable to do the same, aside from technical and political inertia. If you really want to avoid using "real" CAs, then you use DANE or something like that. Regardless, TOFU should be obsolete.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 7:41 UTC (Mon) by anselm (subscriber, #2796) [Link]

I'd say in a corporate environment the solution is to use SSH host certificates. In that case, the only thing you need to deploy to everyone's workstations is a known_hosts file containing the public key of your (SSH) CA, which is much less of a maintenance nightmare.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 10:36 UTC (Mon) by mb (subscriber, #50428) [Link] (3 responses)

>TOFU should be obsolete.

I don't see a problem with TOFU.
Sprinkling CA trust chains does not really solve any real world problem that TOFU doesn't also practically solve. And it's so much simpler.

The real problem is the lack of an automated mechanism to revoke accepted keys. But that has nothing to do with TOFU.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 16:44 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Quite. Using CAs assumes that in some meaningful sense things that CAs sign are more "trusted" than things you trust on first use. Given that nobody can name the CAs they allegedly "trust" (none of which they asked to trust, or actually trust in any conventional sense of the term), and that a whole pile of them have been found to be actually untrustworthy, and that more or less all of them except possibly LetsEncrypt are incentivized to be taken over by total scumbags and they would still be marked trusted more or less everywhere for some time after that, I might suggest that this appears to be not entirely true.

The CA ecosystem makes me shiver. My local SSH key distribution network is a very simple thing involving AuthorizedKeysCommand and curl and private keys on yubikeys and is easy to understand and 100% entirely under my control, and can be used equally easily for machines on the public DNS and machines that are not. It does not make me shiver. Frankly even putting the private keys on a local disk seems a lot less terrifying to me than relying on the snake-infested nightmare zone that is the global PKI infrastructure.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 18:35 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (1 responses)

Nobody is claiming that CAs are perfect. My contention is that, under the CA/B, trust is like tap water. You may not agree with all of the details of how it works, but in practice, it does work, and millions of people rely on it every day. It does fail, but (also like tap water) those failures are both rare and A Big Deal.

OTOH, TOFU is basically the equivalent of grabbing a cup of water out of a river, eyeballing it to make sure it looks vaguely clean-ish, and hoping for the best. You probably won't get sick. I mean, lots of animals drink out of that river, right?

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 18:39 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

I should also point out that, if you are manually checking the keys and verifying them, you aren't practicing TOFU. TOFU means "trust on first use" not "verify on first use." The problem is, most people can't be bothered to do that in practice.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 11:42 UTC (Mon) by kleptog (subscriber, #1183) [Link] (6 responses)

I've known about SSH CAs for a while but never seen them in action. The problem is it's not seen as best practice. A quick check with a few colleagues basically gave the same result: they'd heard of but never seen it.

What I'd like to see is that I can have a private CA stored in an ansible vault, and from there generate the host keys. It turns out there is an actual community.crypto.openssh_cert ansible plugin which looks like it could do the job. The key must be in a separate file which I hope can be encrypted, the docs don't say. When googling for it, google gives me 10(!) hits. I guess this would be number 11.

Maybe someone who has made it work can write an LWN article about it.

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 15:42 UTC (Mon) by liw (subscriber, #6379) [Link]

I don't have Ansible generate host keys. I have more complicated setup:

bare metal machines get a temporary host key and short-lived certificate installed with the operating system
I generate a new key and longer-lived certificate, stored in pass
I renew the certificate from time to time
Ansible installs the key and certificate, retrieving them from pass at run time
virtual machines get a host key and certificate via cloud-init, generated by my libvirt wrapper

Garrett: We need better support for SSH host certificates

Posted Mar 27, 2023 16:21 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> I've known about SSH CAs for a while but never seen them in action.

It's actually super-easy. Much simpler than SSL for HTTPS.

LWN should write an article about it! I can also write a post about it.

Articles

Posted Mar 27, 2023 16:43 UTC (Mon) by corbet (editor, #1) [Link] (2 responses)

You mean we should maybe write an article like this one?

Articles

Posted Mar 27, 2023 21:04 UTC (Mon) by psoberoi (subscriber, #45666) [Link]

Is it possible to make this link more visible (maybe as as postscript to the article)? Right now very few people will ever notice it.

Articles

Posted Mar 28, 2023 10:40 UTC (Tue) by kleptog (subscriber, #1183) [Link]

That article was a good introduction to SSH CAs, which is why I know about it. I was thinking more of the next level, i.e.:

I'm in a team managing 100+ machines with a tool like ansible. What's the best practice for managing the SSH CA host certificates such that (1) users don't get the warnings from SSH, (2) multiple people can do the deployments (or perhaps even automated by a buildbot), (3) secure storage of any secrets.

I have found examples for the management of *user* SSH certificates, for example [1]. Maybe it's so obvious it doesn't get written?

[1] https://engineering.fb.com/2016/09/12/security/scalable-a...

Garrett: We need better support for SSH host certificates

Posted Mar 29, 2023 19:00 UTC (Wed) by michelr (subscriber, #129677) [Link]

At the company I work we use Salt to manage over 3000 servers in multiple HPC clusters: permanent servers on premises and dynamically added and removed servers in the cloud. We have a Salt state for managing the global known_hosts file and the host certificate. The global known_hosts file defines the public key of the CA.

The host certificate is generated at first boot and updated daily, with an expiration date 5 days after certificate creation. A Salt Pillar, implemented in Python, acts as CA, with the private key of the CA stored on the Salt Master and the host public key is added in a Salt Grain.

Especially with continuously adding and removing systems in a large environment, host certificates are really beneficial: no need to update 3000 servers if we add one.