Garrett: We need better support for SSH host certificates
The main problem is that client tooling just doesn't handle this well. OpenSSH has no way to do TOFU for CAs, just the keys themselves. This means there's no way to do a git clone ssh://git@github.com/whatever and get a prompt asking you to trust Github's CA. Instead, you need to add a @cert-authority github.com (key) line to your known_hosts file by hand, and since approximately nobody's going to do that there's only marginal benefit in going to the effort to implement this infrastructure. The most important thing we can do to improve the security of the SSH ecosystem is to make it easier to use certificates, and that means improving the behaviour of the clients.
Posted Mar 24, 2023 20:44 UTC (Fri)
by kokkoro (guest, #139153)
[Link] (2 responses)
If you're using CAs you're probably a large organization and you would want to manage the CA trust explicitly rather than have every employee TOFU the CA.
For example, it would be pretty easy for GitHub to add a one-liner for users to add their CA to known_hosts, and that would be safer than TOFU anyway. Compared to TOFUing individual self-generated host keys, TOFUing a CA is much riskier.
Posted Mar 24, 2023 23:52 UTC (Fri)
by anselm (subscriber, #2796)
[Link]
Where I work we use step-ca, which helps automate dealing with SSH certificates (among other things). When I want to connect to a remote host using SSH for the first time in the morning, I authenticate myself with an OpenID Connect identity provider (could be Google, but in our case it's a simple Django application that talks to our LDAP directory), and, on the strength of that, step-ca issues me with an SSH client certificate that is good for 24 hours. I can use this certificate to SSH into our servers until it expires, at which point I need to re-authenticate using the OIDC IdP. Similarly, the remote servers get their host certificates from step-ca, and these are also updated quite frequently.
To bootstrap this, I need to point the system to the step-ca instance and provide that instance's “fingerprint” for authentication; my SSH client is then set up automatically. It is a convenient system that makes it unnecessary to copy public keys around and also deals with the problem that SSH public keys basically live forever.
Posted Mar 27, 2023 16:35 UTC (Mon)
by floppus (guest, #137245)
[Link]
That depends what you mean by "TOFUing a CA". If that means "trusting this CA to issue certificates for any hostname", the way web browsers do, then yes, that's horribly dangerous.
If it means "trusting this CA to issue certificates for the hostname I'm currently connecting to", that doesn't seem any different from "a one-liner for users to add their CA to known_hosts". And for ssh to do that itself, on first use, would be decidedly safer than requiring users to copy and paste a command they don't understand.
Posted Mar 24, 2023 20:55 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
One my pet peeve is AWS not allowing SSH CAs for the user keys. It would simplify a lot of key management for EC2 instances, but AWS actually made it impossible by adding validation to prohibit the ssh-ca keys.
They can fix that with several lines of code, but I've been trying to reach the appropriate people inside the AWS org for several years with 0 result.
Posted Mar 24, 2023 21:27 UTC (Fri)
by flussence (guest, #85566)
[Link] (2 responses)
I've never looked at the certificate route, is that something it would fix?
Posted Mar 24, 2023 21:43 UTC (Fri)
by geofft (subscriber, #59789)
[Link] (1 responses)
If you really wanted to go this route, you could add some additional secret that you create that GitHub has to prove to you - but this just goes further down the road of shared secrets, which Matthew is basically arguing is the whole problem here, and introduces symmetric crypto with inherent syncing problems. From experience at multiple sites that use Kerberos, which relies on symmetric crypto, key distribution is a giant pain and you basically need to copy private key material around when you have a fleet of servers load-balancing the same hostname, which puts you at immense risk of this sort of incident.
Instead, you want to lean harder on asymmetric crypto, add another layer of indirection, and have some sui-generis CA that signs the GitHub sshds, each one of which has its own key, and have the sshd send you the certificate and ask you to TOFU the CA's public key instead of the individual server's public key. Then each individual server can have its own keypair, the CA only needs to be brought online when provisioning a new server, and the only thing that needs to be sent across GitHub's internal network is public keys and public signatures of keys.
Posted Mar 24, 2023 22:07 UTC (Fri)
by flussence (guest, #85566)
[Link]
The load-bearing word here is “previously”: keys don't exist in a conceptual vacuum, there's a time component by way of them actually being used. A shared secret could be negotiated at first connection and stored near-but-separate-from authorized_keys/known_hosts for later use. An attacker trying to impersonate the server would then have to obtain that value for each user as well as the server's private key.
It only protects against the one specific case of accidentally posting a private key in public, but apparently that's a common enough one that there's several official and unofficial bots constantly monitoring GitHub to catch people doing it.
Posted Mar 24, 2023 22:19 UTC (Fri)
by kjhambrick (guest, #23704)
[Link]
> There's no individual at fault here - there's a series of design decisions that made it possible
We need more of this attitude in the world.
-- kjh
Posted Mar 25, 2023 7:53 UTC (Sat)
by epa (subscriber, #39769)
[Link] (1 responses)
Posted Mar 27, 2023 14:38 UTC (Mon)
by mgedmin (subscriber, #34497)
[Link]
Posted Mar 25, 2023 17:09 UTC (Sat)
by apoelstra (subscriber, #75205)
[Link] (16 responses)
mjg has a point in general, that in most cases everybody just hits "yes" on the TOFU prompt and doesn't even look at the fingerprint, but I think in this case probably a lot of people did do the right thing.
Posted Mar 25, 2023 18:37 UTC (Sat)
by epa (subscriber, #39769)
[Link] (15 responses)
Posted Mar 26, 2023 1:11 UTC (Sun)
by NYKevin (subscriber, #129325)
[Link] (14 responses)
1. The private key has been compromised, and an attacker is trying to rotate victims to using a key that is not known to the original issuer, in order to stymie mitigation efforts. In this case, we should not accept the rotation, because it makes a bad situation worse. In an ideal world (i.e. not the world we currently live in), we would not accept the original key at all, because it would somehow be revoked.
The proper solution is a real PKI, with a real revocation system, and no more of this TOFU nonsense. TOFU should be for my toy computer in the other room (that doesn't have a real DNS entry), not for serious usage like GitHub.
Posted Mar 26, 2023 6:14 UTC (Sun)
by epa (subscriber, #39769)
[Link] (13 responses)
I agree that there’s still a risk of it being an attacker (your first possibility) and perhaps from a strict point of view the risk is just as bad as if the attacker didn’t have the old private key. I think that judgement is one for the end user to make. Ssh should show the information — is this a straightforward key mismatch, or a requested ‘rotation’? — and then the user can decide whether to treat those scenarios differently.
Posted Mar 27, 2023 3:58 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link] (12 responses)
When you have dug a hole of this magnitude, the correct solution is to put down the shovel. No more TOFU. No more ad-hoc SSH keying (except perhaps for toy systems). HTTP transitioned to free, automated, and centralized PKI many years ago. There is no logical reason that SSH is unable to do the same, aside from technical and political inertia. If you really want to avoid using "real" CAs, then you use DANE or something like that. Regardless, TOFU should be obsolete.
Posted Mar 27, 2023 7:41 UTC (Mon)
by anselm (subscriber, #2796)
[Link]
I'd say in a corporate environment the solution is to use SSH host certificates. In that case, the only thing you need to deploy to everyone's workstations is a known_hosts file containing the public key of your (SSH) CA, which is much less of a maintenance nightmare.
Posted Mar 27, 2023 10:36 UTC (Mon)
by mb (subscriber, #50428)
[Link] (3 responses)
I don't see a problem with TOFU.
The real problem is the lack of an automated mechanism to revoke accepted keys. But that has nothing to do with TOFU.
Posted Mar 27, 2023 16:44 UTC (Mon)
by nix (subscriber, #2304)
[Link] (2 responses)
The CA ecosystem makes me shiver. My local SSH key distribution network is a very simple thing involving AuthorizedKeysCommand and curl and private keys on yubikeys and is easy to understand and 100% entirely under my control, and can be used equally easily for machines on the public DNS and machines that are not. It does not make me shiver. Frankly even putting the private keys on a local disk seems a lot less terrifying to me than relying on the snake-infested nightmare zone that is the global PKI infrastructure.
Posted Mar 27, 2023 18:35 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link] (1 responses)
OTOH, TOFU is basically the equivalent of grabbing a cup of water out of a river, eyeballing it to make sure it looks vaguely clean-ish, and hoping for the best. You probably won't get sick. I mean, lots of animals drink out of that river, right?
Posted Mar 27, 2023 18:39 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link]
Posted Mar 27, 2023 11:42 UTC (Mon)
by kleptog (subscriber, #1183)
[Link] (6 responses)
What I'd like to see is that I can have a private CA stored in an ansible vault, and from there generate the host keys. It turns out there is an actual community.crypto.openssh_cert ansible plugin which looks like it could do the job. The key must be in a separate file which I hope can be encrypted, the docs don't say. When googling for it, google gives me 10(!) hits. I guess this would be number 11.
Maybe someone who has made it work can write an LWN article about it.
Posted Mar 27, 2023 15:42 UTC (Mon)
by liw (subscriber, #6379)
[Link]
Posted Mar 27, 2023 16:21 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
It's actually super-easy. Much simpler than SSL for HTTPS.
LWN should write an article about it! I can also write a post about it.
Posted Mar 27, 2023 16:43 UTC (Mon)
by corbet (editor, #1)
[Link] (2 responses)
Posted Mar 27, 2023 21:04 UTC (Mon)
by psoberoi (subscriber, #45666)
[Link]
Posted Mar 28, 2023 10:40 UTC (Tue)
by kleptog (subscriber, #1183)
[Link]
I'm in a team managing 100+ machines with a tool like ansible. What's the best practice for managing the SSH CA host certificates such that (1) users don't get the warnings from SSH, (2) multiple people can do the deployments (or perhaps even automated by a buildbot), (3) secure storage of any secrets.
I have found examples for the management of *user* SSH certificates, for example [1]. Maybe it's so obvious it doesn't get written?
[1] https://engineering.fb.com/2016/09/12/security/scalable-a...
Posted Mar 29, 2023 19:00 UTC (Wed)
by michelr (subscriber, #129677)
[Link]
The host certificate is generated at first boot and updated daily, with an expiration date 5 days after certificate creation. A Salt Pillar, implemented in Python, acts as CA, with the private key of the CA stored on the Salt Master and the host public key is added in a Salt Grain.
Especially with continuously adding and removing systems in a large environment, host certificates are really beneficial: no need to update 3000 servers if we add one.
Posted Mar 25, 2023 18:54 UTC (Sat)
by iabervon (subscriber, #722)
[Link]
In GitHub's example, you've almost certainly copy-pasted the repository URL from the website, which you've already decided to trust, and you can almost certainly also copy-paste another string from the same site almost as easily as typing "yes".
Posted Mar 25, 2023 23:09 UTC (Sat)
by songmaster (subscriber, #1748)
[Link] (4 responses)
GitHub used to do exactly this. My known_hosts files on various different $work machines contained multiple copies of their old public key for a series of nearby IP addresses in more than one block, and yes I do remember having to accept them as new server IPs were added.
I was able to find and delete the host from the file by hand because the entries weren’t hashed. Hashing the known_hosts file entries seems to have come in fairly recently without my reading about it, and this event seems to show a disadvantage of doing that. Is it possible to find and remove duplicate entries in the same way when they are hashed?
Posted Mar 25, 2023 23:44 UTC (Sat)
by ABCD (subscriber, #53650)
[Link] (3 responses)
It appears you can use ssh-keygen -R hostname to remove all keys belonging to the specified hostname from known_hosts. The man page notes that this option is useful to delete hashed hosts.
Posted Mar 26, 2023 0:51 UTC (Sun)
by songmaster (subscriber, #1748)
[Link] (2 responses)
What would be needed is a command that finds all the public host keys in the file that match the key of the named host, and deletes them too.
Posted Mar 26, 2023 13:21 UTC (Sun)
by Smon (guest, #104795)
[Link]
Posted Mar 26, 2023 13:45 UTC (Sun)
by apoelstra (subscriber, #75205)
[Link]
From a user POV this part of things was pretty bad.
Posted Mar 26, 2023 19:53 UTC (Sun)
by cjcox (guest, #60378)
[Link] (9 responses)
Posted Mar 27, 2023 12:02 UTC (Mon)
by ballombe (subscriber, #9523)
[Link] (8 responses)
The simplest alternative (generating a new key each time) leads to teaching users to ignore the ssh warning, which is arguably worse.
As for your main point, github belongs to Microsoft now, so they have to follow their security posture, what ever it is.
Posted Mar 27, 2023 16:00 UTC (Mon)
by walters (subscriber, #7396)
[Link]
Posted Mar 27, 2023 21:09 UTC (Mon)
by Vipketsh (guest, #134480)
[Link] (6 responses)
These sentences sound great and seemingly solve the problem, but in reality they just pass the responsibility around. I've never seen any explanation of what to actually *do* when you get the warning. It's easy to say "the user needs to check", but check what exactly ?
If by some miracle the key/fingerprint is found to be incorrect (let's say it's evidence of a man-in-the-middle) what is the user to do ? Clearly the user wanted to use the service so just give up ? Try to contact someone ? Who exactly ? We all know that there is never anyone to contact and especially anyone who would give a reasonable answer in reasonable time.
Posted Mar 28, 2023 1:22 UTC (Tue)
by NYKevin (subscriber, #129325)
[Link] (5 responses)
1. The key got changed for a legitimate reason. The people who run the service will be very aware of the warning, and so they will not do this if they can avoid it. If it's unavoidable, then they will (or at least, should) proactively put up warnings on their website, which you can consult over HTTPS.
As I mentioned in another comment chain, (1) is a solved problem for HTTPS keys, and from a technical perspective, it would be relatively straightforward to adapt that solution to the SSH use case, but (apparently) some people disagree with doing that. OTOH, (2) will never be possible to solve with technology. The tech can tell you that it is happening, but it cannot tell you what to do about it, because this is a social problem, not a technical problem.
Posted Mar 28, 2023 1:33 UTC (Tue)
by NYKevin (subscriber, #129325)
[Link]
To further expound on this, because I was a little too terse: While it is likely possible to circumvent an MitM attack through the clever use of tunneling, VPNs, Tor, etc., the fundamental issue is that you have been given an insecure internet connection. Someone decided to give you that insecure internet connection. That person is the problem. Remove them from a position of authority over your internet connection, and the problem goes away.
Posted Mar 28, 2023 6:47 UTC (Tue)
by zdzichu (subscriber, #17118)
[Link]
The platform guys checked the situation, basically by going to Github's blog and reading about the change. Then provided company-wide heads-up on Slack. With guidelines how to remove old keys and check the fingerprints of new ones.
I strongly commend the first developer, his behaviour was professional and an example for others.
Posted Mar 28, 2023 9:39 UTC (Tue)
by Vipketsh (guest, #134480)
[Link] (2 responses)
> 1. The key got changed for a legitimate reason.
So there is no problem and the warning is useless and the right thing to do is just bypass it.
> 2. The key got changed because someone is MitM'ing you.
Great, so as per your assessment this can be:
> ISP
Easy! I'll get the CEO replaced tomorrow. How hard could this ever be ?
> IT department
Support desks are known to be very responsive, to the point, give detailed explanations, and fix things immediately. Especially when it comes to security. A support desk brick walling people ? Never heard of it! Very easy to solve, clearly.
> government
No problem! I'll just get a new government installed tomorrow, make the appropriate law changes, issue the right directives and problem solved. It would take me a whole day ? I must be slow.
In summary: it's great that you know you are the target of a MitM attack, but reasonably you have two choices: (i) disconnect from the internet and go live in a cave or (ii) just suck it up and hope for the best. Option (i) isn't reasonable in today's world so that leaves one with (ii) and blindly ignoring warnings.
I'm not saying that this is problem is immaterial to talk about or to try to solve, but it is also quite disingenuous to waive it away with "it's [the user's] social problem" and blame users when they end up ignoring warnings. If anything, the social problem is more on the service providers' side who often don't inform users of problems/changes, put those explanations in places you have to hunt for it and/or try as hard as they can to build a wall between them and those "pesky lusers".
Posted Mar 28, 2023 15:59 UTC (Tue)
by NYKevin (subscriber, #129325)
[Link]
Yes I did. I told you to go read the organization's website, which would have worked in the GitHub case.
> In summary: it's great that you know you are the target of a MitM attack, but reasonably you have two choices: (i) disconnect from the internet and go live in a cave or (ii) just suck it up and hope for the best. Option (i) isn't reasonable in today's world so that leaves one with (ii) and blindly ignoring warnings.
There is also (iii) stop trying to use the service and go do something else instead, or (iv) use it and accept that you don't have any security guarantees.
> blame users when they end up ignoring warnings.
Nowhere in my comment did I blame the user. Please do not put words in my mouth. My position is that this is not a problem the technology is capable of solving. That doesn't make it the user's fault.
Posted Mar 28, 2023 17:49 UTC (Tue)
by geert (subscriber, #98403)
[Link]
You are accessing github at work, so clearly this must be related to your work. So you are interacting with github projects to create products for your customers, or for use in your internal infrastructure. I guess at least someone at a higher level in your company must care if your product or internal infrastructure would become compromised?
Posted Apr 2, 2023 1:56 UTC (Sun)
by IanKelling (subscriber, #89418)
[Link] (1 responses)
echo @cert-authority github.com >> ~/.ssh/config git && git clone ssh://git@github.com/whatever
He didn't bother to ask openssh why they didn't do that. Or github. And he didn't propose any plan for getting the work done. Cue captain hindsight meme.
Posted Apr 2, 2023 12:50 UTC (Sun)
by IanKelling (subscriber, #89418)
[Link]
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
> for a bad outcome to occur, and in a better universe they wouldn't have been necessary. Let's
> work on building that better universe.
Git over https
Git over https
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
2. The private key has been compromised (or is just old), and the original issuer is trying to rotate potential victims to a safe key. In this case, the rotation should succeed, but from the perspective of the victim, who is to say whether we're in this case or the previous one?
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Sprinkling CA trust chains does not really solve any real world problem that TOFU doesn't also practically solve. And it's so much simpler.
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
I don't have Ansible generate host keys. I have more complicated setup:
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
You mean we should maybe write an article like this one?
Articles
Articles
Articles
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
are stored in a GIT repository. So the pressure is great to store the secret key in the GIT repo so that the configuration system can install it. Matthew correctly note that there are no simple, robust alternatives provided by ssh.
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
2. The key got changed because someone is MitM'ing you. Then you need to go convince your ISP, IT department, and/or government to stop MitM'ing you.
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates
>
> Support desks are known to be very responsive, to the point, give detailed explanations, and fix things immediately. Especially when it comes to security. A support desk brick walling people ? Never heard of it! Very easy to solve, clearly.
Garrett: We need better support for SSH host certificates
Garrett: We need better support for SSH host certificates