An attacker that gets access to the private key corresponding to a public key in an
authorized_keys file on your system will have access to it *if and only if that key is
unpassphrased or they determine the passphrase*.
i.e. at worst passphrased keys are as insecure as passwords, and they're much more secure if
the remote host is not compromised itself.
(Unpassphrased keys should only be used inside a trust boundary, and even there with care:
it's best to use an SSH agent instead, and passphrase everything, but that might be tricky if
some of the keys are used by daemons, who can't reasonably ask a human to provide a passphrase
to the agent: if they provide the passphrase themselves, that passphrase becomes as tappable
as the private key, so passphrasing becomes pointless.
It would be nice if SSH had a way to refuse entry to unpassphrased keys, or if I had a way to
determine that the private key corresponding to some public key were unpassphrased, so I could
audit authorized_keys files for unpassphrased keys and remove them. Of course the ability to
do that would itself be an information leak with security consequences...)