Im not sure I accept that ssh does either. If you access a remote host by FQDN, then the host name is what's stored in known_hosts, along with the public key (at least, this seems to be so for my ssh, which is OpenSSH_5.5p1). ssh *can* cache an IP address, to be sure, but for people making use of the DNS, I'm not sure it does.
Similarly, once you tell the browser to cache a certificate, the certificate has the FQDN for which it's valid embedded inside itself (as the CN). That certificate, cached in a trusted cache though it be, can't be used to authenticate another site, even one using the same keypair (which shouldn't happen).