Actually I wouldn't say you're entirely safe if the server is vulnerable and you're not.
There's still the issue of the host key, which is used to prevent the bad guys from
pretending to be the server. If that host key is compromised, then someone can pretend
to be the server. Then you're in a little trouble if they can also get your public key (it's
treated as public, shouldn't be horribly hard), and more trouble if you're using password