Preventing brute force ssh attacks

Yeah. What I'm really interested in here is being confident that an 
intruder who steals the private key of someone with login rights to my 
system cannot use it to log in... but I suppose even passphrases won't 
help there, as if they can steal a key they can almost certainly get root 
and install a keylogger, and the passphrase is toast.