Quite so, ssh-agent is lovely and things like keychain make it usable :)
(Of course it won't protect you if the machine on which the agent is running is rooted: they
could keylog you, install a malicious agent, et seq ad nauseam. But it's useful if machines
you connect to from the agent machine are compromised.)