I put ssh on port 23 to avoid log spam. I am lazy like that.
I use ssh keypairs with passphrase (which is actually 3DES encryption of the private key..) so
if I 'misplace' the key then I am not much worse off then if I just used plain passwords. Then
I disable the use of ssh passwords.
On my desktop (er laptop) I use as my base of operations. I realy don't use any other
computer. So it's the only one that I keep my ssh keys on. On my laptop I have zero network
services (well, other then avahi).. so it's secure. Don't even have sshd running.
For file transfer I just use sshfs.. no samba or nfs for my stuff.
So for my personal stuff brute-force is impossible and the chances of obtaining a key is slim
to none for a attacker. Not also to mention that most of this stuff is firewalled from the
internet via a pc-based router.
for multi-user environment though, especially corporate-land stuff, then passwords are still a
way of life. Oh well.