SSH: passwords or keys?

In general, yes. *Strictly*, no.

Here is an example of when I have used a non-passphrased key. It may seem contrived now, but it was in real use at the time:

Back in ye days of dial-up, I had one machine with a modem in it, connected to the phone line. Dial-on-demand was not an option, as the line was also used for voice, so we needed more control about when to connect, so that left the problem of how to initiate (and terminate) a connection from any other machine. The simplest solution was to use a passphraseless SSH key, permitted to perform both of those tasks and nothing else. None of the users (read: my family) used SSH for anything else, so using an agent would be indistinguishable from not having one.

So, what's the extent of the possible damage?

If somebody had broken into the house and stolen one of the computers with the key on, then they would have gained the ability to connect to the internet the next time they broke in, without having to bring their own modem or subvert the machine plugged in to the phone line. I wouldn't consider that a particularly pressing concern given that *there's somebody in my house dismantling my computers*.

I suppose the most obvious counter-argument is that this is a task which could easily have been done using something other than SSH, but it was still the simplest solution.