True. Perhaps a better way of saying it is that keys which allow the
carrying out of functions which you do not want a random thief to be able
to carry out, or keys which allow anything (J. Random Normal SSH Identity)
should be passphrased. The rest don't need to be, because nothing bad will
happen if random people get the ability to do whatever that key allows.
(Also, keys stored in a location where the key can't be stolen, e.g. in a
Mars rover, are probably safe nonpassphrased. :) )