Yes, a very high level of paranoia is justified when dealing with RNGs that are going to be used in crypto.
Depending on the type of cryptography (and mode of operation) being used, a flawed RNG will allow an attacker to derive some key material or even the entire key at once.
That does mean you need a strong RNG at all times to be safe, not just when generating persistent keys. In practice, it means you have to avoid like the plague any crypto that depends on the quality of the RNG on *both* sides to avoid disaster...
I will leave it to an expert to explain which classes of crypto algorithms and modes of operation are more resistant to bad RNGs.