you don't have to 'wait' if you are willing to trust the /dev/urandom source, you only have to wait if you insist on getting a 'guarantee' that you are getting true randomness as opposed to just the best that the system can do.
The system will use the hardware RNG to modify the output of /dev/random and /dev/urandom, the only thing is that it doesn't increment the "guaranteed randomness available" counter.
realistically, I don't think it makes sense for anyone to depend on /dev/random and the 'available entropy' counter, the few cases that have super strict randomness guarantee requirements should just have a separate source of randomness
think of this as improving the quality of /dev/urandom rather than failing to get the full randomness of the hardware RNG