"Surely you would agree that latency doesn't matter at all if you are e.g. recording singing from a microphone to accompany a guitar track you are playing back."
Nope. Singers like to hear some of their voice fed back to them. It's not uncommon for unconstrained musicians to move to the corner of a room, to best hear the reflection of their voice and instrument (at low latency). If they are using headphones, the audio equipment has to provide that feedback loop. It is not an "offline" operation, where latencies could be ignored (or corrected).