Yes, your perception is wrong. To build professional audio systems you need to care about latency nearly all the time. If your point was "JACK is not necessarily the right choice for playing a 'Ding!' sound when I receive new mail", you have our agreement. But that's not what JACK users are using it for.
It all matters, right down to when I hit a button I want it to take effect immediately. I understand as a programmer that it can't happen literally the moment I press the button, but as the milliseconds drag on the delay becomes perceptible, and then quickly annoying. On a typical PC JACK can get that delay down to the almost imperceptible where a typical non-realtime audio framework struggles to keep it sub-second.
Professional musicians can cope with and compensate for latency up to a certain point. Playing a huge stage, or with some types of instrument that inherently have a latency, you learn. But we're talking low tens of milliseconds at most.
JACK isn't alone here, this a long studied problem. You need buffers, and we have buffers, up to a few thousand frames (samples x channels) which equates to 100+ milliseconds - but beyond that you're just not professional audio any more. And Linux can easily stall non-realtime processes for that long.
If latency didn't matter, there'd be no difference between a realtime NLE and the old batch processed, make a decision and render systems. But in reality the former are tremendously more productive AND require a lot less practice.
[ JACK has a mode for people who don't care about time, freewheel mode, in this mode audio clock time passes as quickly as your CPU can process samples, and of course you don't use RT priority because the machine would appear to freeze indefinitely. ]