If every task releases wakeup sources when it's idling, average number of wakeups goes down.
But if you have lots of tasks doing this at some specific times and especially re-acquiring the wakeup sources at the same time (display unblanking is a good example of where that could happen in mobile devices), you get the "thundering herds" issue. System gets occasional small near-freezes i.e. the latency issue you mention.
I've seen this with user-space processes, but I guess it applies also to kernel tasks...