There is no one setting that is OK for everyone. The most important thing to consider is that hi-fi audio processing can take a lot of time, and the only real improvements available for, say, convolutions are SSE/SSE2 processing that can operate 4 or 2 floats at a time.
Unfortunately for kernel, it can't do floating point mathematics. That limits kernel's operations to integer-only stuff, which is unable to do it well enough. 32-bit integers aren't wide enough for running 96 dB SNR convolutions, so no serious audio freak would accept such a compromise.
So, to summarize: you can't solve this problem from kernel in fashion that is acceptable for everyone.