Heuristics for software-interrupt processing

You're pointing out the fundamental issue with using shared threads to do deferrable, unattributed, work. We see this in ksoftirqd. We see it in workqueues. These threads process generic work that have no associated priority. There is no single policy that one can apply that doesn't degrade some other use-case.

Suppose you have an application thread that contends with ksoftirqd for a CPU, and you feel your application is more important than network processing. You elevate the priority of your application thread over ksoftirqd. To your dismay, you discover that your application is now starving out timers, because thread wakeups are handled by ksoftirqd as well. This may be an unacceptable trade-off, so you submit and let the ksoftirqd network work stomp on your application thread. (I've been forced to make this trade-off half a dozen times.)

Possibly one way forward is to assign CPU-scheduler-oriented priority (i.e., don't use a flag like HI_SOFTIRQ) to all work ksoftirqd work items, and let ksoftirqd's priority float with the work. Adding priority inheritance is the next reasonable step. This is a lot like how some microkernel services work. This would require an API change and would probably be super disruptive!

Maybe another approach would be to assign one thread to each tasklet-class (HI_SOFTIRQ, TIMER_SOFTIRQ, ...). This would at least let one decouple timer processing from network processing.