Eliminating rwlocks and IRQF_DISABLED [LWN.net]

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 6, 2009 16:25 UTC (Sun) by kleptog (subscriber, #1183)
In reply to: Eliminating rwlocks and IRQF_DISABLED by kjp
Parent article: Eliminating rwlocks and IRQF_DISABLED

Aha, glad I'm not the only one with this issue. We recently found out that if you have more than 8 CPUs the round-robin IRQ distribution of the IO-APIC is disabled so that *all* IRQs from a network card (soft and hard) all end up on one CPU. As you point out, the scheduler doesn't handle this very well. (It was the first time I saw a CPU spending 90% of its time in kernel space while the other CPUs were almost idle.)

The only response we got from kernel developers was that "round-robin IRQs suck" which completely sidesteps the point that what happens now doesn't work at all, and any perceived suckyness of round-robin IRQs would at least be spread evenly.

Threaded IRQs do seem to be the solution here, I hope they are implemented soon.

to post comments

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 6, 2009 16:36 UTC (Sun) by dlang (guest, #313) [Link] (3 responses)

there is the userspace tool to ballance interrupts between cpus, have you tried that?

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 6, 2009 16:47 UTC (Sun) by kleptog (subscriber, #1183) [Link] (2 responses)

Sure, it makes it so the CPU with 90% usage is not always the same one, but jumps around every now and then. That doesn't actually solve the problem, actually it makes it worse because then you can't use CPU binding on other processes to avoid them landing on the unlucky CPU.

What I would like is for the IRQs to be distributed over a few CPUs (say 4 CPUs that share the same L3 cache). Anything to avoid all traffic being handled by one CPU.

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 7, 2009 11:26 UTC (Mon) by xav (guest, #18536) [Link] (1 responses)

i don't think it's such a good solution. If all interrupts are related (say, you're receiving data from the network and must process it in an app), distributing the interrupts from CPU to CPU means the application must follow, and with it its cache, so you'll have lots of cache line bouncing, which is expensive.

Round-robin IRQs

Posted Dec 7, 2009 11:56 UTC (Mon) by kleptog (subscriber, #1183) [Link]

Since you cannot run the app on the same CPU as the one receiving the interrupts you're going to get a cache bounce *anyway*, right?

But it's worse than that, it's not *an* app, there are several apps which all need to see the same data and since they are running on different CPUs you're going to get a cache bounce for each one anyway.

What you're basically saying is: round-robin IRQ handling is bad because you're sometimes going to get 6 cache-bounces per packet instead of 5. BFD. Without round-robin IRQs if the amount of traffic doubles we have to tell the client we can't do it.

The irony is, if you buy a more expensive network card you get MSI-X which gets the network card to do the IRQ distribution. You then get the same number of cache bounces as if we programmed the IO-APIC to do round-robin, but the load is more evenly distributed. So we've worked around a software limitation by modifying the hardware!

I'm mostly irked by a built-in feature of the IO-APIC being summarily disabled on machines with 8+ CPUs with the comment "but you don't really want that" while I believe I should at least be given the choice.