Eliminating rwlocks and IRQF_DISABLED [LWN.net]

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 6, 2009 16:47 UTC (Sun) by kleptog (subscriber, #1183)
In reply to: Eliminating rwlocks and IRQF_DISABLED by dlang
Parent article: Eliminating rwlocks and IRQF_DISABLED

Sure, it makes it so the CPU with 90% usage is not always the same one, but jumps around every now and then. That doesn't actually solve the problem, actually it makes it worse because then you can't use CPU binding on other processes to avoid them landing on the unlucky CPU.

What I would like is for the IRQs to be distributed over a few CPUs (say 4 CPUs that share the same L3 cache). Anything to avoid all traffic being handled by one CPU.

to post comments

Eliminating rwlocks and IRQF_DISABLED

Posted Dec 7, 2009 11:26 UTC (Mon) by xav (guest, #18536) [Link] (1 responses)

i don't think it's such a good solution. If all interrupts are related (say, you're receiving data from the network and must process it in an app), distributing the interrupts from CPU to CPU means the application must follow, and with it its cache, so you'll have lots of cache line bouncing, which is expensive.

Round-robin IRQs

Posted Dec 7, 2009 11:56 UTC (Mon) by kleptog (subscriber, #1183) [Link]

Since you cannot run the app on the same CPU as the one receiving the interrupts you're going to get a cache bounce *anyway*, right?

But it's worse than that, it's not *an* app, there are several apps which all need to see the same data and since they are running on different CPUs you're going to get a cache bounce for each one anyway.

What you're basically saying is: round-robin IRQ handling is bad because you're sometimes going to get 6 cache-bounces per packet instead of 5. BFD. Without round-robin IRQs if the amount of traffic doubles we have to tell the client we can't do it.

The irony is, if you buy a more expensive network card you get MSI-X which gets the network card to do the IRQ distribution. You then get the same number of cache bounces as if we programmed the IO-APIC to do round-robin, but the load is more evenly distributed. So we've worked around a software limitation by modifying the hardware!

I'm mostly irked by a built-in feature of the IO-APIC being summarily disabled on machines with 8+ CPUs with the comment "but you don't really want that" while I believe I should at least be given the choice.