Eliminating rwlocks and IRQF_DISABLED
Posted Dec 3, 2009 18:59 UTC (Thu) by adamgundy (subscriber, #5418)
Posted Dec 3, 2009 19:01 UTC (Thu) by kjp (subscriber, #39639)
I'm REALLY looking forward to threaded interrupts. Right now, our network server spends so much time in hard and softirq (napi) that is makes the scheduler make really bad decisions (scheduler appears to not factor in processor interrupt usage).
Posted Dec 6, 2009 16:25 UTC (Sun) by kleptog (subscriber, #1183)
The only response we got from kernel developers was that "round-robin IRQs suck" which completely sidesteps the point that what happens now doesn't work at all, and any perceived suckyness of round-robin IRQs would at least be spread evenly.
Threaded IRQs do seem to be the solution here, I hope they are implemented soon.
Posted Dec 6, 2009 16:36 UTC (Sun) by dlang (subscriber, #313)
Posted Dec 6, 2009 16:47 UTC (Sun) by kleptog (subscriber, #1183)
What I would like is for the IRQs to be distributed over a few CPUs (say 4 CPUs that share the same L3 cache). Anything to avoid all traffic being handled by one CPU.
Posted Dec 7, 2009 11:26 UTC (Mon) by xav (subscriber, #18536)
Posted Dec 7, 2009 11:56 UTC (Mon) by kleptog (subscriber, #1183)
But it's worse than that, it's not *an* app, there are several apps which all need to see the same data and since they are running on different CPUs you're going to get a cache bounce for each one anyway.
What you're basically saying is: round-robin IRQ handling is bad because you're sometimes going to get 6 cache-bounces per packet instead of 5. BFD. Without round-robin IRQs if the amount of traffic doubles we have to tell the client we can't do it.
The irony is, if you buy a more expensive network card you get MSI-X which gets the network card to do the IRQ distribution. You then get the same number of cache bounces as if we programmed the IO-APIC to do round-robin, but the load is more evenly distributed. So we've worked around a software limitation by modifying the hardware!
I'm mostly irked by a built-in feature of the IO-APIC being summarily disabled on machines with 8+ CPUs with the comment "but you don't really want that" while I believe I should at least be given the choice.
Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds