How should interrupts be balanced?

The programmable interrupt controller on modern (SMP) hardware can be set up to route different interrupts to different processors. When properly programmed, the APIC can help system performance by having each interrupt be handled by the processor which is best suited to the job. At the moment, however, there is not much agreement on how the kernel should be programming the APIC.

The 2.5 kernel contains (for the x86 architecture, at least) an in-kernel interrupt balancing routine. It runs as a separate kernel thread ("kirqd") which wakes up every so often and tries to arrange things so that each processor handles approximately the same interrupt rate. If that can't be done (if, for example, most interrupts come from a single source), interrupts are slowly rotated between the processors. This approach works reasonably well much of the time, but it can fail badly for certain loads.

In particular, the interrupt balancer has trouble with networking loads. The networking code goes out of its way to avoid hardware interrupts - when thousands of packets per second are passing through the system, you don't want the network interface bugging you for every one of them. So a great deal of kernel work may result from a single network interface interrupt. To a simple interrupt balancer, which tries to equalize interrupt counts across a system, a processor handling a heavy networking load may look relatively idle. That processor may find that it gets to deal with a SCSI interface as well, even though it is already overloaded. Even worse, a router could end up with multiple interfaces being handled by a single processor, which still looks lightly loaded.

One can certainly imagine ways to tweak the in-kernel interrupt balancer to make it deal properly with the networking case. But many developers believe that IRQ balancing belongs in user space. A user-space solution can contain whatever complexity is needed to make the right sort of decisions; it also, of course, allows site administrators to set their own policies.

A user-space interrupt balancing daemon exists now; it can be downloaded from Arjan van de Ven's web site. The current implementation is relatively simple, depending mostly on interrupt counts like the in-kernel balancer. It does, however, take pains to distribute interrupts from each type of device across processors. That technique will help network routers, since it will at least keep different interfaces on different processors. But the real point is that this policy can be enhanced and customized as needed.

There is some disagreement about moving interrupt balancing to user space. According to some, only the kernel has the knowledge and the ability to react quickly enough to create optimal interrupt routings. But, chances are that user space will be the eventual home for this task. The real question may be whether the in-kernel interrupt balancer is removed before 2.6.0 comes out.

---

(Log in to post comments)