LWN: Comments on "Moving interrupts to threads"

Why threaded interrupt handlers are important

abacus — 2011-04-10T15:01:33+00:00

Don't these threaded interrupt handlers have even more overhead than workqueues? Workqueues are at least handled sequentially in the same thread; using threaded interrupt handlers means there are even more context switches.

I'm afraid that you have missed a key point about real-time systems. Design of a hard real-time system starts with deciding which threads in the system will run at real-time priority (SCHED_FIFO or SCHED_RR) and also at which priority (1..99). This step involves all threads in the system, including threaded interrupt handlers. By assigning a higher priority to certain application threads than the threaded interrupt handlers it becomes possible to obtain a system with low and bounded event response time. Handling some of the interrupt work via workqueues instead of threaded interrupt handlers would result in a larger event response time.

Moving interrupts to threads

shishir — 2008-12-12T00:21:01+00:00

"The top half executes with interrupts disabled, so it is imperative that it do as little as possible to keep the system responsive."

I somehow am confused about this line. Isn't this true only for the interrupts registered with IRQF_DISABLED/SA_INTERRUPT? Any interrupt registered using IRQF_SHARED/SA_SHIRQ can be interrupted by other interrupts.

Moving interrupts to threads

HalfMoon — 2008-10-22T22:27:37+00:00

This will be handy for interrupts issued from I2C and SPI chips, too ... their handlers mostly need to run in task context, since they've got to block while their messages get to the head of the I/O queue and also while waiting for the response. Example: IRQ comes in, block until the IRQ status mask comes back from the chip. (Even when that serial bus was idle when the IRQ came in, the handler must block.)

It's not clear how useful it will be for chained handlers though. As in, an I2C chip may have a few dozen internal IRQ sources, each of which needs to be dispatched as an IRQ in its own right. Not only will the top level dispatcher need to be a thread (possibly shared by the handlers for those internal sources) ... but all the irq_chip methods to update IRQ masks and trigger modes for those internal IRQ sources will need to run in threads, too...

Moving interrupts to threads

neli — 2008-10-16T14:42:56+00:00

Moving interrupts to threads

MisterIO — 2008-10-11T16:34:04+00:00

I always consider the "'this good code' will lead to bad code" argument a
really bad argument. If you consider "this code" to be good and useful,
then you should merge it. If then someone tries to merge "bad code" which
uses some of the features of the "good code", just don't merge the "bad
code".

Moving interrupts to threads

jake — 2008-10-10T14:53:45+00:00

> I don't quite understand, exactly how does threaded interrupt handlers
> reduce the time needed to be spent with interrupts disabled? Couldn't a
> traditional interrupt handler also immediately just ack the interrupt and
> re-enable interrupts?

Sorry I wasn't more clear in the article. Without threaded interrupts, there is no mechanism for the rest of the work to get done without being explicitly arranged for by the top half (i.e. tasklets or workqueues). This necessitates locking and synchronization between the two pieces. Workqueues have a fair amount of overhead and tasklets must still run atomically (i.e. no sleeping, etc.) as well.

Threaded interrupts provide the mechanism for additional work to be done outside of the top half, more or less automatically.

Hopefully I didn't botch this explanation as well :)

jake

Moving interrupts to threads

jlokier — 2008-10-10T14:05:54+00:00

That's correct.

Also threaded interrupts can sleep, take locks and allocate memory as normal threads.

Moving interrupts to threads

jzbiciak — 2008-10-10T13:21:37+00:00

To distill what you just said: The existing bottom halves aren't actually full kernel threads, and therefore aren't subject to the other scheduling mechanisms you might use to prioritize handling.

Did I understand you correctly?

Moving interrupts to threads

jlokier — 2008-10-10T12:55:42+00:00

I think the main _intended_ difference from the old top/bottom-halves is that threaded interrupts are subject to the normal scheduling heuristics.

So if you have some device which is dominating the machine, like say a network device flooded with tiny packets at line rate, in the old scheme userspace would effectively lock up, and in the new scheme, the network device's handlers will get their fair share of the CPU according to the scheduler.

(This is not a great example because many of the popular network drivers have another mechanism to limit their effect already. But the new scheme is more controlled.)

It also means you can manage the priority of interrupts using thread priorities (RT and non-RT, and "nice"), task control groups, and other things used to manage threads, and they can be dispatched to under-utilised CPUs more efficiently, and interact with thermal and power management.

The need to have quick_check_handler separate from the main handler seems to be due to shared interrupt lines. Without quick_check_handler, multiple devices on the same line would need to share the same handler thread, and interfere with each other.

It's possible to abuse quick_check_handler and write code in the same form as the old top/bottom halves. There are probably some latency-sensitive devices where it's essential to do something more than just acknowledging and disabling the device interrupt. However, that's not the intended way of using it, and if a device is latency sensitive, it should probably use a high RT priority on its handler thread instead. If the thread scheduler is too slow at this, it may be possible to tune this particular path through the scheduler, taking a short cut from irq to task, but retaining the standard scheduling semantics.

Moving interrupts to threads

jzbiciak — 2008-10-10T12:10:06+00:00

Other than my terse reply (which only answers "why can't I reenable interrupts in my top half?"), I must say, after re-reading the article, I too am not clear on what the difference is here. The current top half looks similar in principle to the quick_check_handler and the current bottom half looks like the handler thread.

Is it just that even more work is pushed out of the top half into the bottom half via action flags in struct irqaction? I guess I don't know enough kernel internals to appreciate the significance of this change, or why the current top half/bottom half structure couldn't do the same. Is this just a cheap event queuing construct that gets around the "can't sleep" aspect of interrupt handler top halves?

Moving interrupts to threads

jzbiciak — 2008-10-10T04:53:31+00:00

No notion of nested top halves?

Moving interrupts to threads

magnus — 2008-10-09T21:46:58+00:00

I don't quite understand, exactly how does threaded interrupt handlers reduce the time needed to be spent with interrupts disabled? Couldn't a traditional interrupt handler also immediately just ack the interrupt and re-enable interrupts?

The only reason I can come up with is that it is an API issue, i e interrupts are re-enabled when the handler returns and the call to schedule the bottom half has to be done inside the handler..

Moving interrupts to threads

jengelh — 2008-10-09T15:48:42+00:00

The fun thing here is that you can renice the interrupt threads.

Moving interrupts to threads

bronson — 2008-10-09T01:32:56+00:00

With each kernel release, there are usually one or two features that illustrate how nice it is to not have to worry about ABI/API stability.

This sounds like a neat idea and well worth the potential code churn. I look forward to hearing more news as this develops.