LWN: Comments on "Juggling software interrupts and realtime tasks"

Juggling software interrupts and realtime tasks

Cyberax — Sun, 08 Jan 2023 23:40:53 +0000

Bluetooth can be low-latency. Just check the gamer-oriented peripherals.

Juggling software interrupts and realtime tasks

rep_movsd — Sun, 08 Jan 2023 15:40:31 +0000

Speaking about Android and audio latency, BlueTooth, which most people are opting for now, completey screws latency.

Try using some Piano/Keyboard app with a bluetooth headphone - literally unusable

I dont see why Bluetooth has to be so bad

Juggling software interrupts and realtime tasks

excors — Wed, 21 Dec 2022 16:54:49 +0000

> Interrupt priority levels aren't supported in x86??

I'm not an expert but after reading a bit, I think it's roughly:

They are supported by the hardware, but Linux doesn't make much use of them.

On x86-64, there is TPR (task-priority register; writeable) and PPR (processor-priority register; read-only; basically the max of TPR and the currently-active interrupt's priority). An interrupt's 8-bit vector number is used as its priority. The top 4 bits are the priority class. The CPU will only handle an interrupt with a priority class higher than PPR. That means an interrupt can only be preempted by one of higher priority than itself, and TPR can be used to mask all interrupts up to a given priority (e.g. a kernel critical section can disable most interrupts, but still allow preemption by the highest-priority ones to meet certain real-time requirements).

As far as I can tell, x86-64 made TPR more easily accessible (through the CR8 register) but the basic functionality is the same as any x86 since Pentium (specifically ones with APIC).

When an interrupt occurs, the CPU will push EIP/EFLAGS/etc to the stack and clear the IF flag (to disable all interrupts) before running the handler. The handler obviously needs to preserve any other registers on the stack before clobbering them. The handler may do some work and then choose to set IF, to allow itself to be preempted by a higher-priority interrupt. Or it may choose not to. It sounds like Linux used to support nested interrupts but removed it in 2010, because of the complexity of preventing stack overflows: https://lwn.net/Articles/380931/ . (...except the posts in that article make it sound like interrupts on the same vector can nest, and I thought PPR would prevent that? I'm probably confused about something.)

For comparison, I think ARMv7-A/ARMv8-A are similar to x86, except the CPU does not push anything onto the stack - it just copies PC and status register (CPSR/PSTATE) into some special registers. The handler must preserve those on the stack before allowing preemption.

ARMv6/7/8-M are different: they don't disable interrupts before running the handler, so all handlers are preemptible by default. The CPU pushes several registers onto the stack on exception entry, and it can even be preempted in the middle of that entry sequence. That allows lower-latency handling of high-priority interrupts.

Juggling software interrupts and realtime tasks

farnz — Wed, 21 Dec 2022 10:30:56 +0000

The TMS9900 has only three on-chip registers: Program Counter (PC), Status Register (SR), and Workspace Pointer (WP). It then has instructions that operate on 16-bit registers, which are memory relative to the address in WP - so R0 is actually the two bytes at WP and WP + 1, while R12 is the two bytes at WP + 24 and WP + 25. All ALU operations work on numbered registers, not on the three internal registers, and indirect addressing is also relative to the WP relative registers, not one of the internal three. PC and SR work like in other processors; WP has a dedicated load-immediate and store to register instruction pair, plus can be modified by BLWP (branch and load workspace pointer) and RTWP (return with workspace pointer)

An equivalent behaviour on a modified 6809 would be if changing the Direct Page register also changed D, X, Y, U and S - i.e. changing DP changes all but PC and condition code registers in the 6809.

Juggling software interrupts and realtime tasks

jem — Wed, 21 Dec 2022 08:15:08 +0000

The Motorola 68k has the MOVEM instruction which takes a list of registers (a bitmap) to copy to/from consecutive memory locations. The same mnemonic is used for both directions; the assembler determines the variant based on the operands. The instruction does not use a dedicated base register for this, but you can choose from a range of addressing modes for the memory operand. The 68000 is a CISC processor, after all.

Juggling software interrupts and realtime tasks

mpr22 — Tue, 20 Dec 2022 20:00:37 +0000

That's very definitely not the Motorola 68000 (the A24D16-bussed 32-bit microprocessor from 1979 with three zeroes in its model number).

It sounds like the Motorola 6809, an enhanced version of the Motorola 6800 (the A16D8-bussed 8-bit microprocessor from 1974 with two zeroes in its model number).

The 6800's actual registers (8-bit accumulators A and B; 16-bit index register X; 16-bit stack pointer SP; 16-bit program counter PC; 8-bit status register with the two high bits hard-wired to 1) were on-chip. However, it had a "zero page" addressing mode for memory access, which took an 8-bit address and zero-extended it to 16 bits. The speed penalty for going to RAM was a lot lower then than it would be today, so you could use the zero page as "pseudoregisters".

Among other enhancements, the 6809 relabelled the "Zero Page" addressing mode as "Direct Page" and added an 8-bit Direct Page register that controlled the high eight bits of the address presented on the bus when using that addressing mode.

Juggling software interrupts and realtime tasks

Wol — Tue, 20 Dec 2022 11:32:16 +0000

Sounds like a feature I believe the 68000 had would not go amiss here ...

I seem to remember it stored its registers in main ram, with a register that specified the base address. Surely it must have cached them in the cpu for speed, but changing that base register would have done a "flush and restore", so you could simply assign a block of ram as a register stack.

Cheers,
Wol

Juggling software interrupts and realtime tasks

calumapplepie — Mon, 19 Dec 2022 18:48:29 +0000

Interrupt priority levels aren't supported in x86??
That's an interesting choice, given that I've been working with the decade-old PIC32MX processor, which support 7 levels of interrupts-interrupting-interrupts (ie, a handler for priority-1 interrupts can be interrupted by one for priority-2, so on and so forth).

On reflection, the answer to the issue seems clear; while some PIC32 processsors have a second set of registers for holding the interrupted execution context, most don't. That means all interrupts need to save and load every CPU register to memory when starting and stopping; which is a pretty significant chunk of overhead. Having multiple interrupt levels isn't much of a cost on top of that, then.

But I'll bet money that x86 doesn't make that 'shadow register set' optional. And while the simplest AMD64 processor is miles ahead of the most sophisticated PIC32, a complete set of registers is still not a small addition; and why bother having multiple extra sets if one will do?

Juggling software interrupts and realtime tasks

opalmirror — Fri, 09 Dec 2022 01:09:56 +0000

Takes me back. My first post-college job in the late 1980s was helping port V7 UNIX (designed to run on a 68000 board with a segment MMU) to a 68010 (with support for restarting instructions on bus errors - so that sbrk() and brk() could be handled automatically and gracefully without special coordination with the C compiler). We ended up having some complex scheduling and input needs so we ported in bits of BSD into it including a bespoke implementation of select(), the full BSD socket API and network stack. I suppose we should have just tried porting BSD in the first place. It was lots of fun though. Not any hard real-time tasks for that project, but it controlled devices that were hard real-time in hardware.

Juggling software interrupts and realtime tasks

willy — Tue, 06 Dec 2022 03:48:06 +0000

Oh, what I meant to say is that traditional Unix didn't need to distinguish between hard and soft interrupt context. That very document talks about the priority level of interrupts (spl4() thought spl7()). That meant that higher priority interrupts could interrupt lower priority interrupts. As far as I know, x86 didn't (and maybe still doesn't) support that. So Linux disables all interrupts while processing any interrupt, which means that all interrupt routines must be fast and defer anything time-consuming to softirq context (which can be interrupted by hardirqs)

Juggling software interrupts and realtime tasks

willy — Mon, 05 Dec 2022 23:46:27 +0000

Ouch! That's a huge PDF; some warning would have been nice!

Relevant paragraph:

> A number of subroutines are available which are useful to character device drivers. Most of these handlers, for example, need a place to buffer characters in the internal interface between their ‘‘top half’’ (read/write) and ‘‘bottom half’’ (interrupt) routines. For relatively low data-rate devices, the best mechanism is the character queue maintained by the routines getc and putc.

The top half is the bit called in process context. The bottom half is the bit called in interrupt context. Linux's use of bottom half is entirely consistent with that.

Juggling software interrupts and realtime tasks

pabs — Mon, 05 Dec 2022 07:55:59 +0000

I wonder if this Linux patchset would help with the PipeWire issue I had on Debian; choppy audio under high CPU load. I never had this with PulseAudio though.

https://wiki.debian.org/PipeWire#choppy_audio_on_systems_...

Juggling software interrupts and realtime tasks

error27 — Mon, 05 Dec 2022 04:21:10 +0000

For my workload it was the IO scheduler which lead to skips. Switching to BFQ solved the problem.

It might not fix your problem but it's at least easy to try:
sudo echo bfq > /sys/block/sda/queue/scheduler

Juggling software interrupts and realtime tasks

klossner — Sun, 04 Dec 2022 21:25:56 +0000

pre-Linux Unix systems often included the concept of a "bottom half" as a way of deferring work that could not be done in an interrupt handler.

Actually, pre-Linux systems called this the "top half"; the "bottom half" was the interrupt handler. See e.g. UNIX PROGRAMMER’S MANUAL Seven Edition. This reflected the classic stack model: hardware at the bottom, user code at the top. When I first worked on Linux mumble years ago, this was a quite a source of confusion.

Juggling software interrupts and realtime tasks

roc — Sun, 04 Dec 2022 04:21:26 +0000

For many years I've had glitch-free audio playback on Fedora except when the system is under very heavy load.

Juggling software interrupts and realtime tasks

mss — Sat, 03 Dec 2022 14:33:11 +0000

I don't remember ever having a problem with smooth audio playback on Linux desktop or laptop in the last decade.
Even though most of these systems have (had) "spinning rust" HDDs, which are known for their high latency.
But I only use raw ALSA, not stuff like PulseAudio.

Something has to be seriously broken in Fedora then.
Or your system is running out of memory, has crazy high CPU load or is forced to run at a very low power state for some reason.

Juggling software interrupts and realtime tasks

kaali1 — Sat, 03 Dec 2022 00:45:48 +0000

Are the hardware buffer so small on laptop/desktop that the kernel has deadlines missed (underrun) due to software interrupt?
While Android has low latency audio with 2/4ms deadlines, for music playback (especially on a laptop) a much deeper buffer should be used (for both power and performance reason).
I don't know much about the x86 audio subsystem, is the audio ring buffer between north bridge and the ADSP in the south bridge not shared memory? Which would allow for a big enough buffer size (100ms+) to not be impacted by the kernel scheduling (at least for non latency sensitive use case).

Juggling software interrupts and realtime tasks

jhoblitt — Fri, 02 Dec 2022 17:58:28 +0000

It sure would be nice if my fedora desktops/laptops had as smooth audio playback as my android phones... It seems like every fedora kernel update is a gamble on this front (although, the source of the glitches is no doubt a broad number of subsystems).