LWN: Comments on "A realtime developer's checklist"

"priority"

neilbrown — Sat, 05 Dec 2020 01:55:56 +0000

With tongue firmly in cheek, I might suggest that "priority" relates to "prior" meaning "before". And numbers "before" zero are normally negative.

So the first priority has a numerical priority of "-1". This is highest.
My second priority would have the value "-2", which isn't quite as high.

A positive value suggests a "laterity", meaning it can be left until later.

"priority"

dfsmith — Fri, 04 Dec 2020 21:10:31 +0000

Is there a reason that the priority (English word) is sometimes inverted from the English meaning of "priority"? It seems that "importance" would be less ambiguous. B-)
Examples:
"safety is our number one priority" -> set priority to 99.
"snacks are low priority" -> set priority to 1 ("low"/"high" priority works!)
"our second priority is to ensure lunch is ready" -> set priority to 98.
"coffee is way down our priority list" -> set priority to 10 (and ignore the yelling).

musl C and sched.h

mrybczyn — Fri, 04 Dec 2020 07:37:15 +0000

Thank you for the clarification!

musl C and sched.h

Hello71 — Thu, 03 Dec 2020 03:05:19 +0000

This is not accurate. musl implements the majority of the sched_* functions, including sched_setaffinity, sched_getaffinity, sched_get_priority_max, sched_get_priority_min, sched_getcpu, sched_rr_get_interval, and sched_yield. The only ones not implemented are sched_{get,set}{param,scheduler}. As explained by musl author Rich Felker at https://www.openwall.com/lists/musl/2016/03/01/4, this is because on Linux, these syscalls operate on a per-thread basis, but POSIX requires them to operate on a per-process basis. However, as he explains, the fix is to simply use pthread_{get,set}schedparam instead.

A realtime developer's checklist

azz — Fri, 20 Nov 2020 16:13:28 +0000

And in particular, given the present circumstances, real-time music collaboration applications like Jamulus. I've been using Jamulus a lot over the last few months with an RT-patched kernel, and I've seen several non-technical users enthusing lately about JamulusOS (an Ubuntu Studio-based distribution that seems to be pretty decent at getting low-latency/jitter behaviour out of machines that Windows' audio drivers struggle on).

A realtime developer's checklist

glenn — Thu, 19 Nov 2020 18:56:18 +0000

Yes, it's more about predictability than latency, but optimizing latency can improve predictability (or allows one to achieve predictability on less powerful hardware).

Paul McKenney has a great paper on this: "Real Time" vs "Real Fast": How to Choose?".

A realtime developer's checklist

adam820 — Thu, 19 Nov 2020 15:23:44 +0000

Any kind of audio processing app, something like Ardour (JACK). It's hard to play guitar, process the effects, and hear the output if the output lags behind your string hit by half a second. It'd be like talking and hearing your speech played back a half second behind; it's hard to do.

A realtime developer's checklist

hummassa — Thu, 19 Nov 2020 14:03:11 +0000

WRT RT, it's not the length of the deadline, but its predictability :-)

A realtime developer's checklist

mathstuf — Thu, 19 Nov 2020 13:04:03 +0000

Is instant messaging typically RT? Wouldn't the typing latency be the highest thing? Or is this more about "Typing…" notifications? Things interacting with hardware (vsync, constant FPS targets, machine controllers) or wetware deadlines (audio processing, audio/video sync) certainly make sense, but I would have thought typing would have interfered too much. Though if it is, it's almost certainly got longer deadlines than other use cases(?).

Note, I don't do RT development myself, so any misconceptions or naivete is certainly on my side :) .

A realtime developer's checklist

hummassa — Wed, 18 Nov 2020 19:27:19 +0000

Videoconference applications.
VoIP (voice over Internet Protocol)
Online gaming.
Community storage solutions.
Some e-commerce transactions.
Chatting.
IM (instant messaging)

A realtime developer's checklist

meyert — Wed, 18 Nov 2020 18:15:57 +0000

So what is a typical realtime application for Linux? Can you give an example of an Linux realtime "killer" app?

musl C and sched.h

mrybczyn — Wed, 18 Nov 2020 05:35:47 +0000

Thanks for the comment (I'm the author of the article).

Musl does include the sched_* functions, but they are not really implemented. Please check out https://git.musl-libc.org/cgit/musl/tree/src/sched/sched_... for example. It just returns an error. This is what the discussion covered.

A realtime developer's checklist

glenn — Tue, 17 Nov 2020 17:20:42 +0000

> So you're quite happy for a nuclear power plant to blow up because an operator wanted to play a shoot-em-up game on the same computer?

Actually, no. Quite the opposite. I make no such argument.

Mixed-criticality scheduling is a real field of study in real-time systems. It has wide applications in hard real-time environments. In the scheme I described, where SCHED_FIFO can preempt SCHED_DEADLINE, threads with hard real-time requirements are scheduled by SCHED_FIFO, while threads with soft real-time requirements (bounded deadline tardiness, in this case), are scheduled by SCHED_DEADLINE. These ideas are backed by solid real-time scheduling theory.

Mixed-criticality scheduling is not widely supported outside of research kernels, but it is becoming a centerpiece feature of the seL4 microkernel.

Let me defend my specific claim that the SCHED_DEADLINE admissions tests is not useful for many users. An admissions test is useful in an "open system," where the sysadmin (or system designer) cannot predict or control what what is run under the SCHED_DEADLINE policy. If I had to guess at who could benefit from an open linux-rt system, I would guess multi-media and gaming. Contrast this with a "closed system," where the sysadmin/system-designer controls everything that runs on the system. Who are these users? Robots (including autonomous vehicles), high-frequency trading systems, and I would argue pro-audio tools. Admissions tests are less useful in a closed system because the system-designer can account for CPU requirements ahead of time. After all, they already know enough about their execution time requirements provide execution-time parameters to SCHED_DEADLINE. In short, there is no need for an admissions test in a closed system. And this is where it gets in the way of more robust real-time scheduling: Linux could support a flavor of mixed-criticality scheduling if it did away with the admissions test that no one is really using!

Please note that elimination of the admissions test does not mean eliminating budget enforcement. Budget enforcement can still be used to isolate run-away threads.

A realtime developer's checklist

Jonno — Tue, 17 Nov 2020 16:21:57 +0000

> We can't do this today with the current Linux scheduler, since SCHED_DEADLINE preempts SCHED_FIFO. This could be done if we were able to reorder the relative priority of the scheduling classes, or simply let SCHED_FIFO preempt SCHED_DEADLINE. This would weaken (violate) the SCHED_DEADLINE admission test guarantees, but I find this feature less compelling than support for a flavor of mixed-criticality.

In an ideal world, the scheduler would be able to tell whether it would be possible to meet all deadlines even if no SCHED_DEADLINE tasks gets scheduled until the next scheduler tick, and if so give priority to SCHED_FIFO, while still giving priority to SCHED_DEADLINE when necessary to meet deadlines.

Unfortunately that would require doing admission control every time the scheduler has to pick a task (rather than just when changing the scheduling policy of a task) which would probably have a prohibitively high overhead.

A realtime developer's checklist

gus3 — Tue, 17 Nov 2020 15:46:43 +0000

> So you're quite happy for a nuclear power plant to blow up because an operator wanted to play a shoot-em-up game on the same computer?

If the operator wants to play a shoot-em-up while on duty, then the priority inversion is taking place within the operator, not the computer.

> mixed criticality would be lovely, but not at the expense of true real-time operation.

If the constraints of RT can be satisfied in two different domains, whether cores or hosts, it's still RT. Just because they have different scheduling requirements, doesn't make either of them "fake RT".

A realtime developer's checklist

Wol — Tue, 17 Nov 2020 14:21:13 +0000

> This would weaken (violate) the SCHED_DEADLINE admission test guarantees, but I find this feature less compelling than support for a flavor of mixed-criticality.

So you're quite happy for a nuclear power plant to blow up because an operator wanted to play a shoot-em-up game on the same computer?

The whole point of Real Time is that deadlines are deadlines.As in "you're dead if they're missed".

I agree mixed criticality would be lovely, but not at the expense of true real-time operation.

Cheers,
Wol

musl C and sched.h

binkley — Tue, 17 Nov 2020 13:05:13 +0000

Perhaps the situation is rapidly changing, but when I check the musl C source repo, I do see `sched.h`:

https://git.musl-libc.org/cgit/musl/log/?qt=grep&q=sc...

re: _a member of the audience noted that the musl C library does not implement [sched.h]_

What might be meant is that musl's implementation is incomplete for the purposes of the talk.

A realtime developer's checklist

glenn — Tue, 17 Nov 2020 03:58:41 +0000

> ...mixing scheduling classes is "strange".

On the RT kernel as-is, this is strange. However, you should be able to schedule unrelated threads on disjoint sets of CPUs under different scheduling policies (i.e., a SCHED_FIFO cluster and a SCHED_DEADLINE cluster).

There are theoretical grounds for mixing SCHED_FIFO- and SCHED_DEADLINE-style schedulers on shared CPUs. This technique is used in some mixed-criticality schedulers. These schedulers also consider the application-level importance of real-time threads, in addition to scheduling priority (application importance != scheduling priority!!). For example, a simple mixed-criticality scheduler might schedule application-critical threads with fixed-priorities (e.g., SCHED_FIFO) on single CPUs, while scheduling less-application-critical, but still real-time, threads with SCHED_DEADLINE and let these migrate among all CPUs. We can't do this today with the current Linux scheduler, since SCHED_DEADLINE preempts SCHED_FIFO. This could be done if we were able to reorder the relative priority of the scheduling classes, or simply let SCHED_FIFO preempt SCHED_DEADLINE. This would weaken (violate) the SCHED_DEADLINE admission test guarantees, but I find this feature less compelling than support for a flavor of mixed-criticality.

A realtime developer's checklist

dancol — Mon, 16 Nov 2020 21:31:15 +0000

Keep in mind that even if *your* real-time application pre-allocates and locks all memory, the *kernel* code you invoke via system calls isn't written with the same discipline in mind. Kernel mutexes, for example, are not real-time mutexes by default, so you can still get into priority inversion situations with the rest of the system by invoking kernel code that takes locks owned by non-RT tasks -- for example, the task list lock or an inode lock. (The RT patchset is meant to mitigate this problem.)

On a vanilla kernel, you can use *some* OS services without risking priority inversion, but you have to be careful about which ones they are and audit the kernel-side code paths you invoke and make sure that one of your fancy important SCHED_FIFO threads isn't going to get blocked behind some slowpoke non-RT part of the system because the SCHED_FIFO thread made a system call that takes a lock while it's running in the kernel.

If I had my way, we'd just apply the RT patchset universally, *and* make pthreads in gblic use PI by default, *and* implement surrogate execution so that priority inheritance did something for SCHED_OTHER threads (i.e. normal non-RT processes). (Right now, priority inheritance is useless for SCHED_OTHER.) I'm not likely to get my way because my proposal would tank throughput on batch workloads, but personally, I value consistency of performance over throughput.