The realtime preemption pull request

[Posted September 20, 2024 by corbet]

On September 19, Thomas Gleixner delivered the pull request for the realtime preemption enablement patches to Linus Torvalds — in printed form, wrapped in gold, with a ribbon, as Torvalds had requested. It was a significant milestone, marking the completion of a project that required 20 years of effort. Congratulations are due to everybody involved.

Torvalds acted on the pull request the following morning.

[pull request]

to post comments

He listed himself as the commit author!

Posted Sep 20, 2024 11:19 UTC (Fri) by ryanduve (subscriber, #127786) [Link] (4 responses)

Is there an LWN story on what "PREEMPT_RT support" is and why its so important? Sounds like a huge milestone but the term is new to me.

He listed himself as the commit author!

Posted Sep 20, 2024 11:23 UTC (Fri) by lucaswerkmeister (subscriber, #126654) [Link]

There’s a big list of articles at https://lwn.net/Kernel/Index/#Realtime; https://lwn.net/Articles/951337/ is a recent article about the “last holdout” which was resolved since then.

He listed himself as the commit author!

Posted Sep 20, 2024 11:24 UTC (Fri) by corbet (editor, #1) [Link] (2 responses)

There is lots of background material available for those who would like to catch up on 20 years of history. (Sorry, don't have time for a more elaborate answer at the moment).

He listed himself as the commit author!

Posted Sep 20, 2024 11:27 UTC (Fri) by ryanduve (subscriber, #127786) [Link] (1 responses)

No need, that list is probably exactly what I was looking for! If you think this milestone deserves a summary article I would be interested to read that, too.

Thanks for the quick response.

RT history resources

Posted Oct 2, 2024 14:41 UTC (Wed) by tbird20d (subscriber, #1901) [Link]

Steve Rostedt and Sebastien Siewior gave some good talks at kernel recipes a few weeks ago, with lots of good material on the history and features of PREEMPT_RT. See https://kernel-recipes.org/en/2024/schedule/making-the-kernel-suck-less/ and https://kernel-recipes.org/en/2024/schedule/preempt_rt-over-the-years/

Will this be the default?

Posted Sep 20, 2024 14:40 UTC (Fri) by tachoknight (guest, #173552) [Link] (5 responses)

I have a couple of CNC machines that use LinuxCNC that comes with the preemptive kernel and am wondering if this will simply be what everybody gets, even my little raspberry pi, sometime soon-ish.

Will this be the default?

Posted Sep 20, 2024 14:52 UTC (Fri) by daroc (editor, #160859) [Link] (4 responses)

Realtime kernels are unlikely to become the default, simply because there's some small performance overhead from using the realtime config option. But with all the necessary code being part of the mainline kernel, it's certainly possible that some distributions might turn it on by default or make it easier to turn on.

Will this be the default?

Posted Sep 20, 2024 15:21 UTC (Fri) by eru (subscriber, #2753) [Link] (2 responses)

Isn't this useful only if you have applications that need the realtime scheduling, and have been written to use the RT kernel features?
So no benefit for most users from turning the feature on by default.

Will this be the default?

Posted Sep 21, 2024 10:02 UTC (Sat) by tekNico (subscriber, #22) [Link]

Yes to the first question, no to the second.

Will this be the default?

Posted Sep 24, 2024 9:42 UTC (Tue) by dsommers (subscriber, #55274) [Link]

Years ago (when I was involved in some QA work on the RT kernel), I did run it on my desktop. The desktop experience was generally more snappy and responsive under heavier load, typically the mouse and the graphical stack (when given RT priorities) did not have the same amount of lag.

Doing audio and video will also help a lot when those applications have RT scheduling. I would presume gaming could definitely benefit from it as well.

That said, if you enable RT scheduling on each running process .... the benefit will not be visible - in fact it can do more harm to the experienced performance.

I would expect that the next 5-10 years we will see kernel-rt being more commonly available as default in ordinary desktop environments with some performance profiles (managed by tuned-adm and similar tools). And it can definitely help those doing lots of multi-media work. MacOS is known for being quite reliable for such tasks and has for many years. The realtime kernel can definitely help closing that gap, at least to some degree.

Of course, lots of other things has also happened in the Linux world over the last 15-20 years (like Wayland, Pipewire, etc) also improving the desktop experience, as well as better hardware (moving from HDD to SSD to nvme). But I would expect that kernel-rt can be the icing on that improved experience, when properly tuned for the task at hand.

Will this be the default?

Posted Sep 20, 2024 16:27 UTC (Fri) by ukleinek (subscriber, #56625) [Link]

Note that Debian ships an rt-enabled kernel variant on amd64, arm64, armhf and i386. It's not the default, but as easy as possible if you want to dive into rt.

Where you were the 19th of September 2024?

Posted Sep 20, 2024 17:45 UTC (Fri) by alejluther (subscriber, #5404) [Link]

This is history, and both, Linus and Thomas, deserve the respect they already have. As someone working on RTLinux more than 20 years ago, this is a lesson everyone should be proud to learn: give me a challenge labelled with impossible, give me time, and give me people not interested (only) in money, and ...

Congratulations to Thomas and all those contributing to this milestone. What a team!

real time kernel: benefits

Posted Sep 20, 2024 18:49 UTC (Fri) by atai (subscriber, #10977) [Link] (15 responses)

Any concrete measurements of what difference the real time kernel provides vs. when the feature is disabled?

real time kernel: benefits

Posted Sep 20, 2024 20:03 UTC (Fri) by Vorpal (guest, #136011) [Link] (12 responses)

> Any concrete measurements of what difference the real time kernel provides vs. when the feature is disabled?

To a desktop or laptop? None whatsoever. In fact, you will get a small amount of overhead. So it is unlikely to become default for any general purpose distro. For a server? Same. Don't use it.

Realtime is about things that need *guaranteed* reliable timing. Where a lag spike is simply not allowed. Think things like medical devices, or the ABS brake controller in a car. Things that *must* respond within n microseconds to a specific input. Though for many of those use cases you wouldn't be using Linux at all, but some other smaller OS that has safety certifications (not realistic to get the Linux kernel certified, it is far to big and complex).

You *can* find it RT Linux higher level control software for vehicles and industrial equipment (things that aren't *quite* as sensitive as a brake controller). For example: CNC mills and lathes. You don't want a hang of a couple of milliseconds breaking your control loop such that you cut the wrong bit of metal, but it isn't a life and death situation. https://linuxcnc.org/ is a concrete example of open source CNC control software (don't think it is used by any commecial products, but is popular with home built CNC machines and conversions). I wouldn't be surprised if some of the commercial offerings also use RT Linux.

We use RT Linux at work (for high level control of industrial equipment, the critical things like emergency stop etc is handled by dedicated micro controllers with certified software that the high level software talks to and through).

real time kernel: benefits

Posted Sep 20, 2024 20:10 UTC (Fri) by atnot (guest, #124910) [Link] (5 responses)

> don't think it is used by any commecial products, but is popular with home built CNC machines and conversions

Some of the smaller players use it! For example Tormach's PathPilot is based on LinuxCNC.

real time kernel: benefits

Posted Sep 20, 2024 20:22 UTC (Fri) by Vorpal (guest, #136011) [Link]

Oh, cool! From the videos I have seen the open source UIs for LinuxCNC are pretty jank. So I guess they made their own UI? Which, now that I think about it, may also jank to be fair. Many industrial UIs tend to be. Even my 3D printer (a Prusa Mk3.9) has a much slicker UI than most industrial equipment I have seen.

real time kernel: benefits

Posted Nov 18, 2024 6:10 UTC (Mon) by rurban (guest, #96594) [Link] (3 responses)

I would not call the NASA Space Shuttle program a small player. The shuttle was controlled by 3 dSpace machines, ordinary AMD processors with RtLinux. RtLinux had just more and better drivers than the old simple real-time systems. Eg Gigabit Ethernet with DMA support.

Finally the seperate RtLinux is not needed anymore, it's now maintained upstream.

real time kernel: benefits

Posted Nov 18, 2024 6:44 UTC (Mon) by mb (subscriber, #50428) [Link]

> Finally the seperate RtLinux is not needed anymore, it's now maintained upstream.

No, what is maintained upstream is PREEMPT_RT (or sometimes also called Linux-RT).
RtLinux is a different thing which predates PREEMPT_RT.

real time kernel: benefits

Posted Nov 18, 2024 11:06 UTC (Mon) by malmedal (subscriber, #56172) [Link] (1 responses)

> The shuttle was controlled by 3 dSpace machines, ordinary AMD processors with RtLinux.

Uh? The shuttle predated Linux. I believe the modernised cockpit was going to use VxWorks, but the effort was canceled.

real time kernel: benefits

Posted Nov 18, 2024 12:38 UTC (Mon) by excors (subscriber, #95769) [Link]

The Shuttle also predates the company dSPACE, and I can't find any sources indicating any part of the Shuttle program ever used dSPACE products, nor any sources confirming any dSPACE products used RTLinux (though some of their modern products do use a "Linux-based real-time operating system", which I guess might mean PREEMPT_RT). So I think this is a [citation needed].

Apparently the Shuttle's on-board computers were IBM System/4 Pi (based on System/360; the name is a joke about steradians), with a purpose-built OS. The OS was written in assembler and the applications in HAL/S (based on PL/I). Each computer had 424KB memory, could do 450K operations per second, and weighed about 55kg. The Shuttle had four computers in a quadruple-redundant set, plus a fifth running backup flight software developed independently by a different company. (Source: https://klabs.org/DEI/Processor/shuttle/shuttle_primary_c...). Mission Control originally used five IBM System/370 computers (one primary, one backup, three spares) (https://waynehale.wordpress.com/2012/05/28/ground-up-rend...), though I guess that may have been updated over time as it was less safety-critical than the flight computers.

> I believe the modernised cockpit was going to use VxWorks, but the effort was canceled.

Yes, that's discussed in https://ntrs.nasa.gov/api/citations/20070018256/downloads... (which also notes, worryingly, "The project expended a fair amount of time tracking down system problems mostly due to the misuse of pointers and memcpy commands"; and I guess they got a bit carried away with C++ because "It was found that software components utilizing all [the] benefits of [Object Oriented] programming had the worst performance. Several components had to be redesigned and rewritten to fix this problem").

real time kernel: benefits

Posted Sep 20, 2024 22:36 UTC (Fri) by azz (subscriber, #371) [Link]

> To a desktop or laptop? None whatsoever.

There are quite a lot of desktop-ish applications that can benefit from RT. The most common example is anything doing low-latency audio or video - JACK, PulseAudio and PipeWire will all make use of RT scheduling if it's available, which makes it possible for them to work reliably with tiny I/O buffer sizes to minimise processing latency. Other kinds of signal capture and processing benefit from RT scheduling too - I've used it for software-defined radio, and for ld-decode RF signal capture.

I've run RT-patched kernels on my desktop and laptop machines since 2007 or so, and they've always worked really well - it's great to see the last of the work finally making it into the mainline kernel!

real time kernel: benefits

Posted Sep 21, 2024 10:07 UTC (Sat) by tekNico (subscriber, #22) [Link] (3 responses)

You are talking about hard real time while not just ignoring, but actively negating the existence of soft real time. "Don't use it" is bad advice. See another reply by aaz for more details.

real time kernel: benefits

Posted Sep 21, 2024 11:01 UTC (Sat) by mb (subscriber, #50428) [Link]

I understood is as: Don't use it for machines that only run SCHED_OTHER class.

Which is a perfectly valid statement.

But even if you have certain realtime tasks in the system, that still doesn't mean you need PREEMPT_RT.
Rule of thumb: If you don't know whether you need RT, then you probably don't need it.

PREEMPT_RT is *not* a solution that magically makes your machine realtime capable.
Realtime requires that you have a realtime concept for the whole system (including hardware).
The differentiation between "soft" and "hard" is basically useless. In practice what is really needed are numbers for latencies, deadlines and throughput that you have to achieve and then design your whole system accordingly.

real time kernel: benefits

Posted Sep 21, 2024 22:10 UTC (Sat) by simlo (guest, #10866) [Link] (1 responses)

Are we again confusing the term "hard realtime" with safety critical? For me hard realtime means simply that any deadline violations is bug for the purpose of the program. Some audio stuff might be in this category as the missed deadline can destroy a recording. Soft realtime is where the overall application can live with missed deadlines once in a while - the quality of the system simply drops. The linux kernel can with RT support both - but not safety critical, as that requires certification. But then again: Isn't SpaceX using Linux to fly their rockets and manned capsules, or is that a myth?

real time kernel: benefits

Posted Sep 21, 2024 22:35 UTC (Sat) by excors (subscriber, #95769) [Link]

> Isn't SpaceX using Linux to fly their rockets and manned capsules, or is that a myth?

That is true - see https://lwn.net/Articles/540368/ , or e.g. https://old.reddit.com/r/spacex/comments/gxb7j1/we_are_th... :

> All of our on-board computers either run Linux (with the PREEMPT_RT patch) or are microcontrollers that run bare-metal code. For applications running on Linux we are careful to set up the process and kernel thread priorities correctly to avoid priority inversions. We also generally write our code in a way that maximizes determinism, such as avoiding memory allocation at runtime or unbounded loops. Finally, we have telemetry which indicates the performance of all our processes to ensure that they always meet their deadlines across all phases of flight, even in the presence of unexpected or excess inputs.

> We handle radiation fault tolerance by having multiple computers running simultaneously and voting on their outputs. If one of the redundant computers fails due to radiation, the system is generally unaffected. The faulted computer can be rebooted and reincorporated into the system once it recovers, which restores the original fault tolerance.

But some of the most safety-critical things don't use Linux (specifically the thing that detects if the rocket went wrong and might become a risk to people on the ground, and blows it up before it leaves the exclusion zone):

> The Autonomous Flight Safety System (AFSS - it's all about safety) software runs on a set of microcontrollers independent from the flight computer. It receives sensor inputs directly (e.g. IMU measurements) as well as some computed inputs from the flight computer. A Mission Data Load configures the AFSS for which conditions might require termination of the flight, such as the rocket going way off course, losing all acceleration, etc.

real time kernel: benefits

Posted Sep 21, 2024 10:52 UTC (Sat) by mb (subscriber, #50428) [Link]

Just to avoid confusion for readers that don't know:
RTLinux and PREEMPT_RT are two completely different things.

These days LinuxCNC uses PREEMPT_RT instead of RTLinux (or RTAI; yet another completely different thing).

real time kernel: benefits

Posted Sep 21, 2024 13:11 UTC (Sat) by shef (subscriber, #91287) [Link]

4G/5G lower level radio stack requires RT to achieve proper performance (together with time sync) - Intel FlexRAN would be a good example of that

real time kernel: benefits

Posted Sep 21, 2024 13:32 UTC (Sat) by nevets (subscriber, #11875) [Link]

I always say that the biggest difference tht PREEMPT_RT gives you is bounded latency. That is, there's no "unbound priority inversion".

It also flattens the task context. Interrupt handlers and softirqs run at the same context as normal tasks. This allows other tasks of higher priority to preempt them and have better response times.

Most spin locks become mutexes and allow preemption while they are held.

I gave a talk about this back in 2007 that explains this at Ottawa Linux Symposium and most of it still applies

Delivered in real time

Posted Sep 21, 2024 6:21 UTC (Sat) by Lennie (guest, #49641) [Link]

I guess we can say he delivered the patch in real time as well. :-)

Architecture support

Posted Sep 21, 2024 6:59 UTC (Sat) by willy (subscriber, #9762) [Link]

Two of these three architectures (arm64 and riscv) did not exist when this project started.