real time kernel: benefits

Posted Sep 20, 2024 18:49 UTC (Fri) by atai (subscriber, #10977)
Parent article: The realtime preemption pull request

Any concrete measurements of what difference the real time kernel provides vs. when the feature is disabled?

to post comments

real time kernel: benefits

Posted Sep 20, 2024 20:03 UTC (Fri) by Vorpal (guest, #136011) [Link] (12 responses)

> Any concrete measurements of what difference the real time kernel provides vs. when the feature is disabled?

To a desktop or laptop? None whatsoever. In fact, you will get a small amount of overhead. So it is unlikely to become default for any general purpose distro. For a server? Same. Don't use it.

Realtime is about things that need *guaranteed* reliable timing. Where a lag spike is simply not allowed. Think things like medical devices, or the ABS brake controller in a car. Things that *must* respond within n microseconds to a specific input. Though for many of those use cases you wouldn't be using Linux at all, but some other smaller OS that has safety certifications (not realistic to get the Linux kernel certified, it is far to big and complex).

You *can* find it RT Linux higher level control software for vehicles and industrial equipment (things that aren't *quite* as sensitive as a brake controller). For example: CNC mills and lathes. You don't want a hang of a couple of milliseconds breaking your control loop such that you cut the wrong bit of metal, but it isn't a life and death situation. https://linuxcnc.org/ is a concrete example of open source CNC control software (don't think it is used by any commecial products, but is popular with home built CNC machines and conversions). I wouldn't be surprised if some of the commercial offerings also use RT Linux.

We use RT Linux at work (for high level control of industrial equipment, the critical things like emergency stop etc is handled by dedicated micro controllers with certified software that the high level software talks to and through).

real time kernel: benefits

Posted Sep 20, 2024 20:10 UTC (Fri) by atnot (guest, #124910) [Link] (5 responses)

> don't think it is used by any commecial products, but is popular with home built CNC machines and conversions

Some of the smaller players use it! For example Tormach's PathPilot is based on LinuxCNC.

real time kernel: benefits

Posted Sep 20, 2024 20:22 UTC (Fri) by Vorpal (guest, #136011) [Link]

Oh, cool! From the videos I have seen the open source UIs for LinuxCNC are pretty jank. So I guess they made their own UI? Which, now that I think about it, may also jank to be fair. Many industrial UIs tend to be. Even my 3D printer (a Prusa Mk3.9) has a much slicker UI than most industrial equipment I have seen.

real time kernel: benefits

Posted Nov 18, 2024 6:10 UTC (Mon) by rurban (guest, #96594) [Link] (3 responses)

I would not call the NASA Space Shuttle program a small player. The shuttle was controlled by 3 dSpace machines, ordinary AMD processors with RtLinux. RtLinux had just more and better drivers than the old simple real-time systems. Eg Gigabit Ethernet with DMA support.

Finally the seperate RtLinux is not needed anymore, it's now maintained upstream.

real time kernel: benefits

Posted Nov 18, 2024 6:44 UTC (Mon) by mb (subscriber, #50428) [Link]

> Finally the seperate RtLinux is not needed anymore, it's now maintained upstream.

No, what is maintained upstream is PREEMPT_RT (or sometimes also called Linux-RT).
RtLinux is a different thing which predates PREEMPT_RT.

real time kernel: benefits

Posted Nov 18, 2024 11:06 UTC (Mon) by malmedal (subscriber, #56172) [Link] (1 responses)

> The shuttle was controlled by 3 dSpace machines, ordinary AMD processors with RtLinux.

Uh? The shuttle predated Linux. I believe the modernised cockpit was going to use VxWorks, but the effort was canceled.

real time kernel: benefits

Posted Nov 18, 2024 12:38 UTC (Mon) by excors (subscriber, #95769) [Link]

The Shuttle also predates the company dSPACE, and I can't find any sources indicating any part of the Shuttle program ever used dSPACE products, nor any sources confirming any dSPACE products used RTLinux (though some of their modern products do use a "Linux-based real-time operating system", which I guess might mean PREEMPT_RT). So I think this is a [citation needed].

Apparently the Shuttle's on-board computers were IBM System/4 Pi (based on System/360; the name is a joke about steradians), with a purpose-built OS. The OS was written in assembler and the applications in HAL/S (based on PL/I). Each computer had 424KB memory, could do 450K operations per second, and weighed about 55kg. The Shuttle had four computers in a quadruple-redundant set, plus a fifth running backup flight software developed independently by a different company. (Source: https://klabs.org/DEI/Processor/shuttle/shuttle_primary_c...). Mission Control originally used five IBM System/370 computers (one primary, one backup, three spares) (https://waynehale.wordpress.com/2012/05/28/ground-up-rend...), though I guess that may have been updated over time as it was less safety-critical than the flight computers.

> I believe the modernised cockpit was going to use VxWorks, but the effort was canceled.

Yes, that's discussed in https://ntrs.nasa.gov/api/citations/20070018256/downloads... (which also notes, worryingly, "The project expended a fair amount of time tracking down system problems mostly due to the misuse of pointers and memcpy commands"; and I guess they got a bit carried away with C++ because "It was found that software components utilizing all [the] benefits of [Object Oriented] programming had the worst performance. Several components had to be redesigned and rewritten to fix this problem").

real time kernel: benefits

Posted Sep 20, 2024 22:36 UTC (Fri) by azz (subscriber, #371) [Link]

> To a desktop or laptop? None whatsoever.

There are quite a lot of desktop-ish applications that can benefit from RT. The most common example is anything doing low-latency audio or video - JACK, PulseAudio and PipeWire will all make use of RT scheduling if it's available, which makes it possible for them to work reliably with tiny I/O buffer sizes to minimise processing latency. Other kinds of signal capture and processing benefit from RT scheduling too - I've used it for software-defined radio, and for ld-decode RF signal capture.

I've run RT-patched kernels on my desktop and laptop machines since 2007 or so, and they've always worked really well - it's great to see the last of the work finally making it into the mainline kernel!

real time kernel: benefits

Posted Sep 21, 2024 10:07 UTC (Sat) by tekNico (subscriber, #22) [Link] (3 responses)

You are talking about hard real time while not just ignoring, but actively negating the existence of soft real time. "Don't use it" is bad advice. See another reply by aaz for more details.

real time kernel: benefits

Posted Sep 21, 2024 11:01 UTC (Sat) by mb (subscriber, #50428) [Link]

I understood is as: Don't use it for machines that only run SCHED_OTHER class.

Which is a perfectly valid statement.

But even if you have certain realtime tasks in the system, that still doesn't mean you need PREEMPT_RT.
Rule of thumb: If you don't know whether you need RT, then you probably don't need it.

PREEMPT_RT is *not* a solution that magically makes your machine realtime capable.
Realtime requires that you have a realtime concept for the whole system (including hardware).
The differentiation between "soft" and "hard" is basically useless. In practice what is really needed are numbers for latencies, deadlines and throughput that you have to achieve and then design your whole system accordingly.

real time kernel: benefits

Posted Sep 21, 2024 22:10 UTC (Sat) by simlo (guest, #10866) [Link] (1 responses)

Are we again confusing the term "hard realtime" with safety critical? For me hard realtime means simply that any deadline violations is bug for the purpose of the program. Some audio stuff might be in this category as the missed deadline can destroy a recording. Soft realtime is where the overall application can live with missed deadlines once in a while - the quality of the system simply drops. The linux kernel can with RT support both - but not safety critical, as that requires certification. But then again: Isn't SpaceX using Linux to fly their rockets and manned capsules, or is that a myth?

real time kernel: benefits

Posted Sep 21, 2024 22:35 UTC (Sat) by excors (subscriber, #95769) [Link]

> Isn't SpaceX using Linux to fly their rockets and manned capsules, or is that a myth?

That is true - see https://lwn.net/Articles/540368/ , or e.g. https://old.reddit.com/r/spacex/comments/gxb7j1/we_are_th... :

> All of our on-board computers either run Linux (with the PREEMPT_RT patch) or are microcontrollers that run bare-metal code. For applications running on Linux we are careful to set up the process and kernel thread priorities correctly to avoid priority inversions. We also generally write our code in a way that maximizes determinism, such as avoiding memory allocation at runtime or unbounded loops. Finally, we have telemetry which indicates the performance of all our processes to ensure that they always meet their deadlines across all phases of flight, even in the presence of unexpected or excess inputs.

> We handle radiation fault tolerance by having multiple computers running simultaneously and voting on their outputs. If one of the redundant computers fails due to radiation, the system is generally unaffected. The faulted computer can be rebooted and reincorporated into the system once it recovers, which restores the original fault tolerance.

But some of the most safety-critical things don't use Linux (specifically the thing that detects if the rocket went wrong and might become a risk to people on the ground, and blows it up before it leaves the exclusion zone):

> The Autonomous Flight Safety System (AFSS - it's all about safety) software runs on a set of microcontrollers independent from the flight computer. It receives sensor inputs directly (e.g. IMU measurements) as well as some computed inputs from the flight computer. A Mission Data Load configures the AFSS for which conditions might require termination of the flight, such as the rocket going way off course, losing all acceleration, etc.

real time kernel: benefits

Posted Sep 21, 2024 10:52 UTC (Sat) by mb (subscriber, #50428) [Link]

Just to avoid confusion for readers that don't know:
RTLinux and PREEMPT_RT are two completely different things.

These days LinuxCNC uses PREEMPT_RT instead of RTLinux (or RTAI; yet another completely different thing).

real time kernel: benefits

Posted Sep 21, 2024 13:11 UTC (Sat) by shef (subscriber, #91287) [Link]

4G/5G lower level radio stack requires RT to achieve proper performance (together with time sync) - Intel FlexRAN would be a good example of that

real time kernel: benefits

Posted Sep 21, 2024 13:32 UTC (Sat) by nevets (subscriber, #11875) [Link]

I always say that the biggest difference tht PREEMPT_RT gives you is bounded latency. That is, there's no "unbound priority inversion".

It also flattens the task context. Interrupt handlers and softirqs run at the same context as normal tasks. This allows other tasks of higher priority to preempt them and have better response times.

Most spin locks become mutexes and allow preemption while they are held.

I gave a talk about this back in 2007 that explains this at Ottawa Linux Symposium and most of it still applies