Kernel Summit 2006: Realtime [LWN.net]

2006 Kernel Summit coverage on LWN.net.

Ever since the current realtime patch set began to circulate, there have been questions about when (and if) it would be merged into the mainline. Ted Ts'o and Ingo Molnar's session on realtime support started with the note that, to a great extent, this merging has happened. Over the last year, features like robust futexes, priority inheritance, the generic IRQ layer, the core timekeeping rewrite, and the lock validator have all gone into the mainline. For the first time in history, somebody (Ingo) was able to propose the merging of priority inheritance without getting flamed to a crisp by Linus.

As a result of all this merging, the out-of-tree realtime patch set has shrunk considerably in size. About 700KB of patches remain, however. The biggest chunks at this point are the high-resolution timers and sleeping spinlocks.

A realtime kernel can now regularly achieve 20 µsec response times on a heavily loaded system. Anymore, when there are response time issues, the problem is likely to be hardware related. It is, in other words, possible to make a Linux kernel which can provide realtime response - something many people would not have believed even a short time ago.

There is some cost to this, in terms of code complexity and system performance, but the performance hit is often less than people think. Stock kernels do not properly account for time spent handling interrupts, while realtime kernels (as a result of moving interrupt handlers into processes) do. So programs may look like they are using more CPU, but it is really just a matter of the kernel noticing the time lost to interrupts.

With regard to merging, the hard work has been done. Some of the trickiest patches were the ones which cleaned up locking semantics, made the big kernel lock preemptible, etc. One challenging part which remains is identifying the few locks which must remain "raw" (non-preemptible) spinlocks in a realtime kernel. There is a relatively small number of those, but it is important to find them all and change them over to the proper lock type.

One outstanding issue is recovery when a realtime task goes out of control. On a realtime kernel, a high-priority task can block out everything else, including interrupt handlers. Current techniques for handling this situation are limited to those involving the power button. It might be nice to add something like a soft lockup detector. Linus, however, thinks that this is not a problem which should concern people. If you truly want hard realtime, that is what you will get, and vulnerability to runaway high-priority processes is part of the deal.

A nice feature for the future might be realtime disk and filesystem I/O. Essentially, a process which remains within a given bandwidth limit could perform I/O and still have realtime response behavior.

The question was asked: why bother with sleeping locks? Making locks preemptible is seen by some as a way of papering over the real problem: long lock hold times. Why not simply fix those? The answer comes in a couple of parts:

Extensive efforts have been expended toward fixing lock problems for many years, and those efforts will continue into the future. The use of sleeping locks is not being used as an excuse to avoid fixing code which holds locks for too long.
Ensuring realtime response in the absence of preemptible locks requires auditing the entire body of kernel source - all eight million lines or so. That's a big job, and one which is hard to keep up with in an environment where nearly ten thousand lines of code are being changed every day. Sleeping locks reduce the audit requirements to a couple thousand lines - a much more tractable problem. For those who need realtime response, guaranteed, that is a good deal.

Linus indicated that (unlike some others) he has no problem with merging the sleeping locks patch, as long as there is no danger that sleeping locks might be configured in by default. That seems unlikely, given that distributors do not even turn on the much milder kernel preemption feature which has been present throughout the 2.6 kernel series. Since the realtime features ruin performance on large SMP systems, they are not likely to be turned on in enterprise kernels either. Even so, there is said to be pressure on distributors to make realtime kernels available. That pressure becomes harder to resist once a feature makes it into the mainline. It appears that this work may well be merged, however, so distributors may have to show a little backbone.

Next: Embedded systems.

(Log in to post comments)

Realtime - what is it?

Posted Jul 19, 2006 21:47 UTC (Wed) by simlo (subscriber, #10866) [Link]

The editor - along with many other people - seems to equate low latency with realtime. That is not how I view it as a realtime programmer. For me the important issue deterministic latencies. On a system based on fixed priorities, like the real-time priorities in Linux, it is important that the latency for a task running at a given priority only depends on what is running at equal or higher priority. If the OS offers that, you can use it to build a real-time system. How low the latencies are, is ofcourse important for the specific application.

Now some people at LKML claims that for a system to be hard real-time it has to be mathematically proved to be so. For me that is mixing up the issues. People can easily use Linux for other mission critical applications without having a mathematical proof it won't crash. They rely on observation and the fact that the coders consider all crashes as very important bugs. You can then also rely on Linux giving deterministic latencies by observation and if you know the coders have done what they can to avoid unbounded latencies.

The latter can be an issue. The first problem is that 95% of the Linux hackers do not care about real-time. Therefore they wont bother about making code with unbound execution inside spin-locks. The trick about turning spin-locks into mutexes helps that: The unbound code is suddenly preemptible and the higher priority real-time tasks does not have to worry. Sometimes per-cpu variables are used and protected with preempt_disable() or local_irq_disable(). All those places have to be looked at, when people want to use those subsystems in a real-time system.

The second problem is that not even all those coding on real-time systems agree on the rules! So the kernel can still get code from XYZ for real-time which turns out not to be if not properly reviewed. I have noticed stuff added to the preempt-realtime itself which is not entirely safe against Joe User. I.e. Joe User could make an evil program making the latencies very high.

Merging preempt-realtime into the main-line will be a good idea. Then Linux can compete with the dedicated RTOSes - which don't even have to care about Joe User because they typically are single user systems. But for Linux to be really successfull in doing that, someone has to specify the aims and the rules. And there has to be reviews of everything entering the kernel to see if it breaks the real-time behaviour - or at least mark it as unsafe for real-time perposes.

Realtime - what is it?

Posted Jul 19, 2006 23:27 UTC (Wed) by corbet (editor, #1) [Link]

Maybe I should have been clear: the realtime folks claim 20 µsec worst case response times. They are aiming for determinism, not just quickness.

Realtime - what is it?

Posted Jul 20, 2006 17:36 UTC (Thu) by nevets (subscriber, #11875) [Link]

I have to disagree about the comment about 95% of the kernel hackers don't care about real-time. Actually, being more specific, they worry about latency, not to the degree of us real-time folks, but they certainly do care. That's why there is a big push to separate out the latency-tracer patches to be merged into mainline. A lot of clean up code has been happily accepted that fixes latency problems.

This is one of the things that most of the kernel hackers agree on. No matter what you think about the RT patch, it definitely has found and was a source of a fix to numerous bugs. Some of the patches were clean ups, others fixed hard to find deadlocks, and several fixed up latency problems. So in general, the RT patch has been good to the Linux kernel in general.

Priority Inheritance?

Posted Jul 19, 2006 23:30 UTC (Wed) by corbet (editor, #1) [Link]

Priority inheritance is in 2.6.18. See this article for a description of the PI patch. There's not really any difference between user-space and kernel-space PI - it's all implemented in the kernel in any case.

Priority Inheritance?

Posted Jul 20, 2006 0:09 UTC (Thu) by mingo (subscriber, #31122) [Link]

yes. The kernel/rtmutex.c code which implements user-space PI (the new PI-futexes API) is the exact same code used for preemptible spinlocks too in -rt. It was not entirely trivial to merge the two uses into the same subsystem, but it's reality now. (And i dont want to take all credit for it, large portions of this solution were designed and written by Thomas Gleixner, Esben Nielsen and Steven Rostedt.)