LWN: Comments on "LCE: Realtime, present and future"

Most systems also have no proof that a kernel panic won't occur.

gmatht — Sun, 09 Dec 2012 07:41:50 +0000

Even for a non-RT system there is usually no proof that a full system failure won't occur. E.g. Linux has no proof that a kernel panic won't occur. Systems that claim to conform to POSIX standards rarely have a proof of this. If some organisation can trace all the cases of system failure to hardware failures they may choose to trust claims made by software, including hard real time claims, without formal proof.

Real time for what?

mb — Thu, 22 Nov 2012 13:35:24 +0000

> doing so and posting benchamrk numbers showing latency doesn't matter

Ehm what?

> what would matter is if there is anything showing that this makes a difference in the part being machined.

If it would make a difference to the machined parts, either the previous RTAI based implementation or the new Linux-RT-preempt based implementation would be seriously broken.

> That page was big on the 'how to' but utterly lacking in the 'why go to this effort'

We do this, because it simplifies the software a lot and gets rid of all those ugly RTAI kernel modules.

Real time for what?

mpr22 — Thu, 22 Nov 2012 11:24:04 +0000

Naturally. There's a driving examiner in the front passenger seat; getting an accurate picture of someone's everyday driving in such conditions is more or less impossible.

Real time for what?

Otus — Thu, 22 Nov 2012 11:17:01 +0000

> Or the driving test is wildly insufficient to distinguish bad drivers.

That's not the purpose of most driving tests. Usually the purpose is to see if someone is good enough that they'll learn the rest on their own without being too much of a danger to others.

Realistically most people who pass a driving test are going to be bad drivers for a long while. (Unless they are testing for a license in another state/country and have already driven a lot.)

Real time for what?

ceswiedler — Wed, 21 Nov 2012 19:02:19 +0000

Or the driving test is wildly insufficient to distinguish bad drivers.

Real time for what?

dlang — Wed, 21 Nov 2012 09:40:26 +0000

doing so and posting benchamrk numbers showing latency doesn't matter, what would matter is if there is anything showing that this makes a difference in the part being machined.

That page was big on the 'how to' but utterly lacking in the 'why go to this effort'

Real time for what?

mb — Wed, 21 Nov 2012 09:32:22 +0000

There is a subproject for LinuxCNC Preempt-RT support:
http://wiki.linuxcnc.org/cgi-bin/wiki.pl?Debian_Wheezy_Li...

Real time for what?

dashesy — Mon, 19 Nov 2012 15:36:17 +0000

Actually I meant Data Acquisition (DAQ) with additional output channels.

Real time for what?

mathstuf — Mon, 19 Nov 2012 03:20:37 +0000

I assume that "DAQ" is meant to be "DAW" (digital audio workstation)?

Real time for what?

Trelane — Fri, 16 Nov 2012 21:07:59 +0000

NP. Glad you found it informative. :)

Real time for what?

jtc — Fri, 16 Nov 2012 21:06:09 +0000

"At the Professional Hacker level or above:"

Damn. I'm a "starving hacker". I'll have to upgrade once I find a job.

Thanks for the info.

Real time for what?

Trelane — Fri, 16 Nov 2012 19:12:23 +0000

> I believe that as a subscriber you can disable advertisements.

At the Professional Hacker level or above:
https://lwn.net/op/FAQ.lwn#subs

Real time for what?

dlang — Fri, 16 Nov 2012 18:40:31 +0000

I believe that as a subscriber you can disable advertisements.

LWN doesn't go after individual sponsors, they subscribe to an advertisement service that puts the ads in place based on their own criteria.

Real time for what?

jtc — Fri, 16 Nov 2012 18:29:38 +0000

" The software running on your doctor's desktop PC could kill you if it displays the wrong notes and causes the doctor to prescribe the wrong medicine - yet this is not normally an application considered safety-critical where special software methods must be used to guarantee correctness."

It's also requires neither a soft nor hard real-time kernel. :-)

[On a complete tangent: Is anyone else getting completely sick of that constantly changing Perforce ad. that adorns the top and upper right side of almost every page on LWN. I find it quite distracting and have to move another window to cover the right side of the page each time I open a new lwn page, so that I don't have the ad. screaming at me all the time while I'm reading! They've been running this ad. for many weeks - it's about time the found a new sponsor. Argghh!]

Real time for what?

jtc — Fri, 16 Nov 2012 18:20:07 +0000

"what kind of applications require real time these days?"

If Gleixner, when he quoted those download statistics ("About 30% of those went to European corporations, 25% to American corporations, 20% to Asian corporations, 5% to academic institutions"...), had also disclosed who those corporations were, this probably would give a good real-world answer to your question - or at least to the question: "What kinds of applications is the Linux realtime kernel being used or considered for these days?". Has he released this info? I suspect not - I suppose he's justifiably paranoid about being sued for breach of privacy or something like that.

" Linux audio used to be mentioned often, but these days it appears that CONFIG_PREEMPT is enough; is it?"

I can't answer that, but I will speculate that since music and MIDI recording and playback requires hard realtime capability, any companies considering using Linux for this would tend to favor the realtime kernel over the mainline kernel, since although the mainline kernel, perhaps, does fine x% of the time, the realtime kernel is likely (or certain, perhaps) to do fine y% of the time, where y% is significantly greater than x%.

LCE: Realtime, present and future

dmk — Fri, 16 Nov 2012 10:41:30 +0000

Try robotics, where you want to find out where your machine is located out of sensors (gyro, wheelticks). If you have a constant or small variance delay in the measurements, you can estimate it and compensate for it. but if you have high variance in the delay your position estimate will suffer.

LCE: Realtime, present and future

tglx — Thu, 15 Nov 2012 17:29:28 +0000

I'm well aware that safety critical != hard real-time.

Though hard real-time systems must be designed in a way that they guarantee not to ever miss a deadline, simply because missing a single deadline is considered a fatal full system failure. So how do you guarantee that by other means than by mathematical proof?

> And for a practical note: The theoretical upper boundary is not magnitudes higher than what you can measure in tests.

None of those systems including Preempt-RT can specify their theoretical upper boundary, except in safe ranges which make them not at all different :)

Thanks
tglx

LCE: Realtime, present and future

alejluther — Thu, 15 Nov 2012 13:24:50 +0000

Thomas,

That's a good point. But I talked about RTAI or RTLinux because the Xenomai reference. My experience is with RTLinux internals and the virtualization made there is just the simplest one, so you do not have isolation. You know there are today solutions based on full virtualization even with hardware help which, I guess, is a must for real full isolation.

The good part of RTLinux technique (I think RTAI did not follow the rule of "keep simple" strictly) is you can use it with some microcontrollers which will not have support for hardware virtualization.

Don't take me wrong. I think you are doing a fantastic job and with your mindset, maybe you prove me wrong in the future. I just wanted to point out the challenge (probably) needs another revolution regarding kernel design. By the way, the kernel modularity could be the starting point.

LCE: Realtime, present and future

simlo — Thu, 15 Nov 2012 13:00:19 +0000

Everytime this discussion comes up I add a comment like this:

Safety critical != hard real time

Safety critical means that the system is certified and verified and maybe even mathematical proven to be correct. This implies small code and very often hard realtime.

Hard real time means that there per design is an upper boundary to latency. But we all know that a system might not work as designed. Therefore a system can be hard real time by design but fail due to bugs.

Thus by these definitions neither preempt realtime, RT-Linux nor Xenomia can be be safety critical but all can claim "hard real time" - but only if they have made sure that all code paths blocking scheduling of higher priority tasks and interrupts, have an upper boundary.

And for a practical note: The theoretical upper boundary is not magnitudes higher than what you can measure in tests.

Real time for what?

epa — Thu, 15 Nov 2012 11:34:10 +0000

Hmm, you're right. I would draw a distinction between the software running on the air traffic controller's desktop (which has often been Unix for decades now) and the microcontroller which controls the air intake to the engines. The former has fairly relaxed real-time requirements (the display has to update, but once a second might be acceptable) and I would call it 'soft safety-critical'. The software running on your doctor's desktop PC could kill you if it displays the wrong notes and causes the doctor to prescribe the wrong medicine - yet this is not normally an application considered safety-critical where special software methods must be used to guarantee correctness. By contrast, a failure of the jet engine microcontroller (assuming it has one, I am speculating) can cause instant disaster.

LCE: Realtime, present and future

tglx — Thu, 15 Nov 2012 10:26:51 +0000

Your assumption is, that Xenomai, RT-Linux and RTAI are fulfilling the requirements of hard real-time. I agree that the RTOS code base part of those is small, but it's complex enough that I have serious doubts about the ability to formally verify them.

Aside of that all those systems are so interwoven with Linux itself, that you'll have a real hard time to prove that there is no undesired influence possible from the "idle task Linux" to the "safe RTOS". Just look at the obvious lack of fault isolation and explain me how that is fulfilling the requirements.

While Preempt-RT might look more complex in the first place, I dare to claim that building an hybrid system, which fully relies on facilities of the "idle taks Linux" in all its glory without having full isolation in all aspects, is just introducing a different level of hard to understand and hard to verify complexity.

As long as nobody can point me to real proof of why any of the above solutions is verifiably more real-timeish than Preempt-RT, I definitely stand to my word, that Preempt-RT will make those obsolete or already has done so at least partially.

Thanks

    tglx

Real time for what?

anselm — Wed, 14 Nov 2012 12:07:01 +0000

Some time ago I consulted for a leading machine tool manufacturer here in Germany for a day or two and, after my actual work was done, was invited to take a guided tour of the shop floor. I was surprised to find out that they had these huge CNC machines attached to Linux-based controllers. I asked the guy who was showing me around whether they were using real-time Linux, and the answer was no, the normal generic Linux kernel worked fine as far as they were concerned. Considering that they are selling these gadgets to clients all over the world, and that any failure would probably cost them dearly, that put things into perspective for me …

Real time for what?

anselm — Wed, 14 Nov 2012 11:58:49 +0000

I do think they will have to prove their safety far more rigorously than the average human driver, though.

I think a Turing-like test should suffice: If during a driving test, a driving license examiner cannot tell whether a computer or a person is driving the car, and it looks as if the entity in question ought to pass, then – if it was actually the computer driving – the setup is OK.

LCE: Realtime, present and future

alejluther — Wed, 14 Nov 2012 09:00:50 +0000

In another talk, Stephen Rostedt from Redhat explained the RT Preempt using his presentation from 2007 with updates in. Thomas Gleixner was there helping with explanations about how things have changed since 2007.

At the end, there was a question to Thomas Gleixner "Do you consider RT Preempt will make other solutions like Xenomai obsolete?"

"Yes, I do believe so" was the reply.

The answer made me smile since it makes clear what is the mindset of people like Thomas: they love challenges whatever the complexity!

I will not bet against him but moving the Fully-Preemptible Real-Time Linux kernel one step further means HARD real time. The main problems against is the kernel is too big and all we know bugs are out there. Xenomai and other solutions like RTLinux/RTAI are based on simplicity so bugs can be minimized. RTOS vs GPOS is all about determinism and RTOS can achieve this because the code is small enough for verification.

RT Preemtp has brought some sort of revolution with design changes like IRQ threads. IMHO making the RT Preempt hard real time could just be achieved with another revolution which would make happy professor Andrew Tanenbaum (and Linus a bit grumpy).

Real time for what?

iarenaza — Tue, 13 Nov 2012 23:01:49 +0000

I know of at least one Spanish company building protection, control and measurement equipment for electrical grids that use real time linux.

Real time for what?

dlang — Tue, 13 Nov 2012 22:58:40 +0000

failing to shut off the plasma cutter has very little effect (other than possibly a small amount of damage to the piece being cut), if you go off the edge, the plasma isn't going to be sustained, if you leave the cutter in place, the plasma will eat a small amount around the location until the circuit can't be sustained any longer and the plasma will cut off.

These things rely far less on precise timing than you think.

Just about all of the DIY devices rely on stepper motors, which move a specific distance when pulsed, not normal motors run for a specific amount of time.

If the pulses are late, things move a little slower. If there is enough momentum in the system, it's possible for that momentum to cause the equivalent of 'jumping a tooth on a gear' and being slightly out of position, the solution to this problem is to slow the machine down a bit.

synchronization between different pieces is a matter of either moving one motor, then a different motor, then the first one again, or in setting up the movement for both motors and sending a 'move now' pulse. In either case, slight delays don't break anything.

LCE: Realtime, present and future

tdwebste — Tue, 13 Nov 2012 22:51:38 +0000

CPU isolation looks useful particularly for safety certified linux-RT controllers.
IEC6150

It would be very interesting to replace IEC61508 certified RTOS with linux-RT.

Real time for what?

man_ls — Tue, 13 Nov 2012 22:36:17 +0000

I was thinking more along the lines of plasma cutters or robots, also mentioned in the link: failing to switch off the plasma may have bad consequences. Or a cutting machine which needs precise synchronization between different pieces. But perhaps most of these machines are designed to be fail-safe.

Real time for what?

dlang — Tue, 13 Nov 2012 21:48:56 +0000

> Cool! Those look pretty critical to me: having catastrophic consequences if they fail -- at least causing damage to property, and probably to operators.

I have built a CNC machine, so I have first-hand experience here.

It depends how the system fails.

If they fail by not scheduling for 100ms (or even 1s), there is probably no problem besides delaying the work.

If they fail by sending the wrong commands out to the equipment, things can be worse.

But these do not require real-time scheduling to operate.

Real time for what?

tbird20d — Tue, 13 Nov 2012 19:42:05 +0000

At Sony we're using Linux realtime in digital cameras and camcorders, for control of hardware (things like autofocus).

Real time for what?

dashesy — Tue, 13 Nov 2012 16:41:17 +0000

Well not if they are audited for a very narrow use case on dedicated long-life hardware, and fixed on a certain Kernel version without updates and with minimal applications (i.e. instruments).

BTW, one more example I can add is DAQs.

Real time for what?

gregkh — Tue, 13 Nov 2012 16:19:42 +0000

Hm, like MRI machines? Air traffic control systems? Airplane flight navigation? Coal fired power generators? Windmill controls? Yacht stabilizers? Spacecraft control systems? Laser welding robots? High-speed milling machines?

I could go on and on. Linux is used for lots of things like this, and has been for years (something like over 85% of the US power production is controlled directly by Linux machines). A lot of these systems use the real-time patches, and others don't, but to think that Linux isn't being used in these type of situations is wrong.

Real time for what?

mpr22 — Tue, 13 Nov 2012 15:26:08 +0000

Heck, it'd probably have trouble in Boston, MA (which admittedly has one of the most European street layouts of any major city in the USA).

Real time for what?

niner — Tue, 13 Nov 2012 15:23:04 +0000

Well they could hardly do worse than human drivers... speaking as a cyclist who has to endure and survive them every day.

Real time for what?

rvfh — Tue, 13 Nov 2012 15:20:21 +0000

Real test for machine is in difficult situations, where humans may 'feel' that something is not quite right...
Also, I am unsure these cars would work in Europe with crazy drivers and difficult-to-compute timings at some crossings.

(disclaimer: I was born and have lived most of my life in Europe.)

Real time for what?

KSteffensen — Tue, 13 Nov 2012 15:02:19 +0000

Let me qualify that. The first accident involving grave human injury and/or death.

I'm quite willing to believe that on the average these things are far more safe than human drivers since they so rarely have to text their girlfriends or fiddle with the radio or whatever. I do think they will have to prove their safety far more rigorously than the average human driver, though.

Real time for what?

hummassa — Tue, 13 Nov 2012 14:45:55 +0000

Where have you been the last year? ;-)

They already ran about 300,000 miles, with only two accidents: one, the autonomous driving system was offline (i.e., the car was being driven by the human driver) and two, the car was rear-ended. Apparently, they are safer than me (I usually go some 40,000 miles between minor crashes and scrapes, and I have some 400,000 miles total in twenty years of driving, with ten incidents).

Real time for what?

man_ls — Tue, 13 Nov 2012 14:42:07 +0000

Cool! Those look pretty critical to me: having catastrophic consequences if they fail -- at least causing damage to property, and probably to operators.

Real time for what?

KSteffensen — Tue, 13 Nov 2012 14:37:25 +0000

Cool! I did not know that.

Next big hurdle will be the debate the first time one of these is involved in an accident.

Real time for what?

epa — Tue, 13 Nov 2012 14:29:43 +0000

My guess would be systems which are real-time but not safety-critical. In principle, Linux is too complex for use in systems where a bug can cause deaths. In practice I daresay there will be those using it anyway.