LWN: Comments on "The future of realtime Linux in doubt"

The future of realtime Linux in doubt

tlsmith3777 — Tue, 14 Feb 2017 15:19:35 +0000

What is the current state of PREMPT_RT and is it fully integrated into the Linux kernel?

Have the funding issues been resolved and how were they resolved if so?

What are the future plans for the Linux real-time kernel, will it continue to be maintained by the community in the foreseeable future?

How does the Linux real time kernel compare with ROS for real-time determinism and latency? Any papers for comparison, especially for automotive autonomous driving solutions?

safety critical

PaulMcKenney — Sun, 12 Oct 2014 08:38:29 +0000

http://www.linuxplumbersconf.org/2014/ocw/proposals/2583

safety critical

marcH — Sun, 27 Jul 2014 09:53:16 +0000

Sorry I think this is going off on a formal methods tangent. You should not spend that much time answering seriously a post with more exclamation marks than words :-)

What I really meant (and was too lazy to write) is: the quality bar for safety-critical applications should at the very least be one big step up from non safety-critical applications. This means using at the very least all the QA tools and methods which are well-known and *routine* - and a bit more. I think no jury would like to hear that some basic code review process or some common and off the shelf static analyser was ignored. This obviously includes testing as well.

By the way: I would be surprised to hear about some place that does not bother mentioning anything beyond testing to qualify safety critical applications, taking all the rest for granted. Now I would be even more surprised to hear about a place that explicitly states that using other, common QA processes is NOT required!

> And in some cases, the lack of that safety-critical widget will be costing lives,

I'm not sure about this one: there is often the option of doing something simpler without using (too) complex software. Or worst case, not do something at all and wait until it can be done safely (and keep prototyping).

I was just reading http://www.dwheeler.com/essays/heartbleed.html again. Security and safety are close in the sense that the most Software Quality processes can be used for both. Quote from David:
"code should be refactored over time to make it simple and clear, not just constantly add new features. [...] The goal should be code that is obviously right, as opposed to code that is so complicated that I can’t see any problems."

Complexity is exactly why general purpose operating systems cannot DRIVE planes and trains. Cars are probably OK: it's much easier to lie and pretend the driver made a mistake ;-)

> what is required for various safety-critical classifications. And for some of those classifications, the powers that be have determined that a long testing period suffices. Other classifications also require formal validation.

I think this sentence shows that some overdue clarification of the meaning of safety-"critical" is needed. For instance I would not have called "safety-critical" a device that merely monitors and alerts. Because in not all but many situations its failure would not have any effect.

For instance Do-178B seems to use "critical" only for the highest level; even more limited meaning than I thought

http://en.wikipedia.org/wiki/DO-178B

safety critical

PaulMcKenney — Sat, 26 Jul 2014 22:01:11 +0000

Almost.

If the plaintiff's attorney is better than the defense's attorney, the jury will believe that what was required was whatever you did, plus a lot more. There are always more tests that could have been run, and there are always more types of formal validation that you could have brought to bear. Even if you somehow managed to run all conceivable tests and carried out all conceivable formal validation techniques, more will have been conceived of after the fact.

Of course, this would mean that the only safe way to produce a safety-critical widget would be to invest an infinite amount of time and money into it, that is to say, to not produce it at all. And in some cases, the lack of that safety-critical widget will be costing lives, which clearly indicates a need for a balanced approach to this issue.

And this in turn is one reason that there are laws, rules, and regulations that specify what is required for various safety-critical classifications. And for some of those classifications, the powers that be have determined that a long testing period suffices. Other classifications also require formal validation. Which is in fact a reasonably balanced approach to this issue.

safety critical

marcH — Fri, 18 Jul 2014 14:23:40 +0000

Obviously: both!!!

safety critical

PaulMcKenney — Fri, 18 Jul 2014 12:41:44 +0000

Heh! Which would be more convincing to a jury? A huge formal proof based on complicated and unfamiliar assumption? Or a five-year demo? :-)

safety critical

PaulMcKenney — Fri, 18 Jul 2014 12:34:07 +0000

You want names? I suspect that your browser can supply them just as easily as can mine. And I could be wrong, but wherever you live is probably one of those places.

As to places in which lives are put at risk, that would be pretty much anywhere that grants drivers licenses, and especially those that grant drivers licenses to teenagers on the one hand and to the elderly and infirm on the other, now wouldn't it? ;-)

safety critical

marcH — Fri, 18 Jul 2014 11:01:52 +0000

Whatever the respective risk levels are, there is and will always be a massive difference between safety-critical software and carnivore butterflies: you cannot sue carnivore butterflies for millions.

By the way, when talking about safety security is never far away, example:

http://arstechnica.com/security/2013/07/disabling-a-cars-...

Approving a system after ONLY a successful, five years long demo is giving even less confidence about security than about safety. Most other QA tools and processes tackle both at the same time. If you don't use all known working QA options when working on a safety critical system, then you are not really considering it as safety critical. By the way: the MP3 player in the car is probably not safety critical. Unless it can hack into brakes or steering. Modularity and "less is more"; here we go again.

Note about the car example: unlike carnivore butterflies, cars have already killed and will continue to kill millions of people. But again the key question is: for any given specific crash, who can you blame and who can you sue?

safety critical

marcH — Fri, 18 Jul 2014 10:42:52 +0000

Does not look like these laptops are used for any safety-critical feature, are they?

safety critical

ssokolow — Fri, 18 Jul 2014 04:05:55 +0000

Linux. Debian 6, specifically.

safety critical

raven667 — Thu, 17 Jul 2014 21:25:21 +0000

There is a non-zero probability I'm going to be eaten alive by carnivorous butterflies on the way home from work but I am not rearranging my life around that possibility ...

Non-zero doesn't mean significant, there is always risk, managing it is about assessing significance and making subjective value judgements.

8-)

safety critical

marcH — Thu, 17 Jul 2014 21:07:58 +0000

> > in which places lives are put at risk.

> That's probably a bit hyperbolic,

A "risk" is anything with a non-zero probability; leaves plenty of room. It's all in the details.

safety critical

raven667 — Thu, 17 Jul 2014 14:35:31 +0000

> Names? Interesting to know in which places lives are put at risk.

That's probably a bit hyperbolic, I mean clearly you can look outside and see that it is not the apocalypse yet.

safety critical

mathstuf — Thu, 17 Jul 2014 14:24:10 +0000

The ISS laptops used XP until recently (I forget if they went to Windows 7 or Linux though).

safety critical

marcH — Wed, 16 Jul 2014 21:06:42 +0000

> Heh! If you have objections to practical demonstrations being sufficient for some types of safety-critical certifications,

Of course I have, who would not? Which software engineer with more than a tiny bit of experience would believe something as lame as "it worked for five years, so it will work forever even in new situations; no need to look at how it works". Would you? This type of subpar software QA is not even considered acceptable by regular, non-critical software!

What next? "It compiles without any warning, so it's good for production"?

> I suggest you take that up with the authorities in the jurisdictions where this is the case.

Names? Interesting to know in which places lives are put at risk.

The future of realtime Linux in doubt

PaulMcKenney — Wed, 16 Jul 2014 19:19:59 +0000

Agreed, at least assuming that you were not already using ECC and triple modulo redundancy. If you are already using one of these techniques, then adding more hardware can still increase the failure rate, though hopefully at a much smaller rate than for systems not using these techniques. Of course, adding triple modulo redundancy is not a magic wand -- it adds more code, which of course can add complexity and thus more bugs. :-(

The future of realtime Linux in doubt

PaulMcKenney — Wed, 16 Jul 2014 19:13:35 +0000

If you do have to go there, then the certifications will of course be limited to those that can be obtained for the system as a whole.

The future of realtime Linux in doubt

PaulMcKenney — Wed, 16 Jul 2014 19:11:46 +0000

Overprovisioning the hardware does make a lot of sense when feasible.

But if you require deadlines down in the tens of microseconds, overprovisioning may not help much. Last I heard, nuclear power stations had much longer deadlines, but you would know better than I.

safety critical

PaulMcKenney — Wed, 16 Jul 2014 19:06:44 +0000

Heh! If you have objections to practical demonstrations being sufficient for some types of safety-critical certifications, I suggest you take that up with the authorities in the jurisdictions where this is the case.

safety critical

marcH — Tue, 15 Jul 2014 22:29:13 +0000

Whether you are looking at fancy automated proofs or more traditional quality processes (code coverage, code reviews, etc.), breaking down the system into small enough and simple enough modules is required anyway if you want to keep complexity low enough to demonstrate high levels of (very expensive) quality in any possible way. Separating software and hardware is only the start.

Like you mentioned, Linux has grown too big and general purpose to qualify. I think it's OK; it's good at pretty much everything else.

"Practical demonstration" should definitely be a requirement for safety-critical certifications - but not just! Far from it. I mean, testing should never be an excuse not to use other, common quality processes, especially not for safety-critical code.

> And rumor has it that Linux actually is used today in some carefully chosen safety-critical applications.

Meanwhile, people are kept in fear of terrorists...

The future of realtime Linux in doubt

marcH — Tue, 15 Jul 2014 21:37:20 +0000

> In short, at the extremes, you don't have the luxury of defining the software independently of the rest of the system.

So, why do you go there?

On life-critical systems (or... weapon systems) you do have this "luxury". Except for the name; I guess it's rather called "certification requirement" or something similar.

The future of realtime Linux in doubt

marcH — Tue, 15 Jul 2014 21:10:00 +0000

> The problem with any system that you CAN reason and make proofs about it, is that those proofs have nothing whatsoever to do with the real world.

Yes we all know that 2 + 2 is actually 5 in the "real world".

> "As far as the laws of mathematics refer to reality, they are not certain;"

... while "not certain" actually means "nothing whatsover to do" in the same real world.

The future of realtime Linux in doubt

Wol — Tue, 15 Jul 2014 15:11:15 +0000

Or like my Vectra ...

The cambelt fell off!!! Which was a known failure mode :-( the fact it wrecked the engine was a minor point ... I was on a motorway doing 70. Fortunately, late at night, there was no traffic so getting onto the hard shoulder wasn't hard. But if it had been daytime and busy ...

Cheers,
Wol

The future of realtime Linux in doubt

roblucid — Sun, 13 Jul 2014 16:46:06 +0000

The overhead of abstraction, is simply a factor to consider so long if it's predictable.

The cost in slightly more performant hardware, may be a wise investment to reduce system complexity, developer time and improved QA of the result.

If "all you care about" is meeting deadlines, sailing close to limits seems perverse and more a "soft" RT thing, where errors in assumptions are tolerable ie network sound streaming, where no-one dies if it fails in unusual circumstances.

Having seen specs and done a requirements analysis for a nuclear power station project, I can assure you squeezing the most out of the HW was the last thing desired. I suspect (I left the project for another opportunity) any error, would have been in over-provision causing too great an increase in system complexity and making it hard to analyse.

The future of realtime Linux in doubt

roblucid — Sun, 13 Jul 2014 16:34:10 +0000

That's like the first twin engined planes.. unfortunately they relied on the increased engine power so became LESS reliable, as failure chances were doubled.

With enough "more" memory though, things like ECC and a fault tolerant technique like say triple channel with independent implementations allowing a "vote", then you gain greater reliability, like with modern planes which may tolerate multiple engine failures.

The future of realtime Linux in doubt

roblucid — Sun, 13 Jul 2014 16:28:32 +0000

I think that's effectively saying the same thing, using soft sub-goals which mitigate a "miss". By definition of "hard" RT, being allowed to miss a deadline, makes the system no longer "hard" but "soft" so I ruled out this strategy as not meeting spec.

The idea of an "up-clocking" strategy to increase resources "on demand at cost of power" was to mitigate the inherent indeterminism of modern CPU.

I considered how you can make case to be able to meet "hard" deadlines, and assumed any "hard" RT program that risks paging from disk, waits on non predictable event, or as in Mars Pathfinder case blocking on a lower priority process due to inversion is inherently "broken" thus not "hard" RT.

This came out of considering a conversation in late '89 with an RT developer colleague of control systems. Hard RT just loved polling because of it's predictability and simplicity, never mind the performance disadvantages. It seems that just runs counter to philosophy of Linux, which appreciates performance over predictability or simplicity.

A "fast path" which conserves power, but falls back to brute force, polling of registers etc, might be a viable hybrid strategy.

The future of realtime Linux in doubt

PaulMcKenney — Sun, 13 Jul 2014 15:40:34 +0000

It can get much worse.

The more-reliable software might consume more memory. Adding more memory will degrade the overall system reliability, perhaps even to the point that the system containing more memory and more-reliable software is less reliable than the original system. As the old saying goes: "Be careful, it is a real world out there!"

safety critical

PaulMcKenney — Sun, 13 Jul 2014 14:18:30 +0000

I do agree that it will be some time before formal validation tools grow to the point where they can handle something like today's Linux kernel, and if they ever do, the Linux kernel will have grown even larger by that point.

That said, formal proof is but one road to safety-critical certifications. For some applications in some jurisdictions, practical demonstration is permitted. For example, if a given system does the right thing for five years, then it will be considered to be suitable for that particular safety-critical application.

Of course, given that each application needs to a separate long-term safety test, and that any failure means that you start over, the practical-demonstration approach could take even more time and be even more costly than formal proof. The advantage that the practical-demonstration approach has is that the formal-proof approach simply cannot handle anything but the smallest of systems. And rumor has it that Linux actually is used today in some carefully chosen safety-critical applications.

Nicholas McGuire has done some excellent work in this area: http://www.slideshare.net/DTQ4/gnu-linux-for-safety-relat...

The future of realtime Linux in doubt

PaulMcKenney — Sun, 13 Jul 2014 14:06:04 +0000

Indeed! ;-) ;-) ;-)

But it is really hard to believe that article appeared nine years ago!

The future of realtime Linux in doubt

PaulMcKenney — Sun, 13 Jul 2014 14:01:36 +0000

> Not sure why you assume that defining the software forbids defining the rest of the system... strange.

Nice try, but you missed the mark.

The problem is that as you push towards the limits of the system's capabilities, you must be increasingly careful about when and how you apply abstraction. Out at the limits, little things that might otherwise be confined to a given piece of the system start making themselves felt globally. Therefore, out at the limits, premature abstraction is the root of all evil.

In short, at the extremes, you don't have the luxury of defining the software independently of the rest of the system.

One saving grace is that today's hardware can trivially meet requirements that were out at the limits a few decades ago. So the limits have moved out considerably over that time. However, the global awareness required at the limits remains unchanged.

The future of realtime Linux in doubt

mathstuf — Sun, 13 Jul 2014 04:26:07 +0000

Sounds like a problem I had with my Jeep. There's a sensor which detects where the camshaft is to know when to fire the right spark plug. The wire from it shorted on the engine block and rather than firing willy-nilly (and destroying some pistons and/or chambers), it just stopped firing which basically shuts the vehicle off. Granted, there was probably very little ECU involvement here (it is a 1989 after all), but failure modes are important.

The future of realtime Linux in doubt

nevets — Sun, 13 Jul 2014 03:17:00 +0000

This reminds me of a time we talked about "Diamond hard" "Ruby Hard" and, oh yeah, Microsoft's "Feather hard" real-time systems.

http://lwn.net/Articles/143323/

The future of realtime Linux in doubt

Wol — Sat, 12 Jul 2014 22:45:46 +0000

I just apply the standard rule. I will respect you until you throw that respect away.

And in this case, that's exactly what happened - the guy used "real time" in a manner I didn't understand, and when I queried it and said "I think you mean "online" ", his reaction was "well, they are the same thing now, who cares".

Sorry, I do care very much because if we define away the meaning of words, we then have no way of expressing that concept, and the meaning of "real time" is life and death! As others have said, real-time is about deadlines, and when you're dealing with industrial plant, that could well be a DEAD line.

And if you're stupid enough to change one word into another, like here using "real time" when there's a perfectly good word "online", then I'm sorry but you damn well shouldn't be a journalist! The grief people have caused me (when I did PC support) because they refused to learn the difference between "memory" and "disk space", or "PC" and "terminal", and probably various other terms as well. If I have to teach them, then fine. If they refuse to learn, well, I'm sorry if I trashed your system by mistake because you didn't want to express yourself clearly ...

Cheers,
Wol

The future of realtime Linux in doubt

daniel — Sat, 12 Jul 2014 20:11:30 +0000

Crazy low latencies are still needed. Some latency-critical platform components are too complex to implement in gate logic.

The future of realtime Linux in doubt

clump — Sat, 12 Jul 2014 14:27:50 +0000

Hopefully you're more respectful when approaching journalists than you are when referring to them.

The future of realtime Linux in doubt

ianmcc — Sat, 12 Jul 2014 13:40:41 +0000

There are examples of that. I can't immediately point to some links but IIRC it was a car (a BMW?) where the engine spontaneously shut down on the motorway, for some relatively trivial reason. The driver made it out alive. But it brings up a good point, even with 'hard' real-time, coping with a failure mode is very important. And if you can cope adequately with a failure, are you really 'hard' real-time anymore?

I think, going into the future, where even simple microcontrollers have pretty substantial CPU power, the issue of 'hard' real time is less important than robustness under failure conditions. The surrounding hardware has some failure rate too, and the software (1) needs to cope with that as best it can and (2) there is no point going to extreme lengths to engineer software that has a failure rate of X if the failure rate of the installation as a whole is 100000*X.

The future of realtime Linux in doubt

Wol — Sat, 12 Jul 2014 11:27:35 +0000

> The problem with any big and complex system is that you cannot reason and make proofs about it.

The problem with any system that you CAN reason and make proofs about it, is that those proofs have nothing whatsoever to do with the real world.

"As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality." Albert Einstein.

Cheers,
Wol

The future of realtime Linux in doubt

xxiao — Sat, 12 Jul 2014 01:40:21 +0000

The technical discussions are always interesting, but the final question is still, how can we help? or, should be Linux Foundation do something about it since RT is actually an essential part in the Linux ecosystem?
Maybe kernel.org should have a 'support' button to collect some funds for projects like this?

The future of realtime Linux in doubt

dlang — Fri, 11 Jul 2014 20:27:47 +0000

pretty good definitions, but I would not require permanent damage, just explicit recovery for hard-real-time

so audio recording would be hard-real-time. If it's a live source, you permanently loose data, but if it's recording from a different source, you can restart.

P.S. the ECU in a car needs to malfunction for a significant time period before it will do real harm to the engine

The future of realtime Linux in doubt

dlang — Fri, 11 Jul 2014 20:23:42 +0000

the normal kernel will probably provide lower latency >99% of the time, but the -rt kernel will provide lower latency 0.9% of the time

When the -rt work started, this was probably more like 90%/9%

In many cases, while the percentage of cases where -rt helped, the reduction in latency experienced in those cases can be drastic (enough to seriously affect the average, not just the worst-case)

* percentages are not measured, but just my understanding