The perenial "Nuclear Power Plant" example [LWN.net]

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 19:01 UTC (Tue) by sbergman27 (guest, #10767)
Parent article: Approaches to realtime Linux

The nuclear power plant thing is a pet peeve of mine.

Could someone please explain exactly which systems in a nuclear power plant require something to happen within a few microseconds? I only ask because I would think that in any good design, the necessity of that level of response would not be necessary. I don't know about you, but I would be hard pressed to replace a piece of hardware in the event of hardware failure in a few microseconds. It would take me at *least* several milliseconds, if not more.

Are we really sitting on the edge of armageddon, awaiting,in silent terror, the time that some system fails to respond (for any reason) within a microsecond or two?

I sincerely hope that the above is a reducto al absurdum.

Otherwise, I would say that "The End Is Nigh".

to post comments

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 19:37 UTC (Tue) by arget (guest, #5929) [Link] (7 responses)

A nuclear power plant operates with that wonderful oxymoron, a controlled fission chain reaction. A highly energetic neutron hits a Uranium (or Plutonium) atom, and splits it into two smaller atoms, with some heat energy and a neutron or two left over, that in turn can go on to split more Uranium atoms. It's a balancing act, too many neutrons, and the reaction goes "super-critical" and releases exponentially more energy, potentially doubling in sub-second time frames (periods). A bomb is designed to go super-critical very, very quickly. A normally functioning reactor will operate in "critical" with a period of infinity, right on the razor's edge between super-critical and sub-critical (where there are not enough neutrons to sustain a chain reaction). Now, because of some inherent randomness, the reactor is generally a hair one side or the other of critical. Modern reactors are designed so that the geometry is such that things don't get too "hot" (or too "cold") too quickly, and you have some time to adjust as your period drops into positive or negative numbers from infinity. The razor's edge is more like a broad ridge. Even so, you want to be able to respond quickly. You can't wait for a computer to reboot. Is it ever on the order of micro or even milliseconds in a (modern, Western) reactor? Nah, but it could get within minutes, or tens of seconds. Really, space travel is probably a better example of something that needs to be controlled within microseconds.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 19:58 UTC (Tue) by euvitudo (guest, #98) [Link]

I like your description. There has been a bit of discussion about real-time linux in my workplace. A group is writing software that receives a stream of bits from a set of CCDs used for astronomical observations. They chose linux as the platform, but found out that they ended up losing a row of data every so-often (during each readout) due to the kernel going out to make sure its shirt was properly tucked in.

The obvious need here is to not lose track of the stream (in this case, flood) of bits coming from the hardware. I can imagine (though this may not actually be the case) that if a nuclear reactor has been streaming bits to it's warning systems, you certainly do not want to find out that the kernel was taking a short bathroom break. For my needs, I do not require a real-time system; if the kernel pauses for a brief moment to do some catch-up work, I don't care.

OT: safer nuclear reactors

Posted Oct 13, 2004 12:41 UTC (Wed) by jvotaw (subscriber, #3678) [Link] (5 responses)

[ Note: this is definitely not my field; apologies if I get this wrong. ]

For what it's worth, there are some designs of nuclear reactors that are fairly safe. Yes, they're operating in "critical", but it's unlikely that they will go super-critical quickly.

The two broadest, relevant questions about reactor designs include: how stable is the speed of the nuclear reaction? and, if it becomes unstable, does the speed tend to increase or decrease?

Chernobyl uses a fairly unstable design that tends to get hotter if it gets out or control. A counter-example are the CANDU reactors, which are pretty stable and safe.

There are even better designs which have not yet been implemented, such as CAESAR. As I understand it, this design uses depleted, non-radioactive Uranium as fuel. Steam moderates the speed of neutrons to the precise speed where they will cause depleted Uranium to split. If the reactor overheats or underheats, the density of the steam changes, neutrons are no longer moving at the speed necessary to sustain the reaction, and the reaction stops. The advantages of using depleted Uranium as fuel include the ability to have Uranium rods which are 100% fuel, instead of around 5% in traditional reactors, which means ~40 years of power without replacing the fuel rods. Also, the fuel rods are not usable for nuclear weapons either before or after they are used; we'd have the option of building these reactors in unstable countries without increasing nuclear proliferation.

Again, this is definitely not my field, so please forgive me (and correct me) if I'm wrong.

-Joel

OT: safer nuclear reactors

Posted Oct 14, 2004 9:58 UTC (Thu) by nix (subscriber, #2304) [Link] (4 responses)

`non-radioactive Uranium'? An interesting substance: a shame it doesn't exist.

OT: safer nuclear reactors

Posted Oct 14, 2004 13:34 UTC (Thu) by jvotaw (subscriber, #3678) [Link]

I stand corrected. Even pure U-238 is (minimally) radioactive, it seems.

The larger point remains: this is a substance that is widely considered safe enough to be used in ceramic glazing, sailboat keels, race cars, oil drills, etc. (Although, admittedly, not safe enough that you'd want to turn it in to a powder and disperse it into the air or water.)

Thanks, Wikipedia.

-Joel

OT: safer nuclear reactors

Posted Oct 15, 2004 20:07 UTC (Fri) by Baylink (guest, #755) [Link] (2 responses)

I believe the substance in question is "depleted uranium", as used in weapons systems, among other things.

A better analogy, IMHO, for when hard realtime response is necessary, would be industrial robotics: if a 400lb swingarm is about to crush a human, guaranteed millisecond response is in fact essential.

But Linus and I had an exchange about this, a few years back, carboned to this very venue, and he convinced me that if what you need is that hard realtime, then you should probably not be doing anything else with that computer.

http://lwn.net/2000/0713/backpage.php3

OT: safer nuclear reactors

Posted Oct 21, 2004 14:15 UTC (Thu) by alext (guest, #7589) [Link] (1 responses)

Generally true with respect to ordinary OS tasks. Often though you want to respond to specific events within a fixed time limit or always do X at interval Y. Neither things using all the CPU resource, leaving gaps to fill. What you do the rest of the time is low priority things that don't matter them not happening bang on interval Yn to within nanoseconds.

That is my experience from automotive engine controllers. On those we do lots of low priority things. The issue that comes in to play is testing and validation. If you are running other tasks on a controller with safety critical tasks generally you want to test everything to the higher standard if you are mixing on a shared host.

Related to running something like Linux as a low priority task under a hard real time system gives the argued (I have my doubts) ability to sandbox the none safety critical tasks so that they can't do things to interfere with the safety critical portion.

OT: safer nuclear reactors

Posted Oct 21, 2004 17:07 UTC (Thu) by Baylink (guest, #755) [Link]

This is, as always, a tradeoff.

Response latency can usefull be characterized as "M% of the time, the system will successfully respond within N ms." The more important it is to you, the closer to 100 M must be.

But the underlying point is that for values of M less than 100.0, it's often possible to combine soft-real-time techniques with throw-hardware-at-it, and get a useful result. And Linus' assertion, with which I agree now, is that if you really need 100.0%, because people may be hurt or killed, or the value of things which may be destroyed is sufficiently high, that at *best* you should indeed be running Linux as a task under a small, tight, HRT kernel.

LinuxRT and RTAI may be good enough; they may not.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 19:50 UTC (Tue) by jens.richter (subscriber, #20650) [Link] (1 responses)

Modern digital I&C safety systems require a hard realtime OS, yes it's true!

The timer tick of a system I know is 1 ms.

The response time of the I&C safety system to start the shutdown of the reactor in case of emergency is typically in the range below 1s.

You need a realtime OS, but the timig requirements are less dramatic than we think.

The perenial "Nuclear Power Plant" example

Posted Oct 13, 2004 9:52 UTC (Wed) by pwaechtler (guest, #5075) [Link]

>Modern digital I&C safety systems require a hard realtime OS, yes it's true!
>The timer tick of a system I know is 1 ms.

The event is not triggered by the timer - it's interrupt driven.
Speak about interrupt latency in the range of 3-10 us and
scheduling latency in the range of 5-100 us.

http://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sys...

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 20:20 UTC (Tue) by darthscsi (guest, #8111) [Link] (3 responses)

Hard realtime has nothing to do with *fast*, just deturministic response. If I can write a system that always has bounded latency of 1 hour, then I am in the hard realtime realm (though not useful really). If I have a system that has average latency of .00000001 nanoseconds, but on some pathalogical cases cannot be anylized, then we are out of the deturministic (hard realtime) realm, no matter how much faster this second system is.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 21:08 UTC (Tue) by hppnq (guest, #14462) [Link] (2 responses)

But the deterministic response follows from the speed at which operations can take place.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 23:22 UTC (Tue) by gilb (subscriber, #11728) [Link] (1 responses)

Nope, it isn't speed, it the ability to specify an upper bound to the response time that is required for deterministic operation. You need to be able to complete your desired task as well (which relates to speed), but the deterministic requirement simply states that you will always have to the opportunity to do your task every N time intervals.

For example, in a modern plane like the B2 or JSF, you may need to adjust the control surfaces every 10 ms in order to guarantee stable flight. You know that this will work because you ran the simulations that showed that 10 ms will work. If the response time exceeds this, the plane may be stable or it may not be, but you don't want to find out while it is flying.

The perenial "Nuclear Power Plant" example

Posted Oct 13, 2004 10:20 UTC (Wed) by hppnq (guest, #14462) [Link]

You need to be able to complete your desired task as well (which relates to speed)

That's what I mean.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 21:19 UTC (Tue) by Stephen_Beynon (guest, #4090) [Link]

I don't know about about US nuke plants, but in the UK the design aim is
that when something goes wrong the operator should have 30 mins to read
his manual before anything significant needs to be done. That includes
the control computer crashing.

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 21:51 UTC (Tue) by simlo (guest, #10866) [Link] (2 responses)

It could be that they have to switch out the generator from the electrical grid within a few ms. Typically, if the grid outside the plant is short-circuited. You can make a simple hardware solution but then you can't switch the power plant back on again. With a software solution you can switch in resistor banks and burn off the energy for a few ms until the grid is ok again.

That said: Having such systems running somekind of real-time Linux would be insane. Linux caries out too much code which can contain bugs. I would make it from scratch with either no OS at all or only the bare bones (i.e. basicly just a scheduler). Then I would make a simple protocol between this safety critical subsystem and systems running Linux to supervise it.

A realtime Linux is mostly usefull for cheap systems, where you both Linux's server/client capabilities and have a none-safety critical subsystem you need to service as well and you don't wont the extra cost of an extra CPU.

The question is how hard these real time requirements are. Very often it wont be that "hard" in the sense that the application can somewhat survive a missed deadline once in a while but it might be anoying to the user - like Xmms. For many, many applications it is like that: The models say that you have to do things of such and such rate; but in practise you can skip sample points once in a while with no critical problem.

The perenial "Nuclear Power Plant" example

Posted Oct 15, 2004 16:34 UTC (Fri) by iabervon (subscriber, #722) [Link]

Linux doesn't have all that much code if you disable everything. Of course, I'd personally design a nuclear reactor with a microcontroller to handle all the really fast hard realtime stuff, set up a watchdog to make sure the operator's computer is responding within a couple of minutes, and run Linux on the operator's computer. Linux does have far more code than a microcontroller program, no matter what you do. For that matter, with microcontrollers, you could probably set up a set of redundant ones with voting schemes just to make sure that failures don't cause problems. It's not like you're going to blow your power plant budget on microcontrollers.

The perenial "Nuclear Power Plant" example

Posted Oct 21, 2004 14:23 UTC (Thu) by alext (guest, #7589) [Link]

In national power grid systems my experience is of hardware processing that feeds an event to the software which is event driven with no scheduler involved, it just sits and spins waiting to take action. That then triggers nice, very rapid hardware breakers because of the arcing problem switching that kind of voltage (K and M sizes).

The perenial "Nuclear Power Plant" example

Posted Oct 12, 2004 23:27 UTC (Tue) by smoogen (subscriber, #97) [Link]

Most of the controls I know of in a Nuclear Power plant that need Real Time control are also in any 'steam-powered' plant. Let that coal fire plant get out of control and it will blow up.. it just isnt as scary because people dont consider tons of acid rain, mercury, and other heavy metal contaminants as 'sexy' as 3 eyed fish and giant tarantulas.

The perennial "Nuclear Power Plant" example

Posted Oct 13, 2004 3:23 UTC (Wed) by ncm (guest, #165) [Link]

The "Nuclear Power Plant Control System" example is really just a euphemism for the "Nuclear Weapon Control System" example. Really the differences between the two largely amount to how long the control system is expected to continue running (indeed, existing, in solid state) after the chain reaction begins.

The perenial "Nuclear Power Plant" example

Posted Oct 13, 2004 15:56 UTC (Wed) by AJWM (guest, #15888) [Link]

Nothing in a nuclear power plant is dependant on microsecond timing. Ultimately, nuke plants are mechanically controlled -- control rods slide in and out, coolant pumps and valves actuated, etc. and mechanical devices (at least, those big enough to see) just don't react that precisely.

Nuclear plants were operating years before we had solid-state computers, let alone ones with even microsecond cycle times.

The timing in a nuclear bomb is that critical, actually more so, to ensure that the compression wave from the triggering explosives is precisely shaped so as to uniformly squeeze the fissionable material -- an asymmetrical push will let material spew out before the reaction builds to a peak and you get some kind of fizzle yield (worst case the fissionable just melts itself). The reaction is sensitive to surface area to volume ratios (too much area and too many neutrons escape rather than hitting other U or Pu nuclei).