MontaVista unveils "open" hard-real-time Linux project (LinuxDevices) [LWN.net]

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 7:55 UTC (Tue) by simlo (subscriber, #10866) [Link]

That so called hard real time systems "guarantee" deadlines is not entirely correct in practise. It does hold when you start out afresh on a simple system and design it. But as soon as the system grows bigger and "old" the realtime guarantees will be void as noone will be able to analyze the system. At that point you are back at the "test and see if it works" approach - just as you are with the Linux kernel.

That said a hard real time kernel will do better and more consistence at these tests than a Linux kernel where there is not real prioritizing. On Linux you thus will get into trouble once in a while once you get close to 100% CPU utilization, whereas on a hard realtime system you might be able to do the job within the deadlines every time. But you can't guarentee the last statement.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 22:14 UTC (Tue) by Russell (guest, #1453) [Link]

I agree, and I also believe that we don't need a real time kernel for this job, just a low latency one. It's just that we traditionally use hard real time for all our projects, so a lot of noise was made about linux not being real time.

Having real time marketing speak is all a lot of people really need. Even if they are not doing real time.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 22, 2004 6:41 UTC (Fri) by job (guest, #670) [Link]

There's also the important point of *how* it fails. Does one thread stop
momentarily, letting at least one fulfill it's requirements? Is it logged
and reported? Failing gracefully can be very important in a lot of
equipment.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 9:37 UTC (Tue) by simlo (subscriber, #10866) [Link]

I am not entirely convinced making Linux itself into a real time system is either a good idea or will work at all. To my knowledge hard real time systems are designed specifically to their perpose. I.e. if you want hard real time you have to sit down and analyze the system wrt. priorities and resources. Now if you take a Linux kernel with the real time patches you will get a general system. You will do a lot of tuning to make it able to meet your deadlines.

First of all you have to adjust the priorities of all the tasks in the system and check that. Are you depending on sending/recieving realtime data over the network? If so you have to set the task handling the network interrupt high priority. Otherwise you will have to give it low prority.

Secondly, there is a performance problem: When do you want to use a spinlocks and when do you want to use a mutex? Spinlocks are the cheapest seen wrt. through-put but if you are using the a spinlock you increase the interrupt latency by the execution time within the spinlock. If your system can tolerate interrupt latencies of say 1 ms you thus safely can use spinlocks for all resources with access time below 1 ms. This would increase performance a lot over a system using mutexes in all these places. But a general system like Linux have no idea of what the required deadlines are and what CPU's to even meassure them on. Thus needs to make every lock has to be made into a mutex. This, however, will give a suboptimal performance on all the systems not requiring these low latencies.

Also I am in favor of seperating the real-time system handling the hardware (in a wider sense: control and data collection) from the none-real-time system (management, data storage, human interface etc). Having the two parts in "mixed up" potentially leaves you huges problems with shared resources. Even if you have mutex with priority inversion prevention you still get into trouble with none-real-time tasks eating resources from the real-time tasks or the real-time part tries to access none-real-time resources. In theory it works but all the sudden a coder forgets that he is in a realtime task and writes directly to a file. Having seperate CPUs or a system like RTAI- or RT-Linux will automaticly seperate the two systems for you.

Thus the "old system" with a seperate system to take care of the real time problems and Linux to take care of all the rest is not a such bad system!

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 14:37 UTC (Tue) by hppnq (guest, #14462) [Link]

Fortunately at this very moment Ingo Molnar seems to be hacking away at some of the issues you address here. ;-) (Check your favourite LKML mirror, for instance.)

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 15:16 UTC (Tue) by simlo (subscriber, #10866) [Link]

Can you enlighten me, please?

The concerns I mentioned above was about the far from finished full RT Linux kernel. What Ingo Molnar is hacking on is the present kernel moving it slowly towards the RT kernel I have concerns about.

For the order of things: Although I have concerns, I do agree it is the way to go. I just wanted to say that even when (not if!) the technology works, Linux will not out of the box be a hard real time OS and it still takes a lot to make a real time system out of it - more so than smaller OS'es where there are fewer subsystems to worry about.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 12, 2004 20:47 UTC (Tue) by hppnq (guest, #14462) [Link]

The path has to start somewhere.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 9:49 UTC (Wed) by simlo (subscriber, #10866) [Link]

Can I see that math, please? Do you have a link?

As I see it basic concepts for a scaleable SMP system and single CPU real time system are the same. However, you have to tune the various concepts differently. In a SMP system you will want to use spinlocks for locks whereas in a real time system you will more often want a more expensive mutex lock (with a priority inversion prevention) such that you can be preempted within the lock by higher priority tasks. Also in a SMP you just want to run ahead in interrupt whereas in a real time system you often want to do your stuff in a lower priority task which can be preempted.

But these differences can be encapsulated! Make it compile time parameters you can choose in the config along with the priorities of the kernel tasks. I.e. when you pick a driver you should be able to pick a wether it's lock should a spinlock or a mutex and wether the interrupt handler runs in a interrupt context or in a thread and if so at what priority. The driver developer would not have to worry about wether his driver lock is a spinlock or a mutex, nor if his interrupt routine really runs in interrupt context or on some thread - if it runs in interrupt it should also be able to run in a thread.

A whole other issue is the way the kernel allocates memory with kmalloc(). A lot of calls a routed into kmalloc() calls which can take an arbitrary amount of time to complete(?) On a real time system each subsystem preallocates its resources. A layer of resource preallocation is needed as well as the mutex enhancements. That will have the benefit of making the system more reliable for everybody as the memory are no longer taken from the same pool. Thus the subsystems using a lot of memory will be the ones which will run into problems, not everybody else. The negative side is that the parameters for preallocation have to be choosen. How many recieve buffers should your ethernet driver forinstance preallocate??

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 12:07 UTC (Wed) by hppnq (guest, #14462) [Link]

I think lm has a point. Real time systems and time sharing systems share a lot of common concepts but their purposes are very different in lots of respects. (Refrigerators and microwaves share loads of concepts, even thermodynamically, but their purposes are almost orthogonal.)

Also, there is the issue of maintenance. Even if you could pull it off technically, the result would probably be a maintainer's nightmare, even after ironing out the practical obstacles that exist today. And don't forget that real time systems in general only need a very limited subset of the functionality that a Linux desktop or server system needs to provide, but integrating real time systems would affect *all* of the functionality of the bigger system.

I think the question whether it is wise to accept patches like this into mainline is very legitimate.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 12:54 UTC (Wed) by simlo (subscriber, #10866) [Link]

As I said, by using abstractions like the one Igno Molnar have used there is close to no problems of maintance. The basic way of using locks is the same in both cases. What should be done is that on Linus' tree everything tested for the "traditional" setup where spinlocks are used in most places. Then a company like MontaVista can configure their system to use mutexes instead and test that.

The important thing is that the code is the same but only different configurations are used. That would ease the patching and it will not destroy the time sharing properties of the 90% of the direct users. (If Linux is made real time and used in embedded enviroments the number of actual users in that segment would soon equal the number of users on servers and workstations.)

I agree that when you look at the real-time part of a system, it needs far less functionality than a time sharing system - and even if the further functionality is available it you must not use it as it will destroy your real-time properties! However, in a lot of cases you one part of the system having real-time properties and the other part having traditional time sharing properties. A good solution would be to have two CPUs and two OSes. But that is expensive in both hardware and development resources as the developers would have to know and maintain two completely different OS'es.

Where I work the result is that the management force us to put typical sharing applications onto a real-time OS which can not really support it. The main argument is that we don't have development resources to maintain two OS'es for our processor boards.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 19:07 UTC (Wed) by lm (guest, #6402) [Link]

Sure, start with this: http://citeseer.ist.psu.edu/barabanov97linuxbased.html and then read the references.

Without intending to sound patronizing, I would have expected anyone who has an opinion about real time to know this, it's operating systems 101 or was when I went to college in the 1980's. There are lots of texts and papers about real time vs time sharing. I thought most classes on queueing theory will use real time vs time sharing as an illustration of a two different classes of problems in queuing.

Anyway, the point is that this is pretty basic stuff to someone who has a solid foundation in operating systems. Unfortunately, it appears that not many people have that foundation because the idea of mixing realtime into a time sharing system just keeps coming back. I'm baffled by it, in my mind it is like the idea that "2 + 2 = 5" keeps coming back, to me at least, it is that obviously wrong.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 22:31 UTC (Wed) by simlo (subscriber, #10866) [Link]

I have studied physics and only recently started on computer science in my spare time. I have had small courses on real-time programming as part of my job. I spend my time at work working with VxWorks and part of my job is to help the application programmers figure out how to use the real-time primitives in that system. So although I only have 3 years of experience I think I know something about real-time programing.

I have briefly browsed through the references of the paper you send. I looked for the theoretical papers describing rate monotonic scheduling etc. It will take me a long time to find a place where even one of the articles claims that you can't mix time-sharing and real-time systems. Can you point at it for me?

But it would surprise me if you can find such a place along valid arguments. Ofcourse, one of the criteries for it to be doable is that the part of the system having real-time requirements runs in threads with the highest priority and never blocks on resources where there is non-deterministic amount of time before they are available.

If you use priority inheritance on the mutexes you can still calculate the worst case time for entering such a region for your real-time thread: You have to add the maximum time any lower priority thread can hold the mutex to your execution time. Now as any thread holding the mutex is moved up to the priority of your real-time thread it is preempted just as much your real-time thread is. Thus you can in your model assume this extra execution is done in your own thread.

I.e. you can guarantie to a thread real-time execution if it does not call into regions holding a mutex which can be hold for a non-deterministic amount of time. I.e. you can't call the file system but it should be able to put a buffer onto a transmit-queue since the execution time for such an operation is constant.

Ofcourse, fairness have to switched off in the scheduler for priroties over a certain level: real-time tasks should keep their priority even if they use a lot of the CPU.

Most subsystems, even if the locks are changed to mutexes with priority inheritance (or ceiling if you prefer) will not have deterministic locking times. Any real-time thread thus can't call into these subsystems but have to defer the operations to low priority, non critical tasks. But basic stuff like network drivers can easily be changed to behave deterministic.

You will thus be able build a real-time application with Linux if you make sure you only access drivers and other subsystems which behave deterministicly wrt. CPU-time. You wont be able use IP, the filesystem or even allocate memory in the real-time part of your application but have to make low priorty worker threads for doing that.

But again that is how it is under VxWorks right now! Only the very core parts of VxWorks is real-time. The rest (of which a lot is not original written for VxWorks) does not have the properties I mentioned above and you will have to take care.

Thus as I see it Linux can be made into a real-time system with the same deterministic properties as VxWorks with some efford (but not with as good latencies as VxWorks). I still fail to see what it would break on the ordinary time-sharing systems - except for the risc of injecting bugs with the patches.