MontaVista unveils "open" hard-real-time Linux project (LinuxDevices) [LWN.net]

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 9:49 UTC (Wed) by simlo (subscriber, #10866) [Link]

Can I see that math, please? Do you have a link?

As I see it basic concepts for a scaleable SMP system and single CPU real time system are the same. However, you have to tune the various concepts differently. In a SMP system you will want to use spinlocks for locks whereas in a real time system you will more often want a more expensive mutex lock (with a priority inversion prevention) such that you can be preempted within the lock by higher priority tasks. Also in a SMP you just want to run ahead in interrupt whereas in a real time system you often want to do your stuff in a lower priority task which can be preempted.

But these differences can be encapsulated! Make it compile time parameters you can choose in the config along with the priorities of the kernel tasks. I.e. when you pick a driver you should be able to pick a wether it's lock should a spinlock or a mutex and wether the interrupt handler runs in a interrupt context or in a thread and if so at what priority. The driver developer would not have to worry about wether his driver lock is a spinlock or a mutex, nor if his interrupt routine really runs in interrupt context or on some thread - if it runs in interrupt it should also be able to run in a thread.

A whole other issue is the way the kernel allocates memory with kmalloc(). A lot of calls a routed into kmalloc() calls which can take an arbitrary amount of time to complete(?) On a real time system each subsystem preallocates its resources. A layer of resource preallocation is needed as well as the mutex enhancements. That will have the benefit of making the system more reliable for everybody as the memory are no longer taken from the same pool. Thus the subsystems using a lot of memory will be the ones which will run into problems, not everybody else. The negative side is that the parameters for preallocation have to be choosen. How many recieve buffers should your ethernet driver forinstance preallocate??

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 12:07 UTC (Wed) by hppnq (subscriber, #14462) [Link]

I think lm has a point. Real time systems and time sharing systems share a lot of common concepts but their purposes are very different in lots of respects. (Refrigerators and microwaves share loads of concepts, even thermodynamically, but their purposes are almost orthogonal.)

Also, there is the issue of maintenance. Even if you could pull it off technically, the result would probably be a maintainer's nightmare, even after ironing out the practical obstacles that exist today. And don't forget that real time systems in general only need a very limited subset of the functionality that a Linux desktop or server system needs to provide, but integrating real time systems would affect *all* of the functionality of the bigger system.

I think the question whether it is wise to accept patches like this into mainline is very legitimate.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 12:54 UTC (Wed) by simlo (subscriber, #10866) [Link]

As I said, by using abstractions like the one Igno Molnar have used there is close to no problems of maintance. The basic way of using locks is the same in both cases. What should be done is that on Linus' tree everything tested for the "traditional" setup where spinlocks are used in most places. Then a company like MontaVista can configure their system to use mutexes instead and test that.

The important thing is that the code is the same but only different configurations are used. That would ease the patching and it will not destroy the time sharing properties of the 90% of the direct users. (If Linux is made real time and used in embedded enviroments the number of actual users in that segment would soon equal the number of users on servers and workstations.)

I agree that when you look at the real-time part of a system, it needs far less functionality than a time sharing system - and even if the further functionality is available it you must not use it as it will destroy your real-time properties! However, in a lot of cases you one part of the system having real-time properties and the other part having traditional time sharing properties. A good solution would be to have two CPUs and two OSes. But that is expensive in both hardware and development resources as the developers would have to know and maintain two completely different OS'es.

Where I work the result is that the management force us to put typical sharing applications onto a real-time OS which can not really support it. The main argument is that we don't have development resources to maintain two OS'es for our processor boards.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 19:07 UTC (Wed) by lm (guest, #6402) [Link]

Sure, start with this: http://citeseer.ist.psu.edu/barabanov97linuxbased.html and then read the references.

Without intending to sound patronizing, I would have expected anyone who has an opinion about real time to know this, it's operating systems 101 or was when I went to college in the 1980's. There are lots of texts and papers about real time vs time sharing. I thought most classes on queueing theory will use real time vs time sharing as an illustration of a two different classes of problems in queuing.

Anyway, the point is that this is pretty basic stuff to someone who has a solid foundation in operating systems. Unfortunately, it appears that not many people have that foundation because the idea of mixing realtime into a time sharing system just keeps coming back. I'm baffled by it, in my mind it is like the idea that "2 + 2 = 5" keeps coming back, to me at least, it is that obviously wrong.

MontaVista unveils "open" hard-real-time Linux project (LinuxDevices)

Posted Oct 13, 2004 22:31 UTC (Wed) by simlo (subscriber, #10866) [Link]

I have studied physics and only recently started on computer science in my spare time. I have had small courses on real-time programming as part of my job. I spend my time at work working with VxWorks and part of my job is to help the application programmers figure out how to use the real-time primitives in that system. So although I only have 3 years of experience I think I know something about real-time programing.

I have briefly browsed through the references of the paper you send. I looked for the theoretical papers describing rate monotonic scheduling etc. It will take me a long time to find a place where even one of the articles claims that you can't mix time-sharing and real-time systems. Can you point at it for me?

But it would surprise me if you can find such a place along valid arguments. Ofcourse, one of the criteries for it to be doable is that the part of the system having real-time requirements runs in threads with the highest priority and never blocks on resources where there is non-deterministic amount of time before they are available.

If you use priority inheritance on the mutexes you can still calculate the worst case time for entering such a region for your real-time thread: You have to add the maximum time any lower priority thread can hold the mutex to your execution time. Now as any thread holding the mutex is moved up to the priority of your real-time thread it is preempted just as much your real-time thread is. Thus you can in your model assume this extra execution is done in your own thread.

I.e. you can guarantie to a thread real-time execution if it does not call into regions holding a mutex which can be hold for a non-deterministic amount of time. I.e. you can't call the file system but it should be able to put a buffer onto a transmit-queue since the execution time for such an operation is constant.

Ofcourse, fairness have to switched off in the scheduler for priroties over a certain level: real-time tasks should keep their priority even if they use a lot of the CPU.

Most subsystems, even if the locks are changed to mutexes with priority inheritance (or ceiling if you prefer) will not have deterministic locking times. Any real-time thread thus can't call into these subsystems but have to defer the operations to low priority, non critical tasks. But basic stuff like network drivers can easily be changed to behave deterministic.

You will thus be able build a real-time application with Linux if you make sure you only access drivers and other subsystems which behave deterministicly wrt. CPU-time. You wont be able use IP, the filesystem or even allocate memory in the real-time part of your application but have to make low priorty worker threads for doing that.

But again that is how it is under VxWorks right now! Only the very core parts of VxWorks is real-time. The rest (of which a lot is not original written for VxWorks) does not have the properties I mentioned above and you will have to take care.

Thus as I see it Linux can be made into a real-time system with the same deterministic properties as VxWorks with some efford (but not with as good latencies as VxWorks). I still fail to see what it would break on the ordinary time-sharing systems - except for the risc of injecting bugs with the patches.