User: Password:
|
|
Subscribe / Log in / New account

The realtime preemption mini-summit

The realtime preemption mini-summit

Posted Sep 29, 2009 10:59 UTC (Tue) by abacus (guest, #49001)
In reply to: The realtime preemption mini-summit by michaeljt
Parent article: The realtime preemption mini-summit

Priority inheritance is indeed messy. The following paper contains interesting background information: Victor Yodaiken, Against Priority Inheritance, July 2002.


(Log in to post comments)

The realtime preemption mini-summit

Posted Sep 29, 2009 18:14 UTC (Tue) by aleXXX (subscriber, #2742) [Link]

Yes, it's messy.

Still it's a practical tool and works in general.
Another problem is that most RTOSes don't have the full priority
inheritance implemented, but a simplified version.
E.g. eCos (and I think also vxworks) raise the priorities as expected,
but lower them again when all mutexes in the system are released again.
This can be very late.

The poster before said:
"that they should either not wait for potentially low priority processes
in critical paths,"

This is not easy.
I mean, assume you have code like

int get_foo(struct foo* f)
{
lock_mutex(&mutex);
memcpy(f, &source, sizeof(struct foo));
unlock_mutex(&mutex);
}

i.e. you just protect access to the variable "source". You may need this
information in a low priority thread. The code looks innocent, there are
no loops, nothing undeterministic, it will take at less than 10
microseconds.
So why now wait use the same mutex in all other threads ?
The issue is when a medium priority thread comes into play, suddenly the
code above can block a higher priority thread for a time determined by
the medium priority thread (which does not use that mutex at all).

Also, "make sure that the processes waited for already have the right
priority" is basically saying that all threads using the same mutex
should have the same priority ?
Doesn't work.

So, this is a hard issue, and there's no easy solution.
Maybe, try not to use too many shared variables, let your threads
communicate via messages/tokens/etc.
This helps, but everything gets asynchronous, which doesn't make things
necessarily easier.

Alex

The realtime preemption mini-summit

Posted Sep 29, 2009 20:09 UTC (Tue) by michaeljt (subscriber, #39183) [Link]

> I mean, assume you have code like
>
> int get_foo(struct foo* f)
> {
> lock_mutex(&mutex);
> memcpy(f, &source, sizeof(struct foo));
> unlock_mutex(&mutex);
> }
That particular example could be solved by RCU, although I don't want to start a showdown here, as I'm sure you would win it :) I was thinking more on the lines of avoiding contention in the critical path as much as possible though.

The realtime preemption mini-summit

Posted Oct 11, 2009 16:02 UTC (Sun) by efexis (guest, #26355) [Link]

Priority inheritance may often/usually not be the best way to do things by design (ie, try not to rely on it) sure, but is always better to have support for it to avoid inversion just-in-case, than not and have a Pathfinder style incident on your hands :-)

The realtime preemption mini-summit

Posted Sep 30, 2009 7:49 UTC (Wed) by simlo (guest, #10866) [Link]

Well, as for being one of those who actually pushed and implemented a little bit of the priority inheritance in the beginning, I must say that he is just making excuses for not making it in RTLinux, because making it right _is_ indeed very hard. From experience I know it is done wrong even in VxWorks!

But it can be done, and it is done in the current rtmutex in Linux.

From the cases he is talking about, shows me that he has not understood how to use the system at all. He does the usual mistake of not distinguishing between a mutex and a semaphore used as a condition (i.e. waiting for some external event to happen).

Yes, making an RT application work with priority inheritance mutex requires some programming rules: You can't block on a semaphore, socket etc. while holding a lock. But, heck you should always try to avoid that in any multi-threaded program to avoid the whole program eventually locking up because some message didn't arrive over TCP as expected.

In general locks should only be used "inside a module" to protect it's state. The user of the module should not be aware of it. The modules should be ordered such that low level module is not calling a highlevel module with an internal the lock taken - or you can create a deadlock. Or a even simpler rule: Newer make a call to another module with a lock held. In a RT environment with priority inheritance the module can use this to ensure the timing of all the calls to it because all the modules "lower" in the chain have a known timing and you therefore know the maximum time all the internal locks can be held by any thread.

And yes, priority inheritance takes a lot of performance. But in general you should try to avoid congestion and make your program such that the locks are not contended. The locks should only be considered as "safeguards" against a contention, which should not happen very often.

If you know how to use locks, and can avoid the pitfalls, priority inheritance will work for you - provided they are properly implemented by the OS. As is done in Linux.

Wrt. rwlocks: If a high priority, realtime writer wants the lock, it doesn't make sense to boost the readers as you don't know how many there are. What you could do was to limit the number of readers to specific number. Or you could say that writers don't boost the readers but readers can boost the single writer. That way you can't use rwlocks in real time tasks and that would not be a problem in most cases. But the kernel would need a lot of review to be sure and therefore I fully understand the current solution in the preemt RT patche.

The realtime preemption mini-summit

Posted Sep 30, 2009 8:05 UTC (Wed) by michaeljt (subscriber, #39183) [Link]

> Or you could say that writers don't boost the readers but readers can boost the single writer. That way you can't use rwlocks in real time tasks and that would not be a problem in most cases.
So to return to my previous question, this would simply mean not trying to get it "right" for this API and clearly write that on the box.

> But the kernel would need a lot of review to be sure and therefore I fully understand the current solution in the preemt RT patche.
Of course I was naively thinking that the API user would be aware of what locking they are using, but that won't hold if they are doing the locking implicitly through other APIs.

The realtime preemption mini-summit

Posted Oct 4, 2009 13:56 UTC (Sun) by dvhart (subscriber, #19636) [Link]

WRT rwlocks. We actually cap reader count to 1 in PREEMPT_RT for that very reason. This is unfortunate, and one of the causes for performance degradation on -rt for certain workloads. There was some discussion during the rt-summit in Dresden about making kernel rwlocks non-pi-aware for this reason. Some more investigation is needed before we make a decision there.

The realtime preemption mini-summit

Posted Oct 6, 2009 15:37 UTC (Tue) by simlo (guest, #10866) [Link]

As I said: You could leave half-PI-aware : Let readers boost the writer, but not the other way around. This will most likely work in many cases. It means a RT task can't write-lock a rwlock must defer the operation to another task. Config options are needed....


Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds