The state of realtime Linux

If you have some per-cpu data that is never touched in interrupt context, all you need to do to protect it is to disable preemption. This is the same as a lock, since it makes the modification of the data serialized. No one else can modify because you must be on the CPU to modify it and no one can preempt the current user that is modifying the data.

Now if the task is preempted, another task can get on the CPU and modify the data breaking the serialization of the previous task.

Now in PREEMPT_RT, instead we add a special per_cpu_locked() variable. When you grab the per-cpu variable, you grab the lock for that variable (per-cpu). Now you can be preempted, and even migrated. But you will always be touching the data on the original CPU. If someone else tries to touch that variable, it must first grab the per-cpu lock, which will cause it to block until the first task is finished with it.

This solves the serialization, but hurts scalability, since a box of 2048 CPUs can have a bit of cacheline bouncing if tasks are constantly migrating after grabbing a per-cpu variable.