Juggling software interrupts and realtime tasks

For my workload it was the IO scheduler which lead to skips. Switching to BFQ solved the problem.

It might not fix your problem but it's at least easy to try:
sudo echo bfq > /sys/block/sda/queue/scheduler