Tricks and tools for PREEMPT-RT kernel
Testing the latency
Once your PREEMPT-RT kernel is installed, you can test the system latency. For that, you need the Real-Time tests that are available under the rt-tests Debian package.
Then, you can start the cyclictest:
cyclictest -n -p 90 -i 1000
With such a command, you should have an average latency less than 10us. If not, there is probably an error in your setup or your hardware.
The following command will also produce an histogram of the latencies:
cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t 4
Tracking source of latencies
When you experience high latencies, this is possible to trace the call stack and see which function may cause unexpected latencies. The following command may help you for that :
cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t 4 -f -b 35
In fact, using this function, it will make available a kernel trace call so that you can see exactly which function causes the unexpected delay. You can see the trace in /sys/kernel/debug/tracing/trace. On the other hand, for using this functionality, the kernel must be build using tracer functions and debugfs (see kernel hacking option in the kernel configuration and the links on this page).
Disabling the default margin for non-RT tasks
By default, the linux kernel does not allocate 100% of its time and reserve a margin for non-RT tasks. You can see these parameters under the configuration item kernel.sched_rt_period_us and kernel.sched_rt_runtime_us To avoid any margin, the value of kernel.sched_rt_runtime_us must be the same than kernel.sched_rt_period_us So, to supress any margin, you can change the parameter value by issuing the following command:
Changing the priority of interrupt tasks
For changing priorities, you have to use the chrt program as root. It allows you to change the real-time priority associated with a thread. The good aspect is that as interrupts appears as threads/tasks, you can also change their priority. So, for changing the priority of a task, you can use the following command:
chrt -f -p 99 PID
Using this command, it will change the priority of the task PID to 99 and make it using the FIFO scheduling algorithm.
So, if you want to change the priority associated to the Real-Time Clock device, you can do the following:
ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq" 1114 FF 50 90 - [irq/8-rtc0] chrt -f -p 90 1114
The following command shows you all the task information, including the real-time priority.
ps -eLo pid,cls,rtprio,pri,nice,cmd
Changing the c-states of your processor
Some processors have some advanced power management method that could conflict with the real-time nature of the kernel. Switching from processor saving mode to a more performance mode can introduce some latencies and delays that are unexpected. In addition, even when disabled in the BIOS, there is a possibility that the function is still working. For that reason, we night also be willing to disable that in the kernel and set the maximum CState of the processor. To do so, boot the kernel using the following option: processor.max_cstate=1
- OSADL QA Farm: https://www.osadl.org/QA-Farm-Realtime.qa-farm-about.0.html
- Debugging the kernel using Ftrace: http://lwn.net/Articles/365835/