Tricks and tools for PREEMPT-RT kernel

Jump to: navigation, search

Testing the latency

Once your PREEMPT-RT kernel is installed, you can test the system latency. For that, you need the Real-Time tests that are available under the rt-tests Debian package.

Then, you can start the cyclictest:

cyclictest -n -p 90 -i 1000

With such a command, you should have an average latency less than 10us. If not, there is probably an error in your setup or your hardware.

The following command will also produce an histogram of the latencies:

cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t 4

Tracking source of latencies

When you experience high latencies, this is possible to trace the call stack and see which function may cause unexpected latencies. The following command may help you for that :

cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t 4 -f -b 35

In fact, using this function, it will make available a kernel trace call so that you can see exactly which function causes the unexpected delay. You can see the trace in /sys/kernel/debug/tracing/trace. On the other hand, for using this functionality, the kernel must be build using tracer functions and debugfs (see kernel hacking option in the kernel configuration and the links on this page).

Disabling the default margin for non-RT tasks

By default, the linux kernel does not allocate 100% of its time and reserve a margin for non-RT tasks. You can see these parameters under the configuration item kernel.sched_rt_period_us and kernel.sched_rt_runtime_us To avoid any margin, the value of kernel.sched_rt_runtime_us must be the same than kernel.sched_rt_period_us So, to supress any margin, you can change the parameter value by issuing the following command:

sysctl kernel.sched_rt_runtime_us=1000000

Changing the priority of interrupt tasks

For changing priorities, you have to use the chrt program as root. It allows you to change the real-time priority associated with a thread. The good aspect is that as interrupts appears as threads/tasks, you can also change their priority. So, for changing the priority of a task, you can use the following command:

chrt -f -p 99 PID

Using this command, it will change the priority of the task PID to 99 and make it using the FIFO scheduling algorithm.

So, if you want to change the priority associated to the Real-Time Clock device, you can do the following:

ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
 1114  FF     50  90   - [irq/8-rtc0]

chrt -f -p 90 1114

The following command shows you all the task information, including the real-time priority.

ps -eLo pid,cls,rtprio,pri,nice,cmd

Changing the c-states of your processor

Some processors have some advanced power management method that could conflict with the real-time nature of the kernel. Switching from processor saving mode to a more performance mode can introduce some latencies and delays that are unexpected. In addition, even when disabled in the BIOS, there is a possibility that the function is still working. For that reason, we night also be willing to disable that in the kernel and set the maximum CState of the processor. To do so, boot the kernel using the following option: processor.max_cstate=1