Author: Tobit Flatscher (August 2022)
The following sections will outline a few things to consider when setting up a real-time capable systems and optimizations that might help improve its real-time performance. This guide is largely based on the exhaustive Red Hat optimisation guide, focusing on particular aspects of it.
The latency on most computers that are optimised for energy efficiency - like laptops - will be magnitudes larger than the one of a desktop system as can also clearly be seen browsing the OSADL latency plots. It is therefore generally not advisable to use laptops for real-time tests (with and without a Docker). Some single-board computers like the Raspberry Pi, seem to be surprisingly decent but still can't compete with a desktop computer. The Open Source Automation Development Lab eG (OSADL) performs long-term tests on several systems that can be inspected on their website in case you want to compare the performance of your system to others.
Real-time performance might be improved by changing kernel parameters when recompiling the kernel. On the OSADL long-term test farm website you can inspect and download the kernel configuration for a particular system (e.g. here). Important settings that might help reduce latency include disabling all irrelevant debugging feature.
During operation the operating system may scale the CPU frequency up and down in order to improve performance or save energy. Responsible for this are
- The scaling governor: Algorithms that compute the CPU frequency depending on the load etc.
- The scaling driver that interacts with the CPU to enforce the desired frequency
For more detailed information see e.g. Arch Linux.
While these settings might help save energy, they generally increase latency and should thus be de-activated on real-time systems. This will be discussed for Intel systems in more detail in the next section. This should work quite similar on AMD processors as outlined here.
On Intel CPUs the driver offers two possibilities to do so (see e.g. here and here as well as the documentation on the Linux kernel here or here for a technical in-depth overview):
- P-states: The processor can be run at lower voltages and/or frequency levels in order to decrease power consumption
- C-states: Idle states, where subsystems are powered down.
Both states are numbered, where 0 corresponds to operational state with maximum performance, and the higher levels corresponding to power-saving (likely latency-increasing) modes.
Additionally there are other dynamic frequency scaling features, like Turbo-Boost that allow to temporarily raise the operating frequency above the nominal frequency when demanding tasks are run. To what degree this is possible depends on the number of active cores, current and power consumption as well as CPU temperature. Values for different processors can be found e.g. on WikiChip. This feature can be deactivated inside the operating system as well as the BIOS and is often de-activated on servers that are supposed to run 24/7 on full load.
For a better real-time performance the idle states should be turned off and the P-states changed to performance! How this can be done is described here in more detail.
Typically one will reserve a CPU core for a particular real-time task as described for Grub and Linux here and here. If hyper-threading is activated in the BIOS also the corresponding virtual core has to be isolated. The indices of the second virtual cores follow the physical ones.
Additionally it makes sense to also isolate the corresponding hardware interrupts (IRQs) (on the same CPU), disabling the irqbalance daemon and binding the process to a particular CPU as described here. This is in particular crucial for applications that include network and EtherCAT communication.