Performance Optimization

• Do not use shared caches for the processor cores of your controller.

• If you are using Ethernet-based fieldbuses for your target device, then use a physical adapter for them. Do not use a switch architecture.

• Use a real-time kernel.

  We recommend using the RT preempt kernel (https://rt.wiki.kernel.org) for your Linux system. In the case of Debian and Ubuntu distributions, you will find an RT kernel as a package which you can easily install using the apt command. For details about this, see the manual of your distribution.

  • On Debian systems

    sudo apt-get install linux-image-rt-amd64

  • Check which kernel you are using with the uname -a command. for example.

• Avoid using a window manager, GUI/X server, or similar on your system.

  Using these tools could affect the real-time capabilities of your system, which results in high jitter in the IEC application).

Use of "rt-tools":

• Installing "rt-tools": sudo apt install rt-tests

• Starting "cyclictest": sudo cyclictest -p 99 -t -m

• The man cyclictest command shows you more command-line options which you can use to better measure more or specific performance aspects of your system.

• Disable CPU energy-saving mode.

• Disable hyperthreading.

  You can use the following command to disable hyperthreading (example):

  echo off | sudo tee /sys/devices/system/cpu/smt/control

• Disable CPU frequency scaling and switching as much as possible.

  For example, set the minimum and maximum CPU frequency to the same (fixed) value.

• Disable the real-time throttling mechanism of the Linux kernel, because this can lead to jitter on your system.

  For more information, see: The Linux Foundation: Scheduling – RT throttling

  You can use the following command to disable real-time throttling (example):

  echo -1 > /proc/sys/kernel/sched_rt_runtime_us

• Check and change the scheduling/scaling governor.

  • Check the scheduling/scaling governor used:

    cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

  • Change the used scheduling/scaling governor (as root/admin) to Performance:

    echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    # set it for all available cores:
    echo "performance" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
    echo "performance" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
    echo "performance" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor

    Note that the scaling governor has to be reset each time the system is started. You can also set it by means of the kernel configuration.

    The intel_pstate kernel driver can make the correct setting more difficult. Therefore, you should use the cpufreq-info command to check your configuration.

  • You can also use programs such as cpu-freq-utils. This could interfere with the Intel pstate drivers. These drivers normally require a different approach to set the CPU frequency.

  • For more information, see: https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt

• Disable HyperV in the BIOS (if available).

• Create a test application which roughly describes the application for which you want to optimize your system (in terms of required resources, required performance, and project size).

• You can do this iteratively: first create a very rough approximation and then expand it to get closer and closer to the actual application.

• Min. Jitter (µs) / Max. Jitter (µs)

  • The values should be as close as possible to the values determined by the "cycletest" program. Details are described in the section Setting up Linux.

  • For more information about jitter and latency, see: Definitions of Jitter and Latency

  • For more information about the Monitoring tab, see: Tab: Monitoring

• Average Cycle Time (µs) / Max. Cycle Time (µs) Cycle Time (µs)

  • The maximum cycle should never come close to the configured cycle time. This leads to problems as soon as the system experiences a high load.

  • Always keep the cycle time as short as possible.

• Split long-running tasks into multiple smaller tasks.

• When running, the maximum cycle time of the highest-priority task should never reach the configured cycle time. If you cannot avoid this, then you should increase the configured cycle time to make sure of consistent execution times.

• When starting fieldbus systems (for example, EtherCAT or PROFINET), the start cycles may result in a slightly higher CPU load. In this case, it makes sense to monitor the CPU load shortly after starting the application.

The following options cannot be changed in CODESYS Virtual Control SL, but need to be configured on the host:

• Set the value for DisableCpuDmaLatency to 1:

  [SysCpuHandling]
  Linux.DisableCpuDmaLatency=1

  • Note that this is the default setting as of runtime version 4.11.0.0.

  • To check your runtime version, click Extras → Update Linux → System → System info.

• Check whether or not the real-time kernel is really being used.

  • As of version 4.11.0.0, you can use the PLC shell command rt-get kernelinfo to check this.

  • If you have an older version, then you can do this directly on the command line with the command uname -a.

  • If the real-time kernel is not being used, then you need to start again from the beginning.

• You can use the tests from the previous step Configuring the CODESYS runtime system to test your IEC application.

• You can use the Multicore feature in CODESYS.

• Determine the correct priorities for your tasks. Important tasks should have a higher priority.

  Table 5. Mapping of IEC task priorities and Linux thread priorities:

  IEC Task Priority	Linux Priority
  --	88 (SCHED_FIFO)
  --	57 (SCHED_FIFO)
  0 (highest real-time priority)	56 (SCHED_FIFO)
  15 (lowest real-time priority)	41 (SCHED_FIFO)
  16 (not real-time priority)	0 (SCHED_OTHER)
  31 (not real-time priority)	0 (SCHED_OTHER)
  --	0 (SCHED_OTHER)
  Note On a conventional Linux system, most interrupts (IRQs) and kernel workers are at Linux priority 50. Using priorities (with high load) above that can lead to system functions (network/storage) not working as desired.

  
  
  

• You can configure the priority of a task in the respective task configuration.

  

If you do not achieve the desired performance after all the steps mentioned here, then you can look at the following sections:

• Fieldbus-specific information

• [For experts] – Additional tools and topics

• Check the Send Time / Recv Time values on the EtherCAT status page.

  • An x64 CPU with an Intel Core i7 processor and a good adapter should have less than 10µs.

  • ARM processors with an integrated chip will have ~50µs or more.

  

• Check the Send Time / Recv Time values on the PROFINET status page.

  • An x64 CPU with an Intel Core i7 processor and a good adapter should have less than 10µs.

  • ARM processors with an integrated chip will have ~50µs or more.

  

• To sort your priorities and required IRQs, you can use the PLC shell commands irq-list and irq-set-prio.

  • These commands use the normal Linux priorities and not the IEC priorities.

  • This optimization is not possible in CODESYS Virtual Control SL, but rather on the host system.

• Use a separate network adapter for the devices.

• For more information, see:

  • EtherCAT

  • PROFINET

  • Ethernet/IP

  • CANbus, CANopen

• KernelShark / kernel trace

  • This is the program of choice if you find that the cause of your performance problems to be with scheduling. With the help of "kernel trace", you can see whether your task is being interrupted by another task, another service, or an interrupt.

    • trace-cmd record -p function

    • You can use KernelShark to examine the generated trace.dat file.

  • In general, scheduling problems can be divided into two categories:

    • Supersession / preemption

      • If the interrupt or the task preventing execution is unnecessary, then disable it.

      • Increase the priority of your task or reduce the priority of the others.

      • Switch to a different CPU core.

      Important

      Check the effectiveness of the changes by using the programs described in the previous chapter.

    • Execution time

      See below: "Kernel function trace"

• Kernel function trace

  • If you determine that the code execution time is too high, then you can use this tool to pinpoint the problem.

  • If the function with too high execution time is in your own code, then you need to optimize it.

  • If the function with too high execution time is in the kernel, then you can try to achieve the desired functionality with another kernel function. Alternatively, you could also pass configuration parameters to the kernel driver in order to reduce the execution time.

  • If none of these options solve the problem, then you will probably need to use more powerful hardware.

Look at the following points and check whether or not they are the correct tool to achieve your performance goals:

• PREEMPT_FULL

• isolcpu

• rcu_nocbs

• rcu_nocb_poll

• nosoftlockup

• irqbalance disable

• kernel.sched_rt_runtime_us