常见问题

Need technical support?

Warning/error "Cycle too long"

Question

A demanding real-time scenario with GPU based sensors might result in the following warning and potential error:

  • Cycle too long

This happens when the calculation of the GPU based sensors is not completed within the defined time frame. Each calculated sensor is distributed to a sensor cluster in which this time frame is defined by its "latency". If the time frame ended and the sensor is still calculating, the warning mentioned above will be logged and the current calculation cycle for the sensor will be skipped.

Solution

Reasons for the slow computation can be one or more of the following:

  • Demanding sensor parameterization
    The sensor might use too many rays, a very high resolution or other comparable values which require a lot of computation power.
    Solution:
    This can be improved by using better hardware for the calculation. The current computation load can be checked by the following console-command on both Windows and Linux:
    nvidia-smi --loop=1
    This command will print a performance log of all built-in GPUs. The current workload of each GPU is displayed by the value "GPU-Util". If this value is very high (e.g. >90%) the computing power of the GPU is most likely too weak for all assigned sensors.
  • Multiple sensors use the same GPU
    If multiple sensors are combined in a single sensor cluster and/or multiple clusters are assigned to the same GPU, their resulting computation load might add up.
    Solution:
    Split the sensors to different clusters on separate GPUs or deactivate some sensors and only use a subset for each simulation.

  • Power saving modes might be enabled
    If a GPU has low computation load for too long it might switch into a power saving mode. Within this mode its computation power is noticeably lower than normal. When a new sensor calculation cycle is triggered the GPU takes some time to switch back into performance mode. This can affect multiple sensor calculation cycles and will happen more frequently with better GPUs.
    Solution:
    The performance mode should be set as a default on the system containing the GPU in question by using:
    • NVIDIA Control Panel (Windows)
    • nvidia-settings (Linux)
Need technical support?
  • Date: 05.10.2023
  • Product: CarMaker/HIL
  • Version: 12.0
  • Component: Sensor Models
  • Language: English

Tags

GPUSensor
SensCompute