Testing and validation in virtual test driving
Interview with Alexander Frings, Martin Herrmann, IPG Automotive GmbH
The importance of testing and validation in virtual test driving is growing as a result of an increasing number of ECU functions and a nearly unlimited quantity of possible scenarios. In this interview, Alexander Frings (Manager Product Management Engineering Services) and Martin Herrmann (Business Development Manager ADAS and Automated Driving) told us how new possibilities for scenario generation can contribute to the highest test coverage possible.
What role do scenarios play in virtual test driving?
Frings: Scenarios are essential. We want to ensure vehicle safety in all imaginable situations, so it is crucial to test a very large number of scenarios. Of course, we are aware that we will never be able to test all relevant scenarios and therefore never reach full test coverage. There will always be corner cases that we do not know about or that we did not expect.
The more situations are analyzed and transformed into virtual test drives, the more critical scenarios can be tested. A high number of scenarios helps us to challenge developed driving functions.
To reduce uncertainty in virtual validation, we need to take advantage of all available test scenario sources. Some examples are derived test cases from specified system requirements, accident data bases, standardized and normed tests or records from field tests.
Can the method “ScenarioRRR” (Record, Replay, Rearrange) be applied?
Frings: Yes. This is a method that supports scenario generation from real sensor measurement data. First, the trajectories are used for a “Replay” scenario in the simulation. As soon as the traffic objects are modeled in the CarMaker test run, they can be adapted if necessary to allow for variation of the recorded scene. At Open House 2019, we presented how the conversion of road and trajectories and the parameter variation could look like in highway scenarios. For this year’s Open House, we were looking for other freely available data sets to illustrate this method.
Amongst others, we encountered LevelXdata which is a collection of data sets from fka GmbH. The special feature of this data set is that drones measure a fixed point. The recordings can therefore capture a great number of objects.
Are there other possibilities to transfer real scenarios to the virtual world?
Herrmann: We have just started working on a service that transfers highly complex scenarios from the real world to simulation with our partner Scale AI. Scale AI specializes in labeling sensor data from autonomous vehicles. The annotations are typically used as ground truth by perception teams in the development of object detection and sensor fusion algorithms. But they are also an excellent basis for the generation of simulation scenarios for CarMaker with the ScenarioRRR method.
The client tests his vehicle fleet in the field and activates the data recording for example via an intervention by the driver when an error is detected in the system. All sensor data from the previous seconds are then recorded. At the end of the day, the raw sensor data are uploaded to the Scale AI platform where they are annotated: Dynamic objects, traffic signs, road markings etc. are labeled and classified. The abstracted data are then forwarded to the customer’s perception team in form of an object list to train the detection algorithms. They are also sent to us because our job is to extract a CarMaker scenario with the described ScenarioRRR method and to prepare the data for variations. This scenario can then be made available to planning and function development teams in order to integrate it into their test catalogs.
You mentioned accident data bases. To what extend are they suitable sources?
Frings: Accident data bases are a great source for scenarios because they classify real types of accidents and their severity. People interested in connecting CarMaker to accident data bases can be divided into different groups with different interests. First, the accident researchers looking at the consequences of an accident. For them it is crucial that the recorded event of the accident is faithfully rendered. On the one hand, the velocity plot has to be accurate. On the other hand, both the point of collision and the angle at which the vehicles meet have to match precisely. In this case, the simulation is no longer a closed-loop vehicle dynamics simulation. By indicating the position of the ego vehicle externally, we are relocating the vehicle.