With the application of various new and complex technologies like AI methods, testing and validating automated and self-driving functions is becoming more and more difficult. The development cycles for such systems are continuously getting shorter and more challenging. Therefore, without a reliable, efficient, and modern infrastructure foundation to enable agile software development, it could prove to be a very difficult task. For autonomous driving systems, it is necessary to validate not only the system's expected behavior, but also the system's behavior when unintended interactions within its environment occur. Autonomous driving vehicles must function properly in all scenarios within the defined operational design domain (ODD). Tackling this challenge requires a vast amount of test scenarios and test cases which all need to be managed, prepared, documented, and executed in simulation, on the test track, and on the road. Thus, such a testing methodology together with a comprehensive CI/CD framework for all development and testing activities would be required, particularly for ADAS/AD systems.

A comprehensive and reliable agile testing process for automated and autonomous driving functions must follow a reasonable path from ODD input and test scenario generation to test execution and final assessment of the achieved test coverage. In the following article, we will provide an overview of our ontology-based testing approach with a CI/CD framework based on CarMaker and Amazon AWS Cloud. We have developed an efficient, scalable, and customized CI/CD cloud pipeline to orchestrate the automatic dataflow and task control flow using various advanced cloud services (including integration, deployment, scheduling, resource distributing, etc.). The ontology-based testing approach is based on five main pillars: 1) the ODD input, 2) the ontology model set-up, 3) the automated test scenario generation and planning, 4) the test execution and evaluation framework and finally, 5) the test management system. Last, but not least, the test management system ensures a continuous monitoring of the testing process, ensures traceability of requirements, KPIs, and results, and enables test result visualization.