The remarkable progress in autonomous driving technology has revolutionized the automotive industry, bringing us closer to a future of fully autonomous vehicles. However, before these technologies can reach the road, they must undergo extensive testing and validation.
One cost-effective, safe, and efficient way to test these machine learning-based modules is through the use of simulators. Simulators can specifically be used to quickly evaluate a machine learning model before trying it in the real world. Verification involves testing the system against ODD requirements and criteria, such as the system’s ability to correctly and quickly detect and react to other vehicles, pedestrians, and obstacles in the environment, while validation involves acceptance and usability tests against real-world scenarios to ensure that it meets the needs of customers (drivers) and end-users (drivers and pedestrians).
Taking advantage of simulators, the system can be tested against highly realistic and controlled environments that replicate the real-world areas and scenarios that autonomous vehicles may encounter. On the other hand, simulators make it possible to create less likely but more dangerous situations that are costly or difficult to create in the real world. Additionally, all these scenarios can be evaluated without exposing any risk to human life or damaging the environment. All of this can be done while simultaneously recording massive amounts of information that can be used to detect the possible issues and replaying the exact same scenario.
Our team at INFOTIV [7] has done extensive research on use of simulator for validation and testing and in this blog post, in this blog post we briefly go through some of them.
SVL simulator
The Infotiv research group was recently part of a Swedish team for the “2021 IEEE Autonomous Driving AI Test Challenge”. We are proud to announce that out of 119 teams entering the fierce competition our team placed itself among the top 13 that were qualified for the final phase of the challenge. The main challenge was testing the Apollo open autonomous driving platform [3] with the help of SVL simulation [2] platform. In the competition, Infotiv collaborated with researchers from Chalmers University of Technology, Research Institutes of Sweden, and Mälardalen University, as a part of VALU3S research project (“Verification and Validation of Automated Systems’ Safety and Security”). During the challenge, the team designed a tool to aid the automatic generation of various traffic scenarios to verify and validate the safety functionality of autonomous systems. The results of this experiment were presented at IEEE AI Test Conference [10] and will also be published in a paper titled “Efficient and Effective Generation of Test Cases for Pedestrian Detection – Search-based Software Testing of Baidu Apollo in SVL” [1].
Infotiv qualified for final phase in the “2021 IEEE Autonomous Driving AI Test Challenge”
CARLA simulator
CARLA is an unreal engine-based simulator that supports diverse sensor suites including LIDARs, multiple cameras, depth sensors and GPS and gives access to privileged information such as ground truth semantic segmentation and depth information [8]. It supports OpenDrive and OpenScenario. As a part of the project, we look into 3D modelling of the Åkareplatsen area in Gothenburg, Sweden and the road information within the simulator and access to different sensors. Additionally, we used ESMINI [4] to explore and get familiar with the emerging OpenSCENARIO and OpenDRIVE data formats.
DonkeyCar simulator
In a thesis titled “Exploring Machine Learning for Autonomous Drive with the Help of
Simulators Using Deep Reinforcement Learning to Investigate the Extent of Learning Objectives for Autonomous Vehicles” [5], the team explored how simulators and Reinforcement Learning (RL) can be utilized to expand the training process further, as well as investigate their possibilities and limitations. The focus of the work was mainly on implementing an end-to-end autonomous driving system using the DonkeyCar simulator [6] and the double deep Q-network algorithm. The training is executed with different configurations for the simulator and algorithm and the trained models are then evaluated in two regards, training progression and driving capability.
BERGE Simulator
Using the experience gained from using the opensource simulators we assisted our other Swedish partners, BERGE, to define the requirements and help them with verification and validation of their simulator. Their simulator functionalities are including but not limited to:
- scenario creation using a graphical user interface (agent movements)
- scenario definition in a JSON format
- access to sensors such as camera, distorted camera, depth sensor, segmentation sensor
- support for different maps (generic highway, country road and Åkareplatsen area in Gothenburg, as an urban area)
- modification of environment parameters (weather, lighting, time of the day, and colors of objects)
- modification of individual license plate number
Links:
- [1] Ebadi, H., Moghadam, M. H., Borg, M., Gay, G., Fontes, A., & Socha, K. (2021, August). Efficient and effective generation of test cases for pedestrian detection-search-based software testing of Baidu Apollo in SVL. In 2021 IEEE International Conference on Artificial Intelligence Testing (AITest) (pp. 103-110). IEEE.
- [2] https://www.svlsimulator.com/
- [3] https://apollo.auto/
- [4] https://github.com/esmini/esmini
- [5] https://odr.chalmers.se/handle/20.500.12380/305932
- [6] https://github.com/tawnkramer/gym-donkeycar
- [7] https://www.infotiv.se
- [8] https://carla.org/
- [9] https://ieeeaitest.com/
With over two decades of working in the software testing industry Martin has experience from almost all aspects of testing. Through the years Martin has working several different industries including telecom, defense and automotive. The last couple of years he has divided his time between helping large organisation with test challenges and teaching test methodologies at a technical college. The interest for Machine Learning and testing of such functionality has evolved through a series of government and EU funded research projects.
Hamid Ebadi, senior researcher at INFOTIV
Hamid Ebadi did his Ph.D. at the Computer Science and Engineering department at Chalmers University in researching and designing specific programming languages for data science and machine learning with privacy and security in mind. Currently he is exploring the challenges in introducing machine learning-based systems to automated driving within SMILE (Safety analysis and verification/validation of MachIne LEarning based systems) and VALU3S (Verification and Validation of Automated Systems’ Safety and Security) projects.
