Comprehensively Evaluating the Perception Systems of Autonomous Vehicles Against Hazards

Perception systems are vital for the safety of autonomous driving. In complex autonomous driving scenarios, autonomous vehicles must overcome various natural hazards, such as heavy rain or raindrops on the camera lens. Therefore, it is essential to conduct comprehensive testing of the perception systems in autonomous vehicles against these hazards, as demanded by regulatory agency of many countries for human drivers.Since there are many hazard scenarios, each of which has multiple configurable parameters, the challenges are (1) how do we systematically and adequately test an autonomous vehicle against these hazard scenarios, with measurable outcome; and (2) how do we efficiently explore the huge search space to identify scenarios that would induce failure? In this work, we propose a framework {Hazards Generation and Testing (HazGT)} to generate a customizable and comprehensive repository of hazard scenarios for evaluating the perception system of autonomous vehicles. HazGT not only allows us to measure how comprehensively an autonomous vehicle (AV) has been tested against different hazards but also supports the identification of important adversarial hazards through optimization.HazGT supports a total of 70 kinds of hazards relevant to the visual perception of AVs, which are based on industrial regulation.HazGT automatically optimizes the parameter values to efficiently achieve different testing objectives based on Genetic Algorithm.We have implemented HazGT based on two {popular} 3D engines, i.e., Unity and Unreal Engine.For the two mainstream perception models (i.e., YOLO and Faster RCNN), we have evaluated their performance against each hazard through extensive experiments and the results show that both systems have much room to improve. In addition, our experiments also found that ChatGPT4 performs slightly worse than YOLO. Our optimization-based testing engine is effective in finding perceptual errors on the perception models.