Exploring Credibility Scoring Metrics of Perception Systems for Autonomous Driving

The safe operation of autonomous and semi-autonomous vehicles (AVs) requires accurate predictions from the perception module built into the AV. Modern perception modules rely on object detection models to accurately gather information about the surrounding objects and the road. However, there have been several incidents in which erroneous predictions from these object detection models have led to fatal accidents (Wakabayashi, 2018).

Modern object detectors such as YOLO and Fast R-CNN rely heavily upon two main tasks: object localization, the process of locating or identifying the presence of an object in an image, and image classification, predicting the type of object and assigning it a label. As such, the YOLO object detector model outputs a label (specifying the object’s classification), a bounding box (specifying the object’s location), and a confidence score (specifying the model’s confidence in the classification and bounding box). Thus, an error in the object detector can arise in one of two possible ways: $(i)$ incorrectly classifying an object and $(ii)$ incorrectly sizing the bounding box. These propagate into the motion planning modules of an AV, and can result in unsafe behaviors form the AV.

An example of error $(i)$, misclassification of an object, can be seen in Figure 1, in which the pedestrian walking on the side of the road is classified as a cyclist with a very low confidence score of 0.1185. Though the confidence of the model is extremely low, it still chooses to classify this image, which it does so incorrectly. A simple misclassification like this can result in dangerous AV decision making.

On the other hand, an example of error $(ii)$, suboptimal characterization of a bounding box, can be seen in Figure 2, in which the unnecessarily large bounding box of the closest car, hinders the YOLO model’s classification of the adjacent motorcycle. Similar to the effects of misclassification in Figure 1, the suboptimal bounding box and its resulting impact on the classification of another object on the road can pose errors in AV decision making as some of the critical on-road objects are now being overlooked.

These are just two of the many examples that motivate the safer operation of AVs through the lens of error-free object detectors and perception algorithms. Improving these perception systems requires identifying (or creating) intrinsic properties or metrics of an image that suggests whether such a misclassification is probable. Furthermore, these metrics must work online when an AV is driving on the road, meaning they cannot be reliant at run-time on pre-labelled data but instead must be computed from intrinsic properties of the image. This can be used to alert the driver to a potential failure of the autonomous system and take control if needed.

We choose to to further study the credibility of these object detectors rather than the robustness. In the context of reliable autonomous driving, we choose to dive deeper into credibility rather than robustness because credibility is used to determine the reliability of the AV’s judgement, which, if low, can be used to transfer driving control from the AV to the human. Clearly defined, credibility and robustness differ in that credibility measures the effectiveness of object detectors on non-adversarial perturbations, whereas robustness primarily deals with adversarial perturbations.

In this paper, we discuss various metrics that act as indicators of when perception algorithms might fail to correctly classify on-road objects. While these metrics are not directly capable of being deployed online, they still provide a valuable foundation from which online metrics can be designed that incorporate prior knowledge about a classifier’s offline robustness. We hope to use these empirical results to better understand when the self-driving component can be disengaged and control can be shifted back to the driver to make proper decisions.

In the rest of this paper, we discuss some related/prior works that outline metrics for analyzing robustness, we then discuss our core approach and experimental setup, and conclude with qualitative and quantitative core results/takeaways.

Hendrycks et al., [2] focused on benchmarking image classifier robustness. It explored the use of both adversarial perturbations along with visual corruptions on the ImageNet dataset in order to score classifiers on their ability to correctly identify classes within these augmented images. The paper focused on common visual corruptions and also showed how classifiers can be trained to become more robust to such artifacts. In doing so the authors established several unique metrics (mean Corruption Error and Flip Probability), some of which we used as inspiration for our own custom metrics. We further extended their work on classifier robustness to scoring object detector robustness.

Further, Xue et al., 2021 [3] proposes adversarial attacks on object detectors based on natural perturbations. The authors introduce the concept of Real-World Perturbation Score (RPS) which quantifies the similarity of adversarial attacks to real noises in the physical world. While their work focuses on adversarial attack methods to understand robustness of various object detectors (YOLO, Fast R-CNN, and SSD), we extend their work on real-world adversarial perturbations by focusing on understanding the intrinsic properties of an image, measured with non-adversarial metrics, that may suggest misclassification. We maintain their ideas of RPS by using real-world perturbations, including Gaussian Noise/Blur and visual corruptions (fog, sunflare, and snow).

On the other hand, Zhao et al. [4] distinguish the types of adversarial attacks into Hiding Attacks, which make the object detector fail to recognize the object, and Appearing Attacks, which make the object detector mis-recognize the object. Our previous discussion of the two possibilities of error in object detection, either in object localization or image classification, reflects a parallel structure to the study mentioned in Zhao et al. Consequently, the paper suggests Enhanced Realistic Constraints Generation (ERGs), which aim to use context from images such as the background to build an adversarial attack. Though this concept is suggested, the study mainly injects a variety of random patterns and shapes into images, which are unlikely to occur in the physical world. We expand on their study by constructing our perturbations to be non-adversarial, all the while using the paper’s suggestions on ERGs to make our corruptions to the images more natural and realistic.

Other work by Durall et al. [5] looks at images converted to the frequency domain, specifically GAN generated deepfakes, in order to determine their credibility. The idea here is to achieve a metric for credibility that does not depend on labels associated with the data but rather on image intrinsics itself. Our study extends this thesis of image intrinsics being used as a credibility measure to object detectors, in the hopes that the corruption metrics we analyze can contribute to the development on label-independent online methods for scoring these models.

Our first step in studying the credibility of these object detector models was to create a main method to adjust the original image and determine any intrinsic properties that change within the images that may imply misclassification.

We chose to abstain from using adversarial perturbations which while known to confuse deep neural networks (Goodfellow et al., 2014)[6], are not good indicators of whether or not a network is learning robust latent space representations of the input classes.

In our proposed approach, we used several types of non-adversarial corruptions. Image classifiers are known to achieve subhuman accuracy when images are perturbed with Gaussian artifacts (Dodge et al., 2017) [7]. To test this, one set of experiments were hence with normally distributed artifacts in the form of Gaussian noise and Gaussian blur, both of which add some probability-based input to marginally corrupt the image.

The second tactic we used was corruptions that would reflect realistic road conditions that an AV might encounter, such as extreme glare from surrounding vehicles or bad weather like fog and rain.

One advantage of this approach is that it could be used in an online method to make any relevant perception-based decisions given real time constraints. Furthermore, this approach uses perturbations that can be observed in the real world, thus maintaining a high real-world perturbation score and ecological validity (Xue et al., 2021).

One important factor that’s important to highlight in our methodology is that we purposely chose not to train our detector on corrupted images. The intuition behind this decision is as follows. Consider the case of an arbitrary detector whose training data we have no prior knowledge about. Such a detector, regardless of its level of robustness, would be expected to perform poorly on corrupt samples relative to more natural data. The issue with using a detector trained on a corrupt dataset is that it becomes difficult to benchmark performance across a variety of detectors. Our choice of metrics have been designed to be benchmarked using a clean detector so that it can serve as a baseline for more robust detectors that share the same backbone but are trained on corrupt data samples.

For our project setup, we used a YOLOv3 object detector (Redmon et al., 2018) [8] for all the trials carried out in this paper. We used two versions of the detector: one trained on the Berkeley Deep Drive Dataset (Yu et al., 2020) [9] and the other trained on the KITTI Vision Benchmark Suite (Geiger et al., 2012)[10]. We used a combination of noising and blurring perturbations to effectively model real-world scenarios, such as fog, light glare, and snow.

The metrics discussed above extend earlier work on benchmarking the robustness of image classifiers to object detectors. This is a crucial extension since AV safety is a dual task of classifying objects correctly along with localizing them with a high degree of precision. This task is made harder when we consider how easy it is for the input data stream to an AVs perception module to be corrupted in a way that shifts its distribution too far from the training data to a point where the module’s predictions are too volatile to be considered safe.

Benchmarking on YOLO allowed us to see how a state-of-the-art one-stage object detector might perform in an online setting. On clean data our detector achieved a Top-1 Accuracy of 86% indicating strong offline reliability. To simulate online performance on unseen data, we chose to create corruptions in the dataset that mimic real-world noise. This was done through both Gaussian perturbations along with simulated weather conditions. We observed that certain classes that have a lower prior distribution in the training data tend to suffer the most when augmented with any such visual corruption. We also noticed that glare remains a large problem specifically on cars that causes a large drop in mAP. While these metrics are not designed to be run online, they do give us insight into how to better design online metrics that can leverage this base. For instance, knowing which classes have lower prior probabilities of occurring in our training data can allow us to enforce a lower online entropy constraint on the softmax predictions for images containing said classes. Furthermore we could also account for real-time weather conditions such as during times of fog or snow. Poor weather conditions could automatically enforce harsher online constraints on the minimum confidence threshold we require on a prediction to allow the AV to retain control of the driving task.

Developing this base into completely online metrics remains a task for future exploration, but one that is motivated by the knowledge that developing a fuller understanding of these metrics is crucial in making the future of AVs a safe and feasible reality.

We would like to thank Professor Yash Vardhan Pant for meaningful discussions and helpful suggestions during the course of this work. We would also like to thank the COMSNETS Intelligent Transportation Systems Workshop 2022 anonymous reviewers for their helpful comments.