Robust ADAS: Enhancing Robustness of Machine Learning-based Advanced Driver Assistance Systems for Adverse Weather

In Figure 2, we demonstrate that a highly accurate object detector trained on clear images fails to retain its accuracy when employed in adverse weather conditions. In such critical conditions, the driver relies heavily on the assistance of the ML-ADAS. However, with a 4% mAP, there is a high risk of collisions with oncoming traffic since the system is essentially blind; it cannot detect anything. Thus, it is evident that there is a need to robustify ML-ADAS DNNs against adverse weather to guarantee driver safety after deployment in the real world. The object detector, YOLOv8n, specifications and the extended KITTI dataset for adverse weather conditions is described in Section III.

To robustify an ML-ADAS system against multiple adverse weather conditions without retraining all DNNs employed, we propose a preprocessing DNN to construct clear images in adverse weather. Our proposed WUNet is based on the popular UNet architecture and trained on the KITTI dataset extended with augmentations to create synthetic adverse weather images. In extreme fog, the WUNet preprocessing boosts object detection performance from 4% to 70% mAP over the 0.5 IoU threshold. Details on the experimental setup can be found in Evaluation and Discussion. In summary, we make the following key contributions.

1. 1.

   WUNet: we develop a preprocessing DNN to construct clear images in adverse weather, before feeding the input image to the ML-ADAS modules.

2. 2.

   We synthesize an adverse weather dataset by extending the KITTI dataset to include foggy, rainy, and snowy images.

3. 3.

   We devise an in-depth evaluation scheme to measure a denoising algorithm’s impact on object detection performance in varying weather adversities.

4. 4.

   We demonstrate potential to decrease the complexity of a UNet to decrease latency for the specific task of weather-related artifact/noise removal by processing an input image as a batch of crops instead of the whole image.

To address the limitation of publicly available datasets, we decide to create a synthetic dataset containing adverse weather images. With the advent of computer vision, there are multiple open-source image augmentation libraries to choose from. However, we choose the imgaug library based on its effectiveness in producing photo-realistic weather conditions  [10]. With this extensive dataset, we can train a DNN to learn to map adverse weather images to their clear counterparts. In this way, we obtain clear images that not only facilitate the driver visually, but also improve performance for all ML-ADAS features.

While similar efforts have been made to create preprocessing denoising DNNs, often the use-case is restricted to just one weather condition like fog removal  [11]. In contrast, we aim to address all weather cases with one DNN. The popular UNet encoder-decoder architecture performs well in similar tasks such as image super-resolution and depth estimation, where the emphasis is on image reconstruction  [12, 9]. Since its functionality aligns with our strategy, we adopt it in our pipeline.

Training a WUNet on RGB and HSV images entails distinct approaches due to differences in color representation. RGB represents colors using three channels: red, green, and blue, which are additive. In contrast, HSV (Hue, Saturation, Value) represents colors based on perceptually relevant attributes like hue, saturation, and brightness, offering a more intuitive representation. Our exploration will involve training separate WUNet models on RGB and HSV images to discern how each representation affects performance. We plan to compare the models’ ability to generate a clear image from an adverse weather image using the Mean-Squared Error (MSE) metric and comparing the prediction with the original image. Overall, we aim to gain insights into the advantages and limitations of using RGB versus HSV for image enhancement.

As mentioned earlier, the main drawback of this approach, as opposed to retraining the detector, is the additional cost of the WUNet. Figure 4 highlights our strategy to mitigate the additional latency introduced by the WUNet. Before implementing any DNN compression techniques like pruning and quantization  [13], we will study if the WUNet can be reduced in size with a reduction in the scope of the problem. We hypothesize that there will be no difference in WUNet performance when it processes the input image as a whole, compared with dividing it into equal-sized crops and processing this batch of crops instead. We aim to highlight that a DNN does not need to learn an extremely complex function to remove weather artifacts. Thus, with less information as input, the DNN does not need to be as large and can be reduced in size. Our study aims to highlight this potential to shrink the UNet architecture specifically in the task of noise removal. We will evaluate the difference between WUNet versions trained on either whole images or crops.

We rely on the KITTI dataset because it consists of only clear weather images in the day time  [14]. This is an important characteristic because it enables us to clearly evaluate the object detector’s performance when trained on clear images and tested on adverse weather. Consequently, the impact of the WUNet on object detection becomes clear as well.

Using the train/test split provided by the authors of Dist-YOLO  [15], we obtain 6699 images in our train, and 782 images in our test set. Next, we extend this base dataset with synthetic adverse weather images using the imgaug library [10]. We use the Fog, Rain, and Snowflakes augmentations on each image in the dataset, with the instantiation parameters for each augmentation being sampled randomly between low and extreme adversity. Along with the original, these three synthetic versions are saved with the same ground truth annotations as the original image. Thus, our dataset now consists of 26796 images in the train, and 3128 images in the test set encompassing varying intensities of each adverse weather condition. While training the WUNet, a sample is defined as the augmented image (fog, rain, snow, or original) and its ground truth is the original image. For training the object detector, the sample is defined as the image and its ground truth are the corresponding bounding box annotations in the KITTI dataset.

Furthermore, we create a total of 10 validation sets, consisting of 782 images each, by tuning the hyperparameters in the augmentations. These sets are the normal set which is the unaltered KITTI test set, and low, medium, and high intensity variations of each of the three weather augmentations. Figure 5 displays samples from our high adversity validation sets. With this, we can conduct an in-depth evaluation of the WUNet’s weather artifact removal performance in different intensities of different weather conditions.

All experiments were conducted on the NVIDIA GeForce RTX 4090 GPU. The WUNet was trained on the extended and augmented KITTI dataset using the Adam optimizer for 200 epochs with a batch size of 24, learning rate of 0.01, and input image size of (640,200) pixels. Note that the WUNets trained on crops have input image size of (160,100) and a total of 214,368 train and 25,024 test images. This is because we arbitrarily divide each image in the augmented dataset into 8 crops, with each crop being a sample instead of the whole image. In this case, we use batch size of 160. In all experiments, we use the mean-squared loss to supervise training. The WUNet that performs the best on the test set is chosen to be evaluated end-to-end with the object detector.

For object detection, we employ the YOLOv8 variant of size nano (n), small (s), and medium (m) from the open-sourced Ultralytics framework [5]. These detectors have been trained over the base KITTI dataset with the same training settings as the base WUNet except for a 0.001 learning rate. We present the performance evaluation results of each of these variants in Table 1.

Since YOLOv8n has the lowest mAP score, it has the greatest sensitivity to weather artifacts. Thus, we use this variant in our evaluation of the WUNet end-to-end to examine the artifact removal performance in the worst case of highest detector sensitivity.

Figure 7 summarizes the MSE evaluation results we obtain with training each WUNet variant. With MSE validation, we aim to measure the WUNet’s ability to generate high quality clear images. The lower the MSE, the greater the WUNet’s ability in removing weather artifacts. We observe that the RGB learned WUNets outperform the HSV variants by at least 40%. Furthermore, within the RGB WUNets, training over crops yields 15% lower MSE than training over the whole image. This is most likely because of a larger dataset in the case of crops, but we establish that processing via crops is a valid scheme. Figure 8 provides a closer inspection where we observe MSE error in our 10 varying adversity sets. Most notably, we observe that the most challenging and critical case to address is high fog. Next, we aim to employ the YOLOv8n’s mAP performance as a metric to further determine the quality of the clear images produced.

We compare the performance of YOLOv8n, with and without each WUNet variant’s preprocessing, over all the 10 cases. Note that YOLOv8n was trained only on the base KITTI dataset that consists of clear images. The validation set that corresponds to this is the normal set and we expect the highest performance here (Table 1). Figure 9 summarizes our findings. Averaged over all cases, the RGB variants achieve 76% mAP compared to the HSVs’ 67%. Again, we observe that RGB-learning outperforms HSV-learning, with at least a 9% increase in mAP in all categories. Most importantly, we have verified that the WUNet can process images as crops and retain the same performance as processing the entire image. This allows us to continue our work to shrink the size of the WUNet to increase network efficiency.

Lastly, Figure 10 highlights the improvements in YOLOv8n’s accuracy by using the simple RGB WUNet in the pipeline. Our results verify that the YOLOv8n trained on clear images is needs to be robustified against cases like extreme fog. Inference on the clear image provided by the WUNet compared to the extremely foggy image improves mAP score from 4% to 70%, a 1650% improvement. Overall, the most significant performance boosts are seen in all fog and rain cases especially.

Building upon this project, we will evaluate the WUNet on a larger dataset like the Pascal VOC dataset [16]. This will allow us to compare our results to the current state-of-the-art works like the Image-Adaptive YOLO that use larger datasets, as opposed to the KITTI dataset [11]. Additionally, research needs to be conducted to assess the DNN’s performance in terms of its efficiency and additional latency impact on overall system. We have already established that the WUNet is able to process images as a batch of crops which allows us to shrink the network’s size by a factor that is directly proportional to the size of the crop relative to the original image size. This is because removing weather related artifacts from the image does not require learning a very complex function. Thus, we will tune the parameters of network size and image-crop size specifically.

This work was partially supported by the NYUAD Center for Artificial Intelligence and Robotics (CAIR), funded by Tamkeen under the NYUAD Research Institute Award CG010, and the NYUAD Center for Interacting Urban Networks (CITIES), funded by Tamkeen under the NYUAD Research Institute Award CG001.