DA-RAW: Domain Adaptive Object Detection for Real-World
Adverse Weather Conditions

For the reliable perception of environments, numerous methods attempt to train robust object detectors under adverse weather [18, 22, 23, 24]. The most intuitive approach is to utilize annotated datasets of adverse weather conditions [9, 25, 26]. Due to the difficulty of acquiring labeled data for real-world adverse weather, synthetic weather effects are generated on labeled clear images using prior knowledge of image formulation under adverse weather condition [1, 3, 2]. The object detector is then trained in a supervised manner using this synthetic dataset. Other methods train removal networks to restore clear images from adverse weather images using paired data of clear and synthetic weather images with the same background [4, 27, 28], or using unpaired data [29, 30, 7]. To acquire more realistic synthetic data, some methods jointly train a synthetic data generation model and its removal network [5, 6].

In real-world environments, however, the efficacy of methods that utilize synthetic data decreases due to the complexity and diversity of real-world weather corruptions [9]. In addition, removal networks are computationally intensive to be attached to the front of the detection network, and they are trained independently to downstream tasks, which provides insufficient performance improvement for these tasks on real-world images [8, 9]. While some methods have attempted to jointly train the removal network and downstream tasks [31, 32, 33, 34], they still rely on synthetic data or impose a computational burden.

Unsupervised domain adaptation can be used to directly adapt a detector trained on the source domain of clear weather to the target domain of adverse weather [14, 35, 15, 36, 16, 19]. Most UDA methods jointly train a domain classifier and a detector so that the classifier distinguishes between the source and target features, while the detector is optimized to confuse the classifier and align the feature distributions globally [37, 10]. Alignment can be performed on image-level features from various backbones, such as ResNet [11] or Feature Pyramid Network (FPN) [20]. In addition, aligning instance-level features extracted from Region-of-Interest (RoI) can improve domain alignment [13, 38, 39]. Diverging from conventional UDA approaches, some methods employ mathematical formulations of adverse weather conditions to enhance the alignments of features [12, 17]. However, these methods consider the weather corruption as a part of an image’s style and do not distinguish between the style and the weather gap [40, 41], despite their distinct characteristics. This oversight results in suboptimal adaptation performance in real adverse weather conditions with both significant style and weather gaps [21].

Given labeled images of clear weather conditions from the source domain $\mathcal{S}$ and unlabeled images of adverse weather conditions from the target domain $\mathcal{T}$, each minibatch consists of the same number of source and target data. Note that the source and target data are distinct in terms of both weather conditions and the surrounding environment.

We utilize the Faster R-CNN [42] pipeline with an FPN [43] backbone during training. The FPN backbone employs pyramid architecture to generate multi-scale feature maps $(P_{2},P_{3},P_{4},P_{5})$ from an image, allowing the efficient detection of objects of varying scales. The Region Proposal Network (RPN) proposes RoI on these features and extracts instance features from each RoI, following the Region Classification Network (RCN) which makes final class and bounding box predictions. The supervised loss $\mathcal{L}_{\text{sup}}$ obtained from RPN and RCN is applied only to the source data [10].

The overall architecture of our method is depicted in Fig 2. We aim to train a robust object detector that performs well in real-world environments with adverse weather. Based on the architecture of the FPN-based Faster R-CNN framework, we propose two components for domain adaptation to handle both style and weather gaps. First, an image-level style alignment is used to reduce the style gap through adversarial training. An instance-level weather alignment is then utilized to reduce the weather gap and learn the corruption-invariant features. The entire model is trained simultaneously in an end-to-end manner.

The distinct styles of the source and target domains’ environments result in different feature distributions. This style gap between domains is resolved through the alignment of image-level features. Similar to [20], a domain classifier is attached to each layer of the FPN backbone to distinguish between domains. The backbone is then trained to generate the domain-invariant feature by confusing the domain classifier through adversarial training. These objectives are accomplished in a single backpropagation step by a Gradient Reversal Layer (GRL) with a weight coefficient $\lambda$.

We intend to perform image-level feature adaptation by focusing solely on the style properties of images. However, some features are severely degraded due to weather corruption, making it difficult to concentrate on the style differences. To enable the network to emphasize style-related features during alignment, the Convolutional Block Attention Module (CBAM) [44] is employed to emphasize features essential for domain alignment. CBAM is attached to each feature map and applies channel and spatial attention modules to acquire refined features $\mathbf{x}^{p}$ at feature level $p$. The attended feature is then fed into the discriminator $\mathcal{D}_{p}$ that predicts the domain of a feature, leading it to align features through the GRL by concentrating on essential information.

Since low-level features with fine-grained details are more susceptible to weather corruption, only high-level features are used for alignment. Therefore, image-level style alignment is performed on the $P_{4}$ and $P_{5}$ layers of the FPN backbone. The loss for image-level alignment, $\mathcal{L}_{\text{img}}$, is given by the following equation:

	$\displaystyle\mathcal{L}_{\text{img}}=-\sum_{p}\sum_{\mathbf{x}_{i}^{p}}$	$\displaystyle\left[y_{i}\log\mathcal{D}_{p}\left(\mathbf{x}_{i}^{p}\right)\right.$		(1)
		$\displaystyle\left.+\left(1-y_{i}\right)\log\left(1-\mathcal{D}_{p}\left(\mathbf{x}_{i}^{p}\right)\right)\right],$

where $p\in\{P_{4},P_{5}\}$ represents feature level, $\mathbf{x}_{i}^{p}$ represents each feature of level $p$ at location $i$ after CBAM layers, and $y_{i}\in\{0,1\}$ indicates the domain label of each feature at location $i$.

The weather corruption has a local effect on the image and substantially degrades the instance features that are essential for object detection. We aim to obtain corruption-invariant features and reduce the weather gap through prototype-based contrastive learning. In particular, we assume that an instance feature of the target domain image is composed of an object and an arbitrary pattern of weather corruption. Then, the similarity between instance features of object proposals within the same category is encouraged, resulting in instance features invariant to weather corruption.

From the source and target domain training images, instance features are obtained, and each of them is pseudo-labeled as class $\hat{c_{i}}$ using the classwise score of each instance provided by the RCN. To be utilized for contrastive learning, instance features are forwarded to an MLP head to generate instance embeddings $\mathbf{Z}=\{\mathbf{z}_{i}\}_{i=1}^{N}$ with dimension $D$, where $N$ is the total number of instances. Note that MLPs exist independently for each level of features without sharing weights, and only instance features with scores over a threshold $\delta$ from the low-level image feature are utilized to align fine-grained features.

To maximize the similarity between instance embeddings with the same pseudo-labels, prototype-anchored metric learning is used to design the contrastive loss [45]. Using learnable prototypes as representatives of each class, each instance embedding is assigned to prototypes, and a network is learned to increase their similarity. $K$ learnable prototypes are used for each class $c$ as $\mathbf{P}_{c}\in\mathbb{R}^{D\times K}$ to account for the intra-class variation of instance features, and each prototype $\mathbf{p}_{c,k}\in\mathbb{R}^{D}$ serves as the $k^{th}$ cluster center of a class $c$. Also, to further boost the performance of contrastive learning, instance embeddings with the background class also adopt the same number of learnable prototypes, which can be used as negative samples for other instances.

Each instance with a pseudo-label $c$ is assigned to prototypes of the same class by computing the soft assignment matrix for class $c$, $\mathbf{L}_{c}\in\mathbb{R}_{+}^{K\times n_{c}}$. The soft assignment matrix satisfies the condition that the sum of soft assignment probabilities for each instance is one, i.e., $\mathbf{L}_{c}^{\top}\cdot\mathbf{1}^{K}=\mathbf{1}^{n_{c}}$, where $\mathbf{1}^{K}$ and $\mathbf{1}^{n_{c}}$ denotes the vector of all ones with dimensions $K$ and $n_{c}$, respectively. The assignment matrix can be obtained by maximizing the similarity between instance embeddings and the class prototypes, $\mathbf{Q}_{c}=\mathbf{P}_{c}^{\top}\mathbf{Z}_{c}\in\mathbb{R}^{K\times n_{c}}$, where $\mathbf{Z}_{c}\in\mathbb{R}^{D\times n_{c}}$ and $n_{c}$ represent instance embeddings and the number of instances pseudo-labeled as class $c$, respectively.

To avoid a trivial solution in which all the instance embeddings are assigned to a single prototype, an equipartition constraint is added to ensure that instances are equally distributed among prototypes within a class. By adding the entropy regularization term [46] with a parameter $\kappa$ that controls the smoothness of assignment, the objective for obtaining the assignment matrix for class $c$ is as follows:

$$\max_{{\mathbf{L}}_{c}}\text{Tr}\left({\mathbf{L}}_{c}^{\top}\mathbf{Q}_{c}\right)+\kappa~{}\mathcal{H}(\mathbf{L}_{c}),\quad\textit{s.t.}\quad\mathbf{L}_{c}\cdot\mathbf{1}^{n_{c}}=\frac{n_{c}}{K}\cdot\mathbf{1}^{K},$$

(2)

which turns into an optimal transport problem. The solution can be computed by a few iterations of the Sinkhorn-Knopp algorithm [46], which outputs the re-normalization vectors $\mathbf{u}\in\mathbb{R}^{K}$ and $\mathbf{v}\in\mathbb{R}^{n_{c}}$:

$$\mathbf{L}_{c}=\operatorname{diag}(\mathbf{u})\exp\left({\frac{\mathbf{Q}_{c}}{\kappa}}\right)\operatorname{diag}(\mathbf{v}).$$

(3)

After obtaining the soft assignment matrix, the network is trained so that similarities between prototypes and instance embeddings correspond to the soft assignment matrix. The prototypes and instance embeddings are simultaneously optimized by minimizing the following cross-entropy loss between the similarity and assignment matrix:

$$\mathcal{L}_{inst}=-\frac{1}{N\cdot K}\sum_{i=1}^{N}\sum_{j=1}^{K}\mathbf{L}^{i,j}_{\hat{c}_{i}}\cdot\log\frac{\exp{\left(\mathbf{z}_{i}\cdot\mathbf{p}_{\hat{c}_{i},j}/\tau\right)}}{\sum_{c}^{C}\sum_{k}^{K}\exp{\left(\mathbf{z}_{i}\cdot\mathbf{p}_{c,k}/\tau\right)}},$$

(4)

where $\hat{c}_{i}$ is the pseudo-label for the $i^{th}$ instance, and $\mathbf{L}^{i,j}_{\hat{c}_{i}}$ is a soft assignment of the instance to the $j^{th}$ prototype of the pseudo-labeled class. Also, $\tau$ is a temperature parameter, and $C$ denotes the number of classes, including the background class. For each instance $\mathbf{z}_{i}$, minimizing $\mathcal{L}_{inst}$ increases its similarity with assigned prototypes $\mathbf{p}_{\hat{c}_{i},j}$, and decreases its similarity with all the others. Note that the loss is computed on both the source and target domain features using the same prototypes to reduce the domain gap.

As a result, instance embeddings are grouped around their assigned prototypes. This produces corruption-resistant instance features by promoting instance embeddings with similar semantics and variable weather corruption to become closer. The final objective of our method is as follows:

$$\mathcal{L}=\mathcal{L}_{sup}+\alpha\mathcal{L}_{img}+\beta\mathcal{L}_{inst}.$$

(5)

Our source domain dataset is Cityscapes [48], which consists of real-world urban driving images captured under clear weather conditions. For the target domain dataset, multiple datasets are used to validate the efficacy of our method in various environments with adverse weather conditions. Using two synthetic weather datasets, we investigate the efficacy of other methods employing synthetic data or UDA in synthetic weather contexts. On clear images of the Cityscapes, the Rain Rendering [1] generates synthetic rain images with a physical particle simulator for each, and RainCityscapes [47] generates rain and fog effects subject to scene depth. To validate the efficacy of methods in real-world environments, BDD 100K [26], a real-world driving dataset captured in various weather conditions, is employed. The rainy and snowy subsets are used as our target domain to evaluate the efficacy of methods under diverse weather conditions.

To further validate our model across a wider range of environments, we collect Our Dataset in adverse weather conditions using our platform, which is equipped with an external vehicle RGB camera [49, 50]. Several datasets with adverse weather scenarios primarily focus on urban scenes and lack diversity in background environments [51, 52]. To address significant domain gaps, our datasets include data collected from rural and mountainous environments. In comparison to the BDD 100K, which uses a camera mounted inside the windshield of the vehicle, our dataset utilizes an external camera that is consistent with the source domain dataset. In addition, raindrops and snowfalls on the lens result in much more severe blurring of the images. Our dataset is divided into Rainy and Snowy subsets. The rainy subset consists of 2845 training images and 677 validation images, and the snowy subset consists of 1656 training images and 598 validation images. Our dataset comprises three classes: person, car, and motorcycle. For alignment with Cityscapes dataset, Cityscapes classes are assigned to our dataset during the experiment as follows: 1) person, rider to person, 2) car, truck, bus to car, and 3) motorcycle, bicycle to motorcycle.

During training, stochastic gradient descent (SGD) is used as an optimizer with a weight decay of $5e^{-4}$ and momentum of $0.9$. Each batch consists of 16 images, eight from the source domain and eight from the target domain. We resized all the input images so that the shorter side has a length of 800 pixels and applied random horizontal flipping with a probability of 0.5. The entire model is trained with an initial learning rate of $2.5e^{-3}$ for $9.5k$ iterations and then reduced to $2.5e^{-4}$ for another $2k$ iterations.

In addition, SWDA outperforms SADA, despite the fact that SADA incorporates instance-level alignment while SWDA focuses solely on image-level alignment. This suggests that directly aligning the instance-level features that are severely contaminated in real adverse weather conditions reduces performance, necessitating the use of alternative alignment methods. Using image-level style alignment and instance-level weather alignment, our method optimizes feature alignment by resolving both style and weather gaps, thereby improving detection performance.