Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization

For the domain generalization problem, it is necessary to ensure the features of samples with the same label close to each other, while the features of different class samples being far apart. Otherwise, the predictions on the unseen target domain may suffer from ambiguous decision boundaries and performance degradation [20, 8]. This is well aligned with the philosophy of metric learning. Therefore, we design a momentum metric-learning scheme to encourage the network to learn such domain-independent yet class-specific features by considering the mutual relation among samples across domains. Specifically, we propose a novel $K$-hard negative selector for triple loss to improve the training effectiveness by selecting informative triplets in the memory bank, and a momentum updated encoder to guarantee the representation consistency among the embeddings stored in the memory bank.

To broaden the network understanding of the data, we propose to utilize the intrinsic supervision within a single image to impose a regularization into the feature embedding by adding auxiliary self-supervised tasks on all the source domain images. A similar idea has been adopted in domain adaptation and Generative Adversarial Networks training [38, 4]. The auxiliary self-supervised task is able to exploit the intrinsic semantic information within a single image to provide informative feature representations for the main task.

There are plenty of works focusing on designing auxiliary self-supervised tasks, such as rotation degree prediction and relative location prediction of two patches in one image [7, 13, 21]. Here, we employ the recently-proposed solving jigsaw puzzles [32, 2] as our auxiliary task. However, most of the self-supervised tasks focusing on high-level semantic feature learning can be incorporated into our framework. Specifically, we first divide an image into nine ($3\times 3$) patches, and shuffle these patches within the $30$ different combinations following [2]. As pointed by [2], the model achieves the highest performance when the class number is set as $30$ and the order prediction performance decreases when the task becomes more difficult with more orders. A new auxiliary task branch $h_{a}$ follows the extracted feature representation $f_{\theta}$ to predict the ordering of the patches. A Cross-Entropy loss is applied to tackle this order classification task:

where $y^{a}$ and $p^{a}$ are the ground-truth order and predicted order from the auxiliary task branch, respectively. We use $c_{a}=0$ to represent the original images without patch shuffle, leading to a total of 31 classes.

Overall, we formulate the whole framework as a multi-task learning paradigm. The total objective function to train the network is represented as

where $\alpha,\ \beta,\ \text{and}\ \gamma$ are hyper-parameters to balance the weights of the basic classification supervision, extrinsic relationship supervision, and intrinsic self-supervision, respectively.

We evaluate our method on two public domain generalization benchmark datasets: VLCS and PACS. VLCS [10] is a classic domain generalization benchmark for image classification, which includes five object categories from four domains (PASCAL VOC 2007, LabelMe, Caltech, and Sun datasets). PACS [25] is a recent domain generalization benchmark for object recognition with larger domain discrepancy. It consists of seven object categories from four domains (Photo, Art Paintings, Cartoon, and Sketches datasets) and the domain discrepancy among different datasets is more severe than VLCS, making it more challenging.

Our framework is flexible and one can use different network backbones as the feature Encoder. We utilized a fully-connected layer with $31$-dimensional output as the self-supervised auxiliary classification layer following the setting in [2] for a fair comparison. To enable the momentum metric learning, we further employed a fully-connected layer with $128$ output channels following the Encoder part and added an L2 normalization layer to normalize the feature representation $v$ of each sample. The MuEncoder has the same network architecture as the Encoder, and the weight of MuEncoder was initialized with the same weight as Encoder. We followed the previous works in the literature [1, 2, 26, 8] and employed the leave-one-domain-out cross-validation strategy to produce the experiment results, i.e., we take turns to choose each domain for testing, and train a network model with the remaining three domains.

We implemented our framework with the PyTorch library on one NVIDIA TITAN Xp GPU. Our framework was optimized with the SGD optimizer. We totally trained 100 epochs, and the batch size was 128. The learning rate was set as 0.001 and decreased to 0.0001 after 80 epochs. We empirically set the margin of the triplet loss as $2$. We also adopted the same on-the-fly data augmentation as JiGen [2], which includes random cropping, horizontal flipping, and jitter.

We followed the same experiment setting in previous work [2] to train and evaluate our method. The extrinsic metric learning and intrinsic self-supervised learning was developed upon the “FC7” features of AlexNet [22] pretrained on ImageNet [6]. We set the size of the memory bank as $1024$ and the number of negatives $K$ in the triplet loss Eq. (4) as $256$. The hyper-parameters $\alpha$, $\beta$, and $\gamma$ in total objective function Eq. (7) were set as $1$, $0.1$, and $0.05$, respectively. For our results, we report the average performance and standard deviation over three independent runs.

We compare our method with other nine previous state-of-the-art methods. D-MTAE [12] utilized the multi-task auto-encoders to learn robust features across domains. CIDDG [29] was a conditional invariant adversarial network that learns the domain-invariant representations under distribution constraints. CCSA [31] exploited a Siamese network to learn a discriminative embedding subspace with distribution distances and similarities. DBADG [25] developed a low-rank parametrized CNN model for domain generalization. MMD-AAE [28] aligned the distribution through an adversarial auto-encoder by Maximum Mean Discrepancy. MLDG [26] was a meta-learning method by simulating train/test domain shift during training. Epi-FCR [27] was an episodic training method. JiGen [2] solved a jigsaw puzzle auxiliary task based on self-supervision. MASF [8] employed a meta-learning based strategy with two complementary losses for encoder regularization. Moreover, we include the Within domain performance of all the datasets as a comparison to reveal the performance drop due to domain discrepancy. We trained Within domain using a supervised way with training and test images from the same domain.

The comparison results with the above methods are shown in Table 1. It is observed that our EISNet achieves the best performance on both Caltech and Sun datasets and comparable results on PASCAL VOC and LabelMe datasets. Overall, EISNet achieves an average accuracy of $74.67\%$ over four domains, outperforming the previous state-of-the-art method MASF [8]. Our method also outperforms JiGen [2] on three domains and achieves comparable results on the remaining PASCAL VOC domain, demonstrating that utilizing extrinsic relationship supervision can further improve the network generalization ability.

To show the effectiveness of our framework under different network backbones on PACS dataset, we evaluate our method with three different backbones: AlexNet, ResNet-18, and ResNet-50 [16]. The size of memory bank was set as $1024$ and $K$ in the triplet loss Eq. (4) was set as $256$. The hyper-parameters in total objective function Eq. (7) were set as $1$, $0.5$, and $0.7$ for $\alpha$, $\beta$, and $\gamma$, respectively. For our results, we also report the average performance and standard deviation over three independent runs.

Table 2 summarizes the experimental results developed with AlexNet backbone. We compare our methods with eight other methods that achieved previous best results on this benchmark dataset. MetaReg [1] utilized a novel classifier regularization in the meta-learning framework. As we can observe from Table 2, by simultaneously utilizing momentum metric learning and intrinsic self-supervision for images across different source domains, our method achieves the best performance on three datasets. Across all domains, our method achieves an average accuracy of $75.86\%$, setting a new state-of-the-art performance.

We also compare our method with baseline method (DeepAll) and the state-of-the-art method MASF [8] using ResNet-18 and ResNet-50 backbones in Table 3. In the ResNet-50 experiment, we reduce the batch size to 64 to fit the limited GPU memory. The DeepAll method is trained with all the source domains without any specific network design. As shown in Table 3, our method consistently outperforms MASF about 1.11% and 3.17% on average accuracy with ResNet-18 and ResNet-50 backbone, respectively. This indicates that our designed framework is very general and can be migrated to different network backbones. Note that the improvement over MASF is more obvious with a deeper network backbone, showing that our proposed algorithm is more beneficial for domain generalization with deeper feature extractors.

We conduct extensive analysis of our method. Firstly, we investigate the effectiveness of extrinsic and intrinsic supervision using ResNet-50 backbone on PACS dataset, and the experimental results are illustrated in Table 4. The Extrinsic supervision indicates that the momentum metric learning is used, while Intrinsic supervision denotes that the auxiliary self-supervision loss is optimized. The method without these two supervisions is the baseline model, which is the same with DeepAll results in Table 3. From the results in Table 4, we observe that each supervision plays an important role in our framework. Specifically, equipping the extrinsic supervision into the baseline model yields about 2.99% average accuracy improvement. Meanwhile, we also achieve 2.73% average accuracy improvement over the baseline model by incorporating intrinsic self-supervision of the images. By combing extrinsic and intrinsic supervision, performance is further improved across all settings, indicating these two supervisions are complementary.

We then analyze five key components in our framework, that is a) the number of different negative samples $K$ in momentum metric learning, b) the effectiveness of momentum update coefficient $\delta$, c) the effectiveness of hard negative selector, d) the size of memory bank $m$, and e) time cost. All below comparison experiments are implemented with AlexNet backbone on the PACS benchmark.

1. a.

   The number of negative samples $K$ is a key parameter of our designed $K$-hard negative selector in momentum metric learning. We investigate the network performance under different options. We select six $K$ values at different magnitudes, which are 1, 8, 64, 128, 256, and 512. The Sketch dataset results are shown in Fig. 3 (a), We can observe that a large number of negative samples would lead to better results in general and the network generates the best result with $K=256$. However, the performance drops drastically if we set $K=512$, demonstrating that too large $K$ will produce a burden on the metric distance calculation and make the network difficult to learn.

2. b.

   The momentum update coefficient $\delta$ is important to control the feature consistency among different batches of embedded features in the memory bank. We show the accuracy with different momentum coefficient $\delta$ in Fig. 3 (b). It is observed that the network performs well when $\delta$ is relatively large, i.e., 0.999. A small coefficient would degrade the network performance, suggesting that a slow updating MuEncoder is beneficial to the feature consistency.

3. c.

   To validate the effectiveness of $K$-hard negative selector in our proposed metric learning, we compare our proposed $K$-hard negative selector with original random triplet selector and semi-hard negative selector. The Sketch dataset results are shown in Table 5. Equipped with semi-hard negative selector, the accuracy improves $2.70\%$. By selecting more negative pairs from the memory bank, we obtain the accuracy of $70.25\%$, demonstrating the effectiveness of the proposed $K$-hard negative selector.

4. d.

   The size of memory bank $m$ can be adjusted according to different tasks. Here, we show the results of four different settings with the number of negatives changing as well in Table 6. In general, our method is able to generate better results with a large memory bank size and negative samples. However, a too large memory bank will increase the burden to calculate the pair-wise distance in triplet loss. Therefore, we need to balance the accuracy and computation burden.

5. e.

   Apart from the performance improvement over other methods, our method has much lower computation cost. Under the same server setting (one TITAN XP GPU) and AlexNet backbone, our method only takes 1.5 hours to train the network on PACS dataset, while the total training time of the state-of-the-art MASF is about 17 hours. Therefore, our method could save more than $91\%$ time cost on training phase.

We also employ t-SNE [30] to analyze the feature level discrimination of our method and the visualization results are shown in Fig. 4. Compared with the feature extracted from the ImageNet pre-trained network, the distance between different class clusters in our method becomes evident, indicating that equipped with our proposed extrinsic and intrinsic supervision, the model is able to learn more discriminative features among different object categories regardless domains.