TAL: Two-stream Adaptive Learning for Generalizable Person Re-identification

Traditional person re-id methods mainly depend on hand-crafted features, such as [28, 13]. With the development of deep learning, recent re-id models are typically based on deep neural networks, including but not limited to part-based feature representation learning [41, 40, 31], distance metric learning [1, 19, 5, 8, 39], and video-based learning [49, 54, 48]. Despite their remarkable success, these models are fully-supervised. Due to domain shift, their performance will be significantly degraded if directly applied to a novel domain. However, it is difficult and time-consuming to obtain the annotated data for each target domain, which makes it less practical for real-world applications. Therefore, unsupervised domain adaptation (UDA) re-id [38, 15, 16, 30] and domain generalizable (DG) re-id [37, 21, 22, 25, 53] have recently attracted growing interest in the community.

Unsupervised domain adaptation (UDA) is the task of training a model on the source domain with labels and transferring it to the target domain under the unsupervised setting. UDA has been widely studied in recent years and was introduced into re-id task by [29]. Current UDA methods can be generally categorized into two branches: (1) those using generative adversarial networks (GAN) [17] to transfer the style of the source domain labeled images to the target domain [12, 7, 27], and (2) those adapting to the target domain with pseudo labels generated by clustering [38, 15, 16, 52] or assigning soft labels [43]. However, UDA re-id models still rely on target domain images for adaptation, which are not always available. In contrast, DG re-id does not require the target domain data during training, which is more practical.

The objective of domain generalization is to learn a robust, generalizable model, such that the learned model can obtain good performance on an unseen target domain without further updates. Existing DG re-id methods can be divided into two categories: (1) those that train a single model to learn domain-invariant features from single/multi-source domains [37, 25, 22, 53], and (2) those that employ a mixture of experts (MoE) to learn a series of separated experts for the subsets of whole source domains [10]. For the first category, Song et al. [37] designed a Domain-Invariant Mapping Network (DIMN) for DG re-id. Jin et al. [22] proposed Style Normalization and Restitution (SNR) to disentangle the identity-relevant and identity-irrelevant features. Liao et al. [25] proposed a query-adaptive matching mechanism to aggregate the local similarities for DG re-id. Zhao et al. [53] introduced a meta-learning strategy to simulate the training and test process of DG re-id. For the second category, MoE methods have been widely employed in scene parsing [14] and image classification [2, 45]. Dai et al. [10] incorporated this strategy into the DG re-id task by designing a relevance-aware mixture of experts (RaMoE) that utilizes meta-learning for updating the voting network. In this work, we take advantage of both approaches by introducing a two-stream framework to simultaneously exploit the domain-specific and domain-invariant information.

Recently, several works [33, 32, 4, 3, 9] have shown advantages of normalization techniques for boosting the generalization ability of neural networks. One of the most inspiring methods is domain-specific batch normalization (DSBN) [4], which employs exclusive BN layers to represent the statistics of the source and target domains. Nam et al. [32] designed a novel batch-instance normalization (BIN) layer to boost the collaboration of both BN and instance normalization (IN) layers. Choi et al. [9] proposed a generalizable normalization layer by simulating unsuccessful generalization scenarios in the meta-learning pipeline. Inspired by these methods, we propose to employ the DSBN in our DS network, and utilize DSBN parameters to predict the statistics of the unseen domains in the DI network. In this way, both DS and DI information are exploited and thus yielding enhanced generalization ability.

Suppose we have $K$ source domains $\{\mathcal{D}_{1},...,\mathcal{D}_{K}\}$ in the training set, where the $i$-th domain $\mathcal{D}_{i}=\{\mathbf{x}_{i}^{1},...,\mathbf{x}_{i}^{N_{i}}\}$ contains $N_{i}$ training images. Based on a shared backbone (e.g., ResNet-50 [18]), the DS network contains a set of BN layers that are specifically engaged in each domain, as shown in Fig LABEL:subfig-1-1. We denote the DS network as $\mathcal{F}_{DS}=\{\mathcal{F}_{1},...,\mathcal{F}_{K}\}$, which can be regarded as $K$ domain experts that share all the parameters except for the BN layers. Considering there are $P$ stages in the backbone, each domain expert will output a series of domain-specific feature maps:

$$\mathbf{H}_{i}=\mathcal{F}_{i}({\mathbf{x}_{i}}),$$

(1)

where $\mathbf{H}_{i}=\{\mathbf{h}_{i}^{1},...,\mathbf{h}_{i}^{P}\}$, $\mathbf{x}_{i}\in\mathcal{D}_{i}$ and $i=1,...,K$. Subsequently, $\mathbf{H}_{i}$ are fed into a multi-scale variation of the query-adaptive convolutional layer [25] (MS-QAConv), and supervised by a triplet matching loss $\mathcal{L}_{tri}$ (which will be introduced in Sec 3.3).

As each domain expert in our DS network only receives single-source information, it is difficult for them to generalize to unseen domains. To address this, we employ a hybrid source to mimic the target domain, as shown in Fig. LABEL:subfig-1-2, and again train the DS network without domain labels. Specifically, suppose $\mathcal{D}_{HS}$ contains all the training samples regardless of domain labels. Given input $\mathbf{x}_{HS}\in\mathcal{D}_{HS}$, we extract its features from all the domain experts:

$$\mathbf{H}_{i,HS}=\mathcal{F}_{i}({\mathbf{x}_{HS}}),$$

(2)

where $i\in\{1,...,K\}$. The feature maps from all the domain experts $\{\mathbf{H}_{1,HS},...,\mathbf{H}_{K,HS}\}$ are aggregated by a domain-adaptive MS-QAConv layer, which we denote as MSDA-QAConv. The same matching loss $\mathcal{L}_{tri}$ is applied. Note that in this case, the gradients only flow back to the MSDA-QAConv layer, while the backbone parameters are no longer updated. In this way, the domain experts maintain the domain-specific information while being able to generalize to unseen domains.

Although the DS network tries to capture the domain-invariant information given the hybrid source, the feature aggregation only takes place at the last MSDA-QACovn layer, which is incapable of modeling the target domain statistics at different levels. To this end, we further introduce a DI network to normalize the features into domain-invariant representation. In particular, the DI network shares the same backbone with the DS network, but with specifically designed domain-adaptive BN (DABN) layers. To make the DABN layers adaptively approximate the target domain statistics, their normalization results are dynamically predicted from the DSBN layers, as shown in Fig. LABEL:subfig-1-2. Suppose the input feature before the DABN layer is $\mathbf{x}$, we input $\mathbf{x}$ into each DSBN, where the output of the $i$-th domain can be represented as:

$${\rm DSBN}_{i}(\mathbf{x},\gamma_{i},\beta_{i})=\gamma_{i}\hat{\mathbf{x}}+\beta_{i},$$

(3)

where $\gamma_{i}$ and $\beta_{i}$ are the DSBN parameters and

$$\hat{\mathbf{x}}=\frac{\mathbf{x}-\mu_{i}}{\sqrt{\sigma_{i}^{2}+\epsilon}}.$$

(4)

Here, $\mu_{i}$ and $\sigma_{i}$ are the mean and variance of the input feature. In DABN, the normalization result $\tilde{\mathbf{x}}$ can be calculated as a weighted summation of the DSBN outputs:

$${\rm DABN}(\mathbf{x},\alpha,\gamma,\beta)=\sum_{i=1}^{K}\alpha_{i}(\gamma_{i}\hat{\mathbf{x}}+\beta_{i}),$$

(5)

where the weighting terms $\alpha=\{\alpha_{1},...,\alpha_{K}\}$ are estimated in a similar way as squeeze-excitation network [20] (SENet), as shown in Fig. 3. Specifically, the input feature maps are first input into an average pooling layer to extract global information. Then a bottleneck layer with ReLU activation is introduced to reduce the dimension (with reduction rate $r$), followed by another linear layer to predict the weights of each domain. The attention weights are normalized by a softmax layer:

$$\alpha={\rm Softmax}(\mathbf{W}_{2}({\rm ReLU}(\mathbf{W}_{1}{\rm GP}(\mathbf{x})))),$$

(6)

where GP denotes global pooling operation, $\mathbf{W}_{1}$ and $\mathbf{W}_{2}$ are the parameters of the linear layers, respectively. $\alpha$ is applied to Eq. 5 to calculate DABN activation.

To make the model not only adaptive to novel domains, but also flexible to handle new test samples, we train our framework in the same fashion as QAConv [25]. Specifically, QAConv tries to find local correspondences between two feature maps by constructing convolution kernels on-the-fly from one feature map, while performing convolution on the other feature map to calculate the local similarities. In our framework, we design two enhanced variations of QAConv, to further capture the multi-scale information (MS-QAConv), and to aggregate the domain-specific features (MSDA-QAConv).

MS-QAConv. The structure of MS-QAConv is illustrated in Fig. 4. To match a pair of images, we first extract the multi-scale feature maps (e.g., from the res4 to res5 layers) of both the query and gallery images. Then the query feature maps are partitioned into local patches, which are re-organized into convolution kernels. By applying these kernels to the gallery feature maps, i.e., performing QAConv, we can get a set of response maps measuring the similarities between local patches. Then a global max pooling (GMP) operation is applied to these response maps to extract a feature vector that represents the best local correspondence. Vanilla QAConv directly take the vectors from a single scale (res4) into a BN-FC-BN block to calculate the final similarity value. However, as indicated by prior re-id works [6, 44], multi-scale feature representation contains hierarchical information to facilitate re-id learning. Therefore, MS-QAConv first aggregates the multi-scale matching information by concatenating the feature vectors from several scales, and then predicts the final similarity. In this way, the multi-level information is exploited to help our model make better predictions.

MSDA-QAConv. When an image without domain label is input into the DS network, it requires the QAConv block to be able to adaptively aggregate the response maps from different domain experts. Therefore, we further propose MSDA-QAConv to tackle this challenge. Specifically, the basic structure of MSDA-QAConv is similar to MS-QAConv, but it further contains an attention block to weight the response vectors from each domain. The attention weights are calculated in a similar way as in Eq. 6, where the vectors are concatenated and fed into a FC-ReLU-FC-Softmax block to output the domain weights. Subsequently, the weighted summations of response vectors are calculated as the domain-level representation at each scale level, which are further aggregated to yield the final similarity. In this way, both multi-scale and multi-domain information are simultaneously captured for better adaptive feature learning.

Loss Function. A triplet loss [19] with batch hard-negative mining is employed to supervise model training. Within a batch containing $B$ instances, we calculate the pairwise similarities between all the training samples. Specifically, we use $S_{i,p}$ to denote the similarities between the $i$-th sample and its positive pairs, while $S_{i,n}$ denotes the negative similarities. Then the batch hard triplet loss is calculated as follows:

$\displaystyle\mathcal{L}_{tri}=\sum_{i=1}^{B}[m-\min_{p}{S_{i,p}}+\max_{\begin{subarray}{c}n\end{subarray}}{S_{i,n}}]_{+},$

(7)

where $m$ is a hyperparameter denoting the margin between the positive and negative pairs, and $[*]_{+}=\max(*,0)$ ensures the loss value to be non-negative.

Model Inference. In the test phase, the output of the MSDA-QAConv block from the DS network and the output of the MS-QAConv from the DI network are combined to yield the final similarity.

Our implementation is based on the official code of QAConv¹¹1https://github.com/ShengcaiLiao/QAConv [25] and its improved variant (QAConv-GS [26]). We employ the IBN-Net50 [33] pretrained on ImageNet [11] as our default backbone. The features from res4 and res5 layers are employed in our MS-QAConv and MSDA-QAConv blocks. In DABN, we set the reduction rate $r=16$. We set the batch size $B=64$, and we employ the graph sampler [26] to select 16 identities containing 4 samples in each ID. The input images are resized to 384$\times$128, and several data augmentation strategies, including random flipping, cropping, color jittering, are applied. We adopt the SGD optimizer with a weight decay of 0.0005. The initial learning rate is set to 0.005 and is reduced by a factor of 10 after 20 epochs, with the training stage terminating at the 30-th epoch. All the experiments are implemented in PyTorch [34], with four NVIDIA 3090 GPUs.

Datasets.²²2Note that DukeMTMC-reID [36] has been taken down and we did not employ it in our paper. DukeMTMC-reID is replaced by the RandPerson dataset [46] in our experiments. We employ four large-scale datasets, CUHK03 [23], Market-1501 [55], MSMT17 [47], and RandPerson [46] in our experiments, among which RandPerson is a newly released synthesized person re-id dataset and the others are captured from real-world cameras. For CUHK03 dataset, we adopt CUHK03-NP [56] detected subset which contains 767 subjects for training and 700 for evaluation. The widely used Market-1501 dataset includes 1,501 identities and 32,668 pedestrian images captured by 6 different cameras. MSMT17 is currently one of the largest public datasets obtained with real-world cameras. It consists of 4,101 annotated identities and 126,441 images from 12 outdoor and 3 indoor cameras. RandPerson is a synthesized re-id dataset generated by MakeHuman and Unity3D, and it contains a training set of 1,801,816 images and 8,000 synthesized identities. To make a fair comparison, we follow [26] to employ a subset of RandPerson, including 132,145 images of the 8,000 identities. Since RandPerson merely contains the training subset, it is not adopted as the target domain in our experiment.

Evaluation Protocols. To fully verify the effectiveness of our two-stream adaptive learning framework, we design experiments under both single-source and multi-source protocols: (1) Protocol-1: single-source DG re-id. Under this protocol , we perform training on the training set of one dataset or its subset, and evaluate on the test set of another dataset. As a single dataset does not naturally contain different domains, we split domains according to camera views, which display different statistics within a dataset. (2) Protocol-2: multi-source DG re-id. Under this protocol, we employ RandPerson and two of the three real-world datasets (CUHK03, Market-1501 and MSMT17) as the training set, and the remaining one is used for evaluation. In this case, each source dataset can be naturally regarded as a domain. For both settings, we adopt mean average precision (mAP) and CMC top-1, top-5, top-10 accuracy as the evaluation metrics.

Comparison Under Protocol-1. We compare our TAL framework with several DG re-id methods, including MGN [44], MuDeep [35], CBN [57], PCB [42], ADIN [51], SNR [22], QAConv [25] and QAConv-GS [26]. We successively employ Market-1501, MSMT17, and RandPerson as training sources, while the other real-world datasets are used for testing. As shown in Table 1, our proposed TAL clearly outperforms the state-of-the-art models on all three target datasets. TAL further improves our baseline models, i.e., QAConv and QAconv-GS. On common real-world $\to$ real-world tasks, we achieve 0.6% to 3.5% improvements in terms of mAP and up to 3.8% improvements for top-1 accuracy. The improvements on the more challenging synthesized $\to$ real-world task are up to 1.4% for top-1 and 2.1% for mAP. Notably, our experiment confirms the finding in [46], which claims that models trained on a synthesized dataset could achieve competitive generalization ability as those trained on a real dataset.

Comparison Under Protocol-2. Under the multi-source protocol, we respectively adopt Market-1501, CUHK03, and MSMT17 as the test set, while all the other datasets are combined for training. To conduct fair comparisons, we use the authors’ code to re-implement QAConv-GS [26], $\rm M^{3}L$ [53] and MetaBIN [9] with the same training data as our method. From the comparisons shown in Table 2, we can make the following observations. First, our proposed TAL framework improves current methods by a large margin. For example, mAP is improved by 5.1%, and top-1 is increased by 2.1% on Market-1501. Second, state-of-the-arts multi-source DG re-id methods appear to be not stable, as QAConv-GS shows advantages on Market-1501 and MSMT17, but clear poor performance for both mAP and top-1 on CUHK03. In contrast, benefiting from the two-stream design for capturing both domain-specific and domain-invariant information, our method demonstrates notable superiority on all three test datasets.

In this section, we analyze the effectiveness of our TAL framework and conduct comprehensive ablation studies on the detailed components of TAL. We report the results under protocol-2 with the target datasets Market-1501 (M) and CUHK03 (C).

Effectiveness of Two-stream Modeling. Since TAL aims to simultaneously capture the domain-specific and domain-invariant information in a single framework, it is important to investigate how each sub-network influences the overall performance and how they work together. To this end, we perform independent evaluations on the output of each network, and the results are reported in Table 3. With neither the DS nor the DI networks, our framework degenerates into QAConv-GS [26], which could be regarded as our baseline. By introducing the DS and DI networks, the performances are continuously improved. Finally, by combining the results of both the DS and DI networks under the IBN-Net50 backbone, the performance of TAL is further improved, i.e., improving the baseline by 5.1% and 9.2% in mAP on Market-1501 and CUHK03, respectively. In the meantime, significant improvements can also be achieved with the ResNet-50 backbone. These results indicate that both the DS and DI information plays an important role in enhancing the generalization ability of the re-id model, and in TAL, the two-stream structure successfully captures both clues as complements to each other.

Effectiveness of Domain Adaptive BN. To evaluate the effectiveness of DABN in adaptively aggregating domain-specific information, we compare it with an average pooling and a simple BN baseline. From Table 4, it can be observed that by replacing the attention mechanism with an average pooling operation, the performance is decreased by $\sim$1% on both datasets. In the meantime, replacing DABN with a vanilla BN layer also deteriorates the performance. These results indicate that DABN could facilitate the fusion of statistics from the domain-specific BNs and effectively approximate the unseen target domain information.

Effectiveness of Domain Adaptive Matching. In the DS network, we employ MSDA-QAConv to aggregate the correspondence maps and then perform matching. To evaluate its contribution, we compare it with two baselines. First, instead of domain-adaptive attention, we employ the average response map to calculate the matching results. As shown in Table 5, this causes 1.6% to 1.9% performance drop in mAP. Second, we remove the MSDA-QAConv block, and directly sum up the voting results from domains-specific matchers. This also results in inferior performance. These results verify the strong adaptability of the proposed domain-adaptive matching strategy.

Effectiveness of Multi-Scale Matching. We conduct experiments by employing different levels of features in the backbone for calculating the similarity between image pairs. Specifically, we only investigate the features from the res4 and res5 layers, because low-level features from res2 and res3 layers are with relatively high resolution and will cause memory and efficiency issues. As shown in Table 6, although single-level features also yield promising results, combining them further significantly improves the performance. These results demonstrate that multi-scale features not only work well in supervised re-id tasks, but also benefit DG re-id.

To better illustrate the improvements achieved with our framework, we visualize some qualitative matching results in Fig. 5. We can observe that our model (TAL) successfully retrieves the difficult queries from the target domain under various challenging conditions, including occlusion, pose/viewpoint variation, where the baseline model (QAConv-GS) fails. Notably, although QAConv-GS also adopts a local matching strategy, it fails to accurately differentiate persons with similar appearances. In contrast, our model performs much better under this circumstance. We believe this is because our framework simultaneously captures DS and DI information, which makes TAL more robust to the feature variation caused by domain shift.

To better present the working mechanism of the DS and DI networks, we visualize the local correspondences between the query and gallery images. As shown in Fig. 6, the domain experts in the DS network generate different matching patterns. For example, the expert trained on CUHK03 focuses more on the head and leg regions, maybe because these regions in Market-1501 have similar statistics with CUHK03. In contrast, the correspondences generated by the DI network trained with the hybrid dataset tend to locate in more diverse regions, e.g., head, body, and leg, which are critical for identifying a person. This demonstrates that the DS and DI networks provide complementary information to each other to better identify a person in unseen domains.