Domain-adaptive Person Re-identification without Cross-camera Paired Samples

Due to the strong application prospects, person re-ID across long-distance scenes has received significant attention from researchers in the past two years. In 2020, Zhang et al. [15] first proposed the application requirements of the problems across long-distance scenes and designed a single-camera training method to solve the problem caused by the lack of cross-camera paired samples. This method guarantees the discriminability of features by extracting hard-positive samples in the camera and excluding hard-negative sample pairs within-camera and cross-cameras. In cross-camera person identity matching, mitigating cross-camera intra-class variations is significant for person re-ID. However, this method ignores the impact of the above-mentioned problem on recognition performance. Therefore, Ge et al. [16] proposed a method to predict cross-camera pedestrian features of single-camera pedestrians. The predicted results are combined with the original features to form cross-camera positive sample pairs and used to supervise the model training. This method relies heavily on the predicted cross-camera pedestrian features. If the predictions are not good, it most likely hurts the model performance. To alleviate this problem, Wu et al. [17] proposed a camera-conditioned stable feature generation method, which reduces the impact of unideal features on model performance by improving the quality of generated features. Although this method improves the quality of generated cross-camera sample, the problem of model performance being limited by the quality of generated samples remains unresolved. Different from the above methods, the proposed method belongs to the domain adaptive method. It addresses the challenge of missing cross-camera paired samples by improving model generalization and domain adaptability.

The domain-adaptive person re-ID method trains the model through labeled source-domain samples and unlabeled target-domain samples, so that the model has good performance in the target domain. Since this method does not need to label samples in the target domain, which significantly reduces the labor cost of labeling training samples, it has attracted the attention of researchers and proposed a series of effective methods. Existing methods can be roughly classified into three categories, methods based on external model assistance, methods based on domain-adaptive feature extraction, and methods based on pseudo-label prediction. Methods based on external model assistance usually use an external model to assist re-ID models to gain domain adaptability. Commonly used external models include CamStyle transfer model [18], pedestrian pose extraction model [19], etc. In particular, methods such as PT-GAN [20], SPGAN [21], ATNet [22], and CR-GAN [23] all use the CamStyle transfer model to transfer the style of training samples from the source domain to the target domain, so that the model can be trained in a supervised manner on the transferred samples. However, the samples generated after style transfer are prone to corrupt information related to pedestrian identity cues. Utilizing such samples to train a model can easily degrade recognition performance. In pedestrian pose-assisted methods, the pose estimation model is usually used to estimate the pose information of pedestrians in the training set and combined with the generative adversarial mechanism to achieve pedestrian pose-invariant and domain-aligned feature extraction. In addition, to obtain discriminative features with consistent distribution, Yang et al. [24] proposed the part-aware progressive unsupervised domain adaptation person re-ID method. This method utilizes a pedestrian’s pose estimation model to perceive pedestrian body parts. These methods, which rely on external models for data preprocessing, are not conducive to the deployment in real-world scenes.

Methods based on domain-adaptive feature extraction often address the impact of domain discrepancy on recognition performance by learning transferable features. In particular, Yang et al. [25] proposed a patch-based unsupervised learning framework for obtaining transferable discriminative features. This method exploits the similarity between patches to learn features that are transferable across cameras. Wu et al. [26] proposed a camera-aware similarity consistency loss to learn consistent pairwise similarity distributions to address domain shift. Qi et al. [27] proposed an unsupervised camera-aware domain adaptation framework to address domain shift between source and target domains. Zou et al. [28] proposed to purify the target domain representation space through feature disentanglement, thereby enhancing the domain adaptability of the model. Chen et al.[29] used the domain invariance of pedestrian attribute information to improve the performance of visual features in domain-adaptive pedestrian matching. Dong et al. [30] proposed a triple adversarial learning and imaginative reasoning mechanism to obtain domain-adaptive discriminative features. Li et al.[31] proposed a mutual-promotion learning method for feature disentanglement, which achieved the separation of domain-invariant features and domain-related features, improving the performance of the model on the target domain. Although methods based on domain-adaptive feature extraction are effective, their performance is limited due to the underutilization of unlabeled samples from the target domain in model training.

To make full use of unlabeled samples from the target domain in model training, methods based on pseudo-label prediction have been proposed. In particular, Ge et al. [32] proposed an unsupervised mutual mean-teaching (MMT) framework for domain-adaptive person re-ID. The framework improves the accuracy of label prediction through offline refinement of hard pseudo-labels and online refinement of soft pseudo-labels using an alternating training scheme. Yang et al. [33] designed an asymmetric co-teaching framework. The framework suppresses noisy labels through the collaboration of two models. Zhai et al. [34] proposed an augmented discriminative clustering (AD-Cluster) technique to predict pseudo-labels of pedestrians in the target domain. Zeng et al. [35] proposed a hierarchical clustering method with hard-batch triplet loss to improve the quality of pseudo-label prediction. For the problem of model non-convergence in cluster-based pseudo-label prediction methods. Ji et al. [36] proposed an attention-driven two-stage clustering method. To suppress the noise in pseudo-labels, Zhao et al. [37] proposed a noise resistible mutual training (NRMT) mechanism to predict pseudo-labels through interactive instance selection. Shi et al. [38] proposed a reliability exploration with self-ensemble learning (RESL) framework to improve the reliability of pseudo-label prediction in domain-adaptive person re-ID.

To improve the adaptation of the model on the target domain, Chen et al. [39] combined data augmentation methods with a multi-label assignment strategy to decouple semantic features from the source domain. Additionally, a pre-trained model was used to extract various semantic features from the target dataset, and each semantic feature is regarded as a specific domain. The features of each domain are then clustered, and the connections between different clusters are used for self-distillation to generate more reliable pseudo-labels. Li et al. [40] exploited a logical reasoning mechanism that utilizes the logical relationships between samples to refine cross-camera pseudo-labels. When the training set in the target dataset contains a large number of cross-camera positive samples, the above-mentioned methods can predict pseudo-labels relatively reliably. However, across long-distance scenes, the possibility of the same pedestrian appearing under two cameras is extremely low, resulting in the target-domain samples participating in model training being negative samples across cameras.

As shown in Fig. 2, the proposed method consists of the feature extraction network $\bm{E}$, category synergy co-promotion module (CSCM), and cross-camera consistent feature learning module (CCFLM). The $\bm{E}$ is used to extract features from the source-domain and target-domain samples. The CSCM is composed of feature recombination by task (FRT) and interactive promotion learning (IPL). According to the contributions of features to pedestrian and camera identification, FRT divides features into two groups. One group is related to pedestrian identity and the other group is related to camera identity. Then the IPL of features is developed and carried out between the two groups to improve the discriminability of the features. The CCFLM includes the feature distribution alignment (FDA) and cross-camera identity consistent feature learning (ICL) under the supervision of cross-camera unpaired samples. The FDA is mainly used to make the cross-domain features share the same probability distribution. The ICL is mainly used to make the features learned by the model under the supervision of unpaired samples have cross-camera identity consistency.

Assume that the source-domain data sample set is $\bm{D}_{s}=\{\bm{x}_{s,i},y_{s,i},c_{s,i}\}_{i=1}^{N_{s}}$, where $\bm{x}_{s,i}$ represents the $i$-th sample from $\bm{D}_{s}$, $y_{s,i}\in\{1,2,\cdots,n_{s}\}$ and $c_{s,i}\in\{1,2,\cdots,k_{s}\}$ represent the identity label and camera label of the sample $\bm{x}_{s,i}$, $N_{s}$ is the total number of samples in $\bm{D}_{s}$, $n_{s}$ denotes the total category of pedestrian identity, and and $k_{s}$ denotes the total number of cameras. Similarly, the training set in the target domain is defined as $\bm{D}_{t}=\{\bm{x}_{t,j},y_{t,j},c_{t,j}\}_{j=1}^{N_{t}}$, where $y_{t,j}\in\{1,2,\cdots,n_{t}\}$ and $c_{t,j}\in\{1,2,\cdots,k_{t}\}$. In cross-region person re-ID, according to the protocol of [15], it assumes that the training set or domain adaptation in $\bm{D}_{t}$ has no paired images across camera views, i.e., for $\forall i,j$, $y_{t,i}\neq y_{t,j}$ when $c_{t,i}\neq c_{t,j}$. Images of pedestrians with the same identity only appear in a single camera, i.e., $\exists i,j$, such that $y_{t,i}=y_{t,j}$ when $c_{t,i}=c_{t,j}$.

In the feature extraction of pedestrian images, the TransReID [41] framework is used as the backbone denoted as $\bm{E}$. The network consists of 12 transformer layers. In the proposed method, it is necessary to analyze the contribution of extracted features to specific tasks and group them accordingly. Therefore, it is necessary to make $\bm{E}$ have the initial recognition ability. Let $\bm{E}(\bm{x}_{s,i})$ be the global token of the $i$-th pedestrian image $\bm{x}_{s,i}\in{\bm{D}_{s}}$, and $\bm{E}(\bm{x}_{s,i}^{k})$ be the $k$-th local token. The feature extraction network $\bm{E}$ can realize pre-training by minimizing Eq.(1).

where $Ce(\cdot)$ and $Tri(\cdot)$ are the cross-entropy loss and triplet loss functions respectively, ${{\bm{W}}_{id}}\in\mathbb{R}^{m\times n}$ is the identity classifier, $m$ is the total number of pedestrian identities, and $n$ is the number of channels of the feature vector. ${y}_{s,i}$ represents the identity label of the $i$-th sample from the source domain, and ${n_{s}}$ is the number of source-domain samples in a mini-batch.

Since the pedestrian identity classifier and the camera classifier in the proposed method need to promote each other to realize the separation of pedestrian identity and camera information, it is also necessary to pre-train the camera classifier.

where $\bm{x}_{i}\in{\bm{D}_{s}}\cup{\bm{D}_{t}}$, ${c_{i}}$ is the camera label of ${\bm{x}_{i}}$, ${{\bm{W}}_{cam}}$ is the camera classifier, and ${n_{t}}$ is the number of target-domain samples in a mini-batch.

In the domain-adaptive person re-ID, pedestrian discriminative feature extraction, which is not affected by domain shift, plays a crucial role in improving the domain generalization ability of the model. In multi-classification tasks, different features contribute differently to a specific classification task. If these features can be grouped by their contributions to a specific classification task, it is conductive to enhancing the ability of the network to extract task-specific discriminative features. Therefore, this paper proposes a CSCM. This module is mainly composed of FRT and IPL.

This paper aims to address the domain-adaptive person re-ID problem due to the lack of cross-camera paired samples. In this setting, there are no pedestrian across cameras with the same identity in the training samples from the target dataset due to the long distance between cameras. Therefore, it is challenging to learn discriminative features across cameras under the supervision of unpaired samples across cameras. This paper addresses the above problem from two aspects, instance-level distribution-consistency learning of source-domain and target-domain features and cross-camera identity consistency learning.

In the training phase, the samples from source domain are used in the first 100 epochs to perform model pre-training by minimizing Eq.(1). The camera classifier is pre-trained by minimizing Eq.(2) in the 100-200 epochs. Feature distribution alignment is achieved by minimizing Eq.(20) in the 200-300 epochs. In the remaining 100 epochs, the learning of consistent features of the same identity across cameras is achieved by minimizing Eq.(24). In the testing phase, $\bm{E}$ is used for feature extraction, and Euclidean distance is performed for pedestrian identity matching. The optimization algorithm is given in Algo. 1.

To verify the effectiveness of the proposed method, this paper uses three datasets specialized for person re-ID across distant scenes, namely Market-SCT [15], DukeMTMC-SCT (Duke-SCT) [15] and MSMT17-SCT (MSMT-SCT) [16] as the target datasets. These three datasets are derived by resetting the datasets Market-1501 (Market) [45], DukeMTMC (Duke) [46], and MSMT17 [47] according to the characteristics of across distant scenes. In Market, Duke, and MSMT17, each pedestrian image in the training set has the corresponding samples with the same identity across cameras. However, in Market-SCT, Duke-SCT, and MSMT17-SCT, the training set only has single-camera pedestrian samples, and there are no samples with the same pedestrian identity across cameras. In person re-ID across long-distance scenes, the datasets Market, Duke, and MSMT17 are used as source-domain data, and Market-SCT, Duke-SCT, and MSMT17-SCT are used as target-domain data. The specific information of different datasets is shown in Table I. In the objective evaluation of algorithm performance, Cumulative Matching Characteristics (CMC) [48] and mean Average Precision (mAP) [45] are used to evaluate the performance of each method under a single query setting.

This paper uses the TransReID [41] framework as the backbone of the proposed method. All pedestrian images are resized to 256$\times$128 and the batch size is 16. During model training, operations such as horizontal flipping, padding, random cropping, and random erasing are used for data augmentation. The SGD optimizer [49] with a momentum of 0.9, a weight decay rate of $1\times{10^{-4}}$, and a learning rate of $1.6\times{10^{-3}}$ are used to update the parameters of the encoder $\bm{E}$. In this process, the learning rate of ${{\bm{W}}_{id}}$, ${{\bm{W}}_{cam}}$, ${{\bm{W}}_{t,id}}$ and ${{\bm{\tilde{W}}}_{t,id}}$ is $1.2\times{10^{-4}}$. The proposed method is implemented under the PyTorch framework [50], and all experiments are completed on the platform with one NVIDIA GeForce RTX 2080 Ti GPU. The model is trained for a total of 400 epochs. In the first 10 epochs, the learning rate is linearly adjusted by the warm-up strategy [51]. At the 40-th and 70-th epochs, the learning rate decays by 10%.

The proposed method consists of four core components, feature recombination by task (FRT) and interactive promotion learning (IPL), feature distribution alignment (FDA), and identity consistent learning across cameras with single-camera supervision (ICL). The effectiveness of the above core components is verified on the Market$\rightarrow$Duke-SCT and Duke$\rightarrow$MSMT-SCT. After removing the above four core components from the proposed method, the obtained model is used as the baseline method, and the loss function shown in Eq.(1) is used to train the baseline model. In the ablation experiment, the method after adding FRT to Baseline is denoted as Baseline+FRT, the method after adding IPL to Baseline+FRT is denoted as Baseline+FRT+IPL, the method after adding FDA to Baseline+FRT+IPL is denoted as Baseline+FRT+IPL+FDA, the method of the complete model is denoted as Baseline+FRT+IPL+FDA+ICL.

Effectiveness of FRT. To improve the discriminability of features, this paper groups the features on different channels by task and designs FRT. To verify the effectiveness of FRT, FRT is added to Baseline and the performance of Baseline+FRT is compared with that of Baseline. As shown in Table II, compared with Baseline, the recognition performance obtained by Baseline+FRT on both Market$\rightarrow$Duke-SCT and Duke$\rightarrow$MSMT-SCT is significantly improved. Specifically, after adding FRT, the mAP increases by 4.3% and 3.9% on the Market$\rightarrow$Duke-SCT and Duke$\rightarrow$MSMT-SCT, respectively. Additionally, the recognition accuracy on Rank-1 also improves from 66.8% (39.0%) to 69.9% (43.5%). This indicates that FRT plays a positive role in improving model performance.

Effectiveness of IPL. IPL is mainly used to remove camera-related information from features, thereby reducing the limitation of the interference information brought by the camera style on person re-ID. According to the experimental results shown in Table II, when IPL is added to Baseline+FRT, the corresponding performance of the model on the Market$\rightarrow$Duke-SCT (Duke$\rightarrow$MSMT-SCT) increases from 69.9% and 52.2% (43.5% and 20.2%) in Rank-1 and mAP to 78.6% and 63.8% (44.0% and 20.5%). This shows that IPL is effective in removing camera information to improve the domain adaptability of the model.

Effectiveness of FDA. To reduce the distribution inconsistency of source-domain and target-domain features caused by domain shift, FDA module is designed to achieve instance-level feature distribution alignment. As shown in Table II, after adding FDA to Baseline+FRT+IPL, the Rank-1 and mAP performance of Baseline+FRT+IPL+FDA on the Market$\rightarrow$Duke-SCT (Duke$\rightarrow$MSMT-SCT) changes from the original 78.6% and 63.8% (44.0% and 20.5%) to 79.4% and 64.1% (48.1% and 24.6%), respectively. This shows that the FDA plays a positive role in improving the domain adaptability of the re-ID model.

Effectiveness of ICL. The ICL module is introduced to solve the problem of missing cross-camera positive samples. As shown in Table II, compared with Baseline+FRT+IPL+FDA, the corresponding Rank-1 and mAP increase from 79.4% and 64.1% (48.1% and 24.6%) to 83.0% and 69.1% (56.5% and 31.1%) on Market$\rightarrow$Duke-SCT (Duke$\rightarrow$MSMT-SCT) respectively, after the introduction of ICL. The increase rates of Rank-1 and mAP reach 3.6% and 5.0% (8.4% and 6.5%), respectively. This is because ICL enables samples with similar features to show higher similarity. Additionally, it can effectively extend the distance between hard negative samples across cameras, which plays a positive role in pedestrian identity matching. Fig. 5 shows the pedestrian retrieval results after using different modules, from which the contribution of each module is intuitively illustrated. Fig. 6 also qualitatively shows the degree of contribution of the proposed method. Through quantitative and qualitative experiments, the effectiveness of each module in the proposed method has been demonstrated.

Experiments on Market-SCT and Duke-SCT. To verify the effectiveness of the proposed method and its superiority over existing methods, Market-SCT and Duke-SCT are used as the target datasets, and Duke and Market are used as the source-domain datasets respectively. In these experiments, the comparison methods mainly involve both unsupervised learning (USL) and unsupervised domain adaptive (UDA) person re-ID methods. USL methods mainly include MCNL [15], Precise-ICS [52], AGW [53], SimSiam [54], STS [55], ICE [56], CCFP [16], CCSFG [17], and PPLR [57]. UDA methods mainly include MMT-900 [32], SPCL [58], Meb-Net [59], CAC [60], IDM [61], Dual-Refine [62], P2LR [63], and DRDL [31]. The results listed in Table III were obtained by using the codes provided by the corresponding original authors and retraining the corresponding models under the original parameter settings. All comparative experiments only used the optimal preset parameters provided by authors in their papers.

For the USL methods, CCSFG achieves the suboptimal performance on Market-SCT. Its Rank-1 and mAP reach 84.0% and 67.7%, respectively. CCFP also achieves the suboptimal performance on Duke-SCT, with Rank-1 and mAP reaching 80.3% and 64.5%, respectively. In contrast, Rank-1 and mAP obtained by the proposed method are 86.3% and 68.3% (83.0% and 69.1%) respectively on the Market-SCT (Duke-SCT). Compared with the second-best performance, the proposed method improves performance by 2.3% and 0.6% (2.7% and 4.6%) on Market-SCT (Duke-SCT), respectively. This shows that the proposed method has better recognition performance in person re-ID across long-distance scenes than the above-mentioned USL methods. The proposed method belongs to the domain-adaptive recognition methods. Compared with the domain-adaptive methods listed in the Table III, the proposed method also shows better performance. This is mainly because the existing methods are affected by the lack of paired training samples of the target data, which limits their performance on Market-SCT and Duke-SCT.

Experiments on MSMT-SCT. To further prove the effectiveness of the proposed method, MSMT-SCT is used as the target dataset, and both Duke and Market are used as the source-domain datasets to train the model in the second set of experiments. Since MSMT-SCT contains more samples than Market and Duke, Duke$\rightarrow$MSMT-SCT and Market$\rightarrow$MSMT-SCT are more challenging. To verify the superiority of the proposed method over the USL methods, the proposed method is first compared with the USL-based methods, including MCNL [15], Precise-ICS [52], AGW [53], SimSiam [54], STS [55], ICE [56], CCFP [16], CCSFG [17], and PPLR [57], and the experimental results of different methods are shown in Table IV. It can be seen that the proposed method outperforms state-of-the-art USL methods. In addition, compared with the domain-adaptive methods MMT-1000 [32], MMT-2000 [32], SPCL [58], Meb-Net [59], CAC [60], IDM [61], Dual-Refine [62], P2LR [63], and DRDL [31], the proposed method also shows strong competitiveness. As shown in Table V, Rank-1 and mAP obtained by the proposed method reach 56.5% and 31.1% (51.4% and 28.3%) on Duke$\rightarrow$MSMT-SCT (Market$\rightarrow$MSMT-SCT) respectively. Compared with the second-best performance, the proposed method improves performance by 14.6% and 15.5% (15.4% and 14.7%), respectively. This shows that the proposed method has more advantages on the task of spanning from small-size datasets to large-size datasets.

In the proposed method, clustering is adopted to make the features between similar samples have high similarity, so that the model is able to emphasize feature similarity between camera views. Since the optimal solution of hyperparameters cannot be obtained through an optimization method, the optimal hyperparameter values are searched through experiments in this study. Fig. 7 shows the changes in model performance when k takes different values. In the clustering process, the performance of the proposed model is affected, when the number of clusters is too large or too small. When k=700 (k=1,500), the proposed method achieves the best performance on Market$\rightarrow$Duke-SCT (Duke$\rightarrow$MSMT-SCT). Therefore, in the experiments conducted on Market$\rightarrow$Duke-SCT, k is set to 700, while in the experiments conducted on Duke$\rightarrow$MSMT-SCT, k is set to 1,500.