A Benchmark of Video-Based Clothes-Changing Person Re-Identification

The overall pipeline of the proposed method is illustrated in Fig. 2. Given an image sequence $x^{\mathrm{q}}$, whose identity label is denoted as $y^{\mathrm{q}}$, as query, person Re-ID aims to return a ranking list of gallery image sequences where the sequences from the same person, i.e., $\{x^{\mathrm{g}}_{i}|y^{\mathrm{g}}_{i}=y^{\mathrm{q}}\}$, can rank top. To this end, this paper proposes a two-branch confidence-aware re-ranking framework. Specifically, given a query and several galleries, we first extract the appearance and gait features. Base on that, we conduct the inference stage to get the initial ranking list and collect top-K candidates for model training. Then, we construct candidate relation graph and use GCN to estimate the confidence for balancing the two branches. Finally, we fuse the confidence-aware re-weighted appearance and gait representations, and re-rank the galleries as the final retrieval result.

Since the value of $N$ is usually large, we select $K$-nearest neighbors of the query for subsequent training. Specifically, we select the top $\gamma\cdot K\ (0\leq\gamma\leq 1)$ samples of $\mathcal{R}^{\mathrm{a}}$, and supplement the rest with the top samples of $\mathcal{R}^{\mathrm{b}}$. The selected $K$ candidates are denoted as $\mathcal{R}^{\mathrm{t}}=\{t_{1},t_{2},\cdots,t_{K}\}$.
Candidate relation graph building. Given the selected samples $\mathcal{R}^{\mathrm{t}}$, we first construct two graphs with respect to appearance and gait features, respectively. We approximately represent the appearance similarity between the query $x^{\mathrm{q}}$ and sample $x^{\mathrm{g}}_{t_{i}}$ in $\mathcal{R}^{\mathrm{t}}$ by applying min-max normalization to the distance:

$$\begin{split}s_{k}^{a}=\frac{M-d(\mathbf{a}^{\mathrm{q}},\mathbf{a}^{\mathrm{g}}_{t_{k}})}{M-m},\end{split}$$

(1)

where $M$ and $m$ denote the maximum and minimum values of all $d(\mathbf{a}^{\mathrm{q}},\mathbf{a}^{\mathrm{g}}_{t_{k}})$ with $t_{k}\in\mathcal{R}^{\mathrm{t}}$, respectively. The gait similarity $s_{k}^{\mathrm{b}}$ can be represented by the same way.

The similarity of appearance features between the query and samples in $\mathcal{R}^{\mathrm{t}}$ and that of gait features are combined as an initialized similarity matrix $\mathbf{S}_{0}\in\mathbb{R}^{K\times 2}$, and then embedded into a shared matrix $\mathbf{S}\in\mathbb{R}^{K\times D}$ by an encoder. For the appearance graph, the nodes represent all samples, and the features of each node is initialized as the vector of the corresponding row in the matrix $\mathbf{S}$. The edges represent the proximity relationship between the samples. If the gait feature $\mathbf{g}^{\mathrm{g}}_{t_{j}}$ is one of the $n$-nearest neighbors of feature $\mathbf{g}^{\mathrm{g}}_{t_{k}}$, there will be an edge from $t_{j}$ to $t_{k}$, and the edge feature $\mathbf{e}_{jk}$ will be defined as the element-wise multiplication of $\mathbf{a}^{\mathrm{g}}_{t_{j}}$ and $\mathbf{a}^{\mathrm{g}}_{t_{k}}$. In the same way, the gait graph can be constructed.

After the GCN layer, a linear layer is used to map the node feature into a one-dimensional confidence score, which can be formulated as

$$c_{k}=\omega^{\top}\mathbf{H}_{k}^{L},$$

(3)

where $\omega$ denotes the learnable weight of the linear layer, and $c_{k}$ denotes the estimated confidence score for node $k$. Note that the network structure used to extract appearance confidences from the appearance graph and that used to extract gait confidences from the gait graph are the same, while do not share parameters. The appearance confidence for node $k$ can be denoted as $c_{k}^{\mathrm{a}}$ and the gait confidence can be denoted as $c_{k}^{\mathrm{b}}$.
Confidence-aware re-ranking. With the estimated confidence, we propose to adjust the weight of original appearance/gait similarity for each sample in the fusion phase. The final similarity between the query $x^{\mathrm{q}}$ and the sample $x^{\mathrm{g}}_{t_{k}}$ in $\mathcal{R}^{t}$ is defined as:

$$s_{k}=c_{k}^{\mathrm{a}}s_{k}^{\mathrm{a}}+c_{k}^{\mathrm{b}}s_{k}^{\mathrm{b}}+\omega_{0}^{\top}\mathbf{S}(k,:),$$

(4)

where $\mathbf{S}(k,:)\in\mathbb{R}^{D}$ denotes the vector in the k-th row of matrix $\mathbf{S}$, i.e., the initialized node feature, which is mapped into one dimension using a linear layer with weight $\omega_{0}$. Finally, the revised ranking list can be obtained by descending sort of the final similarities.
Confidence loss. To guide the learning of the network, we propose the confidence loss by designing pseudo-labels for the confidence. Specifically, assume the ground-truth label of the sample $x^{\mathrm{g}}_{t_{k}}$ is $b_{k}$. When the identity label of sample $x^{\mathrm{g}}_{t_{k}}$ equals that of the query, i.e., $y^{\mathrm{g}}_{t_{k}}=y^{\mathrm{q}}$, the value of $b_{k}$ is 1, otherwise it is 0. Given the initial appearance similarity $s_{k}^{\mathrm{a}}$ and the ground-truth label $b_{k}$ of sample $x^{\mathrm{g}}_{t_{k}}$, we propose to define the pseudo-label of confidence for appearance feature as

$$\tilde{c}_{k}^{\mathrm{a}}=|(1-b_{k})-s_{k}^{\mathrm{a}}|,$$

(5)

where $|\cdot|$ denotes the symbol of absolute value.

The proof is as follows. Since similarity indicates the relational degree between two samples, i.e., the greater the similarity, the higher the probability that the query and the gallery sample have the same identity label, we define a threshold $\lambda$ to represent the prediction results based on similarities. If $s_{k}^{\mathrm{a}}\geq\lambda$, we consider the prediction result $\hat{b}_{k}^{\mathrm{a}}$ to be 1. In this case, if $b_{k}$ is also 1, i.e., the prediction is correct, the value of confidence should be large. Intuitively, the larger the similarity $s_{k}^{\mathrm{a}}$, the larger the confidence $\tilde{c}_{k}^{\mathrm{a}}$ should be. Therefore, $\tilde{c}_{k}^{\mathrm{a}}$ can be assigned the value of $s_{k}^{\mathrm{a}}$. In contrast, if $b_{k}=0$, i.e., the prediction is wrong, the value of confidence should be small. And the larger the similarity $s_{k}^{\mathrm{a}}$, the smaller the confidence $\tilde{c}_{k}^{\mathrm{a}}$ should be. Therefore, $\tilde{c}_{k}^{\mathrm{a}}$ can be assigned the value of $1-s_{k}^{\mathrm{a}}$. The above can be formulated as

$$\tilde{c}_{k}^{\mathrm{a}}=\begin{cases}s_{k}^{\mathrm{a}},&b_{k}=1\\ 1-s_{k}^{\mathrm{a}},&b_{k}=0\\ \end{cases}.$$

(6)

Similarly, if $s_{k}^{\mathrm{a}}<\lambda$, we consider the prediction result $\hat{b}_{k}^{\mathrm{a}}$ to be 0. In this case, if $b_{k}=1$, i.e., the prediction is wrong, the value of confidence should be small. And the smaller the similarity $s_{k}^{\mathrm{a}}$, the smaller the confidence $\tilde{c}_{k}^{\mathrm{a}}$ should be. Therefore, $\tilde{c}_{k}^{\mathrm{a}}$ can be assigned the value of $s_{k}^{\mathrm{a}}$. In contrast, if $b_{k}=0$, i.e., the prediction is correct, the value of confidence should be large. And the smaller the similarity $s_{k}^{\mathrm{a}}$, the larger the confidence $\tilde{c}_{k}^{\mathrm{a}}$ should be. Therefore, $\tilde{c}_{k}^{\mathrm{a}}$ can be assigned the value of $1-s_{k}^{\mathrm{a}}$. The formulation of the above analysis takes the same form as Eq. 6. Further, Eq. 6 can be rewritten as follows:

$$\tilde{c}_{k}^{\mathrm{a}}=\begin{cases}s_{k}^{\mathrm{a}}-(1-b_{k}),&b_{k}=1\\ (1-b_{k})-s_{k}^{\mathrm{a}},&b_{k}=0\\ \end{cases},$$

(7)

which is exactly the form of classified discussion of Eq. 5. In the same way, the pseudo-label of confidence for gait feature can be defined as

$$\tilde{c}_{k}^{\mathrm{b}}=|(1-b_{k})-s_{k}^{\mathrm{b}}|.$$

(8)

In the training stage, the designed pseudo-labels $\tilde{c}_{k}^{\mathrm{a}}$ and $\tilde{c}_{k}^{\mathrm{b}}$ play a role of ground-truths of confidence to guide the confidence estimation. The confidence loss can be formulated as

$$\mathcal{L}_{c}=\sum_{k=1}^{K}(|\tilde{c}_{k}^{\mathrm{a}}-c_{k}^{\mathrm{a}}|+|\tilde{c}_{k}^{\mathrm{b}}-c_{k}^{\mathrm{b}}|).$$

(9)

Following [12], we collect a large-scale synthetic dataset for CCVReID in surveillance scenarios by exploiting the highly photorealistic video game Grand Theft Auto V. The examples can be seen in Fig. 3(b). We set up 10 surveillance cameras within 5 scenes (2 cameras for each scene) to collect data. After recording the pedestrians for 48 hours in the game at 60 FPS, we use the automatic bounding boxes to crop out RGB sequences of each person. The obtained SCCVReID dataset contains 9,620 sequences from 333 identities and each identity has 2-37 suits of clothes, with an average of 7. Each sequence contains a number of frames ranging between 8 and 165 with an average length of 37. For evaluation, 167 identities with 5,768 sequences are used for training and the rest 166 identities with 3,852 sequences for testing. In the test set, we select the first sequence of each suit of each identity as query. In total, 971 sequences are used as query, and the rest 2,881 as gallery.

We also collect a real dataset named RCCVReID to support researches on CCVReID, which can be seen in Fig. 3(c). All raw videos were recorded in outdoor scenarios. For each video, we first performed ByteTrack [48] to generate human bounding boxes with unified IDs, and used them to crop out RGB sequences of each identity, which is then split into several short sequences (about 200 frames). After that, we merged the sequences of the same identity from different videos and manually labeled the clothes IDs. In total, 6,948 sequences from 34 identities are obtained. Each identity has 2-9 suits of clothes, with an average of 4. We consider two evaluation settings on this dataset. (1) Due to the small number of identities in RCCVReID, we train the models on the large-scale synthetic dataset SCCVReID and only use RCCVReID for performance evaluation. In this case, 2,133 sequences are used as query, and the rest 4,815 sequences as gallery. (2) Despite the small number of identities, the total sample size is sufficient for training models. Thus, we divide the training and test sets as existing Re-ID datasets do. Specifically, 20 identities with 5,532 sequences are reserved for training, and the remaining 14 identities are used for test. In the test set, 486 sequences are used as query, and the rest 932 sequences as gallery.

We perform experiments on three CCVReID datasets, i.e., CCVID [16], SCCVReID, and RCCVReID, to evaluate the proposed method. For evaluation, we focus on two kinds of test settings, i.e., clothes-changing (CC) setting and standard setting. In clothes-changing setting, gallery samples that have the same identity label and clothes label with query are removed, that is, only gallery samples with clothes-changing are considered. While in standard setting, both clothes-consistent and clothes-changing samples are used as gallery to calculate accuracy. Following existing person Re-ID works, we adopt the Rank-1/5/10 (R1/R5/R10) of CMC curve and the mAP (mean average precision) score as the evaluation protocols.

Since there are few works to explore CCVReID task, we compare our method with four kinds of methods for a comprehensive evaluation: 1) video-based person Re-ID methods that do not involve clothes-changing, including AP3D [17], TCLNet [22], and SINet [1], 2) gait recognition methods, including GaitSet [3], GaitPart [13], and GaitGL [29], 3) image-based clothes-changing methods, including ReIDCaps [23], Pixel_sampling [37], and GI-ReID [24], 4) video-based clothes-changing method CAL [16]. Note that image-based methods receive a single frame as input, we report the results under two different settings of such methods, i.e., randomly select a frame for feature extraction (denoted with suffix ”-R”), and extract the features of all frames and take the average as the final feature (denoted with suffix ”-A”).
Results on SCCVReID. As shown in Tab. 1, compared with image-based methods, video-based Re-ID and gait recognition methods achieve better performance, which indicates the importance of spatio-temporal information in improving the accuracy of Re-ID. Besides, our method achieves the best performance under CC setting. Compared with CAL [16], our method improves the mAP and R1 by more than 8.6% and 5.9%, respectively. However, it is observed that under standard setting, our method is lower than SINet [1] in R1, R5, and R10. Intuitively, this can be attributed to the low accuracy of the gait features. While gait information can assist in identification when appearance information is unreliable, i.e., with clothes-changing, it may also introduce certain interference when appearance information are reliable, i.e., with clothes-consistent. Despite this, our method achieve the best mAP under this setting, that is, our method allows the highest average ranking for all gallery samples with the same identity as the query.
Results on RCCVReID. We compare our method with state-of-the-art methods under two evaluation settings, i.e., use SCCVReID for training and use RCCVReID only for testing, or use RCCVReID for both training and testing. The results are shown in Tab. 2. Note that the only test setting can be considered as cross-domain CCVReID, evaluating the generalization ability of the models. Under both evaluation settings, our method achieves the best mAP and R1 under CC setting, and the best mAP under standard setting, which are consistent with the results on SCCVReID.
Results on CCVID. As seen in Tab. 3, our method outperforms all the other methods by a large margin on all evaluation protocols. Since CCVID is built based on gait recognition dataset, the extracted gait features are relatively more discriminative. Thus, the drop in R1 under standard setting caused by the interference of low-quality gait features does not exist here.

In this section, we conduct a series of ablation experiments on SCCVReID to verify each design of our method.
The effectiveness of proposed method. To verify the effectiveness of the proposed method, we compare it with different fusion framework in Tab. 4. The top 2 rows report the performance of using appearance and gait features solely, respectively. The third row report the performance when the initial appearance and gait similarity are directly summed for re-ranking. SEF [43] is the inspiration for this paper. Differently, SEF considers the appearance (gait) features only when building appearance (gait) graph, and uses GCN to directly predict the final similarity. The results show that all fusion strategies bring an improvement in accuracy under CC setting and mAP under standard setting, while a certain drop in R1 under standard setting. The main reason for that has been discussed in Sec. 5.2. However, compared to the other two fusion strategies, our method obtains the highest improvement under clothes-changing setting and the least drop of R1 under standard setting, which indicates the superiority of our method.

The effectiveness of confidence loss. To evaluate the effectiveness of the proposed confidence loss, We first list the accuracy obtained by calculating the final similarity using designed pseudo-label of confidence. As shown in Tab. 4, compared with the simple summation of similarities, our model with pseudo-label, i.e., the ground-truth of confidence, improves 33.0% on mAP and 47.0% on R1 under CC setting, 26.5% on mAP and 17.9% on R1 under standard setting. The results demonstrate the validity of the designed pseudo-label of confidence, which can be regarded as the oracle of the proposed confidence-aware method. We also compare our method with and without appearance/gait confidence loss. In Tab. 4, we can observe that removing the confidence loss of both the appearance and gait branch causes a certain degradation under both clothes-changing and standard setting. This demonstrates that the proposed loss can effectively supervise the learning of confidence.
Discussion of two-branch framework. We conduct ablation experiments to verify each branch of the proposed framework in Tab. 5. We discuss three settings of confidence: 1) the confidence are output by the network, which is represented by $\surd$. 2) the confidence is fixed to 0, that is, the original similarity is not used to calculate the final similarity, only the other branch is considered. However, in this case, the original similarity is not completely neglected, since the information has been implicitly included in the linear layer. 3) the confidence is fixed to 1, that is, the original similarity is used to calculate the final similarity without weight adjustment. From the comparison between the first and second rows, the third and fourth rows, we can see that the model with confidence set to 1 outperforms the model with confidence set to 0. That shows that the importance of the original similarity in the re-ranking stage. From the comparison between the top 4 rows and the fifth to the eighth rows, we can see that the model with confidence fixed for one branch outperforms the model with confidence fixed for both branches. Also, the model without confidence fixed achieves the best performance (the last row), which fully proves the superiority of the proposed network for confidence learning. Besides, we compare the models with and without linear layer (the last 2 rows). The results show that with addition of the linear layer, the performance of model is improved.

Discussion of candidates collection. The parameter $\gamma$ controls the contribution of initial appearance and gait ranking list when selecting training samples. As shown in Tab. 6, we observe that as the value of $\gamma$ increases, the accuracy under both the settings show a trend of first increasing and then decreasing. The model achieves the best performance when $\gamma=0.75$. This means that the optimal neighbors collection strategy is to select most of the samples based on more reliable appearance features, while samples with similar gait features to the query are also considered.