Meta Clustering Learning for Large-scale Unsupervised Person Re-identification

Unsupervised Domain Adaptive (UDA) ReID. This branch usually utilizes transfer learning, e.g., style translation (Zhu et al., 2017), to reduce domain gap between source and target ReID scenarios for adaptation (Wei et al., 2018; Deng et al., 2018; Liu et al., 2019; Dai et al., 2022; Zhang et al., 2022). Their performance is typically inferior to the clustering-based approaches, since there is still a gap between the style-translated images and the realistic person images (Yang et al., 2020; Wang et al., 2020a).

Clustering-based Unsupervised ReID. This branch typically trains ReID model directly on unlabeled dataset with a clustering-based pseudo label estimation (Song et al., 2020; Fan et al., 2018; Yang et al., 2019; Zhang et al., 2019; Fu et al., 2019a; Yang et al., 2020; Zheng et al., 2022). This clustering labeling and training process are usually alternatively performed until the model is stable. In particular, Lin et al.  (Lin et al., 2019) treat each individual sample as a cluster, and then gradually group similar samples into one cluster to generate pseudo labels. Jin et al.  (Jin et al., 2020a) introduce a global distance-distribution separation constraint to handle the sample-wise noisy label. SPCL (Ge et al., 2020b) proposes a self-paced contrastive learning framework to gradually create more reliable clusters for ReID training while updating the hybrid memory containing both source and target domain features. Similarly, ClusterContrast (Dai et al., 2021) further stores feature vectors inside a cluster-level memory to alleviate the inconsistent clustering issue. Recently, Isobe et al.  (Isobe et al., 2021) introduce cluster-wise contrastive learning (CCL), progressive domain adaptation (PDA), Fourier augmentation (FA), and ICE (Chen et al., 2021) introduces inter-instance contrastive encoding to boost the existing class-level contrastive ReID methods. However, all these methods focus on how to get more reliable pseudo labels or how to better leverage them for discriminative feature learning, an important point of computational cost is still under-explored. Besides, the noisy pseudo label issue in these methods has not yet been well addressed.

Reference-based Pseudo Labeling in ReID. Existing representative works, like MAR (Yu et al., 2019), MMCL (Wang and Zhang, 2020), MPRD (Ji et al., 2021), and SSL (Lin et al., 2020) either use labeled source data for pseudo labels generation or assign each unlabeled person image with a multi-class/softened label via pairwise similarity computation. Differently, our MCL creates clustering-free soft pseudo labels with the reference of online updated meta-prototypes that stored in the memory. Such design is more efficient because it does not need source labeled dataset as reference, and meta-prototypes (like a FC layer) could help directly infer out real-valued labels instead of repeat pairwise comparisons. Moreover, more reliable meta-prototypes encourage more accurate pseudo labeling (more effective unsupervised training), and vice versa. They promote each other, achieving a win-win effect.

Feature Extraction and Clustering. As shown in Figure 2, a network $f_{\theta}$ (e.g., ResNet-50 (He et al., 2016), initialized with pre-trained weights on ImageNet (Ge et al., 2020a, b; Dai et al., 2021; Isobe et al., 2021)) is taken as backbone to extract features from ${\mathbb{X}}_{1}$. Then, DBScan (Ester et al., 1996) is used to cluster these features (unclustered outliers are discarded (Chen et al., 2021; Dai et al., 2021)). The IDs of cluster results are assigned to unlabeled samples as the pseudo labels for training.

Query Setup and Meta-prototype Initialization. After obtaining clustered pseudo labels for ${\mathbb{X}}_{1}$, we sample $P$ person identities and $I$ instances for each identity, to set up a mini batch with the size of $P\times I$. Different from works (Song et al., 2020; Jin et al., 2020a) that directly use the instance-wise loss contraints (e.g., triplet loss (Hermans et al., 2017)) for training, we take each batch as a query set and employ a memory dictionary based contrastive learning (Ge et al., 2020b; Dai et al., 2021; Isobe et al., 2021) for optimization.

Meta-prototypes Update and Model Optimization. At each iteration $t$ of epoch, the encoded feature vectors $\{{\bm{q}}\}$ of $P\times I$ query images in each mini-batch would be involved in meta-prototypes update. With the momentum updating (He et al., 2020), the $k$-th cluster prototype ${\bm{w}}_{k}$ is updated by the mean of encoded query features belonging to class $k$,

where ${\mathbb{B}}_{k}^{t}$ denotes the feature vector set belonging to class $k$ in the mini-batch at the $t$-th iteration, and $m\in[0,1]$ is a momentum coefficient, which is empirically set as $0.2$ following (Ge et al., 2020b; Dai et al., 2021). The learned meta-prototypes are taken for model optimization together with query samples in this phase, and also play a role of proxy annotator (see the ‘robot’ in Figure 2,3) for the rest unlabeled subsets ${\mathbb{X}}_{2},{\mathbb{X}}_{3},...,{\mathbb{X}}_{N}$ in the next phase.

Prototype-referenced Labeling. For clarity, we take an unused and unlabeled subset ${\mathbb{X}}_{2}$ as example for illustration. Given ${\mathbb{X}}_{2}=\{x_{i}\}_{i=0}^{N_{u}}$ where each $x_{i}$ is a collected unlabeled person image, the learned meta-prototypes $\{{\bm{w}}_{1},\cdots,{\bm{w}}_{K}\}$ defines the pseudo label space for ${\mathbb{X}}_{2}$ is $[1,K]$. As shown in the ‘robot’ in Figure 3, the meta-prototypes set $\{{\bm{w}}_{k}\}_{k=1}^{K}$ is taken as proxy annotator, a soft real-valued pseudo label ${\bm{y}}_{i}$ can be assigned for $x_{i}$ by comparing $f(x_{i})$ with the reference agents $\{{\bm{w}}_{k}\}_{k=1}^{K}$. This soft prototype-referenced pseudo labeling process is,

where $L(\cdot)$ means a soft pseudo labeling function. This function is epoch-wise and acts like a dimension-variable FC layer (i.e., dot-product). $y_{i}^{j}\in(0,1)$ is the $j$-th entry of ${\bm{y}}_{i}$. All dimensions of ${\bm{y}}_{i}$ add up to 1 and each dimension represents the label likelihood w.r.t. a reference prototype person ID. Different from the vanilla reference-based pseudo labeling (Yu et al., 2019) or using a global classifier for labeling, our prototype-referenced labeling allows the epoch-wise and dimension-variable proxy annotator $\{{\bm{w}}_{k}\}_{k=1}^{K}$ to be updated on the fly. More reliable meta-prototypes encourage more accurate labeling and thus more effective optimization, and vice versa. Besides, the clustering results are different ($K$ is changing) for each epoch in ReID, an immutable global classifier is infeasible.

Polish the Model with Soft Pseudo Labels. An intuitive solution is to ‘harden’ the obtained soft pseudo labels, i.e., regard the dimension with the largest value as person ID, i.e., $J=\mathop{\arg\max}\ \ \{y^{j}\}_{j=1}^{K}$. Based on these identity labels $J$s, we can construct ReID constraints for training, such as cross-entropy loss (Sun et al., 2018; Fu et al., 2019b) and triplet loss (Hermans et al., 2017). In fact, our early attempts along this line have failed to deliver very good results. We analyze that is because the reference-based pseudo labeling itself will inevitably introduce some noisy labels. To exploit the merits of the prototype-referenced labeling while alleviating the noisy label effect, we additionally introduce two well-designed loss constraints to better leverage these unsatisfactory soft pseudo labels for optimization, by considering the intra-identity consistency between different views within each person identity and the inter-identity correlations among different persons. They are shown in the two black boxes in Figure 3 and elaborated below:

Siamese consistency Loss $\mathcal{L}_{sc}$ is the first constraint to promise consistency within the same identity. As shown in the upper black box in Figure 3, $\mathcal{L}_{sc}$ is built on the “swapped” prediction idea of SwAV (Caron et al., 2020) to predict the label of a view from the representation of another view. Given two features ${\bm{f}}_{s}$ and ${\bm{f}}_{t}$ extracted from two different augmentations of the same image, we compute their referenced pseudo labels ${\bm{y}}_{s}$ and ${\bm{y}}_{t}$ following Eq.(3) by matching these features to the learned meta-prototypes $\{{\bm{w}}_{k}\}_{k=1}^{K}$:

where $\mathcal{L}_{ce}({\bm{f}}_{s},{\bm{y}}_{t})$ measures the fit between features ${\bm{f}}_{s}$ and soft pseudo label ${\bm{y}}_{t}$. Intuitively, if these two features capture the same person information, they should be able to predict from each other. $\mathcal{L}_{ce}$ is the cross entropy loss between the label and the probability obtained by taking a softmax of the dot products of ${\bm{f}}$ and all prototypes in $\{{\bm{w}}_{k}\}_{k=1}^{K}$.

Soft-weighted Triplet Loss $\mathcal{L}_{tri}^{sw}$ is the second constraint which softly leverages the relative correlations between identities to construct weighted-triplets for optimization. Considering the soft pseudo labels are continuous real-valued, a soft-weighted triplet loss $\mathcal{L}_{tri}^{sw}$ is enforced to promise the correct relative correlation among person identities. Let $\{x^{a},x^{p},x^{n}\}$ be an input triplet sample and the corresponding feature embeddings are $\{f(x^{a}),f(x^{p}),f(x^{n})\}$, the soft-weighted triplet loss is given by,

Datasets and Evaluation. We evaluate the proposed MCL method on multiple ReID benchmarks (from small to large scale): PersonX (PX) (Sun and Zheng, 2019), Market1501 (Ma) (Zheng et al., 2015), MSMT17 (MT) (Wei et al., 2018), and the largest public ReID dataset (so far) LaST (LS) (Shu et al., 2021). To further show the superiority of MCL in the large-scale data setting, we also conduct experiments on the mixed datasets, e.g., training on multiple datasets PX+Ma+MT+LS while testing on unseen test set of MT. The details about datasets are shown in Table 1.

Implementation Details. The proposed MCL is generic and can be applied to different clustering-based U-ReID backbones. Here, we re-implement ClusterContrast (Dai et al., 2021) as baseline, since it has been dominating the leaderboard in multiple benchmarks w.r.t unsupervised ReID performance, and is considerably more efficient as a source-free purely unsupervised ReID pipeline compared to those competitive adaptive (source data needed) U-ReID algorithms, like MMT (Ge et al., 2020a), SpCL (Ge et al., 2020b),  (Zheng et al., 2021) etc. ResNet-50 (He et al., 2016) is adopted as the backbone of the feature extractor and initialize the model with the parameters pre-trained on ImageNet (Deng et al., 2009).

Memory&Time Cost v.s. U-ReID Performance. Table 7 shows the U-ReID performance resulted by using subsets of size 50%, 33%, 25%, and 20% randomly selected from all unlabeled data as meta-training set ${\mathbb{X}}_{1}$ vs. directly conducting clustering over full data (All). We observe that using partial data for clustering with MCL effectively saves computational costs on both memory and time. For example, the MCL, 50% schemes nearly achieve the same ReID performance but save memory&time cost by over 50%, such savings are particularly obvious on the large datasets: 1761.3MB, 31.1s vs 6251.5MB, 118.3s on MSMT17 and 5779.6MB, 121.2s vs 22398.5MB, 494.8s on LaST. However, we also observe that the ReID performance of mAP/Rank1 has a noticeable drop, especially on the small dataset PersonX (‘blue’ in Table 7). We analyze that’s because the noisy label issue will be enlarged on small datasets.

Necessity of MCL. Someone may think of directly splitting a large-scale dataset into multiple small subsets to do clustering-based U-ReID sequentially. This is also the most straightforward solution to handle the computational issue we focused. To study its feasibility, we deliberately design a scheme named Naive Splitting Training, where multiple subsets picked from one single large data set are sequentially used for clustering$\rightarrow$labeling$\rightarrow$training. Naive Splitting Training also could save memory cost due to its subset-wise clustering, but this operation also inadvertently enlarges the negative effect of time consuming and noisy labeling. As shown in Table 3, two Naive Splitting Training schemes of using 50%/25% subset as training unit, are inferior to MCL by 16.0%/9.7% in mAP on MSMT17, which reveals two facts that 1) naively splitting the holistic large-scale dataset for sequential training is not optimal, and 2) MCL is necessary and more superior.

Influence of Loss Constraints. We study the effectiveness of the proposed siamese consistency loss $\mathcal{L}_{sc}$ and soft-weighted triplet loss $\mathcal{L}_{tri}^{sw}$ in Table LABEL:tab:loss. We see that MCL outperforms MCL w/o $\mathcal{L}_{sc}$ by 4.0%/3.2% in mAP for 50%/25% settings on Market1501. When replace the soft-weighted triplet loss $\mathcal{L}_{tri}^{sw}$ with the basic triplet loss version (Hermans et al., 2017), the scheme of MCL w/o $\mathcal{L}_{tri}^{sw}$ is inferior to MCL by 5.6%/4.4% in mAP for 50%/25% settings on Market1501. Such two constraints facilitate the pseudo label denoising via promising intra-identity consistency and inter-identity correlation. In addition, they are complementary and both vital to MCL, jointly resulting in a superior performance.

Influence of Data Split. As we described before Sec. 3.1, given an unlabeled dataset ${\mathbb{X}}$, we use an random and uniform split strategy to divide the samples into meta training subset ${\mathbb{X}}_{1}$ and the rest subsets $\{{\mathbb{X}}_{2},{\mathbb{X}}_{3},...,{\mathbb{X}}_{N}\}$. Such split is performed before each training epoch. And, the label spaces for different subsets $\{{\mathbb{X}}_{1},{\mathbb{X}}_{2},{\mathbb{X}}_{3},...,{\mathbb{X}}_{N}\}$ are same-size but non-overlapping. Here we study on the influence of different split designs. In Table LABEL:tab:data_split, the scheme of MCL_fixed means we only conduct the data split once at beginning and fix the split results during the training. MCL_same means the scheme where all subsets share the same label space. We can observe that MCL_fixed is inferior to MCL by 11.4%/37.6% in mAP under 50%/25% on Market1501, and MCL_same is inferior to MCL by 15.6%/11.5% in mAP under 25% on Market1501/MSMT17. We analyze that: 1). re-splitting dataset before each epoch plays a role of data re-organization, like the mechanism behind cross-validation (Kohavi et al., 1995), which avoids over-fitting and extremely cases, increasing robustness of MCL. 2). non-overlapped label spaces increase the diversity of training data, like a data augmentation, promoting the discriminative ReID representations learning. Such design brings obvious improvements especially when using less meta-training data. For example, MCL outperforms MCL_same by only 2.1%/1.7% in mAP under 50%, but by 15.6%/11.5% under 25%. More analytic and ablated results (including limitation discussions) are presented in Supplementary.

Visualization on Pseudo Labeling. To further show the proposed prototype-referenced labeling in MCL is superior to the general clustering, we compare these two pseudo-labeling methods by showing the same pseudo-labeled images (i.e., the positive pairs) in Figure 4. Our scheme is MCL, 50%. We observe that: 1). for the general clustering, the grouped entries share the global visual similar appearance. This is not reliable enough. For example, in the most left pair of Figure 4, the two women are dressed very similarly, the only local discriminative clue is they take different items in their hands. 2). the proposed meta-prototype referenced labeling has the capability of discovering fine-grained discriminative clues (bottom in Figure 4) due to the usage of relative comparative characteristic among samples. This also explains why MCL outperforms the baseline scheme of All even with less data for clustering to some extent.

Visualization of Feature Distributions. In Figure 5 (b), we visualize the distributions of the features using t-SNE (Van der Maaten and Hinton, 2008) on MSMT17. We compare the feature distribution with the baseline scheme of All, and observe that the features of different identities are better clearly separated for our scheme MCL, 50%, which demonstrates our learned ReID representations are more discriminative.

Influence of Clustering Hyper-parameters. As discussed in implementation, we use DBScan and Jaccard distance (Zhong et al., 2017) for first-phase training to cluster with $k$ nearest neighbors ($k$=30) following (Ge et al., 2020b; Dai et al., 2021). For DBScan, the maximum distance $d$ between two samples is set as 0.4 for market1501, 0.7 for other datasets, and the minimal number of neighbors in a core point (denoted as $n$) is all set as 4. Here we analyze the influence of these parameters in Figure 7, and conclude that the proposed large-scale unsupervised ReID training method of MCL is robust and stable enough to achieve relatively satisfactory performance with variant hyper-parameters.