Noisy-Correspondence Learning for Text-to-Image Person Re-identification

Give an input image $I_{i}\in\mathcal{V}$, we use the visual encoder $f^{v}$ of CLIP to tokenize the image into a discrete token representation sequence with a length of $N_{\circ}+1$, i.e., $\boldsymbol{V}_{i}=f^{v}(I_{i})=\{\boldsymbol{v}^{i}_{g},\boldsymbol{v}^{i}_{1},\boldsymbol{v}^{i}_{2},\cdots,\boldsymbol{v}^{i}_{N_{\circ}}\}^{\top}\in\mathbb{R}^{(N_{\circ}+1)\times d}$, where $d$ is the dimensionality of the shared latent space. These features include an encoded feature $\boldsymbol{v}^{i}_{g}$ of the [CLS] token and patch-level local features $\{\boldsymbol{v}^{i}_{j}\}^{N_{\circ}}_{j=1}$ of $N_{\circ}$ fixed-sized non-overlapping patches of $I_{i}$, wherein $\boldsymbol{v}^{i}_{g}$ can represent the global representation. For an input text $T_{i}\in\mathcal{T}$, we apply the textual encoder $f^{t}$ of CLIP to obtain global and local representations. Specifically, following IRRA [24], we first tokenize the input text $T_{i}$ using lower-cased byte pair encoding (BPE) with a 49,152 vocab size into a token sequence. The token sequence is bracketed with [SOS] and [EOS] tokens to represent the beginning and end of the sequence. Then, we feed the token sequence into $f_{t}$ to obtain the features $\boldsymbol{T}_{i}=\{\boldsymbol{t}^{i}_{s},\boldsymbol{t}^{i}_{1},\cdots,\boldsymbol{t}^{i}_{N_{\diamond}},\boldsymbol{t}^{i}_{e}\}^{\top}\in\mathbb{R}^{(N_{\diamond}+2)\times d}$, where $\boldsymbol{t}^{i}_{s}$ and $\boldsymbol{t}^{i}_{e}$ are the features of [SOS] and [EOS] tokens and $\{\boldsymbol{v}_{j}^{i}\}^{N_{\diamond}}_{j=1}$ are the word-level local features of $N_{\diamond}$ word tokens of text $T_{i}$. Generally, the $\boldsymbol{t}^{i}_{e}$ can be regarded as the sentence-level global feature of $T_{i}$.

To measure the similarity between any image-text pair $(I_{i},T_{j})$, we can directly use the global features of [CLS] and [EOS] tokens to compute the Basic Global Embedding (BGE) similarity by the cosine similarity, i.e., $S^{b}_{ij}={{\boldsymbol{v}^{i}_{g}}{\boldsymbol{t}^{j}_{e}}^{\top}}\big{/}{\|\boldsymbol{v}^{i}_{g}\|\|\boldsymbol{t}^{j}_{e}\|}$, where the global features represent the global embedding representations of two modalities. However, optimizing the BGE similarities alone may not capture the fine-grained interactions between two modalities, which will limit performance improvement. To address this issue, we exploit the local features of informative tokens to learn more discriminative embedding representations, thus mining the fine-grained correspondences. In CLIP, the global features of the tokens ([CLS] and [EOS]) are obtained by a weighted aggregation of all local token features. These weights reflect the correlation between the global token and each local token. Following previous methods [53, 63], we could select the informative tokens based on these correlation weights to aggregate local features for a more representative global embedding.

In practice, these correlation weights can be obtained directly in the self-attention map of the last Transformer blocks of $f^{v}$ and $f^{t}$, which reflects the relevance among the input $1+N_{\circ}$ (or $2+N_{\diamond}$) tokens. Given the output self-attention map $\boldsymbol{A}^{v}_{i}\in\mathbb{R}^{(1+N_{\circ})\times(1+N_{\circ})}$ of image $I_{i}$, the correlation weights between global token and local tokens are $\{a^{v}_{i,j}\}^{N_{\circ}}_{j=1}=\boldsymbol{a}^{v}_{i}=\boldsymbol{A}^{v}_{i}[0,1:N_{\circ}+1]\in\mathbb{R}^{N_{\circ}}$. Similarly, for text $T_{i}$, the correlation weights are $\{a^{t}_{i,j}\}^{N_{\diamond}}_{j=1}=\boldsymbol{a}^{t}_{i}=\boldsymbol{A}^{t}_{i}[0,1:N_{\diamond}+1]\in\mathbb{R}^{N_{\diamond}}$, where $\boldsymbol{A}^{t}_{i}\in\mathbb{R}^{(2+N_{\diamond})\times(2+N_{\diamond})}$ is the output self-attention map for text $I_{i}$. Then, we select a proportion ($TopK$) of the corresponding token features with higher scores for embedding. Specifically, for $I_{i}$, the selected token sequences and correlation weights are reorganized as $\boldsymbol{V}^{s}_{i}=\{\boldsymbol{v}^{i}_{j}\}_{j\in\boldsymbol{K}^{v}_{i}}$ and $\hat{\boldsymbol{a}}^{v}_{i}=\{a^{v}_{i,j}\}_{j\in\boldsymbol{K}^{v}_{i}}$, where $\boldsymbol{K}^{v}_{i}\in\mathbb{R}^{\lfloor\mathcal{R}N_{\circ}\rfloor}$ is the set of indices for the selected local tokens of $I_{i}$ and $\mathcal{R}$ is the selection ratio. For text $T_{i}$, the selected token sequences and correlation weights are also reorganized as $\boldsymbol{T}^{s}_{i}=\{\boldsymbol{t}^{i}_{j}\}_{j\in\boldsymbol{K}^{t}_{i}}$ and $\hat{\boldsymbol{a}}^{t}_{i}=\{a^{t}_{i,j}\}_{j\in\boldsymbol{K}^{t}_{i}}$, where $\boldsymbol{K}^{t}_{i}\in\mathbb{R}^{\min(\lfloor\mathcal{R}N_{\diamond}^{\prime}\rfloor,N_{\diamond})}$ is the set of indices for the selected local tokens of $T_{i}$. $N_{\diamond}^{\prime}$ is the maximum input sequence length of $f^{t}$. For $I_{i}$ and $T_{i}$, we perform an embedding transformation on these selected token features to obtain subtle representations, instead of using complex fine-grained correspondence discovery used in CFine [53]. The transformation is performed by an embedding module like the residual block [20], as follows:

where $MaxPool(\cdot)$ is the max-pooling function, $MLP(\cdot)$ is a multi-layer perceptron (MLP) layer, $FC(\cdot)$ is a linear layer, $\hat{\boldsymbol{V}}^{s}_{i}=L2Norm({\boldsymbol{V}}^{s}_{i})$, and $\hat{\boldsymbol{T}}^{s}_{i}=L2Norm({\boldsymbol{T}}^{s}_{i})$. $L2Norm(\cdot)$ is the $\ell_{2}$-normalization function to normalize features. Finally, for any pair $(I_{i},T_{j})$, we compute the cosine similarity $S^{t}_{ij}$ between $\boldsymbol{v}_{tse}^{i}$ and $\boldsymbol{t}_{tse}^{j}$ as the Token Selection Embedding (TSE) similarity to measure the cross-modal matching degree for auxiliary training and inference.

To alleviate the negative impact of NC, the key is to filter the possible noisy pairs in the training data, which directly avoids false supervision information. Some previous work in learning with noisy labels [17, 26, 23] are inspired by the memorization effect [1] of DNNs to perform filtrations, i.e., the clean (easy) data tend to have a smaller loss value than that of noisy (hard) data in early training. Based on this, we can exploit the two-component Gaussian Mixture Model (GMM) to fit the per-sample loss distributions computed by the predictions of BGE and TSE to further identify the noisy pairs in the training data. Specifically, given a cross-modal model $\mathcal{M}$, we first define the per-sample loss as:

where $\mathcal{L}$ is the loss function for pair $(I_{i},T_{i})\in\mathcal{P}$ to bring them closer in the shared latent space. In our method, $\mathcal{L}$ is the proposed $\mathcal{L}_{tal}$ defined in Equation 11. Then, the per-sample loss is fed into the GMM to separate clean and noisy data, i.e., assigning the Gaussian component with a lower mean value as a clean set and the other as a noisy one, respectively. Following [26, 23], we use the Expectation-Maximization algorithm to optimize the GMM and compute the posterior probability $p(k|\ell_{i})=p(k)p(\ell_{i}|k)/p(\ell_{i})$ for the $i$-th pair as the probability of being clean/noisy pair, where $k\in\{0,1\}$ is used to indicate whether it is a clean or a noisy component. Then, we set a threshold $\delta=0.5$ to $\{p(k=0|l_{i})\}^{N}_{i=1}$ to divide the data into clean and noisy sets, i.e.,

where $\mathcal{P}^{c}$ and $\mathcal{P}^{n}$ are the divided clean and noisy sets, respectively. For BGE and TSE, the divisions conducted with Equation 13 are $\mathcal{P}=\mathcal{P}^{c}_{bge}\cup\mathcal{P}^{n}_{bge}$ and $\mathcal{P}=\mathcal{P}^{c}_{tse}\cup\mathcal{P}^{n}_{tse}$, separately.

To obtain the final reliable divisions, we propose to exploit the consistency between the two divisions to find the consensus part as the final confident clean set, i.e., $\hat{\mathcal{P}}^{c}=\mathcal{P}^{c}_{bge}\cap\mathcal{P}^{c}_{tse}$. The rest of the data can be divided into noisy and uncertain subsets, i.e., $\hat{\mathcal{P}}^{n}=\mathcal{P}^{n}_{bge}\cap\mathcal{P}^{n}_{tse}$ and $\hat{\mathcal{P}}^{u}=\mathcal{P}-(\hat{\mathcal{P}}^{c}\cup\hat{\mathcal{P}}^{n})$. Finally, we exploit the divisions to further recalibrate the correspondence labels, e.g., for $i$-th pair, the process can be expressed as:

where $Rand(X)$ is the function to randomly select an element from the collection $X$.

The Triplet Ranking Loss (TRL) is a common matching loss that is widely used in cross-modal learning, and achieves promising performance by employing the hardest negatives, e.g., image-text matching [7], video-text retrieval [9], etc. However, we find that this strategy may lead to bad local minima or even model collapse for TIReID under NC in the early stages of training. In contrast, the summation version of TRL that considers all negative samples, namely TRL-S, can maintain better stability and avoid model collapse, but suffers from insufficient performance due to the lack of attention to hard negatives (see Section 3.3.3 for more discussion). Therefore, we propose a novel Triplet Alignment Loss (TAL) to guide TIReID, which differs from TRL in that it relaxes the optimization of the hardest negatives to all negatives with an upper bound (see Lemma 1). Thanks to the relaxation, TAL reduces the risk of the optimization being dominated by the hardest negatives, thereby making the training more stable and comprehensive by considering all pairs. For an input pair $(I_{i},T_{i})$ in a mini-batch $\mathbf{x}$, TAL is defined as

where $m$ is a positive margin coefficient, $\tau$ is a temperature coefficient to control hardness, $S(I_{i},T_{j})\in\{S_{ij}^{b},S_{ij}^{t}\}$, $[x]_{+}\equiv\max(x,0)$, $\exp(x)\equiv e^{x}$, $q_{ij}=1-l_{ij}$, and $K$ is the size of $\mathbf{x}$. From Lemma 1, as $\tau\to 0$, TAL approaches TRL and focuses more on hard negatives. Since multiple positive pairs from the same identity may appear in the mini-batch, $S^{+}_{i2t}(I_{i})=\sum^{K}_{j=1}\alpha_{ij}S(I_{i},T_{j})$ is the weighted average similarity of positive pairs for image $I_{i}$, where $\alpha_{ij}=\frac{l_{ij}\exp{(S(I_{i},T_{j})/\tau)}}{\sum^{N}_{k=1}l_{ik}\exp{(S(I_{i},T_{k})/\tau)}}$. Similarly, $S^{+}_{i2t}(T_{i})$ is the weighted average similarity of positive pairs for text $T_{i}$.

To explore the behaviors of the triplet losses in the noisy case, we record the similarity distributions versus iterations of TRL, TRL-S, and the proposed TAL under 50% noise. From Figure 3(a), one can see that the similarities of all pairs are gradually gathered to 1 during training with TRL, i.e., all samples collapses to a narrow neighborhood space on a hypersphere, resulting in a trivial solution and a bad performance (3.64%).

To delve deeper into the underlying reason, we performed a gradient analysis. For ease of representation and analysis, we only consider one direction since image-to-text retrieval and text-to-image retrieval are symmetrical. And, we suppose that there is only one paired text for each image in the mini-batch. Due to the truncation operation $[x]_{+}$, we only discuss the case of $\mathcal{L}>0$ that could generate gradients. Taking the image-to-text direction as an example, the gradients generated by TRL, TRL-S, and TAL are

where $\mathcal{Z}=\{z\ |\ \big{[}m-S(I_{i},T_{i})+S(I_{i},T_{z})\big{]}_{+}>0,z\neq i,z\in\{0,\cdots,K\}\}$, $\beta_{j}=\frac{\exp(\boldsymbol{v}_{i}^{\top}\boldsymbol{t}_{j}/\tau)}{\sum^{K}_{k\neq i}\exp(\boldsymbol{v}_{i}^{\top}\boldsymbol{t}_{k}/\tau)}$, $\hat{\boldsymbol{t}}_{i}$, $\boldsymbol{t}_{j}$ and $\boldsymbol{t}_{i}$ are the hardest negative sample, negative sample, and positive sample of the anchor sample $\boldsymbol{v}_{i}$, respectively. Since the hardest sample is most similar to the positive one, $\frac{\partial_{trl}}{\partial\boldsymbol{v}_{i}}$ would easily approach $\boldsymbol{0}$ and the gradients for other negative samples except for the hardest negative one are all $\boldsymbol{0}$, which may lead to bad local minima early on in training and even cause the worst-case scenario, i.e., model collapse (see Figure 3(a)). Unlike TRL, TRL-S aims to push all negative samples away from the anchor by a constant margin and produces stronger gradients for the anchor, i.e., $\|\frac{\partial_{trls}}{\partial\boldsymbol{v}_{i}}\|_{2}\geq\|\frac{\partial_{trl}}{\partial\boldsymbol{v}_{i}}\|_{2}$, thus avoiding model collapse (see Figure 3(b)). However, the drawback is that TRL-S treats every negative sample equally while ignoring challenging ones, which limits performance improvement. Different from TRL and TRL-S, from Equation 22, our TAL can comprehensively consider all negative samples and exploits the anchor-negative semantic relationships to adaptively adjust the gradients for each negative, thus paying more attention to hard negatives. As a result, TAL would avoid model collapse under NC while achieving superior performance (63.35% vs. 44.93% vs. 3.64%). More details for the derivations of gradients are provided in the supplementary material.

To train the model robustly, we use the corrected label $\hat{l}_{ii}$ instead of the original correspondence label $l_{ii}$ to compute the final matching loss, i.e.,

where $\mathcal{L}^{b}(I_{i},T_{i})$ and $\mathcal{L}^{t}(I_{i},T_{i})$ are the TAL losses computed by Equation 11 with BGE and TSE similarities, respectively. The training process of RDE is shown in Algorithm 1. For the joint inference, we compute the final similarity of the image-text pair as the average of the similarities computed by both embedding modules, i.e., $S=(S^{b}+S^{t})/2$.

In the experiments, we use the CHUK-PEDES [27], ICFG-PEDES [8], and RSTPReid [62] datasets to evaluate our RDE. We follow the data partitions used in IRRA [24] to split the datasets into training, validation, and test sets, wherein the ICFG-PEDES dataset only has training and validation sets. More details are provided in the supplementary material.

For all experiments, we mainly employ the popular Rank-K metrics (K=1,5,10) to measure the retrieval performance. In addition to Rank-K, we also adopt the mean Average Precision (mAP) and mean Inverse Negative Penalty (mINP) as auxiliary retrieval metrics to further evaluate performance following [24].

As mentioned earlier, we adopt the pre-trained model CLIP [40] as our modality-specific encoders. In fairness, we use the same version of CLIP-ViTB/16 as IRRA [24] to conduct experiments. During training, we introduce data augmentations to increase the diversity of the training data. Specifically, we utilize random horizontal flipping, random crop with padding, and random erasing to augment the training images. For training texts, we employ random masking, replacement, and removal for the word tokens as the data augmentation. Moreover, the input size of images is $384\times 128$ and the maximum length of input word tokens is set to 77. We employ the Adam optimizer to train our model for 60 epochs with a cosine learning rate decay strategy. The initial learning rate is $1e-5$ for the original model parameters of CLIP and the initial one for the network parameters of TSE is initialized to $1e-3$. The batch size is 64. Following IRRA [24], we adopt an early training process with a gradually increasing learning rate. For hyperparameter settings, the margin value $m$ of TAL is set to 0.1, the temperature parameter $\tau$ is set to 0.015, and the selection ratio $\mathcal{R}$ is 0.3.