Learning by Aligning:
Visible-Infrared Person Re-identification using Cross-Modal Correspondences

We use a two-stream CNN to extract feature maps of size $h\times w\times d$ from a pair of RGB/IR person images, where $h$, $w$ and $d$ are the height, width and number of channels, respectively. Assuming that the cross-modal discrepancies between RGB/IR images mainly lie in low-level features [36, 40], we use separate parameters specific to the input modalities for shallow layers, while sharing the remaining ones for others.

The CMAlign module aligns RGB and IR features bidirectionally, i.e., from RGB to IR and from IR to RGB, using dense cross-modal correspondences in a probabilistic way. In the following, we describe an IR-to-RGB alignment. The other case can be performed similarly.

For the IR-to-RGB alignment, we compute local similarities between all pairs of RGB and IR features. Concretely, we compute cosine similarities between RGB and IR features, denoted by ${\bf{f}}_{\mathrm{RGB}}\in\mathbb{R}^{h\times w\times d}$ and ${\bf{f}}_{\mathrm{IR}}\in\mathbb{R}^{h\times w\times d}$, respectively, as follows:

$$C({\bf{p}},{\bf{q}})=\frac{{{\bf{f}}_{\mathrm{RGB}}(\bf{p})}^{\top}{{\bf{f}}_{\mathrm{IR}}(\bf{q})}}{\|{\bf{f}}_{\mathrm{RGB}}({\bf{p}})\|_{2}\|{\bf{f}}_{\mathrm{IR}}({\bf{q}})\|_{2}},$$

(1)

where $\|\cdot\|_{2}$ computes the L2 norm of a vector. We denote by ${\bf{f}}_{\mathrm{RGB}}(\bf{p})$ and ${\bf{f}}_{\mathrm{IR}}(\bf{q})$ RGB and IR features of size $d$ at position $\bf{p}$ and $\bf{q}$, respectively. Based on the similarities, we compute RGB-to-IR matching probabilities using a softmax function as follows:

$$P({\bf{p}},{\bf{q}})=\frac{\exp({\beta C({\bf{p}},{\bf{q}})})}{\sum_{{\bf{q}}^{\prime}}\exp(\beta C({\bf{p}},{\bf{q}}^{\prime}))},\vspace{-1mm}$$

(2)

where we denote by $P$ a matching probability, a 4D tensor of size $h\times w\times h\times w$, and $\beta$ is a temperature parameter. Note that we can establish dense correspondences explicitly from RGB to IR images, by applying an argmax operator to the matching probabilities for each RGB feature, i.e., $\operatorname*{argmax}_{{\bf{q}}}P({\bf{p}},{\bf{q}})$. This offers reliable cross-modal correspondences for semantically similar regions, but aligning IR and RGB features using the hard correspondences is problematic. The correspondences are easily distracted by background clutter and image-specific details (e.g., texture and occlusion), and appearance variations between RGB and IR images are even more significant. Moreover, we could not establish correspondences between different background regions, e.g., from person images captured with different surrounding environments. To alleviate these problems, we instead exploit the matching probabilities, and align IR and RGB features between foreground regions only, typically correspond to persons, via soft warping as follows:

		$\displaystyle\hat{\bf{f}}_{\mathrm{RGB}}(\mathbf{p})=$		(3)
		$\displaystyle M_{\mathrm{RGB}}(\mathbf{p})\mathcal{W}({\bf{f}}_{\mathrm{IR}}(\mathbf{p}))+(1-M_{\mathrm{RGB}}(\mathbf{p})){\bf{f}}_{\mathrm{RGB}}(\mathbf{p}),\vspace{-1.5mm}$

where we denote by $\hat{\bf{f}}_{\mathrm{RGB}}\in\mathbb{R}^{h\times w\times d}$ and $M_{\mathrm{RGB}}\in\mathbb{R}^{h\times w}$ a reconstructed RGB feature by the IR-to-RGB alignment and a person mask, respectively. We denote by $\mathcal{W}$ a soft warping operator that aggregates features using the matching probabilities, defined as follows:

$$\vspace{-1mm}\mathcal{W}({\bf{f}}_{\mathrm{IR}}(\mathbf{p}))=\sum_{{\bf{q}}}P(\mathbf{p},\mathbf{q}){\bf{f}}_{\mathrm{IR}}(\mathbf{q}).\vspace{-1.5mm}$$

(4)

The person mask ensures that the features $\hat{\bf{f}}_{\mathrm{RGB}}$, for person regions are reconstructed by aggregating IR features in a probabilistic way, while others come from original RGB features ${\bf{f}}_{\mathrm{RGB}}$. This reconstruction together with ID consistency and dense triplet losses encourages our model to provide similar person representations, regardless of image modalities, for the corresponding regions. To infer the mask without ground-truth labels, we assume that features, learned with ID labels for the reID task, are highly activated on person regions than other parts, and compute an activation map based on L2 norms of the local feature vectors, denoted by ${\bf{g}}_{\mathrm{RGB}}\in\mathbb{R}^{h\times w}$ for an RGB feature, as follows:

$${\bf{g}}_{\mathrm{RGB}}(\mathbf{p})=\|{\bf{f}}_{\mathrm{RGB}}(\mathbf{p})\|_{2}.$$

(5)

With the activation map of an RGB feature, $\bf{g}_{\mathrm{RGB}}$, at hand, we define a person mask for an RGB feature as follows:

$$M_{\mathrm{RGB}}=f(\bf{g_{\mathrm{RGB}}}),\vspace{-1mm}$$

(6)

where $f$ performs min-max normalization:

$$f(\bf{x})=\frac{x-\min(x)}{\max(x)-\min(x)}.\vspace{-1mm}$$

(7)

The CMAlign, which is a non-parametric module that operates directly on the features obtained from the feature extractor, facilitates learning robust person representations by providing the following advantages in VI-reID: First, a cross-modal alignment helps to alleviate the discrepancies between RGB and IR images in a pixel-level, allowing to suppress modality-related features from person representations more effectively, even with misaligned person images; Second, a dense alignment allows our network to focus on learning local features, especially for person regions, further enhancing the discriminative power of person representations. Note that a pair of RGB and IR images does not have to be of the same identity in our framework, enabling exploiting both positive and negative pairs for training.

As an ID loss, we adopt a sum of classification and hard triplet losses [12] using image-level person representations, which have shown the effectiveness on learning discriminative person features in single-modality person reID. We denote by $\phi(\mathbf{f}_{\mathrm{RGB}})\in\mathbb{R}^{d}$ and $\phi(\mathbf{f}_{\mathrm{IR}})\in\mathbb{R}^{d}$ image-level person representations for the RGB and IR features, respectively, which are obtained by applying a GeM pooling operation [26] to each feature. To compute the classification term, we feed each image-level feature, $\phi(\mathbf{f}_{\mathrm{RGB}})$ and $\phi(\mathbf{f}_{\mathrm{IR}})$, into a same classifier to predict class probabilities, that is, likelihoods of being particular identities for the image-level feature, where the classifier consists of a Batch Normalization layer [14], followed by a fully-connected layer with a softmax activation [22]. We then compute a cross-entropy between the class probabilities and ground-truth identities. The hard triplet term is also computed using image-level person representations, obtained from anchor, positive, and negative images, where the anchor and positive ones share the same ID label, while other pairs do not. Note that the ID loss does not address the cross-modal discrepancies between RGB and IR images explicitly.

We design a term to consider the cross-modal discrepancies between RGB and IR features in an image-level. Suppose that we have a positive pair with the same identity but having different modalities, i.e., RGB and IR images are of the same identity. The features $\hat{{\bf{f}}}_{\mathrm{RGB}}$ for person regions are reconstructed by aggregating IR features ${{\bf{f}}}_{\mathrm{IR}}$, suggesting that the identity of the reconstruction $\hat{{\bf{f}}}_{\mathrm{RGB}}$ should be the same as ground-truth identity for the original features ${{\bf{f}}}_{\mathrm{RGB}}$ and ${{\bf{f}}}_{\mathrm{IR}}$. More specifically, image-level representations of $\phi(\hat{{\bf{f}}}_{\mathrm{RGB}})$ and $\phi(\hat{{\bf{f}}}_{\mathrm{IR}})$ should have the same ID labels as corresponding positive counterparts of different modalities, ${{\bf{f}}}_{\mathrm{IR}}$ and ${{\bf{f}}}_{\mathrm{RGB}}$, respectively. To implement this idea, we define an ID consistency loss as a cross-entropy using image-level representations, similar to the classification term in the ID loss. We instead exploit reconstructed features, $\phi(\hat{{\bf{f}}}_{\mathrm{RGB}})$ and $\phi(\hat{{\bf{f}}}_{\mathrm{IR}})$. Note that we use the same classifier as in the ID loss. The ID consistency loss enforces ID predictions from person images of the same identity but with different modalities to be consistent, allowing to suppress modality-related features from person representations. Moreover, the reconstructions, $\phi(\hat{{\bf{f}}}_{\mathrm{RGB}})$ and $\phi(\hat{{\bf{f}}}_{\mathrm{IR}})$ provide an effect of offering additional samples to train the classifier, further guiding the discriminative person representation learning.

The ID loss facilitates learning discriminative person representations, and the ID consistency term alleviates the cross-modal discrepancies explicitly. They, however, focus on learning image-level person representations, which prohibits discriminative feature learning, especially when the person images are occluded or misaligned. To address this problem, we introduce a dense triplet loss. It locally compares original features and reconstructed ones using the features of different modalities, encouraging final image-level person representations to be discriminative, while alleviating the cross-modal discrepancies in a pixel-level. A straight-forward approach is to compute L2 distances between local features, which is, however, suboptimal in that this does not take occluded regions into consideration. This is particularly problematic when each of person images in a pair depicts disassociated human parts. Enforcing local alignments between the entire person regions in this case is infeasible, and maybe even harmful. To circumvent this issue, we incorporate a co-attention map highlighting person regions visible in both RGB and IR images. This considers feature alignments within mutually visible foreground regions only to compute the dense triplet loss. We define a co-attention map, denoted by $A_{\mathrm{RGB}}\in\mathbb{R}^{h\times w}$ for an RGB image, as follows:

$$A_{\mathrm{RGB}}(\mathbf{p})=M_{\mathrm{RGB}}(\mathbf{p})\mathcal{W}(M_{\mathrm{IR}}(\mathbf{p})).$$

(9)

For $\mathcal{W}(M_{\mathrm{IR}}(\mathbf{p}))$, in this case, we compute the matching probabilities $P$ between $\bf{f}_{\mathrm{RGB}}$ and $\bf{f}_{\mathrm{IR}}$, similar to (4), whereas the person masks are exploited for soft warping. Namely, the co-attention map $A_{\mathrm{RGB}}$ is an intersection of the RGB person mask $M_{\mathrm{RGB}}(\mathbf{p})$ and the warped IR one, w.r.t the RGB image, $\mathcal{W}(M_{\mathrm{IR}}(\mathbf{p}))$. We compute a co-attention map for an IR image similarly, and show in Fig. 3 an example of a co-attention map. Note that we define co-attention maps between positive pairs of the same identity only. Note also that we perform min-max normalization $f$ on the obtained co-attention map, which we omit for notational brevity.

To facilitate training with the dense triplet term, we sample a triplet of anchor, positive, and negative images, where the anchor and other two images have different modalities e.g., an RGB image for the anchor, and IR images for a pair of positive and negative samples. We use the superscripts, $\mathrm{a}$, $\mathrm{p}$, and $\mathrm{n}$ to indicate features from anchor, positive, and negative images, respectively. For example, we denote by $\hat{{\bf{f}}}_{\mathrm{RGB}}^{\mathrm{p}}$ a reconstructed RGB feature using an anchor ${\bf{f}}_{\mathrm{RGB}}^{\mathrm{a}}$ and a positive pair ${\bf{f}}_{\mathrm{IR}}^{\mathrm{p}}$ with the same identity as the anchor ${\bf{f}}_{\mathrm{RGB}}^{\mathrm{a}}$. Similarly, $\hat{{\bf{f}}}_{\mathrm{IR}}^{\mathrm{n}}$ is a reconstructed IR feature using an anchor ${\bf{f}}_{\mathrm{IR}}^{\mathrm{a}}$ and a negative pair ${\bf{f}}_{\mathrm{RGB}}^{\mathrm{n}}$ with the different identity from the anchor ${\bf{f}}_{\mathrm{IR}}^{\mathrm{a}}$. With co-attention maps at hand, we define the dense triplet loss as follows:

$$\vspace{-1mm}\mathcal{L}_{\mathrm{DT}}=\sum_{i\in\{\mathrm{RGB},\mathrm{IR}\}}\sum_{\mathbf{p}}A_{i}(\mathbf{p})[d_{i}^{+}(\mathbf{p})-d_{i}^{-}(\mathbf{p})+\alpha]_{+},$$

(10)

where $\alpha$ is a pre-defined margin and the operation $[\cdot]_{+}$ indicates $\max(0,\cdot)$. $d^{+}_{i}(\mathbf{p})$ and $d^{-}_{i}(\mathbf{p})$ compute local distances between an anchor feature and reconstructed ones from positive and negative images, respectively, as follows:

$$d^{+}_{i}(\mathbf{p})=\|\mathbf{f}_{i}^{\mathrm{a}}(\mathbf{p})-\hat{\mathbf{f}}_{i}^{\mathrm{p}}(\mathbf{p})\|_{2},d^{-}_{i}(\mathbf{p})=\|\mathbf{f}_{i}^{\mathrm{a}}(\mathbf{p})-\hat{\mathbf{f}}_{i}^{\mathrm{n}}(\mathbf{p})\|_{2}.$$

(11)

Note that the reconstructions, $\hat{\mathbf{f}}_{i}^{\mathrm{p}}$ and $\hat{\mathbf{f}}_{i}^{\mathrm{n}}$, are the aggregations of similar features w.r.t the anchor $\mathbf{f}_{i}^{\mathrm{a}}$ from positive and negative images, respectively. We can thus interpret that our loss enforces an aggregation of similar features from negative images to be distant in the embedding space, compared to its positive counterpart by a margin. This is similar to the typical triplet loss [12, 30], but ours penalizes incorrect distances for all local features visible in both anchor and positive images in a soft manner. Note that this local association is possible due to the CMAlign module that performs the dense cross-modal alignment between RGB and IR person images in a probabilistic way.

We use two benchmarks for evaluation: 1) The RegDB dataset [24] contains 412 persons, where each person has 10 visible and 10 far-infrared images collected by dual camera systems. Following the experimental protocol in [24], we divide the dataset into training and test splits randomly, each of which includes non-overlapping 206 identities. We test our model in both visible-to-IR and IR-to-visible settings, which correspond to retrieving IR images from RGB ones and RGB images from IR ones, respectively, and report the results averaged over 10 trials with different training/test splits. 2) SYSU-MM01 [36] is a large-scale dataset for VI-reID, consisting of RGB and IR images obtained by four visible and two near-infrared sensors, respectively. Concretely, it contains 22,258 visible and 11,909 near-infrared images with 395 identities for training. The test set contains 96 identities with 3,803 near-infrared images for a query set and 301 visible images for a gallery set. We adopt the evaluation protocol in [36], which uses all-search and indoor-search modes for testing, where the gallery sets for the former and the latter contain images captured by all four and two indoor visible cameras, respectively. Note that all our results are obtained by taking an average value over $4$ training and test runs.

Following the previous VI-reID methods [39, 21, 3], we adopt ResNet50 [10], trained for ImageNet classification [5], as our backbone network. The backbone networks for visible and infrared images share the parameters, except for the first residual blocks that take images of different modalities, and the stride of the last convolutional block is set to 1. We resize each person image to the size of 288 $\times$ 144, and apply horizontal flipping for data augmentation. We set the size of a person representation $d$ to 2,048. For a mini-batch, we randomly choose 8 identities from each modality and sample 4 person images for each identity. We train our model for 80 epochs with a batch size of 64, using the SGD optimizer with momentum of 0.9 and weight decay of 5e-4. We use a warm-up strategy [22], gradually raising learning rates for the backbones and other parts of the network up to 1e-2 and 1e-1, respectively, which are then decayed by a factor of 10 at the 20th and 50th epochs. We use a grid search to set hyper-parameters: $\lambda_{\text{IC}}=1$, $\lambda_{\text{DT}}=0.5$, $\alpha=0.3$, $\beta=50$. Note that we employ BNN trick [22] during training only, exploiting ResNet50 [10] at test time without any additional parameters. We implement our model with PyTorch [25] and train it end-to-end, taking about 6 and 8 hours for RegDB [24] and SYSU-MM01 [36], respectively, with a Geforce RTX 2080 Ti GPU.

We present in Table 1 a quantitative comparison of our method with the state of the art for VI-reID [39, 21, 3, 17, 34, 35, 38, 4, 37, 36]. We report mean average precision (mAP) (%) and rank-1 accuracy (%) for a single-shot setting on RegDB [24] and SYSU-MM01 [36]. From the table, we can see that our model sets a new state of the art for VI-reID, except for an indoor-search mode on SYSU-MM01 [36], where DDAG [39] shows better results. This method, however, requires additional parameters, other than the ResNet50 [10] backbone, for a feature refinement with self-attention at test time, while being outperformed by ours in other benchmarks. We can also see that our model achieves better results than cm-SSFT¹¹1For cm-SSFT [21], we report in Table 1 the results obtained without using a random erasing technique [50] and a BNN trick [22], similar to ours, for fair comparison. The results are taken from Table 4 of [21]. [21] by a significant margin on both datasets. Note that cm-SSFT [21] uses multiple RGB and IR images to extract person representations, even at test time. That is, it exploits additional images of different modalities, e.g., multiple IR images to extract features from an RGB input. cm-SSFT [21] is thus computationally expensive, and requires a lot of memory. Overall, the experimental results on the standard benchmarks demonstrate that our approach provides person representations robust to the cross-modal discrepancies and intra-class variations across RGB and IR images. Qualitative comparisons along with rank-10 accuracy (%) can be found in the supplementary material.

We compare in Table 2 the average runtime to extract a final person representation. For fair comparison, we measure the average runtime over $50$ executions, for person images of the size $288\times 144$ on the same machine with a Geforce RTX 2080 Ti GPU. Table 2 also compares the number of network parameters required at test time. Our method is fastest among the state of the art, and uses the smallest number of parameters, as it does not use any additional parameters, except the ones for a backbone network, at test time. Other methods on the contrary exploit additional layers or networks.

We show in Table 3 an ablation analysis on training losses and the CMAlign module. We train variants of our model using different combinations of loss terms, $\mathcal{L}_{\text{IC}}$ and $\mathcal{L}_{\text{DT}}$, and co-attention map $A$, while adding CMAlign modules to different layers of the backbone network. We compare the performance in terms of mAP and rank-1 accuracy on SYSU-MM01 [36] under the all-search mode. For the baseline model in the first row, we exclude the CMAlign module and train it using the ID loss alone. Overall, we can see that the baseline shows the worst performance, indicating that incorporating the CMAlign module is beneficial for VI-reID. For example, exploiting the CMAlign module with either the ID consistency term (the second row) or the dense triplet term coupled with the co-attention map (the fourth row) boosts the performance significantly. This is because the ID consistency term mainly addresses cross-modal discrepancies in an image-level and the dense triplet term handles them in a pixel-level, while further enhancing the discriminative power of person representations. From the second, fourth, and last rows, we can observe that using all losses and the co-attention map gives the best results, suggesting that they are complementary to each other. Note that the co-attention map is particularly important for the dense triplet term, as shown in the fifth and last rows. Computing the loss on distractive regions (e.g., occlusions and background clutter) may hinder learning discriminative representations. We also compare in the last three rows our models involving the CMAlign modules in different layers of the backbone network, where the modules are added on top of conv4-6 and/or conv5-3 of ResNet50 [10]. We can see that adding the modules to both conv4-6 and conv5-3 gives the best results, as this allows to consider cross-modal discrepancies in multiple levels of features.

We show in Fig. 4 examples of cross-modal correspondences between RGB and IR images on SYSU-MM01 [36]. We can see that the matches are established well between persons of the same identity, and they are not influenced by cross-modal discrepancies, appearance variations, and background clutter. Specifically, our model provides local features that are robust against scale variations (Fig. 4(a)) and occlusions (Fig. 4(b)). This implies that our model is able to extract discriminative person representations with rich semantics, which are important for the person reID task, while alleviating the cross-modal discrepancies. In particular, our model offers local features that are robust to viewpoint variations (Fig. 4(c)), where a person’s sweatshirt or trousers often matches to its pair regardless of front or side view. This indicates that our network provides local person features that are robust to viewpoint variations, which is particularly useful for VI-reID. This aspect of correspondences for reID is in contrast to typical correspondence tasks, e.g., stereo matching and optical flow estimation, that favor viewpoint-specific matches. We also provide in Fig. 5 a visual comparison of correspondences for different configurations of losses. Our model trained with the ID loss alone is unable to establish reliable matches between cross-modal images and easily distracted by background clutter (Fig. 5(a)), mainly due to cross-modal discrepancies and a lack of discriminative power in local feature representations, particularly for person regions. The ID consistency loss handles the cross-modal discrepancies, establishing correspondences between local person representations from different modalities (Fig. 5(b)). The dense triplet loss further encourages each local feature to be discriminative, which in turn offers matching results focusing on person regions (Fig. 5(c)). The features trained by leveraging dense cross-modal correspondences are more discriminative, establishing matches focusing on person regions, while being robust to the cross-modal discrepancies. More examples can be found in the supplementary material.