Devil’s in the Details: Aligning Visual Clues for Conditional Embedding in Person Re-Identification

Part-based models learn local features of different body parts to enhance the global ReID feature on cross-view matching. One of the most common and effective type of part-based models simply split the output feature-maps of ReID model’s inter-mediate layers into several horizontal stripes and learn local features for each stripe, such as PCB [20], MGN [23], Pyramid [33], RelationNet [14] and VA-ReID [37]. Another type of part-based models [11, 31, 34]segment human body into meaningful body parts and learn local feature for each body parts. SPReID [9] learns a human parsing branch for body part segmentation and fuses local features for different parts by weighted average pooling. DSA-ReID [29] proposes projects human parts into a UV space and uses this UV space branch to guide the learning of a stripe model.

On top of the local features, instead of fusing the local features directly, some methods propose to align parts from a pair of images, and match a pair of images based on the similarity of their aligned part pairs. AlignReID[28] proposes a dynamic programming algorithm to align a stripe in the image to a stripe in another image based on their local feature similarity. DSR [6] and SFR [7] propose a sparse coding method to implicitly look for similar key-point pairs by reconstructing one image’s feature map with another. VPM [19] proposes to align the stripes from two images based on the visibility of each stripes. PGFA [13] exploits pose landmarks to align stripes. HOReID [22] aligns same semantic parts for Occluded ReID by the aid of extra key-points information. CDPM [24] proposes to localize and align local parts by a sliding window method. Some GAN based methods like FD-GAN [4] proposes to align local features by directly transfer the image to the same pose and viewpoint of the target image.

In this paper, Joint Feature Learning refers to methods that feed both images into a model simultaneously to obtain a conditional feature embedding. Existing methods like DCCs [26] and Deep Spatially Multiplicative Integration [27] learns a RNN to iteratively encode couple features step-by-step.

Attention based methods are a type of State-of-the-Art ReID methods that propose to select important regions or channels of a feature-maps to form the ReID feature and discard region irrelevant to recognition such as background. Unlike our CACE-Net that selects crucial region pairs based on a pair of images, most of the existing attention based methods focus on selecting important information from individual images. Method in [32] proposes to predict multiple attention maps for different human parts. HA-CNN [12] uses a Harmonious Attention module to conduct feature selection both spatially and in channel-wise for a individual image. ABDNet [2] proposes a similar spatial and channel attention with an orthogonality constraint. RGA-SC [30] stacks the pairwise relations between single feature and all features together with the feature itself to infer the current position’s attention on spatial and channel dimensions.

Recently some methods propose to use graph-based techniques to learn more complex relationship for ReID model. However, instead of exploring the complex correlation between detailed local regions inside image pairs, most of the existing methods are based on global features. SGGNN [17] use graph to represent the relation between multiple probe images and gallery images and use a graph neural network update samples’ global features with a massage passing method. Group shuffling random walk [18] further extend the probe-gallery relation to gallery-gallery relationship.

Figure 3 shows the general training workflow of CACE-Net. CACE-Net is an end-to-end learning framework containing three stages, namely individual feature embedding, visual clue alignment, and conditional feature embedding. At the first stage, our method extracts individual feature maps for both images. Secondly, given the feature-maps of two images, a correspondence attention module is used to select crucial region pairs both within an image and between image pair. Finally, a novel discrepancy-based GCN is used to extract conditional features from the region correspondence graph. The following three sub-sections elaborate on the implementation and formulation the three stages.

The individual feature embedding stage is responsible for extracting a feature-map for each individual pedestrian image. Any type of CNN backbones for person Re-identification can be applied for the individual feature extraction.

As shown in the blue box in Figure 3, to enforce the backbone network extracting good individual features, an additional training loss branch is attached to the module. Given a training set $\mathcal{D}=\{(I^{(u)},y^{(u)})\}_{u=1}^{N}$, the input image $I^{(u)}$ is first fed into a backbone network. Then, the output feature-map is further fed into a encoder (i.e. a Global Average Pooling followed by a $1*1$ convolution layer) to obtain the individual feature vector for $I^{(u)}$, denoted as $f(I^{(u)})$. Then, a cross entropy based ID loss is used to train the individual feature extractor:

where, $p(c|f(I^{(u)}))=\frac{e^{W^{T}_{C}f(I^{(u)})}}{\sum_{j=1}^{K}e^{W^{T}_{j}f(I^{(u)})}}$, and $W$ is a weight matrix of a fully connect layer to classify $f(I^{(u)})$ into different identities, and $C$ is the total number of identities in the training set.

Visual Clue Alignment select and align crucial regions for pairwise detailed comparison. We denote the feature-map extracted at individual feature embedding stage as $X\in R^{H*W*D}$, where $H$ and $W$ is the height and width of the feature-map and $D$ is the number of feature channels. $X$ is further reshaped into a two-dimensional $HW*D$ matrix denoted as $\hat{X}$. Given the feature-maps of two images indexed as $\hat{X}^{(u)}$ and $\hat{X}^{(v)}$, we select and align the visual clues between them by evaluating the importance of each pair of pixels, denoted as $S^{(u,v)}\in R^{HW*HW}$, where each element $S^{(u,v)}_{ij}$ in the matrix indicate the importance of an aligned pixel pair. Then, The selected pairs of pixels in $X$ forms a undirected graph, whose adjacent matrix $A^{(u,v)}\in R^{HW*HW}$ will be used to extract conditional features of both images with Graph Convolutional Network.

One of the intuitive way to obtain the correlation between two different visual clues is to compute feature similarity. In the ablation study, a cosine similarity is used to evaluate the importance and correlation between a pixel pair from two feature-maps $X^{(u)}_{i},X^{(v)}_{j}$. Then, following a similar strategy in [28], the crucial region pairs are selected by assigning each pixel with the pixel with highest similarity from the other feature-map. Another way to build the pixel correspondence between feature-map pairs is by the human part or other semantic labels of each pixel, where all the corresponding human part are selected with the same importance weight.

As discussed in Section 1, the aforementioned two types of methods for visual clue alignment and selection are based on pre-defined rules and in many cases not able to find the most decisive and discriminative region pairs. Hence, we further propose a novel correspondence attention module that automatically selects crucial visual clue pairs.

Our method not only focuses on finding crucial visual clues between two images, but also discovering intra-relationship between feature-map pixels within an individual image. The importance of pixel pairs within a feature-map $\hat{X}^{(u)}$ is computed as follows:

where $W$ is a diagonal parameter matrix that assigns a learn-able weight for each feature-map channel.

Similarly, given two images whose feature-map is denoted as $X^{(u)}$ and $X^{(v)}$, the inter-image importance is computed as follows:

We combine the importance weight of both intra-image and inter-image visual clues, and select the crucial pairs based on the importance matrix as follow:

where a ReLU activation is used to select the regions pairs with positive importance weights.

Given a pair of images $(I^{(u)},I^{(v)})$, the conditional feature embedding of $I^{(u)}$ should be dynamically adjusted based on $I^{(v)}$, denoted as $f_{cond}(I_{i}|I_{j})$.

In order to fully exploit the detailed information in both images, we propose to extract the conditional feature embedding based on the crucial visual clue pairs between $I^{(u)}$ and $I^{(v)}$. Given the alignment graph $A^{(u,v)}$ of visual clues predicted by the correspondence attention module, we propose a novel discrepancy-based GCN to encode the complex graph structured contextual information into conditional feature vectors.

Like most of the ReID method, the model is evaluated as an image retrieval task. Given a query image set containing $N_{q}$ images and a gallery set containing $N_{g}$ images, we need to retrieve the images with the same identity of $I_{q}$. Our method obtains the similarity between two images with both individual features and conditional features.

We first extract the individual features for all images in query and gallery set, with computational complexity of $O(N_{q}+N_{g})$. Then, for each query, we first sort the gallery images based on the similarity of the individual features. After that, the feature-maps of query image and the top-K images in sorted gallery forms $K$ feature-map pairs, which are fed into the key-point alignment stage and conditional feature embedding stage to obtain conditional features, with computational complexity of $O(N_{q}K)$. Finally, the top-K gallery images are sorted once more by the similarity of coupled features, forming the final ranking result. Compared to the individual feature extraction, much fewer computing operations are needed to obtain conditional feature embedding, so the entire computation cost of CACE-Net is very close to normal ReID method.

Our experiments are conducted on three widely used ReID benchmark datasets.

The input images are resized into $384\times 128$. In training stage, we set batch size to be 16 by sampling 4 identities and 4 images per identity. The ResNet-50 [5] model pretrained on ImageNet is used as the backbone network. Some common data augmentation strategies including horizontal flipping, random cropping, random erasing [36] (with a probability of 0.5) are used. We adopt Gradient Descent optimizer to train our model and set weight decay $5\times 10^{-4}$. The total number of epoch is 80. The learning rate is initialized to $6.25\times 10^{-3}$ and is decayed by cosine method until it equals to 0. At the beginning, we warm up the models for 5 epochs and the learning rate grows linearly from 0 to $6.25\times 10^{-3}$.

In this sub-section, we report the evaluation results of the influence of different components and hyper-parameters of our method.

We evaluate our proposed CACE-Net with the state-of-the-art ReID models. These methods include: (1) the part-based models such as Pyramid, RelationNet; (2) the alignment-based methods like AlignReID, VPM, HOReID; (3) the attention-based methods like ABD-Net, RGA-SC, SCSN; (4) Joint learning methods including DCCs and SMI that learns conditional features with RNN; (5) ReID methods that utilizes graph structure such as Group-shuffling Random Walk and SGGNN. Table 4 shows the performance comparison of CACE-Net with State-of-the-Art methods. As shown in Table 4, thanks to our novel ReID framework that integrates visual clue alignment and conditional feature embedding, CACE-Net outperforms most of the state-of-the-art methods on Market1501, DukeMTMC and MSMT-17.

Furthermore, our CACE-Net can also be applied in ReID in occluded person. As shown in Table 5, other than archieves state-of-the-art method on the general ReID datasets, CACE-Net also outperforms most of the existing occluded ReID methods. It further demonstrates CACE-Net has the ability to solve hard cases like body occlusion.