Pose-guided Feature Disentangling for Occluded Person
Re-identification Based on Transformer

Occluded person Re-ID is more challenging compared with holistic Re-ID due to body information incompleteness. Existing methods can be basically divided into three categories, hand-craft splitting based methods, methods using additional clues, and methods based on the transformer.

Methods based on hand-craft splitting handle the occlusion problem by measuring the similarity relationship of the aligned patches. Sun et al. (Sun et al. 2018) propose a network named Part-based Convolution Baseline (PCB) which uniformly partition the feature map and learn local features directly. Sun et al. (Sun et al. 2019c) propose a region based method VPM which perceives the visible region through self-supervision. Jia et al. (Jia et al. 2021b) propose MoS which formulates the occluded person Re-ID as a set matching problem by using Jaccard similarity coefficient between the corresponding partten set.

Some methods leverage external cues to locate the human body part such as segmentation, pose estimation or body parsing. Song et al.(Song et al. 2018) propose a mask-guided contrastive attention model to learn features separately from the body. Miao et al. (Miao et al. 2019) introduce Pose-Guided Feature Alignment (PGFA) that utilizes pose information to mine discriminative parts. Gao et al. (Gao et al. 2020) propose a Pose-guided Visible Part Matching (PVPM) model to learn discriminative part features with pose-guided attentions. Wang et al. (Wang et al. 2020) propose HOReID that introduces the high-order relation and human-topology information to learn robust features.

Recently, methods based on transformer are emerging, and the transformer has two major capalities. First, transformer has been proven to have powerful feature extraction capabilities. He et al. (He et al. 2021) investigate a pure transformer framework named TransReID that combines the camera perspective information and achieves good performance on both person Re-ID and Vehicle Re-ID. Second, transformer has the ability to learn the disentangled features. Li et al. (Li et al. 2021) is the first one to propose Part Aware Transformer (PAT) for occluded person Re-ID, which could disentangle robust human part discovery.

Different from above methods, our method combines the pose information and transformer architecture to clearly disentangle more discriminative features and effectively alleviate the failure of feature learning caused by the occlusion.

Transformer (Vaswani et al. 2017) has made great achievements in the field of natural language processing. Inspired by the self-attention mechanism, many researchers apply transformers in computer vision. For example, ViT (Dosovitskiy et al. 2020) processes images directly as sequences and achieves state-of-the-art performance in image recognition. DETR (Carion et al. 2020) performs cross-attention between object query and feature map to transform detection problem into a one-to-one matching problem, which eliminates the need for hand-crafted components in object detection.

We build our encoder based on transformer-based image classification model ($i.e.$ViT(Dosovitskiy et al. 2020)). Given a person image $x\in{\mathbb{R}^{H\times{W\times{C}}}}$, where $H$,$W$,$C$ denote the height, width, and channel dimension respectively. We first split the $x$ into $N$ fixed-sized patches $\{x_{p}^{i}|i=1,2,...,N\}$ by using a sliding window. The step size can be denoted as $S$, the size of each image patch as $P$, and the number of patches $N$ can be described as:

where $\lfloor\cdot\rfloor$ is the floor function. When $S$ is equal to patch size $P$, The divided patches are non-overlapping. When $S$ is smaller than $P$, the generated patches are overlapping, which can alleviate the loss of the spatial neighborhood information of the image. The transformer encoder only requires sequence as input, so a trainable linear projection function $f$($\cdot$) is performed on flatten patches to map the patches to $D$ dimensions, and finally obtain the patch embeddings $\textbf{E}\in{\mathbb{R}^{N\times{D}}}$, ($i.e.,\textbf{E}_{i}=f(x_{i}),i=1,2,...,N$). A learnable [class] token $x_{class}$ is prepended to the patch embeddings, and the output [class] token serves as encoder global feature representation $f_{gb}$. In order to retain the position information, we apply learnable positional encodings. However, the features are very susceptible to the variation of the camera, so we follow the method in (He et al. 2021) to learn camera perspective information. Then the final input sequence can be described as:

where $P_{E}$ is positional embeddings, $C_{id}\in{\mathbb{R}^{(N+1)\times{D}}}$ is camera embeddings, and the $C_{id}$ is same for the same images. $\lambda_{cm}$ is a hyper-parameter to balance the weight of camera embeddings. Next, the input embeddings $E_{input}$ will be processed by $m$ transformer layers. The final output $f_{en}\in\mathbb{R}^{(N+1)\times D}$ of encoder can be divided into two parts (encoder global features and part features): $f_{gb}\in\mathbb{R}^{1\times D}$ and $f_{part}\in\mathbb{R}^{N\times D}$. In order to learn more discriminative features on human parts, the part features $f_{part}$ are split into $K$ groups in order, and the size of each group is $(N//K)\times D$. Then, each group that concatenates the encoder global feature $f_{gb}\in\mathbb{R}^{1\times D}$ will be fed into a shared transformer layer to learn $K$ group part local feature $f_{gp}=[f_{gp}^{1},f_{gp}^{2},...,f_{gp}^{K}]$.
Encoder Supervision Loss. We adopt cross-entropy loss as identity loss and triplet loss for encoder global features and group features. The encoder loss function can be formulated as:

where $\mathcal{P}(\cdot)$ denotes the probability prediction function.

Occluded person images suffer from less body information, and non-body part information can be ambiguous, which causes performance degradation. Thus, we employ a human pose estimator to extract keypoint information from images.

Pose Estimation. Given a person image $x$, the estimator extract $M$ landmarks from input image. Then the landmarks are utilized to generate heatmaps $\textbf{H}=[h_{1},h_{2},...,h_{M}]$. Each heatmap is downsampled to the size of $(H/4)\times(W/4)$. The maximum response point of each heatmap corresponds to a joint point. We set a threshold $\gamma$ to filter out high confidence landmarks and low confidence landmarks. But unlike (Miao et al. 2019), we do not set the heatmap whose landmark is smaller than $\gamma$ to 0. Instead, a label $l_{i}\in\{0,1\},i=1,...,M$ will be assigned for each heatmap. Formally, heatmap label can be illustrated as:

where $c_{i}$ denotes the confidence score of i-th landmark.

Pose-guided Feature Aggregation. In order to integrate the the pose information, we set $K=M$, which is exactly equal to the number of keypoints. Then, a fully connected layer is applied to heatmaps H to obtain the heatmaps $\textbf{H}^{{}^{\prime}}$, whose dimension is same as the group part local feature $f_{gp}$. Next, the heatmaps $\textbf{H}^{{}^{\prime}}$ mutiply $f_{gp}$ element-wisely and obtain the pose-guided feature $\textbf{P}=[P_{1},P_{2},...,P_{M}]$. Although P has explicitly encoded the information of different parts of the human body, we hope to find the part of the information from $f_{gp}$ that contributes the most to a certain body part. Thus, we develop a matching and distributing mechanism, which regards the part local feature and pose-guided feature as a set similarity measurement problem. Finally, we can obtain the pose-guided feature aggregation set $\textbf{S}=\{S_{i}|i=1,2,...,M\}$. For each $P_{i}$, we can find the most similar feature in $f_{gp}$, and then two features are added to form the $S_{i}$. Formally,

where $i=1,2,...,K$, $\left\langle{\cdot,\cdot}\right\rangle$ denotes the inner product, $f_{gp}^{k}$ denotes the most similar one to $P_{i}$ in $f_{gp}$.

In this section, we define a set of learnable semantic part views to learn the discriminative body parts. The learnable semantic part views can be denoted as $\textbf{Z}=\{Z_{i}|i=1,2,...,N_{v}\}$, and $\textbf{Z}\in\mathbb{R}^{N_{v}\times D}$ will be added to each cross-attention layer as queries. As shown in Figure 2, the keys and values are from the combination of pose heatmap H and the output of encoder $f_{en}$. An average pooling layer is applied to heatmap H and then multiply the $f_{en}$, finally output the $f_{de}\in\mathbb{R}^{(N+1)\times D}$. Formally, queries, keys and values can be formulated as:

where $i=1,2,...,N_{v},j=1,2,...,D$, linear projections $\textbf{W}_{q}\in\mathbb{R}^{D\times d_{k}}$, $\textbf{W}_{k}\in\mathbb{R}^{D\times d_{k}}$, and $\textbf{W}_{v}\in\mathbb{R}^{D\times d_{v}}$ are applied to semantic part views and feature $f_{de}$, respectively. Next, we can obtain the $N_{v}$ part views set $v=\{v_{i}|i=1,2,...,N_{v}\}$ by implementing the multi-head attention mechanism and two fully-connected layer, which is the same as (Vaswani et al. 2017).

Pose-View Matching Module. In the cross-attention mechanism, the $N_{v}$ part semantic views can learn some discriminative features. However, it is unknown which part or what kind of the information has been learned. Therefore, in order to obtain features related to the human skeleton, we propose a pose-view matching module. Since each feature of the pose-guided feature aggregation set S is related to a certain keypoint information of the human body, we can find the part view $v_{i}$ related to the certain keypoint of the human body by calculating the similarity between the part view $v_{i}$ and $S_{i}$. The matched semantic part view $v_{i}$ and the pose-guided feature aggregation feature $S_{i}$ are added to produce the final view feature set $\textbf{F}_{v}=\{f_{v}^{i}|i=1,2,...,N_{v}\}$. Formally,

since the confidence score of the landmarks can indicate which part of feature contains human body information, the heatmap label $l_{i}$ can guide us to split the view feature set $\textbf{F}_{v}$ into two parts. Features with heatmap label $l_{i}=1$ in view set feature form a high-confidence keypoint view feature set $\textbf{F}_{h}=\{f_{h}^{i}|i=1,2,...,L\}$, and the rest form the low-confidence keypoint view feature set $\textbf{F}_{l}=\{f_{l}^{i}|i=1,2,...,N_{v}-L\}$, where $L$ denotes the number of features in $\textbf{F}_{v}$ whose heatmap label is equal to 1.

Decoder Supervision Loss. In order to focus on more non-occluded body features, we propose a Pose-guided Push Loss:

where $B$ denotes the training batch size. The motivation of this loss is obvious. Human body parts and non-human body parts should not have strong similarities. If $\textbf{F}_{h}$ and $\textbf{F}_{l}$ are similar, then $\mathcal{L}_{p}$ will be large and the learnable sematic part views will adjust themselves adaptively. In order to guide the decoder view feature representation learning, an average-pooling layer is applied to high-confidence keypoints view feature set $\textbf{F}_{h}$ to obtain the pose-guided decoder global feature $f_{ph}$, then identity loss and triplet loss are employed to guide pose-guided decoder global feature $f_{ph}$ and high-confidence feature $f_{h}$ learning as in Eq.13.

In the training stage, the pose estimation uses a pre-trained model, and the rest of components (such as encoder, decoder and so on) are trained together with the overall objective loss, which is formulated as Eq.14.

where $\lambda_{en}$ and $\lambda_{de}$ are the scale factor of encoder loss and decoder loss respectively, and both are set to 0.5.

In the test stage, we concatenate the encoder global feature $f_{gb}$, pose-guided decoder global feature $f_{ph}$, group local part feature $f_{gp}$ and high-confidence keypoint view feature $\textbf{F}_{h}$ as representation $\mathcal{F}$, ingnoring the low-confidence keypoint view feature $\textbf{F}_{l}$. However, high-confidence keypoint view feature $\textbf{F}_{h}$ has variable length, and the network is difficult to implement. Thus, we fix the length of it to $N_{v}$ by padding zeros.

To illustrate the effectiveness of our method, We evaluate our method on five Re-ID datasets for two tasks including occluded person Re-ID and holistic person Re-ID.

Occluded-Duke (Miao et al. 2019) consists of 15,618 training images, 2,210 occluded query images and 17,661 gallery images. It is a subdataset of DukeMTMC-reID (Zheng, Zheng, and Yang 2017), whichtians occluded images and remove some overlapping images.

Occluded-REID (Zhuo et al. 2018) is captured by the mobile phone, which consist of 2,000 images of 200 occluded persons.Each identity has five full-body person images and five occluded person images with different types of severe occlusions.

Market-1501 (Zheng et al. 2015) contains 1,501 identities observed from 6 camera viewpoints, 12,936 training images of 751 identities, 19,732 gallery images, and 2,228 queries.

DukeMTMC-reID (Zheng, Zheng, and Yang 2017) contains 36,411 images of 1,404 identities captured from 8 camera viewpoints. It contains 16,522 training images, 17,661 gallery images and 2,228 queries.

MSMT17 (Wei et al. 2018) contains 125,441 images of 4101 identities captured from 15 camera viewpoints. It contains 32,621 training images. During inference, 82,161 images are randomly selected as gallery and other 11,659 images are considered as query.

Evaluation Metrics. We adopt Cumulative Matching Characteristic (CMC) curves and Mean average precision (mAP) to evaluate the quality of different Re-ID models.

Both training and testing images are resized to 256 $\times$ 128. The training images are augmented with random horizontal flipping, padding, random cropping and random erasing (Zhong et al. 2020). The initial weights of encoder are pre-trained on ImageNet-21K and then finetuned on ImageNet-1K. In this paper, the number of the split group $K$ and the number of the estimated human landmarks are both set to 17. The number of decoder layer is set to 2 on Occluded-Duke and 6 on the other datasets. The hidden dimension $D$ is set to 768. The transformer decoder is same with (Vaswani et al. 2017). The batch size is set to 64 with 4 images per ID. The learing rate is initialized at 0.008 with cosine learning rate decay. To detect landmarks from images, we adopt HRNet (Sun et al. 2019b) pre-trained on the COCO dataset. The threshold $\gamma$ is set to 0.2.

We compare our method with the state-of-the-art methods on five benchmarks including occluded person ReID and holistic person ReID.

Results on Occluded-Duke and Occluded-REID. Table 1 shows the results on two occluded datasets. As table shows, three kinds of methods are compared: (1) hand-crafted splitting based methods including Part-Aligned (Zhao et al. 2017) and PCB (Sun et al. 2018). (2) occluded ReID methods including Part Bilinear (Suh et al. 2018), PD-GAN (Ge et al. 2018), Ad-Occluded (Huang et al. 2018), FPR (He et al. 2019), PGFA (Miao et al. 2019), PVPM (Gao et al. 2020), GASM (He and Liu 2020), HOReID (Wang et al. 2020), ISP (Zhu et al. 2020) and MoS (Jia et al. 2021b). (3) Transformer based occluded ReID methods including PAT (Li et al. 2021) and TransReID (He et al. 2021). From the table, we can observe that our proposed method PFD achieves 67.7%/79.8% Rank-1 accuracy and 60.1%/81.3% mAP on Occluded-Duke and Occluded-REID datasets, respectively, and outperforms all kinds of methods in Occluded-Duke. Futher PFD^∗ achieves higher Rank-1 and mAP with a small step sliding-window setting. Compared with PGFA, PVPM and HOReID, which are SOTA methods with keypoints information, our method surpasses them by at least +12.6% Rank-1 accuracy and +16.3% mAP on Occluded-Duke dataset. Compared to the competing transformer based methods PAT, our method surpasses it by at least +3.2% Rank-1 accuracy and +6.5% mAP on Occluded-Duke and +9.2% mAP on Occluded-REID.

The reasons for the superior performance of PFD can be attributed to the following points. First, compared with CNN, the transformer has better feature representation ability and can pay better attention to discriminative features. Second, the disentangled features obtained from our method can indicate the body part information in cluttered scenes, leading to clear semantic guidance when matching, which is more effective than spatial alignment. Third, the proposed pose-guided push loss efficiently weakens the interference of occlusions and background clutters.

Results on Holistic ReID datasets. To verify the effectiveness of our model on the holistic ReID task, we conduct experiments on three holistic ReID datasets including Market-1501, DukeMTMC-reID and MSMT17. Table 2 shows the results on Market-1501 and DukeMTMC-reID datasets. There are four types of methods in the comparison: (1) part feature based methods including PCB (Sun et al. 2018), DSR (He et al. 2018), BOT (Luo et al. 2019b) and VPM (Sun et al. 2019c). (2) global feature based methods including MVPM (Sun et al. 2019a), SFT (Luo et al. 2019a), CAMA (Yang et al. 2019), IANet (Hou et al. 2019) and Circle (Sun et al. 2020). (3) extra cue based methods including SPReID (Kalayeh et al. 2018), P²Net (Guo et al. 2019), PGFA (Miao et al. 2019), AANet (Tay, Roy, and Yap 2019) and HOReID (Wang et al. 2020). (4) transformer based methods including TransReID (He et al. 2021) and PAT (Li et al. 2021). From the table, we can observe that our proposed method achieve competitive results. Specifically, our method achieves SOTA performance (95.5%/90.6% Rank-1 accuracy and 89.5%/82.2% mAP, respectively) on Market-1501 and DukeMTMC-reID datasets. Compared with transformer based method PAT, our method surpasses it by +1.6% mAP on Market-1501 and +1.8%/+4% Rank-1 accruacy/mAP on DukeMTMC. We also conduct experiments on the proposed method on the MSMT17 dataset. Several methods are compared, including MVPM (Sun et al. 2019a), SFT (Luo et al. 2019a), OSNet (Zhou et al. 2019), IANet (Hou et al. 2019), DG-Net (Zheng et al. 2019), CBN (Zhuang et al. 2020), Cirecle (Sun et al. 2020), RGA-SC (Zhang et al. 2020), and SAN (Jin et al. 2020). From the table 3 we can see that proposed PFD achieves competitive performance. Specifically, our method achieves 82.7% Rank-1 accuracy and 65.1% mAP on MSMT17. It can be seen that although our method is not designed for holistic reid tasks, it can still achieve competitive results, which reflects the robustness of our proposed method.

In this part, we conduct ablation studies on Occluded-Duke dataset to analyze the effectiveness of each component.

Effectiveness of proposed Modules. Table 4 shows the experimental results. Index-1 denotes that the pure transformer encoder-decoder architecture. We can see that the performance can reach 58.2% rank-1 accuracy and 48.3% mAP, which even shows better performance than pose-guided SOTA method HOReID. This is because the self-attention mechanism can focus on more discriminative features than CNN. From index-2, when pose-guided feature aggregation is added, the performance is greatly improved by +5.5% rank-1 accuracy and +7.9% mAP. This shows that the introduction of pose information and correct aggregation can bring good performance improvements. From index-3, we can see that our proposed PVM is also effective. And by comparing index-3 and index-5, we discover that combination of PFA and PVM can increase performance by +8.8% rank-1 accuracy and +11.2% mAP, which indicates that pose information and correct matching is very important. From index-5 and index-6, we can see that our overall model can achieve optimal performance, which shows the effectiveness of the Pose-guided Push Loss.

Analysis of the number of Semantic views. The number of semantic views $N_{v}$ determines the granularity of view features. As shown in Table 5, the performance of our proposed PFD is robust to $N_{v}$. With $N_{v}$ increases, the performance keeps improving before $N_{v}$ arrives 17, which is exactly equal to the number of keypoints. So, we conclude that 17 semantic views may be able to capture the corresponding 17 key point features.

Analysis of the number of Transformer Layers. We perform quantitative experiments to find the most suitable number of decoder layer. As shown in Figure 3(a), when the decoder is removed, the performance of the model is greatly reduced. It can be seen that only the features obtained by the encoder are not robust enough, and the learnable semantic view in the decoder can implicitly learn more important features, which enhances the features from the encoder. we observe that when the number of decoder layer is set to 2, the best performance can be achieved. And with the increase of the number of layers, there is almost no improvement in performance. This is because the resolution of the images in the data set is small, and the content is relatively simple.

The Impact of the Threshold $\gamma$. The threshold $\gamma$ is defined in Eq 4 to indicate the high confidence landmarks, which could help PVM explicitly match visible body parts. We conduct ablation study on threshold $\gamma$ by changing it from 0 to 0.7. From the Figure 3(b), when $\gamma$ is set to 0.2, we can get the best performance. When the value of $\gamma$ is too small, PVM may consider all landmarks as human body areas, thereby introducing noise. Conversely, when the $\gamma$ is too large, a certain body area information may be lost. It is worth noting that when gamma is set to 0, a lot of noise is introduced, but our method can still achieve 65.5% Rank-1 accuracy, which is still SOTA performance on Occluded-Duke. This shows that our method is robust to pose noise, and further indicates why it could achieve good results on the holistic datasets.

The Impact of Pose Estimation. We adopt three different pose estimation algorithms, HRNet (Sun et al. 2019b), AlphaPose (Fang et al. 2017), and OpenPose (Cao et al. 2017) in PFD. From the Table 6, the results shows that the PFD still could achieve state-of-the-art performance by using less reliable landmark estimators. Besides, we add Gaussian noise $\mathcal{N}(\mu,\sigma)$ to the estimated heatmap by changing $\sigma$ from 0.1 to 20. From Fig 4, we find that the model is robust to pose noise when $\sigma$ is less than 10.

We visualize decoder cross-attention for the different learnable semantic views and fuse them together to form attention heatmap. As Figure 5 shows, the fused learnable semantic views can almost accurately localize the unobstructed part of the human body, which proves the effectiveness of our proposed method.