Efficient Person Search: An Anchor-Free Approach

The basic framework of the proposed AlignPS is based on FCOS [21], one of the most popular one-stage anchor-free object detectors. Differently, we adhere to the “re-id first” principle to put emphasis on learning robust feature embeddings for the re-id subtask, which is crucial for enhancing the overall performance of person search.

As illustrated in Fig. 2, our model simultaneously localizes multiple people in the image and learns re-id embeddings for them. Specifically, an AFA module is developed to aggregate features from multi-level feature maps in the backbone network. To learn re-id embeddings, which is the key of our method, we directly take the flattened features from the output feature maps of AFA as the final embeddings, without any extra embedding layers. For detection, we employ the detection head from FCOS which is good enough for the detection subtask. The detection head consists of two branches, both of which contain four 3$\times$3 conv layers. In the meantime, the first branch predicts regression offsets and centerness scores, while the second makes foreground/background classification. Finally, each location on the output feature map of AFA will be associated with a bounding box with classification and centerness scores, as well as a re-id feature embedding.

Following FPN [18], we make use of different levels of feature maps to learn detection and re-id features. As the key of our framework, the proposed AFA performs three levels of alignment, beyond the original FPN, to make the output re-id features more discriminative.

Scale Alignment. As shown in Fig. 3, the original FCOS model employs different levels of features to detect objects of different sizes. This significantly improves the detection performance since the overlapped ambiguous samples will be assigned to different layers. For the re-id task, however, the multi-level prediction could cause feature misalignment between different scales. In other words, when matching a person of different scales, re-id features are inconsistently taken from different levels of FPN, thus preventing the re-id features from being robust to scale variations. Furthermore, the people in the gallery set are of various scales, which could eventually make the multi-level model fail to find correct matches for the query person. Therefore, in our framework, we only make predictions based on a single layer of AFA, which explicitly addresses the feature misalignment caused by scale variations. Specifically, we employ the $\{C_{3},C_{4},C_{5}\}$ feature maps from the ResNet-50 backbone, and AFA sequentially outputs $\{P_{5},P_{4},P_{3}\}$, with strides of 32, 16, and 8, respectively. We only generate features from $\{P_{3}\}$, which is the largest output feature map, for both the detection and re-id subtasks, and $\{P_{6},P_{7}\}$ are no longer generated as in the original FPN. Although this design may slightly influence the detection performance, we will show in Sec. IV-C that it achieves a good trade-off between the detection and re-id subtasks, which adapts well to the person search task.

Region Alignment. On the output feature map of AFA, each location perceives the information from the whole input image based on a large receptive field. Due to the lack of the ROI-Align operation as in Faster R-CNN, it is difficult for our anchor-free framework to learn more accurate features within the pedestrian bounding boxes, and thus leading to the issue of region misalignment. The re-id subtask is even more sensitive to this issue as background features could greatly impact the discriminative capability of the learned features. In AlignPS, we address this issue from three perspectives. First, we replace the 1$\times$1 conv layers in the lateral connections with 3$\times$3 deformable conv layers. As the original lateral connections are designed to reduce the channels of feature maps, a 1$\times$1 conv is enough. In our design, moreover, the 3$\times$3 deformable conv enables the network to adaptively adjust the receptive field on the input feature maps, thus implicitly fulfilling region alignment. Second, we replace the “sum” operation in the top-down pathway with a “concatenation” operation, which can better aggregate multi-level features. Third, we again replace the 3$\times$3 conv with a 3$\times$3 deformable conv for the output layer of FPN, which further aligns the multi-level features to finally generate a more accurate feature map. The above three designs work seamlessly to address the region misalignment issue, and we notice that these simple designs are extremely effective when accommodating the basic anchor-free model for our person search task.

Task Alignment. Existing person search frameworks typically treat pedestrian detection as the primary task, i.e., re-id embeddings are just generated by stacking an additional layer after the detection features. A recent work [64] investigated a parallel structure by employing independent heads for the two tasks to achieve robust multiple object tracking results. In our task of person search, we find the inferior re-id features largely hinder the overall performance. Therefore, we opt for a different principle to align these two tasks by treating re-id as our primary task. Specifically, the output features of AFA are directly supervised with a re-id loss (which will be introduced in the following subsection), and then fed to the detection head. This “re-id first” design is based on two considerations. First, the detection subtask has been relatively well addressed by existing person search frameworks, which directly inherit the advantages from existing powerful detection frameworks. Therefore, learning discriminative re-id embeddings is our primary concern. As we discussed, re-id performance is more sensitive to region misalignment in an anchor-free framework. Therefore, it is desirable for the person search framework to be inclined towards the re-id subtask. We also show in our experiments that this design significantly improves the discriminative capability of the re-id embeddings, while having negligible impact on detection. Second, compared with “detection first” and parallel structures, the proposed “re-id first” structure does not require an extra layer to generate re-id embeddings, and is thus more efficient.

Existing works typically employ the OIM loss to supervise the training of the re-id subtask. Specifically, OIM stores the feature centers of all labeled identities in a lookup table (LUT), $V\in\mathbb{R}^{D\times L}=\{v_{1},...,v_{L}\}$, which contains $L$ feature vectors with $D$ dimensions. Meanwhile, a circular queue $U\in\mathbb{R}^{D\times Q}=\{u_{1},...,u_{Q}\}$ containing the features of $Q$ unlabeled identities is maintained. At each iteration, given an input feature $x$ with label $i$, OIM computes the similarity between $x$ and all the features in the LUT and circular queue by $V^{T}x$ and $Q^{T}x$, respectively. The probability of $x$ belonging to the identity $i$ is calculated as:

where $\tau=0.1$ is a hyperparameter that controls the softness of the probability distribution. The objective of OIM is to minimize the expected negative log-likelihood:

Although OIM effectively employs both labeled and unlabeled samples, we still observe two limitations. First, the distances are only computed between the input features and the features stored in the lookup table and circular queue, while no comparisons are made between the input features. Second, the log-likelihood loss term does not give an explicit distance metric between feature pairs.

To improve OIM, we propose a specifically designed triplet loss. For each person in the input images, we employ the center sampling strategy as in [65]. As shown in Fig. 4, for each person, a set of features located around the person center are considered as positive samples. The objective is to pull the feature vectors from the same person close, and push the vectors from different people away. Meanwhile, the features from the labeled persons should be close to the corresponding features stored in the LUT, and away from the other features in the LUT.

More specifically, suppose we sample $S$ vectors from one person; we get $X_{m}=\{x_{m,1},...,x_{m,S},v_{m}\}$ and $X_{n}=\{x_{n,1},...,x_{m,S},v_{n}\}$ as the candidate feature sets for the persons with identity labels $m$ and $n$, respectively, where $x_{i,j}$ denotes the $j$-th feature of person $i$, and $v_{i}$ is the $i$-th feature in the LUT. Given $X_{m}$ and $X_{n}$, positive pairs can be sampled within each set, while negative pairs are sampled between the two sets. The triplet loss can be calculated as:

where $M$ denotes the distance margin, and $D_{\text{pos}}$ and $D_{\text{neg}}$ denote the Euclidean distances between the positive pair and the negative pair, respectively. Finally, the Triplet-aided OIM (TOIM) loss is the summation of these two terms:

ROI-AlignPS. Although the proposed AFA module implicitly addresses the region misalignment issue with deformable convolution, it is still desirable to investigate the explicitly aligned learning schemes, e.g., cropping pedestrians for re-id learning [1, 7] and the ROI-Align operation in Faster R-CNN [2, 12]. However, learning based on the cropped pedestrians inevitably transfers the whole framework into a two-step model, which significantly downgrades the efficiency. In the meantime, the dense region proposals in Faster R-CNN also make it inefficient during inference. To address these issues, we propose a novel variant, which we name ROI-AlignPS, by augmenting our efficient anchor-free framework with an explicit region alignment module.

In the training phase, two parallel branches are simultaneously trained with implicit and explicit region alignment, respectively. As shown in Fig. 5a, the bottom branch is the original AlignPS model, while the top branch follows the architecture of NAE [12], which includes a RPN and ROI-Align operation to output the re-id features of a certain region. Both branches share the ResNet-50 backbone from res1 to res4 layers, while having separate res5 layers. In the test phase, as illustrated in Fig 5b, the bottom AlignPS branch first outputs bounding boxes and re-id features. Then, the processed bounding boxes are fed to the top branch to extract explicitly aligned re-id features corresponding to each person. Finally, the re-id features from these two branches are combined to generate a more robust representation.

Although ROI-AlignPS contains an anchor-free branch and an anchor-based branch, the dense anchors in Faster R-CNN are only involved in the training stage. During inference, RPN is discarded and only a small number of detected bounding boxes are processed with ROI-Align. In this way, ROI-AlignPS can still remain highly efficient during the inference stage. Moreover, the explicit and implicit region alignment strategies are complementary to each other. Specifically, the ROI-Align branch extracts re-id features based on bounding boxes, while AlignPS dynamically adapts to certain regions. Meanwhile, the performance of both methods may be influenced by occlusions and backgrounds. Therefore, combining these two kinds of features intuitively reduces such impacts, and the effectiveness of this design is validated in Sec IV-D.

Branch-Level Mutual Learning. Prior works [53, 55, 66, 67] found that feature interaction and knowledge distillation can benefit feature learning in neural networks. In our case, it would be desirable if the prediction of the two branches can reach a better consensus. To fulfill this, we investigate several branch interaction strategies as follows.

(1) Mutual Information Maximization. Mutual information is defined to measure the dependency between two random variables $X$ and $Y$:

where $p(x,y)$ is the joint probability density function, $p(x)$ and $p(y)$ are the marginal probability density functions of $X$ and $Y$, respectively. As demonstrated in prior works [68, 69], mutual information is able to depict the mutual dependence between $X$ and $Y$, no matter how nonlinear the dependence is. Thus, mutual information is more flexible than the measurement of correlation. In this work, random variables $X$ and $Y$ represent the re-id features from the two branches in ROI-AlignPS, respectively. We aim to maximize the mutual information between $X$ and $Y$, such that the overall representation will focus on the identity information. As it is non-trivial to directly calculate the mutual information, we employ a neural estimator [70] to maximize the lower bound of the mutual information. Specifically,

where $\mathbb{P}_{{X}{Y}}$ is the joint distribution and $\mathbb{P}_{{X}\otimes{Y}}$ represents the marginal distribution, $T_{\theta}$ is a neural network parameterized by $\theta\in\Theta$. To maximize $\mathcal{I}_{\Theta}(X,Y)$, we minimize its negative value:

(2) KL-Divergence Minimization. The Kullback-Leibler (KL) divergence is widely employed to measure the difference between two distributions, including successful applications in feature distillation in person search [19, 63]. In our case, we minimize the KL-divergence between identity predictions of the two branches. Suppose $p^{A}$ and $p^{R}$ denote the output probabilities of the AlignPS branch and the ROI-Align branch, respectively, the KL-divergence is calculated as:

where $p_{i}^{A}$ and $p_{i}^{R}$ are calculated from Eq. 1, and $L$ is the number of identities in the training set. In this way, the output features could reach better prediction-level consensus.

(3) Diversity Maximization. Rather than pursuing a consensus between the two branches, we aim to diversify the features belonging to the same identity, to yield more robust ensemble results. To enhance diversity, we promote the output features of the two branches to be different. Specifically, suppose $x_{i}^{A}$ and $x_{i}^{R}$ denote the features of the $i$-th person from the AlingPS branch and ROI-Align branch, respectively, we aim to minimize the cosine similarity between the corresponding features:

where $B$ denotes the number of persons in a mimi-batch, and $\rm cos$ denotes the cosine similarity.

Discussions. A recent work, SeqNet [71], proposes a sequential architecture based on Faster R-CNN, which first predicts the detection results, and then extracts re-id features with an additional ROI-Align branch. Although ROI-AlignPS shares certain spirits with SeqNet, there exist several significant differences. (1) The motivation of SeqNet is to improve the quality of proposals, such that the re-id features could be better learned. In ROI-AlignPS, rather than considering the detection issue, the ROI-Align branch serves as an explicitly aligned re-id feature extractor, to complement the features from the AlignPS branch. (2) Despite the strong performance, SeqNet relies on a post graph matching technique. In contrast, the re-id features in ROI-AlignPS could be mutually promoted and the ensemble results are highly discriminative without post-processing. (3) In the inference stage, ROI-AlignPS still runs in an anchor-free way. Consequently, our framework is more efficient than SeqNet. The comparative results between ROI-AlignPS and SeqNet can be found in Sec. IV-E.

CUHK-SYSU [2] is a large-scale person search dataset which contains 18,184 images, with 8,432 different identities and 96,143 annotated bounding boxes. The images come from two kinds of data sources (i.e., real street snaps and movies/TV), covering diverse scenes and including variations of viewpoints, lighting, resolutions, and occlusions. We utilize the standard training/test split, where the training set contains 5,532 identities and 11,206 images, and the test set contains 2,900 query persons and 6,978 images. This dataset also defines a set of protocols with gallery sizes ranging from 50 to 4,000. We report the results using the default gallery size of 100 unless otherwise specified.

PRW [1] was captured using six static cameras in a university campus. All the images are extracted from the surveillance videos, which consist of 11,816 video frames in total. Person identities and bounding boxes are manually annotated, resulting in 932 labeled persons with 43,110 bounding boxes. The dataset is split into a training set of 5,704 images with 482 different identities, and a test set of 2,057 query persons and 6,112 images. Results are reported based on this split.

Evaluation Metric. We employ the mean average precision (mAP) and top-1 accuracy to evaluate the performance for person search. We also employ recall and average precision (AP) to measure the detection performance.

We employ ResNet-50 [72] pretrained on ImageNet [73] as the backbone. We set the batch size to 4, and adopt the stochastic gradient descent (SGD) optimizer with weight decay of 0.0005. The initial learning rate is set to 0.001 and is reduced by a factor of 10 at epoch 16 and 22, with a total of 24 epochs. We use a warmup strategy for 300 steps. We employ a multi-scale training strategy, where the longer side of the image is randomly resized from 667 to 2000 during training, while zero padding is utilized to fit the images with different resolutions. For inference, we rescale the test images to a fixed size of 1500$\times$900. Following [74], we add a focal loss [28] to the original OIM loss. All the experiments are implemented based on PyTorch [75] and MMDetection [76], with an NVIDIA Tesla V100 GPU. It takes around 38 and 24 hours for ROI-AlignPS to finish training on CUHK-SYSU and PRW, respectively.

Baseline. We directly add a re-id head in parallel with the detection head to the FCOS model and take it as our baseline. As shown in Fig. 6, each of the alignment strategies brings notable improvements to the baseline, and combining all of them yields 23% and 26.1% improvements in mAP on CUHK-SYSU and PRW, respectively.

Scale Alignment. To evaluate the effects of scale alignment, we employ feature maps from different levels of AFA and report the results in Table I. Specifically, we evaluate the features from $P_{3}$, $P_{4}$, and $P_{5}$ with strides of 8, 16, and 32, respectively. As can be observed, features from the largest scale $P_{3}$ yield the best performance on both CUHK-SYSU and PRW, due to the fact that they absorb different levels of features from AFA, providing richer information for detection and re-id. Similar to FCOS, we also evaluate the performance by assigning people of different scales to different feature levels. We set the size ranges for {$P_{3},P_{4}\}$ as [0, 128] and [128, $\infty$], while the prediction ranges for {$P_{3},P_{4},P_{5}\}$ are [0, 128], [128, 256], and [256, $\infty$], respectively. We can see that these dividing strategies achieve slightly better detection results w.r.t. the recall rate. However, they bring back the scale misalignment issue to person re-id, resulting in worse performance compared with single-scale features. Also note that this issue is not well addressed with the multi-scale training strategy. All the above results demonstrate the necessity and effectiveness of the proposed scale alignment strategy.

Region Alignment. We conduct experiments with different combinations of lateral deformable conv, output deformable conv and feature concatenation, and analyze how different region alignment components influence the overall performance. The results are reported in Table II. Without all these modules, the framework only achieves 83.4% and 27.4% in mAP on CUHK-SYSU, and PRW, respectively, which is 9.7% and 18.5% lower than the full model. The individual components of lateral deformable conv and output deformable conv improve the model by $\sim$7% and $\sim$8% on CUHK-SYSU, respectively. Feature concatenation also brings $\sim$1% improvements. By combining two of the three components, we observe consistent improvements. Finally, employing all the three modules yields 93.1% in mAP and 93.4% in top-1 accuracy on CUHK-SYSU, significantly boosting the performance. On PRW, we also observe consistent improvement by introducing these strategies. These ablation studies thoroughly demonstrate the effectiveness of region alignment.

To further illustrate how the deformable convolutions work in our framework, we visualize the learned offsets of the deformable filters in Fig. 7. We observe that the proposed framework is capable of learning adaptive receptive field according to the layout of the human body, and is robust to occlusion, crowding, and scale/illumination variations. We also observe that the lateral deformable conv in $C_{3}$ learns tighter offsets around the body center, while the offsets in the $C_{4}$ layer cover larger regions, which makes the two layers complementary to each other.

Task Alignment. Since person search aims to simultaneously address detection and re-id subtasks in a single framework, it is important to understand how different configurations of the two subtasks influence the overall task and which subtask should be paid more attention to. To this end, we design several structures to compare different training options (as shown in Fig. 8), the performance of which is summarized in Table III. As can be observed, the structures of $T_{1}$ and $T_{2}$, where re-id features are shared with the regression and classification heads, respectively, yield significantly lower performance in re-id compared with our design. This indicates that the detection task takes advantage of the shared heads. As for $T_{3}$ where re-id and detection have independent feature heads, it achieves slightly better performance compared with $T_{1}$ and $T_{2}$, but still remarkably underperforms our design. These results indicate that our “re-id first” structure achieves the best task alignment among all these designs.

TOIM Loss. We evaluate the performance of our framework when adopting different loss functions and report the results in Table IV. We find that directly employing a triplet loss brings slight improvement. When employing the items in the LUT, the TOIM improves the mAP and top-1 accuracy on CUHK-SYSU by 0.7% and 0.5%, respectively. These two terms also slightly improve the performance on PRW. This indicates that it is beneficial to consider the relations between the input features and the features stored in the LUT.

Deformable Conv in the Backbone. As shown in Table V, inserting deformable convolutions into the backbone network has positive effects on our framework. However, the contribution of the deformable conv layers in the backbone network is less significant than the deformable conv layers in our AFA module, e.g., with all the res3 & res4 & res5 deformable conv layers, only $\sim$1% and 0.2% improvements are observed on CUHK-SYSU and PRW, respectively. These results indicate that the proposed AFA works as the key module for successful feature alignment.

We analyze the effectiveness of ROI-AlignPS by answering the following questions.

Is the ROI-Align branch necessary? In ROI-AlignPS, the ROI-Align branch extracts the explicitly aligned re-id features, complementary to the implicitly aligned features in the AlignPS branch. To evaluate the effectiveness of this design, we test each branch under different settings. (1) Separate Training/Test, i.e., two person search models are separately trained and evaluated. During inference, we also concatenate the features from the ROI-Align model (with RPN) and the AlignPS model. (2) Joint Training w/o RPN, where the ROI-Align branch and the AlignPS branch are jointly trained, and both the training and test pipelines follow the structure in Fig. 5b, i.e., no RPN is employed during training and inference. (3) Joint Training w/ RPN, which denotes the proposed ROI-AlignPS framework, where the training pipeline follows Fig. 5a, and the test pipeline follows Fig. 5b. In this case, RPN is only employed in the ROI-Align branch during training. As shown in Table VI, although the separately trained models both obtain promising results, directly concatenating their re-id features does not yield better performance, which may be because the detection results of the two models are not perfectly aligned. In our joint training setting, in contrast, the branch-level features are well aligned. When the ROI-Align branch is trained without RPN, although the performance of each branch is similar to the separately trained models, the concatenated re-id features are notably improved by 1.3% and 4.1% w.r.t. mAP on CUHK-SYSU and PRW, respectively. Moreover, when the ROI-Align branch is trained with RPN, the performance of this branch is further improved, which in turn yields enhanced performance by concatenating the re-id features. These results not only indicate that the jointly training pipeline can better regularize features in the shared layers (i.e., res1-4), but also validate the effectiveness and necessity of the ROI-Align branch, which is complementary to AlignPS.

How does the ROI-Align branch influence the efficiency? In ROI-AlignPS, the ROI-Align branch inevitably brings additional computational overhead. As shown in Table VI, in the inference stage, it takes 75 milliseconds (ms) to process an image with both AlignPS and ROI-Align, in comparison to 61 ms for AlignPS only. However, this is still less than the Faster R-CNN based ROI-Align model (83 ms), because our model does not require dense region proposals during inference. Therefore, ROI-AlignPS achieves significant performance improvement and remains highly efficient.

Are the mutual learning strategies effective? As mentioned in Sec. III-D, we investigate several mutual learning strategies to enhance the representation capabilities of the re-id features in ROI-AlignPS. By applying these strategies, we observe 0.1%-1.3% performance gains in mAP on the two datasets, as shown in Table VII. However, combining several strategies together does not yield better performance, because the objectives of these strategies are partially overlapped. Furthermore, we visualize the distribution of re-id features based on t-SNE. As shown in Fig. 9, the features generated from the two branches are separable, which further validates the complementary property of these two branches. We also observe that the features with mutual learning strategies are more separable, where the overall performance with mutual information loss is slightly better than employing other strategies. Therefore, in the following, we denote ROI-AlignPS as the model trained with $\mathcal{L}_{mi}$.

We compare our framework with state-of-the-art one-step models [2, 50, 9, 10, 74, 11, 19, 12, 77, 63, 71] and two-step models [7, 8, 20, 62, 49].

Results on CUHK-SYSU. As shown in Table VIII, ROI-AlignPS outperforms all the existing person search models. Notably, ROI-AlignPS outperforms the current best-performing SeqNet [71] by 0.6% and 0.3% in mAP and top-1 accuracy, respectively. Note that SeqNet requires a graph matching strategy as post-processing to achieve its best performance, while ROI-AlignPS does not need such a process. We also observe from the table that our model outperforms all the two-step models, even though they employ two separate models for detection and re-id. In contrast, our model allows joint inference with a very simple structure, whilst running at a higher speed.

We visualize the results of AlignPS and ROI-AlignPS w.r.t. mAP with various gallery sizes and compare our model with both one-step and two-step models. Fig. 11 illustrates the detailed results, where ROI-AlignPS outperforms all the models by notable margins, in terms of all the gallery sizes.

Results on PRW. PRW contains less training data; therefore, all the models achieve worse performance on this dataset. Nevertheless, as can be observed from Table VIII, ROI-AlignPS still outperforms all the one-step models in terms of mAP. We notice that DKD [63], PGA [77], IGPN [62], and TCTS [49] achieve higher top-1 accuracy on PRW. These methods also seek to address the feature misalignment issue, but they either resort to feature distillation from the cropped instances [63, 62, 49], or apply attention mechanisms [77]. Differently, our model efficiently addresses this issue with the proposed AFA module.

Efficiency Comparison. Since different methods are evaluated with different GPUs, it is difficult to conduct a fair comparison of the efficiency among all the models. Here, we compare our method with OIM¹¹1We test the PyTorch implementation at https://github.com/serend1p1ty/person_search. [2], NAE/NAE+ [12] and SeqNet [71] on the same Tesla V100 GPU. All the test images are resized to 1500$\times$900 before being fed to the networks. As shown in Table IX, our anchor-free AlignPS only takes 61 ms to process an image, which is 27% and 38% faster than NAE and NAE+, respectively. Meanwhile, ROI-AlignPS also runs faster than prior state-of-the-art models [12, 71]. This is because our models are anchor-free during inference, validating the advantage of the proposed framework.

Qualitative Results. Some qualitative results are illustrated in Fig. 10. We can observe that AlignPS and ROI-AlignPS are more robust in handling occlusions and scale/viewpoint variations, where OIM [2] and NAE [12] fail. A failure case is illustrated in the last row, where our models fail in distinguishing very tiny objects that share similar appearances. In our future work, we will work towards this direction.