An End-to-End Transformer Model for Crowd Localization

The detection-based methods [31, 24, 44] mainly follow the pipeline of Faster RCNN [29]. Specifically, PSDDN [24] utilizes the nearest neighbor distance to initialize the pseudo bounding boxes and update the pseudo boxes by choosing smaller predicted boxes in the training phase. LSC-CNN [31] also uses a similar mechanism to generate the pseudo bounding boxes and propose a new winner-take-all loss for better training at higher resolutions. These methods [31, 24, 44] usually use NMS to filter the predicted boxes, which is not end-to-end trainable.

Map-based methods are the mainstream of the crowd localization task. Idress et al. [13], and Gao et al. [7] utilize small Gaussian kernel density maps, and the head locations are equal to the maxima of density maps. Even though using the small kernel can generate sharp density maps, it still exists overlaps in the extremely dense region, making the head location undistinguishable. To solve this, some methods [46, 18, 8, 1] focus on designing new maps to handle the extremely dense region, such as the distance label map [46], Focal Inverse Distance Transform Map (FIDTM) [18] and Independent Instance Map (IIM) [8]. These methods can effectively avoid overlap in the dense region, but they need post-processing (“find-maxima”) to extract the instance location and rely on multi-scale feature maps, which is not simple and elegant.

Just a few research works focus on regression-based. We note a recent paper [34], P2PNet, also a regression-based framework for crowd localization. But this is a concurrent work that has appeared while this manuscript is under preparation. P2PNet [34] defines surrogate regression on a large set of proposals, and the model relies on pre-processing, such as producing $8\times W\times H$ point proposals. In contrast, our method replaces massive fixed point proposals with a few trainable instance queries, which is more elegant and unified.

Recently, visual transformers [4, 37, 23, 2, 28] have gone viral in computer vision. In particular, DETR [2] utilizes the Transformer-decoder to model object detection in an end-to-end pipeline, successfully removing the need for post-processing. Based on DETR [2], Conditional DETR [28] further adopts the spatial queries and keys to a band containing the object extremity or a region, accelerating the convergence of DETR [2]. In the crowd analysis, Liang et al. [17, 26] propose TransCrowd, which reformulates the weakly-supervised counting problem from a sequence-to-count perspective. Several methods [36, 35] demonstrate the power of transformers in point-supervised crowd counting setup. Method [6] adopts the IIM [8] in the swin transformer [25] to implement crowd localization.

We use a $1\times 1$ convolution to reduce the channel dimension of the extracted feature maps $F$ from $\mathbb{R}^{\frac{H}{32}\times\frac{W}{32}\times C}$ to $\mathbb{R}^{\frac{H}{32}\times\frac{W}{32}\times c}$ ($c$ set as 256). Due to the transformer-encoder adopt a $1D$ sequence as input, we reshape the extracted features $F$ and add position embedding, resulting in $F_{p}\in\mathbb{R}^{\frac{HW}{32^{2}}\times c}$. The $F_{p}$ are then fed into the transformer-encoder layers to generate the encoded features $F_{e}$. Here the encoder contains many encoder layers, and each layer consists of a self-attention ($SA$) layer and a feed-forward ($FC$) layer. The $SA$ consists of three inputs, including query ($Q$), key ($K$), and value ($V$), defined as follow:

where $Q$, $K$ and $V$ are obtained from the same input $Z$ (e.g., $Q=ZW_{Q}$). In particular, we use the multi-head self-attention ($MSA$) to model the complex feature relation, which is an extension with several independent $SA$ modules: $MSA=[SA_{1};SA_{2};\cdots;SA_{m}]W_{O}$, where $W_{O}$ is a re-projection matrix and $m$ is the number of attention heads set as 8.

The transformer-decoder consists of many decoder layers, and each layer is composed of three sub-layers: (1) a self-attention ($SA$) layer. (b) a cross attention ($CA$) layer. (3) a feed-forward ($FC$) layer. The $SA$ and $FC$ are the same as the encoder. The $CA$ module takes two different embeddings as input instead of the same inputs in $SA$. Let us denote two embeddings as $X$ and $Z$, and the $CA$ can be written as $CA=SA(Q=XW_{Q},K=ZW_{K},V=ZW_{V})$. Following [28], we adopt the conditional cross-attention, i.e., the $Q$ are concatenated by the trainable embeddings $Q_{h}$ and content query (from decoder self-attention). The decoder output the decoded features $F_{d}$, which are used to predict the point coordinates (regression head) and confidence scores (classification head).

To train the model, we need to match the predictions and GT by one-to-one, and the unmatched predicted points are considered to the “background” class.

Let us denote the predicted points set as $\hat{y}=\{\hat{y}_{j}\}_{j=1}^{N}$ and GT points set as $y=\{y_{i}\}_{i=1}^{M}$. $N$ and $M$ refer to the number of predicted heads and GT, respectively. $N$ is larger than $M$ to ensure each GT matches a prediction, and the rest of the predictions match failed can be classified as negative. Next, we need to find a bipartite matching between these two sets with the lowest cost. A straightforward way is to take the $L1$ distance and confidence as matching cost:

where $||*||_{1}$ means the $L1$ distance and $\hat{C_{j}}$ is the confidence of the $j$-th predicted point. $y^{p}_{i}$ is a vector that defines the $i$-th GT point coordinates. Accordingly, $\hat{y}_{j}^{p}$ is formed as the point coordinates of $j$-th predicted head. Based on the $L_{m}$, we can utilize the Hungarian [14] to implement one-to-one matching. However, we find that merely taking the $L1$ with confidence maybe generate unsatisfactory matching results (seen Fig. 1(d)). Another toy example is shown in Fig. 3, given a pair of GT and prediction set (Fig. 3(a)), from the whole perspective, the $\hat{y}_{1}$ should match the $y_{1}$ ideally (just like $\hat{y}_{2}$ match $y_{2}$). Using Eq. 2 for matching cost¹¹1Here, we ignore the $\hat{C_{j}}$ for simply illustrating since heads usually report similar confidence score., it will match the $\hat{y}_{1}$ and $y_{4}$ since the $L1$-based Hungarian lack of context information. Thus, we introduce the KMO-based Hungarian, which effectively utilizes the context as auxiliary matching cost, formulated as $L_{m}^{k}$:

where $d^{k}_{i}$ means the distance between $i$-th GT point and its $k$-th neighbour. $y^{k}_{i}$ refer to the average neighbour-distance of the $i$-th GT point. $\hat{d}^{k}_{j}$ and $\hat{y}^{k}_{j}$ have similar definitions as $d^{k}_{i}$ and $y^{k}_{i}$, respectively. Taking inspirations from [24, 31], in our experiments, $\hat{y}^{k}_{j}$ is predicted by the network. The proposed $L_{m}^{k}$, revisiting the label assignment from a context view, turns to find the whole optimum. As shown in Fig. 3(c), the proposed KMO-based Hungarian makes sure the $\hat{y}_{1}$ successfully matches the $y_{1}$. Regarding the predictions as a point set containing the geometric relationships. Assignment on Fig. 3(b) is not wrong, but it is a local view without context, and it will break the internal geometric relationships of the point set. Assignment on Fig. 3(c) considers the context information from the nearby heads, pursuing the whole optimum and maintaining the geometric relationships of the point set, making the model easier to be optimized, which is more reasonable. For the case in Fig. 1(d), when multiple gts tend to match the same predicted point, the KMO-based Hungarian will resolve their conflicts by using the context information. Note that the matcher is just used in the training phase.

After obtaining the one-to-one matching results, we calculate the loss for back-propagate. We make point predictions directly. The loss consists of point regression and classification. For the point regression, we employ the commonly-used $L_{1}$ loss, defined as:

where $\hat{y}_{\sigma(i)}^{p}$ is the matched subset from $y_{i}^{p}$ by using the proposed KMO-based Hungarian. It is noteworthy that we normalize all ground truth point range to $[0,1]$ for scale invariance. We utilize the focal loss as the classification loss $L_{cls}$, and the final loss $L$ is defined as:

where $\lambda$ is a balance weight, set as 2.5. These two losses are normalized by the number of instances inside the batch.

We use the ResNet50 [10] as the backbone. The number of transformer encoder layers and decoder layers are both set to 6. The $N$ is set to 500 (number of instance queries $Q_{h}$). We augment the training data using random cropping, random scaling, and horizontal flipping with a 0.5 probability. The crop size is set as $128\times 128$ for ShanghaiTech Part A, $256\times 256$ for the rest datasets. We use Adam with the learning rate of 1e-4 to optimize the model. For the large-scale datasets (i.e., UCF-QNRF, JHU-Crowd++, and NWPU-Crowd), we ensure the longer size is less than 2048, keeping the original aspect ratio. The batch size is set to 16. $k$ is set as 4 for all datasets. During the testing phase, each image is split into non-overlapped patches (size same as training phase). Zero padding is adopted if a cropped patch is smaller than the predefined size. And a simple confidence threshold (set to $0.35$) is used to filter the “background” class.

We evaluate our method on five challenging public datasets, each being elaborated below.

NWPU-Crowd [42] is a large-scale dataset collected from various scenes, consisting of 5,109 images. The images are randomly split into training, validation, and test sets, which contain 3109, 500, and 1500 images, respectively. This dataset provides point-level and box-level annotations.

JHU-Crowd++ [33] is a challenging dataset containing 4372 crowd images. This dataset consists of 2272 training images, 500 validation images, and 1600 test images. And the total number of people in each image ranges from 0 to 25791.

UCF-QNRF [13], a dense dataset, contains 1535 images (1201 for training and 334 for testing) and about one million annotations. The average number of pedestrians per image is 815, and the maximum number reaches 12865.

ShanghaiTech [48] is divided into Part A and Part B. Part A consists of 300 training images and 182 test images. Part B consists of 400 training images and 316 test images.

This paper mainly focuses on crowd localization, and counting is an incidental task, i.e., the total count is equal to the number of predicted points.

Localization Metrics. In this work, we utilize the Precision, Recall, and F1-measure as the localization metrics, following  [42, 13]. If the distance between a predicted point and GT point is less than the predefined distance threshold $\sigma$, this predicted point will be treated as True Positive (TP). For the NWPU-Crowd dataset [42], containing the box-level annotations, we set $\sigma$ to ${\sqrt{{w^{2}}+{h^{2}}}}/2$, where $w$ and $h$ are the width and height of each head, respectively. For the ShanghaiTech dataset, we utilize two fixed thresholds, including $\sigma=4$ and $\sigma=8$. For the UCF-QNRF, we use various threshold ranges from [1, 100], following CL [13].

Counting Metrics. The Mean Absolute Error (MAE) and Mean Square Error (MSE) are used as counting metrics, defined as: $MAE=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\left|P_{i}-G_{i}\right|$, $MSE=\sqrt{\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\left|P_{i}-G_{i}\right|^{2}}$, where $N_{c}$ is the total number of images, $P_{i}$ and $G_{i}$ are the predicted and GT count of the $i$-th image, respectively.

We first evaluate the localization performance with some state-of-the-art localization methods [39, 1, 46], as shown in Table 1, Table 2, and Table 3. For the NWPU-Crowd (see Table 1), a large-scale dataset, our CLTR outperforms GL [39] and AutoScale [46] at least 5.8% (resp. 2.5%) for F1-measure on the validation set (resp. test set). It is noteworthy that this dataset provides precise box-level annotations, and the TopoCount [1] relies on the labeled box in the training phase instead of using the point-level annotations. Even though our method is just based on point-annotation, a more weak label mechanism, it can still achieve competitive performance against the TopoCount [1] on the NWPU-Crowd (test set). For the dense dataset, UCF-QNRF (see Table 2), our method achieves the best Average Precision, Average Recall and F1-measure. For the ShanghaiTech Part A (see Table 3), a sparse dataset, our CLTR outperforms the state-of-the-art method TopoCount [1] by 2.1% F1-measure for the strict setting ($\sigma=4$), and still get ahead for the less strict settings ($\sigma=8$). These results demonstrate that the proposed method can cope with various scenes, including large-scale, dense and sparse scenes. Note that all of other localization methods [46, 39, 1] adopt multi-scale or higher-resolution feature that potentially benefit our approach, which is currently not our focus and left as our future work.

In this section, we compare the counting performance with various methods (including density map regression-based and localization-based), as shown in Table 4, Table 5 and Table 6. Although our method only uses a single-scale and low-resolution ($\frac{1}{32}$ of input image) feature map, it can achieve state-of-the-art or highly competitive performance in all experiments. Specifically, our method achieves the first MAE and MSE on the NWPU-Crowd test set (see Table 4). Compared with the localization-based counting methods (the bottom part of Table 5), which can provide the position information, our method achieves the best counting performance in MAE and MSE on ShanghaiTech Part A and Part B datasets. On the UCF-QNRF dataset, our method achieves the best MSE and reports comparable MAE. On the JHU-Crowd++ dataset ( Table 6), our method outperforms the state-of-the-art method GL [39] by a significant margin of 18.9 MSE. Furthermore, CLTR has superior performance on the extremely dense (the “High” part) and degraded set (the “Weather” part).

We further give some qualitative visualizations to analyze the effectiveness of our method, as shown in Fig. 4. The samples are selected from some typical scenes on the NWPU-Crowd dataset (validation set), including negative, sparse, extremely dense and dark scenes. In the first row, CLTR shows a strong robust on the negative sample (“dense fake humans”). CLTR performs well in different congested scenes, such as the sparse scene (row 2) and extremely dense scene (row 3). Additionally, we find that CLTR can also make promising localization results in dark scenes (row 4). These impressive visualizations demonstrate the effectiveness of our method in crowd localization and counting.

The ablation studies are carried out on the UCF-QNRF dataset, a large and dense dataset, which can effectively avoid overfitting.

Our method has some limitations. For instance, due to the CLTR crop a fixed-size (i.e., $256\times 256$) sub-image for training and testing, it may fail on extremely large heads, which are significantly larger than the crop size, as shown in Fig. 5(a). This problem can be solved by resizing the image into a small resolution. Another case of unsatisfied localization is shown in Fig. 5(b), where there are some confused background regions (containing “fake” people that do not need localization). This failure case can be solved using more modalities, such as thermal images.

This work was supported by National Key R&D Program of China (Grant No. 2018YFB1004602).