Scale-Aware Crowd Count Network with Annotation Error Correction

Traditional methods cast the counting task as a density regression problem (Lempitsky and Zisserman 2010; Sindagi and Patel 2017; Wan and Chan 2020). For a given image $\mathcal{I}$ with $N$ people that we aim to count, let ${\mathrm{\textbf{H}}}_{i}$ denote the true position of the $i^{th}$ person. For any pixel location $x$ in the image $\mathcal{I}$, we model the crowd density $y$ at $x$ as a Gaussian kernel centered at each annotation point. Let $\beta$ denote the annotation variance of the Gaussian kernel and $\sum_{i=1}^{N}\mathcal{N}(x|\mu,\Sigma)$ be the Probability Density Function (PDF) for a multivariate Gaussian with mean $\mu$ and covariance matrix $\Sigma$. We calculate the squared Mahalanobis distance as $\left\|x\right\|^{2}_{\Sigma}=X^{T}\Sigma^{-1}X$, where $X$ is the feature vector of $x$ extracted from a network backbone. The crowd density $y$ at position $x$ is calculated as:

$$y(x)=\sum_{i=1}^{N}\mathcal{N}(x|{\mathrm{\textbf{H}}}_{i},\beta\mathrm{\textbf{I}})=\sum_{i=1}^{N}\frac{1}{\sqrt{2\pi}\beta}exp(-\frac{||x-{\mathrm{\textbf{H}}}_{i}||_{\beta\mathrm{\textbf{I}}}^{2}}{2}),$$

(1)

For all annotated head positions $\{{\mathrm{\textbf{H}}}_{i}\}^{N}_{i=1}$ in the image $\mathcal{I}$, the density map $y$ is estimated by learning a regressor $f(\mathcal{I})$ based on the L2 loss $\mathcal{L}(y,f(\mathcal{I}))$ = $\left\|y-f(\mathcal{I})\right\|^{2}$ or a Bayesian loss (Wan and Chan 2020). The crowd count is calculated as the sum of the map $y$ from all pixels in $\mathcal{I}$.

We next overview the Scale-Aware Crowd Counting Network (SACC-Net) architecture that generates the density map $y(x)$ as in Fig. 2. In most of CNN backbones, the pooling or convolution with stride 2 usually down-samples the image to half and produces feature maps of $\frac{1}{2}$, $\frac{1}{4}$, $\frac{1}{8}$, and so on. We hypothesize that such scale gap is too large, which causes the features fusion within the layers to be uneven. To address this issue, we propose a novel Synthetic Fusion Module (SFM) in Fig. 3 to generate new synthetic layers between the original layers, such that an improved set of density maps can be produced for accurate crowd counting. We further propose an Intra-block Fusion Module (IFM) in Fig. 4 to allow all feature layers within the same convolution block to be fused, such that more fine-grained information can be sent to the decoder for effective crowd counting. At the end of the SACC-Net, we adopt the ASPP (Chen et al. 2017) and CAN (Liu, Salzmann, and Fua 2019) modules to leverage atrous convolutions with different rates to extract multi-scale features for accurate counting.

The Synthetic Fusion Module (SFM) creates various synthetic layers between the original layers to scale the prediction maps to $\frac{1}{2}$, $\frac{1}{3}$, $\frac{1}{4}$, $\frac{1}{6}$ and $\frac{1}{8}$, as in Fig. 3. This provides a denser scale space samples to fit the ground truth whose scale changes continuously. The synthetic layer generated by SFM can take either two or three inputs depending on its position in the SACC-Net, as shown in the brown blocks in Fig. 2. SFM first performs linearly scaling of the inputs then merge them via an $1\times 1$ convolution. The results are then fused using a $3\times 3$ convolution. SFM thus synthesizes a new feature layer from the two original adjacent layers, which yields a smoother scaling space for crowd counting. Details of the linear down-sampling and up-sampling operations and their time complexities are discussed in the supplementary.

Intra-block Fusion Module (IFM): For most CNN architectures such as VGG, various convolutions are performed sequentially to extract features. Only the feature maps from the last layer of a convolution block are sent to the next module. In contrast, we argue that, not only the last layer but also all other layers within the convolution block can provide fine-grained features to generate an accurate density map. To reflect these ideas, our proposed IFM is different in design in comparison to the structure of the DenseNet (Huang et al. 2017). The convolution block of the DenseNet in Fig. 4(a) uses a fully connected structure to link all layers, which might lead to difficulties in training such as complicated computation in back-propagation, excessive RAM usage during model training, and inefficiency during inference. Fig. 4(b) shows the structure of our IFM, which uses fewer connections over DenseNet to generate the required feature maps. IFM contains three advantages over DenseNet: (1) IFM requires fewer memory usage, as it uses $1\times 1$ convolution to directly obtain the output. (2) IFM can obtain more representative features by aggregating all layers of information. (3) IFM contains fewer parameters and is more efficient over DenseNet. Time complexity of IFM is analyzed in the supplementary.

Light-Weight Architecture: Density-based crowd counting methods can achieve good counting accuracy, but existing methods are inefficient for running in real-time applications. The processing of feature maps in SACC-Net goes in two routes, as shown in the Conv-2-1 block in Fig. 2: one branch goes through a VGG block, and the other goes directly to the convolution block with two simple convolutions and move on (see Fig. 2 in the Supplementary). This separation can balance the computation load of each layer and reduce the memory load. In each convolution block, $e.g.$, the green block “Conv-n-x” in Fig. 2), only half the number of channels instead of the original convolution on all channels are sent to the next convolution block, $e.g.$, the block “Conv-(n+1)-x” in Fig. 2. This design can greatly reduce the number of model parameters while maintaining stable accuracy performance. Additional details regarding this lightweight design are provided in the supplementary.

We aim to deal with uncertainty in the manually labeled annotations as shown in Fig. 1(a). Our assumption is that the point-wise head annotations can inevitably come with annotation errors. This annotation errors will cause the crowd density $y$ in Eq. (1) to be incorrectly estimated. We next derive a solution to address this.

Let $\tilde{\mathrm{\textbf{H}}}_{i}$ denote the annotated head position of the $i^{th}$ person with potential annotation error, and $\varepsilon_{i}$ denote its annotation noise, $\tilde{\mathrm{\textbf{H}}}_{i}=\mathrm{\textbf{H}}_{i}+\varepsilon_{i}$. We assume the annotation noise is independent and identically distributed (i.i.d), $\varepsilon_{i}\overset{i.i.d}{\sim}\mathcal{N}(0,\alpha\mathit{\textbf{I}})$, where $\alpha$ is a annotation variance parameter. Considering the case of annotation noise, we model the density $\mathbb{D}(x)$ at location $x$ as:

	$\displaystyle\mathbb{D}(x)$	$\displaystyle=\sum_{i=1}^{N}\mathcal{N}(x|\mathrm{\tilde{\mathrm{\textbf{H}}}_{i}},\beta\mathit{\textbf{I}})=\sum_{i=1}^{N}\mathcal{N}(x|{\mathrm{\textbf{H}}}_{i}+\varepsilon_{i},\beta\mathit{\textbf{I}})$		(2)
		$\displaystyle=\sum_{i=1}^{N}\mathcal{N}(q_{i}|\varepsilon_{i},\beta\textbf{I})\cong\sum_{i=1}^{N}\phi_{i}.$

where $\beta$ is defined in Eq. (1), $q_{i}=x-{\mathrm{\textbf{H}}}_{i}$ denoting the position difference between the $i^{th}$ annotation and $x$, and $\phi_{i}$ denotes the individual Gaussian kernel for the $i^{th}$ annotation. In the literature, (Wan and Chan 2020) did not distinguish the range of annotation errors among small and large objects. They proposed a fixed-scale model using the NoiseCC loss to rectify the annotation noise. One key problem for all SoTA methods (Lempitsky and Zisserman 2010; Sindagi and Patel 2017; Wan and Chan 2020) regarding Eq. (2) is that a fixed $\beta$ with constant value is used to model $\mathbb{D}(x)$ of the crowd density around each head position. As aforementioned, the head sizes appear in different sizes according to their distances w.r.t. the observing camera. To this end, our design makes $\beta$ scale-aware and adaptive to the size of each head appearing in the image.

As shown in Fig. 1, the distribution of $\beta$ is positively skewed (small heads occupy more). With this observation, this section will propose a scale-adaptive Gussian model for heat map generation and annotation error correction. Assume that there are $S$ scales used to model a head with annotation errors. Then, Eq. (2) can be rewritten as:

$\displaystyle\mathbb{D}(x)$

$\displaystyle=\sum_{i=1}^{N}\sum_{s=1}^{S}w_{s}\mathcal{N}(q_{i}|\varepsilon_{i},\beta_{s}\textbf{I})\cong\sum_{s=1}^{S}w_{s}\sum_{i=1}^{N}\phi_{i}^{s},\vspace{-3mm}$

(3)

where the density of a head is modeled with a mixed Gaussian model, and $\sum_{s=1}^{S}w_{s}=1$. In addition, $\phi_{i}^{s}=\mathcal{N}(q_{i}|\varepsilon_{i},\beta_{s}\textbf{I})$, $i.e$., the Gaussian kernel placed in the $i$th annotation at the scale $s$ and parameterized with the annotation error $\varepsilon_{i}$ and the variance $\beta_{s}$. Let $\mathbb{D}_{s}=w_{s}\sum_{i=1}^{N}\phi_{i}^{s}$. In addition, we denote $\mathcal{I}_{s}$ as the scaled-down version of $\mathcal{I}$ at scale $s$. For all pixels $x_{j}$ in $\mathcal{I}_{s}$, a multivariate random variable for the density map $\mathbb{D}(x)$ at the scale $s$ can be constructed as:

$\displaystyle\mathbb{D}_{s}=[\mathbb{D}_{s}(x_{1}),\cdots,\mathbb{D}_{s}(x_{j}),\cdots,\mathbb{D}_{s}(x_{J_{s}})],\vspace{-2mm}$

(4)

where $J_{s}$ is the number of pixels in $\mathcal{I}_{s}$.

Since the dimension of $\boldsymbol{\Sigma}_{x_{j},x_{k}}^{s}$ is huge, $i.e$, $J_{s}\times J_{s}$, this section will derive its low-rank approximation with its non-zero rows and columns for efficiency improvement. Let $\hat{\boldsymbol{\Sigma}}^{s}$ denote the approximation to $\boldsymbol{\Sigma}^{s}$ using Singular Value Decomposition (SVD) calculated as:

$$\hat{\boldsymbol{\Sigma}}^{s}\cong\boldsymbol{\Sigma}^{s}=\textbf{U}^{s}\textbf{C}_{L}^{s}\textbf{V}^{T},$$

(9)

where $\textbf{U}^{s}$ is an $J_{s}\times J_{s}$ orthogonal matrix, $\textbf{C}_{L}^{s}$ is a nonnegative $J_{s}\times J_{s}$ diagonal matrix with diagonal entries sorted from high to low, and $\textbf{V}^{T}$ is a $J_{s}\times J_{s}$ orthogonal matrix. Let $v_{j}^{s}$ = $\boldsymbol{\Sigma}_{x_{j},x_{j}}^{s}$. To obtain this low-rank approximation, each pixel $x_{j}$ is first ordered by $v_{j}^{s}$. Then, the top-$M$ pixels whose percentages of variance are larger than 0.8, $i.e$.

$$\frac{\sum_{j=1}^{M}v_{j}^{s}}{\sum_{i=1}^{J}v_{j}^{s}}>0.8,$$

(10)

are selected from $\mathcal{I}_{s}$ for this low-rank approximation. Let the set of indices of the top $M$ pixels be denoted by $L$, $i.e.$, $L=\{l_{1},l_{2},\dots,l_{m},\dots,l_{M}\}$. Then, only the elements in $L$ are selected to approximate $\boldsymbol{\Sigma}^{s}$. The approximation of a matrix $\boldsymbol{\Sigma}^{s}$ by a rank-$M$ matrix requires a representation of $\boldsymbol{\Sigma}^{s}$ as the sum of several ingredients ordered by their importance. SVD lends itself to this task by transforming $\boldsymbol{\Sigma}^{s}$ into the sum of rank-1 matrices (weighted by the corresponding singular values), namely, $\boldsymbol{\Sigma}^{s}=\textbf{U}^{s}\textbf{C}_{L}^{s}{\textbf{V}^{s}}^{T}$ is equivalent to:

$$\boldsymbol{\Sigma}^{s}=\sum^{J_{s}}_{i=1}c_{i}^{s}\cdot\textbf{u}^{s}_{i}{\textbf{v}_{i}^{s}}^{T},\vspace{-2mm}$$

(11)

where the scale $s=1,...,S$, $c_{i}^{s}$ is the $i$th singular value and $\textbf{u}^{s}_{i},\textbf{v}_{i}^{T}$ are the corresponding left and right singular vectors. A natural idea is to keep only the top $M$ terms on the right-hand side of Eq. (11). That is, for $\boldsymbol{\Sigma}^{s}$ as in Eq. (11) and a target rank $M$, the proposed rank-$M$ approximation is:

$$\hat{\boldsymbol{\Sigma}}^{s}\cong\sum^{M}_{i=1}c_{i}^{s}\cdot\textbf{u}^{s}_{i}{\textbf{v}_{i}^{s}}^{T},\vspace{-2mm}$$

(12)

where the singular values ( $s_{1}\geq s_{2}\geq\cdots\geq s_{J_{s}}\geq 0$) have been sorted, and $\textbf{u}^{s}_{i},\textbf{v}_{i}^{T}$ denote the $i$th left and right singular vectors. With $\hat{\boldsymbol{\Sigma}}^{s}$, the rank-M approximate negative log-likelihood function is:

$$-\mathrm{log}\hat{p}(\textbf{D}_{s})=-\mathrm{log}\mathcal{N}(\textbf{D}_{s}|\mu_{s},\hat{\boldsymbol{\Sigma}}^{s})\propto||\textbf{D}_{s}-\mu_{s}||^{2}_{\hat{\boldsymbol{\Sigma}}^{s}}.$$

(13)

The time complexities to calculate the right-hand sides of Eq.(12) and Eq.(13) take $O(M^{3})$ and $O(M^{2})$, in contrast to $O(J_{s}^{3})$ and $O(J_{s}^{2})$ required to calculate the original matrix $\boldsymbol{\Sigma}^{s}$ and the distance $||\textbf{D}_{s}-\mu_{s}||^{2}_{{\boldsymbol{\Sigma}}^{s}}$, respectively.

The Gaussian approximation to $\textbf{D}_{s}$ can be obtained based on Eqs. (12) and (13). To ensure that the predicted density map near each annotation satisfies the density sum to 1, for the $i$-th annotation point, we define the regularizer $\mathcal{R}_{i}^{s}$ as:

$$\mathcal{R}_{i}^{s}=|\sum_{j}\mathbb{D}_{s}(x_{j})\frac{\phi_{i}^{s}(x_{j})}{\sum_{i=1}^{N}\phi_{i}^{s}{(x_{j})}}-1|,$$

(14)

where $\mathbb{D}_{s}(x_{j})$ is the $j^{th}$ term of $\textbf{D}_{s}$. Let $\bar{\textbf{D}}_{s}=\textbf{D}_{s}-\mu_{s}$. Then, the final loss function is:

$$\mathcal{L}=\sum_{s=1}^{S}\bar{\textbf{D}}_{s}^{T}(\hat{\Sigma}^{s})^{-1}\bar{\textbf{D}}_{s}+\sum_{s=1}^{S}\sum_{i=1}^{N}\mathcal{R}_{i}^{s}.$$

(15)

Our method was pre-trained on ImageNet (Deng et al. 2009) with the Adam optimizer. Since the image dimensions in the used datasets are different, patches with a fixed size are cropped at random locations, then randomly flipped (horizontally) with probability 0.5 for data augmentation. The learning rates used in the training process are $1e^{-5}$, $1e^{-5}$, $1e^{-5}$, and $1e^{-4}$ for the UCF-QNRF, UCF CC 50, NWPU, and ShanghaiTech datasets, respectively. To stabilize the training loss change, we use batch sizes 10, 10, 15, and 10, respectively. All parameters used in the training stage are listed in Table 1. Similarly to other SoTA methods (Li, Zhang, and Chen 2018; Xu et al. 2019; Bai et al. 2020; Xiong et al. 2019; Varior et al. 2019; Jiang et al. 2020; Thanasutives et al. 2021; Zhu et al. 2019), the mean average error (MAE) and the mean squared error (MSE) are used to evaluate the performance of our architecture.

As shown in Fig. 1(a), Fig. 5(a), and Fig. 6(a), the number of heads of the crowd decreases according to the head size $h$. This $P_{head}(h)$ distribution is positive-skewed and can be easily obtained by accumulating from the training data. In Eq. (1), the variance parameter $\beta$ is proportional to the head size. We can set the mean of $h$ as the initial value of $\beta_{1}$, $\beta_{1}=\sum_{h}hP_{head}(h)$. In a CNN backbone such as VGG19, the pooling operation will reduce the feature map size by half, thus also reduce the head size in the feature map. Given $\beta_{s}$, the value of $\beta_{s+1}$ can be obtained in a recursive form: $\beta_{s+1}=\beta_{s}/2$. Our analysis yields $\beta$ to be approximately 8.3. This subsampling operation will also make small heads to eventually disappear. Then, $w_{s}$ is set to $P_{head}(\beta_{(S+1-s)})$, where $S$ is the largest scale used to model $\mathbb{D}(x)$ in Eq. (3). We set $S=3$. After normalization, we have $\sum_{s=1}^{S}w_{s}=1$.

We compare the proposed loss function against L2, BL (Ma et al. 2019), NoiseCC (Wan and Chan 2020), DM-count (Wang et al. 2020), and the generalized loss (Wan, Liu, and Chan 2021) under different backbones to evaluate the effectiveness. Table 2 shows the performance evaluation results. Clearly, our proposed scale-aware loss function outperforms other SoTA loss functions under various backbones. Since human head sizes are different, the same annotation error causes different effects to affect the accuracy of crowd counting. Although NoiseCC (Wan and Chan 2020) has pointed out that the annotation noise will affect the accuracy of crowd counting, their work did not address the scaling issues. Our scale-aware loss function can properly handle it and outperforms other losses on UCF-QNRF.

To further evaluate the performance of our proposed method, 14 SoTA methods are compared here for performance evaluation; that is, CSRNet (Li, Zhang, and Chen 2018), CAN (Liu, Salzmann, and Fua 2019), S-DCNet (Xiong et al. 2019), SANet (Cao et al. 2018), BL (Ma et al. 2019), SFANet (Zhu et al. 2019), DM-Count (Wang et al. 2020), RPnet (Zhang et al. 2015), AMSNet (Hu et al. 2020) M-SFANet (Thanasutives et al. 2021), TEDnet (Jiang et al. 2019), P2PNet (Song et al. 2021), GauNet (Cheng et al. 2022), and MAN (Lin et al. 2022). Table 3 shows the comparative results among these SoTA methods on four benchmark datasets. Clearly, our method achieves the best MAE on all the above datasets, especially for large-scale datasets such as UCF-QNRF, NWPU-Crowd, and ShanghaiTech Part A. As to the MSE metric, our method outperforms all SoTA methods except MAN (Lin et al. 2022).

To demonstrate the effectiveness of our fusion approach, we conducted an ablation study on how the addition of “fusion” and the number of scales used to improve crowd counting accuracy. Most objects in the UCF-QNRF dataset are smaller than those in other datasets. Thus, UCF-QNRF is adopted here to fairly evaluate the effect of our proposed fusion module. In Table 4, we can see that using our fusion module is significantly better than not using it. For example, our SACC-Net with this module reduces the error rates significantly from 85.42 to 77.12 in MAE and from 145.44 to 124.25 in MSE for the UCF-QNRF dataset.

We also evaluated the effects of the number of scales on improving the accuracy of crowd count. There are five pooling layers created in VGG19 that cause the original image to be scaled down to only 1/32$\times$1/32 ratio. The feature map in the last layer cannot provide enough information to calculate the required covariance matrix. The first layer is too primitive for crowd counting. Since three layers provide optimal performance results, we set $S$ to three in Eq. (3). Table 4 shows the accuracy comparisons between three combinations of three scales (corresponding to layer 2, layer 3, and layer 4). The three-scale scale-aware loss function significantly improves the accuracy of crowd counting on the UCF-QNRF dataset, especially in the MAE metric.

Visualization results of heat maps: Fig. 5 shows the visualization results when different loss functions were used. The ground truth of head counting in (a) is 855. The heat maps generated by the MSE loss and the Bayesian loss (Ma et al. 2019) are visualized in (b) and (c) with the predicted results of 772.69 and 901.30, respectively. Clearly, the MSE loss performs better than the BL method. (d) is the visualization result generated by our scale-aware loss with the predicted value 884.1. Compared to (b) and (c), our loss function can generate a more detailed heat map for heads, since annotation errors are taken into account in crowd modeling. NoiseCC (Wan and Chan 2020) also points out that noise from annotation will affect the accuracy of crowd counting. Fig. 6 shows the visualization results generated by our method wo/ and w/ IFM. The detailed heat map of smaller heads generated in (c) leads to better crowd counting accuracy, which justifies the effectiveness of IFM. The SMF can generate various synthetic layers to construct better density maps for crowd counting. Refer to the supplementary for detailed performance evaluations of our lightweight model.