Counting from Sky: A Large-scale Dataset for Remote Sensing Object Counting and A Benchmark Method

Contemporary works for object counting can be roughly classified into three categories: 1) detection-based; 2) regression-based and 3) density map estimation based methods. Detection-based methods are inchoate counting techniques, which estimate the number of objects from the detection of object instances. The performance of these methods relies on sophisticated detectors and is not satisfactory owing to the fact that object detection is still in its primitive stage in the early years. In recent years, with the advent of deep learning techniques, object detection has achieved significant progress. Object detectors such as Faster RCNN [37], SSD [39], YOLO [38] have shown remarkable performance in various object detections such as face [42], pedestrian [43] and so on. Counting is completed spontaneously after the detection by simply summarizing the number of detection instances. However, these approaches can only work well in the easy case, i.e., objects are sparsely located and in relatively large scales, they may show poor performance in densely congested scenes, particularly in remote sensing images, since ground objects such as dense buildings, small-vehicles, large-vehicles, and ships are far beyond current detection performance.

Regression-based methods, also known as global-regression based methods, count objects via mapping the high-dimension images into the real number space through regression models such as linear regression [44] or Gaussian mixture regression [45]. Elaborately designed hand-crafted features such as SIFT [46], GLCM [47] are usually employed to represent images. These methods are successful when dealing with objects with homogeneity scales and uniform distributions, however, may fail when objects are with varying scales and cluster distributions.

Lempitsky et al. [16] set a milestone for subsequent counting researches by casting the visual object counting problem as image density estimation, in which the integral of the density map gives the count of objects within it. Following this work, Pham et al. [48] learn a non-linear mapping function by using random forest regression. Recently, with the entering of the deep learning era, many CNN based counting methods have been proposed and achieved great success, especially in crowd counting. Zhang et al. [1] propose a multi-column neural network (MCNN) with three branches, each of which constructed with different kernel sizes for capturing varisized objects. Sindagi et al. [49] improve the MCNN by jointly learning crowd count classification and density map estimation, and integrate them into an end-to-end cascaded network. Sam et al. [5] propose a switching CNN to select the most suitable regressor by training a switch classifier. Instead of designing a wider multi-column network, Li et al. [12] propose to devise a deeper single column network that utilizes dilated convolution to enlarge the receptive fields so as to boost counting performance. In this paper, we also intend to design a CNN based model for estimating density maps of remote sensing images by taking advantage of recently developed techniques such as attention mechanism.

Attention mechanism has been incorporated into deep neural networks for improving performance in various computer vision tasks, including image captioning [50, 51], image question answering [52], video analysis [53], image classification [54], object counting [7, 55] and countless others. Bahdanau et al. [56] are among the first to introduce the attention mechanism and successfully apply it to machine translation. Chen et al. [57] propose to encode the spatial- and channel-wise attention sequentially for improving image captioning performance. Wang et al. [58] put forward a non-local neural network to compute the response at a position as a weighted sum of feature maps. Woo et al. [59] devise a convolution block attention module (CBAM) to enrich feature representations. It can be plugged into many feed-forward convolutional networks. The attention mechanism has also been introduced into the counting field. For instance, Liu et al. [7] incorporate an attention module to adaptively decide the appropriate counting mode for different locations on the image based on its real density conditions. Zhang et al. [55] propose an attentional neural field network, in which they introduce the no-local attention module [58] to expand the receptive field to the entire image such that the method can deal with large scale variations. Our work arranges the channel and spatial attention modules in different manners. The rational arrangement will be demonstrated in Section V-C. With the attention module incorporated in our proposed remote sensing object counting framework, we can achieve the goal of highlighting the region of interest and alleviating the interference of cluttered backgrounds.

Dilated convolution has been widely used in a bunch of vision tasks, due to its prominent characteristics of enlarging the receptive field of networks without increasing the computation complexity. Yu et al. [60] design a dilated convolution-based network to capture multi-scale contextual features for better segmenting objects. Song et al. [61] leverage pyramid dilated convolution for video salient object detection. Li et al. [62] utilize dilated convolution for image de-raining. In counting community, CSRNet [12] also employs dilated convolution to design a deep convolutional neural network for crowd counting in congested scenes.

Different from previous approaches, our proposed work incorporates the attention mechanism to capture more contextual features and then concatenates several parallel dilated convolutions with different dilation rates to extract multi-scale features. Furthermore, a set of deformable convolution layers is leveraged to generate high-quality density maps to accurately locate the position of objects, to address the issue that orientation arbitrariness due to overhead perspective in the remote sensing images.

For a given remote sensing image with an arbitrary size, we first feed it into a feature extractor, which is composed of the first 10 convolution layers of the pre-trained VGG-16 [40]. The VGG-16 is widely used in counting tasks as backbone to build models for its strong generalization ability [5, 6, 12], thus in this work we also employ VGG-16 as the basis to construct our network. VGGs were initially designed and trained for image classification task [63]. Although it has been proved that VGGs could generalize well to remote sensing images after fun-tuning [64], additional designs should be considered to address counting tasks, since there is a significant gap between counting and classification. The features should be enhanced to better represent crowded objects.

Inspired by recent development in attention mechanism, especially [59] and [65], we incorporate attention modules to achieve the goal of capturing more contextual and high-level semantics. We consider both channel- and spatial-attention, which are described below:

There are multiple max-pooling operations in the front-end stage, leading to dramatically decrease in the size of the feature map. The size of the output map is only 1/64 of the input image. This will bring two drawbacks. Firstly, a density map indicates the localization of objects in the image. Pooling operation makes the features invariance to local translations thus is good for classification however is harmful for object localization, in here is a barrier to generating high-quality density map. Secondly, the information of small objects is weakened as the spatial resolution of the feature map decreases, making the network blind to the small objects.

To address these problems, inspired by [66], we introduce a scale pyramid module (SPM) that is built with four parallel dilated convolution layers. The dilated convolution is convolution with holes as illustrated in Fig. 3. It was first introduced by Yu et al. [60] in the segmentation task to expand the receptive fields without losing spatial resolution of feature maps. In addition, there are no extra parameters or computations, making it an excellent solution for dense prediction tasks, e.g., scene parsing [67] and object counting [12, 66].

In this work, all the four layers have the same channels however different dilation rates to capture different scale information. We use dilation rates 2, 4, 8 and 12 as suggested in [66]. In this way, we build a pyramid with different receptive fields that can not only keep the spatial resolution of feature maps unchanged but also can be invariant to scale variations. The structure of the SPM is depicted in Fig. 3.

Deformable convolution [68] is an operation that adds an offset, whose magnitude can be learnable, on each point in the receptive field of the feature map. After the offset, the shape of the receptive field matches the actual shape of the object rather than a square. The advantage of the offset is that no matter how deformable the object is, the region of convolution always covers the object. The diagram and the visualization of deformable convolution are illustrated in Fig. 4 and Fig. 5, respectively.

For a normal convolution, a given sampling location $\mathbf{p}_{m}$ with $3\times 3$ convolution kernel of dilation 1, which can be represented as $\mathbf{p}_{m}\in\mathcal{M}=\{(-1,-1),(0,-1).\ldots,(1,0),(1,1)\}$. And then the output feature map $\mathbf{y}(\mathbf{p})$ on the location $\mathbf{p}$ can be formulated as

where $\mathbf{w}$ represents weighted parameters and $\mathbf{x}$ means the input feature map.

In contrast with normal convolution, deformable convolution adds an adaptive learnable offset $\Delta\mathbf{p}_{m}$, which can be optimized via training [68]. Therefore, the deformable convolution of feature map $\mathbf{y}(\mathbf{p})$ can be represented as

Specifically, we adopt three deformable convolution layers with the kernel size $3\times 3$ followed by ReLU activation after each layer. And then a $1\times 1$ convolution is leveraged to generate the density map. The final counting number of objects can be computed by summing all the pixel values of the density map. With the dynamic sampling scheme in deformable convolution, the orientation arbitrariness due to the overhead perspective in the remote sensing imagery can be well addressed.

We generate the ground truth density maps following the procedure of density map generation in previous works [1, 12, 66]. Assuming that there is an object instance at pixel ${x_{i}}$, it can be represented by a delta function $\delta({x-{x_{i}}})$. Therefore, for an image with $N$ annotations, it can be represented as follows:

To generate the density map $F$, we convolute $H(x)$ with a Gaussian kernel, which can be defined as follows:

where $\sigma_{i}$ represents the standard deviation. Empirically, we adopt the fixed kernel with $\sigma=15$ for all the experiments. The ground truth density maps after Gaussian convolution operation are visualized in Fig. 6.

We adopt Euclidean distance as loss function to evaluate the difference between the predicted density maps and the ground truths, which is widely adopted by many other counting works [1, 3, 5]. The $L_{2}$ loss function is defined as follows:

where $N$ means the batch size, ${X_{i}}$ represents the input image and $\Theta$ indicates the trainable parameters, $F\left({{X_{i}};\Theta}\right)$ and $F_{i}^{GT}$ are an estimated density map and its corresponding ground truth, respectively.

Two widely used metrics, Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), are employed to evaluate the performance of the proposed and comparison methods. MAE measures the accuracy of the model, while RMSE measures the robustness. These two metrics are defined as follows:

where $n$ is the number of test images, ${\hat{C}_{i}}$ denotes the predicted count for the $i$th image and ${C_{i}}$ indicates the ground truth count.

We implement the proposed ASPD-Net in PyTorch [69], train and test it on an NVIDIA 2080Ti GPU. The first 10 convolution layers are fine-tuned from VGG-16 [40], and the other layers are initialized through a Gaussian noise with 0.01 standard deviation. ASPD-Net is trained in an end-to-end manner. During training, we employ stochastic gradient descent (SGD) to optimize the network and set the learning rate as le-5. For the building subset, we adopt a batch size of 32 and set a batch size of 1 for the other three subsets. For ShanghaiTech crowd counting dataset, we inherit the training manner in [12]. All training will reach convergence in 400 epochs.

We apply data augmentation to generate more training samples. For each image from the training set, 9 sub-images with 1/4 size of the original image are cropped, of which four are adjacent non-overlapping sub-images and the other five are cropped randomly. After then, a mirror flip is applied to double them. Since the ship, large-vehicle and small-vehicle images are with large sizes, which will lead to out-of-memory of GPU in the training phase, therefore, we resize all the large images into 1024 $\times$ 768 pixels before data augmentation.

There are several configurations when integrating channel- and spatial-attention, as shown in Fig. 10. Here we test all the combinations on top of the MCNN [1] by embedding attention modules into the MCNN. We choose MCNN because it is easy to implement and train. An example test model is shown in Fig. 11. We conduct experiments on the RSOC_building subset. The results are illustrated in Table II. We can find that incorporating attention module, either spatial or channel attention, could significantly improve the performance by a large margin. Channel attention performs slightly better than spatial attention. Combining two attentions together could further boost the performance. And sequential assemblies are superior to parallel one. This is consistent with [59]. The ‘channel+spatial’ configuration obtains the best result, thus in the following experiments, we employ it as the attention module to embed into the proposed ASPD-Net.

To better quantify the contribution of different modules of our method, we conduct an ablation study on the RSOC_building subset with simplified models.

The results of the ablation experiments are tabulated in Table III. From the table, we can see that each component in our network contributes to performance improvement. Specifically, the naive baseline method does not perform the most optimal performance. Using the attention module captures the global and local dependencies of the feature maps. The use of scale pyramid module further improves the performance by capturing the multi-scale information. A toy example for the visualization of density maps on RSOC_building subset is depicted in Fig. 12. It can be intuitively observed that by incorporating attention module, scale pyramid module, and deformable convolution module into the unified framework, our proposed ASPDNet can obtain the superior counting performance and accurate predicts.

We make comparison with several state-of-the-art counting methods, which are first raised for crowed counting, however, they can be applicable for object counting in remote sensing images. We report the results in Table IV. It can be observed that our approach achieves the best performance on the RSOC dataset. Specifically, compared with the second best state-of-the-art method, ASPDNet improves the performance with 1.94% MAE and 7.14% RMSE on RSOC_building subset, 0.47% MAE and 0.77% RMSE on RSOC_small vehicle subset, 35.40% MAE and 33.09% RMSE on RSOC_large vehicle subset and 3.86% MAE and 4.18% RMSE on RSOC_ship subset, respectively. From the evaluation metrics, we can find that even though we have achieved the best performance on the proposed RSOC dataset, there is still a large margin to be improved, especially for small objects such as small-vehicles and ships. This is consistent with other counting tasks that higher congested the scene is, more challenging the counting task will be. This also indicates the challenging nature of the proposed RSOC is and we hope this will encourage more research efforts to put on our dataset.

Figs. 13 depicts the visualization results for some sample images from RSOC_building, RSOC_small vehicle, RSOC_large vehicle and RSOC_ship subsets. It can be observed that our proposed method obtains high-quality density maps with small count errors. It also indicates even under the extreme conditions of scale variation, complex clutter backgrounds, and orientation arbitrariness, our proposed approach still performs strong robustness capacity. Meanwhile, from the generated density maps, we can find that our model has accurate localization ability to some extent.

To further demonstrate the effectiveness and generalization ability of our proposed method, we apply it on one crowd counting dataset, i.e., ShanghaiTech dataset [1]. This dataset is composed of 1,198 annotated images with a total of 330,165 presidents, which is split into two parts, Part_A and Part_B. Part_A contains 482 highly congested images which are randomly crawled from the Internet, where 400 images serve as training and the remaining are for testing. Part_B contains 716 image with relatively sparse people which are taken from the busy streets of metropolitan areas in Shanghai, China, where 400 images are for training and the rest 316 are for testing.

We report the comparison results in Table V. It can be seen that compared with the state-of-the-art methods, our proposed method can obtain the best results, which demonstrates great robustness and generalization ability. Specifically, compared with the baseline method, CSRNet [12], it gains 10.85% / 16.35% (MAE / RMSE) on Part_A, and 32.08% / 34.38% (MAE / RMSE) on Part_B, respectively. Some visualizations of density maps are depicted in Fig. 13, which shows the proposed method still can generate accurate predicts for diverse objects from sparse to dense scenarios.

To validate the stability of our proposed method, following [76], we also report the standard deviations of our methods on the constructed dataset. See Table VI for detail, note that the standard deviations are computed using 5 trials.