High-Order Paired-ASPP Networks for Semantic Segmentation

A encoder generally reduces the spatial size of feature maps to get semantic context information and enlarge the receptive field. Then the decoder recovers the predictions. As the first encoder-decoder approach on semantic segmentation, FCN [1] shows good performance. It discards the fully connected layer to support input of arbitrary sizes. Based on FCN [1], many works change the upsample methods to generate better results. The SegNet [17] stores the max-pooling indices of the feature maps and uses them in its decoder network to make upsample convincing. The DeconvNet [18] uses deconvolutions to perform the decoding pass. Besides, multi-level feature fusion in semantic segmentation is a hot topic, many encoder-decoder methods try to solve this problem. The U-Net [19] gradually fuses high-level low-resolution features with low-level but high-resolution features, which is helpful for decoder to generate better results. This architecture is adopted in many works [20, 21, 22]. RefineNet [22] exploits all the information available along the down-sampling process to enable high-resolution predictions using long-range residual connections. ICNet [23] incorporates multi-resolution branches under proper label guidance to address the challenge of real-time semantic segmentation. FCDenseNet [24] adjust DenseNet [25] for semantic segmentation. ExFuse [12] assigns auxiliary supervisions to all stages to force low-level features to encode more semantic concepts and high-level features to learn more spatial information. DFN [15] uses two sub-networks, one smooth network and one border network to select the more discriminative features for later segmentation. Besides, Conditional Random Field [26] can be used in encoder-decoder structures for semantic segmentation. Combining with CNNs, the methods [27, 28, 29, 30, 31] adopted this strategy, making the deep network end-to-end trainable. The encoder-decoder structure guarantees the competitive performance. However, it is difficult to discover disciminative feature and fuse the different level features using only simple encoder-decoder structure.

Context embedding has shown promising improvements on semantic segmentation. GCN [32] utilizes $1\times k$, $k\times 1$ convolution to constitude a global convolution block. ParseNet [33] uses global pooling to clarify local confusions. Both of these two methods are intended to enlarge the receptive field and capture context information. Recently, self-attention mechanism [8] is widely used for semantic segmentation to get global dependencies. The self-attention [8] is originally proposed to solve the machine translation [34], [35], and the following work [36] further proposed the non-local neural network for various tasks such as object detection, video classification and instance segmentation. DANet [9] appends two types of attention modules which model the semantic interdependencies in spatial and channel dimensions. CCNet [10] harvest the contextual information of its surrounding pixels on the criss-cross path through a novel criss-cross attention module. As variants of self-attention structure, EMANet [11] and ANN [37] present new self-attention modules to reduce the computation and memory consumption without sacrificing the performance. To extract the multi-scale information and global context efficiently, SPP [29] based methods are proposed. Deeplabv2 [30] proposes ASPP module, where parallel atrous convolution layers with different rates to capture multi-scale information, then concatenate the different scales features to fuse the local and global context. PSPNet [6] performs spatial pooling at several grid scales to aggregate global contextual information, the local and global clues together make the final prediction more reliable. The ASPP module is improved in DeepLabV3 [5] by integrating a global pooling branch. DenseASPP [38] connects a set of atrous convolutional layers in a dense way, such that it generates multi-scale features that not only cover a larger scale range, but also cover that scale range densely, besides the module doesn’t increase the computation cost too much. All of the SPP [29] variants are stacked at the top of their backbone networks for prediction, which ignores the spatial information. These works achieve great performance on semantic segmentation. However, the low-level and high-level features in these methods are first-order and lack of spatial information. Thus, we propose the High-Order Representation module and Paired-ASPP module to exploit high-order statistics and capture more discriminative spatial information.

Recently, some works of image classification and recognition [39, 40, 41, 42, 43] show that the integration of high-order statistics significantly improves their performance. Global high-order pooling is utilized to aggregate the information and represent whole images [41, 42, 43], in which the sum of outer product of convolutional features is firstly computed, then element-wise power normalization [42], matrix logarithm normalization [41] and matrix power normalization [43] are performed. G2DeNet [44] adds a trainable layer of one global Gaussian as an image representation into CNNs, which exploits first-order and second-order information. All these methods generate very high dimensional representations, which is not suitable for other tasks such as semantic segmentation. Some works [39, 40] adopt polynomial and Gaussian RBF Kernel functions to generate low-dimensional high-order representations. MLKP [45] proposes a location-aware kernel approximation method to exploits high-order statistics for improving performance of object detection. MHN [46] proposes the High-Order Attention module, using complex and high-order statistics to increase the discrimination of proposals. It improves the performance of person-ReID task. All these methods only extract high-order information from single scale. In this paper, we generate high-order features from different layers and fuse them. Therefore, our high-order statistics contain plenty of spatial details and semantic concepts. To the best of our knowledge, our method is the first work to apply high-order information in semantic segmentation.

With the Eq 3, we can calculate any arbitrary order of representation by learning the parameters of weight $\mathrm{w^{1}}$, $\alpha^{r}$ and $u_{s}^{r,d}$. For the $z^{r,d}=\prod_{s=1}^{r}\langle u_{s}^{r,d},\mathrm{x}\rangle$, we can first deploy $\{u_{s}^{r,d}\}_{d=1,\cdots,D_{r}}$ as a set of $D_{r}$ $1\times 1$ convolutional filters on $\mathcal{X}$ to generate a set of feature maps $\mathcal{Z}_{s}^{r}$ of dimension $D^{r}\times M\times N$. In general, the dimension of $D^{r}$ should be large enough to make a good approximation. However, in practice, we should use a relative low dimension because the high-dimension increases the computational cost and causes the over-fitting problem. The feature maps $\{\mathcal{Z}_{s}^{r}\}_{s=1,\cdots,r}$ are combined by element-wise product to get $\mathcal{Z}^{r}=\mathcal{Z}_{1}^{r}\odot\cdots\odot\mathcal{Z}_{r}^{r}$. The $\{u_{s}^{r,d}\}_{d=1,\cdots,D_{r}}$ can be regarded as a polynomial module to learn degree-r polynomial features.

Implementation: As shown in Figure 2, the HR module of $R=3$ can be easily implemented. At each stage of the backbone, we can get a feature map $\mathcal{X}$. Then we adopt $\{u_{s}^{r,d}\}_{d=1,\cdots,D_{r}}$ as a set of $1\times 1$ convolution layers on $\mathcal{X}$. So we can get $6$ representations with channels $D^{r}$, $\mathcal{Z}_{1}^{1},\cdots,\mathcal{Z}_{3}^{3}$. These feature maps are combined by element-wise product to generate $\mathcal{Z}^{1},\mathcal{Z}^{2}$, and $\mathcal{Z}^{3}$. And $\alpha^{1},\alpha^{2},\alpha^{3}$ can also be implemented by three $1\times 1$ convolution layers. Finally we concatenate the three feature maps as the third-order statistics of this stage.

In a general encoder-decoder structure, high-level features of the encoder output depict the semantic information of the input images. Although the highest-level features contain much semantic information, it is also necessary to make use of the low-level features. Neglecting low-level features leads to the lack of multi-scale information and concrete spatial details. To extract discriminative information from features in shallow layers, we introduce the Paired-ASPP (PA) module. The PA module collects high-order statistics from all stages of the backbone and fuses them via a pairing manner.

The normal ASPP module applies four parallel atrous convolutions with different atrous rates on the last stage. This enable the ASPP module to extract multi-scale information and whole image context. We argue that only using the highest-level feature is inappropriate. Therefore, we combine the high-order statistics of the first three stages to the last stage in a pairing manner. In this way we can capture more scales information than normal ASPP while retaining the spatial information from low-level layers.

As shown in Figure 4, we get four feature maps $\mathcal{Y}_{1}$, $\mathcal{Y}_{2}$, $\mathcal{Y}_{3}$, and $\mathcal{Y}_{4}$ where $\{\mathcal{Y}_{s}\}_{s=1,\cdots,4}$ is the high-order representation of each stage. Then we concat the first three feature maps $\mathcal{Y}_{1},\mathcal{Y}_{2},\mathcal{Y}_{3}$ with $\mathcal{Y}_{4}$ in a pairing manner:

We can regard the $\mathcal{V}_{1,4},\mathcal{V}_{2,4},\mathcal{V}_{3,4}$ as multi-view information of the scene, each of which contains two scales high-order statistics. We apply four atrous $3\times 3$ convolutions with $rates=(3,6,12,18)$ and one global pooling on $\mathcal{V}_{1,4},\mathcal{V}_{2,4},\mathcal{V}_{3,4}$ and $\mathcal{Y}_{4}$. In this way, each atrous convolution can capture information of two scales. We want to use different atrous rates convolutions to cover the span of scales as large as possible. Therefore, we apply atrous convolution with the biggest rate 18 on $\mathcal{V}_{3,4}$ since the receptive field of $\mathcal{Y}_{3}$ and $\mathcal{Y}_{4}$ is large. Meanwhile, atrous convolution with the small rates ${6,12}$ is employed on $\mathcal{V}_{1,4},\mathcal{V}_{2,4}$. The main reason is that low-level features need low-rate atrous convolutions to maintain their spatial information. Moreover, we apply normal convolution and global average pooling on original $\mathcal{Y}_{4}$. The normal convolution also extracts one scale information and the global average pooling captures global context information.

Then we concatenate $\{\mathcal{U}_{i}\}_{i=1,\cdots,5}$. It is followed by several convolutional layers with batch normalization and ReLU activation. Finally the fused features are fed into segmentation layer to give the prediction.

The architecture of our High-Order Paired-ASPP Network is shown in Figure 3. We choose the ResNet [48] as our backbone network. We remove the last two stages with atrous convolution and make both strides to 1 in order to keep the spatial information. Each stage of the ResNet has an output $\mathcal{X}_{i}$. The four feature maps $\{\mathcal{X}_{i}\}_{i=1,2,3,4}$ are used as the input for four High-Order Representation modules (R=3) to capture high-order statistics $\{\mathcal{Y}_{i}\}_{i=1,2,3,4}$. Then, all the high-order statistics $\{\mathcal{Y}_{i}\}_{i=1,2,3,4}$ are sent to the Paired-ASPP module for multi-scale feature extraction and feature fusion. All the convolutional layers are followed with batch normalization and ReLU activation. In the Paired-ASPP module, all the feature maps are concatenated finally. And we will use it for final semantic segmentation. Comparing with other methods which use segmentation heads, our proposed method models the high-order disparity between confusing objects. Besides we take full advantage of low-level features, making high-level features more comprehensive and discriminative. The entire network is trained in an end-to-end manner driving by cross-entropy loss defined on the segmentation benchmarks.

In this section, we analyze the High-Order Representation module and High-Order Paired-ASPP module. Ablation experiments are conducted on the validation set of Cityscapes with only single scale testing if not specified.

We provide the visualization of results in Figure 5 to further illustrate the effectiveness of the High-Order Representation module. We choose some challenging region (the $whitecircles$ in the images) to illustrate that the confusing regions can be progressively corrected with the order increased. It proves the effectiveness of high-order statistics for semantic segmentation.

In Figure 6, we visualize the scales coverage of combination-1 and combination-2. The receptive field becomes larger when the network goes deeper and feature map becomes smaller. Therefore, if we apply atrous convolution with rate 18 on $\mathcal{V}_{1,4}$, we can not obtain the smallest scale information as shown in combination-2 of Figure 6. In combination-2, some similar scales information are repeatedly captured. By adopting the combination-1, we can cover the span of scales as large as possible, making features more comprehensive and discriminative.

As shown in Table VII, the two modules improve the performance remarkably. Compared with the DeepLabV3 (ResNet-50), employing PA module yields a result of $79.11\%$ in mIoU, which brings $1.2\%$ improvement. Meanwhile, employing HR module further improve the performance by $0.77\%$. Moreover, when we adopt the deeper pre-trained ResNet-101, the network with two modules significantly improves the segmentation performance over the DeepLabV3 by $1.83\%$. Results show that PA and HR modules bring great benefit to scene segmentation.

At last, we show the experiments results with different settings in Table VIII. We can see that the network with two modules significantly improves the segmentation performance over the DeepLabV3 module by $1.83\%$. The results demonstrate the efficacy of the proposed High-Order Representation module and High-Order Paired-ASPP module.