Adaptive Dilated Convolution For Human Pose Estimation

Multi-scale fusion is widely adopted in many high-level vision tasks, such as detection [13, 14, 15], semantic segmentation [16], etc. On the one hand, these tasks involve both location and classification. They need multi-scale information to make a balance between these two sub-tasks. On the other hand, these tasks need to tackle objects of various sizes. They scale-invariant representations to get more stable performances. In these tasks, most methods [7, 13, 16] firstly extract a feature pyramid, which contains feature maps of different spatial sizes, and then fuse feature maps to obtain multi-scale information. However, as we have discussed above, the fused features may be not well spatially-aligned. This non-alignment may hurt the accuracy of location. For detection and segmentation, this influence could be ignored, because their evaluation metrics (IOU) are less sensitive to the accuracy of location. While HPE methods are evaluated by OKS, which will be heavily influenced by pixel-level errors. Thus the non-alignment may restrict the performance of HPE methods. In the proposed adaptive dilated convolution, multi-scale features are of the same spatial sizes, which may be more friendly to human pose estimation.

The main idea of dilated convolution is to insert zeros between pixels of convolution kernels. It is widely used in segmentation [17, 18] to enlarge the receptive fields while keeping the resolutions of feature maps. As the size of its receptive field can be easily changed by adjusting its dilation rate, dilated convolution is also used to aggregate multi-scale context information. For example, in [11], the outputs of convolutional layers with different dilation rates are fused to exploit multi-scale context information. And in [12], a similar idea is adopted in an atrous spatial pyramid pooling (ASPP) module. However, these dilation rates of different layers are manually set and can only be integers, which are not flexible enough. Instead, the dilation rates in ADC can be fractional and are adaptively generated, which enables it to learn more suitable receptive fields for objects of various sizes. Besides, every dilation group in ADC can represent features at a scale, which enables ADC to fuse richer multi-scale information yet in a simpler way.

As shown in Figure 3, original dilated convolution can be decomposed into two steps: 1) sampling according to a index set $\mathcal{I}$ over the input feature map $\mathbf{x}$; 2) matrix multiplication of the sampled values and convolutional kernel $\mathbf{w}$. The index set $\mathcal{I}$ is defined by the dilation rate $r$ and size of kernel $k\times k$:

$$\begin{gathered}\mathcal{I}=\{(i\cdot r,j\cdot r,c)\},\\ s.t.\quad\lfloor-k/2\rfloor\leq i,j\leq\lfloor k/2\rfloor,\quad 0\leq c<C_{in},\end{gathered}$$

(1)

where $C_{in}$ is the number of channels in $x$, $\lfloor\cdot\rfloor$ denotes rounding down to the nearest integer. Specially, if $k=3$ and $C_{in}=1$ then

	$\displaystyle\mathcal{I}=$	$\displaystyle\{(-r,-r,0),(-r,-r+1,0),\dots,$		(2)
		$\displaystyle(r,r-1,0),(r,r,0)\}.$

For value at location $(i,j,c)$ of the output feature map $\mathbf{y}$, we have

$$\mathbf{y}(i,j,c)=\sum_{\Delta\mathbf{p}\in\mathcal{I}}\mathbf{w}^{c}(\Delta\mathbf{p})\cdot\mathbf{x}((i,j,0)+\Delta\mathbf{p}),$$

(3)

where $\Delta\mathbf{p}$ enumerates the indexes in $\mathcal{I}$, and $\mathbf{w}^{c}$ denotes the corresponding convolutional kernel for the $c^{th}$ output channel.

The receptive field for each channel in convolutional layer is defined as the square covered by index set $\mathcal{I}$. In original dilated convolution, the receptive fields of all channels are the same. Their sizes are:

	$\displaystyle Area=$	$\displaystyle(\lfloor k/2\rfloor\cdot r-\lfloor-k/2\rfloor\cdot r)^{2}$		(4)
	$\displaystyle=$	$\displaystyle(kr-r+1)^{2}.$

Specially, when $r=1$, the size of receptive field is $k^{2}$.

In adaptive dilated convolution, the dilation rates are no longer manually set. As shown in Figure 2, we use a dilation-rates regression module (DRM) to adaptively generate the dilation rates for different channels. DRM consists of a global average pooling layer and two fully connected layers with nonlinear activations. Suppose DRM is denoted as a function $\phi(\cdot)$, then generated dilation rate $\mathbf{r}$ is

$$\mathbf{r}=\phi(\mathbf{x}).$$

(5)

We divide the input channels into $g$ dilation groups. Each group contains $C_{in}/g$ channels. The channels in the same group shares the same dilation rate. Thus the shape of $\mathbf{r}$ is $g\times 1$. And the dilation rate of the $c^{th}$ input channel is $\mathbf{r}^{\lfloor c/g\rfloor}$. If $g=C_{in}$, then each channel has its own dilation rate. If $g=1$, then all channels share the same dilation rate.

Consequently, the sampling index set becomes

$$\begin{gathered}\mathcal{I}=\{(i\cdot\mathbf{r}^{\lfloor c/g\rfloor},j\cdot\mathbf{r}^{\lfloor c/g\rfloor},c)\}\\ s.t.\quad-k/2\leq i,j\leq k/2,\quad 0\leq c<C_{in}.\end{gathered}$$

(6)

In cases where $\mathbf{r}$ is fractional, as shown in Figure 3, we use bilinear interpolation to get the sampling values. Suppose $M(\mathbf{x},(i,j,c))$ denotes the interpolated value at $(i,j,c)$ on $\mathbf{x}$, then we have:

$$\mathbf{y}(i,j,c)=\sum_{\Delta\mathbf{p}\in\mathcal{I}}\mathbf{w}^{c}(\Delta\mathbf{p})\cdot M(\mathbf{x},(i,j,0)+\Delta\mathbf{p}).$$

(7)

Similarly, in the $c^{th}$ channel of adaptive dilated convolution, the size of receptive field is:

	$\displaystyle Area=$	$\displaystyle(\lfloor k/2\rfloor\cdot\mathbf{r}^{\lfloor c/g\rfloor}-\lfloor-k/2\rfloor\cdot\mathbf{r}^{\lfloor c/g\rfloor})^{2}$		(8)
	$\displaystyle=$	$\displaystyle(k\cdot\mathbf{r}^{\lfloor c/g\rfloor}-\mathbf{r}^{\lfloor c/g\rfloor}+1)^{2}.$

Consequently, different dilation groups have different sizes of receptive fields. And thus ADC can fuse multi-scale information in a single layer.

Since all involved operators are numerical differentiable [19, 20], the proposed adaptive dilated convolution can be easily plugged into existing models trained end-to-end by standard back-propagation.

We plug ADC into the backbones of frequently used HPE models, including the family of SimpleBaseline [22] and HRNet [4]. Their backbones are built up with residual blocks [23]. As shown in Figure 4, we replace one ordinary convolution layer in the original residual block by ADC. The weights of the last layer in the dilated-rates regression module are initialized as zeros and its bias are initialized as ones. Thus, the generated dilation rates in ADC are initialized are ones. The dilation groups $g$ are set as $g=C_{in}$, in which case each group contains only one channel. Thus every channel can exploit context information at different scales, and ADC could fuse as much richer multi-scale information as it can. We also experimentally demonstrate that the performance is positively correlated to $g$ in Sec IV-A4.

The models are optimized by Adam [26] optimizer, and the initial learning rate is set as $1\times 10^{-3}$. For the family of HRNet, each model is trained for 210 epochs and the learning rate decays to $1\times 10^{-4}$ and $1\times 10^{-5}$ at $170th$ and $200th$ epoch respectively. For the family of SimpleBaseline, each model is trained for 140 epochs and the learning rate decays to $1\times 10^{-4}$ and $1\times 10^{-5}$ at $90th$ and $110th$ epoch respectively. All models are trained and tested on $4$ Tesla V100 GPUs. More details can be referred to the Github repository Pose^***https://github.com/leoxiaobin/deep-high-resolution-net.pytorch.git.

Similar to human pose estimation, semantic segmentation also requires rich multi-scale information to make a balance between local and semantic features. Thus the proposed ADC should also benefit the performance of semantic segmentation models. In this section, we plug ADC into different models to demonstrate its benefits on semantic segmentation.

We use CityScapes [28] as our training (2975 images) and validation (500 images) datasets. We use FCN [29], PSANet [30], DeepLabV3 [12] and DeepLabV3+ [17] as our baseline models. The input sizes are set as $769\times 769$. All models are trained for $40K$ iterations. More details can be referred to the Github repository mmsegmention^†††https://github.com/open-mmlab/mmsegmentation.git. As shown in Table IV, ADC can bring consistent improvements to different Semantic Scene Parsing models. For FCN, ADC even improves the mIOU by $4.27$.