DO-Conv: Depthwise Over-parameterized Convolutional Layer

We use $\mathbb{T}_{ijk}$ to denote the $(i,j,k)$-th element of a 3D tensor $\mathbb{T}\in R^{I\times J\times K}$. A tensor can be reshaped without changing the number of its elements, or their values, e.g., a 4D tensor $\mathbb{T}\in R^{I\times J\times K\times L}$ can be reshaped to a 3D tensor $\mathbb{T}^{\prime}\in R^{I\times(J\times K)\times L}$, or $\mathbb{T}^{\prime}\in R^{I\times M\times L}$, where $M=J\times K$. For simplicity, we will refer to a tensor and its reshaped version(s) with the same symbol. Furthermore, $\mathbb{T}_{ijkl}$ and $\mathbb{T}_{i(jk)l}$ refer to the same element of $\mathbb{T}\in R^{I\times J\times K\times L}$ and its reshaped version $\mathbb{T}\in R^{I\times(J\times K)\times L}$, respectively.

Given an input feature map, a convolutional layer processes it in a sliding window fashion, applying a set of convolution kernels to a patch of corresponding size, at each window position. For the purposes of our exposition, it is convenient to think of a patch as a 2D tensor $\mathbb{P}\in R^{(M\times N)\times C_{in}}$, where $M$ and $N$ are the spatial dimensions of the patch,

and $C_{in}$ is the number of channels in the input feature map. The trainable kernels of a convolutional layer²²2For simplicity, the bias terms are not included in our exposition. can be represented as a 3D tensor $\mathbb{W}\in R^{C_{out}\times(M\times N)\times C_{in}}$, where $C_{out}$ is the number of channels in the output feature map. The output of a convolution operator $\ast$ is a $C_{out}$-dimensional feature $\mathbb{O}=\mathbb{W}\ast\mathbb{P}$:

$$\mathbb{O}_{c_{out}}=\sum_{i}^{(M\times N)\times C_{in}}\mathbb{W}_{c_{out}i}\mathbb{P}_{i}.$$

(1)

This is illustrated in Figure 1 (better viewed in color), where $M\times N=4$, $C_{in}=3$ with different input channels in different cube frame colors, $C_{out}=2$ with different output channels in different cube face colors, and each element of $\mathbb{O}$ is produced by a dot-product between one kernel (one horizontal slice of $\mathbb{W}$) and the patch $\mathbb{P}$.

In a convolutional layer, dot products are computed between each of the $C_{out}$ kernels and the entire input patch tensor $\mathbb{P}$. In contrast, in depthwise convolution, each of the $C_{in}$ channels of $\mathbb{P}$ is involved in $D_{mul}$ separate dot-products. Thus, each input patch channel (a $M\times N$-dimensional feature) is transformed into a $D_{mul}$-dimensional feature. $D_{mul}$ is often referred to as depth multiplier. As depicted in Figure 2, the trainable depthwise convolution kernel can be represented as a 3D tensor $\mathbb{W}\in R^{(M\times N)\times D_{mul}\times C_{in}}$. Since each input channel is converted into a $D_{mul}$-dimensional feature, the output of the depthwise convolution operator $\circ$ is a $D_{mul}\times C_{in}$-dimensional feature $\mathbb{O}=\mathbb{W}\circ\mathbb{P}$:

$$\mathbb{O}_{d_{mul}c_{in}}=\sum_{i}^{M\times N}\mathbb{W}_{id_{mul}c_{in}}\mathbb{P}_{ic_{in}}.$$

(2)

This is illustrated in Figure 2 (better viewed in color), where $M\times N=4$, $D_{mul}=2$, $C_{in}=3$ with different input channels in different cube frame colors, and each element of $\mathbb{O}$ is computed by the dot product between each vertical column of $\mathbb{W}$ and the elements in the corresponding channel of $\mathbb{P}$ (those with the same color).

is a composition of a depthwise convolution with trainable kernel $\mathbb{D}\in R^{(M\times N)\times D_{mul}\times C_{in}}$ and a conventional convolution with trainable kernel $\mathbb{W}\in R^{C_{out}\times D_{mul}\times C_{in}}$, where $D_{mul}\geq M\times N$. Given an input patch $\mathbb{P}\in R^{(M\times N)\times C_{in}}$, the output of a DO-Conv operator $\circledast$ is, the same as that of a convolutional layer, a $C_{out}$-dimensional feature $\mathbb{O}=(\mathbb{D},\mathbb{W})\circledast\mathbb{P}$. More specifically, as illustrated in Figure 3, the depthwise over-parameterized convolution operator can be applied in two mathematically equivalent ways as:

	$\displaystyle\mathbb{O}$	$\displaystyle=(\mathbb{D},\mathbb{W})\circledast\mathbb{P}$				(3)
		$\displaystyle=\mathbb{W}\ast(\mathbb{D}\circ\mathbb{P})$		(Fig. 3-a, feature composition)
		$\displaystyle=(\mathbb{D}^{T}\circ\mathbb{W})\ast\mathbb{P},$		(Fig. 3-b, kernel composition),

where $\mathbb{D}^{T}\in R^{D_{mul}\times(M\times N)\times C_{in}}$ is a transpose of $\mathbb{D}\in R^{(M\times N)\times D_{mul}\times C_{in}}$ on the first and second axis. Using feature composition, as depicted in Figure 3-a, the depthwise convolution operator $\circ$ first applies the kernel weights $\mathbb{D}$ to the patch $\mathbb{P}$, yielding a transformed feature $\mathbb{P}^{\prime}=\mathbb{D}\circ\mathbb{P}$, and then a conventional convolution operator $\ast$ applies the kernels $\mathbb{W}$ to $\mathbb{P}^{\prime}$, yielding $\mathbb{O}=\mathbb{W}\ast\mathbb{P}^{\prime}$. In contrast, using kernel composition, as depicted in Figure 3-b, the composition of $\circ$ and $\ast$ is achieved through a composite kernel $\mathbb{W}^{\prime}$, i.e., a depthwise convolution operator first uses the trainable kernels $\mathbb{D}^{T}$ to transform $\mathbb{W}$, yielding $\mathbb{W}^{\prime}=\mathbb{D}^{T}\circ\mathbb{W}$, and then a conventional convolution operator is applied between $\mathbb{W}^{\prime}$ and $\mathbb{P}$, yielding $\mathbb{O}=\mathbb{W}^{\prime}\ast\mathbb{P}$.

Note that the receptive field of DO-Conv is still $M\times N$. This is apparent in Figure 3-b, where the depthwise and conventional convolution kernels are composited to process a single input patch of spatial size $M\times N$. This is perhaps less obvious in the feature composition view (Figure 3-a), where the conventional convolution operator is applied to $\mathbb{P}^{\prime}\in R^{C_{in}\times D_{mul}}$, while $D_{mul}$ could be greater than $M\times N$. Note, however, that the values of $\mathbb{P}^{\prime}$ are obtained by transforming an $M\times N$ spatial patch.

This statement can be easily verified using the kernel composition formulation (Figure 3-b). The linear transformation performed by a convolutional layer can be concisely parameterized by $C_{out}\times(M\times N)\times C_{in}$ trainable weights. However, in DO-Conv, a linear transformation of equivalent expressiveness is parameterized by two sets of trainable kernel weights: $\mathbb{D}\in R^{(M\times N)\times D_{mul}\times C_{in}}$ and $\mathbb{W}\in R^{C_{out}\times D_{mul}\times C_{in}}$, where $D_{mul}\geq M\times N$. Thus, even for the case where $D_{mul}=M\times N$, the number of parameters is increased by $(M\times N)\times D_{mul}\times C_{in}$. Note that the condition $D_{mul}\geq M\times N$ is necessary, otherwise the composite operator $\mathbb{W}^{\prime}$ cannot express the same family of linear transformations as $\mathbb{W}$ in a conventional convolutional layer.

The interface of DO-Conv is same as that of a convolutional layer, thus DO-Conv layers can easily replace convolutional layers in CNNs. Since the $\circledast$ operator defined in Equation 3 is differentiable, both $\mathbb{D}$ and $\mathbb{W}$ of DO-Conv can be optimized with the gradient descent based optimizer that is used for training CNNs with convolutional layers. After the training phase, $\mathbb{D}$ and $\mathbb{W}$ are folded into $\mathbb{W}^{\prime}=\mathbb{D}^{T}\circ\mathbb{W}$, and this single $\mathbb{W}^{\prime}$ is used for the inference. Since $\mathbb{W}^{\prime}$ is in the same shape as the kernel of a convolutional layer, the computation of DO-Conv at the inference phase is exactly same as that of a conventional convolutional layer.

The two ways for computing DO-Conv are mathematically equivalent, but have different training efficiency. The number of multiply and accumulate operations (MACC), is often used for measuring the amount of computation and serves as an indicator of the efficiency. The MACC cost of feature and kernel composition, when they are applied on an feature map in $R^{H\times W\times C_{in}}$ (assuming $stride=1$), can be calculated as follows:

	Feature composition	$\displaystyle:$
	Kernel composition	$\displaystyle:$

where $H$ and $W$ are the height and width of the feature map, respectively. Note that $\mathbb{W^{\prime}}$ is computed once for an entire feature map. The MACC costs of feature and kernel composition depend on the values of the involved hyper-parameters. However, in most cases, since $H\times W\gg C_{out}$ and $D_{mul}\geq M\times N$, kernel composition typically incurs fewer MACC operations than feature composition. Similarly, the memory consumed by $\mathbb{W^{\prime}}$ in kernel composition is typically smaller than that consumed by $\mathbb{P^{\prime}}$ in feature composition. Therefore, kernel composition is preferable for the training phase.

The feature composition of DO-Conv (Figure 3-a) is equivalent to applying a $M\times N$ depthwise convolutional layer on an input feature map, yielding an intermediate feature map in $D_{mul}\times C_{in}$ channels, and then applying a $1\times 1$ convolutional layer on the intermediate feature map. This process is exactly same as that of a depthwise separable convolutional layer, where the $1\times 1$ convolution is often referred to as pointwise convolution. However, the motivation of DO-Conv and depthwise separable convolution is quite different. Depthwise separable convolution is introduced as an approximation and alternative to a conventional convolution for saving MACC to ease the deployment of CNNs especially on edge devices, thus it requires³³3In earlier versions of Tensorflow [1], setting $D_{mul}>M\times N$ in the depthwise separable convolutional layer is considered as an error, though this constraint has been removed since commit 1822073. $D_{mul}<M\times N$, and $D_{mul}=1$ is probably the choice most widely used in practice, for example in Xception [7] and MobileNets [18, 30, 17].

The equivalence between feature and kernel composition holds not only for $D_{mul}\geq M\times N$ (DO-Conv), but also for $D_{mul}<M\times N$ (depthwise separable convolution). We have shown that for DO-Conv, kernel composition often saves MACC and memory, compared with feature composition. For depthwise separable convolution, it is easy to see that feature composition is more economical. While the MACC is highly correlated with the running speed on edge devices that are often computation capability bounded, this might not be the case on NVIDIA GPUs, as they are more memory access bounded. Depthwise separable convolution has a significantly lower MACC cost than a conventional convolution, but it often runs slower on NVIDIA GPUs, which the CNNs are often trained on. To speedup the training of depthwise separable convolution layers on NVIDIA GPUs, kernel composition can be used, after which feature composition can be used for the deployment on edge devices. This handy trick is not the focus of this paper, but a by-product of the feature and kernel composition equivalence that may greatly interest many deep learning practitioners.

Depthwise over-parameterization can not only be applied over a conventional convolution to yield DO-Conv, but also be applied over a depthwise convolution, which leads to DO-DConv. Following the same principle we used for establishing DO-Conv, as shown in Figure 4, DO-DConv also can be computed in two mathematically equivalent ways as:

	$\displaystyle\mathbb{O}$	$\displaystyle=(\mathbb{D},\mathbb{W})\circledcirc\mathbb{P}$				(4)
		$\displaystyle=\mathbb{W}\circ(\mathbb{D}\circ\mathbb{P})$		(Fig. 4-a, feature composition)
		$\displaystyle=(\mathbb{D}^{T}\circ\mathbb{W}^{T})^{T}\circ\mathbb{P},$		(Fig. 4-b, kernel composition)

DO-DConv layer can be used as an alternative to depthwise convolutional layer, which is widely used in MobileNets [18, 30, 17]. Same as in DO-Conv, both $\mathbb{D}$ and $\mathbb{W}$ are used in the training of DO-DConv, while the folded $\mathbb{W^{\prime}}$ is used for the inference. Following the same principle used for yielding DO-Conv and DO-DConv, grouped convolutional layer [23], which is a general case of depthwise convolutional layer and is shown to be effective in ResNeXt [34], can be over-parameterized, yielding DO-GConv. We refer to the over-parameterized (depthwise/group) convolutional layer as DO-Conv for simplicity, in cases where there is no ambiguity.

Note that when $M\times N=1$, the underlying convolution is a pointwise convolution. In this case, there is no spatial patch for the application of the depthwise convolution, thus DO-Conv is not applied. When $D_{mul}=M\times N$, the kernel of the underlying convolution $\mathbb{W}$ is of the same shape as the conventional one, and the over-parameterizing kernel $\mathbb{D}\in R^{(M\times N)\times D_{mul}\times C_{in}}$ can be viewed as $C_{in}$ square $D_{mul}\times D_{mul}$ matrices. We initialize these square matrices to identity $\mathbb{I}$, such that $\mathbb{W}^{\prime}=\mathbb{W}$ in the beginning. In this case, the over-parameterization does not initially change the operation of the original CNN. It also facilitates DO-Conv by reusing the parameters from the pre-trained model. Moreover, since the diagonal elements in $\mathbb{D}$ could be overly suppressed by $L_{2}$ regularization, we optimize $\mathbb{D}^{\prime}=\mathbb{D}-\mathbb{I}$, where $\mathbb{D}^{\prime}$ is initialized with zeros, instead of optimizing $\mathbb{D}$ directly. Unless otherwise noted, we use $D_{mul}=M\times N$ in our experiments.

The performance of a CNN can be affected by a wide range of factors. To demonstrate the effectiveness of DO-Conv, we take several notable CNNs as baselines, and merely replace the non-pointwise convolutional layers with DO-Conv, without any change to other settings. In other words, the replacement is the one and only difference between the baseline and our method. This guarantees that the observed performance change is due to the application of DO-Conv, but not other factors. Furthermore, this also means that no hyper-parameter is tuned to favor DO-Conv. Following this protocol, we demonstrate the effectiveness of DO-Conv on image classification, semantic segmentation and object detection tasks on several benchmark datasets.