CASSOD-Net: Cascaded and Separable Structures of Dilated Convolution for Embedded Vision Systems and Applications

In previous works [3, 6, 7, 9, 11, 14, 17, 20, 22], the dilated filters are generated based on $3\times 3$ filters. A total of 9 filter weights are used to compute the convolution results for one input feature map. The FOV is expanded when the dilation rate $D$ is larger than 1. The larger the dilation rate $D$, the larger the FOV.

The dilated filters which are generated based on $2\times 2$ filters, are proposed in this work. The output result of the proposed dilated filters is calculated based on the following equation.

where $O_{(i,j)}$ is the output feature map, $I_{(c,i,j)}$ is the $c$-th input feature map, and $W_{(c,x,y)}$ is the filter weight for the $c$-th input feature map. The parameters $(i,j)$ represent the index of the pixels in the feature maps, and the ranges of both $i$ and $j$ depend on the size of feature maps. The parameter $C$ denotes the number of channels. There are only 4 filter weights for 1 input feature map and the ranges of both $x$ and $y$ are $[0,1]$.

Two examples of the $2\times 2$ filters and dilated convolutions are shown in Fig. 2. Fig. 2(a) shows an example where the dilation rate $D$ is $2$. Two cascaded $2\times 2$ dilated convolutions where the dilation rate $D$ is $2$ can be an approximation to a $3\times 3$ dilated convolution with the same dilation rate since the positions of zero weights are exactly the same. Similarly, as shown in Fig. 2(b), two cascaded $2\times 2$ dilated convolutions with the dilation rate $D=4$ can be an approximation to a $3\times 3$ dilated convolution with the same dilation rate. The dilation rate $D$ is always set to a multiple of 2.

The concept of the proposed CASSOD module is shown in Fig. 1. The upper part of the figure shows an example of dilated convolutions [19] where the dilation rate $D$ is $2$. There are $C_{1}$ channels in the input feature maps, and $C_{2}$ channels in the output feature maps. The number of filter weights of the dilated convolutions is $3\times 3\times C_{1}\times C_{2}$.

The lower part of the figure shows an example of the proposed CASSOD module where the dilation rate $D$ is $2$. There are two convolution layers in this module and 3 sets of feature maps. Either of the first convolution layer or the second convolution layer can be a depthwise convolution [5] layer. The variations (CASSOD-A,C,D) are shown in Table 1.

In the CASSOD-A module, there are $C_{1}$, $C_{1}$, and $C_{2}$ channels in the first, the second, and the third set of feature maps, respectively. The numbers of channels in the first and the second set of feature maps are the same because the first convolution layer includes depthwise separable operations. The first convolution layer includes a series of $2\times 2$ depthwise and dilated filters, which are different from the traditional $3\times 3$ dilated filters. The second convolution layers includes a series of $2\times 2$ dilated filters. The number of filter weights of the dilated convolutions is $2^{2}\times C_{1}\times(1+C_{2})$. Table 1 also shows the comparison of the number of filter weights. It can be observed that, when $C_{2}$ is large, the number of filter weights in the CASSOD-A module is close to $44.4\%$ (4/9) of the number of filter weights in the traditional dilated convolution.

In the CASSOD-C module, there are $C_{1}$, $C_{1}$ (or $C_{2}$), and $C_{2}$ channels in the first, the second, and the third set of feature maps, respectively. Both convolution layers include a series of $2\times 2$ dilated filters. The number of filter weights of the dilated convolutions is $2^{2}\times(C_{1}+C_{2})\times C_{1}$ or $2^{2}\times(C_{1}+C_{2})\times C_{2}$. It can be observed that, when $C_{1}$ is equal to $C_{2}$, the number of filter weights of the CASSOD-C module is close to $88.9\%$ (8/9) of the number of filter weights in the traditional dilated convolution.

The CASSOD-A,C modules have lower computational costs than the traditional $3\times 3$ dilated convolutions. The number of filter weights is also proportional to the computational cost, which is equal to the number of MAC operations. Compared with the traditional $3\times 3$ dilated convolutions, 1 additional convolution layer is required to implement the CASSOD modules. The batch normalization operations and the activation functions (e.g. ReLU) can be included in the additional convolution layer if necessary. The CASSOD-A,C modules are alternatives to the dilated convolutions, and the CASSOD-D module is an alternative to the depthwise and dilated convolutions. The number of filter weights of the CASSOD-D module is close to $88.9\%$ (8/9) of the depthwise and dilated convolutions.

Table 2 shows an example of the computational time of dilated convolutions when the size of input image is $320\times 320$ pixels. A network with 3 layers is used to measure the computational time of CPU, and a network with 10 layers is used to measure the computational time of GPU. There are 64 input channels and 64 output channels in each layer of the networks, and the dilation rates $D$ of the $3\times 3$ convolution layers are set to the same value.

The results show that the processing time does not change much when the dilation rate $D$ is larger than 1 for either CPU or GPU. The computational time of standard convolutions, where the dilation rate is set to 1, is the shortest, and the reason can be that the framework (e.g. CUDA library) includes some optimized operations to accelerate the computations of standard convolutions. The speed of dilated convolutions may depend on the performance of system platforms, but the dilated convolutions with a dilation rate larger than 1 cannot necessarily achieve the same level of speed as standard convolutions with a dilation rate of 1. The goal of this work is to design a hardware system which can handle both dilated convolutions and standard convolutions efficiently.

For hardware implementation, the pixel values in the feature maps are usually stored in a shift register array. One solution to handle dilated convolutions using a shift register array is to add zeros to the filter weights and compute the results of standard convolutions. To handle a dilated $3\times 3$ filter where the dilation rate $D$ is $2$, it is necessary to compute the products of 25 filter weights and 25 input pixels. A total of 16 filter weights equal to zero can be skipped. The computational time is proportional to the size of dilated filters, and the efficiency of filter processing is relatively low compared with CPU and GPU.

To solve this problem, a new hardware architecture is proposed to speed up dilated convolutions. An example of the architecture of the proposed hardware system is shown in Fig. 3, which includes 6 modules and DRAM. The filter weights stored in the “Filter Weight Memory” are sent to the “Filter Weight Cache,” and the feature maps of the current convolution layer stored in the “Pixel Memory” are sent to the “Pixel Array.” The filter weights and the pixels of feature maps are sent to the “Convolution Processor” to compute the convolution results, and the convolution results are sent to the “Activation and Pooling Unit” to generate the feature maps for the next convolution layer. The “Pixel Array” includes multiple hierarchical stages of pixel caches, which can generate the input pixels for dilated convolutions with a different dilation rate $D$.

The “Pixel Array,” including multiple hierarchical stages, can handle dilated convolutions with different dilation rates efficiently. The proposed hardware architecture can be implemented in a Field-Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC). Fig. 4 shows an example of the architecture and the interconnections of the proposed shift register array, where there are 3 hierarchical stages. In this example, there are $6\times 6$ pixel buffers (or selectors) in each stage, and each pixel buffer (or selector) is connected to 4 other pixel buffers (or selectors) in the up, right, down, and left directions. It means that the pixel stored in the pixel buffer can be transferred to one of the 4 connected pixel buffers (or selectors). In the first hierarchical stage, the input of a pixel buffer is connected to the input of its neighboring pixel buffer and the pixel selector with the same position in the second stage. The difference of the shifted index and the original index of the pixel, $X_{0}$, is 0 or 1 ($2^{0}$). Similarly, in the second hierarchical stage, the difference of the shifted index and the original index of the pixel, $X_{1}$, is 0 or 2 ($2^{1}$), and in the third hierarchical stage, the difference of the shifted index and the original index of the pixel, $X_{2}$, is 0 or 4 ($2^{2}$).

The supported dilation rate, which is equal to the total difference of the shifted index and the original index of the pixel in all hierarchical stages ($X_{0}$, $X_{1}$, $X_{2}$, … , $X_{H-1}$) is shown in the following equation.

where $H$ is the total number of the hierarchical stages. In this example, $H$ is equal to 3, but $H$ can be set to any number to support $2^{H}-1$ dilation rates ($D=[1,2^{H}-1]$) according to the network architectures. When the gate count of each stage is the same, the total hardware cost is proportional to the number of hierarchical stage, $H$.

The proposed network architecture, CASSOD-Net, is evaluated based on the RetinaFace [2], which is designed for face detection tasks. The context module, inspired by the SSH face detector [16], is used to increase the FOV and enhance the rigid context modeling power. To compare the accuracy, the convolution layers in the context modules is replaced by the Feature Enhance Module (FEM), which includes dilated convolution structures, in the Dual Shot Face Detector (DSFD) [8]. Then, the dilated convolutions in the DSFD [20] are replaced by the proposed CASSOD modules. The network architecture of the modified context module is shown in Fig. 5. To keep the computational costs at the same level, the number of input channels of the FEM is changed from 256 to 64, and the number of input channels in the remaining layers is reduced by the same ratio. Fig. 5(a) shows the original context module in the RetinaFace, and Fig. 5(b) shows the modified context module with FEM, which includes dilated convolution layers with a dilation rate ($D$) of 2.

MobileNet-0.25 [5] is used as the backbone network. The accuracy of face detection is shown in Table 3. After replacing the dilated convolutions with the CASSOD-C and CASSOD-A modules, the accuracy does not decrease. Besides, the proposed CASSOD module can achieve higher accuracy than the previous architecture with batch normalization and activation. The CASSOD-C module has better performance than the CASSOD-A module on the 3 categories (easy, medium, and hard), and the reason can be that the CASSOD-C module has more parameters than the CASSOD-A module. It is shown that the CASSOD module is a good alternative to the original dilated convolutions. The accuracy of the CASSOD-A module shows that the new network achieves higher accuracy than the FEM-based network with only 47% of filter weights in the dilated convolution layers of the context module.

The proposed network architecture, CASSOD-Net, is evaluated based on the FastFCN [20], which is designed for semantic image segmentation tasks. The JPU module, which is included in the FastFCN, combines the up-sampled feature maps by using 4 groups of dilated convolutions with different dilation rates. The JPU modules also contain depthwise convolution layers. It can extract multiple-scale context information and increase the accuracy of image segmentation.

To compare the accuracy, the dilated convolutions in the JPU [20] are replaced by the proposed CASSOD-D modules. The network architecture of the modified JPU is shown in Fig. 6. ResNet-50 [4] is used as a backbone network and trained with Pascal Context [15] and ADE20K datasets [21]. The accuracy of image segmentation is shown in Table 4. The results [20] and our re-implementation results are very close. After replacing the dilated convolutions with the CASSOD modules, the accuracy does not decrease. Besides, the proposed CASSOD module can achieve higher accuracy than the previous architecture with batch normalization and activation. It is shown that the CASSOD module is a good alternative to the original dilated convolutions for image segmentation. Also, the proposed CASSOD modules can be applied to networks with depthwise and dilated convolutions. The computational time of GPU increases after applying the CASSOD modules because an extra layer is added. This problem can be solved by using the proposed hardware architecture.

To show the advantages of the proposed hardware system, the hardware architecture and the CASSOD modules are compared with previous works. The previous works refer to the hardware systems [1, 13] which do not accelerate the algorithms with the $3\times 3$ dilated convolutions shown in Sec. 2. Since the hardware systems of the related works are not available, we re-implement a hardware system on our platforms without using the “Pixel Array” shown in Sec. 3.2 for comparison. The results are shown in Fig. 7.

Fig. 7(a) shows the relation between the computational time and the dilation rate ($D$) with $3\times 3$ filters. The number of cycles is equivalent to the computational time. In previous works, since there are no special hardware architectures to handle dilated convolutions, it is necessary to add zero-values to the filter weights after a filter is dilated. The computational time increases as the dilation rate $D$ increases and is roughly proportional to a square of the dilation rate $(D)$. In the proposed hardware system, the input pixels of dilated convolutions can be adjusted according to the dilation rate $(D)$ and dumped consecutively, and the computational time does not vary with the dilation rate $D$. It can be observed that the proposed hardware system can handle both dilated convolutions and standard convolutions ($D=1$) efficiently. When the dilation rate ($D$) of a $3\times 3$ filter is 2, the proposed hardware system is 2.78 times faster than the previous work.

Fig. 7(b) shows the relation between the computational time and the dilation rate ($D$). By replacing the traditional dilated convolutions with the CASSOD module, the computational cost and the parameter size can be reduced even though 1 additional convolution layer is required. In the proposed hardware system, the time to set the parameters for 1 additional layer is relatively small. The results show that the computational time of the proposed hardware system can be further reduced by using the CASSOD module, which achieves similar accuracy as the traditional dilated convolutions.

Fig. 7(c) shows the relation between the gate count of the “Pixel Array” and the maximum dilation rate ($D$). It can be observed that the overhead of hardware cost increases as the dilation rate $D$ increases, but the difference of hardware cost between the previous work and the proposed work is not proportional to the maximum dilation rate, $D$. The proposed hardware system scales well because it can support $2^{3}-1$ dilation rates ($D=[1,7]$) with less than $3$ times of hardware costs.

The supported dilation rates also depend on the filter size and the interface between modules. Table 5 shows the specifications of the proposed hardware architecture. Different from previous works, the proposed hardware system can handle $2\times 2$ and $3\times 3$ dilated convolutions efficiently. The maximum supported dilation rate for $2\times 2$ filters is 6, and the maximum supported dilation rate for $3\times 3$ filters is 3. By replacing the traditional dilated convolutions with the CASSOD module, an approximated $3\times 3$ dilated filters with a dilation rate of 6 can be implemented with cascaded $2\times 2$ filters with a dilation rate of 6. The computational costs and memory footprints can also be reduced. Since the ”Pixel Array” with 3 hierarchical stages only occupies only 21% of the total area, the overhead to support the dilated convolutions is relatively small. A comparison of the processing speed of the proposed system is shown in Table 6. The resolution of the input image is $512\times 512$ pixels, and the network is RetinaFace [2] with FEM [8], which is shown in Table 3. By using the proposed hardware architecture with the shift register array, 23% of processing time can be reduced. By replacing the dilated $3\times 3$ filters with the proposed CASSOD modules, 9% of processing time can be further reduced. The result clearly shows the advantages in terms of computational speed.