Condensation-Net: Memory-Efficient Network Architecture with
Cross-Channel Pooling Layers and Virtual Feature Maps

Figure 3 shows the comparison of spatial pooling operations and cross-channel pooling operations. An illustration of spatial pooling is shown in Figure 3(a), where the stride and the window size are both 2 pixels (2 $\times$ 2 pixels) in the spatial domain. The size of input feature map is reduced from 4 $\times$ 4 pixels to 2 $\times$ 2 pixels in the spatial domain. The number of feature maps does not change. An illustration of cross-channel pooling is shown in Figure 3(b), where the stride and the window size are both 2 pixels in the channel direction. In this example, the window size for cross-channel pooling has no connection with the spatial domain. The number of input feature maps is 2, and the number of output feature maps is reduced to 1. The resolution of feature maps does not change after cross-channel pooling. The operations of spatial pooling and cross-channel pooling are similar, but the numbers of filter weights required to compute the input feature maps for the 2 pooling algorithms can be different. The number of filter weights for a convolution layer with cross-channel pooling operations might be 2 times larger than the one with spatial pooling operations. The output pixel of cross-channel pooling can be computed based on the max operations, the average operations, the min operations, and so on.

Figure 4 shows the proposed algorithms. There are 4 levels in the nested loop structure. In the 1st level of loops, each convolution layer of the networks is processed sequentially, and the parameters of the corresponding layer are set. In the 2nd level of loops, each output channel of the layer is processed sequentially. In the 3rd level of loops, each block of the output channel is processed sequentially. In the 4th level of loops, each input channel is processed sequentially. The filter weights of the $i$-th convolution layer, the $j$-th output channel, the $n$-th input channel are set, and the convolution results of $\alpha$ output blocks are computed. If cross-channel pooling operations are enabled, the output block to the next layer is calculated and the number of feature maps becomes 1 since the output blocks are reduced by a factor of $\alpha$ in the channel direction. Otherwise, the output result to the next layer includes the convolution results of $\alpha$ blocks (channels) and no operations are required.

Figure 5 shows the relation between the “virtual feature maps” and the feature map memory. There are $N_{\mathsf{ch},i}$ input feature maps (channels) in the upper part of the feature map memory and $N_{\mathsf{ch},i+1}$ output feature maps in the lower part of the feature map memory. A block is extracted from the input feature maps, and the result of convolution and activation, which is a part of virtual feature maps, is generated. There are $\alpha N_{\mathsf{ch},i+1}$ virtual feature maps in total. After cross-channel pooling, the number of output feature maps is reduced to $N_{\mathsf{ch},i+1}$, and a block of the output feature maps is generated. The same operation are repeated until all blocks are processed.

The memory can store $N_{\mathsf{ch},i}$ channels for the input and $N_{\mathsf{ch},i+1}$ channels for the output. If all the virtual feature maps are stored, we need an extra storage for $\alpha N_{\mathsf{ch},i+1}$ channels. Since the result of cross-channel pooling can be calculated partially, it is not necessary to store all virtual feature maps. By employing the proposed algorithm, only the storage of 1 block is required to compute the result of cross-channel pooling. The results of cross-channel pooling are also called virtual feature maps since it is not necessary to store them in the physical memory.

The activation functions mentioned in the previous sections can be implemented by using Rectified Linear Unit (ReLU) functions, sigmoid functions, tangent functions, leaky ReLU functions, and so on. When the bit width of feature maps and filter weights are small, the activation functions can also be implemented with quantization functions [22, 2]. The zeros in the feature maps can be removed by using cross-channel pooling operations without losing much information especially for low-bit networks (e.g. 1 bit or 2 bits) since the pixels of feature maps contain lots of zeros.

The proposed network with cross-channel pooling can be trained by using the back-propagation algorithm. The following is an example of the training algorithm for cross-channel pooling with the max operations.

For a cross-channel pooling layer with a stride of 2 pixels and a window size of 2 pixels in the channel direction, the maximum value of 2 input pixels is computed in the forward-propagation process, and the corresponding channel which has the maximum values is recorded. Then, the gradient value is sent to the recorded channel of the cross-channel pooling layer in the back-propagation process. The steps of forward-propagation process and back-propagation process are repeated until the training results converge.

We evaluate the 2 networks shown in Table 1. The first one is Tiny-YOLOv2, which is a compact network for object detection [14, 18]. The second one is the proposed Condensation-Net, where the parameters are extended from Tiny-YOLOv2. In Condensation-Net, the 1st – the 4th convolution layers are replaced by the combinations of convolution layers and cross-channel pooling layers shown in Figure 1. The reason to choose the 1st – the 4th convolution layers is that the memory cost of filter weights in these layers is relatively small compared to the total cost. In the 1st convolution layer, the filter size of Condensation-Net is $16\alpha\times 3\times 3$, which means that the number of output channels, $N_{\mathsf{ch},1}$, is $16\alpha$, and both the width and the height are 3. The number of input channels is 3 since there are 3 channels (RGB) in the input image. For Condensation-net, the value of $\alpha$ is set to 2 or 4, and Tiny-YOLOv2 can be regarded as a special case of Condensation-Net when $\alpha=1$. The cross-channel pooling layers can be added to different layers according to the requirements of applications.

The images in the training set of Wider Face [19] are used to train the 2 networks, Tiny-YOLOv2 and Condensation-Net. We compare the accuracy of the networks on the Face Detection Data Set and Benchmark (FDDB) [8], which contains 5,171 faces in 2,845 test images. The ROC curves of the networks are shown in Figure 7, which shows that Condensation-Net achieves better detection rate than Tiny-YOLOv2 especially when the number of false positive is small. To further evaluate the performance of the face detector, we compare the detection rate (true-positive rate) when the false-positive rate is 0.1 (1 false positive / 10 test images), which corresponds to $\lfloor 2,845\times 0.1\rfloor=284$ faces in 2,845 test images. The comparison results are shown in Table 2.

As mentioned in Sec. 1, one method to reduce the memory size is quantization. We quantize the network using the Half-Wave Gaussian Quantization (HWGQ) algorithm, where full-precision filter weights and full-precision feature maps are replaced with 1-bit filter weights and 2-bit feature maps, respectively [2]. For quantized networks and full-precision networks, Condensation-Net achieves higher accuracy than Tiny-YOLOv2 because Condensation-Net contains more filter weights than Tiny-YOLOv2. It means that the proposed cross-channel pooling layers can be applied to either quantized networks or full-precision networks to increase the accuracy. However, when $\alpha$ is increased from 2 to 4, the number of filter weights in the 1st – the 4th convolution layer doubles, but the accuracy decreases. The reason can be that some information contained in the parameters, which is essential to increasing the accuracy of face detection, is removed by cross-channel pooling. Similar to spatial pooling operations, where the window size and the stride are set to $2\times 2$ pixels for many applications, the parameter $\alpha=2$ is found to be the optimal value for the proposed cross-channel pooling on face detection tasks according to the experimental result.

In the spatial pooling layers, we re-use the parameters of Tiny-YOLOv2 and employ the max operations to compute the result of spatial pooling. However, to compare the accuracy, we employ the max operations and the average operations to compute the result of cross-channel pooling in the 1st – the 4th convolution layers. The accuracy of these two kinds of operations is similar, but the max operations achieve slightly better performance than the average operations except for the full-precision network with $\alpha=2$. According to the experimental results, we employ the max operations for cross-channel pooling on face detection tasks.

The comparison of the required memory size of the proposed hardware architecture for different networks is shown in Table 3. As shown in Figure 6, the proposed hardware architecture stores the data of 2 consecutive feature maps and all of the filter weights in the network. The maximum size of input images is $512\times 512$ pixels $\times 8$ bits. For the convolution layers where $\alpha$ is set to 2 or 4, we only need to store $1/\alpha$ of feature maps after convolution operations because of cross-channel pooling. The size of the weight memory is 1,924 KB for Tiny-YOLOv2, and the sizes of the weight memory are 1,935 KB and 1,959 KB for Condensation-Net when $\alpha=2$ and $\alpha=4$, respectively. The memory sizes of the feature map memory for Tiny-YOLOv2 and Condensation-Net ($\alpha=2$ or $\alpha=4$) are both 4,096 KB. They are exactly the same since we do not have to store the virtual feature maps as mentioned in Sec. 2.2. The numbers of feature maps of Tiny-YOLOv2 and Condensation-Net are different after convolution operations, but they become the same after applying cross-channel pooling to Condensation-Net. The memory sizes of the weight memory for Tiny-YOLOv2 and Condensation-Net are slightly different because Condensation-Net requires more filter weights than Tiny-YOLOv2 to compute the results of convolution. However, as shown in Table 1, since the numbers of channels in the 1st – the 4th convolution layers are small, the difference of filter weights between the 2 networks is also small.

As shown in Table 2 and Table 3, the total size of the weight memory and the feature map memory for Tiny-YOLOv2 is 6,020 KB, and the detection rate of face detection is 89.87%. By using the proposed techniques, we can increase the detection rate by adding memory storage with negligible costs. The total size of the weight memory and the feature map memory for Condensation-Net ($\alpha=2$) is 6,031 KB, but the detection rate of face detection is 91.82%, which is higher than Tiny-YOLOv2. The overhead of the proposed hardware architecture is 11 KB, which is only 0.2% of the total memory size.

Figure 8 shows the comparison of the required memory size of all layers. In our hardware architecture, we only need to store the feature map of 2 consecutive layers, but the memory access of feature maps in all layers is inevitable. When $\alpha=2$, the required memory size of Condensation-Net is 9,788 KB, but we can reduce the memory access to 7,740 KB with the virtual feature maps. It means that we can save 26.5% of memory bandwidth. The larger the feature map, the more memory access can be saved.

The specifications of the proposed hardware architecture, which is designed to process the quantized Condensation-Net ($\alpha=2$), are shown in Table 4. We use the 28-nm CMOS technology library for the experiments. The performance is 320 Multiply-Accumulate operations (MACs) per clock cycle for 2-bit feature maps and 1-bit filter weights, which means that the value of $M$ is set to 320. As shown in Figure 6, since the cross-channel pooling operations can be enabled or disabled in different layers, the hardware architecture can also process Tiny-YOLOv2, which includes no cross-channel pooling operations. The gate counts of the CLPU and the PLPU are 197K and 2K, respectively. The gate count of the PLPU is a small number compared to the gate count of the whole hardware architecture, 199K. It means that we can support cross-channel pooling functions by increasing only 1.0% of the hardware resources. Compared with Tiny-YOLOv2, it takes 1.3 times of processing time to compute the inference result of Condensation-Net. Since the proposed hardware can support both Tiny-YOLOv2 and Condensation-Net, the target networks can be switched according to the target processing time.

The comparison of hardware architectures for low-bit quantized CNNs is shown in Table 5. It is difficult to compare the proposed hardware architecture with the related works [10, 21] because they are implemented on FPGA platforms. Since the proposed architecture is synthesized with the cell-based design library, it generally achieves higher clock frequency than FPGAs. The proposed hardware architecture can achieve 7.72 GOPS / kLUT for 1-bit – 2-bit operations. The performance density is higher than the related work [21] without considering the cost of DSPs in FPGA, and the supported bit width is larger than the related work  [10].