A Structurally Regularized CNN Architecture via Adaptive Subband Decomposition

Fig. 1 depicts the ASD structure with $M$ layers. Similar to $HP$ and $LP$ in WSD [30], the $U$ and $L$ blocks represent decomposition filters. Specifically, each block comprises a collection of $c$ filters operating in parallel on the $c$ channels of the input. The filter for the $i$th channel, $i=1,\ldots,c$, is a real 2-dimensional filter of order $(2N+1)\times(2N+1)$, where $N$ is an integer, with impulse response denoted by $W(k,l;i)$. Representing the input by a tensor of dimensions $a\times b\times c$, $X(m,n;i)$, the output tensor $Y(m,n;i)$ is given by the convolution sum

The $U$ and $L$ blocks are followed by decimation-by-2 along the $x$ and $y$ dimensions only. Hence, at each layer, each channel of the input is split into four subbands with half the resolution along each of the $x$ and $y$ dimensions. This filtering-and-decimation process gets repeated for each subband as we move to the next decomposition layer. For example, for an input of size 224$\times$224 with 3 channels, i.e., 224$\times$224$\times$3, an a 1-layer subband decomposition provides four subbands with dimensions 112$\times$112$\times$3. For 2-layer decomposition, the same input yields 16 subbands of dimensions 56$\times$56$\times$3 each. We note that the net dimension of the input and the final stack of the generated subbands are the same.

The network is end-to-end trained by back-propagating the error gradient up to the subband structures. It is, therefore, possible that, in contrast to WSD, the subbands may not be orthogonal leading to an overall lossless decomposition of the input signal. It is interesting to note that, as we show later, the subband structure learns filters of known types, such as band-pass, band-stop, low-pass, and high-pass structures, typically used in signal processing.

We can obtain a lossless subband decomposition that preserves the net information content of the input signal by further constraining the $U$ and $L$ filters into being spectrally complementary. The resulting CASD structure is shown in Fig. 2. As we can see, CASD is a special case of ASD with $L=1-U$, i.e., $U$ has the same structure as in the case of ASD described in Section II-A1. The input-output relationship is given by (1) with $W=U$. Similar to ASD, the network is end-to-end trained.

One of the benefits of the CASD structure is that it reduces the number of filter weights and, therefore, the computational cost of the subband decomposition structure at training by half; the cumulative effect of this reduction across training iterations can be significant. Another benefit of CASD is a higher degree of regularization compared to ASD which, depending on the dataset, can be beneficial. As we will see in the experimental results section, CASD (worse performance but also lower computational complexity) allows us to achieve a different tradeoff point than ASD which could be preferable depending on the application.

Note that the decomposition structures shown in Fig. 1 and Fig. 2 are regular convolution filter structures, and hence the standard back-propagation mechanism is applicable. We illustrate this by deriving the back-propagation equation for the CASD structure. Referring to Fig. 3, let $L$ be the loss function of the overall CNN, $X$ be the input, and $Y_{1}$ and $Y_{2}$ be the two outputs going to the next stage of the decomposition structure. Here $Y_{1}$ and $Y_{2}$ are complementary to each other in terms of spectral content [31], [32], ([c. 4, p. 125)]. Also, $\frac{\partial L}{\partial Y_{i}}$ is the back-propagating error derivative wrt the output $Y_{i}$, $i\in\{1,2\}$, $\frac{\partial L}{\partial W}$ the error derivative wrt the weights $W$, $\frac{\partial L}{\partial X}$ the error derivative wrt the input $X$ or the gradient passing down to the downstream module and $\frac{\partial Y}{\partial X_{1}}$ the local gradient of the filter $W$. Then, we can write

Computing $\frac{\partial L}{\partial X}$ is needed to propagate the error derivative while $\frac{\partial L}{\partial W}$ is required to update the filter weights.

Fig. 4 shows the generalized $L$-level MSR-CNN architecture in which the subbands are processed separately by individual CNNs, each with $I$ layers. This restricts the field of view of each CNN, making them indifferent to the rest of the subbands. The convolutional layers are similar to that of Alexnet [1] and/or VGG16 [7], with the difference of having a residual layer connection [9] before the pooling layer, as per Fig. 4. The addition of the residual layers, inspired by ResNet [9], significantly helps in speeding up training by allowing the error gradients to easily propagate to the initial layers. Finally, $N$ FC layers combine the feature outputs of the subband CNNs and perform classification.

We note that even though the input to the subband CNNs is already reduced in dimension, we still need pooling layers in the CNN to extract the dominant features per subband. This is done with a hierarchy of convolutional layers followed by a pooling layer that keeps the outputs with the largest magnitude, discarding the rest. In contrast, the decimation-by-2 blocks in the subband decomposition structure drop every 2nd filter output sample along the x and y dimensions, regardless of their magnitude. Therefore, the primary function of the wavelet subband decomposition is not to do dominant feature extraction. Rather, it assists in the extraction of the dominant features by the CNNs that follow.

Fig. 5(a) shows the general architecture of SSR-CNN. The SSR-CNN architecture shares the same front-end subband decomposition structure as the MSR-CNN, i.e., ASD, CASD, or WSD. An $M$-layer subband decomposition structure, with input tensor of dimensions $a\times b\times c$, where $a$, $b$, and $c$ are the number of rows, columns, and channels, will result in ${4}^{M}$ total subbands, each represented by a tensor of dimensions $(a/2^{M})\times(b/2^{M})\times c$. In contrast to the MSR-CNN architecture, all the channels of the subbands are stacked together and treated as a single input of dimension $((a/2^{M})\times(b/2^{M})\times(c\cdot 4^{M})$) to a single CNN, followed by a fully connected layer to classify.

Both network architectures are end-to-end trained, meaning that the front-end structures (ASD or CASD), CNN and fully connected layers, are trained together by backpropagating the error derivative from the output layer all the way back to the front-end. As we will see, the ASD structure gives better accuracy over both CASD and WSD of [30], at the expense of higher computational cost. Furthermore, the CNN architecture with a CASD frontend performs better than with a WSD frontend, as shown in [30]. The WSD structure offers the least computation cost, while the CASD structure lies between the ASD and the WSD structures, balancing computation cost and performance.

In Fig. 6, we show examples of the 2D frequency responses of the first layer of the ASD filter structure (cf. Fig. 1). The top row shows the 2D frequency response of the filters in the first upper and lower paths that decompose the input signal into two subbands. The bottom row shows the 2D frequency response of the filters in the second set of upper and lower paths that completes the first layer decomposing the input into four subbands. Note that the first-row filter responses are low-pass (left) and high-pass (right), while the second row of filters consists of low-pass, wider range low-pass, band-pass, and high-pass filter frequency responses, from left to right, respectively. Interestingly, the learned filters in the proposed structure resemble subband decomposition filters, i.e., pairs of highpass and lowpass filters, typically used in signal processing.

Fig. 7 shows the input image and the 4 subbands of a 1-layer 2D-DWT decomposition filter structure while Fig. 8 shows the 4 subbands of a 1-layer ASD decomposition filter structure in MSR-CNN for the same input image. Fig. 9 shows the output of the first convolutional layer of MSR-CNN with a 1-layer ASD structure. Though all subbands have a mix of low and high-frequency components, the subbands to the left have more low-frequency components while the subbands to the right have more high-frequency components as seen in the activation map. Fig. 10 shows the output at the 10th convolutional layer. In contrast to Fig. 9, Fig. 10 has lost the details in LL and UL subbands containing mainly low frequency or DC components, while the other two subbands still resemble the input and highlight specific contours of the input, indicating high-frequency components. The FC layer utilizes this collective information to partition the output hyperspace.

In Table III we present the top-1 accuracy comparison across the BCNN, SSR-CNN, and MSR-CNN architectures, for both 1- and 2-layer ASD and CASD structures. We have used a filter of order five for both the ASD and CASD filter structures. Generally, we notice a drop in performance going to a 2-layer decomposition compared to a 1-layer decomposition structure. This drop is a tradeoff between performance and computation cost. We also observe that the ASD structure with the MSR-CNN performs the best across different datasets.

Fig. 11 compares the test-set error curves of the BCNN, SSR-CNN, and MSR-CNN architectures. Compared to both the BCNN and the SSR-CNN curves, we find that the accuracy of the MSR-CNN has a milder monotonically increasing trajectory before settling down to a higher accuracy value. The BCNN increases quickly, then remains relatively stable, and finally shows an increase in accuracy before settling to a lower value compared to MSR-CNN and the SSR-CNN. The SSR-CNN on the other hand makes an initial jump in accuracy like that of the MSR-CNN and thereafter improves in accuracy in an almost linear fashion until settling between the performance of the MSR-CNN and the BCNN. This indicates that, with a 1-layer ASD structure, the better regularization of the MSR-CNN architecture results in more effective training compared to the BCNN and the SSR-CNN architectures.

On the ImageNet-2012 dataset, the MSR-CNN architecture with 1-layer ASD structure achieves top-5 and top-1 validation set accuracy of 86.91% and 69.45%, respectively. Unlike most popular networks, we did not perform architectural hyper-parameter optimization due to the lack of heavy computational resources, leaving room for further improvements.

In Table IV, we compare the total computation cost for both inference and training operations, of MSR-CNN and SSR-CNN architectures with the ASD structure. The reported numbers take into account every addition and multiplication operation performed by each network. The total addition and multiplication operations involved in the convolution operations dominate the overall computational cost of the CNNs. In the case of the back-propagation path, we take into account the number of operations per iteration. In the case of the inference path, which is not iterative, we report the total number of operations. With a 1-layer ASD structure, the MSR-CNN achieves nearly two orders of magnitude reduction in total addition and multiplication operations relative to the BCNN. The total number of addition and multiplication operations reduces further going to a 2-layer subband decomposition structure. When integrated over the period of training, it significantly reduces the total operations or the total energy spent. The above significant computational cost reductions are directly attributed to the number of filters in each architecture. We note that the number of filters in the ASD filter is $2(4^{M}-1)$ filters, where $M$ is the number of layers. In comparison, the CASD filter structure requires $(4^{M}-1)$ filters. For example, a 3-layer ASD structure would have 126 such filters to compute, while the 3-layer CASD structure would only have 63 such filters to compute.

In Table V we present a comprehensive comparison of well-established CNN architectures [54],[55],[8],[1],[14],[9],[7], and [11] against the best performer among the proposed CNN architectures. The comparison is in terms of time taken, number of multiply-and-accumulate (MAC) units, number of total parameters, and CNN performance accuracy percentage for the ImageNet-2012 dataset. The computation time was evaluated on a DELL PowerEdge R740xd server with 256 GB DDR4 2666MT/S, dual Intel-Xeon Gold 5220 CPUs with 18 cores each, and an Nvidia GTX 1080Ti GPU with Nvidia CUDA and cuDNN libraries. The results were obtained using 1000 images with a batch size of 8 images for all models. As we can see, the MSR-CNN with the 1-layer ASD structure performs the best, with an accuracy of 86.91% with 169.5M MAC operations and 42.05M parameters, which takes around 1.52 seconds to process 1000 images with a batch size of 8 images.

Both the 1-layer and 2-layer subband decomposed MSR-CNNs give a best-in-class reduction in total MAC operations needed. The MSR-CNN with 1-layer decomposition takes 169.5M MAC operations while the 2-layer decomposition takes 46.34M MAC operations. Other models range between 300M to 15.5B MAC operations. On the number of parameters front, the MSR-CNN architecture performs fairly. The parameters of all compared CNNs require between 0.6 to 65.8 MBytes, with the 1-layer and 2-layer MSR-CNN architectures at 20.1 and 6.5 MBytes, respectively. In practice, the difference between 6.5 and 0.6 MBytes can be ignored, whereas a 10$\times$ reduction in total MAC operations can significantly improve computation time. Finally, based on our experimental setup, the 1-layer, and 2-layer MSR-CNN take 1.52 secs and 0.79 sec respectively, while the other models range between 1.49 secs and 14.63 secs.

In Table VI, we show the effect of different levels of input quantization for the BCNN, SSR-CNN, and MSR-CNN architectures with WSD, 1-layer CASD, and 1-layer ASD. We quantize the input at 1, 2, 4, 6, and 8 bits. All internal computations are performed using the IEEE 32-bit floating-point precision. The objective of this experiment is to analyze the effect of input quantization noise on CNN accuracy results over various CNN architectures, subband decomposition structures, quantization levels, and datasets. We clearly observe that all three subband decomposition structures, WSD, CASD, and ASD are highly robust towards input quantization noise compared to the full-band BCNN architecture. The results in Table VI indicate that: 1) The subband decomposition structures provide robustness to input quantization error compared to full-band CNN architecture. 2) The effect of input quantization on the subband decomposition architectures is not drastic, even when we go down to 1-bit quantization. This is not so surprising since even a human can fairly recognize an image well with 1-bit quantization. For instance, grayscale newspaper printing used 1-bit greyscale with dithering [56]. Further, screens often use only 4-to-5 bits of color per channel natively and yet display good quality images, thanks to dithering [57]. Dithering fundamentally manipulates quantization noise by reshaping and spreading the noise unevenly across the spectrum. The subband decomposition process offers better input quantization even at 1-bit, compared to BCNN. 3) For applications requiring low transmission data rates or low storage and that intend to save power and computation resources, the flexibility offered by MSR-CNN with an adaptive subband decomposition front-end to quantize the input images to a greater extent without significantly affecting performance is advantageous from a system design consideration.

Weight-and-bias quantization plays a very important role in the implementation of CNNs in real-life systems [58]. Without weight and bias quantization, it would have been impractical to implement a CNN in an embedded system, where storage space and computation resources are limited. The ability to quantize at 8 bits compared to a 32-bit and 64-bit implementation is a matter of $1/4^{th}$ and $1/8^{th}$ savings in direct storage space, respectively. Quantization of weight and bias also results in a lower memory footprint during the computation of the CNN model. In Table VII, we investigate the effect of weight and bias quantization noise on inference accuracy for the BCNN, SSR-CNN, and MSR-CNN architectures with 1-layer WSD, CASD, and ASD for four different datasets, MNIST, CIFAR-10/100, and Caltech-101. In this experiment, we consider different quantization levels such as 8-bit, 16-bit, and 32 bits. All internal computations are performed using the IEEE 32-bit floating-point precision. Results from Table VII indicate that: 1) The subband decomposition structure provides robustness to weight and bias quantization error compared to full-band CNN architecture. 2) The MSR-CNN with the ASD subband decomposition structure, in general, performs better than the other architectures. 3) The ability of MSR-CNN with an adaptive subband decomposition front-end to perform better with heavily quantized weights and biases makes it an attractive choice for applications that are highly sensitive to computation and storage resources.

Both Tables VI and VII indicate that the MSR-CNN with the ASD decomposition structure performs the best. For applications with an additional need for reducing computation resources, the CASD structure may be a suitable compromise between computation cost and performance. As a final note, we would like to emphasize that the proposed architectures can be combined with sophisticated weight quantization methods, e.g., range batch normalization. Investigation of which method complements better the proposed architectures is beyond the scope of this work.