ConvBLS: An Effective and Efficient Incremental Convolutional Broad Learning System
for Image Classification

Suppose that the input data set $\bm{X}\in\mathbb{R}^{N\times C\times H\times W}$, where $N$, $C$, $H$, and $W$ denote the number of samples, the number of channels, height, and width, respectively. Taking the $k$th image $\bm{x}^{(k)}\in\mathbb{R}^{C\times H\times W}$ as the input, the output of CF_i (i.e., the $i$th feature node group) can be defined as follows:

where $\bm{W}_{f_{i}}$ and $\bm{\beta}_{f_{i}}$ are the weight matrix and bias matrix, respectively. Among them, $\bm{W}_{f_{i}}$ is obtained by the SKM algorithm. Similar to [44], we set $\bm{\beta}_{f_{i}}$ to be a zero matrix to simplify the learning of CF layers. Because it is difficult for the SKM algorithm to learn features in a very high-dimensional feature space, we split the high-dimensional feature space into several sub-spaces and utilize the SKM algorithm to learn filters separately in each subspace. Thus, to divide the subspace and ensure that the filters are only used to extract features in the subspace where they are trained, we use group convolution here represented as $GConv(\cdot)$. In other words, the features from different feature sub-spaces (i.e., different feature groups) do not interact. And $c_{i}$ is the number of feature maps for each feature group in the CF_i layer. Particularly, we set $c_{1}$ to be the number of channels of the input image for the CF₁ layer. As for the subsequent CF layers, $c_{i}$ and $c_{r}$ can be different for $i\neq r$. Without loss of generality, we have $c_{i}=c_{r}$, where $i,r\in\{2,3,...,n\}$. At last, $\bm{\phi}(\cdot)$ represents the ReLU activation function. In other words, each CF layer utilizes the output of its precursor CF layer to obtain more abstract features. Furthermore, in the CF₁ layer, the input $\bm{z}_{0}^{(k)}$ is defined as $\bm{x}^{(k)}$. Unlike the group convolution typically used in DNNs, we pre-process all patches extracted from previous feature maps by normalization and whitening before convolution.

Denote $\bm{z}^{(k)}\equiv[\bm{z}_{1}^{(k)},\bm{z}_{2}^{(k)},...,\bm{z}_{n}^{(k)}]$, which is the concatenation of all feature node groups. To keep the spatial size of the feature nodes consistent, pooling operations are used in appropriate locations. Since each feature node is a feature map obtained by convolution, there are still significant spatial relationships amongst the feature values within each map. Consequently, we use convolution for two-dimensional feature enhancement instead of the one-dimensional feature enhancement used in BLS to preserve the spatial relationships of feature values. The output of CE_j (i.e., the $j$th enhancement node group) can be formulated as follows:

where $\bm{W}_{e_{j}}$ and $\bm{\beta}_{e_{j}}$ are randomly generated weight matrix and bias matrix, respectively. $Conv(\cdot)$ stands for convolutional operation and $\bm{\varphi}(\cdot)$ denotes a selected activation function. Similarly, $\bm{h}_{0}^{(k)}$ is equivalent to $\bm{z}^{(k)}$ in the CE₁ layer.

To obtain promising performance, all feature node groups and enhancement node groups (with various feature scales) are concatenated directly to yield the first stage multi-scale features, which can be expressed as follows:

where $\bm{\psi}(\cdot)$ is a concatenation function and $s_{1}$ represents the first stage multi-scale features. Subsequently, more reasonable and comprehensive two-stage multi-scale features are attained by the typical SPP technique. The second stage multi-scale features are defined as follows:

where $d$ is the number of feature pyramid layers and $b_{l}$ denotes the size of the feature maps for the $l$th layer feature pyramid. $\bm{\tau}(\cdot)$ represents a function combination of SPP and flattening. Suppose that the shape of $a^{(k)}_{s_{1}}$ is $(C_{f}+C_{e})\times W_{s_{1}}\times H_{s_{1}}$, in which $W_{s_{1}}$ and $H_{s_{1}}$ are usually equal, thus we set $W_{s_{1}}=H_{s_{1}}=v$. Take the $l$th layer pyramid as an example, to obtain the features with the specified scale, we have a pooling layer with the window size $win=\lceil v/b_{l}\rceil$ and stride $str=\lfloor v/b_{l}\rfloor$, where $\lceil\cdot\rceil$ and $\lfloor\cdot\rfloor$ represent ceiling and floor operations. Besides, $s_{2}$ denotes the second stage multi-scale features. Combining Eq. (3) and Eq. (4), the TSMS features $\bm{a}^{(k)}_{s_{2}}$ of the $k$th sample can be obtained. For simplicity, the subscript $s_{2}$ is omitted in the rest of this paper.

The final representation of each sample is a comprehensive feature vector, which is, to some extent, translation invariant and scales invariant inherited from convolutions and has rich multi-scale semantic information. Similar to the vanilla BLS, the output layer of ConvBLS is still a plain linear classification layer that can be represented as the equation of the form:

where $\bm{W}_{out}$ is the weight matrix and $\bm{y}^{(k)}$ denotes the final output of ConvBLS for input image $\bm{x}^{(k)}$. Furthermore, the outstanding classification performance of ConvBLS can also be ensured by the efficient training algorithm, which will be illustrated later.

The weights of convolutional filters for CF layers are trained using the SKM algorithm in a greedy manner. When training the CF_i layer, the extracted features for all training samples at the CF_i-1 layer are available. We collect the features of all training samples in the CF_i-1 layer as $\bm{Z}_{i-1}\in\mathbb{R}^{N\times e^{(i-2)}C_{1}\times\frac{H}{2^{i-1}}\times\frac{W}{2^{i-1}}}$. The equation for $\bm{Z}_{i-1}$ is denoted as $\bm{Z}_{i-1}=[\bm{z}_{i-1}^{(1)},\bm{z}_{i-1}^{(2)},...,\bm{z}_{i-1}^{(N)}],i=1,2,...n$ , where $\bm{Z}$ represents the features of all samples and $\bm{z}$ denotes the features of one sample. Moreover, when the CF₁ layer is trained, $\bm{Z}_{0}$ is defined as $\bm{X}$. Given the number of feature maps $\bm{c}_{i}$ within each feature group for the CF_i layer, the features extracted from the CF_i-1 layer can be divided into $\lceil e^{(i-2)}C_{1}/{\bm{c}_{i}}\rceil$ groups. Additionally, for $\bm{Z}_{i-1,q}\in\mathbb{R}^{N\times c_{i}\times\frac{H}{2^{i-1}}\times\frac{W}{2^{i-1}}}$, where $q$ represents the $q$th feature group, the training procedure begins by extracting random patches from the feature group $\bm{Z}_{i-1,q}$. Each patch has dimension $w$-by-$w$-by-$c_{i}$, with $w$ referred to as the receptive field size. After that, pre-processing operations are necessary. First, every patch is normalized by subtracting the mean and dividing by the standard deviation of its elements to normalize the brightness and contrast. Subsequently, to overcome the correlations between adjacent pixels, the ZCA whitening transform[45] should be used. We then gather all of the patches and construct a new dataset for the training of this set of convolutional filters. The dataset is represented as $\bm{P}=\{\bm{p}^{(1)},\bm{p}^{(2)},...,\bm{p}^{(S)}\}$, where $\bm{p}^{(s)}\in\mathbb{R}^{w\times w\times c_{i}}$, and $S$ represents the number of patches.

After pre-processing, the core issue becomes how to learn the weights of convolution filters from patches. As we know, the principle of deep neural networks essentially involves a template-matching problem. For CNNs, each convolution filter is a template (i.e., pattern) used to extract the corresponding feature from the response of the precursor layer. Thus, a set of excellent templates must produce similar responses on the same class of samples and vice versa, which is crucial for the classification task. However, the RCF-based methods[14, 18, 19, 21, 22, 23] whose convolutional filters are generated randomly can not meet the above requirements. Inspired by the earlier works that use the K-means algorithm for unsupervised feature learning[39, 40, 41, 42, 43], we utilize the SKM to learn better convolutional filters (i.e., templates) for CF layers. In this context, the data points to be clustered are randomly extracted patches, and the centroids are the convolutional filters used to extract features from the corresponding output of the predecessor CF layer. The algorithm finds the convolutional filters as follows:

where $\bm{u}^{(s)}_{q}$ is the code vector associated with the input $\bm{p}^{(s)}$, and $\bm{w}_{f_{i},q,t}$ is the $t$-th convolutional filter in CF_i for the $q$th feature group $\bm{Z}_{i-1,q}$. Note that the convolution filters learned in the $q$th feature group can only be utilized to extract more abstract features in the $q$th feature group eventually. Recalling the feature grouping method mentioned earlier, we can attain the weights of convolutional filters associated with other feature groups in a similar manner. Finally, the convolutional filters for all CF layers can be fine-tuned in this way.

The weights of the output layer are trained using the ridge regression algorithm. To this end, the TSMS features of all training samples that are robust to object deformation are first calculated using Eq. (4). Denoting the final TSMS feature matrix as $\bm{A}$ and the real label vector as $\hat{\bm{Y}}$, the optimization problem of the output layer is expressed as follows:

where $\lambda$ represents the regularization coefficient to balance the mean squared error term and L2 normalization term. Also, the problem is convex, and the solution can be obtained by the ridge regression theory, which produces an approximation to the Moore-Penrose generalized inverse by adding a positive number to the diagonal of $\bm{A}^{\top}\bm{A}$ or $\bm{A}\bm{A}^{\top}$. Therefore, the weights of output layer $\bm{W}_{out}$ can be calculated as follows:

where $I$ is an identity matrix with the same shape as $\bm{A}^{\top}\bm{A}$.

By combining the above two phases, all the weights to be trained in our ConvBLS can be optimized. It should be noted that since the weights of CF layers are optimized without tedious fine-tuned processes using the BP algorithm, the training procedure of the entire model is extremely efficient. Thanks to the powerful feature extraction capability of the CF layer and CE layer, the training of the output layer can achieve excellent classification performance without requiring a lot of labeled data. Thus, our method also has remarkable performance in semi-supervised classification.

Recall that the weights and biases of CE layers do not require training. Therefore, we add additional enhancement nodes for CE layers to improve the performance quickly. For convenience to introduce the incremental learning algorithm of the enhancement nodes, we rewrite the CE layer with four-dimensional tensors. In other words, the entire training set $\bm{X}$ rather than a single image $\bm{x}^{(k)}$ are fed into the ConvBLS at once to calculate the enhancement features $\bm{H}$. Next, we detail the broad expansion method for adding $C^{a}_{e}$ additional enhancement nodes in the CE₁ layer. The output of the CE_j layer for additional enhancement nodes can be formulated as follows:

where

and $\bm{\beta}_{e_{j}}^{C_{2}+1}\in\mathbb{R}^{e^{j-1}C_{e}^{a}}$ are randomly generated. Specifically, similarly to Eq. (2), $\bm{H}_{0}^{C_{2}+1}$ is defined as $\bm{Z}$. And then, the new TSMS features can be formulated as follows:

where $\bm{\sigma}(\cdot)$ is a function combination of $\bm{\psi}(\cdot)$ and $\bm{\tau}(\cdot)$ in Eq. (3) and Eq. (4). Then, we deduce the pseudoinverse of the new matrix as

where $\bm{D}=(\bm{A}^{C_{2}})^{+}\bm{\sigma}(\bm{H}_{1}^{C_{2}+1},\bm{H}_{2}^{C_{2}+1},...,\bm{H}_{m}^{C_{2}+1})$,

and $\bm{Q}=\bm{\sigma}(\bm{H}_{1}^{C_{2}+1},\bm{H}_{2}^{C_{2}+1},...,\bm{H}_{m}^{C_{2}+1})-\bm{A}^{C_{2}}\bm{D}$

Again, the new weight matrix is

The incremental learning algorithm of additional enhancement nodes is listed in Algorithm 1.

In some cases, due to the insufficient feature nodes, adding enhancement nodes solely does not meet the performance requirements. Here, we describe the incremental learning for newly incremental feature nodes. Similarly, we rewrite the CF layer in four-dimensional tensors. Assume that there are $C_{1}$ and $C_{2}$ output channels in the CF₁ and CE₁ layers of the initial architecture, respectively. Consider adding $C^{a}_{f}$ feature nodes to the CF₁ layer, the additional output of the $i$th feature layer can be expressed as follows:

where $\bm{Z}_{0}^{C_{1}+1}$ is defined as $\bm{X}$. Denote $\bm{Z}^{C_{1}+1}\equiv[\bm{Z}_{1}^{C_{1}+1},\bm{Z}_{2}^{C_{1}+1},...,\bm{Z}_{n}^{C_{1}+1}]$, the corresponding enhancement nodes are randomly generated as follows:

where $\bm{H}_{0}^{C_{1}+1}$ is defined as $\bm{Z}^{C_{1}+1}$. Then, the TSMS features can be expressed as follows:

Similarly, we deduce the pseudoinverse of the new matrix as

where $\bm{D}=(\bm{A}^{C_{1}})^{+}\bm{\sigma}(\bm{Z}_{1}^{C_{1}+1},...,\bm{Z}_{n}^{C_{1}+1},\bm{H}_{1}^{C_{1}+1},...,\bm{H}_{m}^{C_{1}+1})$,

and $\bm{Q}=\bm{\sigma}(\bm{Z}_{1}^{C_{1}+1},...,\bm{Z}_{n}^{C_{1}+1},\bm{H}_{1}^{C_{1}+1},...,\bm{H}_{m}^{C_{1}+1})-\bm{A}^{C_{1}}\bm{D}$,

Again, the new weight matrix is

The incremental learning algorithm of additional feature nodes is listed in Algorithm 2.

Inspired by the great success of deep transfer learning, we also use the pre-trained deep convolutional neural network as the CF layer. Compared with the CF layer trained by the SKM algorithm, using the pre-training model as the CF layer also maintains the training efficiency of BLS and has better performance on more complex tasks.

The dataset[46] contains 60 000 training samples and 10 000 test samples, which are evenly distributed over 10 classes of handwritten digital images. Among them, every sample is a gray-scale image with 28$\times$28 pixels.

The dataset[47] is consistent with the MNIST dataset, except that it has more complex image features. Besides, all samples fall into 10 categories, including 1) T-shirt/top, 2) trouser, 3) pullover, 4) dress, 5) coat, 6) sandal, 7) shirt, 8) sneaker, 9) bag, and 10) ankle boot.

The dataset[48] is a more complicated dataset compared with MNIST and Fashion-MNIST datasets, which is composed of 48 600 images with the size of 2 $\times$ 32 $\times$ 32 pixels. The NORB contains images of 50 different 3-D toy objects labeled by five distinct categories: 1) animals, 2) humans, 3) airplanes, 4) trucks, and 5) cars. Here, 24 300 images of 25 objects are used for training, and the other 24 300 images are used for testing.

Our results are shown in Table I. The experimental results of the comparison methods, including SAE, DBN, and MLELM, are cited from [49], while that of Stacked BLS are cited from [50]. For our ConvBLS, we set the initial number of output channels in the CF₁ layer as $32$ and the corresponding expansion ratio as $2$.

We can observe that ConvBLS has the highest test accuracy of $99.280\%$, even with the extremely short training time. Specifically, our method have a speedup of more than $200$ times compared to the most time-consuming method, which is attributed to the TSMS feature extracted by the effective model architecture and the efficient training algorithm.

Table II presents the results. To make the experimental conclusions more reliable, the test accuracy of several comparison methods, including AlexNet, VGG16, GoogLeNet, and DenseNet, are cited from [54] and [55]. Conversely, the training time of these models are obtained by re-running them on our experimental platform with the same training details as the original papers for a fair comparison. For CNNBLS and CNN+BLS, the model structures and training details we adopted are the same as that on the MNIST dataset. At last, given the more challenging data set, the number of initial output channels and expansion ratio was set to $64$ and $1.5$, respectively.

As shown in Table II, similar to the MNIST dataset case, our method achieve the state-of-the-art performance among the existing approaches with a superfast speed in computation. Thus, the proposed ConvBLS model is very appealing. In particular, our approach surpasses many classical DNNs in both time and accuracy.

The experimental results are reported in Table III. In the first methods, the k-means and triangle activation are used for single-layer feature extraction, and a support vector machine is used for classification, which generally requires a large number of features. For reference, K-means-BLS, one of the most similar works to ours, that use k-means to extract features and use a complete BLS to classify features, are reproduced on the NORB dataset, and the relevant hyper-parameters are as follows: the number of features in K-means feature extraction is set to $1600$, and BLS is constructed by total $100\times 10$ feature nodes and $1\times 9000$ enhancement nodes. Moreover, to our knowledge, no work attempts to perform classification tasks on NORB datasets using typical CNNs. Therefore, we instantiate some common CNNs as deep topology-based comparison methods. In the first convolutional layer, AlexNet-small, AlexNet-base, and AlexNet-large have $16$, $32$, and $64$ output channels, respectively, and the number of output channels of the remaining layers vary accordingly. For the comparison methods trained by the BP algorithm, similar to [49] and [13], the training procedure is set as $100$ epochs, and the remaining training hyperparameters are finetuned to make the models converge. Lastly, the ConvBLS structure keeps same as that on Fashion-MNIST dataset.

The results also indicate that ConvBLS outperforms all of the comparison methods. In particular, our method significantly outperforms the broad topology-based and deep topology-based comparison methods in terms of test accuracy with a very short training time, which also validates our conjecture that convolution is very effective in processing image data, yet the arduous BP-based training for convolution filters is not necessary on relatively small-scale image datasets.

Our experiments consider several activation functions (which we will discuss in the next section) and the number of output channels for the CF₁ layer. As shown in Fig. 6, our methods with different activation functions generally achieve higher performance by learning more feature maps as expected.

However, as the number of output channels increases, the accuracy of all methods rapidly saturates. In other words, we can attempt to increase the width (number of output channels) of the ConvBLS to improve accuracy, which is the basis of the incremental learning algorithm for ConvBLS.

The selection of the activation function is also an important issue. In this part, four activation functions, including Tanh, ReLU, Sigmoid, and Triangle, are validated. As shown in Fig. 6, the Tanh function works best, while the Triangle has the worst accuracy. Besides, the ReLU and Sigmoid activation functions have similar accuracy. Given that Triangle is computationally expensive and its accuracy is the worst, we further investigate the remaining three activation functions in the following experiments. Lastly, considering the simplicity of ReLU, we use the ReLU activation function to construct our ConvBLS model.

Different kernel sizes can capture features of different scales. Hence, we test kernel sizes of $6$, $8$, $10$, and $12$. As shown in Fig. 7, the methods with a smaller kernel size in CF layers work better. The reason for the above phenomenon is as follows. In particular, as the kernel size increases, the dimension of the feature space where the unsupervised learning algorithm works also becomes larger. This makes it difficult for the algorithm to discover the selective filters.

We vary the stride over $1$, $2$, and $4$. Fig. 8 depicts the results.

Overall, the 1-pixel stride works best and the larger the stride the worse the accuracy. Because the larger the stride, the less the number of features we can extract. Thus, if we have computational resource to spare, our results suggest that it is better to spend it on reducing the stride and selecting a small kernel size.