Quantized Neural Networks via $\{-1,+1\}$ Encoding Decomposition and Acceleration

As the basic computation in most layers of neural networks, matrix multiplication costs lots of resources and is also the most time-consuming operation. Modern computers store and process data in binary format, thus non-negative integers can be directly encoded by $\{0,1\}$. Therefore, we propose a novel decomposition scheme to accelerate matrix multiplication as follows: Let $x\!=\![\mathrm{x}^{1},\mathrm{x}^{2},...,\mathrm{x}^{N}]^{T}$ and $w\!=\![\mathrm{w}^{1},\mathrm{w}^{2},...,\mathrm{w}^{N}]^{T}$ be two vectors of non-negative integers, where $\mathrm{x}^{i},\mathrm{w}^{i}\!\in\!\{0,1,2,...\}$ for $i\!=\!1,2,...,N$. The dot product of those two vectors is written as follows:

All of the above operations consist of $N$ multiplications and $(N\!-\!1)$ additions. Based on the above $\{0,1\}$ encoding scheme, the vector $x$ can be encoded to a binary form using $M$ bits, i.e.,

Then the right-hand side of the formulation (2) can be converted into the following form:

where

In such an encoding scheme, the number of represented states is not greater than $2^{M}$. We also encode another vector $w$ with $K$-bit representation in the same way. Therefore, the dot product of the two vectors can be computed as follows:

where ${d}$ is the encoding element of $\mathrm{w}$. From the above formula, the dot product is decomposed into $M\times K$ sub-operations, in which every element is $0$ or $1$. Because of the restriction of encoding and without using the sign bit, the above representation can only be used to encode non-negative integers. However, it is impossible to limit the weights and the values of the activation functions to non-negative integers.

In order to extend encoding space to negative integer and reduce the computational complexity, we propose a new encoding scheme, which uses $\{-1,+1\}$ as the basic elements of our encoder rather than $\{0,1\}$. Except for the difference of basic elements, the encoding scheme is similar to the rules in Eq. (4), and is formulated as follows:

where $M$ denotes the number of encode bits, and can represent $2^{M}$ states. And the encoding space is extended to $\{...,-3,-1,1,3,...\}$. We can use multiple bitwise operations (i.e., xnor and bitcount) to effectively achieve the above vector multiplications. Therefore, the dot product of those two vectors can be computed as follows:

where $\emph{XnorPopcount}(\cdot)$ denotes the bitwise operation, and its details will be defined below. Therefore, this operation mechanism is suitable for all vector/matrix multiplications.

In various neural networks, matrix multiplication is one basic operation in some commonly used layers (e.g., fully connected layers and convolution layers). Based on the above computational decomposition mechanism of matrix multiplication, we can decompose the quantized networks into a more efficient architecture.

Let $M$ be the number of encoding bits, and thus the encode space can be defined as: $\{-(2^{M}\!-\!1),-(2^{M}\!-\!3),...,(2^{M}\!-\!3),(2^{M}\!-\!1)\}$, in which each state is an integer value. The boundaries of the values are determined by the number of encoding bits, $M$. However, the parameters in DNNs are usually distributed within the interval $[-r,+r]$. In order to apply our decomposition mechanism to network model decomposition, we use the following formulation for the computation of common layers in neural networks.

where $Op(\cdot)$ is the basic operations for multiple layers (e.g., fully connected operations and convolution operations), $M$ is the number of bits to encode input $x$, $K$ is the number of bits to encode weights $w$, and $EnC(\cdot)$ is the encoding process. Here, ${r}/{[(2^{M}\!\!-\!1)(2^{K}\!\!-\!1)]}$ can be viewed as a scaling factor, and $r$ is usually set to $1$. If there is a batch normalization layer after the above layer, the scaling factor is not used in the computation of our decomposition scheme.

For instance, we use a $2$-bit encoding scheme (i.e., $M\!\!=\!\!K\!\!=\!\!2$) for a fully connected layer to introduce the mechanism of our decomposition model, as shown in Fig. 1. We define an “Encoder” that can be used in the 2BitEncoder function (i.e., $\varphi_{2}^{1}(\cdot)$ and $\varphi_{2}^{2}(\cdot)$ defined below) for encoding input data. Here, $x$ is encoded by $x_{1}\!\in\!\{-1,+1\}^{N}$ and $x_{2}\!\in\!\{-1,+1\}^{N}$, where $x_{2}$ is high bit data and $x_{1}$ is low bit data. Therefore, we have the following formula: $x\!=\!x_{1}\!+\!2x_{2}$. In the same way, the weight $w$ can be converted into $w_{1}\!\in\!\{-1,+1\}^{M\times N}$ and $w_{2}\!\in\!\{-1,+1\}^{M\times N}$. After cross multiplications, we get the four intermediate variables {$y_{1},y_{2},y_{3},y_{4}$}. In other words, each multiplication can be considered as a binarized fully connected layer, whose elements are $-1$ or $+1$. This decomposition can produce multi-branch layers, and thus we call our networks as multi-branch binary networks (MBBNs). For instance, we decompose the $2$-bit fully connection operation into four branches binary operations, which can be accelerated by bitwise operations, and then the weighted sum of those four results by the fixed scaling factor (e.g., $1/9$ in Fig. 1) is the final output. Besides fully connected layers, our decomposition scheme is suitable for the convolution and deconvolution layers in various deep neural networks.

As an important part of various DNNs, activation functions can enhance the nonlinear characterization of networks. In our model decomposition method, the encoding function also plays a critical role and can encode input data to multi-bits ($-1$ or $+1$). Those numbers represent the encoding of input data. For some other QNNs, several quantization functions have been proposed. However, it is not clear what the nonlinear mapping is between quantized numbers and encode bits. In this subsection, a list of $M$-bit encoding functions are proposed to produce the element of each bit that follows the rules for encoding data.

Before encoding, the data should be limited to a fixed numerical range. Table 1 lists the four activation functions. $HTanh(\cdot)$ is used to project input data to $[-1,+1]$, and it consists of the sign function to achieve binary encoding of weights and activations [17, 23]. Since the convergence of SGD for training the networks with $ReLU(\cdot)$ is faster than other activation functions, we propose a new activation function $HReLU(\cdot)$, which retains the linear characteristics in the specific range and limits the range of input data to $[0,1]$. Different from general activation functions mentioned above, the output of our $M$-bit encoding function defined below should be $M$ numbers, which are $-1$ or $+1$. Those numbers represent the encoding of input data. Therefore, the dot product can be computed by Eq. (9). In addition, in the above described experimental condition, when we use $2$-bit to encode the data $x$ and constrain to $[-1,+1]$, there are 4 encoded states, as shown in Table 2. The nonlinear mapping between quantized real numbers and their corresponding encoded states is given in Table 2. From the table, we can see that there is a linear factor $\alpha$ between quantized real numbers and encoded states (i.e., $\alpha=3$ for Table 2). When we use Eq. (9) to compute the multiplication of two encoded vectors, the value will be expanded $\alpha^{2}$ times. Therefore, the result can multiply its scaling factor to get the final result, e.g., $1/9$ as in Fig. 1.

From the above results, we obviously need an activation function to achieve the transformation from the quantized number to its corresponding encoded state. The most straightforward transformation method is to determine the encoded states according to the number value through the look-up table, which is suitable for many hardware platforms (e.g., FPGA and ASIC). However, its derivative is hardly be defined, and thus this method can not be applied in training process. Therefore, we need a kind of derivable encoding function as activation functions to train the network.

This is a typical single-input multiple-output problem. The Fourier neural network (FNN) [53, 54] can be used to transform a nonlinear optimization problem into a combination of multiple linear ones [55]. It has been successfully used for stock prediction [54], aircraft engine fault diagnostics [56], harmonic analysis [57], lane departure prediction [58], and so on. In[59], it has verified that the trigonometric activation function can achieve faster and better results than those using established monotonic functions for some certain tasks. Fig. 2 shows the illustration of the $2$-bit and $3$-bit encoding functions. We can see that those encoding functions are required to be periodic, and they should have different periods. Inspired by FNN, we apply trigonometric functions as the basic encoding functions, whose lines are highlighted in red. Finally, we use the sign function to determine whether the value is $-1$ or $+1$. Hence, we use several trigonometric functions to approximate the nonlinear mapping between quantized numbers and their encoded states, which is called as the $MBitEncoder$ function, and is formulated as follows:

when $\varphi_{M}^{M}(x)$ (i.e., $m\!=\!M$) is the encoding function of the highest bit of the $MBitEncoder$ function. For example, for the 2-bit case, the $2BitEncoder$ function is formulated as follows:

where $\varphi_{2}^{1}(x)$ is the encoding function of the first bit ($x^{i}_{1}$) of the $2BitEncoder$ function, and $\varphi_{2}^{2}(x)$ denotes the encoder function of the second bit ($x^{i}_{2}$) of the $2BitEncoder$ function. The periodicity is clearly different from the others because it needs to denote more states.

Generated by the decomposition of QNNs, MBBNs need to use $M$-bit encoding functions as activation functions to get the elements of each bit, which can be used by more efficient bitwise operations to replace arithmetic operations. As described in Section 3.3, $M$-bit encoding functions are the combination of a series of trigonometric and sign functions. The sign function of the encoder makes it difficult to implement the BP process. Thus, we approximate the derivative of the encoding function with respect to $x$ as follows:

We take the 2-bit encoding as an example to describe the optimization method of MBBNs.

Besides activations, all the weights of networks also need to be quantized to binary values. Therefore, MBBN can be viewed as a special binarized neural network [17]. In this training method, we use the same mechanisms to constrain and update parameters, and use $w\!\in\![-1,1]$ and $w_{b}\!\in\!\{-1,1\}$ to represent the real-valued weights and its binarized weights, respectively. $w$ is constrained between $-1$ and $1$ by the HTanh() function to avoid excessive growth. Instead of $M$-bit encoding functions that are complex, we use the sign function directly to binarize parameters. Therefore, the binarization function for transforming $w$ to $w_{b}$ can be directly formulated as follows:

For this function, we define the gradient function of each component to constrain the search space. In the training process, we apply $w_{b}$ to compute the loss and gradients, which are used to update $w$. That is, the input of the sign function can be constrained to $[-1,+1]$ by $HTanh(x)$, and it can also speed up convergence. The parameters of the whole network are updated in the condition of differentiability.

We give the detailed training algorithm of MBBN in Algorithm 1, where we describe the main computing process of the network, and some details about batch normalization, pooling and loss layers are omitted. Note that the forward() operation is the basic operation in DNNs, such as convolutional and full-connected operations. Those operations can be accelerated by binary bitwise operations. We use backward() and BackMBitEncoder() to specify how to backpropagate through forward() and MBitEncoder() operations. Update() specifies how to update the parameters with their known gradients. For example, we usually use Adam or SGD as our optimization algorithm.

The above training scheme is proposed to optimize binary networks, which are transformed by our model decomposition method. However, this decomposition method can produce multiple times as many parameters as the original network. If we optimize the binary network, it may easily fall into local optimal solutions and has slow convergence speed. Based on the nonlinear mapping between quantized numbers and their encoded states, we can directly optimize the quantized network and then utilize our equivalent transformation method to decompose quantized network. The decomposed parameters can be used by MBBN in the inference process to achieve computational acceleration.

Two quantization schemes are usually applied in QNNs [17, 20, 29], and include linear quantization and logarithmic quantization. Due to the requirement of our encoding mechanism, linear quantization is used to quantize our networks. We quantize a tensor $x$ linearly into $K$-bit, which can be formulated as follows:

where the clamp function is used to truncate all values into the range of $[-1,1]$, $t$ is a learned parameter, and the scaling factor $d$ is defined as: $d=t/(2^{K-1}-1)$. The rounding operation can be used to quantize a real number to a certain state, we call it a hard ladder function, which can segment input space to multi-states. Table 2 lists the four states quantized by Eq. (11). However, the derivative of this function is almost zero everywhere, and thus it cannot be applied in the training process. Inspired by STE [38], we use the same technique to speed up computing process and yield better performance. We use the loss computed by quantized parameters to update full precision parameters. Note that for our encoding scheme with low-precision quantization (e.g., binary), we use Adam to train our model, otherwise SGD is used. Algorithm 2 shows the main training process of our quantized network, where $BackQuantize_{M}$() specifies how to backpropagate through the $Quantize_{M}$() function with $M$-bit quantization.

CIFAR-10 is an image classification dataset, which has $50,000$ training and $10,000$ testing images. All the images are $32\!\times\!32$ color images representing airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships and trucks.

We validated our method with different bit encoding schemes, in which activations and weights are equally treated, that is, both of them use the same number of encoding bits. Table 3 lists the results of our method and many state-of-the-art models, including BWN [33], BNN [17], XNOR-Net [11], XNOR-Net [11], TWN [18], DoReFa-Net [20], ABC-Net [12], INQ [21], and SYQ [60]. Here we used the same network architecture as in [33] and [17], except for the encoding functions. We used $\emph{HTanh}(\cdot)$ as the activation function and employed Adam to optimize all the parameters of the networks. From all the results, we can see that the representation capabilities of our networks encoded by 1-bit or 2-bit are enough for small-scale datasets, e.g., CIFAR-10. Our method with low-precision encoding (e.g., $2$-bit) achieved nearly the same classification accuracy as its high precision versions (e.g., $8$-bit) and full-precision models, while we can attain $\sim\!16\times$ memory saving compared with its full-precision counterpart. When both activations and weights are constrained to $1$-bit, our network structure is similar to BNN [17], and our method yielded even better accuracy mainly because of our proposed encoding functions.

ImageNet: We further compared the performance of our method with all the other methods mentioned above on the ImageNet (ILSVRC-2012) dataset [61]. This dataset consists of 1K categories images, and has over $1.2$M images in the training set and $50$K images in the validation set. Here, we used ResNet-18 and ResNet-50 to illustrate the effectiveness of our method. and used Top-1 and Top-5 accuracies to report classification performance. For large-scale training sets (e.g., ImageNet), it usually costs plenty of time and requires sufficient computing resources for classical full-precision models. Moreover, it will be more difficult to train quantized networks, and thus the initialization of parameters is particularly important. Here, we used $\emph{HReLU}(\cdot)$ as the activation function to constrain activations. In particular, the full-precision model parameters activated by $\emph{ReLU}(\cdot)$ can be directly used as the initialization parameters for our $8$-bit quantized network. After a little number of fine-tuning, our $8$-bit quantized networks can be well-trained. Similarly, we used the $8$-bit model parameters as the initialization parameters to train $7$-bit quantized networks, and so on. There is a special case, when we use $\emph{HReLU}(\cdot)$ and the 1BitEncoder function to encode activations, and all the activations are constrained to $1$. Thus, we used $\emph{HTanh}(\cdot)$ as the activation function for $1$-bit encoding. Note that we used SGD to optimize parameters when the number of encoding bits is not less than $3$, and the learning rate was set to $0.1$. When the number of the encoding bits is $1$ or $2$, the convergent speed of Adam is faster than SGD, as discussed in [17, 11].

Table 3 lists the performance (e.g., accuracies and compression ratios) of our method and many state-of-the-art models mentioned above. Experimental results show that our method performs much better than its counterparts. Similarly, our method with $5$-bit encoding significantly outperforms ABC-Net[$5$-bit] [12]. Moreover, our networks can be trained in a very short time to achieve the accuracies in our experiments, and only dozens of fine-tuning are needed. Different from BWN and TWN, whose weights are only quantized rather than activation values, our method quantifies both weights and activation values, simultaneously. Although BWN and TWN can obtain little higher accuracies than our $1$-bit quantization model, we obtain significantly better speedups, and the speedup ratios of existing methods such as BWN and TWN are limited to $2\times$. Due to limited and fixed expression ability, existing methods (such as BWN, TWN, BNN and XNOR-Net) cannot satisfy various quantization precision requirements. In particular, our method can provide $64$ available encoding choices, and hence our encoded networks with different encoding precisions have different speedup ratios, memory requirements and experimental accuracies.

In the above experiments, all activation values and weights are constrained to the same bit precision (e.g., $M\!\!=\!\!K\!\!=\!\!6$). However, our method also has an advantage that it can support mixed precisions (e.g., $M\!\!=\!\!6$ and $M\!\!=\!\!5$), that is, weights and activations for each layer can have different bit precisions. This makes model quantification and acceleration be more flexible, and thus meets more requirements for model design. We used ResNet-18 as an example to verify the effect of mixed precisions. Table 4 lists the network structures, bit precision of each layer, compression ratios, and classification accuracies on ImageNet. For the convenience, we set the same bit precision for each residual block of ResNet-18. The experimental results show that the mixed precisions can obtain higher classification accuracies. As shown in Table 4, the classification results of Precision#1 are higher than those of their full-precision counterparts, while obtaining a 5$\times$ compression ratio. We also set the bit precision of all the activation values to 4, and different bit precisions for each weight in Table 4, and achieve $14.1\times$ compression ratio and $69.32\%$ Top-1 accuracy.

Besides image classification, QNNs can also be used for object detection tasks [62, 63]. Hence, we use our method to quantize some trained networks with the Single Shot MultiBox Detector (SSD) framework [64] for object detection tasks, (i.e., a coordinate regression task and a classification task). The regression task has higher demands on the encoding precision, and thus the application of object detection presents a new challenge for QNNs.

In this experiment, all the models were trained on the PASCAL VOC2007 and VOC2012 trainval set, and tested on the VOC2007 test set. Here, we used MobileNetV2 [9], InceptionV3 [65] and ResNet-50 [66] as the basic networks to extract features in the SSD framework. Note that we quantize the basic network layers, and keep the other layers in full-precision. All the networks are trained on a large input image of size $512\times 512$. In order to quickly achieve convergence of the network parameters, we adopted the same training strategy in the above experiments, in which the parameters of high-bit models are used to initialize low-bit models. We used the average precision (AP) to measure the performance of the detector on each category and mean average precision (mAP) to evaluate multi-target detector performance. Table IV shows the object detection results of our method and its counterparts, including its full-precision counterpart and BWN. It is clear that our method with $8$-bit encoding scheme can yield very similar performance as its full-precision counterparts. Especially for ResNet-50, we achieve the best results of $78.4\%$ mAP in the case of $8$-bit precision that is better than its full-precision counterpart by $1.2\%$ mAP. When we use $6$-bit to encode both activation values and weights, the evaluation index drops slightly. If the number of encode bits is constrained to $4$, the performance of this task has visibly deteriorated. Especially for MobileNetV2, when the number of the encoding bits is converted to $4$ from $6$, the mAP result drops monotonically from $70.5\%$ to $55.1\%$. For some categories (e.g., bottle, chair and plant), the AP results dramatically drop. In our experiments, we also implemented the binary weight form of different basic networks. Obviously, our method with $4$-bit encoding has a higher accuracy advantage over BWN [33]. For object detection tasks, fixed extremely low-bit precision network seems powerless. Especially for MobileNetV2, BWN achieves the result of $21.8\%$ mAP that is not suitable for many industrial applications.

To compare the performance of different models with different precisions more conveniently, Fig. 3 shows some detection results on the examples of the VOC2007 test set. Detections are shown with scores higher than $0.5$. The detection output pictures, which are tested by different precisions, are shown in the same column. From all the results, we can see that $8$-bit and $6$-bit encoding precisions have similar performance as their full-precision counterparts. There is almost no obvious difference between those detection examples. However, when the encoding precision is constrained to $4$-bit, we can see that some offsets are produced on the bounding box (e.g., bike). Meanwhile, some small objects and obscured targets may be missed (e.g., cow and sheep).

In fact, it is not necessary to constrain both activation values and weights to the same bit precision. Table 5 shows the results of our method with four different bit precisions based on the InceptionV3 network. For the first case, we constrain activation values and weights to $8$-bit and $4$-bit, respectively. For the second case, we constrain activation values and weights to $4$-bit and $8$-bit, respectively. The two cases have the same computational complexity, however, they have different compression ratios and results. The results of the third case, where activation values and weights are constrained to $6$-bit and $3$-bit, respectively, are obviously better than the fourth case, where activation values and weights are constrained to $3$-bit and $6$-bit, respectively. Therefore, the diversity of activation values is more important than weights. In order to mitigate the performance degradation of QNNs, we should encode activation values with a higher precision than weights.

For the object detection task, our method with a relatively low encoding precision (e.g., $6$-bit) yields good performance on the SSD framework. Similarly, it can be applied to other frameworks, e.g., R-CNN [67], Fast R-CNN [68], SPP-Net [69] and YOLO [70].

As a typical task in the field of unmanned driving, image semantic segmentation has attracted many scholars and companies to invest in research. The application scenario for this task usually relies on small devices, and therefore, the model compression and computational acceleration seems particularly important for real-world applications. We evaluate the performance of our method on the PASCAL VOC2012 semantic segmentation benchmark, which is consisted of $20$ foreground object classes and one background class. We use pixel accuracy (pixAcc) and mean IoU (mIoU) as the metrics to report the performance.

We used DeepLabV3 [71] based on ResNet-101 [66] as the basic network in this experiment. Similarly, we use different encoding precisions to evaluate the semantic segmentation performance of our method. Fig. 5 shows the evaluating indicators of different encoding precisions, where the dotted line denotes the mIoU of the full-precision network, and the solid line denotes the pixAcc of the full-precision network. From the other curves, we can see that with the increase of encoding precisions, the semantic segmentation performance is becoming better. For instance, when the encoding precision is $7$-bit, our method achieves the best performance ($98.9\%$ mIoU and $95.2\%$ pixAcc), which is much better than its full-precision counterpart ($98.5\%$ mIoU and $92.9\%$ pixAcc). Fig. 4 shows some visualization results on VOC2012 with different encoding precisions. Obviously, the segmentation results can become more precise with higher encoding bits. For example, from the fourth image we can see that as the increase of encoding precisions, the detail of segmentation of horse’s legs is becoming more and more clear.

In this experiment, we evaluated the performance of our method based on the DeepLabV3 framework. However, it can also be applied to FCN [72] and PSP [73]. Besides semantic segmentation tasks, our method can be applied to other real-word tasks, e.g., instance segmentation.

Courbariaux et al. [17] used a $8192\times 8192\times 8192$ matrix multiplication on a GTX750 NVIDIA GPU to illustrate the performance of XNOR kernel, and can achieve $23\times$ speedup ratio than the baseline kernel. Therefore, we will use the same matrix multiplication to validate the performance of our method by theoretical analysis and experimental study.

In this subsection, we analyze the speed-up ratios of our method with different encoding bits. Suppose there are two vectors $x\!=\![x^{1},x^{2},...,x^{64}]$ and $w\!=\![w^{1},w^{2},...,w^{64}]$, which are composed by $64$ real numbers. If we use the $32$-bit floating-point data type to represent each element of the two vectors, the element-wise vector multiplication should require $64$ multiplications. However, after each element of the two vectors is quantized to a binary state, we use $1$-bit to represent their elements, and can use the $64$-bit data types to represent the two vectors. Then the element-wise multiplication of the two vectors can be computed at once. This method is sometimes called SIMD (single, instuction, multiple data) within a register. The details are shown in Fig. 6. The general computing platform (i.e., CPU or GPU) can perform a $64$-bit binary operation in one clock cycle. In addition, an Intel SSE (e.g., AVX or AVX-512) instruction can perform $128$ (or 256 and 512) bits binary operations. This mechanism is very suitable for the FPGA platform, which is designed for arithmetic and logical operations. Therefore, our method can achieve model compression, computational acceleration and resource saving in the inference process.

Suppose the length of two vectors is $N$. We use $S_{MK}$ to denote the speedup ratio of our method by constraining activation values to $M$-bit and weights to $K$-bit, where $M,K\!\in\!\{1,2,...,8\}$, and $S_{MK}$ can be formulated as follows:

where $\gamma$ is the speedup ratio between the multiply-accumulate operation (MAC) and bitwise operations, $\beta$ denotes the speedup ratio between addition compared and bitwise operations, and $L$ represents the length of register (e.g., 8, 16, 32, and 64). If we use $O(1)$ to represent the time complexity of $1$-bit operations, we can use $O(MK)$ to denote the time complexity of our method by constraining activation values to $M$-bit and the values of weights to $K$-bit. Following TBN [19], we can know that $\gamma\!=\!1.91$ and $L\!=\!64$. According to the speedup ratio achieved by BWN [33], we can safely assume $\beta\!=\!\gamma/2$. Thus, the matrix multiplication after encoded by $2$-bit can obtain at most $\sim\!\!15.13\times$ speedups over its full-precision counterpart.

Besides the theoretical analysis, we also implemented the multiplication of two matrices via our $\{-1,+1\}$ encoding scheme on GTX 1080 GPU. Please refer to [17] for the experimental design and analysis. In our experiments, we first get the encoding numbers of two matrices, and then store those numbers by bits. The matrix multiplication after encoded by $2$-bit has obtained at most $\sim\!15.89\times$ speedup ratio than the baseline kernel (a quite unoptimized matrix multiplication kernel). Because of the parallel architecture of GPU, the experimental result is slightly higher than its theoretical speedup ratio. Fig. 7 shows the speedup ratios of different encoding precisions (from $1$-bit to $6$-bit), where both $M$ and $K$ denote the number of encoding bits for weights and activations, respectively. From all the results, we can see that the computational complexity is gradually increasing with the increase of encoding bits. Thus, users can easily achieve different encoding precisions arbitrarily according to their requirements (e.g., accuracy and speed) and hardware resources (e.g., memory). The code can be available at: https://github.com/qigongsun/BMD.

As described in [20], there exists a nonlinear mapping between quantized numbers and their encoded states. The quantized values are usually restricted to a closed interval $[-1,1]$. For example, the mapping is formulated as follows: