TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers

We can consider two representative SNN models, integrate-and-fire (IF), and leaky-integrate-and-fire (LIF) ones. It is well-known that the IF model is easily converted from an ANN, considered as the SNN model throughout our work. In the IF model, at time step $t$, the neuron $i$ in the $l^{th}$ layer has the summation of weighted spike input, $z^{l}_{i}$, as follows.

, where $W^{l}_{ij}$ is the synaptic weight, $b^{l}_{i}$ refers to the bias of the neuron, and $\theta^{l-1}_{j}$ is the spike input from the neuron $j$ that is in the previous layer. In the $l^{th}$ layer, the spike output of the neuron, $\theta^{l}_{i}$, remains zero until the membrane potential, $V^{l}_{i}$, reaches the threshold $V^{l}_{thr}$. At the time that $V^{l}_{i}$ becomes larger than or equal to $V^{l}_{thr}$, the spike output is fired. Hence,

After the firing, the membrane potential, $V^{l}_{i}$, is reset. There are two approaches to reset $V^{l}_{i}$: reset-to-zero and reset-by-subtraction. Since the reset-to-zero suffers from considerable information loss [10], the reset-by-subtraction is employed throughout this work. Under this situation, at the time step of $t$, $V^{l}_{i}$ can be modeled as follows.

The ReLU function is widely used as the activation function of ANNs, given by the following one.

By comparing (4) to (1), an ANN-to-SNN converting algorithm can be obtained. In SNNs, spike output signals are binary, only ’1’ or ’0’, implying that the spike outputs do not have negative values. Therefore, (4) can be simply mapped to (1) by converting the ReLU output to the spike rates of SNNs. For the conversion, both weights and biases need to be normalized, namely data-normalization, where weights ($W^{l}$) and biases ($b^{l}$) are normalized by (5), and then, the threshold voltage of neurons is set to $1.0$.

, where $\lambda^{l}$ is the normalization factor of the current layer, so called norm-factor, $\lambda^{l-1}$ is the norm-factor of the previous layer. The decision of norm-factors is more discussed in Section III-A.

It is well-known that it is difficult to model max-pooling and batch-normalization in SNNs [10]. The max-pooling can be replaced with the other pooling techniques such as average-pooling, well-modeled in SNNs. The authors of [10] remove the batch-normalization, of (6) after the training of ANNs.

, where $a$ is the input, $\mu$ and $\sigma$ are mean and variance of mini-batch, and $\gamma$ and $\beta$ are two learned parameters of batch-normalization. To prevent the accuracy loss of ANNs due to the removal of the batch-normalization, they scale weights and biases of corresponding convolution layers by using the following equation:

Our ANN-to-SNN conversion is based on the above discussion. We apply the data-normalization based on (5) and remove batch-normalization by using (7). We replace max-pooling layers with average-pooling ones. Instead of using a soft-max layer, not modeled in SNNs, we simply count the number of spiking signals and take the maximum to classify outputs. At the first SNN layer, we feed input signals with analog values, so called real coding, same as the technique which is used in [10, 13, 15].

In ANN-to-SNN conversion, an activation of ANNs, $a_{i}$, is mapped to a spike rate of SNNs, $f_{i}$, as mentioned above. The spike rate, $f_{i}$, can be estimated by following equation.

, where $a_{limit}$ is the maximum value of activations mapped to the maximum spike rate ($f_{max}=1$), and $\Delta$ is the quantization resolution. In ANN-to-SNN conversion, the activations of ANNs are quantized and then, the quantized activations are mapped to their corresponding spike rate, where some quantization error is unavoidable. To reduce the quantization error due to ANN-to-SNN conversion, we need to make smaller $\Delta$, which can be obtained by increasing the latency, $T$, or decreasing $a_{limit}$. This is the reason why after ANN-to-SNN conversion, latency and accuracy have trade-off relation. In the ANN-to-SNN conversion with data-normalization, $\lambda$ is proportional to $a_{limit}$, where smaller $\lambda$ with the fixed $T$ tends to reduce $a_{limit}$. This results in the reduction of the quantization error. However, from a certain point, $a_{limit}$ is lower than $a_{max}$ and then, we have the conversion loss due to the clipping of activations, shown in Fig. 1.B, degrading the accuracy of converted SNNs as well. On the other hand, larger $\lambda$ accompanies the increment of the quantization error, as shown in Fig. 1.C.

The work in [9] determined the norm-factor of each layer by taking the maximum value among the activation parameters of the layer. However, in order to reduce quantization error, this approach requires large latency in SNNs. The authors of [10] and [13] decreased $\lambda$ by the following technique. In ANNs, most activations, roughly 99.0% to 99.99%, are placed in the range of of $[0,\;max/3]$. From this observation, they decide $\lambda$ by selecting the value of 99.9%. The authors of [11] and [12] chose $\lambda$ based on the operation of SNN. They sequentially run SNN layers and take the maximum weighted spike output for $\lambda$, then scaled with the factor of $0.7\sim 0.9$. Although the smaller $\lambda$ due to these techniques reduces quantization error, the conversion loss due to clipping error still causes significant accuracy degradation in SNNs.

In this work, we propose a novel technique to alleviate the trade off between quantization and clipping errors, where we decrease $a_{max}$ as small as possible while maintaining the accuracy of ANNs accuracy. Such an approach provides both low latency and high accuracy in SNNs, discussed in Section III-B.

To decide the norm-factor of a certain layer, instead of analyzing activations of the corresponding ANN layer, we add a clipping layer after the ReLUs of the ANN layer, shown in Fig. 2. The forward function of the clipping layer is described in (9). Please, note that the clipping layer has a trainable parameter, $\lambda$, which becomes the norm-factor for the data-normalization. When backward computations of ANNs are processed, the gradients of $\lambda$ are formulated as (10). $\lambda$ is trained based on the learning rule of (11), where L2-regularization is used to optimize $\lambda$.

, where $\eta$ is the learning rate, $\alpha$ is the L2-regularization coefficient, and $L$ is the ANN loss. We visualize the training process of norm-factors, shown in Fig. 3. The results are obtained from the training of VGG-16 for two datasets, CIFAR-10 and ImageNet. In (11), the L2-regularization term ($-\eta\alpha\lambda$) tends to decrease $\lambda$, the norm-fact of the current layer, as already mentioned. From a certain point, the decrement of $\lambda$ causes training loss. Under this situation, the optimization term ($-\eta\frac{\partial\bar{a}}{\partial\lambda}\frac{\partial L}{\partial\bar{a}}$) increases $\lambda$ to complement the training loss. After some epochs, $\lambda$ starts to be stabilized to the optimal value with respect to the training loss. Fig. 4 shows training and test losses of each training epoch. We compare training and test losses of the original ANN training, where our TCL is not used, to those of our TCL-based ANN training. Further, we observe validation accuracies of each epoch, shown in Fig. 5, clearly demonstrating that our TCL hardly affect the training results of VGG-16.

To further prove the efficacy of our TCL, we investigate how our TCL affect the loss of ANN-to-SNN conversion by comparing the conversion loss due to our TCL to those due to various norm-factors, 99.99%, 99.9%, 99.8%, 99.7%, and 99.5%, shown in Fig. 6. The conversion loss due to our TCL is significantly lower compared to the others, well showing that our TCL improves the accuracy of SNNs compared to [10, 11, 12, 13].

In our TCL, the L2-regularization coefficient of (11), $\alpha$, affects the accuracy of SNNs with the following two styles. Firstly, as shown in Fig. 7, larger $\alpha$ tends to provide smaller norm-factors, resulting in enhanced SNN accuracy under very small latency constraints below 50 cycles. Secondly, in our TCL-based training the accuracy of ANNs shows the trend to become lower as $\alpha$ increases, potentially lowering corresponding SNN accuracies with moderate latency conditions above 200 cycles. The above two trends clearly appear in our experiment results, shown in Table I. Since the above two trends oppositely influence the accuracy of SNNs, we need to find the optimal $\alpha$, the largest value not to affect the accuracy of ANNs. However, the algorithm to find the optimal values of hyper-parameters such as the L2-regularization coefficient is not well-established in previous researches yet, not explored in this work. From some trial and errors, we properly selected the values of $\alpha$ in this work.

Our experiment results are summarized and compared to those of state-of-the-arts (SOTAs) related to ANN-to-SNN conversion in Table II. For CIFAR-10, [8], [10], [11], and [12] achieve good SNN accuracies after the ANN-to-SNN conversion. However, large latencies are required for those techniques. Although the authors of [14, 15] alleviate the large latency problem by using their proposed hybrid training technique, where they reduce their latency to the cycles of 20$\sim$250, the accuracy loss due to the ANN-to-SNN conversion is still significant, larger than $0.5\%$. Our TCL technique makes the following two significant accomplishments compared to SOTAs.

• •

  In spite of limiting the range of activations, our TCL technique hardly affect the accuray of ANNs.

• •

  After ANN-to-SNN conversion, SNNs show accuracies comparable to their ANN counterparts with moderate latency conditions.

In the dataset of CIFAR-10, with the latency of 100$\sim$150 cycles, the accuracy loss due to the ANN-to-SNN conversion is almost negligible, less than $0.5\%$. The results of ImageNet further validate our contributions, where the training results of ANNs based on our TCL are almost same to their original accuracies. In addition, with the moderate latency of 250 cycles, we obtain good SNN accuracies, almost comparable to their ANN counterparts.

For very low latencies below 50 cycles , we train ANNs with a little larger $\alpha$. The results show almost comparable accuracy to the hybrid training of [15], optimized for small latency constraints. Unlike [15], where the direct SNN training phase in addition to the training one of ANN requires very large computational overhead, our TCL delivers good SNN accuracy with the only ANN training phase, our significant contribution as well.