SpikeDyn: A Framework for Energy-Efficient Spiking Neural Networks with Continual and Unsupervised Learning Capabilities in Dynamic Environments

We performed experiments that provide dynamic scenarios by feeding consecutive task (i.e., class²²2The SNN community uses term “task” which refers to “class” in dataset.) changes to the network. First, a stream of samples for digit-0 is fed. Then, the task is changed to digit-1. This process is repeated for other tasks without re-feeding previous tasks, and each task has the same number of samples. More details of the experimental setup are presented in Section IV. The results are presented in Fig. 1, from where we make the following key observations.  The baseline does not efficiently learn new tasks from digit-2 onward, as most of the synapses are already occupied by previously learned tasks (digit-0 and digit-1), and mix new information with the existing ones.  The state-of-the-art improves the accuracy over the baseline at a cost of an energy overhead due to: (a) a large number of weights and neuron parameters from excitatory and inhibitory layers, and (b) the complex exponential calculations for computing multiple spike traces, membrane-potential and threshold-potential decay, and weight decay. These observations expose several challenges that need to be solved to address the targeted problem, as discussed below.

• •

  The SNN system should employ a simple yet effective learning algorithm at run time that achieves high accuracy, in both the dynamic and non-dynamic scenarios.

• •

  It should reduce the non-significant weights or neuron parameters, and the complex exponential calculations, to optimize the energy consumption.

• •

  The memory- and energy-constraints should be considered in the design process to meet many design scenarios.

To address the above challenges, we propose SpikeDyn framework (Section III) for developing energy-efficient SNNs considering both the training and the inference phases, with continual and unsupervised learning capabilities in dynamic environments. The SpikeDyn employs the following key mechanisms (see Fig. 2).

1. 1.

   Reducing the energy consumption of the neuronal operations (Section III-B) by replacing the inhibitory neurons with the direct lateral inhibitory connections.

2. 2.

   An SNN model search algorithm (Section III-C) under the given memory- and energy-constraints. It quickly estimates the memory footprint and energy consumption of the investigated SNN models using our analytical models that leverage the network parameters, the bit precision, the energy for processing an input, and the number of samples.

3. 3.

   An algorithm for achieving a continual and unsupervised learning (Section III-D) through the following means: (a) adaptive learning rates, (b) synaptic weight decay, (c) adaptive membrane threshold potential, and (d) reduction of the spurious weight updates.

Key Results: We evaluated our SpikeDyn for accuracy and energy consumption using Python-based simulations with the MNIST on the Embedded GPU and the GPGPUs, to show the generality of our solution. For a network with 400 excitatory neurons, when compared to the state-of-the-art [7], SpikeDyn reduces the energy consumption by up to 66% (avg. 51%) for training and up to 54% (avg. 37%) for inference. It also improves the accuracy by up to 29% (avg. 21%) for the most recently learned task and by up to 37% (avg. 8%) for the previously learned tasks, in dynamic scenarios.

Fig. 3 shows the detailed flow of our SpikeDyn framework with the following key steps, which are explained in the subsequent sections.

1. 1.

   Reducing the operations in SNN model to minimize the energy consumption, through a replacement of the inhibitory neurons with the direct lateral inhibitions. Thus, the operations in the inhibitory layer are eliminated.

2. 2.

   An SNN model search algorithm that explores different number of excitatory neurons to find a set of Pareto-optimal SNN models, while considering the memory and energy constraints. To quickly perform the search, we also propose analytical models that achieve less than 5% errors compared to the actual run, by incorporating:

   1. (a)

      the number of weights and neuron parameters, and the bit precision, to estimate the memory footprint,

   2. (b)

      the energy consumption for processing an input sample, and the number of samples that will be processed, to estimate the energy consumption.

3. 3.

   A continual and unsupervised learning algorithm that employs the following techniques.

   1. (a)

      Adaptive learning rates that determine the potentiation and depression factors in the STDP-based learning, using the presynaptic and postsynaptic spike activities.

   2. (b)

      Synaptic weight decay that helps the network to gradually remove the weak synaptic connections (which represent old and insignificant information), thereby enabling the synapses to learn new information.

   3. (c)

      Adaptive membrane threshold potential that provides balance in the neurons’ internal, so that the neuron generates spikes only when the corresponding synapses need to learn the input features. It is determined by the threshold potential value and its decay rate.

   4. (d)

      Reduction of the spurious weight updates in the presynaptic and postsynaptic spike event by employing timestep to carefully perform weight potentiation and depression.

Previous works in the continual and unsupervised learning for SNNs, used the network architecture shown in Fig. 1(a), which consists of the input, excitatory, and inhibitory layers [7][12]. We observe that the inhibitory neurons have different parameters from the excitatory ones to be saved in memory. Therefore, employing such an architecture will consume high memory and energy. To address this issue, we reduce the neuronal operations to substantially decrease the memory footprint and the energy consumption, as shown in Figs. 4(a)-4(c). We also observe that, the optimized architecture still achieves similar accuracy profile as of the baseline, as shown in Fig. 4(d). Therefore, we will improve the accuracy of the optimized architecture with our learning mechanism, as discussed in the Section III-D.

Each application has memory- and energy-constraints that need to be considered in the model generation. Towards this, we propose an algorithm to search for an appropriate model size for a given SNN architecture that meets the design constraints (see Alg. 1). The idea is to explore different sizes of SNN model and estimate their memory and energy consumption in both training and inference phases, using the proposed analytical models.

For each investigated SNN models’ size, the memory footprint ($mem$) is estimated using $mem=(P_{w}+P_{n})\cdot BP$, that leverages the number of weights ($P_{w}$) and neuron parameters ($P_{n}$), and the bit precision ($BP$). The reason is that, the above aspects are dominant factors in determining the size of an SNN model. Meanwhile, the total energy ($E$) is estimated using $E=E_{1}\cdot N$, that leverages the energy for processing a single sample ($E_{1}$), and the number of samples that will be processed ($N$). The number of samples $N$ is important, as the deployed systems might have different number of samples available from the environment. If the estimated memory $mem$ and energy $E$ are within the memory constraint ($mem_{c}$) and energy constraint ($E_{c}$) respectively, then the investigated SNN model is selected as the candidate of solution. Our algorithm then selects the largest-sized SNN model from the candidates as the solution, since larger network usually can achieve higher accuracy [6]. We validated our analytical models against the actual execution runs, see the results presented in Fig. 5(a) for memory footprint, and Figs. 5(b)-5(c) for energy consumption of training and inference, respectively. The results show that our analytical models achieve less than 5% errors compared to the actual runs. Thus, they are suitable to the fast estimation need. Employing our algorithm is beneficial, rather than actually running all possible SNN configurations and selecting the desired one at the end, since it saves the exploration time, as shown in Figs. 5(d)-5(e).

Our SpikeDyn employs an algorithm that incorporates the following techniques (see pseudo-code in Alg. 2).

Adaptive Learning Rates: Our algorithm employs the potentiation factor ($k_{p}$) and the depression factor ($k_{d}$) to define the learning rates for weight potentiation and depression. The idea is adjusting the potentiation factor $k_{p}$ to have high value when the corresponding synapses need to learn input features, which is indicated by the occurrences of postsynaptic spikes. The value of $k_{p}$ is obtained by normalizing the maximum accumulated postsynaptic spikes ($maxSp_{post}$) with the spike threshold ($Sp_{th}$); see Eq. 1(a). Meanwhile, the depression factor $k_{d}$ provides weight depression when the corresponding synapses need to weaken the connections, which is indicated by no occurrences of postsynaptic spikes. The value of $k_{d}$ is obtained by the ratio between the maximum accumulated postsynaptic spikes ($maxSp_{post}$) and presynaptic spikes ($maxSp_{pre}$); see Eq. 1(b). These factors are incorporated into the improved STDP-based learning algorithm; see Eq. 2. Here, $\Delta w$ denotes the weight change, $\eta_{pre}$ and $\eta_{post}$ denote the learning rate for a pre- and post-synaptic spike, $x_{pre}$ and $x_{post}$ denote the pre- and post-synaptic traces, respectively.

Synaptic Weight Decay: We employ a weight decay to gradually remove the old and insignificant information, which are represented by small weight values. It follows equation $\tau_{decay}\cdot(dw/dt)=-w_{decay}\cdot w$, with $\tau_{decay}$ denotes the decay time constant and $w_{decay}$ denotes the weight decay rate. In this manner, weak connections will get more disconnected over the training period. We define the value of $w_{decay}$ to be inversely proportional to the size of network ($w_{decay}\propto 1/n_{exc}$), with $n_{exc}$ denotes the number of excitatory neurons. The reason is that, a smaller network has less number of synapses for learning new information, while retaining the old ones. Therefore, it needs to forget the old information at a faster rate than the larger network. We observe that an appropriate $w_{decay}$ can improve accuracy, as shown by the label- in Fig. 6.

Adaptive Membrane Threshold Potential: The threshold potential is defined by $V_{th}+\theta$, as discussed in Section II. We observe that the adaptation potential $\theta$ has an important role to determine whether a neuron would generate spikes easily for later inputs. If $\theta$ is too high, the neurons will not spike easily for later inputs, since the threshold potential is already adjusted for recognizing earlier inputs. In the context of dynamic scenarios, the network will face difficulties when learning new tasks. If $\theta$ is too low, the neurons will spike easily for any inputs. In the context of dynamic scenarios, the network will quickly forget old information. Thus, the threshold potential should be balanced, so that some neurons are available for recognizing new features, while the others retain the old yet significant information. Towards this, we define the adaptation potential $\theta$ to be proportional to its decay rate $\theta_{decay}$ and the presentation time of a sample $t_{sim}$, and can be stated as $\theta=c_{\theta}\cdot\theta_{decay}\cdot t_{sim}$, with $c_{\theta}$ denotes the adaptation constant. An appropriate $\theta$ can improve accuracy, as shown by the label- in Fig. 6.

Reducing the Spurious Weight Updates: Previous work [3] has observed that there are spurious updates in SNNs, which can degrade the accuracy. These are observed in two cases: (1) when the neurons spike unpredictably, due to the random weight initialization; and (2) when a neuron spikes for patterns that belong to different classes, due to the overlapped features. We exploit this observation in a novel way to reduce the spurious updates that are induced by both the pre- and post-synaptic spikes. The idea is to employ a timestep, and then monitor whether at least one postsynaptic spike happens. If so, then the weight potentiation will be conducted, and otherwise, the weight depression will be conducted (see Fig. 7).

Dynamic Environments: We evaluated the classification accuracy for two cases, i.e., (1) classifying the most recently learned task, which represent the capability of learning new task; and (2) classifying the previously learned tasks, which represent the capability of retaining old information.

Case-1: Figs. 9(a.1) and 9(b.1) show the accuracy when the network classifies the most recently learned task (or learning new task), for N200 and N400 respectively. The ASP achieves better accuracy than the baseline since the ASP employs adaptive learning rate and weight decay, while the baseline does not consider the dynamic tasks in its learning. Here, our SpikeDyn improves the learning capabilities more than the ASP. The SpikeDyn improves the accuracy by up to 38% (avg. 23%) than the ASP for N200, and by up to 29% (avg. 21%) for N400. The reason is that, the SpikeDyn employs: (1) a more careful mechanism to determine the rates of weight potentiation and depression for learning new features, (2) an appropriate threshold potential to adjust some neurons to be active in the learning process, (3) weight decay rate that effectively removes the old and insignificant information, and (4) reduction of the spurious weight updates.

Case-2: Figs. 9(a.2) and 9(b.2) show the accuracy when the network classifies the previously learned tasks (or retaining old information), for N200 and N400 respectively. The baseline performs the worst as it does not decrease the weights, thereby suffering from mixed information in its synapses. Here, our SpikeDyn shows comparable accuracy to the ASP. The SpikeDyn improves the accuracy by up to 36% (avg. 4%) than the ASP for N200, and by up to 37% (avg. 8%) for N400. The reason is that, the SpikeDyn employs: (1) a threshold potential that tunes some neurons to be inactive in the learning process, thereby retaining the old yet significant information, and (2) weight decay rate that does not remove the old yet significant connections. Furthermore, we also observe that, some classes are relatively difficult to learn in dynamic scenarios, especially in the case of retaining old information. For instance, in N400 case, the accuracy for classifying digit-4 is low, as indicated by label- in Fig. 9(b.2). It is because a considerable number of misclassification happens when the digit-4 is recognized as another digit (i.e., digit-9), as indicated by label- in Fig. 10(b). The reason is that, the learned features from digit-4 are gradually changed to represent the features of digit-9 over a training period, due to their overlapped features and the sequence of learning tasks. Therefore, some neurons that recognize digit-4 at the early of the training period, are changed to recognize digit-9 at the end of the training period.

Non-dynamic Environments: Figs. 9(c.1) and 9(c.2) show the accuracy over the presentation of training samples for N200 and N400, respectively. The results show that our SpikeDyn achieves comparable accuracy to other techniques. The reason is that, our SpikeDyn employs effective learning rates to potentiate and decrease the weights, while reducing the spurious updates. In this manner, each weight update is adjusted appropriately, and hence the accuracy is maintained. Such observations are important, as SNN systems may have a different number of training samples available from the environment. Thus, the users can devise a strategy to define the minimum training samples for achieving the targeted accuracy.

Fig. 11 shows that SpikeDyn reduces the energy consumption compared to other techniques, across different sizes of network and different GPUs, for both the dynamic and non-dynamic environments. For N200, our SpikeDyn reduces the energy consumption from the ASP by up to 59% (avg. 57%) for training, and by up to 54% (avg. 51%) for inference. For N400, our SpikeDyn reduces the energy consumption from the ASP by up to 66% (avg. 51%) for training, and by up to 54% (avg. 37%) for inference. The energy savings in training come from the elimination of the inhibitory neurons, the reduction of spurious updates and exponential calculations. Meanwhile, the energy savings in inference mainly come from the elimination of inhibitory neurons. Furthermore, we also observe the processing time of the SpikeDyn for training and inference phases (see Table II). The results show that running an SNN model on the Embedded GPU (Jetson Nano) requires a longer time than the GPGPUs, as the Embedded GPU has less number of cores, memory size and bandwidth. Therefore, the users should devise a strategy for defining the number of samples in the training and inference phases, to comply with the use cases’ requirements, especially on the embedded applications.