A $0.11-0.38$ pJ/cycle Differential Ring Oscillator in $65$ nm CMOS for Robust Neurocomputing

From the analysis of conventional ring oscillators in section II and simulations shown later in this section (Fig. 5), we know that current starved fully differential structure has both better PSRR and CMRR. To offer differential inputs, outputs and a positive feedback hysteresis, each stage of the conventional differential RO uses four inverters (i.e. $8$ transistors) except for the current starved transistors (Fig. 2). Our novelty stems from the observation that only four transistors seem necessary to realize oscillation as shown in Fig. 3(a). MP1 and MN2 are the pull-up and pull-down differential I/O pair while MN3 and MP4 serve as their loads respectively. At the same time, MN3 and MP4 constitute the simplest dynamic positive feedback latch. Another way to interpret this proposed structure is as follows: MP1 and MN3 constitute a dynamic inverter since $outn$ and $inp$ are almost the same (with a certain phase shift), similarly, MP4 and MN2 constitute another dynamic inverter since $outp$ and $inn$ are almost the same (with a certain phase shift). Likewise, the two dynamic inverters compose the regeneration cross pair to generate positive feedback and offer extra hysteresis phase shift. Every transistor acts as an active device for itself and a load device for another transistor. The push-pull nature and transistor reuse make the cell compact and efficient.

However, one problem is that $inp$ and $inn$ are strictly inverse only when oscillations have been sustained, while $outn$ and $inn$ are inverse with a phase delay of $180^{\circ}/N$. If for some reason, the initial condition of $inp$ and $inn$ (Fig. 3 (a)) are same at say $0$ V, then $outp$ is guaranteed to be pulled up to $V_{dd}$ but the condition of $outn$ stays at initial condition since both MN2 and MP4 are OFF. If initial condition of $outn$ was $V_{dd}$, then the outputs of this stage are both at high voltage and the oscillator gets locked in this state and cannot start oscillating. So a start-up circuit is necessary to maintain robust oscillations. Only a NMOS or PMOS or both (Fig. 3(b)) can eliminate the possible stable state and work as start-up circuit. In the above example, when both $inn$ and $inp$ are $0$ V, now with added MP2, it will force $outn$ to $V_{dd}$ forcing the outputs of this stage to be opposite in polarity thus breaking the lock state earlier. From SPICE simulations, the structure with start-up of both NMOS and PMOS as shown as Fig. 3(b), gives the best frequency performance (see Section V) and is adopted in the rest of the paper.

The novelty in our design of reducing the number of transistors should lead to an increased oscillation frequency per unit current. However, the charging currents also change and hence the combined effect is not obvious. Intuitively, the proposed delay stage offers less charging and discharging current by removing MP3 and MN4, which cuts down on power consumption. To precisely compare the charging/discharging current and the effect on the output frequency, we analyze the load capacitance and charging current model of a single stage in more details to present a theoretical prediction.

First, the load capacitor of each stage for the conventional structure (Fig. 2) is given by:

In comparison, for the proposed structure (Fig. 3(b)), it reduces to

Thus, ${C_{L\_prop}}=3/4{C_{L\_conv}}$. It is therefore reasonable to expect that the load capacitor decreases by $25\%$ since the number of transistors of each stage decreases by the same amount.

Next, we analyze the charging process of the node of $outp$ in Fig. 3(b), while the discharging process is same as the node of $outn$. Fig. 3(c) illustrates the charging period, which is divided into four phases (A, B, C and D) according to the operation of the transistors. $V_{max}$ and $V_{min}$ are the swing range of the oscillation, which is lower than rail-to-rail because of the current starved structure. In our design, $V_{max}\approx 900$ mV, $V_{min}\approx 300$ mV, and $V_{cm}\approx 600$ mV. For simplicity, we assume that $V_{TN}=|{V_{TP}}|=V_{T}$, $\Delta V_{1}=\Delta V_{4}=50$ mV and $\Delta V_{2}=\Delta V_{3}=250$ mV such that $\sum_{i=1}^{4}\Delta V_{i}=V_{max}-V_{min}=600$ mV.

where, $I_{i\_conv}$ and $I_{i\_prop}$ represent charging current in four different phases of conventional and proposed stage respectively, which are given by

The equations and approximated average values of $I_{MP1}$, $I_{MP3}$ and $I_{MN3}$ in all phases are listed in Table I, where the charging process is divided into the four phases A, B, C and D as mentioned earlier. The regions of operation of the MOSFET are referred to as 0, 1 and 2 for cut-off, linear and saturation respectively.

The drain current of short-channel MOSFETs is assumed to follow the widely used alpha power law  [22],  [23]. The carrier velocity saturation coefficient $\alpha$ is between $1.2$ to $1.5$ for sub-micron CMOS technology. Then, according to the above equations and the estimated values in the table, we can get the propagation delay relationship between the proposed and conventional RO as $t_{d\_prop}=88.4\%t_{d\_conv}$. Thus, the relation between the frequencies are:

This means that the frequency of proposed RO increases $15.6\%$ compared with conventional structure for the same input current. From equation 1, the average current relationship is expressed as:

This implies the average current of the proposed RO decreases by $13.3\%$ compared to the conventional structure.

To verify these theoretical models, we conducted the current-frequency (I-F) transfer curve simulations of the conventional and proposed $4$ stage RO using $65$ nm CMOS models in SPICE. Fig. 4 shows the simulation results and the theoretical prediction results of the output frequency. As predicted, the proposed oscillator indeed produces higher frequency than the conventional one for the same input current. However, the theoretical values are slightly greater than that of simulation because of the inaccuracy in estimation of the current of the transistor MP3.

While the increase in oscillation frequency is good, it is not useful if the new oscillator has a degradation in other key metrics such as robustness and jitter. We first evaluate the robustness of the proposed oscillator in comparison to the conventional one through simulations. The simulation results of line sensitivity ($\%/V$) and temperature sensitivity ($\%/T$) of four types of RO-CCO is shown in Fig. 5. As expected, the characteristics of differential current starved structures, both proposed and conventional, are better ($\approx 18\%$ for line sensitivity and $<0.014\%$ for temperature sensitivity) than non current starved pseudo differential RO ($\approx 139\%$ for line sensitivity and $\approx 0.14\%$ for temperature sensitivity). For current starved structures, differential structures are better than single ended ($\approx 61\%$ for line sensitivity and $\approx 0.06\%$ for temperature sensitivity) as expected. It can be seen that the line sensitivity and temperature sensitivity of the proposed differential structure is not degraded compared to the conventional differential structure. The relatively high value of sensitivity to power supply is traced back to the tail current sources coming out of saturation at power supply voltages lower than $0.9$V–this can be solved by reducing the current range. Confined to $1-1.2$V for power supply, the sensitivity is only around $1.5\%$.

Following [24] , the variance of period jitter for a RO can be expressed as:

where,

$N$ is the number of delay stage. $\gamma_{N}$ and $\gamma_{P}$ are technology-dependent noise factors for NMOS and PMOS respectively. Thus,

In our implementation, the load capacitor and average current decrease by $25\%$ and $13.3\%$ respectively.

So, the jitter of the proposed CCO is expected to be approximately the same as the conventional structure. This has been confirmed in simulations and measurement–we present these results in the next section.

For usage within a neural network system, the raw frequencies of the CCO need to be often converted to a digital word. The easiest method to do this is to pass the CCO output as a clock to a counter[12]. However, this method incurs a large conversion time ($\propto 2^{N}$) and concomitantly large conversion energy. A different approach, following [11] is used in our work as described next.

Fig. 6 shows the diagram of time to digital converter (TDC) with a four delay stage differential CCO and the sequential counters. Although the four stage differential CCO outputs eight phases totally, only half are unique with separate phase information while the rest are exactly the inverse of the unique phases and offer the same phase information. So only four phases with $45^{\circ}$ phase shift of each other, as shown in Fig. 6 (a), are utilized to generate the phase code counter (PC-CNT). Only $360^{\circ}/45^{\circ}=8$ codes are generated by the four stages differential CCO and hence only 3-bits are available in the PC-CNT acting as fine counter. The sequence diagrams of the four choice outputs and the corresponding complementary symmetry phase codes are illustrated in Fig. 6(b). The opposite phase of the last phase of PC-CNT is chosen to clock a Gray code counter (GC-CNT) serving as coarse counter. The number of bits of GC-CNT depends on the input current range while the number of bits in PC-CNT determines the accuracy of the converter. A Gray code counter is selected for higher reliability in neural networks with tightly packed layout. To enable wide dynamic range testing, a 12-bit GC-CNT is used in our design but the bit width can be optimized in neural network applications.

Both the phase code counter and Gray code counter possess the merits of energy efficiency and reliability since only one bit trips during state transition. Regarding the choice of number of stages in the CCO, we note that more stages of CCO structure ($\times M$) offers more valid output phases ($\times M$) and thus more bits in PC-CNT but has lower frequency and thus less bits in GC-CNT. Overall, the total number of bits is constant but more CCO stages consume more area overhead ($\times M$). Assuming the same input current for both cases, the tail current for CCO core is constant and hence energy per conversion due to CCO is constant. Energy dissipated in the counter is less for more CCO stages, but this is much smaller than the CCO energy dissipation. Hence, due to the need of small footprint of the CCO, a $4$-stage CCO core is designed in our work.

In this section, we explore potential application of the proposed CCO as a neuron in neural network (NN) implementations. The custom activation function of the proposed neuron is modelled using experimental results of the CCO (Fig. 9 and Fig. 11). The I-F transfer curve of CCO has a high sensitivity to process variations and hence the neurons on a chip are expected to have mismatch. Fig. 13 shows the mismatch of CCO through Monte Carlo (MC) simulation. As seen from the figure, the standard variation is $1.8$ MHz at the mean frequency of $39.7$ MHz leading to a coefficient of variation of $\approx 5\%$. The mismatch of CCO is observed by performing MC runs while sweeping input currents. The variability in the slope of this I-F curve is included in the NN simulation as shown next. Note that synaptic non-idealities are a separate issue and has to be considered for the full system as well. This is beyond the scope of this work but many strategies to handle synaptic non-idealities such as mismatch, low-precision and write non-linearity have been published[12, 37, 38, 39, 40].

At first, the neural networks are trained and tested using ReLu activation function to establish a benchmark for further evaluation of proposed activation function. For hardware simulation, we adopted two strategies. In the first one, the networks are trained using ReLu activation and the custom activation function is introduced only during inference. In the second strategy, the custom activation is used both during training and inference. The simulation details are described in detail in algorithm 1.

We used three different datasets (MNIST [41], fashion MNIST [42], CIFAR-10 [43]) and two network topologies (one ANN and one CNN) to evaluate the performance of the proposed neuron model. Since the primary aim of these simulations is to verify the viability of the CCO based neuron, instead of experimenting with different network architectures, we used the same ANN and CNN architectures for all the datasets and compared the performance of the hardware simulations with their pure software counterpart. The ANN comprises of two fully connected hidden layers with 800 and 300 neurons respectively. The CNN network architecture used can be described as: $A\times B\times C-32c5-m2-64c5-m2-128c1-m2-fc512-0.5d-fc10$, where the input dimension is $A\times B\times C$, $XcY$ represents a convolution layer with $X$ convolution filters with $Y\times Y$ size, $mZ$ represents a $Z\times Z$ 2D max-pooling layer, $Pd$ represent a dropout layer with dropout rate $P$ and $fcL$ represent a fully connected dense layer with $L$ neurons. All models are trained and tested through google colaboratory on a server with Intel Xeon CPU and NVIDIA Tesla K80 GPU. All the models are trained with categorical crossentropy loss and Adam optimizer and the testing accuracies for all the datasets and network topologies are averaged over 3 trials.

The results (mean across 3 trials) are shown in table IV. The key observations from the results are as follows: firstly, for simple datasets like MNIST, there is almost no loss of accuracy even if the custom activation function is introduced only during inference. For a relatively more complex fashion MNIST dataset, there is a small loss ($\sim 1\%$) in testing accuracy, while for more complex CIFAR10 dataset, the loss of accuracy becomes significant ($\sim 2.5-4\%$). Secondly, the loss in accuracy can be almost completely recovered even for complex datasets like CIFAR-10, if the custom activation is introduced during training i.e. weight learning through back-propagation is able to correct for any non-ideality resulting from the custom activation. Thirdly, in some cases, when the custom activation is introduced during training, the testing accuracies seem to surpass the software accuracy. This can be attributed to Gaussian Noise injection in the custom activation acting as noise regularization and improving overall network performance [44]. For all the datasets and both hardware simulation settings, the loss of accuracy is similar for both ANN and CNN network topologies. Since the ANN uses only two hidden layers while the CNN architecture consists of large number of layers, this observation establishes the scalability of the proposed neuron model. Finally, we evaluated in detail the variability in the performance of the classifiers caused by random variation of the custom activation function described in step $5$ of algorithm 1. For MNIST classification task with ANN and CNN, we evaluated the classification accuracies over $20$ trials where the slopes of the custom activation are randomly generated for each trial. The standard deviation of classification accuracy for ANN and CNN are $0.15\%$ and $0.10\%$ respectively. This goes to show that the performance variation introduced by the slope variability is very small and therefore, further proves the viability of the proposed neuron model.

Changes in power supply voltage and temperature are ideally common mode perturbations and affect the CCO less due to differential structure. Moreover, due to sharing of power supply, if all neurons are affected by the same factor, RELU based neural networks will not be ideally affected. However, due to variations between neurons, there will be some effect in real implementations. To capture this effect, Monte-Carlo simulations were done to find distributions of supply voltage sensitivity (temperature sensitivity is almost negligible as shown in Section III). The resulting sensitivity distribution had a mean of $-0.78\%$ and standard deviation of $1.74\%$ when $V_{dd}$ varied in the range of $1-1.2$ V. Including this as an additional perturbation for the neuron outputs during inference, the results for CIFAR-10 were re-evaluated. We observed negligible reduction in accuracy of $<0.5\%$ confirming our hypothesis that the network is relatively robust against supply variations.

The earlier sections showed the usage of the CCO as a rate encoded neuron similar to its use in [12, 11]. However, it may also be used as a spiking neuron with precisely timed output spikes as needed in spiking neural networks (SNN). While one way to use the CCO as a spiking neuron is shown in [9], we show next how some of the critical features can be included in the CCO much easily. We identify one of the phases of the CCO as the spike phase and the corresponding output of the fine counter is plotted as spike output.

First, we show in SPICE simulations that adding a leak current source[45] next to the PMOS input current mirror in Fig. 8 can act to suppress spurious neuron spikes at low input currents. The resulting I-F curve plotted in Fig. 14(a) shows that indeed valid neuron firing starts for input currents larger than $I_{leak}=1nA$. Next, the response of the CCO neuron to a step of magnitude $I_{in}=1uA$ is shown in Fig. 14(b). As expected, the neuron fires spikes only when the step is applied. Lastly, to see how to integrate such a neuron in a network, we show the response of the CCO neuron to spike inputs that may come from other neurons in the network. Plotted in Fig. 14(c), it shows that for input synaptic current pulses of magnitude $100nA$, the neuron integrates the charge on the PMOS current mirror and produces an output spike for every $4$ input spikes.

In Section III, we proposed the novel structure of CCO and explained the necessity of a start-up circuit. A robust oscillation can be ensured by a start-up circuit with only a PMOS transistor or a NMOS transistor or both. As shown in Fig. 15 (a), a start-up circuit is equipped at the differential output nodes to eliminate the uncertainty state. The start-up circuit would be a single PMOS acting as a pull-up device to eliminate the uncertainty of $outn$, as shown in Fig. 15 (b), or a single NMOS acting as a pull-down device to eliminate the uncertainty of $outp$, as shown in Fig. 15 (c), or both a PMOS and a NMOS to eliminate the uncertainty of both the differential outputs, as shown in Fig. 15 (d).

The different start-up circuits give different oscillation frequencies. Fig. 16 compares the simulated output frequencies of conventional and proposed structure with three different start-up circuits. The CCO with both PMOS and NMOS start-up performs $\times 11\%$ higher frequency than the conventional one while the CCOs with a single PMOS or NMOS start-up degrade the frequencies around $\times 13\%$ compared to the conventional one. Hence, we only use the startup with both PMOS and NMOS in the proposed structure.