Covariant spatio-temporal receptive fields for neuromorphic computing

The generalized Gaussian derivative model for receptive fields [Lindeberg, 2016, Lindeberg, 2021] defines a spatio-temporal scale-space representation

$$L(x,y;\;\Sigma,\tau,v)=\int_{\xi\in\mathbb{R}^{2}}\int_{u\in\mathbb{R}}T(\xi,u;\;\Sigma,\tau,v)\,f(\xi,u)\,d\xi\,du$$

(1)

over joint space-time $(x\in\mathbb{R}^{2},t\in\mathbb{R})$ by convolving any video sequence $f(x,t)$ with a spatio-temporal convolution kernel

$$T(x,t;\;\Sigma,\tau,v)=g(x-vt;\;\Sigma)\,h(t;\;\tau)$$

(2)

where $g(x;\;\Sigma)$ is a 2-dimensional spatial Gaussian kernel with its shape determined by a $2\times 2$ spatial covariance matrix, $v$ denotes image velocity, and $h(t;\;\tau)$ is a scale-covariant temporal smoothing kernel with temporal scale parameter $\tau$. In the original work on the generalized Gaussian derivative model, the temporal smothing kernel was chosen as either a 1-dimensional Gaussian kernel

$$h(t;\;\tau)=g_{1D}(t;\;\tau)=\frac{1}{\sqrt{2\pi\tau}}\exp(-t^{2}/(2\tau))$$

(3)

or as the time-causal limit kernel [Lindeberg, 2016, Lindeberg, 2023a]

$$h(t;\;\tau)=\Psi(t;\;\tau,c),$$

(4)

which represents the convolution of an infinite number of truncated exponential kernels in cascade, with especially chosen time constants, as depending on the distribution parameter $c$, to obtain temporal scale covariance. Here, we extend that temporal model to leaky integrators and integrate-and-fire neurons.

Image data are subject to natural image transformations, which to first-order of approximation can be modelled as a combination of three types of geometric image transformations:

• •

  spatial affine transformations $x^{\prime}=A\,x$, where $A$ is an affine transformation matrix,

• •

  Galiean transformations $x^{\prime}=x+u\,t$, where $u$ is a velocity vector, and

• •

  temporal scaling transformations $t^{\prime}=S_{t}\,t$, where $S_{t}$ is a temporal scaling factor.

Under a composition of these transformations of the form

	$\displaystyle x^{\prime}$	$\displaystyle=A\,(x+u\,t),$		(5)
	$\displaystyle t^{\prime}$	$\displaystyle=S_{t}\,t,$		(6)

with two video sequences $f^{\prime}$ and $f$ related according to $f^{\prime}(x^{\prime},t^{\prime})=f(x,t)$, the spatio-temporal scale-space representations $L^{\prime}$ and $L$ of $f^{\prime}$ and $f$ are related according to

$$L^{\prime}(x^{\prime},t^{\prime};\;\Sigma^{\prime},\tau,^{\prime}v^{\prime})=L(x,t;\;\Sigma,\tau,v),$$

(7)

provided that the parameters of the receptive fields are related according to

	$\displaystyle\Sigma^{\prime}$	$\displaystyle=A\,\Sigma\,A^{T},$		(8)
	$\displaystyle\tau^{\prime}$	$\displaystyle=S_{t}^{2}\,\tau,$		(9)
	$\displaystyle v^{\prime}$	$\displaystyle=(A\,v+u)/S_{t}.$		(10)

This result can be proved by composing the individual image transformations in cascade, as treated in Lindeberg [Lindeberg, 2023a].

In practice, this result means that if spatio-temporal video data are subject to these classes of natural image transformations, it will always be possible to capture these transformations in the receptive field responses, provided that the parameters of the receptive fields are matched accordingly. Figure 1 illustrates the covariance property in a commutative diagram. Such a matching property can substantially improve the performance of computer vision algorithms operating on spatio-temporal receptive fields. We will next extend this result to leaky integrator neuron models and event-driven computations.

Gaussian kernels are symmetric around the origin and therefore impractical to use as temporal kernels in real-time situations, because they violate the principle of temporal causality. Lindeberg and Fagerström [Lindeberg and Fagerström, 1996] showed that one-sided, truncated exponential temporal scale-space kernels constitute a canonical class of temporal smoothing kernels, in that they are the only time-causal kernels that guarantee non-creation of new structure, in terms of either local extrema or zero-crossings, from finer to coarser levels of scale:

$$h_{exp}(t,\mu)=\begin{cases}\frac{1}{\mu}\,e^{-t/\mu}&t>0,\\ 0&t\leqslant 0,\end{cases}$$

(11)

where the time constant $\mu$ represents the temporal scale parameter corresponding to $\tau=\mu^{2}$ in (2).

Coupling $K$ such temporal filters in parallel, for $k\in[1,K]$, yields a temporal multi-channel representation in the limit when $K\to\infty$:

$$h_{composed}(\cdot;\;\tau_{k})=h_{exp}(\cdot;\;\mu_{k}).$$

(12)

A specific temporal scale-space channel representation $L$ for some signal $f$ [Lindeberg and Fagerström, 1996, Eq. (14)] can be written as

$$L(t;\;\mu)=\int_{u=0}^{\infty}f(t-u)\,h_{exp}(\cdot;\;\mu)\,du.$$

(13)

Turning to the domain of neuron models, a first-order integrator with a leak has the following integral representations for a given time $t$, time constant $\mu$ and some time-dependent input $I$ [Gerstner et al., 2014, Eq. (1.11)]:

$$u(t;\;\mu)=\int_{0}^{\infty}\frac{1}{\mu}\,e^{-\xi/\mu}\,I(t-\xi)\ d\xi.$$

(14)

This precisely corresponds to the truncated exponential kernel in (11). Inserting into (13), we apply the temporal scaling operation $t^{\prime}=S_{t}\,t$ for a temporal scaling factor $S_{t}>0$ to the temporal input signals $f^{\prime}(t^{\prime})=f(t)$:

	$\displaystyle L(t^{\prime};\;\mu^{\prime})$	$\displaystyle=\int_{u^{\prime}=0}^{\infty}f^{\prime}(t^{\prime}-u^{\prime})\,\frac{1}{\mu^{\prime}}\,e^{-t^{\prime}/\mu^{\prime}}du^{\prime}$		(15)
		$\displaystyle\stackrel{{\scriptstyle 1}}{{=}}\int_{u=0}^{\infty}f^{\prime}\left(S_{t}\,(t-u)\right)\frac{1}{S_{t}\,\mu}\,e^{-S_{t}t/S_{t}\mu}\,S_{t}\,du$
		$\displaystyle=\int_{u=0}^{\infty}f(t-u)\,\frac{1}{\mu}\,e^{-t/\mu}\,du$
		$\displaystyle=L(t;\;\mu),$

where the step (1) sets $u^{\prime}=S_{t}\,u$, $du^{\prime}=S_{t}\,du$, and $\mu^{\prime}=S_{t}\,\mu$. This establishes temporal scale covariance for a single temporal filter $T(t;\;\tau)=h_{exp}(t;\;\mu)$, while a set of temporal filters will be scale-covariant across multiple scales, provided the scales are logarithmically distributed [Lindeberg, 2023a, section 3.2].

Continuing with the thresholded model of the first-order leaky integrator equation, we turn to the Spike Response Model [Gerstner et al., 2014]. This model generalizes to numerous neuron models, but we will restrict ourselves to the leaky integrate-and-fire equations (given in the Supplementary Material, section D), which can be viewed as the composition of three filters: a membrane filter $\kappa$, a threshold filter $H$, and a membrane reset filter $\eta^{\prime}$. The membrane filter and membrane reset filter describe the subthreshold dynamics as follows:

$$u(t)=\int_{0}^{\infty}\eta^{\prime}(s)\,\Gamma(t-s)+\kappa(s)\,I(t-s)\,ds$$

(16)

In the case of the leaky integrate-and-fire model, we know that the resetting mechanism, as determined by $\eta^{\prime}$, depends entirely on the spikes, and we can decouple it from the spike function $\Gamma$. If we further assume that the membrane reset follows a linear decay, governed by a time constant $\mu_{r}$, of the form described in equation (29) in the Methods section, we arrive at the following formula for the leaky integrate-and-fire model:

$$z(t)=-\theta_{thr}\ e^{-(t-t_{f})/\mu_{r}}+\int_{0}^{\infty}\kappa(s)\,I(t-s)\,ds.$$

(17)

Considering a temporal scaling transformation $t^{\prime}=S_{t}\,t$ and $t^{\prime}_{f}=S_{t}\,t^{\prime}_{f}$ for the temporal signals $f^{\prime}(t^{\prime})=f(t)$ in the concrete case of $\kappa=h_{exp}$, we follow the steps in equation (15):

	$\displaystyle L(t^{\prime};\;\mu^{\prime},\mu^{\prime}_{r})$	$\displaystyle=-\theta_{thr}\ e^{-(t^{\prime}-t_{f}^{\prime})/\mu_{r}^{\prime}}+\int_{z^{\prime}=0}^{\infty}f^{\prime}(t^{\prime}-z^{\prime})\,\frac{1}{\mu^{\prime}}\,e^{-t^{\prime}/\mu^{\prime}}\,dz^{\prime}$		(18)
		$\displaystyle\stackrel{{\scriptstyle 1}}{{=}}-\theta_{thr}\,e^{-S_{t}\,(t-t_{f})/S_{t}\,\mu_{r}^{\prime}}+\int_{z=0}^{\infty}f^{\prime}\left(S_{t}\,(t-z)\right)\frac{1}{S_{t}\,\mu}\,e^{-S_{t}\,t/S_{t}\,\mu}\,S_{t}\,dz$
		$\displaystyle=-\theta_{thr}\,e^{-(t-t_{f})/\mu_{r}}+\int_{z=0}^{\infty}f(t-z)\,\frac{1}{\mu}\,e^{-t/\mu}\,dz$
		$\displaystyle=L(t;\;\mu,\mu_{r}).$

This shows that a single leaky integrate-and-fire filter $z$ is scale-covariant over time, and can be extended to a spectrum of temporal scales as above. Combined with the spatial affine covariance as well as Galilean covariance from equation (7), this finding establishes spatio-temporal covariance to spatial affine transformations, Galilean transformations and temporal scaling transformations in the image plane for leaky integrators and leaky integrator-and-fire neurons.

It is well known that spiking neural networks are comparatively harder to train than non-spiking neural networks for event-based vision, and they are not yet comparable in performance to methods used for training non-spiking deep networks [Neftci et al., 2019, Schuman et al., 2022]. To guide the training process for spiking networks, we propose to initiate the receptive fields with priors according to the idealized model for covariant spatio-temporal receptive fields. Following this idea, we used the purely spatial component of the spatio-temporal receptive fields by scale-normalized affine directional derivative kernels according to [Lindeberg, 2021, Equation (31) for $\gamma=1$]

$$T_{{\varphi}^{m_{1}}{\perp{\varphi}}^{m_{2}},norm}(x_{1},x_{2};\;\Sigma)=\sigma_{\varphi}^{m_{1}}\,\sigma_{\bot{\varphi}}^{m_{2}}\,\partial_{\varphi}^{m_{1}}\,\partial_{\bot\varphi}^{m_{2}}\,\left(g(x_{1},x_{2};\;s\Sigma)\right),$$

(19)

where

• •

  $\partial_{\varphi}=\cos\varphi\,\partial_{x_{1}}+\sin\varphi\,\partial_{x_{2}}$ and $\partial_{\bot\varphi}=-\sin\varphi\,\partial_{x_{1}}+\cos\varphi\,\partial_{x_{2}}$ denote directional derivatives in directions $\varphi$ and $\bot\varphi$ parallel to the two eigendirections of the spatial covariance matrix $\Sigma$,

• •

  $m_{1}$ and $m_{2}$ denote the orders of spatial differentiation and

• •

  $\sigma_{\varphi}=\sqrt{\lambda_{\varphi}}$ and $\sigma_{\bot\varphi}=\sqrt{\lambda}_{\bot\varphi}$ denote spatial scale parameters in these directions, with $\lambda_{\varphi}$ and $\lambda_{\bot\varphi}$ denoting the eigenvalues of $\Sigma$.

This receptive field model directly follows the spatial component in the covariant receptive field model in Section 2.2, while restricted to the special case when the image velocity parameter $v=0$, and complemented with spatial differentiation, to make the receptive fields more selective. Receptive fields from this family have been demonstrated to be very similar to the receptive fields of simple cells in the primary visual cortex (see Figures 16 and 17 in [Lindeberg, 2021]). For the temporal scales, we initialize logarithmically distributed time constants in the interval $[1,4]$ for the leaky integrate-and-fire neurons, as described Section 4.1. Figure 2 (a) visualizes a subset of the spatial receptive fields that we used in the experiments. Panels (b) and (d) visualize the convolving of the spatial receptive fields with conventional image data and event data, respectively, and panels and (c) and (e) show the temporal traces change according to time scale. providing an implicit stabilization mechanism for the event-based data.

We proceed to test the covariance properties of our model for affine transformations. The affine case is more challenging than the translation setting above, but also the most realistic due to its close relation to image transformations. Figure 4 (a) shows the prediction accuracies of the models subject when trained on a dataset where shapes are subject to fully affine transformations at different velocities. Firstly, we observe higher errors than the translation case which is expected since we are applying the same model to a significantly more complex task. Secondly, we observe the same effect of the initialization scheme as in the translation case: the leaky integrator initialized according to theory outperforms the uniform initialization. The effect for the leaky integrate-and-fire model is less pronounced than in the translation case, particularly for sparser signals. Thirdly, the leaky integrator outperforms the naive ANN baseline. We ascribe this to the logarithmically distributed memory in the neuromorphic model, naïvely because it is the only difference between the models, but also because it aligns with our theoretical understanding of joint spatio-temporal covariance. Figure 2 (c) and (e) shows the importance of the temporal component for event-based vision: temporal covariance is critical to create stable representations, in particular when the input is sparse and noisy.

Lindeberg and Gårding [Lindeberg and Gårding, 1997] established a relationship between two scale-space representations under linear affine transformations [Lindeberg, 2013]. Concretely, if two spatial images $f(x)$ and $f^{\prime}(x^{\prime})$ are related according to $x^{\prime}=A\,x$, and provided that the two associated covariance matrices $\Sigma$ and $\Sigma^{\prime}$ are coupled according to [Lindeberg and Gårding, 1997, Eq. (16)],

$$\Sigma^{\prime}=A\,\Sigma\,A^{T}$$

(20)

then two spatially smoothed scale-space representations, $L$ and $L^{\prime}$, are related according to,

$$L(x;s,\Sigma)=L^{\prime}(x;s,\Sigma^{\prime})$$

(21)

where $s$ parameterizes scale. When scaling from $s$ to $s^{\prime}$ by a spatial scaling factor of $S_{x}$, we can relate the two scales by $s^{\prime}=S_{x}^{2}\>s$ in the following scale-space representations [Lindeberg, 2023a, Eqs. (35) & (36)]

$$L(x;s,\Sigma)=L^{\prime}(x;s^{\prime},\Sigma)$$

(22)

We exploit these relationships to initialize our spatial and temporal receptive fields by sampling across the parameter space for the spatial affine and temporal scaling operations.

For the spatial transformations, we sample the space of covariance matrices for Gaussian kernels parameterized by their orientation, scale, and skew up to their second derivative. For computing directional derivatives from the images smoothed by affine Gaussian kernels, we apply compact directional derivative masks to the spatially smoothed image data, according to Section 5 in [Lindeberg, 2024], which constitutes a computationally efficient way of computing multiple affine-Gaussian-smoothed directional derivatives for the same input data. Examples of spatial Gaussian derivative kernels and their directional derivatives of different orders are shown in Figure 2 (a). Combined with the convolutional operator, this guarantees covariance under natural image transformations according to Equation (21) in [Lindeberg, 2023a].

To handle temporal scaling transformations, we choose time constants according to a geometric series, which effectively models logarithmically distributed memories of the past [Lindeberg, 2023b, Eqs. (18)-(20)]

$$\tau_{k}=c^{2(k-K)}\tau_{\text{max}}$$

(23)

where $c$ is a distribution parameter that we set to $\sqrt{2}$, $K$ is the number of scales, and $\tau_{\text{max}}$ is the maximum temporal scale. Further details are available in Supplementary Material Section B.1.

This logarithmic space of scales which guarantees time-causal covariance for temporal image scaling operations [Lindeberg and Fagerström, 1996, Lindeberg, 2023a].

We use the leaky integrate-and-fire formalization from Lapicque [Blair, 1932, Abbott, 1999], stating that a neuron voltage potential $u$ evolves over time, according to its time constant, $\mu$, and some input current $I$

$$\mu\,\dot{u}=-u+I$$

(24)

with the Heaviside threshold function parameterized by $\theta_{threshold}$

$$z(t)=\begin{cases}1&u>=\theta_{threshold}\\ 0&u<\theta_{threshold}\end{cases}$$

(25)

and a jump condition for the membrane potential, which resets to $\theta_{reset}$ when the threshold is crossed

$$u(t)=\begin{cases}\theta_{reset}&z(t)=1\\ u(t)&z(t)<1\end{cases}$$

(26)

The firing part of the leaky integrate-and-fire model implies that neurons are coupled solely at discrete events, known as spikes.

We simulate an event-based dataset consisting of sparse shape contours, as shown in Figure 6, based on Pedersen et al. [Pedersen et al., 2024]. We use three simple geometries, a triangle, a square, and a circle, that move around the scene according to four different types of transformations: translation, scaling, rotation, and affine (adding shearing). Events are generated by applying some transformation $A$ to the shapes and subtracting two subsequent frames. We then integrate those differences over time to approximate the dynamics of event cameras, until reaching a certain threshold and emitting events of either positive or negative polarity. To obtain sub-pixel accuracy we perform all transformations in a high-resolution space of $2400\times 2400$ pixels, which is downsampled bilinearly to the dataset resolution of $300\times 300$.

We simulate six logarithmically distributed velocities to model varying temporal scales from 0.08 to 2.56. The velocities are normalized to the subsampled pixel space such that, for translational velocity for example, a velocity of one in the x-axis corresponds to the shape moving one pixel in the x-axis per timestep. The velocity parameterizes the sparsity of the dataset, as seen in Figure 6 panel (a), where the shape contours are barely visible compared to the faster-moving objects in panel (c). We add 5‰  Bernoulli-distributed background noise.

Note that the data is extremely sparse, even with high velocities. A temporal velocity of 0.08 provides around 3‰ activations while a scale of 2.56 provides around 4‰ activations, including noise. Furthermore, random guessing yields an average $\ell^{2}$ error of 152 pixels when both points are drawn uniformly from a $300\times 300$ cube [OEIS Foundation Inc., 2024, A091505]. If one of the points is fixed to the center of the cube, the error averages to 76 pixels.

We model our network around four convolutional blocks, shown in Figure 7, with interchangeable activation functions. Each block in the model consists of a spatial convolution, an activation function, a batch normalization layer, and a dropout layer with a $5\%$ probability. Four different models are created. The first two such functions are the leaky integrator, coupled with a rectified linear unit (ReLU) to induce a (spatial) nonlinearity, and the leaky integrate-and-fire models. To provide baseline comparisons to stateless models, we additionally construct a stateless Single Frame (SF) model that uses ReLU activations instead of the temporal kernels and a Multiple Frame (MF) model that operates on six consecutive frames instead of one. We chose six frames in favour of the frame-based model, since an exponentially decaying leaky integrator with $\mu=2$ only retains $1/e^{2}=0.135$ of the signal. To convert the latent activation space from the fourth block into 2-dimensional coordinates on which we can regress, we apply a differentiable coordinate transformation layer introduced in [Pedersen et al., 2023b], that finds the spatial average in two dimensions for each of the three shapes, as shown in Figure 7. Note that the coordinate transform uses leaky integrator neurons for both the spiking and non-spiking model architectures.

The convolutions in the first two blocks are initialized according to theory as described in Section 4.1. All other parameters, both spatial and temporal, are initialized according to the uniform distribution $\mathcal{U}(-\sqrt{k},\sqrt{k})$ where $k=1/\sqrt{c}$ and $c$ is the number of input channels.

We train the models using backpropagation through time on datasets consisting of 4000 movies with 50 frames of 1ms duration, with 20% of the dataset held for model validation. The full sets of parameters, along with steps to reproduce the experiments, are available in the Supplementary Material and online at https://github.com/jegp/nrf.