Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth

Sinusoidal networks are neural networks with sine nonlinearities, instead of the traditional ReLU or hyperbolic tangent. They have been recently popularized, particularly for applications in implicit representation models, in the form of SIRENs (Sitzmann et al., 2020). However, despite their popularity, many aspects of their behavior and comparative advantages are not yet fully understood. Particularly, some initialization and parametrization choices for sinusoidal networks are often defined arbitrarily, without a clear understanding of how to optimize these settings in order to maximize performance.

In this paper, we first propose a simplified version of such sinusoidal networks, that allows for easier implementation and theoretical analysis. We show that these simple sinusoidal networks can match and outperform SIRENs in implicit representation learning tasks, such as fitting videos, images and audio signals. We then analyze sinusoidal networks from a neural tangent kernel (NTK) perspective (Jacot et al., 2018), demonstrating that their NTK approximates a low-pass filter with adjustable bandwidth. We confirm, through an empirical analysis this theoretically predicted behavior also holds approximately in practice. We then use the insights from this analysis to inform the choices of initialization and parameters for sinusoidal networks. We demonstrate we can optimize the performance of a sinusoidal network by tuning the bandwidth of its kernel to the maximum frequency present in the input signal being learned. Finally, we apply these insights in practice, demonstrating that “well tuned” sinusoidal networks outperform other networks in learning implicit representation models with good interpolation outside the training points, and in learning the solution to differential equations.

Sinusoidal networks.   Sinusoidal networks have been recently popularized for implicit modelling tasks by sinusoidal representation networks (SIRENs) (Sitzmann et al., 2020). They have also been evaluated in physics-informed learning settings, demonstrating promising results in a series of domains (Raissi et al., 2019b; Song et al., 2021; Huang et al., 2021b; a; Wong et al., 2022). Among the benefits of such networks is the fact that the mapping of the inputs through an (initially) random linear layer followed by a sine function is mathematically equivalent to a transformation to a random Fourier basis, rendering them close to networks with Fourier feature transformations (Tancik et al., 2020; Rahimi & Recht, 2007), and possibly able to address spectral bias (Basri et al., 2019; Rahaman et al., 2019; Wang et al., 2021). Sinusoidal networks also have the property that the derivative of their outputs is given simply by another sinusoidal network, due to the fact that the derivative of sine function is simply a phase-shifted sine.

Neural tangent kernel.   An important prior result to the neural tangent kernel (NTK) is the neural network Gaussian process (NNGP). At random initialization of its parameters $\theta$, the output function of a neural network of depth $L$ with nonlinearity $\sigma$, converges to a Gaussian process, called the NNGP, as the width of its layers $n_{1},\dots,n_{L}\rightarrow\infty$. (Neal, 1994; Lee et al., 2018). This result, though interesting, does not say much on its own about the behavior of trained neural networks. This role is left to the NTK, which is defined as the kernel given by $\Theta(x,\tilde{x})=\left\langle\nabla_{\theta}f_{\theta}(x),\nabla_{\theta}f_{\theta}(\tilde{x})\right\rangle$. It can be shown that this kernel can be written out as a recursive expression involving the NNGP. Importantly, Jacot et al. (2018) demonstrated that, again as the network layer widths $n_{1},\dots,n_{L}\rightarrow\infty$, the NTK is (1) deterministic at initialization and (2) constant throughout training. Finally, it has also been demonstrated that under some assumptions on its parametrization, the output function of the trained neural network $f_{\theta}$ converges to the kernel regression solution using the NTK (Lee et al., 2020; Arora et al., 2019). In other words, under certain assumptions the behavior of a trained deep neural network can be modeled as kernel regression using the NTK.

Physics-informed neural networks.   Physics-informed neural networks (Raissi et al., 2019a) are a method for approximating the solution to differential equations using neural networks (NNs). In this method, a neural network $\hat{u}(t,x;{\theta})$, with learned parameters ${\theta}$, is trained to approximate the actual solution function $u(t,x)$ to a given partial differential equation (PDE). Importantly, PINNs employ not only a standard “supervised” data loss, but also a physics-informed loss, which consists of the differential equation residual $\mathcal{N}$. Thus, the training loss consists of a linear combination of two loss terms, one directly supervised from data and one informed by the underlying differential equations.

There are many details that complicate the practical implementation of current sinusoidal networks. We aim to propose a simplified version of such networks in order to facilitate theoretical analysis and practical implementation, by removing such complications.

As an example we can look at SIRENs, which have their layer activations defined as $f_{l}(x)=\mathrm{sin}{(\omega(W_{l}x+b_{l}))}$. Then, in order to cancel the $\omega$ factor, layers after the first one have their weight initialization follow a uniform distribution with range $[-\frac{\sqrt{6/n}}{\omega},\frac{\sqrt{6/n}}{\omega}]$, where $n$ is the size of the layer. Unlike the other layers, the first layer is sampled from a uniform distribution with range $[-1/n,1/n]$.

We instead propose a simple sinusoidal network, with the goal of formulating an architecture that mainly amounts to substituting its activation functions by the sine function. We will, however, keep the $\omega$ parameter, since (as we will see in future analyses) it is in fact a useful tool for allowing the network to fit inputs of diverse frequencies. The layer activation equations of our simple sinusoidal network, with parameter $\omega$, are defined as

Finally, instead of utilizing a uniform initialization as in SIRENs (with different bounds for the first and subsequent layers), we propose initializing all parameters in our simple sinusoidal network using a default Kaiming (He) normal initialization scheme. This choice not only greatly simplifies the initialization scheme of the network, but it also facilitates theoretical analysis of the behavior of the network under the NTK framework, as we will see in Section 4.

This simple Lemma and relates to Lemma 1.7 in Sitzmann et al. (2020), showing that the initialization we propose here has the same variance as the one proposed for SIRENs. Using this result we can translate the result from the main Theorem 1.8 from Sitzmann et al. (2020), which claims that the SIREN initialization indeed has the desired properties, to our proposed initialization:¹¹1We note that despite being named Theorem 1.8 in Sitzmann et al. (2020), this result is not fully formal, due to the Gaussian distribution being approximated without a formal analysis of this approximation. Additionally, a CLT result is employed which assumes infinite width, which is not applicable in this context. We thus refrain from calling our equivalent result a theorem. Nevertheless, to the extent that the argument is applicable, it would still hold for our proposed initialization, due to its dependence solely on the variance demonstrated in Lemma 1 above.

For a uniform input in $[-1,1]$, the activations throughout a sinusoidal network are approximately standard normal distributed before each sine non-linearity and arcsine-distributed after each sine non-linearity, irrespective of the depth of the network, if the weights are distributed normally, with mean $0$ and variance $\frac{2}{n}$, where $n$ is a layer’s fan-in.

Empirical evaluation of initialization scheme.   To empirically demonstrate the proposed simple initialization scheme preserves the properties from the SIREN initialization scheme, we perform the same analysis performed by Sitzmann et al. (2020). We observe that the distribution of activations matches the predicted normal (before the non-linearity) and arcsine (after the non-linearity) distributions, and that this behavior is stable across many layers. These results are reported in detail in the Appendix B.

In the following we derive the NTK for sinusoidal networks. This analysis will show us that the sinusoidal networks NTK is approximately a low-pass filter, with its bandwidth directly defined by $\omega$. We support these findings with an empirical analysis as well in the following section. Finally, we demonstrate how the insights from the NTK can be leveraged to properly “tune” sinusoidal networks to the spectrum of the desired signal. Full derivations and extensive, detailed analysis are left to Appendix D.

We can see that for values of $\omega>2$, the second term quickly vanishes due to the $e^{-2\omega^{2}}$ factor. This leaves us with only the first term, which has a Gaussian form. Due to the linear scaling term $x^{T}\tilde{x}$, this is only approximately Gaussian, but the approximation improves as $\omega$ increases. We can thus observe that this kernel approximates a Gaussian kernel, which is a low-pass filter, with its bandwidth defined by $\omega$. Figure 1 presents visualizations for NTKs for the simple sinusoidal network, compared to a (scaled) pure Gaussian with variance $\omega^{-2}$, showing there is a close match between the two.

If we write out the NTK for networks with more than one hidden layer, it quickly becomes un-interpretable due to the recursive nature of the NTK definition (see Appendix D). However, as shown empirically in Figure 1, these kernels are still approximated by Gaussians with variance $\omega^{-2}$.

For deeper SIREN networks, the kernels defined by the later layers are in fact Gaussian too, as discussed in Appendix D. This leads to an NTK that is approximated by a product of a sinc function and a Gaussian. These SIREN kernels are also presented in Figure 1.

As shown above, neural tangent kernel theory suggests that sinusoidal networks work as low-pass filters, with their bandwidth controlled by the parameter $\omega$. In this section, we demonstrate empirically that we can observe this predicted behavior even in real sinusoidal networks. For this experiment, we generate a $512\times 512$ monochromatic image by super-imposing two orthogonal sinusoidal signals, each consisting of a single frequency, $f(x,y)=\mathrm{cos}(128\pi x)+\mathrm{cos}(32\pi y)$. This function is sampled in the domain $[-1,1]^{2}$ to generate the image on the left of Figure 2.

To demonstrate what we can expect from applying low-pass filters of different bandwidths to this signal, we perform a discrete Fourier transform (DFT), cut off frequencies above a certain value, and perform an inverse transform to recover the (filtered) image. The MSE of the reconstruction, as a function of the cutoff frequency, is shown in Figure 3. We can see that due to the simple nature of the signal, containing only two frequencies, there are only three loss levels. If indeed the NTK analysis is correct and sinusoidal networks act as low-pass filters, with bandwidth controlled by $\omega$, we should be able to observe similar behavior with sinusoidal networks with different $\omega$ values. We plot the final training loss and training curves for sinusoidal networks with different $\omega$ in Figure 3. We can observe, again, that there are three consistent loss levels following the magnitude of the $\omega$ parameter, in line with the intuition that the sinusoidal network is working as a low-pass filter. This is also observable in Figure 2, where we see example reconstructions for networks of various $\omega$ values after training.

However, unlike with the DFT low-pass filter (which does not involve any learning), we see in Figure 3 that during training some sinusoidal networks shift from one loss level to a lower one. This demonstrates that sinusoidal networks differ from true low-pass filters in that their weights can change, which implies that the bandwidth defined by $\omega$ also changes with learning. We know the weights $W_{1}$ in the first layer of a sinusoidal network, given by $f_{1}(x)=\mathrm{sin}{\left(\omega\cdot W_{1}^{T}x+b_{1}\right)}$, will change with training. Empirically, we observed that the spectral norm of $W_{1}$ increases throughout training for small $\omega$ values. We can interpret that as the overall magnitude of the term $\omega\cdot W_{1}^{T}x$ increasing, which is functionally equivalent to an increase in $\omega$ itself. In Figure 3, we observe that sinusoidal networks with smaller values of $\omega$ take a longer time to achieve a lower loss (if at all). Intuitively, this happens because, due to the effect described above, lower $\omega$ values require a larger increase in magnitude by the weights $W_{1}$. Given that all networks were trained with the same learning rate, the ones with a smaller $\omega$ require their weights to move a longer distance, and thus take more training steps to achieve a lower loss.

As shown in the previous section, though the bandwidth of a network can change throughout training, the choice of $\omega$ still influences how easily and quickly (if at all) it can learn a given signal. The value of the $\omega$ parameter is thus crucial for the learning of the network. Despite this fact, in SIRENs, for example, this value is not adjusted for each task (except for the audio fitting experiments), and is simply set empirically to an arbitrary value. In this section, we seek to justify a proper initialization for this parameter, such that it can be chosen appropriately for each given task.

Moreover, it is often not the case that we simply want to fit only the exact training samples but instead want to find a good interpolation (i.e., generalize well). Setting $\omega$ too high, and thus allowing the network to model frequencies that are much larger than the ones present in the actual signal is likely to cause overfitting. This is demonstrated empirically in Figure 4.

Consequently, we want instead to tune the network to the highest frequency present in the signal. However, we do not always have the knowledge of what is the value of the highest frequency in the true underlying signal of interest. Moreover, we have also observed that, since the network learns and its weights change in magnitude, that value in fact changes with training. Therefore, the most we can hope for is to have a good heuristic to guide the choice of $\omega$. Nevertheless, having a reasonable guess for $\omega$ is also likely sufficient for good performance, precisely due to the ability of the network to adapt during training and compensate for a possibly slightly suboptimal choice.

Choosing $\omega$ from the Nyquist frequency.    One source of empirical information on the relationship between $\omega$ and the sinusoidal network’s “learnable frequencies” is the previous section’s empirical analysis. Taking into account the scaling, we can see from Fig. 3 that around $\omega=16$ the network starts to be able to learn the full signal (freq. $128$). We can similarly note that at about $\omega=4$ the sinusoidal network starts to be able to efficiently learn a signal with frequency $32$, but not the one with frequency $128$. This scaling suggests a heuristic of setting $\omega$ to about $1/8$ of the signal’s maximum frequency.

For natural signals, such as pictures, it is common for frequencies up to the Nyquist frequency of the discrete sampling to be present. We provide an example for the “camera” image we have utilized so far in Figure 21 in Appendix E, where we can see that the reconstruction loss through a low-pass filter continues to decrease significantly up to the Nyquist frequency for the image resolution.

In light of this information, analyzing the choices of $\omega$ for the experiments in Section 3.1 again suggests that $\omega$ should be set around $1/8$ of the Nyquist frequency of the signal. These values of $\omega$ are summarized in Table 2 in the “Fitting $\omega$” column. For example, the image fitting experiment shows that, for an image of shape $512\times 512$ (and thus Nyquist frequency of $256$ for each dimension), this heuristic suggests an $\omega$ value of $256/8=32$, which is the value found to work best empirically through search. We find similar results for the audio fitting experiments. The audio signals used in the audio fitting experiment contained approximately $300,000$ and $500,000$ points, and thus maximum frequencies of approximately $150,00$ and $250,000$. This suggests reasonable values for $\omega$ of $18,750$ and $31,250$, which are close to the ones found empirically to work well. In examples such as the video fitting experiments, in which each dimension has a different frequency, it is not completely clear how to pick a single $\omega$ to fit all dimensions. This suggests that having independent values of $\omega$ for each dimension might be useful for such cases, as discussed in the next section.

Finally, when performing the generalization experiments in Section 7, we show the best performing $\omega$ ended up being half the value of the best $\omega$ used in the fitting tasks from Section 3.1. This follows intuitively, since for the generalization task we set apart half the points for training and the other half for testing, thus dividing the maximum possible frequency in the training sample in half, providing further evidence of the relationship between $\omega$ and the maximum frequency in the input signal.

In this section, we first perform experiments to demonstrate how the optimal value of $\omega$ influences the generalization error of a sinusoidal network, following the discussion in Section 6. After that, we demonstrate that sinusoidal networks with properly tuned $\omega$ values outperform traditional physics-informed neural networks in classic PDE tasks.

In this work, we have present a simplified formulation for sinusoidal networks. Analysis of this architecture from the neural tangent kernel perspective, combined with empirical results, reveals that the kernel for sinusoidal networks corresponds to a low-pass filter with adjustable bandwidth. We leverage this information in order to initialize these networks appropriately, choosing their bandwidth such that it is tuned to the signal being learned. Employing this strategy, we demonstrated improved results in both implicit modelling and physics-informed learning tasks.