Adaptive Resolution Residual Networks — Generalizing Across Resolutions Easily and Efficiently

In this section, we introduce Laplacian pyramids (Burt & Adelson, 1987), as they have often been used in vision techniques to decompose signals across a range of resolutions, and as they closely relate to Laplacian residuals.

Laplacian pyramids take some signal $s$, and perform a series of $m$ convolutions with smoothing kernels $\phi_{n}$ to generate lower bandwidth signals $p^{\text{low}}_{n}$ that each are incrementally smoother than the previous one. Laplacian pyramids then generate difference signals $p^{\text{diff}}_{n}$ that isolate the part of the signal that was lost at each incremental bandwidth reduction, which intuitively correspond to a certain level of detail of the original signal. The operations that compose a Laplacian pyramid can be captured by a base definition and two simple recursive definitions:

	$\displaystyle p^{\text{low}}_{0}$	$\displaystyle=s\ast\phi_{1}$	$\displaystyle\in S_{1}$		(1)
	$\displaystyle p^{\text{low}}_{n}$	$\displaystyle=p^{\text{low}}_{n-1}\ast\phi_{n+1}$	$\displaystyle\in S_{n+1}$		(2)
	$\displaystyle p^{\text{diff}}_{n}$	$\displaystyle=p^{\text{low}}_{n-1}-p^{\text{low}}_{n}$	$\displaystyle\in S_{n}\setminus S_{n+1}$		(3)

Laplacian pyramids are convenient to implement, since each reduction in bandwidth allows a reduction in resolution, as hinted by the notation above, which explicitly includes the bandwidth $S_{n}$ to highlight the coinciding resolution $|X_{n}|$ that may be used.

Laplacian pyramids also allow reconstructing the original signal up to an arbitrary bandwidth using only the last lower bandwidth signal $p^{\text{low}}_{m}$ and a variable number of difference signals $p^{\text{diff}}_{n}$:

$\displaystyle s\ast\phi_{n}=p^{\text{diff}}_{n}+p^{\text{diff}}_{n+1}+\cdots+p^{\text{diff}}_{m-1}+p^{\text{diff}}_{m}+p^{\text{low}}_{m}\in S_{n}$

(4)

In Figure 1, we summarize the recursive formulation of Laplacian pyramids in a three-block pyramid; this is intended to allow easy comparison with the Laplacian residuals we illustrate in Figure 2.

Laplacian pyramids are typically constructed using Gaussian smoothing kernels, which violates Equation 43, and introduces errors in the sampling process as a consequence. We require perfect Whittaker-Shannon smoothing kernels for the exact computation reduction property we show in Laplacian pyramids just below and in Laplacian residuals later in subsection 3.1, as these smoothing kernels are uniquely able to satisfy Equation 44. However, we show in subsection 4.5 that Laplacian dropout enables ARRNs to learn to correct these approximation errors.

\columnratio

0.5 {paracol}2 {nthcolumn}0 Laplacian pyramids have the desirable ability to adapt to low resolutions by skipping computations, as we may intuit from Equation 4 and Figure 1. We formally show this by considering a signal in discrete form $s[\ \cdot\ ]_{u}$ with a low resolution $|X_{u}|$ and a corresponding low bandwidth constraint $S_{u}$. We are interested in what happens up to the level $n$ of the Laplacian pyramid, where $n$ is the highest level that sits just at or above the resolution $|X_{n}|\geq|X_{u}|$ and the bandwidth constraint $S_{n}\supseteq S_{u}$ of the original signal. We observe that all of the smoothing filters $\phi_{1},\cdots,\phi_{n}$ associated with prior levels of the Laplacian pyramids leave the original signal $s$ unchanged since $s\in S_{u}\subseteq S_{n}$ and since $S_{n}\subset\cdots\subset S_{1}$, which induces a trail of zero terms in the expansion of the recursive terms Equation 2 and Equation 3, as shown in Equation 14 on the right. {nthcolumn}1

	$\displaystyle\implies p^{\text{low}}_{0}$	$\displaystyle=s\ast\phi_{1}$		(5)
		$\displaystyle=s$		(6)
	$\displaystyle\implies p^{\text{low}}_{1}$	$\displaystyle=p^{\text{low}}_{0}\ast\phi_{2}$		(7)
		$\displaystyle=s$		(8)
	$\displaystyle\implies p^{\text{diff}}_{1}$	$\displaystyle=p^{\text{low}}_{0}-p^{\text{low}}_{1}$		(9)
		$\displaystyle=0$		(10)
		$\displaystyle\vdots$
	$\displaystyle\implies p^{\text{low}}_{n-1}$	$\displaystyle=p^{\text{low}}_{n-2}\ast\phi_{n}$		(11)
		$\displaystyle=s$		(12)
	$\displaystyle\implies p^{\text{diff}}_{n-1}$	$\displaystyle=p^{\text{low}}_{n-2}-p^{\text{low}}_{n-1}$		(13)
		$\displaystyle=0$		(14)

This allows directly setting $p^{\text{low}}_{n-1}=s$ and carrying out computation only to recover difference terms starting at $p^{\text{diff}}_{n}$, skipping all difference terms $p^{\text{diff}}_{1},\ \dots\ ,p^{\text{diff}}_{n-1}$. We later design Laplacian residuals to reproduce exactly this desirable behaviour such that we can adapt deep learning architectures to lower resolutions by skipping computations.

Laplacian residuals are alike to Laplacian pyramids in the way they separate a signal into a sum of progressively lower bandwidth signals, and in the way they are able to operate at lower resolution by simply skipping computations. However, Laplacian residuals crucially differ in their ability to incorporate neural architectural blocks that enable deep learning.

Laplacian residuals are formulated as adaptive-resolution layers (operators on continuous signals) $r_{n}:(\mathbf{X}\to\mathbb{R}^{f_{n}})\in S_{n}\to(\mathbf{X}\to\mathbb{R}^{f_{n+1}})\in S_{n+1}$ that incorporate neural architectural blocks $b_{n}:(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})\to(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})$ that are fixed-resolution layers (operators on discrete signals). This formulation enables wide compatibility with mainstream fixed-resolution layers that is unseen in prior adaptive-resolution methods. This compatibility is only conditional on the neural architectural block $b_{n}$ producing a constant signal when its input is zero Equation 15, which is trivially guaranteed by linear layers, activation layers, convolutional layers, batch normalization layers, some transformer layers, and any composition of layers that individually meet this condition:

$\displaystyle b_{n}\{0\}=a\ \text{where}\ a\in\mathbb{R}^{f_{n}}$

(15)

The base case and recursive cases seen in Laplacian residuals are nearly identical to those of Laplacian pyramids, aside from including a linear projection $\mathbf{A}_{0}:\mathbb{R}^{f_{1}\times f_{0}}$ to raise the feature dimensionality from $f_{0}\in\mathbb{N}$ to $f_{1}\in\mathbb{N}$ before the first neural architectural block:

	$\displaystyle r_{0}$	$\displaystyle=\mathbf{A}_{0}s\ast\phi_{1}$	$\displaystyle\in S_{1}$		(16)
	$\displaystyle r^{\text{low}}_{n}$	$\displaystyle=r_{n-1}\ast\phi_{n+1}$	$\displaystyle\in S_{n+1}$		(17)
	$\displaystyle r^{\text{diff}}_{n}$	$\displaystyle=r_{n-1}-r^{\text{low}}_{n}$	$\displaystyle\in S_{n}\setminus S_{n+1}$		(18)

The neural architectural block $b_{n}$ receives the difference signal $r^{\text{diff}}_{n}$ as its input, and then sums its output with the lower bandwidth signal $r^{\text{low}}_{n}$ before passing it to the next Laplacian residual. The signals are combined with some additional processing, which we define below and motivate more concretely next:

$\displaystyle r_{n}=\mathbf{A}_{n}(b_{n}\{r^{\text{diff}}_{n}\}\ast\psi\ast\phi_{n+1}+r^{\text{low}}_{n})\in S_{n+1}$

(19)

In Figure 2, we summarize the recursive formulation of Laplacian residuals into a diagram that illustrates a three-block ARRN, which allows easy comparison with the Laplacian pyramid shown in Figure 1.

We include the kernel $\psi$ in Equation 19 to allow Laplacian residuals to replicate the same computation skipping behaviour seen in Laplacian pyramids. We achieve this by setting the kernel $\psi$ to a constant rejection kernel; the convolution against $\psi$ effectively subtracts the mean and ensures the neural architectural block $b_{n}$ contributes zero to the residual signal $r_{n}$ if the difference signal $r^{\text{diff}}_{n}$ is zero, as shown in Equation 20. We obtain this result thanks to the constraint set on the neural architectural block $b_{n}$ in Equation 15:

$\displaystyle b_{n}\{0\}\ast\psi=0$

(20)

We include a smoothing kernel $\phi_{n+1}$ in Equation 19 to ensure the output bandwidth of $r_{n}$ coincides with the input bandwidth of $r_{n+1}$, which is necessary for Laplacian residuals to follow the same general structure as Laplacian pyramids.

We also incorporate a projection matrix $\mathbf{A}_{n}:\mathbb{R}^{f_{n+1}\times f_{n}}$ in Equation 19 to allow raising the feature dimensionality from $f_{n}\in\mathbb{N}$ to $f_{n+1}\in\mathbb{N}$ at the end of each Laplacian residual, so that more capacity can be allocated to later Laplacian residuals.

We note that the neural architectural block $b_{n}\{\ \cdot\ \}$ in Equation 19 is written in shorthand, and stands for the more terse expression $\mathfrak{I}_{n}\{b_{n}\{\mathfrak{S}_{n}\{\ \cdot\ \}\}\}$. This is a formal trick that enables our analysis by casting the neural architectural block from an operator on discrete signals $(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})\to(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})$ to an operator on continuous signals $(\mathbf{X}\to\mathbb{R}^{f_{n}})\in S_{n}\to(\mathbf{X}\to\mathbb{R}^{f_{n}})\in S_{n}$ that is equivalent in the sense of Equation 44. This does not directly reflect the implementation of the method, as all linear operators are analytically composed then cast to their discrete form to maximize computational efficiency.

We add that we can alter the formulation of Equation 19 to let the neural architectural block perform a parameterized downsampling operation $(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})\to(\mathbf{X}_{n+1}\to\mathbb{R}^{f_{n}})$ by changing the interpolation operator discussed above from $\mathfrak{I}_{n}:(\mathbf{X}_{n}\to\mathbb{R}^{f_{n}})\to(\mathbf{X}\to\mathbb{R}^{f_{n}})\in S_{n}$ to $\mathfrak{I}_{n+1}:(\mathbf{X}_{n+1}\to\mathbb{R}^{f_{n}})\to(\mathbf{X}\to\mathbb{R}^{f_{n}})\in S_{n+1}$ and by dropping the smoothing kernel $\phi_{n+1}$ from Equation 19.

In this section, we introduce Laplacian dropout, a training augmentation that is specially tailored to improve the performance of our method by taking advantage of the structure of Laplacian residuals, and that comes at effectively no computational cost.

We formulate Laplacian dropout by following the intuition that Laplacian residuals can be randomly disabled during training to improve generalization. We only allow disabling consecutive Laplacian residuals using the logical or operator to ensure that Laplacian dropout does not cut intermediate information flow:

	$\displaystyle d^{\text{indep}}_{n}$	$\displaystyle\sim B(1-p_{n})$		(38)
	$\displaystyle d^{\text{chain}}_{n}$	$\displaystyle=d^{\text{indep}}_{n}\oplus d^{\text{chain}}_{n-1}$		(39)
	$\displaystyle r^{\text{diff}}_{n}$	$\displaystyle=d^{\text{chain}}_{n}(r_{n-1}-r^{\text{low}}_{n})$		(40)

Next, we provide a theoretical interpretation that identifies two distinct purposes that Laplacian dropout fulfills in our method. We see this dual utility as a highly desirable feature of Laplacian dropout.

We demonstrate that our method allows for greater low-resolution robustness than mainstream methods without compromise in high-resolution performance. Figure 3 shows four ablations of our method corresponding to permutations of two sets: either with Laplacian dropout (red lines) or without Laplacian dropout (black lines); and either with adaptation (solid lines) or without adaptation (dashed lines). We see that with Laplacian dropout and with adaptation (full red lines), our method outperforms every baseline method across every resolution and every dataset. We also find that, in contrast, without Laplacian dropout (black lines), our method shows much weaker generalization across resolutions, clearly demonstrating Laplacian dropout’s effectiveness as a regularizer for robustness to diverse resolutions.

We confirm the computational savings granted by adaptation by performing time measurements on the previous experiment. We use CUDA event timers and CUDA synchronization barriers around the forward pass of the network to eliminate other sources of overhead, such as data loading, and sum these time increments over all batches of the full dataset. We repeat this process $10$ times and pick the median to reduce the effect of outliers. Figure 4 shows the inference time of ARRNs with adaptation (full red lines) and without adaptation (dashed red lines). Our method significantly reduces its computational cost (highlighted by the shaded area) by requiring the evaluation of a lower number of Laplacian residuals at lower resolutions. Our method also has a reasonable inference time relative to well-engineered standard methods overall.

We demonstrate the ease of use of our method and its capability to incorporate various layer types by constructing adaptive-resolution architectures from a range of mainstream fixed-resolution architectures (ResNet18, ResNet50, ResNet101, WideResNet50V2, WideResNet101V2, MobileNetV3Small and MobileNetV3Large). Figure 5 compares the accuracy of architectures in adaptive-resolution form (red box plots) and in fixed-resolution form (green box plots). The distribution of accuracies of the seven underlying architectures in the adaptive-resolution group and fixed-resolution group is visually conveyed in a decluttered format by drawing a small box plot at every resolution. Our method consistently delivers better low-resolution performance, and similar or better high-resolution performance. Our method achieves this while generalizing beyond the abilities of prior adaptive-resolution architectures, as it is able to incorporate the fixed-resolution layers used in this experiment.

We perform an ablation study to verify our theoretical guarantee for numerically identical adaptation. Figure 6 displays a set of experiments that use perfect quality Whittaker-Shannon smoothing kernels (in the upper row of graphs in green) implemented through the Fast Fourier Transform (Cooley & Tukey, 1965). We showcase the usual set of ablations within this group of experiments; with Laplacian dropout (bright green lines) or without Laplacian dropout (dark green lines); with adaptation (full lines) or without adaptation (dashed lines). Our method evaluates practically identically whether unnecessary Laplacian residuals are discarded (with adaptation, full lines, Equation 36), or whether all Laplacian residuals are preserved (without adaptation, dashed lines, Equation 35), with imperceptible discrepancies that are exactly zero, or that are small enough to be attributed to the numerical limitations of floating point computation. Our method is able to skip computations without numerical compromises, as predicted by our theoretical guarantee.

We extend the previous ablation study to verify our theoretical analysis of the dual effect of Laplacian dropout. Figure 6 introduces a set of experiments that relies on fair quality approximate Whittaker-Shanon smoothing kernels (in the middle row of graphs in red), and on poor quality truncated Gaussian smoothing kernels (in the bottom row of graphs in blue). Figure 7 displays these same results in the form of a decision tree to help recognize the trends that are relevant to our discussion. This decision tree factors the impact of choosing a specific filter quality, choosing whether to use Laplacian dropout or not, and choosing whether to use adaptation or not. This analysis considers accuracy averaged over all resolutions and all datasets as the metric of interest. The numerical values displayed on each node correspond to the average multiplicative change in the metric once a decision is made. We claimed in subsection 3.2 that Laplacian dropout has two distinct purposes: it regularizes for robustness across a wider distribution of resolutions; it also mitigates numerical discrepancies caused by approximate smoothing kernels when using adaptation. We have demonstrated the first effect in subsection 4.1 and can also observe this effect clearly in the decision tree. We can only observe the second effect if we consider the choice of smoothing kernel, which motivates this experimental setup. Our method will conform to our theoretical interpretation if absence of error terms at training time yields better performance in the absence of error terms at inference time, and vice versa; that is to say on the last two levels of the decision tree, on the black and white nodes, tracing a path across two nodes of identical colour should yield a multiplier greater than $1$ at the last node, and conversely, tracing a path across two nodes of opposing colour should yield a multiplier smaller than $1$ at the last node. Our method displays exactly this behaviour, with the discrepancy at the last level of the decision tree growing monotonically with decreases in filter quality. Our method therefore leverages Laplacian dropout not just to improve robustness across a wider distribution of resolutions, but to compensate numerical errors induced by imperfect smoothing kernels, which enables the use of computationally greedy implementations.