Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Let $H$ be a Hilbert space and $\mathbb{D}\subset H$ a dictionary of basis functions. An important problem in machine learning, statistics, and signal processing is the non-linear approximation of a target function $f$ by a sparse linear combinations of dictionary elements

where $g_{i}\in\mathbb{D}$ depend upon the function $f$. Typical examples include non-linear approximation by redundant wavelet frames [25], shallow neural networks, which correspond to non-linear approximation by dictionaries of ridge functions [20], and gradient boosting [15], which corresponds to non-linear approximation by a dictionary of weak learners. Another important example is compressed sensing [13, 6, 8], where a function $f$ which is a sparse linear combination of dictionary elements is recovered from a small number of linear measurements.

In this work, we study the problem of algorithmically calculating an expansion of the form (1) to approximate a target function $f$. A common class of algorithms for this purpose are greedy algorithms, specifically the pure greedy algorithm [25],

which is also known as matching pursuit, the relaxed greedy algorithm [17, 3, 2],

and the orthogonal greedy algorithm (also known as orthogonal matching pursuit) [28]

where $r_{k}=f-f_{k}$ is the residual and $P_{k}$ is the orthogonal projection onto the span of $g_{1},...,g_{k}$.

When applied to general dictionaries $\mathbb{D}\subset H$, greedy algorithms are often analyzed under the assumption that the target function $f$ lies in the convex hull of the dictionary $\mathbb{D}$. Specifically, following [31, 10, 19] we write the closed convex hull of $\mathbb{D}$ as

and denote the gauge norm [30] of this set, which is often called the variation norm with respect to $\mathbb{D}$, as

A typical assumption then is that the target function $f$ satisfies $\|f\|_{\mathcal{K}_{1}(\mathbb{D})}<\infty$. One notable exception is compressed sensing, where it is typically assumed that $f$ is a linear combination of a small number of elements $g_{i}\in\mathbb{D}$. In this case, however, one needs an additional assumption on the dictionary $\mathbb{D}$, usually incoherence [14] or a restricted isometry property (RIP) [7]. The analysis of greedy algorithms under these assumptions is given in [35, 26, 27, 34] for instance. In this work we are interested in the case of general dictionaries $\mathbb{D}\subset H$ which do not satisfy any incoherence or RIP conditions, however, so we will consider the convex hull condition on $f$ given above.

Given a function $f\in B_{1}(\mathbb{D})$, a sampling argument due to Maurey [29] implies that there exists an $n$-term expansion which satisfies

Here $|\mathbb{D}|:=\sup_{g\in\mathbb{D}}\|g\|_{H}$ is the maximum norm of the dictionary and

is the set of $n$-term non-linear dictionary expansions. One can even take the expansion $f_{n}$ in (7) to satisfy $\sum_{i=1}^{n}|a_{i}|\leq 1$. It is also known that for general dictionaries $\mathbb{D}$, the approximation rate (7) is the best possible [19] up to a constant factor. The key problem addressed by greedy algorithms is whether the rate in (7) can be achieved algorithmically.

For simplicity of notation, we assume in the following that $|\mathbb{D}|\leq 1$, i.e. that $\|g\|_{H}\leq 1$ for all $g\in\mathbb{D}$, which can always be achieved by scaling the dictionary appropriately. Relaxing this assumption changes the results in a straightforward manner. It has been shown that under these assumptions the orthogonal greedy algorithm [12] and a suitable version of the relaxed greedy algorithm [17, 3] satisfy

for a suitable constant $K$ (here and in the following $K$, $M$, and $C$ represent unspecified constants). Thus the orthogonal and relaxed greedy algorithms are able to algorithmically attain the approximation rate in (7) up to a constant. The behavior of the pure greedy algorithm is much more subtle. A sequence of improved upper bounds on the convergence rate of the pure greedy algorithm have been obtained in [12, 21, 33], which culminate in the bound

where $\gamma\approx 0.182$ satisfies a particular non-linear equation. Conversely, in [22] a dictionary is constructed for which the pure greedy algorithm satisfies

Thus the precise convergence rate of the pure greedy algorithm is still open, but it is known that it fails to achieve the rate (7) by a significant margin. A further interesting open problem concerns the pure greedy algorithm with shrinkage $s\in(0,1]$, which is given by

where again $r_{k}=f-f_{k}$ is the residual. This algorithm is important for understanding gradient boosting [15]. It is shown in [33] that the convergence order improves as $s$ decreases to a maximum of about $0.305$ as $s\rightarrow 0$, but it is an open problem whether the optimal rate (7) can be achieved in the limit as $s\rightarrow 0$.

The preceding results hold under the assumption that $\|f\|_{\mathcal{K}_{1}(\mathbb{D})}<\infty$, i.e. that the target function is in the scaled convex hull of the dictionary. An important question is how robust the greedy algorithms are to noise. This is captured by the more general assumption that $f$ is not itself in the convex hull, but rather that $f$ is close to an element in the convex hull. Specifically, we assume that $h\in\mathcal{K}_{1}(\mathbb{D})$ and that $\|f-h\|_{H}$, which can be thought of as noise, is small. The convergence rate of the orthogonal and relaxed greedy algorithms are analyzed under these assumptions in [3]. Specifically, it is shown that for any $f\in H$ and any $h\in\mathcal{K}_{1}(\mathbb{D})$, the iterates of the orthogonal greedy algorithm satisfy

This result is important for the statistical analysis of the orthogonal greedy algorithm, for instance in showing the universal consistency of the estimator obtained by applying this algorithm to fit a function $f$ based on samples $f(x_{1}),...,f(x_{n})$ [3]. It also gives a convergence rate for the orthogonal greedy algorithm on the interpolation spaces between $\mathcal{K}_{1}(\mathbb{D})$ and $H$. The $K$-functional of $H$ and $\mathcal{K}_{1}(\mathbb{D})$ is defined by

The interpolation space $X_{\theta}:=[H,\mathcal{K}_{1}(\mathbb{D})]_{\theta,\infty}$ is defined by the norm $\|f\|_{X}=\sup_{t>0}K(f,t)t^{-\theta}$. Intuitively, the interpolation spaces measure how efficiently a function can be approximated by elements with small $\mathcal{K}_{1}(\mathbb{D})$-norm. We refer to [3] and [11], Chapter 6, for more information on interpolation spaces.

Although the approximation rate (7) is sharp for general dictionaries, for compact dictionaries the rate can be improved using a stratified sampling argument [24, 18, 36]. Specifically, the dictionary

where the activation function $\sigma_{0}$ is the Heaviside function and $\Omega\subset\mathbb{R}^{d}$ a bounded domain was analyzed in [24]. There it was shown that for $f\in B_{1}(\mathbb{P}_{0}^{d})$ the approximation rate (7) can be improved to

This corresponds to an improved approximation rate for shallow neural networks with Heaviside (and also sigmoidal) activation function. For shallow neural networks with ReLU^k activation function, the relevant dictionary [32] is

where $S^{d-1}$ is the unit sphere is $\mathbb{R}^{d}$ and $c$ is the diameter of $\Omega$ (assuming without loss of generality that $0\in\Omega$). Using the smoothness of the dictionary $\mathbb{P}_{k}^{d}$, it has been shown that for $f\in B_{1}(\mathbb{P}_{k}^{d})$ an approximation rate of

can be achieved, and that this rate is optimal if the coefficients of $f_{n}$ are bounded [32]. Given these improved theoretical approximation rates for shallow neural networks, it has been an important open problem whether they can be achieved algorithmically by greedy algorithms.

In this work, we show that the improved approximation rates (18) can be achieved using the orthogonal greedy algorithm. More generally, we show that the orthogonal greedy algorithm improves upon the rate (7) whenever the convex hull of the dictionary $\mathbb{D}$ has small metric entropy. Specifically, we recall that the (dyadic) metric entropy of a set $A$ in a Banach space $X$ is defined by

The metric entropy gives a measure of compactness of the set $A$ and a detailed theory can be found in [23], Chapter 15. We show that if the dictionary $\mathbb{D}$ satisfies

then the orthogonal greedy algorithm (4) satisfies

where $K^{\prime}$ only depends upon $K$ and $\alpha$. More generally, this analysis is also robust to noise in the sense considered in [3], i.e. for any $f\in H$ and $g\in\mathcal{K}_{1}(\mathbb{D})$ we have

Utilizing the metric entropy bounds proven in [32], this implies that the orthogonal greedy algorithm achieves the rate (18) for shallow neural networks with ReLU^k activation function. We provide numerical experiments using shallow neural networks with the Heaviside activation function which confirm that these optimal rates are indeed achieved. Additional numerical experiments which confirm the theoretical approximation rates for shallow networks with ReLU^k activation function can be found in [16], where the orthogonal greedy algorithm is used to solve elliptic PDEs.

We conclude the manuscript with a lower bound and negative result concerning the orthogonal greedy algorithm. Consider approximating $f\in B_{1}(\mathbb{D})$ by dictionary expansions with $\ell^{1}$-bounded coefficients, i.e. from the set

It is known that the corresponding approximation rates are lower bounded by the metric entropy up to logarithmic factors. In particular, if for some $M>0$ and any $f\in B_{1}(\mathbb{D})$, we have

then we must have $\epsilon_{n\log{n}}(B_{1}(\mathbb{D}))\leq Kn^{-\frac{1}{2}-\alpha}$. The result (21) shows that the orthogonal greedy algorithm achieves this rate. However, we note that these lower bounds do not a priori apply to the orthogonal greedy algorithm since the expansions generated will in general not have coefficients uniformly bounded in $\ell^{1}$. In fact, we give an example demonstrating that the iterates generated by the orthogonal greedy algorithm may have arbitrarily large $\mathcal{K}_{1}(\mathbb{D})$-norm. Nonetheless, we will show that the rate (21) is sharp and cannot be further improved for general dictionaries satisfying 20. We do not know whether there exists a greedy algorithm which can attain the improved approximation rate (21) while also guaranteeing a uniform $\ell^{1}$-bound on the coefficients.

The paper is organized as follows. In the Section 2, we derive the improved convergence rate (21). Next, in Section 3 we provide numerical experiments which demonstrate the improved rates. Then, in Section 4, we give an example showing that the iterates generated by the orthogonal greedy algorithm cannot be bounded in $\mathcal{K}_{1}(\mathbb{D})$ and also show that the improved rates are tight. Finally, we give concluding remarks and further research directions.

The intuituve idea behind this lemma is that if (26) fails, then the convex hull $B_{1}(\mathbb{D})$ must contain a relatively large skewed simplex. A comparison of volumes then gives a lower bound on its entropy.

Finally, we come to the main result of this section, which shows that the orthogonal greedy algorithm (4) achieves a convergence rate which matches the entropy for dictionaries $\mathbb{D}$ whose entropy decays faster than $O(n^{-\frac{1}{2}})$. In fact, we prove a bit more, generalizing the result from [3] to obtain improved approximation rates for the interpolation spaces between $\mathcal{K}_{1}(\mathbb{D})$ and $H$ as well.

We note that although $f_{n}$ is a linear combination of $n$ dictionary elements $g_{1},...,g_{n}$, we do necessarily have that $f_{n}$ can be bounded in $\mathcal{K}_{1}(\mathbb{D})$, as we show in Proposition 1. This is due to the fact that the coefficients in the expansion generated by the orthogonal greedy algorithm (4) cannot be bounded in $\ell^{1}$.

Further, the bound (36) shows that the improved convergence behavior of the orthogonal greedy algorithm is robust to noise. This is critical for the statistical analysis of the algorithm, for instance for proving statistical consistency [3]. Further, it enables one to prove a convergence rate for functions $f$ in the interpolation space $X_{\theta}=[H,\mathcal{K}_{1}(\mathbb{D})]_{\theta,\infty}$, analogous to the results in [3]. We remark that in [3] a Fourier integrability condition is given which guarantees membership in $X_{\theta}$ when $\mathbb{D}=\mathbb{P}_{0}^{d}$ is the dictionary corresponding to shallow neural networks with sigmoidal activation function. This generalizes the Fourier condition introduced by Barron in [2].

In this section, we give numerical experiments which demonstrate the improved convergence rates derived in Section 2. The setting we consider is a special case of the situation considered in [20]. We consider the dictionary

where $\sigma_{0}$ is the Heaviside activation function. Nonlinear approximation from this dictionary corresponds to approximation by shallow neural networks with Heaviside activation function. The results proved in [32] imply that the metric entropy of the convex hull of $\mathbb{P}_{0}^{2}$ satisfies

We use the orthgonal greedy algorithm to approximate the target function

by a non-linear expansion from the dictionary $\mathbb{P}_{0}^{2}$. Note that the target function $f$ is smooth so that $f\in\mathcal{K}_{1}(\mathbb{P}_{0}^{2})$ [2]. We approximate the $L^{2}$ norm on the domain $[0,1]^{2}$ by the empirical $L^{2}$-norm on a set of $N$ sample points $X_{N}=\{x_{1},...,x_{N}\}$ drawn uniformly at random from the square $[0,1]^{2}$. The subproblem

becomes the combinatorial problem of determining the optimal splitting of the sample points $x_{1},...,x_{N}$ by a hyperplane $\omega\cdot x+b$ such that

is maximized. There are $O(N^{2})$ such hyperplane splittings and the optimal splitting can be determined using $O(N^{2}\log(N))$ operations using a simple modification of the algorithm in [20] which is specific to the case of $\mathbb{R}^{2}$.

In this section, we show that the the iterates generated by the orthogonal greedy algorithm cannot be bounded in $\mathcal{K}_{1}(\mathbb{D})$ and that the rate derived in Theorem 1 cannot be further improved.