Sparse recovery by non-convex optimization – instance optimality

As mentioned above, $\Delta_{0}(Ax)=x$ exactly if $x$ is sufficiently sparse depending on the matrix $A$. However, the associated optimization problem is combinatorial in nature, thus its complexity grows quickly as $N$ becomes much larger than $M$. Naturally, one then seeks to modify the optimization problem so that it lends itself to solution methods that are more tractable than combinatorial search. In fact, in the noise-free setting, the decoder defined by $\ell^{1}$ minimization, given by

$$\Delta_{1}(b):=\arg\min_{y}\|y\|_{1}\text{ subject to }Ay=b,$$

(3)

recovers $x$ exactly if $x$ is sufficiently sparse and the matrix $A$ has certain properties (e.g., [6, 4, 14, 15, 9, 26]). In particular, it has been shown in [4] that if $x\in\Sigma_{S}^{N}$ and $A$ satisfies a certain restricted isometry property, e.g., $\delta_{3S}<1/3$ or more generally $\delta_{(k+1)S}<\frac{k-1}{k+1}$ for some $k>1$ such that $k\in\frac{1}{S}\mathbb{N}$, then $\Delta_{1}(Ax)=x$ (in what follows, $\mathbb{N}$ denotes the set of positive integers, i.e., $0\notin\mathbb{N}$). Here $\delta_{S}$ are the $S$-restricted isometry constants of $A$, as introduced by Candès, Romberg and Tao (see, e.g., [4]), defined as the smallest constants satisfying

$$(1-\delta_{S})\|c\|_{2}^{2}\leq\|Ac\|_{2}^{2}\leq(1+\delta_{S})\|c\|_{2}^{2}$$

(4)

for every $c\in\Sigma_{S}^{N}$. Throughout the paper, using the notation of [30], we say that a matrix satisfies $\text{RIP}(S,\delta)$ if $\delta_{S}<\delta$.

Checking whether a given matrix satisfies a certain RIP is computationally intensive, and becomes rapidly intractable as the size of the matrix increases. On the other hand, there are certain classes of random matrices which have favorable RIP. In fact, let $A$ be an $M\times N$ matrix the columns of which are independent, identically distributed (i.i.d.) random vectors with any sub-Gaussian distribution. It has been shown that $A$ satisfies $\text{RIP}\left(S,\delta\right)$ with any $0<\delta<1$ when

$$S\leq c_{1}M/log(N/M),$$

(5)

with probability greater than $1-2e^{-c_{2}M}$ (see, e.g., [1],[5],[6]), where $c_{1}$ and $c_{2}$ are positive constants that only depend on $\delta$ and on the actual distribution from which $A$ is drawn.

In addition to recovering sparse vectors from error-free observations, it is important that the decoder be robust to noise and stable with regards to the “compressibility” of $x$. In other words, we require that the reconstruction error scale well with the measurement error and with the “non-sparsity” of the signal (i.e., (C2) above). For matrices that satisfy $\text{RIP}((k+1)S,\delta)$, with $\delta<\frac{k-1}{k+1}$ for some $k>1$ such that $k\in\frac{1}{S}\mathbb{N}$, it has been shown in [4] that there exists a feasible decoder $\Delta_{1}^{\epsilon}$ for which the approximation error $\|\Delta_{1}^{\epsilon}(b)-x\|_{2}$ scales linearly with the measurement error $\|e\|_{2}\leq\epsilon$ and with $\sigma_{S}(x)_{\ell^{1}}$. More specifically, define the decoder

$$\Delta_{1}^{\epsilon}(b)=\arg\min_{y}\|y\|_{1}\text{ subject to }\|Ay-b\|_{2}\leq\epsilon.$$

(6)

The following theorem of Candès et al. in [4] provides error guarantees when $x$ is not sparse and when the observation is noisy.

In the noise free case, i.e., when $\epsilon=0$, the reconstruction error in Theorem 1 is bounded above by $\sigma_{S}(x)_{\ell^{1}}/\sqrt{S}$, see (7). This upper bound would sharpen if one could replace $\sigma_{S}(x)_{\ell^{1}}/\sqrt{S}$ with $\sigma_{S}(x)_{\ell^{2}}$ on the right hand side of (7) (note that $\sigma_{S}(x)_{\ell^{1}}$ can be large even if all the entries of the reconstruction error are small but nonzero; this follows from the fact that for any vector $y\in\mathbb{R}^{N}$, $\|y\|_{2}\leq\|y\|_{1}\leq\sqrt{N}\|y\|_{2}$, and consequently there are vectors $x\in\mathbb{R}^{N}$ for which $\sigma_{S}(x)_{\ell^{1}}/\sqrt{S}\gg\sigma_{S}(x)_{\ell^{2}}$, especially when $N$ is large). In [10] it was shown that the term $C_{2,S}\sigma_{S}(x)_{\ell^{1}}/\sqrt{S}$ on the right hand side of (7) cannot be replaced with $C\sigma_{S}(x)_{\ell^{2}}$ if one seeks the inequality to hold for all $x\in\mathbb{R}^{N}$ with a fixed matrix $A$, unless $M>cN$ for some constant $c$. This is unsatisfactory since the paradigm of compressed sensing relies on the ability of recovering sparse or compressible vectors $x$ from significantly fewer measurements than the ambient dimension $N$.

Even though one cannot obtain bounds on the approximation error in terms of $\sigma_{S}(x)_{\ell^{2}}$ with constants that are uniform on $x$ (with a fixed matrix $A$), the situation is significantly better if we relax the uniformity requirement and seek for a version of (7) that holds “with high probability”. Indeed, it has been recently shown by Wojtaszczyk that for any specific $x$, $\sigma_{S}(x)_{\ell^{2}}$ can be placed in (7) in lieu of $\sigma_{S}(x)_{\ell^{1}}/\sqrt{S}$ (with different constants that are still independent of $x$) with high probability on the draw of $A$ if (i) $M>cS\log{N}$ and (ii) the entries ${A}$ is drawn i.i.d. from a Gaussian distribution or the columns of $A$ are drawn i.i.d. from the uniform distribution on the unit sphere in $\mathbb{R}^{M}$ [30]. In other words, the encoder $\Delta_{1}=\Delta_{1}^{0}$ is “(2,2) instance optimal in probability” for encoders associated with such $A$, a property which was discussed in [10].

Following the notation of [30], we say that an encoder-decoder pair $(A,\Delta)$ is $(q,p)$ instance optimal of order $S$ with constant C if

$$\|\Delta({Ax})-x\|_{q}\leq C\frac{\sigma_{S}(x)_{\ell^{p}}}{S^{1/p-1/q}}$$

(9)

holds for all $x\in\mathbb{R}^{N}$. Moreover, for random matrices $A_{\omega}$, $(A_{\omega},\Delta)$ is said to be $(q,p)$ instance optimal in probability if for any $x$ (9) holds with high probability on the draw of $A_{\omega}$. Note that with this notation Theorem 1 implies that $(A,\Delta_{1})$ is (2,1) instance optimal (set $\epsilon=0$), provided $A$ satisfies the conditions of the theorem.

The preceding discussion makes it clear that $\Delta_{1}$ satisfies conditions (C1) and (C2), at least when $A$ is a sub-Gaussian random matrix and $S$ is sufficiently small. It only remains to note that decoding by $\Delta_{1}$ amounts to solving an $\ell^{1}$ minimization problem, and is thus tractable, i.e., we also have (C3). In fact, $\ell^{1}$ minimization problems as described above can be solved efficiently with solvers specifically designed for the sparse recovery scenarios (e.g. [27],[16], [11]).

We have so far seen that with appropriate encoders, the decoders $\Delta_{1}^{\epsilon}$ provide robust and stable recovery for compressible signals even when the measurements are noisy [4], and that $(A_{\omega},\Delta_{1})$ is (2,2) instance optimal in probability [30] when $A_{\omega}$ is an appropriate random matrix. In particular, stability and robustness properties are conditioned on an appropriate RIP while the instance optimality property is dependent on the draw of the encoder matrix (which is typically called the measurement matrix) from an appropriate distribution, in addition to RIP.

Recall that the decoders $\Delta_{1}$ and $\Delta_{1}^{\epsilon}$ were devised because their action can be computed by solving convex approximations to the combinatorial optimization problem (2) that is required to compute $\Delta_{0}$. The decoders defined by

	$\displaystyle\Delta_{p}^{\epsilon}({b})$	$\displaystyle:=\arg\min_{y}\|{y}\|_{p}\text{ s.t. }{\|Ay-b\|_{2}\leq\epsilon},\ \text{and}$		(10)
	$\displaystyle\Delta_{p}(b)$	$\displaystyle:=\arg\min_{y}\|{y}\|_{p}\text{ s.t. }{Ay=b},$		(11)

with $0<p<1$ are also approximations of $\Delta_{0}$, the actions of which are computed via non-convex optimization problems that can be solved, at least locally, still much faster than (2). It is natural to ask whether the decoders $\Delta_{p}$ and $\Delta^{\epsilon}_{p}$ possess robustness, stability, and instance optimality properties similar to those of $\Delta_{1}$ and $\Delta^{\epsilon}_{1}$, and whether these are obtained under weaker conditions on the measurement matrices than the analogous ones with $p=1$.

Early work by Gribonval and co-authors [22, 21, 20, 19] take some initial steps in answering these questions. In particular, they devise metrics that lead to sufficient conditions for uniqueness of $\Delta_{1}({b})$ to imply uniqueness of $\Delta_{p}({b})$ and specifically for having $\Delta_{p}({b})=\Delta_{1}({b})=x$. The authors also present stability conditions in terms of various norms that bound the error, and they conclude that the smaller the value of $p$ is, the more non-zero entries can be recovered by (11). These conditions, however, are hard to check explicitly and no class of deterministic or random matrices was shown to satisfy them at least with high probability. On the other hand, the authors provide lower bounds for their metrics in terms of generalized mutual coherence. Still, these conditions are pessimistic in the sense that they generally guarantee recovery of only very sparse vectors.

Recently, Chartrand showed that in the noise-free setting, a sufficiently sparse signal can be recovered perfectly with $\Delta_{p}$, where $0<p<1$, under less restrictive RIP requirements than those needed to guarantee perfect recovery with $\Delta_{1}$. The following theorem was proved in [7].

Note that, for example, when $p=0.5$ and $k=3$, the above theorem only requires $\delta_{3S}+27\delta_{4S}<26$ to guarantee perfect recovery with $\Delta_{0.5}$, a less restrictive condition than the analogous one needed to guarantee perfect reconstruction with $\Delta_{1}$, i.e., $\delta_{3S}+3\delta_{4S}<2.$ Moreover, in [8], Staneva and Chartrand study a modified RIP that is defined by replacing $\|Ac\|_{2}$ in (4) with $\|Ac\|_{p}$. They show that under this new definition of $\delta_{S}$, the same sufficient condition as in Theorem 5 guarantees perfect recovery. Steneva and Chartrand also show that if $A$ is an $M\times N$ Gaussian matrix, their sufficient condition is satisfied provided $M>C_{1}(p)S+pC_{2}(p)S\log(N/S)$, where $C_{1}(p)$ and $C_{2}(p)$ are given explicitly in [8]. It is important to note is that $pC_{2}(p)$ goes to zero as $p$ goes to zero. In other words, the dependence on $N$ of the required number of measurements $M$ (that guarantees perfect recovery for all $x\in\Sigma^{N}_{S}$) disappears as $p$ approaches 0. This result motivates a more detailed study to understand the properties of the decoders $\Delta_{p}$ in terms of stability and robustness, which is the objective of this paper.

In what follows, we present generalizations of the above results, giving stability and robustness guarantees for $\ell^{p}$ minimization. In Section 2.1 we show that the decoders $\Delta_{p}$ and $\Delta_{p}^{\epsilon}$ are robust to noise and (2,p) instance optimal in the case of appropriate measurement matrices. For this section we rely and expand on our note [25]. In Section 2.3 we extend [30] and show that for the same range of dimensions as for decoding by $\ell^{1}$ minimization, i.e., when $A_{\omega}\in\mathbb{R}^{M\times N}$ with $M>cS\log(N)$, $(A_{\omega},\Delta_{p})$ is also (2,2) instance optimal in probability for $0<p<1$, provided the measurement matrix $A_{\omega}$ is drawn from an appropriate distribution. The generalization follows the proof of Wojtaszczyk in [30]; however it is non-trivial and requires a variant of a result by Gordon and Kalton [18] on the Banach-Mazur distance between a $p$-convex body and its convex hull. In Section 3 we present some numerical results, further illustrating the possible benefits of using $\ell^{p}$ minimization and highlighting the behavior of the $\Delta_{p}$ decoder in terms of stability and robustness. Finally, in Section 4 we present the proofs of the main theorems and corollaries.

While writing this paper, we became aware of the work of Foucart and Lai [17] which also shows similar $(2,p)$ instance optimality results for $0<p<1$ under different sufficient conditions. In essence, one could use the $(2,p)$-results of Foucart and Lai to obtain $(2,2)$ instance optimality in probability results similar to the ones we present in this paper, albeit with different constants. Since neither the sufficient conditions for $(2,p)$ instance optimality presented in [17] nor the ones in this paper are uniformly weaker, and since neither provide uniformly better constants, we simply use our estimates throughout.

We begin with a deterministic stability and robustness theorem for decoders $\Delta_{p}$ and $\Delta_{p}^{\epsilon}$ when $0<p<1$ that generalizes Theorem 1 of Candès et al. Note the associated sufficient conditions on the measurement matrix, given in (12) below, are weaker for smaller values of $p$ than those that correspond to $p=1$. The results in this subsection were initially reported, in part, in [25].

In what follows, we say that a matrix $A$ satisfies the property $P(k,S,p)$ if it satisfies

$$\delta_{kS}+k^{\frac{2}{p}-1}\delta_{(k+1)S}<k^{\frac{2}{p}-1}-1,$$

(12)

for $S\in\mathbb{N}$ and $k>1$ such that $k\in\frac{1}{S}\mathbb{N}$.

Let $A$ be an $M\times N$ matrix and suppose $\delta_{m}$, $m\in\{1,\dots,\lfloor M/2\rfloor\}$ are its $m$-restricted isometry constants. Define $S_{p}$ for $A$ with $0<p\leq 1$ as the largest value of $S\in\mathbb{N}$ for which the slightly stronger version of (12) given by

$$\delta_{(k+1)S}<\frac{k^{\frac{2}{p}-1}-1}{k^{\frac{2}{p}-1}+1}$$

(18)

holds for some $k>1$, $k\in\frac{1}{S}\mathbb{N}$. Consequently, by Theorem 6, $\Delta_{p}(Ax)=x$ for all $x\in\Sigma^{N}_{S_{p}}$. We now establish a relationship between $S_{1}$ and $S_{p}$.

In this section, we show that $(A_{\omega},\Delta_{p})$ is $(2,2)$ instance optimal in probability when $A_{\omega}$ is an appropriate random matrix. Our approach is based on that of [30], which we summarize now. A matrix $A$ is said to possess the $\text{LQ}_{1}(\alpha)$ property if and only if

$${A}(B_{1}^{N})\supset\alpha B_{2}^{M},$$

where $B_{q}^{n}$ denotes the $\ell^{q}$ unit ball in $\mathbb{R}^{n}$. In [30], Wojtaszczyk shows that random Gaussian matrices of size $M\times N$ as well as matrices whose columns are drawn uniformly from the sphere possess, with high probability, the $\text{LQ}_{1}(\alpha)$ property with $\alpha=\mu\sqrt{\frac{\log{(N/M)}}{M}}$. Noting that such matrices also satisfy $\text{RIP}((k+1)S,\delta)$ with $S<c\frac{M}{log{(N/M)}}$, again with high probability, Wojtaszczyk proves that $\Delta_{1}$, for these matrices, is (2,2) instance optimal in probability of order $S$. Our strategy for generalizing this result to $\Delta_{p}$ with $0<p<1$ relies on a generalization of the $\text{LQ}_{1}$ property to an $\text{LQ}_{p}$ property. Specifically, we say that a matrix $A$ satisfies $\text{LQ}_{p}(\alpha)$ if and only if

$${A}(B_{p}^{N})\supset\alpha B_{2}^{M}.$$

We first show that a random matrix $A_{\omega}$, either Gaussian or uniform as mentioned above, satisfies the $\text{LQ}_{p}(\alpha)$ property with

$$\alpha=\frac{1}{C(p)}\left(\mu^{2}{\frac{\log{(N/M)}}{M}}\right)^{(1/p-1/2)}.$$

Once we establish this property, the proof of instance optimality in probability for $\Delta_{p}$ proceeds largely unchanged from Wojtaszczyk’s proof with modifications to account only for the non-convexity of the $\ell^{p}$-quasinorm with $0<p<1$.

Next, we present our results on instance optimality of the $\Delta_{p}$ decoder, while deferring the proofs to Section 4. Throughout the rest of the paper, we focus on two classes of random matrices: ${A}_{\omega}$ denotes $M\times N$ matrices, the entries of which are drawn from a zero mean, normalized column-variance Gaussian distribution, i.e., ${A}_{\omega}=(a_{i,j})$ where $a_{i,j}\sim\mathcal{N}(0,1/\sqrt{M})$; in this case, we say that $A_{\omega}$ is an $M\times N$ Gaussian random matrix. ${\widetilde{A}}_{\omega}$, on the other hand, denotes $M\times N$ matrices, the columns of which are drawn uniformly from the sphere; in this case we say that $\widetilde{A}_{\omega}$ is an $M\times N$ uniform random matrix. In each case, $(\Omega,P)$ denotes the associated probability space.

We start with a lemma (which generalizes an analogous result of [30]) that shows that the matrices $A_{\omega}$ and $\widetilde{A}_{\omega}$ satisfy the $\text{LQ}_{p}$ property with high probability.

The above lemma for $p=1$ can be found in [30]. As we will see in Section 4, the generalization of this result to $0<p<1$ is non-trivial and requires a result from [18], cf. [23], relating certain “distances” of $p$-convex bodies to their convex hulls. It is important to note that this lemma provides the machinery needed to prove the following theorem, which extends to $\Delta_{p}$, $0<p<1$, the analogous result of Wojtaszczyk [30] for $\Delta_{1}$.

In what follows, for a set $T\subseteq\{1,\dots,N\}$, $T^{c}:=\{1,\dots,N\}\setminus T$; for $y\in\mathbb{R}^{N}$, $y_{T}$ denotes the vector with entries $y_{T}(j)=y(j)$ for all $j\in T$, and $y_{T}(j)=0$ for $j\in T^{c}$.

Finally, our main theorem on the instance optimality in probability of the $\Delta_{p}$ decoder follows.

First, note that for any $A\in\mathbb{R}^{M\times N}$, $\delta_{m}$ is non-decreasing in $m$. Also, the map $k\mapsto\frac{k-1}{k+1}$ is increasing in $k$ for $k\geq 0$.

Set

$$L:=(k+1)S_{1},\quad\widetilde{\ell}=k^{\frac{p}{2-p}},\quad\text{and $\widetilde{S}_{p}=\frac{L}{\widetilde{\ell}+1}$.}$$

Then

$$\delta_{(\widetilde{\ell}+1)\widetilde{S}_{p}}=\delta_{(k+1)S_{1}}<\frac{k-1}{k+1}=\frac{\widetilde{\ell}^{\frac{2-p}{p}}-1}{\widetilde{\ell}^{\frac{2-p}{p}}+1}.$$

We now describe how to choose $\ell$ and $S_{p}$ such that $\ell\geq\widetilde{\ell}$, $S_{p}\in\mathbb{N}$, and $(\ell+1)S_{p}=L$ (this will be sufficient to complete the proof using the monotonicity observations above). First, note that this last equality is satisfied only if $(\ell,S_{p})$ is in the set

$$\{(\frac{n}{L-n},L-n):\ n=1,\dots,L-1\}.$$

Let $n^{*}$ be such that

$$\frac{n^{*}-1}{L-n^{*}+1}<\widetilde{\ell}\leq\frac{n^{*}}{L-n^{*}}.$$

(22)

To see that such an $n^{*}$ exists, recall that $\widetilde{\ell}=k^{\frac{p}{2-p}}$ where $0<p<1$. Also, $(k+1)S_{1}=L$ with $S_{1}\in\mathbb{N}$, and $k>1$. Consequently, $1<\widetilde{\ell}<k\leq L-1$, and $k\in\{\frac{n}{L-n}:\ n=\lceil\frac{L}{2}\rceil,\dots,L-1\}$. Thus, we know that we can find $n^{*}$ as above. Furthermore, $\frac{n^{*}}{L-n^{*}}>1$. It follows from (22) that

$$L-n^{*}\leq\widetilde{S}_{p}<L-n^{*}+1.$$

We now choose

$$\ell=\frac{n^{*}}{L-n^{*}},\quad\text{and}\ S_{p}=\lfloor\widetilde{S}_{p}\rfloor=L-n^{*}.$$

Then $(\ell+1)S_{p}=L$, and $\ell\geq\widetilde{\ell}$. So, we conclude that for $\ell$ as above and

$$S_{p}=\lfloor\widetilde{S}_{p}\rfloor=\left\lfloor\frac{k+1}{k^{\frac{p}{2-p}}+1}S_{1}\right\rfloor,$$

we have

$$\delta_{(\ell+1)S_{p}}<\frac{\ell^{\frac{2-p}{p}}-1}{\ell^{\frac{2-p}{p}}+1}.$$

Consequently, the condition of Corollary 10 is satisfied and we have the desired conclusion. ∎