Testing Positive Semidefiniteness Using Linear Measurements ^††thanks: Authors DN and WS were partially supported by NSF DMS $\#$2011140 and NSF DMS $\#$2108479. DW was supported by NSF CCF $\#$1815840 and Office of Naval Research Grant N00014-18-1-2562.

We make particular note of the distinction between one-sided and two-sided testers. In some settings one is interested in a tester that produces one-sided error. When such a tester outputs False, it must be able to produce a proof that $A$ is not PSD. The simplest such proof is a witness vector $v$ such that $v^{T}Av<0$, and indeed we observe that in the matrix-vector model, any one-sided tester can produce such a $v$ when it outputs $\text{False}.$ This may be a desirable feature if one wishes to apply these techniques to saddle point detection for example: given a point that is not a local minimum, it would be useful to produce a descent direction so that optimization may continue. In the vector-matrix-vector model the situation is somewhat more complicated in general, but all of our one-sided testers produce a witness vector whenever they output False.

We provide optimal bounds for one-sided testers for both matrix-vector and vector-matrix-vector models. The bounds below are stated for constant probability algorithms. Here $\widetilde{O}(f)=f\cdot\text{poly}(\log f)$.

1. 1.

   In the matrix-vector query model, we show that up to a factor of $\log d$, $\widetilde{\Theta}(1/\epsilon^{p/(2p+1)})$ queries are necessary and sufficient for an $\ell_{p}$-tester for any $p\geq 1$. In the $p=1$ case, we note that the $\log d$ factor may be removed.

2. 2.

   In the vector-matrix-vector query model, we show that $\widetilde{\Theta}(d^{1-1/p}/\epsilon)$ queries are necessary and sufficient for an $\ell_{p}$-tester for any $p\geq 1$. Note that when $p=1$ we obtain a very efficient $\widetilde{O}(1/\epsilon)$-query algorithm. In particular, our tester for $p=1$ has query complexity independent of the matrix dimensions, and we show a sharp phase transition for $p>1$, showing in some sense that $p=1$ is the largest value of $p$ possible for one-sided queries.

The matrix-vector query complexity is very different than the vector-matrix-vector query complexity, as the query complexity is $\text{poly}(1/\epsilon)$ for any $p\geq 1$, which captures the fact that each matrix-vector query response reveals more information than that of a vector-matrix-vector query, though a priori it was not clear that such responses in the matrix-vector model could not be compressed using vector-matrix-vector queries.

Our main technical contribution for two-sided testers is a bilinear sketch for PSD-testing with respect to the Frobenius norm, i.e. $p=2$. We consider a Gaussian sketch $G^{T}AG$, where $G$ has small dimension $\widetilde{O}(\frac{1}{\epsilon^{2}}).$ By looking at the smallest eigenvalue of the sketch, we are able to distinguish between $A$ being PSD and being $\epsilon$-far from PSD. Notably this tester may reject even when $\lambda_{\min}(G^{T}AG)>0$, which results in a two-sided error guarantee. This sketch allows us to obtain tight two-sided bounds in the vector-matrix-vector model for $p\geq 2$, both for adaptive and non-adaptive queries.

Surprisingly, we show a separation between one-sided and two-sided testers in the vector-matrix-vector model. For the important case of the Frobenius norm, i.e., $p=2$, we utilize our bilinear sketch to construct a $\widetilde{O}(1/\epsilon^{2})$ query two-sided tester, whereas by our results above, any adaptive one-sided tester requires at least $\Omega(\sqrt{d}/\epsilon)$ queries.

We also show that for any $p>2$, any possibly adaptive two-sided tester requires $d^{\Omega(1)}$ queries for constant $\epsilon$, and thus in some sense, $p=2$ is the largest value of $p$ possible for two-sided queries.

We also study the role of adaptivity in both matrix-vector and vector-matrix-vector models. In both the one-sided and two-sided vector-matrix-vector models we show a quadratic separation between adaptive and non-adaptive testers, which is the largest gap possible for any vector-matrix-vector problem [Sun+19].

In the matrix-vector model, each query reveals more information about $A$ than in the vector-matrix-vector model, allowing for even better choices for future queries. Thus we have an even larger gap between adaptive and non-adaptive testers in this setting.

While the two-sided tester discussed above yields optimal bounds for PSD testing, it does not immediately give a way to estimate the negative eigenvalue when it exists. Via a different approach, we show how to give such an approximation with $\epsilon\left\|A\right\|_{F}$ additive error. In fact, we show how to approximate all of the top $k$ eigenvalues of $A$ using $O(k^{2}\text{poly}\frac{1}{\epsilon})$ non-adaptive vector-matrix-vector queries, which may be of independent interest.

We note that this gives an $O(k^{2}\text{poly}\frac{1}{\epsilon})$ space streaming algorithm for estimating the top $k$ eigenvalues of $A$ to within additive Frobenius error. Prior work yields a similar guarantee for the singular values [AN13], but cannot recover the signs of eigenvalues.

For the case of adaptive matrix-vector queries, we show that Krylov iteration starting with a single random vector yields an optimal $\ell_{p}$-tester for all $p$. Interestingly, our analysis is able to beat the usual Krylov matrix-vector query bound for approximating the top eigenvalue, as we modify the usual polynomial analyzed for eigenvalue estimation to implicitly implement a deflation step of all eigenvalues above a certain threshold. We do not need to explicitly know the values of the large eigenvalues in order to deflate them; rather, it suffices that there exists a low degree polynomial in the Krylov space that implements this deflation.

We note that this idea of implicitly deflating the top eigenvalues first appeared in [AL86]. More recently this observation was applied in a setting very similar to ours by [SW09] who deflated eigenvalues that are large relative to the trace.

Further, we show that this technique is tight for all $p\geq 1$ by showing that any smaller number of matrix-vector products would violate a recent lower bound of [Bra+20] for approximating the smallest eigenvalue of a Wishart matrix. This lower bound applies even to two-sided testers.

We start by describing our result for $p=1$. We give one of the first examples of an algorithm in the vector-matrix-vector query model that leverages adaptivity in an interesting way. Most known algorithms in this model work non-adaptively, either by applying a bilinear sketch to the matrix, or by making many independent queries in the case of Hutchinson’s trace estimator [Hut89]. Indeed, the algorithm of [AN13] works by computing $G^{T}AG$ for a Gaussian matrix $G$ with $1/\epsilon$ columns, and arguing that all eigenvalues that are at least $\epsilon\|A\|_{1}$ can be estimated from the sketch. The issue with this approach is that it uses $\Omega(1/\epsilon^{2})$ queries and this bound is tight for non-adaptive testers! One could improve this by running our earlier matrix-vector algorithm on top of this sketch, without ever explicitly forming the $1/\epsilon\times 1/\epsilon$ matrix $G^{T}AG$; however, this would only give an $O(1/\epsilon^{4/3})$ query algorithm.

To achieve our optimal $\widetilde{O}(1/\epsilon)$ complexity, our algorithm instead performs a novel twist to Oja’s algorithm [Oja82], the latter being a stochastic gradient descent (SGD) algorithm applied to optimizing the quadratic form $f(x)=x^{T}Ax$ over the sphere. In typical applications, the randomness of SGD arises via randomly sampling from a set of training data. In our setting, we instead artificially introduce randomness at each step, by computing the projection of the gradient onto a randomly chosen direction. This idea is implemented via the iteration

for a well-chosen step size $\eta.$ If $f$ ever becomes negative before reaching the maximum number of iterations, then the algorithm outputs False, otherwise it outputs True. For $p=1$, we show that this scheme results in an optimal tester (up to logarithmic factors). Our proof uses a second moment analysis to analyze a random walk, that is similar in style to [Jai+16], though our analysis is quite different. Whereas [Jai+16] considers an arbitrary i.i.d. stream of unbiased estimators to $A$ (with bounded variance), our estimators are simply $gg^{T}A,$ which do not seem to have been considered before. We leverage this special structure to obtain a better variance bound on the iterates throughout the first $\widetilde{O}(1/\epsilon)$ iterations, where each iteration can be implemented with a single vector-matrix-vector query. Our algorithm and analysis gives a new method for the fundamental problem of approximating eigenvalues.

Our result for general $p>1$ follows by relating the Schatten-$p$ norm to the Schatten-$1$ norm and invoking the algorithm above with a different setting of $\epsilon$. We show our method is optimal by proving an $\Omega(d^{2-2/p}/\epsilon^{2})$ lower bound for non-adaptive one-sided testers, and then using a theorem in [RWZ20] which shows that adaptive one-sided testers can give at most a quadratic improvement. We note that one could instead use a recent streaming lower bound of [INW22] to prove this lower bound, though such a lower bound would depend on the bit complexity.

The key technical ingredient behind all of our two-sided testers is a bilinear sketch for PSD-testing. Specifically, we show that a sketch of the form $G^{T}AG$ with $G\in R^{d\times k}$ is sufficient for obtaining a two-sided tester for $p=2$. In contrast to the $p=1$ case, we do not simply output False when $\lambda_{\min}:=\lambda_{\min}(G^{T}AG)<0$ as such an algorithm would automatically be one-sided. Instead we require a criterion to detect when $\lambda_{\min}$ is suspiciously small. For this we require two results.

The first is a concentration inequality for $\lambda_{\min}(G^{T}AG)$ when $A$ is PSD. We show that $\lambda_{\min}\geq\operatorname{Tr}(A)-\widetilde{O}(\sqrt{k})\left\|A\right\|_{F}$ with very good probability. This result is equivalent to bounding the smallest singular value of $A^{1/2}G$, which is a Gaussian matrix whose rows have different variances. Although many similar bounds for constant variances exist in the literature [Lit+05, Ver18], we were not able to find a bound for general covariances. In particular, most existing bounds do not seem to give the concentration around $\operatorname{Tr}(A)$ that we require.

When $A$ has a negative eigenvalue of $-\epsilon$, we show that $\lambda_{\min}\leq\operatorname{Tr}(A)-\epsilon O(k)$. By combining these two results, we are able to take $k=\widetilde{O}(1/\epsilon^{2})$, yielding a tight bound for non-adaptive testers in the vector-matrix-vector model. In fact this bound is even tight for general linear sketches, as we show by applying the results in [LW16].

We also utilize this bilinear sketch to give tight bounds for adaptive vector-matrix-vector queries, and indeed for general linear measurements. By first (implicitly) applying the sketch, and then shifting by an appropriate multiple of the identity we are able to reduce to the $(\epsilon^{2},\ell_{1})$-testing problem, which as described above may solved using $\widetilde{O}(1/\epsilon^{2})$ queries.

A natural approach for approximating the eigenvalues of an $n\times n$ matrix $A$ is to first compute a sketch $G^{T}AG$ or a sketch $G^{T}AH$ for Gaussian matrices $G$ and $H$ with a small number of columns. Both of these sketches appear in [AN13]. As noted above, $G^{T}AG$ is a useful non-adaptive sketch for spectrum approximation, but the error in approximating each eigenvalue is proportional to the Schatten-$1$ norm of $A$. One could instead try to make the error depend on the Frobenius norm $\|A\|_{2}$ of $A$ by instead computing $G^{T}AH$ for independent Gaussian matrices $G$ and $H$, but now $G^{T}AH$ is no longer symmetric and it is not clear how to extract the signs of the eigenvalues of $A$ from $G^{T}AH$. Indeed, [AN13] are only able to show that the singular values of $G^{T}AH$ are approximately the same as those of $A$, up to additive $\epsilon\|A\|_{2}$ error. We thus need a new way to preserve sign information of eigenvalues.

To do this, we show how to use results for providing the best PSD low rank approximation to an input matrix $A$, where $A$ need not be PSD and need not even be symmetric. In particular, in [CW17a] it was argued that if $G$ is a Gaussian matrix with $O(k/\epsilon)$ columns, then if one sets up the optimization problem $\min_{\textrm{rank k PSD}\ Y}\|AGYG^{T}A^{T}-A\|_{F}^{2}$, then the cost will be at most $(1+\epsilon)\|A_{k,+}-A\|_{F}^{2}$, where $A_{k,+}$ is the best rank-$k$ PSD approximation to $A$. By further sketching on the left and right with so-called affine embeddings $S$ and $T$, which have $\text{poly}(k/\epsilon)$ rows and columns respectively, one can reduce this problem to $\min_{\textrm{rank k PSD}\ Y}\|SAGYG^{T}A^{T}T-SAT\|_{F}^{2}$, and now $SAG$, $G^{T}A^{T}T$ and $SAT$ are all $\text{poly}(k/\epsilon)\times\text{poly}(k/\epsilon)$ matrices so can be computed with a $\text{poly}(k/\epsilon)$ number of vector-matrix-vector products. At this point the optimal $Y$ can be found with no additional queries and its cost can be evaluated. By subtracting this cost from $\left\|A\right\|_{F}^{2}$, we approximate $\left\|A_{+,i}\right\|_{F}^{2}$, and $\left\|A_{-,i}\right\|_{F}^{2}$ for all $i\in[k]$, which in turn allows us to produce (signed) estimates for the eigenvalues of $A$.

When $A$ is PSD, we note that Theorem 1.2 in [AN13] is able to reproduce our spectral approximation guarantee using sketching dimension $O(\frac{k^{2}}{\epsilon^{8}})$, compared to our sketch of dimension $O(\frac{k^{2}}{\epsilon^{12}})$. However as mentioned above, our guarantee is stronger in that it allows for the signs of the eigenvalues to be recovered, i.e. our guarantee holds even when $A$ is not PSD. Additionally, we are able to achieve $O(\frac{k^{2}}{\epsilon^{8}})$ using just a single round of adaptivity.

To prove lower bounds for one-sided non-adaptive testers, we first show that a one-sided tester must be able to produce a witness whenever it outputs False. In the matrix-vector model, the witness is a vector $v$ with $v^{T}Av<0$, and in the vector-matrix-vector model, the witness is a PSD matrix $M$ with $\left\langle M,A\right\rangle<0$. In both cases we show that even for simplest non-PSD spectrum $(-\lambda,1,\ldots,1)$, that it takes many queries to produce a witness when $\lambda$ is small. In the matrix-vector model, our approach is simply to show that the $-\lambda$ eigenvector is typically far from span of all queried vectors, when the number of queries is small. This will imply that $A$ is non-negative on the queried subspace, which precludes the tester from producing a witness. In the vector-matrix-vector model our approach is similar, however now the queries take the form of inner products against rank one matrices $x_{i}x_{i}^{T}.$ We therefore need to work within the space of symmetric matrices, and this requires a more delicate argument.

Several works have considered Oja’s algorithm in the context of streaming PCA, [Sha16, Jai+16, AL17]. [Jai+16] gives a tight convergence rate for iteratively approximating the top eigenvector of a PSD matrix, given an eigengap, and [AL17] extends this to a gap free result for $k$-PCA.

As mentioned above, PSD-testing has been investigated in the bounded entry model, where one assumes that the entries of $A$ are bounded by $1$ [BCJ20], and one is allowed to query the entries of $A$. This is a restriction of the vector-matrix-vector model that we consider where only coordinate vectors may be queried. However since we consider a more general query model, we are able to give a better adaptive tester – for us $\widetilde{O}(1/\epsilon)$ vector-matrix-vector queries suffice, beating the $\Omega(1/\epsilon^{2})$ lower bound given in [BCJ20] for entry queries.

Another work on PSD-testing is that of [Han+17], who construct a PSD-tester in the matrix-vector model. They first show how to approximate a general trace function $\sum f(\lambda_{i})$ for sufficiently smooth $f$, by using a Chebyshev polynomial construction to approximate $f$ in the sup-norm over an interval. This allows them to construct an $\ell_{\infty}$-tester by taking $f$ to be a smooth approximation of a shifted Heaviside function. Unfortunately this approach is limited to $\ell_{\infty}$-testers, and does not achieve the optimal bound; they require $\Omega((\log d)/\epsilon)$ matrix-vector queries compared to the $\widetilde{O}((\log d)/\sqrt{\epsilon})$ queries achieved by Krylov iteration.

The closely-related problem of spectrum estimation has been considered several times, in the context of sketching the largest $k$ elements of the spectrum [AN13] discussed above, and approximating the entire spectrum from entry queries in the bounded entry model [BMR21a].