A General Algorithm for Solving Rank-one Matrix Sensing

Lianke Qin lianke@ucsb.edu. UCSB. Zhao Song zsong@adobe.com. Adobe Research. Ruizhe Zhang ruizhe@utexas.edu. The University of Texas at Austin.

Matrix sensing has many real-world applications in science and engineering, such as system control, distance embedding, and computer vision. The goal of matrix sensing is to recover a matrix $A_{\star}\in\mathbb{R}^{n\times n}$ , based on a sequence of measurements $(u_{i},b_{i})\in\mathbb{R}^{n}\times\mathbb{R}$ such that $u_{i}^{\top}A_{\star}u_{i}=b_{i}$ . Previous work [39] focused on the scenario where matrix $A_{\star}$ has a small rank, e.g. rank- $k$ . Their analysis heavily relies on the RIP assumption, making it unclear how to generalize to high-rank matrices. In this paper, we relax that rank- $k$ assumption and solve a much more general matrix sensing problem. Given an accuracy parameter $\delta\in(0,1)$ , we can compute $A\in\mathbb{R}^{n\times n}$ in $\widetilde{O}(m^{3/2}n^{2}\delta^{-1})$ , such that $|u_{i}^{\top}Au_{i}-b_{i}|\leq\delta$ for all $i\in[m]$ . We design an efficient algorithm with provable convergence guarantees using stochastic gradient descent for this problem.

1 Introduction

Matrix sensing is a generalization of the famous compressed sensing problem. Informally, the goal of matrix sensing is to reconstruct a matrix $A\in\mathbb{R}^{n\times n}$ using a small number of quadratic measurements (i.e., $u^{\top}Au$ ). It has many real-world applications, including image processing [4, 37], quantum computing [1, 9, 20], systems [29] and sensor localization [16] problems. For this problem, there are two important theoretical questions:

•

Q1. Compression: how to design the sensing vectors $u\in\mathbb{R}^{n}$ so that the matrix can be recovered with a small number of measurements?
•

Q2. Reconstruction: how fast can we recover the matrix given the measurements?

[39] initializes the study of the rank-one matrix sensing problem, where the ground-truth matrix $A_{\star}$ has only rank- $k$ , and the measurements are of the form $u_{i}^{\top}A_{\star}u_{i}$ . They want to know the smallest number of measurements $m$ to recover the matrix $A_{\star}$ . In our setting, we assume $m$ is a fixed input parameter and we’re not allowed to choose. We show that for any $m$ and $n$ , how to design a faster algorithm for solving an optimization problem which is finding $A\approx A_{\star}$ . Thus, in some sense, previous work [39, 7] mainly focuses on problem Q1 with a low-rank assumption on $A_{\star}$ . Our work is focusing on Q2 without the low-rank assumption.

We observe that in many applications, the ground-truth matrix $A_{\star}$ does not need to be recovered exactly (i.e., $\|A-A_{\star}\|\leq n^{-c}$ ). For example, for distance embedding, we would like to learn an embedding matrix between all the data points in a high-dimensional space. The embedding matrix is then used for calculating data points’ pairwise distances for a higher-level machine learning algorithm, such as $k$ -nearest neighbor clustering. As long as we can recover a good approximation of the embedding matrix, the clustering algorithm can deliver the desired results. As we relax the accuracy constraints of the matrix sensing, we have the opportunity to speed up the matrix sensing time.

We formulate our problem in the following way:

Problem 1.1 (Approximate matrix sensing).

Given a ground-truth positive definite matrix $A_{\star}\in\mathbb{R}^{n\times n}$ and $m$ samples $(u_{i},b_{i})\in\mathbb{R}^{n}\times\mathbb{R}$ such that $u_{i}^{\top}A_{\star}u_{i}=b_{i}$ . Let $R=\max_{i\in[m]}|b_{i}|$ . For any accuracy parameter $\delta\in(0,1)$ , find a matrix $A\in\mathbb{R}^{n\times n}$ such that

\displaystyle(u_{i}^{\top}Au_{i}-u_{i}A_{\star}u_{i})^{2}\leq\delta,~{}~{}~{}\forall i\in[m]

(1)

\displaystyle(1-\delta)A_{\star}\preceq A\preceq(1+\delta)A_{\star}.

(2)

We make a few remarks about Problem 1.1. First, our formulation doesn’t require the matrix $A_{\star}$ to be low-rank as literature [39]. Second, we need the measurement vectors $u_{i}$ to be “approximately orthogonal” (i.e., $|u_{i}^{\top}u_{j}|$ are small), while [39] make much stronger assumptions for exact reconstruction. Third, the measure approximation guarantee (Eq. (1)) does not imply the spectral approximation guarantee (Eq. (2)). We mainly focus on achieving the first guarantee and discuss the second one in the appendix.

This problem is interesting for two reasons. First, speeding up matrix sensing is salient for a wide range of applications, where exact matrix recovery is not required. Second, we would like to understand the fundamental tradeoff between the accuracy constraint $\epsilon$ and the running time. This tradeoff can give us insights on the fundamental computation complexity for matrix sensing.

This paper makes the following contributions:

•

We design a potential function to measure the distance between the approximate solution and the ground-truth matrix.
•

Based on the potential function, we show that gradient descent can efficiently find an approximate solution of the matrix sensing problem. We also prove the convergence rate of our algorithm.
•

Furthermore, we show that the cost-per-iteration can be improved by using stochastic gradient descent with a provable convergence guarantee, which is proved by generalizing the potential function to a randomized potential function.

Technically, our potential function applies a $\cosh$ function to each “training loss” (i.e., $u_{i}^{\top}Au_{i}-b_{i}$ ), which is inspired by the potential function for linear programming [5]. We prove that the potential is decreasing for each iteration of gradient descent, and a small potential implies a good approximation. In this way, we can upper bound the number of iterations needed for the gradient descent algorithm.

To reduce the cost-per-iteration, we follow the idea of stochastic gradient descent and evaluate the gradient of potential function on a subset of measurements. However, we still need to know the full gradient’s norm for normalization, which is a function of the training losses. It is too slow to naively compute each training loss. Instead, we use the idea of maintenance [5, 28, 3, 2, 19, 11, 34, 35, 13, 31] and show that the training loss at the $(t+1)$ -th iteration (i.e., $u_{i}^{\top}A_{t+1}u_{i}-b_{i}$ ) can be very efficiently obtained from those at the $t$ -th iteration (i.e., $u_{i}^{\top}A_{t}u_{i}-b_{i}$ ). Therefore, we first preprocess the initial full gradient’s norm, and in the following iterations, we can update this quantity based on the previous iteration’s result.

We state our main result as follows:

Theorem 1.2 (Informal of Theorem 6.1).

Given $m$ measurements of matrix sensing problems, there is an algorithm that outputs a $n\times n$ matrix $A$ in $\widetilde{O}(m^{3/2}n^{2}R\delta^{-1})$ time such that $|u_{i}^{\top}Au_{i}-b_{i}|\leq\delta$ , $\forall i\in[m]$ .

2 Related Work

Linear Progamming

Linear programming is one of foundations of the algorithm design and convex optimization. many problems can be modeled as linear programs to take advantage of fast algorithms. There are many works in accelerating linear programming runtime complexity [25, 26, 5, 28, 3, 2, 33, 8, 19, 10].

Semi-definite Programming

Semidefinite programming optimizes a linear objective function over the intersection of the positive semidefinite cone with an affine space. Semidefinite programming is a fundamental class of optimization problems and many problems in machine learning, and theoretical computer science can be modeled or approximated as semidefinite programming problems. There are many studies to speedup the running time of Semidefinite programming [30, 12, 27, 15, 14, 11, 10].

Matrix Sensing

Matrix sensing [22, 32, 17, 39, 7] is a generalization of the popular compressive sensing problem for the sparse vectors and has applications in several domains such as control, vision etc. a set of universal Pauli measurements, used in quantum state tomography, have been shown to satisfy the RIP condition [23]. These measurement operators are Kronecker products of $2\times 2$ matrices, thus, they have appealing computation and memory efficiency. Rank-one measurement using nuclear norm minimization is also used in other work [6, 21]. There is also previous work working on low-rank matrix sensing to reconstruct a matrix exactly using a small number of linear measurements. ProcrustesFlow [36] designs an algorithm to recover a low-rank matrix from linear measurements. There are other low-rank matrix recovering algorithms based on non-convex optimizations [38, 24].

3 Preliminary

Notations.

For a positive integer, we use $[n]$ to denote set $\{1,2,\cdots,n\}$ . We use $\cosh(x)=\frac{1}{2}(e^{x}+e^{-x})$ and $\sinh(x)=\frac{1}{2}(e^{x}-e^{-x})$ . For a square matrix, we use $\operatorname{tr}[A]$ to denote the trace of $A$ . An $n\times n$ symmetric real matrix $A$ is said to be positive-definite if $x^{\top}Ax>0$ for all non-zero $x\in\mathbb{R}^{n}$ . An $n\times n$ symmetric real matrix $A$ is said to be positive-semidefinite if $x^{\top}Ax\geq 0$ for all non-zero $x\in\mathbb{R}^{n}$ . For any function $f$ , we use $\widetilde{O}(f)=f\cdot\operatorname{poly}(\log f)$ .

3.1 Matrix hyperbolic functions

Definition 3.1 (Matrix function).

Let $f:\mathbb{R}\rightarrow\mathbb{R}$ be a real function and $A\in\mathbb{R}^{n\times n}$ be a real symmetric function with eigendecomposition

\displaystyle A=Q\Lambda Q^{-1}

where $\Lambda\in\mathbb{R}^{n\times n}$ is a diagonal matrix. Then, we have

\displaystyle f(A):=Qf(\Lambda)Q^{-1},

where $f(\Lambda)\in\mathbb{R}^{n\times n}$ is the matrix obtained by applying $f$ to each diagonal entry of $\Lambda$ .

We have the following lemma to bound $\cosh(A)$ and delay the proof to Appendix A.3.

Lemma 3.2.

Let $A$ be a real symmetric matrix, then we have

\displaystyle\|\cosh(A)\|=\cosh(\|A\|)\leq\operatorname{tr}[\cosh(A)].

We also have

\displaystyle\|A\|\leq 1+\log(\operatorname{tr}[\cosh(A)]).

3.2 Properties of $\sinh$ and $\cosh$

We have the following lemma for properties of $\sinh$ and $\cosh$ .

Lemma 3.3 (Scalar version).

Given a list of numbers $x_{1},\cdots x_{n}$ , we have

•

$(\sum_{i=1}^{n}\cosh^{2}(x_{i}))^{1/2}\leq\sqrt{n}+(\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}$ ,
•

$(\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}\geq\frac{1}{\sqrt{n}}(\sum_{i=1}^{n}\cosh(x_{i})-n)$ .

Proof.

For the first equation, we can bound $(\sum_{i=1}^{n}\cosh^{2}(x_{i}))^{1/2}$ by:

	$\displaystyle(\sum_{i=1}^{n}\cosh^{2}(x_{i}))^{1/2}=$	$\displaystyle~{}(n+\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\sqrt{n}+(\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}$

where the first step comes from fact A.5, and the second step follows from $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ .

For the second equation, we can bound $(\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}$ by:

	$\displaystyle(\sum_{i=1}^{n}\sinh^{2}(x_{i}))^{1/2}\geq$	$\displaystyle~{}\frac{1}{\sqrt{n}}(\sum_{i=1}^{n}\sinh(x_{i}))$
	$\displaystyle\geq$	$\displaystyle~{}\frac{1}{\sqrt{n}}(\sum_{i=1}^{n}\cosh(x_{i})-n)$

where the first step follows that $\sqrt{\frac{\sum_{i=1}^{n}x_{i}^{2}}{n}}\geq\frac{\sum_{i=1}^{n}x_{i}}{n}$ , and the second step follows from fact A.5 and $\sqrt{x^{2}-1}\geq\sqrt{x}-1$ . ∎

We also have a lemma for the matrix version.

Lemma 3.4 (Matrix version).

For any real symmetric matrix $A$ , we have

•

$(\operatorname{tr}[\cosh^{2}(A)])^{1/2}\leq\sqrt{n}+\operatorname{tr}[\sinh^{2}(A)]^{1/2}$ ,
•

$(\operatorname{tr}[\sinh^{2}(A)])^{1/2}\geq\frac{1}{\sqrt{n}}(\operatorname{tr}[\cosh(A)]-n)$ .

Proof.

Part 1. We have

	$\displaystyle(\operatorname{tr}[\cosh^{2}(A)])^{1/2}=$	$\displaystyle~{}(n+\operatorname{tr}[\sinh^{2}(A)])^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\sqrt{n}+\operatorname{tr}[\sinh^{2}(A)]^{1/2}.$

where the first step follows from $\cosh^{2}(A)-\sinh^{2}(A)=I$ .

Part 2. Let $\sigma_{i}$ denote the singular value of $\cosh(A)$

	$\displaystyle(\operatorname{tr}[\sinh^{2}(A)])^{1/2}=$	$\displaystyle~{}(\operatorname{tr}[\cosh^{2}(A)]-n)^{1/2}$
	$\displaystyle=$	$\displaystyle~{}(\sum_{i=1}^{n}\sigma_{i}^{2}-1)^{1/2}$
	$\displaystyle\geq$	$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sqrt{\sigma_{i}^{2}-1}$
	$\displaystyle\geq$	$\displaystyle~{}\frac{1}{\sqrt{n}}(\sum_{i=1}^{n}\sigma_{i}-1)$
	$\displaystyle=$	$\displaystyle~{}\frac{1}{\sqrt{n}}(\operatorname{tr}[\cosh(A)]-n)$

where the second step follows from $\|\cdot\|_{2}\geq\frac{1}{\sqrt{n}}\|\cdot\|_{1}$ , the third step follows from $\sigma_{i}\geq 1$ .

∎

4 Technique Overview

We first analyze the convergence guarantee of our matrix sensing algorithm based on gradient descent and improve its time complexity with stochastic gradient descent under the assumption where $\{u_{i}\}_{i\in[m]}$ are orthogonal vectors. We then analyze the convergence guarantee of our matrix sensing algorithm under a more general assumption where $\{u_{i}\}_{i\in[m]}$ are non-orthogonal vectors and $|u_{i}^{\top}u_{j}|\leq\rho$ .

Gradient descent.

We begin from the case where $\{u_{i}\}_{i\in[m]}$ are orthogonal vectors in $\mathbb{R}^{n}$ . We can the following entry-wise potential function:

\displaystyle\Phi_{\lambda}(A):=\sum_{i=1}^{m}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i}))

and analyze its progress during the gradient descent according to the update formula defined in Eq. (4) for each iteration. We split the gradient of the potential function into diagonal and off-diagonal terms. We can upper bound the diagonal term and prove that the off-diagonal term is zero. Combining the two terms together, we can upper bound the progress of update per iteration in Lemma 5.3 by:

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}.

By accumulating the progress of update for the entry-wise potential function over $T=\widetilde{\Omega}(\sqrt{m}R\delta^{-1})$ iterations, we have $\Phi(A_{T+1})\leq O(m)$ . This implies that our Algorithm 1 can output a matrix $A_{T}\in\mathbb{R}^{n\times n}$ satisfying guarantee in Eq. (23), and the corresponding time complexity is $O(mn^{2})$ .

We then analyze the gradient descent under the assumption where $\{u_{i}\}_{i\in[m]}$ are non-orthogonal vectors in $\mathbb{R}^{n}$ , $|u_{i}^{\top}u_{j}|\leq\rho$ and $\rho\leq\frac{1}{10m}$ . We can upper bound the diagonal entries and off-diagonal entries respectively and obtain the same progress of update per iteration in Lemma D.1. Accumulating in $T=\widetilde{\Omega}(\sqrt{m}R\delta^{-1})$ iterations, we can prove the approximation guarantee of the output matrix of our matrix sensing algorithm.

Stochastic gradient descent.

To further improve the time cost per iteration of our approximate matrix sensing, by uniformly sampling a subset ${\cal B}\subset[m]$ of size $B$ , we compute the gradient of the stochastic potential function:

\displaystyle\nabla\Phi_{\lambda}(A{,{\cal B}}):={\frac{m}{|{\cal B}|}\sum_{i\in{\cal B}}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda(u_{i}^{\top}Au_{i}-b_{i})),

and update the potential function based on update formula defined in Eq. (8). We upper bound the diagonal and off-diagonal terms respectively and obtain the expected progress on the potential function in Lemma 6.3.

Over $T=\widetilde{\Omega}(m^{3/2}B^{-1}R\delta^{-1})$ iterations, we can upper bound $\Phi(A_{T+1})\leq O(m)$ with high probability. With similar argument to gradient descent section, we can prove that the SGD matrix sensing algorithm can output a solution matrix satisfying the same approximation guarantees with high success probability in Lemma 6.4. The optimized time complexity is $O(Bn^{2})$ where $B$ is the SGD batch size.

For the more general assumption where $\{u_{i}\}_{i\in[m]}$ are non-orthogonal vectors in $\mathbb{R}^{n}$ and $|u_{i}^{\top}u_{j}|$ has an upper bound, We also provide the cost-per-iteration analysis for stochastic gradient descent by bounding the diagonal entries and off-diagonal entries of the gradient matrix respectively. Then we prove that the progress on the expected potential satisfies the same guarantee as the gradient descent in Lemma E.2. Therefore, our SGD matrix sensing algorithm can output an matrix satisfying the approximation guarantee after

\displaystyle T=\widetilde{\Omega}(m^{3/2}B^{-1}R\delta^{-1})

iterations under the general assumption.

5 Gradient descent for entry-wise potential function

In this section, we show how to obtain an approximate solution of matrix sensing via gradient descent. For simplicity, we start from a case that $\{u_{i}\}_{i\in[m]}$ are orthogonal vectors in $\mathbb{R}^{n}$ ¹¹1We note that $A^{\prime}:=\sum_{i=1}^{m}b_{i}u_{i}u_{i}^{\top}$ is a solution satisfying $u_{i}^{\top}A^{\prime}u_{i}=b_{i}$ for all $i\in[m]$ . However, we pretend that we do not know this solution in this section., which already conveys the key idea of our algorithm and analysis and we generalize the solution to the non-orthogonal case (see Appendix D). We show that $\widetilde{\Omega}(\sqrt{m}/\delta)$ iterations of gradient descent can output a $\delta$ -approximate solution, where each iteration takes $O(mn^{2})$ -time. Below is the main theorem of this section:

Theorem 5.1 (Gradient descent for orthogonal measurements).

Suppose $u_{1},\dots,u_{m}\in\mathbb{R}^{n}$ are orthogonal unit vectors, and suppose $|b_{i}|\leq R$ for all $i\in[m]$ . There exists an algorithm such that for any $\delta\in(0,1)$ , performs $\widetilde{\Omega}(\sqrt{m}R\delta^{-1})$ iterations of gradient descent with $O(mn^{2})$ -time per iteration and outputs a matrix $A\in\mathbb{R}^{n\times n}$ satisfies:

\displaystyle|u_{i}^{\top}Au_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

In Section 5.1, we introduce the algorithm and prove the time complexity. In Section 5.2 - 5.4, we analyze the convergence of our algorithm.

5.1 Algorithm

The key idea of the gradient descent matrix sensing algorithm (Algorithm 1) is to follow the gradient of the entry-wise potential function defined as follows:

\displaystyle\Phi_{\lambda}(A):=\sum_{i=1}^{m}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i})).

(3)

Then, we have the following solution update formula:

\displaystyle A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}.

(4)

Lemma 5.2 (Cost-per-iteration of gradient descent).

Each iteration of Algorithm 1 takes $O(mn^{2})$ -time.

Proof.

In each iteration, we first evaluate $u_{i}^{\top}A_{t}u_{i}$ for all $i\in[m]$ , which takes $O(mn^{2})$ -time. Then, $\nabla\Phi_{\lambda}(A_{t})$ can be computed by summing $m$ rank-1 matrices, which takes $O(mn^{2})$ -time. Finally, at Line 6, the solution can be updated in $O(n^{2})$ -time. Thus, the total running time for each iteration is $O(mn^{2})$ . ∎

Algorithm 1 Matrix Sensing by Gradient Descent.

1:procedure GradientDescent(

\{u_{i},b_{i}\}_{i\in[m]}

)

\triangleright

Theorem 5.1

\tau\leftarrow\max_{i\in[m]}b_{i}

A_{1}\leftarrow\tau\cdot I

4: for

t=1\to T

\nabla\Phi_{\lambda}(A_{t})\leftarrow\sum_{i=1}^{m}u_{i}u_{i}^{\top}\lambda\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))

\triangleright

Compute the gradient

A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}

7: end for

8: return

A_{T+1}

9:end procedure

5.2 Analysis of One Iteration

Throughout this section, we suppose $A\in\mathbb{R}^{n\times n}$ is a symmetric matrix.

We can compute the gradient of $\Phi_{\lambda}(A)$ with respect to $A$ as follows:

		$\displaystyle~{}\nabla\Phi_{\lambda}(A)$
	$\displaystyle=$	$\displaystyle~{}\sum_{i=1}^{m}u_{i}u_{i}^{\top}\lambda\sinh\left(\lambda(u_{i}^{\top}Au_{i}-b_{i})\right)\in\mathbb{R}^{n\times n}.$		(5)

We can compute the Hessian of $\Phi_{\lambda}(A)$ with respect to $A$ as follows

		$\displaystyle~{}\nabla^{2}\Phi_{\lambda}(A)$
	$\displaystyle=$	$\displaystyle~{}\sum_{i=1}^{m}(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\lambda^{2}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i})).$

The Hessian $\nabla^{2}\Phi_{\lambda}(A)\in\mathbb{R}^{n^{2}\times n^{2}}$ and $\otimes$ is the Kronecker product.

Lemma 5.3 (Progress on entry-wise potential).

Assume that $u_{i}\perp u_{j}=0$ for any $i,j\in[m]$ and $\|u_{i}\|^{2}=1$ . Let $c\in(0,1)$ denote a sufficiently small positive constant. Then, for any $\epsilon,\lambda>0$ such that $\epsilon\lambda\leq c$ ,

we have for any $t>0$ ,

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}

Proof.

We defer the proof to Appendix B.1 ∎

5.3 Technical Claims

We prove some technical claims in below.

Claim 5.4.

For $Q_{1}$ defined in Eq. (18), we have

\displaystyle Q_{1}\leq\Big{(}\sqrt{m}+\frac{1}{\lambda}\|\nabla\Phi_{\lambda}(A_{t})\|_{F}\Big{)}\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}.

Proof.

For simplicity, we define $z_{t,i}$ to be

\displaystyle z_{t,i}:=\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}).

Recall that

\displaystyle\nabla^{2}\Phi_{\lambda}(A_{t})=\lambda^{2}\cdot\sum_{i=1}^{m}(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\cosh(z_{t,i}).

For $Q_{1}$ , we have

$\displaystyle Q_{1}=$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})\sum_{i=1}^{m}\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\operatorname{tr}[\sum_{i=1}^{m}\cosh(z_{t,i})(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\cdot\sum_{i=1}^{m}\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\operatorname{tr}[\cosh(z_{t,i})\cdot\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top}u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top}u_{i}u_{i}^{\top})]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\cosh(z_{t,i})\sinh^{2}(z_{t,i})$
$\displaystyle\leq$	$\displaystyle~{}\lambda^{2}\cdot(\sum_{i=1}^{m}\cosh^{2}(z_{t,i}))^{1/2}\cdot(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$
$\displaystyle\leq$	$\displaystyle~{}\lambda^{2}\cdot B_{1}\cdot B_{2},$	(6)

where the first step comes from the definition of $Q_{1}$ , the second step comes from the definition of $\nabla^{2}\Phi_{\lambda}(A_{t})$ , the third step follows from $(A\otimes B)\cdot(C\otimes D)=(AC)\otimes(BD)$ and $u_{i}^{\top}u_{j}=0$ , the fourth step comes from $\|u_{i}\|=1$ and $\operatorname{tr}[(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})]=1$ .

For the term $B_{1}$ , we have

	$\displaystyle B_{1}=$	$\displaystyle~{}(\sum_{i=1}^{m}\cosh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F},$

where the second step follows Part 1 of Lemma 3.3.

For the term $B_{2}$ , we have

	$\displaystyle B_{2}=$	$\displaystyle~{}(\sum_{i=1}^{m}\sinh^{4}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2},$

where the second step follows from $\|x\|_{4}^{2}\leq\|x\|_{2}^{2}$ . This implies that

	$\displaystyle Q_{1}\leq$	$\displaystyle~{}\lambda^{2}\cdot B_{1}\cdot B_{2}$
	$\displaystyle\leq$	$\displaystyle~{}\lambda^{2}\cdot(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
	$\displaystyle=$	$\displaystyle~{}(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}.$

∎

Claim 5.5.

For $Q_{2}$ defined in Eq. (19), we have $Q_{2}=0$ .

Proof.

Because in $Q_{2}$ we have :

	$\displaystyle~{}\sum_{\ell=1}^{m}(u_{\ell}u_{\ell}^{\top}\otimes u_{\ell}u_{\ell}^{\top})\sum_{i\neq j}(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top})$
$\displaystyle=$	$\displaystyle~{}\sum_{\ell=1}^{m}\sum_{i\neq j}(u_{\ell}u_{\ell}^{\top}u_{i}u_{i}^{\top})\otimes(u_{\ell}u_{\ell}^{\top}u_{j}u_{j}^{\top})$
$\displaystyle=$	$\displaystyle~{}0,$	(7)

where the first step follows from $(A\otimes B)\cdot(C\otimes D)=(AC)\otimes(BD)$ , the second step follows that $u_{i}^{\top}u_{j}=0$ if $i\neq j$ and $\ell\neq i$ or $\ell\neq j$ always holds in Eq. (5.3).

Therefore, we get that $Q_{2}=0$ . ∎

5.4 Convergence for multiple iterations

The goal of this section is to prove the convergence of Algorithm 1:

Lemma 5.6 (Convergence of gradient descent).

Suppose the measurement vectors $\{u_{i}\}_{i\in[m]}$ are orthogonal unit vectors, and suppose $|b_{i}|$ is bounded by $R$ for $i\in[m]$ . Then, for any $\delta\in(0,1)$ , if we take $\lambda=\Omega(\delta^{-1}\log m)$ and $\epsilon=O(\lambda^{-1})$ in Algorithm 1, then for $T=\widetilde{\Omega}(\sqrt{m}R\delta^{-1})$ iterations, the solution matrix $A_{T}$ satisfies:

\displaystyle|u_{i}^{\top}A_{T}u_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

Proof.

We defer the proof to Appendix B.2 ∎

Theorem 5.1 follows immediately from Lemma 5.2 and Lemma 5.6.

6 Stochastic gradient descent

In this section, we show that the cost-per-iteration of the approximate matrix sensing algorithm can be improved by using a stochastic gradient descent (SGD). More specifically, SGD can obtain a $\delta$ -approximate solution with $O(Bn^{2})$ , where $0<B<m$ is the size of the mini batch in SGD. Below is the main theorem of this section:

Theorem 6.1 (Stochastic gradient descent for orthogonal measurements).

Suppose $u_{1},\dots,u_{m}\in\mathbb{R}^{n}$ are orthogonal unit vectors, and suppose $|b_{i}|\leq R$ for all $i\in[m]$ . There exists an algorithm such that for any $\delta\in(0,1)$ , performs $\widetilde{O}(m^{3/2}B^{-1}R\delta^{-1})$ iterations of gradient descent with $O(Bn^{2})$ -time per iteration and outputs a matrix $A\in\mathbb{R}^{n\times n}$ satisfies:

\displaystyle|u_{i}^{\top}Au_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

The algorithm and its time complexity are provided in Section 6.1. The convergence is proved in Section 6.2 and 6.3. The SGD algorithm for the general measurement without the assumption that the $\{u_{i}\}_{i\in[m]}$ are orthogonal vectors is deferred to Appendix E.

6.1 Algorithm

We can use the stochastic gradient descent algorithm (Algorithm 2) for matrix sensing. More specifically, in each iteration, we will uniformly sample a subset ${\cal B}\subset[m]$ of size $B$ , and then compute the gradient of the stochastic potential function:

\displaystyle\nabla\Phi_{\lambda}(A{,{\cal B}}):={\frac{m}{|{\cal B}|}\sum_{i\in{\cal B}}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda(u_{i}^{\top}Au_{i}-b_{i})),

(8)

which is an $n$ -by- $n$ matrix. Then, we do the following gradient step:

\displaystyle A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}.

(9)

Lemma 6.2 (Running time of stochastic gradient descent).

Algorithm 2 takes $O(mn^{2})$ -time for preprocessing and each iteration takes $O(Bn^{2})$ -time.

Proof.

The time-consuming step is to compute $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ . Since

\displaystyle\nabla\Phi_{\lambda}(A_{t})=\sum_{i=1}^{m}u_{i}u_{i}^{\top}\lambda\sinh\left(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})\right),

and $u_{i}\bot u_{j}$ for $i\neq j\in[m]$ , we know that $u_{i}$ is an eigenvector of $\nabla\Phi_{\lambda}(A)$ with eigenvalue $\lambda\sinh\left(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})\right)$ for each $i\in[m]$ . Thus, we have

	$\displaystyle\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}=$	$\displaystyle~{}\sum_{i=1}^{m}\lambda^{2}\sinh^{2}\left(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})\right)$
	$\displaystyle=$	$\displaystyle~{}\sum_{i=1}^{m}\lambda^{2}\sinh^{2}(\lambda z_{t,i}),$

where $z_{t,i}:=u_{i}^{\top}A_{t}u_{i}-b_{i}$ for $i\in[m]$ . Then, if we know ${z_{t,i}}_{i\in[m]}$ , we can compute $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ in $O(m)$ -time.

Consider the change $z_{t+1,i}-z_{t,i}$ :

		$\displaystyle z_{t+1,i}-z_{t,i}$
	$\displaystyle=$	$\displaystyle~{}u_{i}^{\top}(A_{t+1}-A_{t})u_{i}$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}}\cdot u_{i}^{\top}\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})u_{i}$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon\lambda m}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}B}\sum_{j\in{\cal B}_{t}}u_{i}^{\top}u_{j}u_{j}^{\top}u_{i}\cdot\sinh(\lambda z_{t,j})$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon\lambda m\sinh(\lambda z_{t,i})}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}B}\cdot{\bf 1}_{i\in{\cal B}_{t}},$

where the last step follows from $u_{i}\bot u_{j}$ for $i\neq j$ . Hence, if we have already computed $\{z_{t,i}\}_{i\in[m]}$ and $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ , $\{z_{t+1,i}\}_{i\in[m]}$ can be obtained in $O(B)$ -time.

Therefore, we preprocess $z_{1,i}=u_{i}^{\top}A_{1}u_{i}-b_{i}$ for all $i\in[m]$ in $O(mn^{2})$ -time. Then, in the $t$ -th iteration ( $t>0$ ), we first compute

\displaystyle\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})=\frac{m}{B}\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda z_{t,i})

in $O(Bn^{2})$ -time. Next, we compute $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ using $z_{t,i}$ in $O(m)$ -time. $A_{t+1}$ can be obtained in $O(n^{2})$ -time. Finally, we use $O(B)$ -time to update $\{z_{t+1,i}\}_{i\in[m]}$ .

Hence, the total running time per iteration is

\displaystyle O(Bn^{2}+m+n^{2}+B)=O(Bn^{2}).

∎

Algorithm 2 Matrix Sensing by Stochastic Gradient Descent.

1:procedure SGD(

\{u_{i},b_{i}\}_{i\in[m]}

)

\triangleright

Theorem 6.1

\tau\leftarrow\max_{i\in[m]}b_{i}

A_{1}\leftarrow\tau\cdot I

z_{i}\leftarrow u_{i}^{\top}A_{1}u_{i}-b_{i}

for

i\in[m]

5: for

t=1\to T

6: Sample

{\cal B}_{t}\subset[m]

of size

B

uniformly at random

\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})\leftarrow\frac{m}{B}\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda z_{i})

\|\nabla\Phi_{\lambda}(A_{t})\|_{F}\leftarrow\left(\sum_{i=1}^{m}\lambda^{2}\sinh^{2}(\lambda z_{i})\right)^{1/2}

A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}

10: for

i\in{\cal B}_{t}

11:

z_{i}\leftarrow z_{i}-\epsilon\lambda m\sinh(\lambda z_{i})/(\|\nabla\Phi_{\lambda}(A_{t})\|_{F}B)

12: end for

13: end for

14: return

A_{T+1}

15:end procedure

6.2 Analysis of One Iteration

Suppose $A\in\mathbb{R}^{n\times n}$ . Let ${\cal B}_{t}$ be a uniformly random $B$ -subset of $[m]$ at the $t$ -th iteration, where $B$ is a parameter.

We can compute the gradient of $\Phi_{\lambda}(A{,{\cal B}})$ with respect to $A$ as follows:

		$\displaystyle~{}\nabla\Phi_{\lambda}(A{,{\cal B}})$
	$\displaystyle=$	$\displaystyle~{}{\frac{m}{\|{\cal B}\|}\sum_{i\in{\cal B}}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda(u_{i}^{\top}Au_{i}-b_{i})),$

where $\nabla\Phi_{\lambda}(A{,{\cal B}})\in\mathbb{R}^{n\times n}$ .

We can also compute the Hessian of $\Phi_{\lambda}(A{,{\cal B}})$ with respect to $A$ as follows:

		$\displaystyle~{}\nabla^{2}\Phi_{\lambda}(A{,{\cal B}})$
	$\displaystyle=$	$\displaystyle~{}{\frac{m}{\|{\cal B}\|}\sum_{i\in{\cal B}}}(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\lambda^{2}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i}))$

where $\nabla^{2}\Phi_{\lambda}(A{,{\cal B}})\in\mathbb{R}^{n^{2}\times n^{2}}$ and $\otimes$ is the Kronecker product.

It is easy to see the expectations of the gradient and Hessian of $\Phi_{\lambda}(A,{\cal B})$ over a random set ${\cal B}$ :

	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{{\cal B}\sim[m]}[\nabla\Phi_{\lambda}(A,{\cal B})]=\nabla\Phi_{\lambda}(A),$
	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{{\cal B}\sim[m]}[\nabla^{2}\Phi_{\lambda}(A,{\cal B})]=\nabla^{2}\Phi_{\lambda}(A)$

Lemma 6.3 (Expected progress on potential).

Given $m$ vectors $u_{1},u_{2},\cdots,u_{m}\in\mathbb{R}^{n}$ . Assume $\langle u_{i},u_{j}\rangle=0$ for any $i\neq j\in[m]$ and $\|u_{i}\|^{2}=1$ , for all $i\in[m]$ . Let $\epsilon\lambda\leq 0.01\frac{|{\cal B}_{t}|}{m}$ , for all $t>0$ .

Then, we have

\displaystyle\operatorname*{{\mathbb{E}}}[\Phi_{\lambda}(A_{t+1})]\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot{\Phi_{\lambda}(A_{t})}+\lambda\epsilon\sqrt{m}.

Proof.

We first express the expectation as follows:

	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{A_{t+1}}[\Phi_{\lambda}(A_{t+1})]-\Phi_{\lambda}(A_{t})$
$\displaystyle\leq$	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{A_{t+1}}[\langle\nabla\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\rangle]$
$\displaystyle+$	$\displaystyle~{}O(1)\cdot\operatorname*{{\mathbb{E}}}_{A_{t+1}}[\langle\nabla^{2}\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})\rangle],$	(10)

which follows from Corollary A.2.

We choose

\displaystyle A_{t+1}=A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t}{,{\cal B}_{t}})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}.

Then, we can bound

	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{A_{t+1}}[-\operatorname{tr}[\nabla\Phi_{\lambda}(A_{t})\cdot(A_{t+1}-A_{t})]]$
$\displaystyle=$	$\displaystyle~{}\operatorname*{{\mathbb{E}}}_{{\cal B}_{t}}\Big{[}\operatorname{tr}\Big{[}\nabla\Phi_{\lambda}(A_{t})\cdot\frac{\epsilon\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}}\Big{]}\Big{]}$
$\displaystyle=$	$\displaystyle~{}\epsilon\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}$	(11)

We define for $t>0$ and $i\in[m]$ ,

\displaystyle z_{t,i}:=u_{i}^{\top}A_{t}u_{i}-b_{i}.

We need to compute this $\Delta_{2}$ . For simplificity, we consider $\Delta_{2}\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}$ ,

	$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})]\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
	$\displaystyle=$	$\displaystyle~{}(\lambda\epsilon)^{2}\cdot(\frac{m}{\|{\cal B}_{t}\|})^{2}\cdot\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\sinh(z_{t,i})\otimes(\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\sinh(z_{t,i})\Big{]}.$		(12)

Ignoring the scalar factor in the above equation, we have

$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i,j\in B_{t}}\sinh(z_{t,i})\sinh(z_{t,i})\cdot(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}$
$\displaystyle=$	$\displaystyle\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\in B_{t}}\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))\Big{]}$
$\displaystyle+$	$\displaystyle\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\neq j\in B_{t}}\sinh(z_{t,i})\sinh(z_{t,i})\cdot(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}$
$\displaystyle=:$	$\displaystyle~{}\widetilde{Q}_{1}+\widetilde{Q}_{2},$	(13)

where the first step follows that we extract the scalar values from Kronecker product, the second step comes from splitting into two partitions based on whether $i=j$ , the third step comes from the definition of $\widetilde{Q}_{1}$ and $\widetilde{Q}_{2}$ where $\widetilde{Q}_{1}$ denotes the diagonal term, and $\widetilde{Q}_{2}$ denotes the off-diagonal term. Taking expectation, we have

	$\displaystyle~{}\operatorname*{{\mathbb{E}}}[\Delta_{2}\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}]$
$\displaystyle=$	$\displaystyle~{}(\lambda\epsilon)^{2}\cdot(\frac{m}{\|{\cal B}_{t}\|})^{2}\operatorname*{{\mathbb{E}}}[\widetilde{Q}_{1}]$
$\displaystyle=$	$\displaystyle~{}(\lambda\epsilon)^{2}\cdot(\frac{m}{\|{\cal B}_{t}\|})^{2}\cdot\frac{\|{\cal B}_{t}\|}{m}\cdot Q_{1}$
$\displaystyle\leq$	$\displaystyle~{}(\lambda\epsilon)^{2}\cdot\frac{m}{\|{\cal B}_{t}\|}\cdot(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$	(14)

where the first step comes from extracting the constant terms from the expectation and Claim 5.5, the second step follows that $\operatorname*{{\mathbb{E}}}[\widetilde{Q}_{1}]=\frac{|{\cal B}_{t}|}{m}\cdot Q_{1}$ , and the third step comes from the Claim 5.4. Therefore, we have:

		$\displaystyle~{}\operatorname*{{\mathbb{E}}}[\Phi_{\lambda}(A_{t+1})]-\Phi_{\lambda}(A_{t})$
	$\displaystyle\leq$	$\displaystyle-\operatorname{{\mathbb{E}}}[\Delta_{1}]+O(1)\cdot\operatorname{{\mathbb{E}}}[\Delta_{2}]$
	$\displaystyle\leq$	$\displaystyle-\epsilon(1-O(\epsilon\lambda)\cdot\frac{m}{\|{\cal B}_{t}\|})\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle-0.9\epsilon\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle-0.9\epsilon\lambda\frac{1}{\sqrt{m}}(\Phi_{\lambda}(A_{t})-m)+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle-0.9\epsilon\lambda\frac{1}{\sqrt{m}}\Phi_{\lambda}(A_{t})+\epsilon\lambda\sqrt{m},$

where the first step comes from Eq. (6.2), the second step comes from Eq. (6.2) and Eq. (6.2), the third step follows from $\epsilon\leq 0.01\frac{|{\cal B}_{t}|}{\lambda m}$ , the forth step follows from Eq. (B.1), and the last step follows from $\epsilon\lambda\in(0,0.01)$ . ∎

6.3 Convergence for multiple iterations

The goal of this section is to prove the convergence of Algorithm 2.

Lemma 6.4 (Convergence of stochastic gradient descent).

\displaystyle T=\widetilde{\Omega}(m^{3/2}B^{-1}R\delta^{-1})

iterations, with high probability, the solution matrix $A_{T}$ satisfies:

\displaystyle|u_{i}^{\top}A_{T+1}u_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

Proof.

Similar to the proof of Lemma 5.6, we can bound the initial potential by:

\displaystyle\Phi(A_{1})\leq 2^{O(\lambda R)}.

In the following iterations, by Lemma 6.3, we have

\displaystyle\operatorname*{{\mathbb{E}}}[\Phi_{\lambda}(A_{t+1})]\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot{\Phi_{\lambda}(A_{t})}+\lambda\epsilon\sqrt{m},

as long as $\epsilon\leq 0.01\frac{|B_{t}|}{\lambda m}$ , where $B_{t}$ is a uniformly random subset of $[m]$ of size $B$ .

It suffices to take $\epsilon=O(\lambda^{-1}m^{-1}B)$ .

Now, we can apply Lemma 6.3 for $T$ times and get that

\displaystyle\operatorname*{{\mathbb{E}}}[\Phi(A_{T+1})]\leq 2^{-\Omega(T\epsilon\lambda/\sqrt{m})+O(\lambda R)}+2m.

By taking

\displaystyle T=\widetilde{\Omega}(m^{3/2}B^{-1}R\delta^{-1}),

we have

\displaystyle\Phi(A_{T+1})\leq O(m)

holds with high probability. By the same argument as in the proof of Lemma 5.6, we have

\displaystyle|u_{i}^{\top}A_{T+1}u_{i}-b_{i}|\leq\delta~{}\forall i\in[m].

The lemma is thus proved. ∎

References

Aar [07] Scott Aaronson. The learnability of quantum states. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 463(2088):3089–3114, 2007.
BLSS [20] Jan van den Brand, Yin Tat Lee, Aaron Sidford, and Zhao Song. Solving tall dense linear programs in nearly linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 775–788, 2020.
Bra [20] Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 259–278. SIAM, 2020.
CLMW [11] Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? J. ACM, 58(3), 2011.
CLS [19] Michael B. Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, page 938–942, New York, NY, USA, 2019. Association for Computing Machinery.
CZ [15] T Tony Cai and Anru Zhang. Rop: Matrix recovery via rank-one projections. The Annals of Statistics, 43(1):102–138, 2015.
DLS [23] Yichuan Deng, Zhihang Li, and Zhao Song. An improved sample complexity for rank-1 matrix sensing. arXiv preprint arXiv:2303.06895, 2023.
DLY [21] Sally Dong, Yin Tat Lee, and Guanghao Ye. A nearly-linear time algorithm for linear programs with small treewidth: A multiscale representation of robust central path. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1784–1797, 2021.
FGLE [12] Steven T Flammia, David Gross, Yi-Kai Liu, and Jens Eisert. Quantum tomography via compressed sensing: error bounds, sample complexity and efficient estimators. New Journal of Physics, 14(9):095022, 2012.
GS [22] Yuzhou Gu and Zhao Song. A faster small treewidth sdp solver. arXiv preprint arXiv:2211.06033, 2022.
HJS⁺ [22] Baihe Huang, Shunhua Jiang, Zhao Song, Runzhou Tao, and Ruizhe Zhang. Solving sdp faster: A robust ipm framework and efficient implementation. In FOCS, 2022.
HRVW [96] Christoph Helmberg, Franz Rendl, Robert J Vanderbei, and Henry Wolkowicz. An interior-point method for semidefinite programming. SIAM Journal on optimization, 6(2):342–361, 1996.
HSWZ [22] Hang Hu, Zhao Song, Omri Weinstein, and Danyang Zhuo. Training overparametrized neural networks in sublinear time. arXiv preprint arXiv:2208.04508, 2022.
JKL⁺ [20] Haotian Jiang, Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS), pages 910–918. IEEE, 2020.
JLSW [20] Haotian Jiang, Yin Tat Lee, Zhao Song, and Sam Chiu-wai Wong. An improved cutting plane method for convex optimization, convex-concave games, and its applications. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 944–953, 2020.
JM [13] Adel Javanmard and Andrea Montanari. Localization from incomplete noisy distance measurements. Found. Comput. Math., 13(3):297–345, jun 2013.
JMD [10] Prateek Jain, Raghu Meka, and Inderjit Dhillon. Guaranteed rank minimization via singular value projection. Advances in Neural Information Processing Systems, 23, 2010.
JN [08] Anatoli Juditsky and Arkadii S Nemirovski. Large deviations of vector-valued martingales in 2-smooth normed spaces. arXiv preprint arXiv:0809.0813, 2008.
JSWZ [21] Shunhua Jiang, Zhao Song, Omri Weinstein, and Hengjie Zhang. Faster dynamic matrix inverse for faster lps. In STOC. arXiv preprint arXiv:2004.07470, 2021.
KKD [15] Amir Kalev, Robert L Kosut, and Ivan H Deutsch. Quantum tomography protocols with positivity are compressed sensing protocols. npj Quantum Information, 1(1):1–6, 2015.
KRT [17] Richard Kueng, Holger Rauhut, and Ulrich Terstiege. Low rank matrix recovery from rank one measurements. Applied and Computational Harmonic Analysis, 42(1):88–116, 2017.
LB [09] Kiryung Lee and Yoram Bresler. Guaranteed minimum rank approximation from linear observations by nuclear norm minimization with an ellipsoidal constraint. arXiv preprint arXiv:0903.4742, 2009.
Liu [11] Yi-Kai Liu. Universal low-rank matrix recovery from pauli measurements. Advances in Neural Information Processing Systems, 24, 2011.
LMCC [19] Yuanxin Li, Cong Ma, Yuxin Chen, and Yuejie Chi. Nonconvex matrix factorization from rank-one measurements. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1496–1505. PMLR, 2019.
LS [14] Yin Tat Lee and Aaron Sidford. Path finding methods for linear programming: Solving linear programs in o (vrank) iterations and faster algorithms for maximum flow. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 424–433. IEEE, 2014.
LS [15] Yin Tat Lee and Aaron Sidford. Efficient inverse maintenance and faster algorithms for linear programming. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 230–249. IEEE, 2015.
LSW [15] Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 1049–1065. IEEE, 2015.
LSZ [19] Yin Tat Lee, Zhao Song, and Qiuyi Zhang. Solving empirical risk minimization in the current matrix multiplication time. In COLT, 2019.
LV [10] Zhang Liu and Lieven Vandenberghe. Interior-point method for nuclear norm approximation with application to system identification. SIAM Journal on Matrix Analysis and Applications, 31(3):1235–1256, 2010.
NN [94] Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM, 1994.
QSZZ [23] Liank Qin, Zhao Song, Lichen Zhang, and Danyang Zhuo. An online and unified algorithm for projection matrix vector multiplication with application to empirical risk minimization. In AISTATS, 2023.
RFP [10] Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471–501, 2010.
SY [21] Zhao Song and Zheng Yu. Oblivious sketching-based central path method for solving linear programming problems. In 38th International Conference on Machine Learning (ICML), 2021.
SYZ [21] Zhao Song, Shuo Yang, and Ruizhe Zhang. Does preprocessing help training over-parameterized neural networks? Advances in Neural Information Processing Systems, 34, 2021.
SZZ [21] Zhao Song, Lichen Zhang, and Ruizhe Zhang. Training multi-layer over-parametrized neural network in subquadratic time. arXiv preprint arXiv:2112.07628, 2021.
TBS⁺ [16] Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning, pages 964–973. PMLR, 2016.
WSB [11] Andrew Waters, Aswin Sankaranarayanan, and Richard Baraniuk. Sparcs: Recovering low-rank and sparse matrices from compressive measurements. Advances in neural information processing systems, 24, 2011.
WZG [17] Lingxiao Wang, Xiao Zhang, and Quanquan Gu. A unified computational and statistical framework for nonconvex low-rank matrix estimation. In Artificial Intelligence and Statistics, pages 981–990. PMLR, 2017.
ZJD [15] Kai Zhong, Prateek Jain, and Inderjit S Dhillon. Efficient matrix sensing using rank-1 gaussian measurements. In International conference on algorithmic learning theory, pages 3–18. Springer, 2015.

Appendix

Roadmap.

We first provide the proofs for matrix hyperbolic functions and properties of $\sinh$ and $\cosh$ in Appendix A. Then we provide the proofs for the gradient descent and stochastic gradient descent convergence analysis in Appendix B. We consider the spectral potential function with ground-truth oracle scenario in Appendix C. We analyze the gradient descent with non-orthogonal measurements in Appendix D. We provide the cost-per-iteration analysis for stochastic gradient descent under non-orthogonal measurements in Appendix E.

Appendix A Proofs of Preliminary Lemmas

A.1 Calculus tools

We state a useful calculus tool from prior work,

Lemma A.1 (Proposition 3.1 in [18]).

Let $\Delta$ be an open interval on the axis, and $f$ be $C^{2}$ function on $\Delta$ such that for certain $\theta_{\pm},\mu_{\pm}\in\mathbb{R}$ one has

	$\displaystyle~{}\forall(a<b,a,b\in\Delta):$
	$\displaystyle~{}\theta_{-}\cdot\frac{f^{\prime\prime}(a)+f^{\prime\prime}(b)}{2}+\mu_{-}\leq\frac{f^{\prime}(b)-f^{\prime}(a)}{b-a}$
	$\displaystyle~{}\frac{f^{\prime}(b)-f^{\prime}(a)}{b-a}\leq\theta_{+}\cdot\frac{f^{\prime\prime}(a)+f^{\prime\prime}(b)}{2}+\mu_{+},$

where $f^{\prime}$ and $f^{\prime\prime}$ means the first- and second-order derivatives of $f$ , respectively.

Let, further, ${\cal X}_{n}(\Delta)$ be the set of all $n\times n$ symmetric matrices with eigenvalues belonging to $\Delta$ . Then ${\cal X}_{n}(\Delta)$ is an open convex set in the space $S^{n}$ of $n\times n$ symmetric matrices, the function

\displaystyle F(X)=\operatorname{tr}[f(X)]:{\cal X}_{n}(\Delta)\rightarrow\mathbb{R}

is $C^{2}$ , and for every $X\in{\cal X}_{n}(\Delta)$ and every $H\in S^{n}$ one has

	$\displaystyle~{}\theta_{-}\cdot\operatorname{tr}[Hf^{\prime\prime}(X)H]+\mu_{-}\cdot\operatorname{tr}[H^{2}]\leq D^{2}F(X)[H,H]$
	$\displaystyle~{}D^{2}F(X)[H,H]\leq\theta_{+}\cdot\operatorname{tr}[Hf^{\prime\prime}(X)H]+\mu_{+}\cdot\operatorname{tr}[H^{2}],$

where $D$ means directional derivative.

We will use below corollary to compute the trace with a map $f:\mathbb{R}\rightarrow\mathbb{R}$ .

Corollary A.2.

Let $f:\mathbb{R}\rightarrow\mathbb{R}$ be a $C^{2}$ function. Let $A$ and $B$ be two symmetric matrices. We have

	$\displaystyle\operatorname{tr}[f(A)]\leq$	$\displaystyle~{}\operatorname{tr}[f(B)]+\operatorname{tr}[f^{\prime}(B)(A-B)]$
		$\displaystyle~{}+O(1)\cdot\operatorname{tr}[f^{\prime\prime}(B)(A-B)^{2}].$

A.2 Kronecker product

Suppose we have two matrice $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{p\times q}$ , we use $A\otimes B$ denote the Kronecker product:

\displaystyle A\otimes B=\left[\begin{array}[]{ccc}A_{1,1}{B}&\cdots&A_{1,n}{B}\\ \vdots&\ddots&\vdots\\ A_{m,1}{B}&\cdots&A_{m,n}{B}\end{array}\right].

We state a fact and delay the proof into Section A.

Fact A.3.

Suppose we have two matrices $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{n\times k}$ , we have

\displaystyle(A\otimes B)\cdot(C\otimes D)=(AC)\otimes(BD).

Proof.

From the definition of Kronecker product we have:

		$\displaystyle~{}(A\otimes B)\cdot(C\otimes D)$
	$\displaystyle=$	$\displaystyle~{}{\left[\begin{array}[]{ccc}A_{1,1}B&\ldots&A_{1,n}B\\ \vdots&\ddots&\vdots\\ A_{m,1}B&\ldots&A_{m,n}B\end{array}\right]\left[\begin{array}[]{ccc}C_{1,1}D&\ldots&C_{1,k}D\\ \vdots&\ddots&\vdots\\ C_{n,1}D&\ldots&C_{n,k}D\end{array}\right]}$
	$\displaystyle=$	$\displaystyle~{}\left[\begin{array}[]{ccc}(\sum_{i=1}^{n}A_{1,i}C_{i,1}){BD}&\cdots&(\sum_{i=1}^{n}A_{1,i}C_{i,k}){BD}\\ \vdots&\ddots&\vdots\\ (\sum_{i=1}^{n}A_{m,i}C_{i,1}){BD}&\cdots&(\sum_{i=1}^{n}A_{m,i}C_{i,k}){BD}\end{array}\right]$
	$\displaystyle=$	$\displaystyle~{}\left[\begin{array}[]{ccc}(AC)_{1,1}BD&\cdots&(AC)_{1,k}BD\\ \vdots&\ddots&\vdots\\ (AC)_{m,1}BD&\cdots&(AC)_{m,k}BD\end{array}\right]$
	$\displaystyle=$	$\displaystyle~{}(AC)\otimes(BD)$

Thus we complete the proof. ∎

A.3 Proof of $\cosh(A)$ upper bound

Lemma A.4 (Restatement of Lemma 3.2).

Let $A$ be a real symmetric matrix, then we have

\displaystyle\|\cosh(A)\|=\cosh(\|A\|)\leq\operatorname{tr}[\cosh(A)].

We also have $\|A\|\leq 1+\log(\operatorname{tr}[\cosh(A)])$ .

Proof.

Note that for each eigenvalue $\lambda$ of $A$ , we know that it corresponds to $\cosh(\lambda)$ for $\cosh(A)$ . The second inequality follows from the fact that $\cosh(A)$ is psd.

For the second part, we know that $\exp(x)/2\leq\cosh(x)$ , hence, $\exp(\|A\|)/2\leq\cosh(\|A\|)$ , and

	$\displaystyle\\|A\\|=$	$\displaystyle~{}\log(\exp(\\|A\\|))$
	$\displaystyle\leq$	$\displaystyle~{}\log(2\cosh(\\|A\\|))$
	$\displaystyle\leq$	$\displaystyle~{}1+\log(\operatorname{tr}[\cosh(A)]),$

where the second step is by the monotonicity of $\log(\cdot)$ and $\exp(\|A\|)\leq 2\cosh(\|A\|)$ , the last step is by $\cosh(\|A\|)\leq\operatorname{tr}[\cosh(A)]$ . ∎

We state a fact as follows:

Fact A.5.

For any real number $x$ , $\cosh^{2}(x)-\sinh^{2}(x)=1$

From the definition of $\cosh(x)$ and $\sinh(x)$ we have:

		$\displaystyle~{}\cosh^{2}(x)-\sinh^{2}(x)$
	$\displaystyle=$	$\displaystyle~{}\frac{1}{4}(e^{2x}+2+e^{-2x})-\frac{1}{4}(e^{2x}-2+e^{-2x})$
	$\displaystyle=$	$\displaystyle~{}1$

We also have the following lemma for matrix.

Lemma A.6.

Let $A$ be a real symmetric matrix, then we have

\displaystyle\cosh^{2}(A)-\sinh^{2}(A)=

\displaystyle~{}I.

Proof.

Since $A$ is real symmetric, we write it in the eigendecomposition form: $A=U\Lambda U^{\top}$ , then

		$\displaystyle~{}\cosh^{2}(A)-\sinh^{2}(A)$
	$\displaystyle=$	$\displaystyle~{}U\cosh^{2}(\Lambda)U^{\top}-U\sinh^{2}(\Lambda)U^{\top}$
	$\displaystyle=$	$\displaystyle~{}U(\cosh^{2}(\Lambda)-\sinh^{2}(\Lambda))U^{\top}$
	$\displaystyle=$	$\displaystyle~{}UU^{\top}$
	$\displaystyle=$	$\displaystyle~{}I,$

where the first step follows from $\cosh$ and $\sinh$ can be expressed as $\exp$ , the third step is by applying entrywise the identity $\cosh^{2}(x)-\sinh^{2}(x)=1$ . ∎

Appendix B Proofs of GD and SGD convergence

In this section, we provide proofs of convergence analysis the gradient descent and stochastic gradient descent matrix sensing algorithms.

B.1 Proof of GD Progress on Potential Function

We start with the progress of the gradient on the potential function in below lemma.

Lemma B.1 (Restatement of Lemma 5.3).

we have for any $t>0$ ,

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}

Proof.

We first Taylor expand $\Phi_{\lambda}(A_{t+1})$ as follows:

	$\displaystyle~{}\Phi_{\lambda}(A_{t+1})-\Phi_{\lambda}(A_{t})$
$\displaystyle\leq$	$\displaystyle~{}\langle\nabla\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\rangle+O(1)\langle\nabla^{2}\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})\rangle$
$\displaystyle:=$	$\displaystyle~{}\Delta_{1}+O(1)\cdot\Delta_{2},$	(15)

which follows from Lemma A.1.

We choose

\displaystyle A_{t+1}=A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}.

We can bound

	$\displaystyle\Delta_{1}=$	$\displaystyle~{}\operatorname{tr}[\nabla\Phi_{\lambda}(A_{t})(A_{t+1}-A_{t})]$
	$\displaystyle=$	$\displaystyle~{}-\epsilon\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}.$		(16)

Next, we upper-bound $\Delta_{2}$ . Define

\displaystyle z_{t,i}:=\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}).

and consider $\Delta_{2}\cdot(\lambda\epsilon)^{-2}\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}$ , which can be expressed as:

	$\displaystyle\Delta_{2}\cdot(\lambda\epsilon)^{-2}\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}(\lambda\epsilon)^{-2}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})]\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(z_{t,i}))\otimes(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(z_{t,i}))\Big{]}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i,j}\sinh(z_{t,i})\sinh(z_{t,i})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i=1}^{m}\sinh^{2}(z_{t,i}))(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))\Big{]}$
$\displaystyle+$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\neq j}\sinh(z_{t,i})\sinh(z_{t,j})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}$
$\displaystyle=:$	$\displaystyle~{}Q_{1}+Q_{2},$	(17)

where

\displaystyle Q_{1}:=\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i=1}^{m}\sinh^{2}(z_{t,i}))(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))\Big{]}

(18)

denotes the diagonal term, and

\displaystyle Q_{2}:=

\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\neq j}\sinh(z_{t,i})\sinh(z_{t,j})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}

(19)

denotes the off-diagonal term. The first step comes from the definition of $\Delta_{2}$ , the second step follows fromr eplacing $A_{t+1}-A_{t}$ using Eq (4), the third step follows that we extract the scalar values from Kronecker product, the fourth step comes from splitting into two partitions based on whether $i=j$ , the fifth step comes from the definition of $Q_{1}$ and $Q_{2}$ .

Thus,

$\displaystyle\Delta_{2}=$	$\displaystyle~{}(\epsilon\lambda)^{2}(Q_{1}+Q_{2})/\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}(\epsilon\lambda)^{2}(Q_{1}+0)/\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}(\epsilon\lambda)^{2}\cdot(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}).$	(20)

where the second step follows from Claim 5.5, and the third step follows from Claim 5.4.

Hence, we have

		$\displaystyle~{}\Phi_{\lambda}(A_{t+1})-\Phi_{\lambda}(A_{t})$
	$\displaystyle\leq$	$\displaystyle~{}\Delta_{1}+O(1)\cdot\Delta_{2}$
	$\displaystyle\leq$	$\displaystyle~{}-\epsilon\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}+O(1)(\epsilon\lambda)^{2}(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\\|\Phi_{\lambda}(A_{t})\\|_{F}+O(\epsilon\lambda)^{2}\sqrt{m}$

where the first step follows from Eq. (B.1), the second step follows from Eq. (22) and Eq. (B.1), the third step follows from $\epsilon\lambda\in(0,0.01)$ .

For $\|\Phi_{\lambda}(A_{t})\|_{F}$ , we have

	$\displaystyle~{}\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{2}]$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\sum_{i=1}^{m}(u_{i}u_{i}^{\top})^{2}\sinh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))]$
$\displaystyle=$	$\displaystyle~{}\sum_{i=1}^{m}\sinh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))$
$\displaystyle\geq$	$\displaystyle~{}\frac{1}{m}(\sum_{i=1}^{m}\cosh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))-m)^{2}$
$\displaystyle=$	$\displaystyle~{}\frac{1}{m}(\Phi_{\lambda}(A_{t})-m)^{2},$	(21)

where the first step comes from Eq. (5.2), the second steps follow from $u_{i}^{\top}u_{j}=0$ , the third step follows from $\|u_{i}\|_{2}=1$ , the forth step follows from Part 2 in Lemma 3.3, the fifth step follows from the definition of $\Phi_{\lambda}(A)$ .

Thus, we get that

\displaystyle\|\Phi_{\lambda}(A_{t})\|_{F}^{2}\geq~{}\lambda\epsilon\cdot\frac{1}{\sqrt{m}}|\Phi_{\lambda}(A_{t})-m|,

(22)

It implies that

		$\displaystyle\Phi_{\lambda}(A_{t+1})-\Phi_{\lambda}(A_{t})$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\lambda\frac{1}{\sqrt{m}}\|\Phi_{\lambda}(A_{t})-m\|+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\lambda\frac{1}{\sqrt{m}}\|\Phi_{\lambda}(A_{t})-m\|+0.1\epsilon\lambda\sqrt{m},$

where the second step follows from extracting the constant term from the summation.

Then, when $\Phi(A_{t})>m$ , we have

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}.

When $\Phi(A_{t})\leq m$ , we have

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1+0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})-0.8\lambda\epsilon\sqrt{m}.

The lemma is then proved. ∎

B.2 Proof of GD Convergence

In this section, we provide proofs of convergence analysis of gradient descent matrix sensing algorithm.

Lemma B.2 (Restatement of Lemma 5.6).

\displaystyle|u_{i}^{\top}A_{T}u_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

Proof.

Let $\tau=\max_{i\in[m]}b_{i}$ . At the beginning, we choose the initial solution $A_{1}:=\tau I_{n}$ where $I_{n}\in\mathbb{R}^{n\times n}$ is the identity matrix, and we have

	$\displaystyle\Phi(A_{1})=$	$\displaystyle~{}\sum_{i=1}^{m}\cosh(\lambda\cdot(\tau-b_{i}))$
	$\displaystyle\leq$	$\displaystyle~{}e^{\lambda\tau}\sum_{i=1}^{m}e^{-\lambda b_{i}}\leq 2^{O(\lambda R)},$

where the last step follows from $|b_{i}|\leq R$ for all $i\in[m]$ .

After $T$ iterations, we have

	$\displaystyle\Phi(A_{T+1})\leq$	$\displaystyle~{}(1-\frac{\epsilon\lambda}{\sqrt{m}})^{T}\Phi(A_{1})+2m$
	$\displaystyle\leq$	$\displaystyle~{}(1-\frac{\epsilon\lambda}{\sqrt{m}})^{T}\cdot 2^{O(\lambda R)}+2m$
	$\displaystyle\leq$	$\displaystyle~{}2^{-\Omega(T\epsilon\lambda/\sqrt{m})+O(\lambda R)}+2m$

where the first step follows from applying Lemma 5.3 for $T$ times, and $\sum_{i=1}^{T}(1-\epsilon\lambda/\sqrt{m})^{i-1}\epsilon\lambda\sqrt{m}\leq 2m$ .

As long as $T=\Omega(R\sqrt{m}/\epsilon)=\Omega(R\sqrt{m}\lambda)$ , then we have

\displaystyle\Phi(A_{T+1})\leq O(m).

This implies that for any $i\in[m]$ ,

	$\displaystyle\|u_{i}^{\top}A_{T+1}u_{i}-b_{i}\|\leq$	$\displaystyle~{}\lambda^{-1}\cdot\cosh^{-1}(O(m))$
	$\displaystyle=$	$\displaystyle~{}\lambda^{-1}\cdot O(\log m)$
	$\displaystyle=$	$\displaystyle~{}\delta,$

where we take $R=\Omega(\delta^{-1}\log m)$ .

Therefore, with $T=\widetilde{\Omega}(\sqrt{m}R\delta^{-1})$ iterations, Algorithm 1 can achieve that

\displaystyle|u_{i}^{\top}A_{T+1}u_{i}-b_{i}|\leq\delta~{}~{}~{}\forall i\in[m].

(23)

The theorem is then proved. ∎

Appendix C Spectral Potential function with ground-truth oracle

In this section, we consider the matrix sensing with spectral approximation; that is, we want to obtain a matrix $A$ that is a $\delta$ -spectral approximation of the ground-truth matrix $A_{\star}$ , i.e.,

\displaystyle(1-\delta)A_{\star}\preceq A\preceq(1+\delta)A_{\star}.

To do this, instead of performing a series of quadratic measurements, we assume that we have access to an oracle ${\cal O}_{A_{\star}}$ such that for any matrix $A\in\mathbb{R}^{n\times n}$ , the oracle will output a matrix $A_{\star}^{-1/2}AA_{\star}^{-1/2}$ . Algorithm 3 implements a matrix sensing algorithm with spectral approximation guarantee with the assumption of oracle ${\cal O}_{A_{\star}}$ .

We define the spectral loss function as follows:

\displaystyle\Psi_{\lambda}(A):=\operatorname{tr}[\cosh(\lambda(I-(A_{\star})^{-1/2}A(A_{\star})^{-1/2}))].

We will show that $\Psi_{\lambda}(A)$ can characterize the spectral approximation of $A$ with respect to $A_{\star}$ .

It is easy to see that if we can query an arbitrary $A$ to the ground-truth oracle ${\cal O}_{A_{\star}}$ , then we can definitely recover $A_{\star}$ exactly by querying ${\cal O}_{A_{\star}}(I)$ . Instead, in Algorithm 3, we focus on the following process: the initial matrix $A_{1}$ is given, and in the $t$ -th iteration, we first compute

\displaystyle X_{t}=\lambda(I-A_{\star}^{-1/2}A_{t}A_{\star}^{-1/2})

and do eigendecompsotion of $X_{t}$ to obtain $\Lambda_{t}$ such that $X_{t}=Q_{t}\Lambda_{t}Q_{t}^{\top}$ . Then we update the matrix $A_{t+1}$ by:

\displaystyle A_{t+1}=A_{t}+\epsilon\cdot A_{\star}^{1/2}\sinh(X_{t})A_{\star}^{1/2}/\|\sinh(X_{t})\|_{F}.

We are interested in the number of iterations needed to make $A_{t}$ be a $\delta$ -spectral approximation. We believe this example will provide some insight into this problem, and we leave the question of spectral-approximated matrix sensing without the ground-truth oracle to future work.

Algorithm 3 Matrix Sensing with Spectral Approximation.

1:procedure GradientDescent(

{\cal O}_{A_{\star}}

A_{1}

)

2: for

t=1\to T

X_{t}\leftarrow\lambda\cdot(I_{n}-{\cal O}_{A_{\star}}(A_{t}))

Q_{t}\Lambda_{t}Q_{t}^{\top}\leftarrow

Eigendecomposition of

X_{t}

\triangleright

It takes

O(n^{\omega})

-time

Y_{t}\leftarrow Q_{t}\cdot\sinh(\Lambda_{t})\cdot Q_{t}^{\top}

\triangleright

Y_{t}=\sinh(X_{t})

. It takes

O(n^{2})

-time

A_{t+1}\leftarrow A_{t}+\epsilon\cdot{\cal O}_{A_{\star}}(Y_{t})/\|Y_{t}\|_{F}

\triangleright

It takes

O(n^{2})

-time

7: end for

8: return

A_{T+1}

9:end procedure

Lemma C.1 (Progress on the spectral potential function).

Let $c\in(0,1)$ denote a sufficiently small positive constant. We define $X_{t}$ as follows:

\displaystyle X_{t}:=\lambda(I-(A_{\star})^{-1/2}A_{t}(A_{\star})^{-1/2})

Let

\displaystyle A_{t+1}=A_{t}+\epsilon\cdot\lambda(A_{\star})^{1/2}\sinh(X_{t})(A_{\star})^{1/2}/\|\lambda\cdot\sinh(X_{t})\|_{F}.

For any $\epsilon\in(0,1)$ and $\lambda\geq 1$ such $\lambda\epsilon\leq c$ , we have for any $t>0$ ,

\displaystyle\Psi_{\lambda}(A_{t+1})\leq(1-0.9\epsilon\lambda/\sqrt{n})\Psi_{\lambda}(A_{t})+\epsilon\lambda\sqrt{n}.

Proof.

We can compute

	$\displaystyle~{}\Psi_{\lambda}(A_{t+1})-\Psi_{\lambda}(A_{t})$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\cosh(X_{t+1})]-\operatorname{tr}[\cosh(X_{t})]$
$\displaystyle\leq$	$\displaystyle~{}-\lambda\cdot\operatorname{tr}[\sinh(X_{t})\cdot((A_{\star})^{-1/2}(A_{t+1}-A_{t})(A_{\star})^{-1/2})]$
$\displaystyle+$	$\displaystyle~{}O(1)\cdot\lambda^{2}\cdot\operatorname{tr}[\cosh(X_{t})\cdot((A_{\star})^{-1/2}(A_{t}-A_{t+1})(A_{\star})^{-1/2})^{2}]$
$\displaystyle=$	$\displaystyle~{}-\Delta_{1}+O(1)\cdot\Delta_{2},$	(24)

the first step is by expanding by definition, the second step is by Taylor expanding the first term at the point $I-(A_{\star})^{-1/2}A_{t}(A_{\star})^{-1/2}$ (via Lemma A.1), and the last step is by definition of $\Delta_{1}$ and $\Delta_{2}$ .

To further simplify proofs, we define

	$\displaystyle\nabla\Psi_{\lambda}(A_{t}):=$	$\displaystyle~{}\lambda\cdot(A_{\star})^{1/2}\sinh(X_{t})(A_{\star})^{1/2}$
	$\displaystyle\widetilde{\nabla}\Psi_{\lambda}(A_{t}):=$	$\displaystyle~{}\lambda\cdot\sinh(X_{t})$
	$\displaystyle\widetilde{\Delta}\Psi_{\lambda}(A_{t}):=$	$\displaystyle~{}\lambda\cdot\cosh(X_{t})$

To maximize the gradient progress, we should choose

\displaystyle A_{t+1}=A_{t}+\epsilon\cdot\nabla\Psi_{\lambda}(A_{t})/\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\|_{F}

Then

$\displaystyle\Delta_{1}=$	$\displaystyle~{}(\epsilon\lambda^{2})\cdot\operatorname{tr}[\sinh^{2}(X_{t})]/\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}$
$\displaystyle=$	$\displaystyle~{}\epsilon\cdot\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}^{2}/\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}$
$\displaystyle=$	$\displaystyle~{}\epsilon\cdot\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}$	(25)

and

$\displaystyle\Delta_{2}=$	$\displaystyle~{}\epsilon^{2}\lambda^{4}\cdot\operatorname{tr}[\cosh(X_{t})\cdot\sinh^{2}(\lambda(X_{t})]/\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\epsilon^{2}\lambda\cdot\operatorname{tr}[\widetilde{\Delta}\Psi_{\lambda}(A_{t})\cdot\widetilde{\nabla}\Psi_{\lambda}(A_{t})^{2}]/\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle\leq$	$\displaystyle~{}\epsilon^{2}\lambda\cdot\\|\widetilde{\Delta}\Psi_{\lambda}(A_{t})\\|_{F}\cdot\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})^{2}\\|_{F}/\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle\leq$	$\displaystyle~{}\epsilon^{2}\lambda\cdot\\|\widetilde{\Delta}\Psi_{\lambda}(A_{t})\\|_{F}$
$\displaystyle\leq$	$\displaystyle~{}\epsilon^{2}\lambda\cdot(\lambda\sqrt{n}+\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F})$	(26)

where the first step follows from the definition of $\Delta_{2}$ , the second step comes from the definition of $\widetilde{\Delta}\Psi_{\lambda}(A_{t})$ and $\widetilde{\nabla}\Psi_{\lambda}(A_{t})$ , the third step follows that $\|AB\|_{F}\leq\|A\|_{F}\|B\|_{F}$ , the forth step follows from $\|x\|_{4}^{2}\leq\|x\|_{2}^{2}$ , and the fifth step follows from Part 1 of Lemma 3.4.

Now, we need to lower bound $\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\|_{F}$ , we have

$\displaystyle\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}=$	$\displaystyle~{}(\operatorname{tr}[\lambda^{2}\sinh^{2}(X_{t})])^{1/2}$
$\displaystyle\geq$	$\displaystyle~{}\frac{\lambda}{\sqrt{n}}(\operatorname{tr}[\cosh(X_{t})]-n)$
$\displaystyle=$	$\displaystyle~{}\frac{\lambda}{\sqrt{n}}(\Psi_{\lambda}(A_{t})-n)$	(27)

where the second step follows from Part 2 in Lemma 3.4.

We know that

Then, we have

		$\displaystyle~{}\Psi_{\lambda}(A_{t+1})-\Psi_{\lambda}(A_{t})$
	$\displaystyle\leq$	$\displaystyle~{}-\epsilon\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}+\epsilon^{2}\lambda(\sqrt{n}+\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F})$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\\|\widetilde{\nabla}\Psi_{\lambda}(A_{t})\\|_{F}+\epsilon^{2}\lambda^{2}\sqrt{n}$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\lambda\frac{1}{\sqrt{n}}\Psi_{\lambda}(A_{t})+\epsilon\lambda\sqrt{n}$

where the first step follows from Eq. (C) and Eq. (C) , the second steps comes from $\epsilon\in(0,0.01)$ , the third step comes from Eq. (C) and $\epsilon\lambda\leq 1$ .

Finally, we complete the proof.

∎

Lemma C.2 (Small spectral potential implies good spectral approximation).

Let $A\in\mathbb{R}^{n\times n}$ be symmetric, and $\lambda>0$ . Suppose $\Psi_{\lambda}(A)\leq p$ for some $p>1$ . Then, we have

\displaystyle(1-\delta)A_{\star}\preceq A\preceq(1+\delta)A_{\star}

for $\delta=O(\lambda^{-1}\log p)$ .

Proof.

By the definition of $\Psi_{\lambda}(A)$ , $\Psi_{\lambda}(A)\leq p$ implies that for any $i\in[n]$ ,

\displaystyle\cosh(\lambda(1-\lambda_{i}(A_{\star}^{-1/2}AA_{\star}^{-1/2})))\leq p,

or equivalently,

\displaystyle\left|(1-\lambda_{i}(A_{\star}^{-1/2}AA_{\star}^{-1/2}))\right|\leq O(\lambda^{-1}\log p).

Hence, we have

\displaystyle(1-\delta)I_{n}\preceq A_{\star}^{-1/2}AA_{\star}^{-1/2}\preceq(1+\delta)I_{n},

where $\delta:=O(\lambda^{-1}\log p)$ . Therefore, by multiplying $A_{\star}^{-1/2}$ on both sides, we get that

\displaystyle(1-\delta)A_{\star}\preceq A\preceq(1+\delta)A_{\star},

which completes the proof of the lemma. ∎

Appendix D Gradient descent with General Measurements

In this section, we analyze the potential decay by gradient descent with non-orthogonal measurements. The main result of this section is Lemma D.1 in below.

We first recall the definition of the potential function $\Phi_{\lambda}(A)$ :

\displaystyle\Phi_{\lambda}(A):=\sum_{i=1}^{m}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i})),

its gradient $\nabla\Phi_{\lambda}(A)\in\mathbb{R}^{n\times n}$ :

\displaystyle\nabla\Phi_{\lambda}(A)=\sum_{i=1}^{m}u_{i}u_{i}^{\top}\lambda\sinh\left(\lambda(u_{i}^{\top}Au_{i}-b_{i})\right),

(28)

and its Hessian $\nabla^{2}\Phi_{\lambda}(A)\in\mathbb{R}^{n^{2}\times n^{2}}$ :

\displaystyle\nabla^{2}\Phi_{\lambda}(A)=\sum_{i=1}^{m}(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\lambda^{2}\cosh(\lambda(u_{i}^{\top}Au_{i}-b_{i})).

Lemma D.1 (Progress on entry-wise potential with general measurements).

Assume that $|u_{i}^{\top}u_{j}|\leq\rho$ and $\rho\leq\frac{1}{10m}$ , for any $i,j\in[m]$ and $\|u_{i}\|^{2}=1$ . Let $c\in(0,1)$ denote a sufficiently small positive constant. Then, for any $\epsilon,\lambda>0$ such that $\epsilon\lambda\leq c$ , we have for any $t>0$ ,

\displaystyle\Phi_{\lambda}(A_{t+1})\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}

Proof.

We first have

	$\displaystyle~{}\Phi_{\lambda}(A_{t+1})-\Phi_{\lambda}(A_{t})$
$\displaystyle\leq$	$\displaystyle~{}\langle\nabla\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\rangle+O(1)\langle\nabla^{2}\Phi_{\lambda}(A_{t}),(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})\rangle$
$\displaystyle:=$	$\displaystyle~{}-\Delta_{1}+O(1)\cdot\Delta_{2},$	(29)

which follows from Corollary A.2.

We choose

\displaystyle A_{t+1}=A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}.

(30)

We can bound

	$\displaystyle\Delta_{1}=$	$\displaystyle~{}-\operatorname{tr}[\nabla\Phi_{\lambda}(A_{t})(A_{t+1}-A_{t})]$
	$\displaystyle=$	$\displaystyle~{}\epsilon\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}.$		(31)

For $\|\Phi_{\lambda}(A_{t})\|_{F}^{2}$ ,

	$\displaystyle~{}\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{2}]$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\sum_{i=1}^{m}\sinh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))]$
$\displaystyle+$	$\displaystyle~{}\operatorname{tr}[\sum_{i=1}^{m}\sum_{j\neq i}^{m}(u_{i}u_{i}^{\top})(u_{j}u_{j}^{\top})\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))\cdot\sinh(\lambda(u_{j}^{\top}A_{t}u_{j}-b_{j}))]$
$\displaystyle\geq$	$\displaystyle~{}0.9\operatorname{tr}[\sum_{i=1}^{m}\sinh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))]$
$\displaystyle\geq$	$\displaystyle~{}0.9\frac{1}{m}(\sum_{i=1}^{m}\cosh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))-m)^{2}$
$\displaystyle=$	$\displaystyle~{}0.9\frac{1}{m}(\Phi_{\lambda}(A_{t})-m)^{2},$	(32)

where the first step follows from Eq. (28), the second steps follow from partitioning based on whether $i=j$ and $\|u_{i}\|_{2}=1$ , the third step comes from Claim D.2, the fourth step in Eq. (D) follows from Part 2 in Lemma 3.3, the fifth step follows from the definition of $\Phi_{\lambda}(A)$ .

Thus,

	$\displaystyle\Delta_{1}=$	$\displaystyle~{}-\operatorname{tr}[\nabla\Phi_{\lambda}(A_{t})(A_{t+1}-A_{t})]$
	$\displaystyle\geq$	$\displaystyle~{}\lambda\epsilon\cdot\frac{1}{\sqrt{m}}(\Phi_{\lambda}(A_{t})-m).$		(33)

For simplicity, we define

\displaystyle z_{t,i}:=\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}).

We need to compute this $\Delta_{2}$ . For simplificity, we consider $\Delta_{2}\cdot(\frac{1}{\epsilon\lambda})^{2}\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}$ , which can be expressed as:

	$\displaystyle\Delta_{2}\cdot(\frac{1}{\epsilon\lambda})^{2}\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\frac{1}{(\lambda\epsilon)^{2}}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(A_{t+1}-A_{t})\otimes(A_{t+1}-A_{t})]\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(z_{t,i}))\otimes(\sum_{i=1}^{m}u_{i}u_{i}^{\top}\sinh(z_{t,i}))\Big{]}$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})(\sum_{i,j}\sinh(z_{t,i})\sinh(z_{t,i})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))]$
$\displaystyle=$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})(\sum_{i=1}^{m}\sinh^{2}(z_{t,i}))(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))]$
$\displaystyle+$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})(\sum_{i\neq j}\sinh(z_{t,i})\sinh(z_{t,j})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))]$
$\displaystyle=$	$\displaystyle~{}Q_{1}+Q_{2},$	(34)

where

\displaystyle Q_{1}:=\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i=1}^{m}\sinh^{2}(z_{t,i}))(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))\Big{]}

(35)

denotes the diagonal term, and

\displaystyle Q_{2}:=\operatorname{tr}\Big{[}\nabla^{2}\Phi_{\lambda}(A_{t})\cdot(\sum_{i\neq j}\sinh(z_{t,i})\sinh(z_{t,j})(u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top}))\Big{]}

(36)

denotes the off-diagonal term. The first step comes from the definition of $\Delta_{2}$ , the second step follows from replacing $A_{t+1}-A_{t}$ using Eq. (30), the third step follows that we extract the scalar values from Kronecker product, the fourth step comes from splitting into two partitions based on whether $i=j$ , the fifth step comes from the definition of $Q_{1}$ and $Q_{2}$ .

Thus,

	$\displaystyle\Delta_{2}\leq$	$\displaystyle~{}(\epsilon\lambda)^{2}(Q_{1}+Q_{2})/\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
	$\displaystyle=$	$\displaystyle~{}1.3(\epsilon\lambda)^{2}\cdot(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}).$		(37)

where the second step follows from Claim D.3 and Claim D.5.

Hence, we have

		$\displaystyle~{}\Phi_{\lambda}(A_{t+1})-\Phi_{\lambda}(A_{t})$
	$\displaystyle\leq$	$\displaystyle~{}-\Delta_{1}+O(1)\cdot\Delta_{2}$
	$\displaystyle\leq$	$\displaystyle~{}-\epsilon\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}+O(1)(\epsilon\lambda)^{2}(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\\|\Phi_{\lambda}(A_{t})\\|_{F}+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\lambda\frac{1}{\sqrt{m}}(\Phi_{\lambda}(A_{t})-m)+O(\epsilon\lambda)^{2}\sqrt{m}$
	$\displaystyle\leq$	$\displaystyle~{}-0.9\epsilon\lambda\frac{1}{\sqrt{m}}\Phi_{\lambda}(A_{t})+\epsilon\lambda\sqrt{m},$

where the first step follows from Eq. (D), the second step follows from Eq. (D) and Eq. (D), the third step follows from $\epsilon\lambda\in(0,0.01)$ , the fourth step follows from Lemma A.6, and the final step follows that extracting the constant term from the summation.

The lemma is then proved. ∎

We prove some technical claims in below.

Claim D.2.

It holds that:

		$\displaystyle\sum_{i\neq j\in[m]}\langle u_{i},u_{j}\rangle^{2}\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))\sinh(\lambda(u_{j}^{\top}A_{t}u_{j}-b_{j}))$
	$\displaystyle\leq$	$\displaystyle~{}0.1\sum_{i=1}^{m}\sinh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))$

Proof.

We define $R_{i,j}$ and $R$ as follows:

	$\displaystyle R_{i,j}=$	$\displaystyle~{}\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))\sinh(\lambda(u_{j}^{\top}A_{t}u_{j}-b_{j}))$
	$\displaystyle R=$	$\displaystyle~{}\operatorname{tr}[\sum_{i=1}^{m}\sum_{j\neq i}^{m}(u_{i}u_{i}^{\top})(u_{j}u_{j}^{\top})\sinh(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}))\cdot\sinh(\lambda(u_{j}^{\top}A_{t}u_{j}-b_{j}))]$

Then we can upper bound $|R|$ by:

	$\displaystyle\|R\|=$	$\displaystyle~{}\operatorname{tr}[\sum_{i=1}^{m}\sum_{j\neq i}^{m}\|(u_{i}u_{i}^{\top})(u_{j}u_{j}^{\top})\|\|R_{i,j}\|]$
	$\displaystyle\leq$	$\displaystyle~{}\rho^{2}\operatorname{tr}[\sum_{i=1}^{m}\sum_{j\neq i}^{m}\|R_{i,j}\|]$
	$\displaystyle\leq$	$\displaystyle~{}\frac{\rho^{2}}{2}\operatorname{tr}[\sum_{i=1}^{m}\sum_{j\neq i}^{m}(R_{i,i}+R_{j,j})]$
	$\displaystyle\leq$	$\displaystyle~{}m\rho^{2}\operatorname{tr}[\sum_{i=1}^{m}R_{i,i}]$
	$\displaystyle\leq$	$\displaystyle~{}0.1\operatorname{tr}[\sum_{i=1}^{m}R_{i,i}]$

where the first step follows $|ab|=|a||b|$ , the second step follows $|u_{i}^{\top}u_{j}|\leq\rho$ , the third step follows that $|ab|\leq\frac{a^{2}+b^{2}}{2}$ , the fourth step follows from the summation over $j$ , and the fifth step comes from $m\rho^{2}\leq 0.1$ . ∎

Claim D.3.

For $Q_{1}$ defined in Eq. (35), we have

\displaystyle Q_{1}\leq 1.1\Big{(}\sqrt{m}+\frac{1}{\lambda}\|\nabla\Phi_{\lambda}(A_{t})\|_{F}\Big{)}\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}.

Proof.

For simplicity, we define $z_{t,i}$ to be

\displaystyle z_{t,i}:=\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i}).

Recall that

\displaystyle\nabla^{2}\Phi_{\lambda}(A_{t})=\lambda^{2}\cdot\sum_{i=1}^{m}(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\cosh(z_{t,i}).

For $Q_{1}$ , we have

$\displaystyle Q_{1}=$	$\displaystyle~{}\operatorname{tr}[\nabla^{2}\Phi_{\lambda}(A_{t})\sum_{i=1}^{m}\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top}\otimes u_{i}u_{i}^{\top}))]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\operatorname{tr}[\sum_{i=1}^{m}\cosh(z_{t,i})(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})\cdot\sum_{i=1}^{m}\sinh^{2}(z_{t,i})(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\operatorname{tr}[\cosh(z_{t,i})\sinh^{2}(z_{t,i})\cdot(u_{i}u_{i}^{\top}u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top}u_{i}u_{i}^{\top})]$
$\displaystyle+$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\sum_{j\neq i}^{m}\operatorname{tr}[\cosh(z_{t,i})\sinh^{2}(z_{t,j})\cdot(u_{i}u_{i}^{\top}u_{j}u_{j}^{\top})\otimes(u_{j}u_{j}^{\top}u_{i}u_{i}^{\top})]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\cosh(z_{t,i})\sinh^{2}(z_{t,i})$
$\displaystyle+$	$\displaystyle~{}\lambda^{2}\cdot\sum_{i=1}^{m}\sum_{j\neq i}^{m}\operatorname{tr}[\cosh(z_{t,i})\sinh^{2}(z_{t,j})\cdot(u_{i}u_{i}^{\top}u_{j}u_{j}^{\top})\otimes(u_{j}u_{j}^{\top}u_{i}u_{i}^{\top})]$
$\displaystyle\leq$	$\displaystyle~{}1.1\lambda^{2}\cdot(\sum_{i=1}^{m}\cosh^{2}(z_{t,i}))^{1/2}\cdot(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$
$\displaystyle\leq$	$\displaystyle~{}1.1\lambda^{2}\cdot B_{1}\cdot B_{2},$	(38)

where the first step comes from the definition of $Q_{1}$ , the second step comes from the definition of $\nabla^{2}\Phi_{\lambda}(A_{t})$ , the third step follows from $(A\otimes B)\cdot(C\otimes D)=(AC)\otimes(BD)$ and partition the terms based on whether $i=j$ , the fourth step comes from $\|u_{i}\|=1$ and $\operatorname{tr}[(u_{i}u_{i}^{\top})\otimes(u_{i}u_{i}^{\top})]=1$ , and the fifth step comes from Cauchy–Schwarz inequality and Claim D.4.

Claim D.4.

We can bound the off-diagonal entries by:

		$\displaystyle~{}\|\lambda^{2}\cdot\sum_{i=1}^{m}\sum_{j\neq i}^{m}\operatorname{tr}[\cosh(z_{t,i})\sinh^{2}(z_{t,j})\cdot(u_{i}u_{i}^{\top}u_{j}u_{j}^{\top})\otimes(u_{j}u_{j}^{\top}u_{i}u_{i}^{\top})]\|$
	$\displaystyle\leq$	$\displaystyle~{}0.1\lambda^{2}(\sum_{i=1}^{m}(\cosh(z_{t,1}))^{1/2}\cdot(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$

Proof.

		$\displaystyle~{}\|\lambda^{2}\cdot\sum_{i=1}^{m}\sum_{j\neq i}^{m}\operatorname{tr}[\cosh(z_{t,i})\sinh^{2}(z_{t,j})\cdot(u_{i}u_{i}^{\top}u_{j}u_{j}^{\top})\otimes(u_{j}u_{j}^{\top}u_{i}u_{i}^{\top})]\|$
	$\displaystyle\leq$	$\displaystyle~{}\rho^{2}\lambda^{2}\|\sum_{i=1}^{m}\sum_{j\neq i}^{m}\cosh(z_{t,i})\sinh^{2}(z_{t,j})\|$
	$\displaystyle\leq$	$\displaystyle~{}\rho^{2}\lambda^{2}(\sum_{i=1}^{m}\sum_{j\neq i}^{m}(\cosh^{2}(z_{t,i}))^{1/2}\cdot(\sum_{i=1}^{m}\sum_{j\neq i}^{m}\sinh^{4}(z_{t,j}))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}m\rho^{2}\lambda^{2}(\sum_{i=1}^{m}(\cosh^{2}(z_{t,i}))^{1/2}\cdot(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}0.1\lambda^{2}(\sum_{i=1}^{m}(\cosh^{2}(z_{t,i}))^{1/2}\cdot(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$

where the first step comes from $|\langle u_{i},u_{j}\rangle|\leq\rho$ , the second step comes from Cauchy–Schwarz inequality, the third step follows from summation over $m$ terms, and the fourth step comes from $\rho^{2}m\leq 0.1$ . ∎

For the term $B_{1}$ , we have

	$\displaystyle B_{1}=$	$\displaystyle~{}(\sum_{i=1}^{m}\cosh^{2}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F},$		(39)

where the second step follows Part 1 of Lemma 3.3.

For the term $B_{2}$ , we have

	$\displaystyle B_{2}=$	$\displaystyle~{}(\sum_{i=1}^{m}\sinh^{4}(\lambda(u_{i}^{\top}A_{t}u_{i}-b_{i})))^{1/2}$
	$\displaystyle\leq$	$\displaystyle~{}\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2},$		(40)

where the second step follows from $\|x\|_{4}^{2}\leq\|x\|_{2}^{2}$ . This implies that

	$\displaystyle Q_{1}\leq$	$\displaystyle~{}1.1\lambda^{2}\cdot B_{1}\cdot B_{2}$
	$\displaystyle\leq$	$\displaystyle~{}1.1\lambda^{2}\cdot(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\frac{1}{\lambda^{2}}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
	$\displaystyle=$	$\displaystyle~{}1.1(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}.$

This completes the proof. ∎

Claim D.5.

For $Q_{2}$ defined in Eq. (36), we have:

\displaystyle Q_{2}\leq 0.2\lambda^{2}(\sqrt{m}+\frac{1}{\lambda}\|\nabla\Phi_{\lambda}(A_{t})\|_{F})\cdot\|\nabla\Phi_{\lambda}(A_{t})\|_{F}^{2}

Proof.

Because in $Q_{2}$ we have :

$\displaystyle Q_{2}=$	$\displaystyle~{}\lambda^{2}\operatorname{tr}[\sum_{\ell=1}^{m}(\cosh(z_{t,\ell})\cdot u_{\ell}u_{\ell}^{\top}\otimes u_{\ell}u_{\ell}^{\top})\cdot\sum_{i\neq j}^{m}(\sinh(z_{t,i})\sinh(z_{t,j})\cdot u_{i}u_{i}^{\top}\otimes u_{j}u_{j}^{\top})]$
$\displaystyle=$	$\displaystyle~{}\lambda^{2}\operatorname{tr}[\sum_{\ell=1}^{m}\sum_{i\neq j}^{m}\cosh(z_{t,\ell})\sinh(z_{t,i})\sinh(z_{t,j})\cdot(u_{\ell}u_{\ell}^{\top}u_{i}u_{i}^{\top})\otimes(u_{\ell}u_{\ell}^{\top}u_{j}u_{j}^{\top})]$
$\displaystyle\leq$	$\displaystyle~{}\lambda^{2}\rho^{2}\sum_{\ell=1}^{m}\sum_{i\neq j}^{m}\cosh(z_{t,\ell})(\sinh^{2}(z_{t,i})+\sinh^{2}(z_{t,j}))$
$\displaystyle\leq$	$\displaystyle~{}2m\lambda^{2}\rho^{2}\sum_{\ell=1}^{m}\sum_{i=1}^{m}\cosh(z_{t,\ell})\sinh^{2}(z_{t,i})$
$\displaystyle\leq$	$\displaystyle~{}2m^{2}\lambda^{2}\rho^{2}\sum_{i=1}^{m}\cosh(z_{t,i})\sinh^{2}(z_{t,i})$
$\displaystyle\leq$	$\displaystyle~{}2m^{2}\lambda^{2}\rho^{2}(\sum_{i=1}^{m}(\cosh^{2}(z_{t,i}))^{1/2}(\sum_{i=1}^{m}\sinh^{4}(z_{t,i}))^{1/2}$
$\displaystyle\leq$	$\displaystyle~{}0.2\lambda^{2}(\sqrt{m}+\frac{1}{\lambda}\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F})\cdot\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$	(41)

where the second step follows from $(A\otimes B)\cdot(C\otimes D)=(AC)\otimes(BD)$ , the third step follows Cauchy–Schwarz inequality and $|\langle u_{i},u_{j}\rangle|\leq\rho$ , the fourth step follows from combining $\sinh^{2}(z_{t,i})$ and $\sinh^{2}(z_{t,j})$ , the fifth step comes from summation over $m$ terms, and the sixth step comes from Cauchy–Schwarz inequality and the seventh step follows from Eq. (D) and Eq. (D) and $m^{2}\rho^{2}\leq 0.1$ .

∎

Appendix E Stochastic Gradient Descent for General Measurements

In this section, we further extend the general measurement where $\{u_{i}\}_{i\in[m]}$ are non-orthogonal vectors and $|u_{i}^{\top}u_{j}|\leq\rho$ to the convergence analysis of the stochastic gradient descent matrix sensing algorithm. Algorithm 4 implements the stochastic gradient descent version of the matrix sensing algorithm.

In Algorithm 4, at each iteration $t$ , we first compute the stochastic gradient descent by:

\displaystyle\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})\leftarrow\frac{m}{B}\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda z_{i})

then we update the matrix with the gradient:

\displaystyle A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}

At the end of each iteration, we update $z_{i}$ by:

\displaystyle z_{i}\leftarrow z_{i}-\epsilon\lambda mw_{i,j}^{2}\sinh(\lambda z_{j})/(\|\nabla\Phi_{\lambda}(A_{t})\|_{F}B\;\;\forall i\in[m],j\in{\cal B}_{t}

We are interested in studying the time complexity and convergence analysis under the general measurement assumption.

Lemma E.1 (Cost-per-iteration of stochastic gradient descent for general measurements).

Algorithm 4 takes $O(mn^{2})$ -time for preprocessing and each iteration takes $O(Bn^{2}+m^{2})$ -time.

Proof.

Since $u_{i}$ ’s are no longer orthogonal, we need to compute $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ in the following way:

		$\displaystyle\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}^{2}$
	$\displaystyle=$	$\displaystyle~{}\operatorname{tr}\Big{[}\Big{(}\sum_{i=1}^{m}u_{i}u_{i}^{\top}\lambda\sinh(\lambda(\lambda z_{t,i}))\Big{)}^{2}\Big{]}$
	$\displaystyle=$	$\displaystyle~{}\lambda^{2}\sum_{i,j=1}^{m}\langle u_{i},u_{j}\rangle^{2}\sinh(\lambda(\lambda z_{t,i}))\sinh(\lambda(\lambda z_{t,j}))$
	$\displaystyle=$	$\displaystyle~{}\lambda^{2}\sum_{i,j=1}^{m}w_{i,j}^{2}\sinh(\lambda(\lambda z_{t,i}))\sinh(\lambda(\lambda z_{t,j})).$

Hence, with $\{z_{t,i}\}_{i\in[m]}$ , we can compute $\|\nabla\Phi_{\lambda}(A_{t})\|_{F}$ in $O(m^{2})$ -time.

Another difference from the orthogonal measurement case is the update for $z_{t+1,i}$ . Now, we have

		$\displaystyle z_{t+1,i}-z_{t,i}$
	$\displaystyle=$	$\displaystyle~{}u_{i}^{\top}(A_{t+1}-A_{t})u_{i}$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}}\cdot u_{i}^{\top}\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})u_{i}$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon\lambda m}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}B}\sum_{j\in{\cal B}_{t}}u_{i}^{\top}u_{j}u_{j}^{\top}u_{i}\cdot\sinh(\lambda z_{t,j})$
	$\displaystyle=$	$\displaystyle~{}-\frac{\epsilon\lambda m}{\\|\nabla\Phi_{\lambda}(A_{t})\\|_{F}B}\sum_{j\in{\cal B}_{t}}w_{i,j}^{2}\cdot\sinh(\lambda z_{t,j}).$

Hence, each $z_{t+1,i}$ can be computed in $O(B)$ -time. And it takes $O(mB)$ -time to update all $z_{t+1,i}$ .

The other steps’ time costs are quite clear from Algorithm 4. ∎

Lemma E.2 (Progress on expected potential with general measurements).

\displaystyle\operatorname*{{\mathbb{E}}}[\Phi_{\lambda}(A_{t+1})]\leq(1-0.9\frac{\lambda\epsilon}{\sqrt{m}})\cdot\Phi_{\lambda}(A_{t})+\lambda\epsilon\sqrt{m}

The proof is a direct generalization of Lemma 6.3 and is very similar to Lemma D.1. Thus, we omit it here.

Algorithm 4 Matrix Sensing with Stochastic Gradient Descent (General Measurements).

1:procedure SGD_General(

\{u_{i},b_{i}\}_{i\in[m]}

)

\triangleright

Lemma E.1

\tau\leftarrow\max_{i\in[m]}b_{i}

A_{1}\leftarrow\tau\cdot I

z_{i}\leftarrow u_{i}^{\top}A_{1}u_{i}-b_{i}

for

i\in[m]

\triangleright

z\in\mathbb{R}^{m}

w_{i,j}\leftarrow\langle u_{i},u_{j}\rangle

for

i,j\in[m]

\triangleright

w\in\mathbb{R}^{m\times m}

6: for

t=1\to T

7: Sample

{\cal B}_{t}\subset[m]

of size

B

uniformly at random

\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})\leftarrow\frac{m}{B}\sum_{i\in{\cal B}_{t}}u_{i}u_{i}^{\top}\lambda\sinh(\lambda z_{i})

\triangleright

It takes

O(Bn^{2})

-time

\|\nabla\Phi_{\lambda}(A_{t})\|_{F}\leftarrow\lambda\left(\sum_{i,j=1}^{m}w_{i,j}^{2}\sinh(\lambda z_{i})\sinh(\lambda z_{j})\right)^{1/2}

\triangleright

It takes

O(m^{2})

-time

10:

A_{t+1}\leftarrow A_{t}-\epsilon\cdot\nabla\Phi_{\lambda}(A_{t},{\cal B}_{t})/\|\nabla\Phi_{\lambda}(A_{t})\|_{F}

\triangleright

It takes

O(n^{2})

-time

11: for

i\in[m]

\triangleright

Update

z

. It takes

O(mB)

-time

12: for

j\in{\cal B}_{t}

13:

z_{i}\leftarrow z_{i}-\epsilon\lambda mw_{i,j}^{2}\sinh(\lambda z_{j})/(\|\nabla\Phi_{\lambda}(A_{t})\|_{F}B)

14: end for

15: end for

16: end for

17: return

A_{T+1}

18:end procedure

A General Algorithm for Solving Rank-one Matrix Sensing

1 Introduction

Problem 1.1 (Approximate matrix sensing).

Theorem 1.2 (Informal of Theorem 6.1).

2 Related Work

Linear Progamming

Semi-definite Programming

Matrix Sensing

3 Preliminary

Notations.

3.1 Matrix hyperbolic functions

Definition 3.1 (Matrix function).

Lemma 3.2.

3.2 Properties of sinh\sinh and cosh\cosh

Lemma 3.3 (Scalar version).

Proof.

Lemma 3.4 (Matrix version).

Proof.

4 Technique Overview

Gradient descent.

Stochastic gradient descent.

5 Gradient descent for entry-wise potential function

Theorem 5.1 (Gradient descent for orthogonal measurements).

5.1 Algorithm

Lemma 5.2 (Cost-per-iteration of gradient descent).

Proof.

5.2 Analysis of One Iteration

Lemma 5.3 (Progress on entry-wise potential).

Proof.

5.3 Technical Claims

Claim 5.4.

Proof.

Claim 5.5.

Proof.

5.4 Convergence for multiple iterations

Lemma 5.6 (Convergence of gradient descent).

Proof.

6 Stochastic gradient descent

Theorem 6.1 (Stochastic gradient descent for orthogonal measurements).

6.1 Algorithm

Lemma 6.2 (Running time of stochastic gradient descent).

Proof.

6.2 Analysis of One Iteration

Lemma 6.3 (Expected progress on potential).

Proof.

6.3 Convergence for multiple iterations

Lemma 6.4 (Convergence of stochastic gradient descent).

Proof.

References

Appendix

Roadmap.

Appendix A Proofs of Preliminary Lemmas

A.1 Calculus tools

Lemma A.1 (Proposition 3.1 in [18]).

Corollary A.2.

A.2 Kronecker product

Fact A.3.

Proof.

A.3 Proof of cosh⁡(A)\cosh(A) upper bound

Lemma A.4 (Restatement of Lemma 3.2).

Proof.

Fact A.5.

Lemma A.6.

Proof.

Appendix B Proofs of GD and SGD convergence

B.1 Proof of GD Progress on Potential Function

Lemma B.1 (Restatement of Lemma 5.3).

Proof.

B.2 Proof of GD Convergence

Lemma B.2 (Restatement of Lemma 5.6).

Proof.

Appendix C Spectral Potential function with ground-truth oracle

Lemma C.1 (Progress on the spectral potential function).

Proof.

Lemma C.2 (Small spectral potential implies good spectral approximation).

Proof.

Appendix D Gradient descent with General Measurements

Lemma D.1 (Progress on entry-wise potential with general measurements).

Proof.

Claim D.2.

3.2 Properties of $\sinh$ and $\cosh$

A.3 Proof of $\cosh(A)$ upper bound