Structure vs. Randomness for Bilinear Maps

In this paper we prove a tight relation between the slice rank and the analytic rank of bilinear maps, or $3$-tensors. Our proof crucially uses the notion of geometric rank as an intermediary, enabling the use of tools from algebraic geometry to ultimately prove that these three notions of rank are in fact equivalent up to a constant.

A $3$-tensor (or sometimes simply a tensor) over a field $\mathbb{F}$ is a three-dimensional matrix $(a_{i,j,k})_{i,j,k}\in\mathbb{F}^{n_{1}\times n_{2}\times n_{3}}$ with entries $a_{i,j,k}\in\mathbb{F}$. Equivalently, a tensor can be thought of as a degree-$3$ polynomial, namely, a trilinear form $T(\mathbf{{x}},\mathbf{{y}},\mathbf{{z}})=\sum_{i,j,k}a_{i,j,k}x_{i}y_{j}z_{k}$ with coefficients $a_{i,j,k}\in\mathbb{F}$, where $\mathbf{{x}}=(x_{1},\ldots,x_{n_{1}})$, $\mathbf{{y}}=(y_{1},\ldots,y_{n_{2}})$, $\mathbf{{z}}=(z_{1},\ldots,z_{n_{3}})$.¹¹1A trilinear form means that every monomial has exactly one variable from $\mathbf{{x}}$, one from $\mathbf{{y}}$, and one from $\mathbf{{z}}$, and so is linear separately in each of $\mathbf{{x}}$,$\mathbf{{y}}$, and $\mathbf{{z}}$. Note that a tensor is just a symmetric way to think of a bilinear map $f\colon\mathbb{F}^{n_{1}}\times\mathbb{F}^{n_{2}}\to\mathbb{F}^{n_{3}}$, where $f=(f_{1},\ldots,f_{n_{3}})$ with $f_{k}(\mathbf{{x}},\mathbf{{y}})=\sum_{i,j}a_{i,j,k}x_{i}y_{j}$; indeed, each $f_{k}$ corresponds to a slice $(a_{i,j,k})_{i,j}$ of $T$.²²2Yet another point of view is that a $3$-tensor is a member of the vector space $V_{1}\otimes V_{2}\otimes V_{3}$, where $V_{1}=\mathbb{F}^{n_{1}},V_{2}\in\mathbb{F}^{n_{2}},V_{3}=\mathbb{F}^{n_{3}}$ are finite-dimensional vector spaces over $\mathbb{F}$. As opposed to matrices, which have only one notion of rank, there are multiple notions of rank for $3$-tensors. The notions of rank of $3$-tensors we consider are defined as follows:

• •

  The slice rank of $T$, denoted $\operatorname{SR}(T)$, is the smallest $r\in\mathbb{N}$ such that $T$ can be decomposed as $T=\sum_{i=1}^{r}f_{i}g_{i}$ where $f_{i}$ is an $\mathbb{F}$-linear form in either the $\mathbf{{x}}$, $\mathbf{{y}}$, or $\mathbf{{z}}$ variables and $g_{i}$ is an $\mathbb{F}$-bilinear form in the remaining two sets of variables, for each $i$.

• •

  The analytic rank of $T$ over a finite field $\mathbb{F}$ is given by $\operatorname{AR}(T)=-\log_{|\mathbb{F}|}\operatorname*{\mathbb{E}}_{\mathbf{{x}},\mathbf{{y}},\mathbf{{z}}}\chi(T(\mathbf{{x}},\mathbf{{y}},\mathbf{{z}}))$,³³3We use $\operatorname*{\mathbb{E}}_{\mathbf{{x}}}$ to denote averaging, so $\operatorname*{\mathbb{E}}_{\mathbf{{x\in\mathbb{F}^{n}}}}$ stands for $|\mathbb{F}|^{-n}\sum_{\mathbf{{x}}\in\mathbb{F}^{n}}$. where we fix $\chi$ to be any nontrivial additive character of $\mathbb{F}$ (e.g., $\chi(x)=\exp(2\pi ix/p)$ when $\mathbb{F}=\mathbb{F}_{p}$ is prime).

• •

  The geometric rank of $T$, viewed as a bilinear map $f$,⁴⁴4Although permuting $\mathbf{{x}},\mathbf{{y}},\mathbf{{z}}$ gives rise to three distinct bilinear maps corresponding to $T$, the definition of $\operatorname{GR}(T)$ is invariant under them (see Theorem 3.1 in [30]). is defined as $\operatorname{GR}(T)=\operatorname{codim}\ker f$, the codimension of the algebraic variety $\ker f=\{(\mathbf{{x}},\mathbf{{y}})\in\overline{\mathbb{F}}^{n_{1}}\times\overline{\mathbb{F}}^{n_{2}}\mid f(\mathbf{{x}},\mathbf{{y}})=\mathbf{{0}}\}$.

We note that all three notions above generalize matrix rank. Moreover, just like matrix rank, for any $T\in\mathbb{F}^{n\times n\times n}$ all three quantities lie in the range $[0,n]$. Furthermore, for the $n\times n\times n$ identity tensor $I_{n}$ we have $\operatorname{SR}(I_{n})=\operatorname{GR}(I_{n})=n$ and $\operatorname{AR}(I_{n})=(1-o_{|\mathbb{F}|}(1))n$. The slice rank was defined by Tao [40] in the context of the solution of the cap-set problem (similar notions have been considered before in other areas of research). The analytic rank was introduced by Gowers and Wolf [19] in the context of higher-order Fourier analysis. Roughly, this notion measures how close to uniform is the distribution of the values of the polynomial corresponding to the tensor. The geometric rank is an algebro-geometric notion of rank that was recently introduced in [30]. (A related notion was applied by Schmidt [37] in the context of number theory.) Intuitively, it measures the number of “independent” components of the corresponding bilinear map. We use it as a geometric analogue of the bias of the (output distribution of the) bilinear map.

Understanding the structure of $d$-tensors, or $d$-dimensional matrices, that have low analytic rank is important in many applications of the structure-vs-randomness dichotomy, in additive combinatorics, coding theory and more (see, e.g., [4, 20]). A recent breakthrough, obtained independently by Milićević [33] and by Janzer [25], showed that the partition rank of a $d$-tensor, which is a generalization of slice rank to $d$-tensors, is bounded from above by roughly $\operatorname{AR}(T)^{2^{2^{{\sf poly}(d)}}}$. For fixed $d$ this is a polynomial bound, which proves a conjecture of Kazhdan and Ziegler [28]. Lovett [31], as others have, asks whether in fact a linear upper bound holds. For $3$-tensors, the best known bound until this work was $\operatorname{SR}(T)\leq O(\operatorname{AR}(T)^{4})$ by Haramaty and Shpilka [22]. They write: “It is an interesting open question to decide whether we can do only with the $\sum_{j=1}^{O(\log_{|\mathbb{F}|}1/\delta)}\ell_{i}\cdot q_{i}$ part”, which refers to a linear upper bound $\operatorname{SR}(T)\leq O(\operatorname{AR}(T))$. Our main result is as follows.

We note that the reverse inequalities are easy: $\operatorname{GR}(T)\leq\operatorname{SR}(T)$ (see Theorem 4.1 in [30]) and $\operatorname{AR}(T)\leq\operatorname{SR}(T)$ (see Lemma 2.2 of [27] or [31]). Thus, as mentioned above, an immediate—and perhaps surprising—corollary of Theorem 1 is that the combinatorial notion $\operatorname{SR}(T)$, the algebro-geometric notion $\operatorname{GR}(T)$, and the analytic notion $\operatorname{AR}(T)$ are all, up to a constant, equivalent notions of rank. In particular, if one wants to estimate the slice rank of a $3$-tensor, as Tao did in a solution of the cap-set problem [40], then it is necessary and sufficient to instead estimate the bias of the tensor.

The importance of bilinear maps in theoretical computer science cannot be overstated. One example, in the area of algebraic algorithms, is matrix multiplication. Note that the operation of multiplying two matrices $X,Y\in\mathbb{F}^{m\times m}$ is a bilinear map ${\sf{MM}}_{n}\colon\mathbb{F}^{n}\times\mathbb{F}^{n}\to\mathbb{F}^{n}$ with $n=m^{2}$, as every entry of $XY$ is a bilinear form in the entries of $X$ and $Y$. It has been a persistent challenge to upper bound the arithmetic complexity of matrix multiplication, that is, the minimum number of $+,-,\cdot,\div$ operations over $\mathbb{F}$ required to express ${\sf{MM}}_{n}$ in terms of its variables. Current research puts the complexity of ${\sf{MM}}_{n}$ below $O(n^{1.2})$ (the state of the art is $O(n^{1.18643})$ due to Alman and Williams [2]), with the ultimate goal of getting all the way down to $n^{1+o(1)}$. For another example of the challenge of bilinear maps, this time in the area of circuit complexity, we mention that explicitly finding even a single bilinear map⁶⁶6Or, equivalently, a single degree-$3$ polynomial $\sum_{k=1}^{N}f_{k}(\mathbf{{x}},\mathbf{{y}})z_{k}$. $f\colon\mathbb{F}^{n}\times\mathbb{F}^{n}\to\mathbb{F}^{n}$ with provably superlinear arithmetic complexity, say $\Omega(n^{1.001})$, would imply the first such lower bound in circuit complexity. This should be compared with the fact that almost every bilinear map $f\colon\mathbb{F}^{n}\times\mathbb{F}^{n}\to\mathbb{F}^{n}$ has arithmetic complexity $\Theta(n^{2})$. Finally, in the area of identity testing, it was shown by Valiant [41] that identity testing of formulas reduces to deciding whether a given bilinear map $f\colon\mathbb{F}^{n}\times\mathbb{F}^{n}\to\mathbb{F}^{m}$ has full commutative rank, meaning a linear combination of its components $f_{i}$ has full rank $n$. Whether this can be decided efficiently remains an open question, despite being raised by Edmonds [14] in the early days of computer science, and it has close ties with a variety of other topics, from perfect matchings in bipartite graphs to matrix scaling (see [18]).

Given the importance of bilinear maps, we propose studying other foundational questions of theoretical computer science via the lens of bilinear maps. Consider Mahaney’s Theorem [32], a classical result in computational complexity. It states that, assuming ${\sf{P}}\neq~{}{\sf{NP}}$, no ${\sf{NP}}$-hard language is sparse. Phrased differently, if a boolean function $f\colon\{0,1\}^{*}\to\{0,1\}$ is “extremely” biased in the sense that $|f^{-1}(1)\cap\{0,1\}^{n}|\leq{\sf poly}(n)$, then it is not ${\sf{NP}}$-hard. Multiple other classical results in the same vein have been proved (see [17, 24, 34, 8, 9, 21]), giving implications of such extreme bias for various complexity classes. This raises the following fundamental question.

We make progress towards the above Question for the class of bilinear maps $f$; our notion of complexity is multiplicative complexity $\operatorname{C}^{*}(f)$, which is the number of (non-scalar) multiplications needed to compute $f$ by an arithmetic circuit; our notion of bias is the min-entropy $\operatorname{H_{\infty}}(f)$ of the output distribution of $f$. See Section 5 for a more formal discussion and the proof.

Another closely related classical result says that, assuming ${\sf{P}}\neq{\sf{NP}}$, it is impossible to efficiently solve ${\sf{SAT}}$ on all but at most polynomially many inputs (this is sometimes phrased as saying that ${\sf{SAT}}$ is not “${\sf{P}}$-close”). Again, this raises a fundamental question: For a given class of functions, what is the best possible approximation of a hard function? Put differently, what is the best possible worst-case to average-case reduction? For bilinear maps, we give an optimal answer to this question; again, see Section 5 for a more formal discussion and the proof. We say that maps $f,g$ are $\delta$-close if $\Pr_{x}[f(x)=g(x)]=\delta$,⁸⁸8We use $\Pr_{x}$ to denote probability under a uniform choice of $x$. and denote by $\operatorname{SR}(f)$ the slice rank of the $3$-tensor corresponding to a bilinear map $f$.

We note that Kaufman and Lovett [26] prove such a reduction for degree-$d$ polynomials over general finite fields, improving a previous result by Green and Tao [20]. However, their reduction is qualitative in nature and the implied bounds are far from optimal (see the next subsection for more discussion on previous results and techniques).

Our proof for the bound $\operatorname{SR}(T)=O(\operatorname{AR}(T))$ in Theorem 1 (and ultimately for complexity-vs-bias trade-offs for bilinear maps in Section 5) goes through an algebraically closed field—despite the statement ostensibly being about polynomials over finite fields. We use the concepts of dimension and tangent spaces from algebraic geometry to obtain our slice rank decomposition, which ends up yielding the bound $\operatorname{SR}(T)=O(\operatorname{GR}(T))$ (Theorem 3.1). To finish the proof of Theorem 1, we prove a new generalization of the Schwartz-Zippel lemma appropriate for our setting, which yields $\operatorname{GR}(T)=O(\operatorname{AR}(T))$ (Proposition 4.1).

To obtain the slice rank decomposition mentioned above, we first prove a result about linear spaces of matrices in which low-rank matrices have high dimension: we show that in any such space, one can always find a somewhat large decomposable subspace (Proposition 3.4, Item (2)); following Atkinson and Lloyd [3] (see also [16]), a space of matrices is decomposable if, roughly speaking, there is a basis where all the matrices have the same block of zeros. This result is proved by looking at a tangent space to a determinantal variety at a non-singular point, which turns out to be a decomposable matrix space in the sense above (Proposition 2.4). We further show that such a matrix space can be thought of as a $3$-tensor of low slice rank (Lemma 2.3). Since the above is proved by applying algebro-geometric tools on varieties, which naturally live over the algebraic closure of our finite field, the resulting slice rank decomposition has coefficients in the closure; however, using a result of Derksen [11], one can convert a small slice rank decomposition over the closure into a decomposition over the base field (Proposition 3.2). Finally, we use a result of [30] to combine the slice rank information we obtained above into a bound on the geometric rank (Fact 3.5).

We note that our arguments diverge from proofs used in previous works. In particular, we do not use results from additive combinatorics at all, nor do we use any “regularity lemma” for polynomials or notions of quasi-randomness. Instead, our arguments use a combination of algebraic and geometric ideas, which perhaps helps explain why we are able to obtain linear upper bounds.

We will need only a very small number of basic concepts from algebraic geometry, which we quickly review next. All the material here can be found in standard textbooks (e.g., [23, 36]). A variety $\mathbf{{V}}$ is the set of solutions, in an algebraically closed field, of some finite set of polynomials. More formally, for a field $\mathbb{F}$,⁹⁹9We henceforth denote by $\overline{\mathbb{F}}$ the algebraic closure of the field $\mathbb{F}$. the variety $\mathbf{{V}}\subseteq\overline{\mathbb{F}}^{n}$ cut out by the polynomials $f_{1},\ldots,f_{m}\in\mathbb{F}[x_{1},\ldots,x_{n}]$ is

We say that $\mathbf{{V}}\subseteq\overline{\mathbb{F}}^{n}$ is defined over $\mathbb{F}$ as it can be cut out by polynomials whose coefficients lie in $\mathbb{F}$. The ideal of $\mathbf{{V}}$ is $\operatorname{I}(\mathbf{{V}})=\{f\in\mathbb{F}[\mathbf{{x}}]\mid\forall p\in\mathbf{{V}}\colon f(p)=0\}$. Any variety $\mathbf{{V}}$ can be uniquely written as the union of irreducible varieties, where a variety is said to be irreducible if it cannot be written as the union of strictly contained varieties. The dimension of a variety $\mathbf{{V}}$, denoted $\dim\mathbf{{V}}$, is the maximal length $d$ of a chain of irreducible varieties $\emptyset\neq\mathbf{{V}}_{1}\subsetneq\cdots\subsetneq\mathbf{{V}}_{d}\subsetneq\mathbf{{V}}$. The codimension of $\mathbf{{V}}\subseteq\overline{\mathbb{F}}^{n}$ is simply $\operatorname{codim}\mathbf{{V}}=n-\dim\mathbf{{V}}$.

In the rest of the paper we will often find it convenient to identify, with a slight abuse of notation, a bilinear map $f\colon\mathbb{F}^{n_{1}}\times\mathbb{F}^{n_{2}}\to\mathbb{F}^{m}$ (or tensor) with a linear subspace of matrices, or matrix space, $\mathbf{{L}}\preceq\mathbb{F}^{n_{1}\times n_{2}}$. If $f=(f_{1},\ldots,f_{m})$, we will identify $f$ with the linear subspace $\mathbf{{L}}$ spanned by the $m$ matrices corresponding to the bilinear forms $f_{1},\ldots,f_{m}$. Note that this identification is not a correspondence, as it involves choosing a basis for $\mathbf{{L}}$. Importantly, however, since the notions of tensor rank that we study are invariant under the action of the general linear group $\mathrm{GL}_{n}$ on each of the axes, the choice of basis we make is immaterial in the definition of rank, meaning that $\operatorname{GR}(\mathbf{{L}})$, $\operatorname{SR}(\mathbf{{L}})$, $\operatorname{AR}(\mathbf{{L}})$ are nevertheless well defined.

For the reader’s convenience, we summarize below the different perspectives of tensor/bilinear map/matrix space that we use, and how they relate to each other:

• •

  A tensor $T=(a_{i,j,k})\in\mathbb{F}^{n_{1}\times n_{2}\times n_{3}}$, or a multilinear form $T(\mathbf{{x}},\mathbf{{y}},\mathbf{{z}})=\sum_{i,j,k}a_{i,j,k}x_{i}y_{j}z_{k}$.

• •

  A bilinear form $f=(f_{1},\ldots,f_{n_{3}})\colon\mathbb{F}^{n_{1}}\times\mathbb{F}^{n_{2}}\to\mathbb{F}^{n_{3}}$ with $f_{k}(\mathbf{{x}},\mathbf{{y}})=\sum_{i,j}a_{i,j,k}x_{i}y_{j}$.

• •

  A matrix space $\mathbf{{L}}\preceq\mathbb{F}^{n_{1}\times n_{2}}$ spanned by $\{A_{1},\ldots,A_{n_{3}}\}$ where $A_{k}=(a_{i,j,k})_{i,j}$.

For a variety $\mathbf{{V}}\subseteq\mathbb{K}^{n}$, the tangent space $\mathbf{T}_{p}\mathbf{{V}}$ to $\mathbf{{V}}$ at the point $p\in\mathbf{{V}}$ is the linear subspace

Equivalently, for any choice of a generating set $\{g_{1},\ldots,g_{s}\}\subseteq\mathbb{K}[x_{1},\ldots,x_{n}]$ for the ideal $\operatorname{I}(\mathbf{{V}})$ (which is finitely generated by Hilbert’s basis theorem), the tangent space at $p\in\mathbf{{V}}$ is $\mathbf{T}_{p}\mathbf{{V}}=\ker\operatorname{\mathbf{J}}_{p}$, where $\operatorname{\mathbf{J}}_{p}$ is the Jacobian matrix

We will need the following basic fact about tangent spaces (for a proof see, e.g., Theorem 2.3 in [36]).

We will also need the following easy observation about the interplay between tangents and intersections.

We henceforth denote by $\mathbf{{M}}_{r}=\mathbf{{M}}_{r}(\mathbb{K}^{m\times n})\subseteq\mathbb{K}^{m\times n}$ the variety of matrices in $\mathbb{K}^{m\times n}$ of rank at most $r$. Note that $\mathbf{{M}}_{r}$ is indeed a variety, as it is cut out by a finite set of polynomials: all $(r+1)\times(r+1)$ minors. It is therefore referred to in the literature as a determinantal variety.

The following crucial lemma shows that certain tangent spaces of the variety $\mathbf{{M}}_{r}=\mathbf{{M}}_{r}(\mathbb{K}^{m\times n})$, which are matrix spaces, have a small slice rank (recall Subsection 2.2 for the terminology).

To prove Lemma 2.3 will need the following result, which explicitly describes the tangent space to $\mathbf{{M}}_{r}$ at any matrix of rank exactly $r$. It can be deduced from Example 14.16 in [23]. We prove it below for completeness.

We note that any matrix $A$ with $\operatorname{rank}(A)=r$ is a nonsingular point of $\mathbf{{M}}_{r}$ (i.e., $\dim\mathbf{T}_{A}\mathbf{{M}}_{r}=\dim\mathbf{{M}}_{r}$), whereas any matrix $B$ with $\operatorname{rank}(B)<r$ is a singular point, and in fact, $\mathbf{T}_{B}\mathbf{{M}}_{r}(\mathbb{K}^{m\times n})=\mathbb{K}^{m\times n}$.

For $\mathbf{{L}}\preceq\mathbb{K}^{m\times n}$ a matrix space we define the variety $\mathbf{{L}}_{r}=\mathbf{{L}}\cap\mathbf{{M}}_{r}$ (here $\mathbf{{M}}_{r}=\mathbf{{M}}_{r}(\mathbb{K}^{m\times n})$) of all matrices in $\mathbf{{L}}$ of rank at most $r$. We next bound the slice rank of a matrix space using these linear sections of a determinantal variety. We denote by $\operatorname{codim}_{L}\mathbf{{X}}$ the codimension of a variety $\mathbf{{X}}\subseteq L$ inside a linear space $L$; that is, $\operatorname{codim}_{L}\mathbf{{X}}=\dim L-\dim\mathbf{{X}}$.

We note that an immediate corollary of Proposition 3.4 is a slice rank upper bound of $2r$ for any subspace of matrices of rank at most $r$.

To prove Theorem 3.1 we also need the following characterization of geometric rank. Recall that $\operatorname{GR}(T)=\operatorname{codim}\ker T$ where $\ker T=\{(\mathbf{{x}},\mathbf{{y}})\mid T(\mathbf{{x}},\mathbf{{y}},\cdot)=\mathbf{{0}}\}$.

Fact 3.5 is proved via the decomposition

using a result from algebraic geometry on the dimensions of fibers, and the fact that the codimension of a finite union of varieties is the minimum of their codimensions. We refer to Theorem 3.1 in [30] for the formal proof.

We are now ready to prove the main result of this section. First, we show how to obtain Proposition 3.2 from the results in [11].