VDOO: A Short, Fast, Post-Quantum Multivariate Digital Signature Scheme

In the context of this endeavor, we summarize our contributions below.

• •

  We review related multivariate signature schemes and provide a comprehensive analysis of their design and performance in Section 2.

• •

  We present Vinegar-Diagonal-Oil-Oil (VDOO), a novel multivariate signature scheme based on unbalanced oil and vinegar in Section 3. Compared to other UOV schemes VDOO boasts three primary benefits: simplicity, efficiency, and security (see Sections 4 and 5). To the best of our knowledge, we are the first to introduce a diagonal layer within the UOV framework, demonstrating that it enhances efficiency without compromising security.

• •

  We establish that VDOO effectively withstands all current attacks and outline the EUF-CMA security of our scheme. Through meticulous parameter selection, our findings reveal that it achieves a remarkably compact smallest signature size of 96 bytes (see Sections 4 and 5), contrasting favorably with NIST-standardized post-quantum signatures (Crystals-dilithium [26], Falcon [32], and SPHINCS+ [4]).

Here we briefly describe a generic construction for multivariate signature schemes. Due to the NP-hardness of inverting a randomly generated quadratic system [38]. However, signers can leverage a specially structured quadratic system to efficiently perform the inversion. This specialized system is commonly referred to as the central map and is typically denoted as $\mathcal{F}=(f_{1},\cdots,f_{m})$, where each $f_{i}$ represents a specifically structured multivariate quadratic polynomial. Signers must conceal this unique structure from third parties to prevent forgery attacks. To achieve this objective, signers employ one or two random invertible linear maps: $\mathcal{S}$ and $\mathcal{T}$. Consequently, the public key is constructed by composing these linear maps along with the central map, denoted as $\mathcal{P}=~{}\mathcal{S}\circ\mathcal{F}\circ\mathcal{T}~{}:~{}\mathbb{F}_{q}^{n}\longrightarrow~{}\mathbb{F}_{q}^{m}$.

The secret key comprises $\mathcal{S},~{}\mathcal{T}$ and $\mathcal{F}$. A hash function, denoted as $\mathcal{H}:\{0,1\}^{*}\longrightarrow\mathbb{F}_{q}^{m}$, is employed to generate a vector $\mathbf{m}\in\mathbb{F}_{q}^{m}$ from a message $msg\in\{0,1\}^{*}$. The signature generation process unfolds as follows: first, compute $\mathbf{d}\leftarrow\mathcal{S}^{-1}(\mathbf{m})$, then $\mathbf{d}^{\prime}\leftarrow\mathcal{F}^{-1}(\mathbf{d})$, and finally $\mathbf{s}\leftarrow\mathcal{T}^{-1}(\mathbf{d}^{\prime})$. The signer sends the signature $\mathbf{s}$ for the message $msg$ to the verifier. The verifier simply evaluates the polynomial map $\mathcal{P}$ on $\mathbf{s}$ and checks whether it matches the hash of the message, i.e., whether $\mathbf{m}=\mathcal{P}(\mathbf{s})$ holds or not.

The Oil-Vinegar (OV) signature scheme was initially introduced by Patarin [46]. However, due to the Kipnis-Shamir’s [40] invariant subspace attack, this scheme was modified by increasing the number of vinegar variables. This is known as the Unbalanced Oil-Vinegar (UOV) signature scheme [39].

Consider the OV central map, denoted as $\mathcal{F}$. Split all variables of $\mathbf{x}=(x_{1},\cdots,x_{v}$ $,\cdots,x_{n})$ into two buckets: the first bucket has first $v$ variables representing vinegar, and the second bucket contains next $o$ variables representing oil, where $n=v+o$ and $o=m$. To create a multivariate quadratic homogeneous polynomial, combine variables involving vinegar $\times$ vinegar and vinegar $\times$ oil, while excluding all oil $\times$ oil terms.

Notably, if anyone randomly fixes vinegar variables, then the remaining part would be linear in the oil variables. Therefore, the quadratic system reduces to a linear system of $o$ linear equations with $o$ unknowns.

Rainbow is a multi-layer variant of UOV [24]. For simplicity consider a two-layer Rainbow. Suppose $n=v+o_{1}+o_{2}$, where the first $v$ variables are vinegar and the next $o_{1}$ and $o_{2}$ variables are the first and second layer of oil variables respectively. This can be viewed as a UOV map with $v+o_{1}$ variables and $o_{1}$ oil variables and the next layer $v+o_{1}+o_{2}$ variables and $o_{2}$ oil variables.

For a better view of cryptanalysis on Rainbow, Beullens explained the construction of Rainbow via subspaces [10]. Using this description, he derived the simple attack [11]. To elaborate this idea, initially, we define a differential polar form of a polynomial map.

The differential polar map of a polynomial map $\mathcal{P}$ is denoted by $\mathcal{DP}:~{}\mathbb{F}_{q}^{n}\times\mathbb{F}_{q}^{n}~{}\rightarrow~{}\mathbb{F}_{q}^{m}$ and defined as $\mathcal{DP}(\mathbf{x},~{}\mathbf{w})=~{}\mathcal{P}(\textbf{x}+\textbf{w})-~{}\mathcal{P}(\textbf{x})-~{}\mathcal{P}(\textbf{w}).$ Note that, we only consider homogeneous quadratic polynomials, so throughout this paper, $\mathcal{P}(0)=0$.

Trapdoor information. This part describes the trapdoor information of $l$-layer Rainbow. At first, signer chooses a secret chain of nested subspaces: input subspaces $O_{1}\ \supset O_{2}\ \supset\cdots\ \supset O_{l}$ and output subspaces $Q_{1}\ \supset Q_{2}\ \supset\cdots\ \supset Q_{l}=\{0\}$. Using this secret, one can construct a public polynomial map as follows.

• •

  $\mathcal{P}$ maps each $O_{i}$ to $Q_{i}$ and

• •

  for any $\mathbf{x}\in_{U}\mathbb{F}_{q}^{n}$, $\mathcal{DP}_{\mathbf{x}}\colon O_{i}\to Q_{i-1}\,$ is a linear map (see Figure 2)

Inversion. In this methodology, the goal is to compute $\mathbf{x}\in\mathbb{F}_{q}^{n}$ from given $\mathbf{y}\in\mathbb{F}_{q}^{m}$ such that $\mathbf{y}=\mathcal{P}(\mathbf{x})$. The knowledge of nested sequences of input and output subspaces is used in this computation. At first glance, for $l$-layer Rainbow, the value of the unknown $\mathbf{x}$ can be represented as $\mathbf{v}+\mathbf{o_{1}}+\cdots+\mathbf{o}_{l}$ where all of the $\mathbf{o_{i}}\in O_{i}$. Fix $\mathbf{v}\in_{U}\mathbb{F}_{q}^{n}$. Then $\mathcal{P}$ is used in conjunction with the $i$th-layer’s output subspace $Q_{i}$ to calculate $\mathbf{o}_{i}$. For the sake of clarity, let’s define the quotient space $\overline{O}_{i}:=O_{i}/O_{i+1}$.

Using the knowledge of sequences of subspaces, the goal is to find $\textbf{o}_{i}$ for all $i$. This will lead to computing the preimage of any element from $\mathbb{F}_{q}^{n}$. For computing $\overline{\mathbf{o}}_{i}\in\overline{O}_{i}$, use the following relation (note that, from definition, $\mathcal{P}\ (\overline{\mathbf{o}}_{i})=0$),

Earlier v is fixed, so the quadratic system reduces to a linear system. The number of constraints and variables are the same for the linear system. This implies that a unique solution can be obtained with probability $(1-\frac{1}{q})$. Repeatedly running this procedure, one can compute all $\textbf{o}_{i}$, which implies that preimage $\mathbf{x}$ will be computed.

In 2022, Beullens [11] reduced the security level of Rainbow. He showed for small $n-m$, recovering all subspaces is significantly efficient. Also, the small finite field size accelerates the attack.

The NIST additional signature submission call [20] received a total of eleven multivariate signature schemes e.g. Mayo [12], QR-UOV [34], TUOV [23], etc. Most of them are based on the old unbalanced Oil-Vinegar structure. For example, Mayo [12] employed a UOV structure along with a new whipped-up MQ (WMQ) approach. QR-UOV is another variant of UOV where the public key is represented by block matrices, with each element corresponding to an element in a quotient ring [34]. Also, in 2022, a new proposal, called IPRainbow [17] was made by perturbing the central polynomials of the second layer by $s$ variables. This change although decreases the attack probability by $1/q^{s}$, the running time significantly increases due to the usage of Gröbner basis technique for inversion.

Here, we describe other approaches used in the cryptanalysis of multivariate signatures apart from the direct solution of MQ equations.

1. 1.

   Min-rank. Let $M_{1},\;M_{2},\;\cdots,\;M_{k}\;\in\;\mathbb{F}_{q}^{n\times m}$ be the given matrices and $r\in\mathbb{N}$, find a non-trivial linear combination (with $m_{1},m_{2},\cdots,m_{k}\in\mathbb{F}_{q}$) so that $\mbox{rank }(\,\sum_{i=1}^{k}\,m_{i}M_{i}\,)\leq\;r.$ This problem is called the min-rank problem and has been shown to be NP-hard [16]. The min-rank problem appeared as a cryptanalytic tool in multivariate cryptography [41, 30, 6, 10]. This attack helps to find a linear combination of public matrices which sums up to a low-rank matrix.

2. 2.

   EIP. Find an equivalent composition of $\mathcal{P}=\mathcal{S^{\prime}}\circ\mathcal{F^{\prime}}\circ\mathcal{T^{\prime}}$, where $\mathcal{S^{\prime}}\mbox{ and }\mathcal{T^{\prime}}$ are equivalent affine maps, and $\mathcal{F^{\prime}}$ is an equivalent central map. The above problem is the Extended Isomorphism of Polynomials (EIP) problem. No such hardness classification is known (though it subsumes graph isomorphism problem [1, 2]), but for some instances, polynomial time algorithms exist [40].

To construct polynomial maps we need to define parameters associated with this. In this phase algorithm takes input the security parameter $\lambda$ and output the parameter tuple, that is $\mathsf{params}=(\;q,\ v,~{}d,~{}o_{1},~{}o_{2}~{})\leftarrow\mathsf{VDOOSetUp}(1^{\lambda})$. Here,

• •

  Finite field $\mathbb{F}_{q}$ which has $q$ elements.

• •

  Positive integers $v,\ d,\ o_{1},$ and $o_{2}$, where $v$ denotes the number of vinegar variables, $d$ is the number of diagonal variables, $o_{1}$ and $o_{2}$ stands for the number of first and second layer oil variables respectively. Therefore, total number of variables is $n=v+d+o_{1}+o_{2}$, and number of equations is $m=d+o_{1}+o_{2}$.

Construction of central polynomial map $\mathcal{F}:\mathbb{F}_{q}^{n}\to\mathbb{F}_{q}^{m}$ plays an important role in the multivariate signature schemes. To the best of our knowledge, we are the first to propose a central polynomial map that involves vinegar, diagonal, and oil variables in a three-layer construction.

• •

  Diagonal Layer. Here, we explain the structure of any central polynomial $f_{k}$ for the diagonal layer $k\in[v+1:v+d]$. Each $f_{k}$ is defined as follows.

  $$f_{k-v}(x_{1},x_{2},\cdots,x_{n})=\sum_{i=1}^{k-1}\alpha_{i,k}^{(k)}x_{i}x_{k}+\sum_{i,j=1,i\leq j}^{k-1}\beta_{i,j}^{(k)}x_{i}x_{j}$$

  Each coefficient $\alpha_{ij}^{(k)}$, and $\beta_{ij}^{(k)}\in_{U}\mathbb{F}_{q}$. The subroutine $\mathsf{DiagPoly}(q,\;k)$ is used to generate such central polynomial $f_{k}$ in the diagonal layer.

• •

  First Oil Layer. In this oil layer, we use $v+d$ variables as vinegar variables and next $o_{1}$ variables as oil variables. All these variables help us to construct $o_{1}$ homogeneous quadratic polynomials of the following form.

  $$f_{k-v}(x_{1},x_{2},\cdots,x_{n})~{}=~{}\sum_{i=1}^{v+d}~{}\sum_{j=1}^{v+d}~{}\alpha_{ij}^{(k)}~{}x_{i}x_{j}+\sum_{i=1}^{v+d}~{}\sum_{j=v+d+1}^{v+d+o_{1}}~{}\beta_{ij}^{(k)}~{}x_{i}x_{j}$$

  where $k\in[v+d+1:v+d+o_{1}]$, $\alpha_{ij}^{(k)}$, and $\beta_{ij}^{(k)}\in_{U}\mathbb{F}_{q}$.

• •

  Second Oil Layer. The topmost oil layer has $v+d+o_{1}$ vinegar and $o_{2}$ oil variables. That means, it has $o_{2}$ quadratic equations. Those equations are of the form

  $$f_{k-v}(x_{1},x_{2},\cdots,x_{n})~{}=~{}\sum_{i=1}^{v+d+o_{1}}~{}\sum_{j=1}^{v+d+o_{1}}\alpha_{ij}^{(k)}~{}x_{i}x_{j}~{}+\sum_{i=1}^{v+d+o_{1}}~{}\sum_{j=v+d+o_{1}+1}^{v+d+o_{1}+o_{2}}~{}\beta_{ij}^{(k)}x_{i}x_{j},$$

  where $k\in[v+d+o_{1}+1:v+d+o_{1}+o_{2}=n]$ and $\alpha_{ij}^{(k)}$, and $\beta_{ij}^{(k)}\in_{U}\mathbb{F}_{q}$. We denote this as $\mathsf{OVPoly}(q,\ v,\ o)$ to generate a oil-vinegar central polynomial (according to 2.2) which has $v$ vinegar variables and $o$ oil variables.

Here, Algorithm 1, uses $\mathsf{OVPoly}$ and $\mathsf{DiagPoly}$ to generate a VDOO central map $\mathcal{F}$.

Inversion. The main computational bottleneck of UOV-based constructions is the inversion of the central polynomial. It requires Gaussian elimination which runs in $O(N^{3})$. However, in our scenario inversion of the diagonal polynomials is straightforward as there is only one unknown variable. Nevertheless, the inversion of OV polynomials in the remaining two layers each needs a Gaussian elimination. Therefore, inverting VDOO central polynomial map needs two Gaussian elimination only. This is shown in Algorithm 2. Following two algorithms help to compute the inverse of the VDOO central polynomial.

$\mathsf{ST}$

The subroutine Substitution or $\mathsf{ST}$ converts a bunch of oil-vinegar polynomials to a bunch of linear polynomials consists of oil variables by fixing the vinegar variables. That means, $\mathsf{ST}$ substitutes vinegar variables $x_{1},\cdots,x_{v}$ by random values (in $\mathbb{F}_{q}$) in the bunch of $o$ oil-vinegar polynomials $(f_{i})_{i=1}^{o}$ and converts it to a bunch of linear polynomials of $o$ oil variables $(\tilde{f}_{i})_{i=1}^{o}$.

$\mathsf{GE}$

The $\mathsf{GE}_{(q,l)}$ denotes Gaussian elimination for $l$ unknowns over the linear system of equations $(~{}\tilde{f}_{i}=y_{i}~{})_{i=1}^{l}$ over $\mathbb{F}_{q}$. It returns a failure when the rank of the matrix representing the linear system is less than $l$.

The $\mathsf{VDOOKeyGen}$ in Algorithm 3 generates two random invertible affine maps $\mathcal{S}:\mathbb{F}_{q}^{m}\to\mathbb{F}_{q}^{m}$ and $\mathcal{T}:\mathbb{F}_{q}^{n}\to\mathbb{F}_{q}^{n}$ along with the VDOO-central map $\mathcal{F}:\mathbb{F}_{q}^{n}\to\mathbb{F}_{q}^{m}$. Here, secret/signing key is $\mathcal{S},~{}\mathcal{F}$, and $\mathcal{T}$ and public/verification key is the composition map $\mathcal{P}$, where $\mathcal{P}=\mathcal{S}\circ\mathcal{F}\circ\mathcal{T}:\mathbb{F}_{q}^{n}\to\mathbb{F}_{q}^{m}$. Note that, the individual information of secret maps allows user to compute the inverse of $\mathcal{P}$ efficiently. We denote $S\leftarrow\mathsf{randomMatrix}\;(q,\;m,\;seed)$ to generate a random $m\times m$ matrix over $\mathbb{F}_{q}$ from a $seed$, $\mathsf{invMat}\;(q,\;m,\;S)$ helps to compute the inverse of a $m\times m$ matrix $S$ over $\mathbb{F}_{q}$, and $\mathsf{Affine}(S,\mathbf{a})$ computes $\mathcal{S}\leftarrow S\cdot\mathbf{x}+\mathbf{a}.$

Similar to the other OV based constructions [12, 23, 39, 24], we use the hash-and-sign paradigm for our signature algorithm as shown in Algorithm 4. We use a hash function $\mathcal{H}:\{0,1\}^{*}\to\mathbb{F}_{q}^{m}$. Signer knows each polynomial map, so it can compute the inverse of each map i.e. $\mathcal{S}^{-1}$, $\mathcal{F}^{-1}$, and $\mathcal{T}^{-1}$. If $\mathsf{GE}$ reports a failure during the computation of $\mathcal{F}^{-1}$, we restart the process by regenerating the salt and repeating the entire procedure. Finally, the signature is computed as $\mathcal{P}^{-1}(\mathcal{H}(\mathcal{H}(msg)||salt))$.

Our verification procedure is simple. It needs a polynomial evaluation of $\mathcal{P}$, requiring just $O(N^{3})$ field operations. Compute $\mathbf{d}^{\prime}=\mathcal{P}(\mathbf{s})$ from public key $\mathcal{P}$ and signature $\sigma=(\mathbf{s},\;salt)$ The signatures is accepted if $\mathbf{d}^{\prime}=\mathcal{H}(\mathcal{H}(msg)~{}||~{}salt)$ holds, else rejected.

Our VDOO contains one diagonal layer and two UOV layers. The size of the private key is determined first, followed by the size of the public key.

• •

  Size of the central map $\mathcal{F}$ for a diagonal layer having $d$-diagonal polynomials is $\sum_{i=1}^{d}\left(\dfrac{v_{i}(v_{i}+1)}{2}+v_{i}\right)$ field elements. The first diagonal layer has $v_{1}=n-m$ vinegar variables. In any diagonal layer, a central polynomial $f_{i}$ has $v_{i}$ vinegar variables and $f_{i+1}$-th polynomial has $v_{i+1}=v_{i}+1$ vinegar variables.

• •

  Size of the central map $\mathcal{F}$ for a UOV layer is around $o\times\left(\dfrac{v(v+1)}{2}+ov\right)$ field elements. Such UOV layer has $v$ vinegar variables and $o$ oil variables.

The sizes of the two affine transformations are as follows: for $\mathcal{S}$ we need $m(m+1)$, while for $\mathcal{T}$ we need $n(n+1)$, field elements. These maps can be generated using a random seed.

Now we are interested in computing the size of the public key of standard VDOO. Each $n$-variate quadratic polynomial requires $\frac{(n+1)(n+2)}{2}$ field elements. Therefore, the size of the public key is $m\frac{(n+1)(n+2)}{2}$. Further optimization of public key is possible [48, 49]. It optimized the public key size from $O(mn^{2}\log q)$ to $O(m^{3}\log q)$.

Our scheme can be explained through Beullens’s subspace descriptions [10]. This description is useful to understand the cryptanalysis of VDOO. In this case, we have $d+2$ input and output subspaces. These sequences of nested subspaces are as follows.

• •

  Input subspaces $\mathbb{F}_{q}^{n}\supset D_{1}\;\supset\;D_{2}\;\supset\;\cdots\;\supset\;D_{d}\;\supset\;O_{1}\;\supset\;O_{2}\,.$

• •

  Output subspaces $\mathbb{F}_{q}^{m}\supset Q_{1,1}\supset Q_{1,2}\supset\cdots\supset Q_{1,d}~{}\supset Q_{2}\supset\,Q_{3}=~{}\{0\}\,.$

In the Figure 3 (single arrow denotes $\mathcal{P}$ and bold arrow denotes $\mathcal{DP}(\mathbf{x},\cdot)$), these following relations will hold: $\dim(D_{i})=\dim(D_{i+1})+1$ and $\dim(Q_{1,i})=\;\dim(Q_{1,i+1})+1$ for $1\leq i<d$. Also, $\dim(D_{1})=m$, $\dim(D_{i})=\dim(Q_{1,i-1})$ for $1<i\leq d$. In addition, $\dim(O_{1})=\dim(Q_{1,d})=o_{1}+o_{2}$, $\dim(O_{2})=\dim(Q_{2})=o_{2}$.

The signer first fixes $\textbf{v}\in_{U}\mathbb{F}_{q}^{n}$. Since $\dim(\tilde{D}_{i})=\dim(D_{i})-\dim(D_{i+1})=1$, so for diagonal layer computing $\textbf{d}_{1}\ ,\cdots\ ,\textbf{d}_{d}$ is very easy. Once these vectors are found, then update $\textbf{v}\leftarrow\textbf{v}+\textbf{d}_{1}+\cdots+\textbf{d}_{d}$. Now, signer needs to solve for $\tilde{\textbf{o}}_{1}\in\tilde{O_{1}}(=O_{1}/O_{2})$, so that the following relation holds. Note that, $\dim(\tilde{O_{1}})=o_{1}$.

We know that the above equation is a linear system of $o_{1}$ variables and $o_{1}$ equations. With the probability $(1-1/q)$, the signer will able to compute $\mathbf{o}_{1}$. Then signer again updates $\mathbf{v}\leftarrow\mathbf{v}+\mathbf{o}_{1}$ and follow a similar strategy to find $\textbf{o}_{2}\in O_{2}$. Thus the signer can finally compute the pre-image of $\mathbf{t}$.

The direct attack is the fundamental methodology for forging any multivariate signature scheme. To counterfeit a VDOO signature, an attacker aims to solve an underdetermined system with $n$ variables and $m$ homogeneous equations ($n>m$), to find $\mathbf{s}$ such that $\mathcal{P}(\mathbf{s})=\mathbf{t}$. The basic approach involves converting this underdetermined system into a determined one by fixing $n-m$ variables. Subsequently, quadratic system-solving techniques like the Wiedemann XL algorithm [58, 21] or Gröbner basis methods such as F4 or F5 [27, 28] are applied. Another approach named hybrid approach [9] involves guessing $k$ variables prior to solving the system. The time complexity of this attack, using the approach outlined in [9], is expressed in terms of field multiplications as:

Here, $k$ denotes the number of variables fixed during the algorithm, and $d$ represents the smallest integer for which the coefficient of $t^{d}$ in the series $\frac{(1-t^{2})^{m}}{(1-t)^{m-k}}$ is non-positive.

In 2022, Beullens proposed the simple attack against Rainbow [24]. For Rainbow, this highly effective attack reduces $n$-unknown and $m$-constraints in the quadratic system to $n-m$-unknown and $m$-constraints. Now an attacker can apply the same methodology on VDOO to recover the secret key. Recall from Figure. 3, $\mathcal{P}$ is the public polynomial map, and sequences of nested input and output subspaces are,

• •

  Input subspaces $\mathbb{F}_{q}^{n}\;\supset D_{1}\;\supset D_{2}\;\supset\;\cdots\;\supset D_{d}\;\supset O_{1}\;\supset\;O_{2}\,.$

• •

  Output subspaces $\mathbb{F}_{q}^{m}\;\supset~{}Q_{1,1}\supset\;Q_{1,2}\;\supset\;\cdots\supset\;Q_{1,d}\;\supset Q_{2}\;\supset\;Q_{3}\;=\;\{0\}\,.$

The main crux of the simple attack lies in finding a vector within $O_{2}$ (as depicted in Figure. 3). To achieve this, the attacker must solve a quadratic system with $n-m$ unknowns and $m$ constraints using the XL algorithm. This computational step constitutes the most significant component of the entire attack. Here is a step-by-step outline detailing the cryptanalysis of our scheme using the simple attack.

Input:

Public polynomial map $\mathcal{P}$.

Output:

Recover sequences of subspaces.

Find a vector $\textbf{o}\in O_{2}$:

Choose $\textbf{v}\in_{U}\mathbb{F}_{q}^{n}$. Then from Figure. 3, $\mathcal{DP}_{\textbf{v}}~{}:~{}\mathbb{F}_{q}^{n}\rightarrow\mathbb{F}_{q}^{m}$ is a linear map, in particular it maps $O_{2}$ to $Q_{2}$. The attacker uses this linear relation to reduce the number of unknowns present in the quadratic system. Therefore, to find a vector, an attacker should solve the following system.

	$\displaystyle\mathcal{DP}_{\textbf{v}}(\textbf{o})$	$\displaystyle=0$
	$\displaystyle\mathcal{P}(\textbf{o})$	$\displaystyle=0$

With probability $\approx 1/q$, the attacker successfully guesses a vector in $O_{2}$. Later, the attacker deploys the XL algorithm to solve the quadratic system of $n-m$-unknowns and $m$-constraints. Thus attacker recovers $\mathbf{o}$.

Recover $Q_{2}$:

Attacker will retrieve $Q_{2}$ using the information $\textbf{o}\in O_{2}$. Note that, $\mathcal{DP}_{\mathbf{o}}:O_{2}\rightarrow Q_{2}$ is a linear map. Therefore,

$$\mbox{Span}\{~{}\mathcal{DP}_{\textbf{o}}(\textbf{e}_{1}),~{}\cdots,~{}\mathcal{DP}_{\textbf{o}}(\textbf{e}_{n})~{}\}\subseteq Q_{1}$$

for some linearly independent vectors $\mathbf{e}_{i}$. For enough such $\textbf{e}_{i}$’s equality will hold.

Recover $O_{2}$:

To recover $O_{2}$, solve the following system of linear equations. Because with high probability kernel of $\mathcal{DP}_{\mathbf{o}}$ matches with $O_{2}$.

	$\displaystyle\mathcal{DP}_{\textbf{o}}(\textbf{e}_{1})$	$\displaystyle\equiv 0\mod~{}Q_{2}$
	$\displaystyle\mathcal{DP}_{\textbf{o}}(\textbf{e}_{2})$	$\displaystyle\equiv 0\mod~{}Q_{2}$
	$\displaystyle\vdots$
	$\displaystyle\mathcal{DP}_{\textbf{o}}(\textbf{e}_{n})$	$\displaystyle\equiv 0\mod~{}Q_{2}$

Recover a vector $\mathbf{o}^{\prime}\in O_{1}$:

Now the quadratic system $\mathcal{P}$ reduces to $m^{\prime}=m-o_{2}$ equations and $n^{\prime}=n-o_{2}$ variables. To recover $O_{1}$, the goal of the attacker is to find a vector in $\mathbf{o}\in O_{1}$. Again attacker will guess a vector $\mathbf{v}^{\prime}\in\mathbb{F}_{q}^{n^{\prime}}$. Like above, a similar argument shows that $\mathcal{DP}_{\mathbf{v}^{\prime}}:~{}O_{1}\rightarrow Q_{1,1}$ is a linear map and the attacker tries to solve the following systems mod $Q_{2}$.

	$\displaystyle\mathcal{DP}_{\textbf{v}^{\prime}}(\textbf{o}^{\prime})$	$\displaystyle=0\mod~{}Q_{2}$
	$\displaystyle\mathcal{P}(\textbf{o}^{\prime})$	$\displaystyle=0\mod~{}Q_{2}$

The attacker runs the XL algorithm to solve the quadratic system of $n^{\prime}-m^{\prime}$-unknowns and $m^{\prime}$-constraints.

Recover $O_{1}$:

Attacks follows same approach as recovering $O_{2}$ to recover $O_{1}$. Here, an attacker solves a system $\mathcal{DP}_{\mathbf{o}^{\prime}}(\mathbf{e}_{i}^{\prime})\equiv 0\mod~{}Q_{1}$ for $i\leq n^{\prime}.$

Recovering vectors from diagonal layer:

The only task that remains is to find all the diagonal vectors. The attacker can apply Wolf et al.’s [59] trick to find all the diagonal vectors in the layer. Here observe that the computation of finding a vector in $O_{2}$, dominates the computation of finding a vector in $O_{1}$.

Attack Complexity. The complexity of the first steps dominates the complexity of other steps involved in this algorithm. Basically, a system of $n$ variables and $m$ non-linear equations reduces to a system of $m$ homogeneous equations with $n-m$ variables. This computation can be performed via XL algorithm and it requires

field operations, where $d$ is the operating degree of the algorithm. It means, $d$-is the smallest positive integer so that the coefficient of $t^{d}$ in the power series $(1-t^{2})^{m}/(1-t)^{n-m}$ is non-positive.

Example for SL-1 parameters. Apply Beullens’ trick to guess a vector in $O_{2}$, which happens with probability $1/q$. Finding one vector on $O_{2}$ asks to solve a quadratic system of $100$-variables 60-unknowns. This computation is the most costly in the entire algorithm. Solving this quadratic system needs $2^{134}$ field operations. The guessing needs $1/q$ search and cost of one $\mathbb{F}_{16}$- multiplication needs 36 gates. Therefore, this parameter set provides approximately at-least 128-bit security.

Rectangular min-rank attack is proposed by Beullens [10]. We first describe the attack against VDOO and then compute the required attack complexity to perform this attack against VDOO. Attacker starts with $n\times m$-rectangular matrices $M_{1},\;M_{2}\;,\cdots,\;M_{n}$ over $\mathbb{F}_{q}$ where each $M_{i}$ is defined as

where $(\mathbf{s}_{i})_{i=1}^{n}$ is a basis of $\mathbb{F}_{q}^{n}$.

Let $\mathbf{o}_{2}\in\mathbb{F}_{q}^{n}$. The bi-linearity of $\mathcal{DP}$ implies