Enumeration of Anti-Invariant Subspaces and Touchard’s
Formula for the Entries of the $q$-Hermite Catalan Matrix

Let $q$ be a prime power, and let $\mathbf{F}_{q}$ denote a finite field of order $q$. For nonnegative integers $n$ and $k$, let ${n\brack k}_{q}$ denote the $q$-binomial coefficient, which is the number of $k$-dimensional subspaces of $\mathbf{F}_{q}^{n}$. Recall that for a linear operator $T\in M_{n}(\mathbf{F}_{q})$, a subspace $W\subset\mathbf{F}_{q}^{n}$ is said to be $T$-invariant if $T(W)\subset W$.

The computation of $X^{T}_{j}$ from the similarity class of $T$ as a polynomial in $q$ is easy [13, Section 2] and has been implemented in SageMath [10]. The formula in Eq. (1) can be recast in the following form, which will be used in the proof.

Anti-invariant subspaces were introduced by Barría and Halmos [3]. A matrix $T\in M_{n}(\mathbf{F}_{q})$ is said to be $\ell$-transitive if each $\ell\times\ell$ matrix over $\mathbf{F}_{q}$ appears as the upper left submatrix of some matrix similar to $T$. Using the fact that $T$ is $\ell$-transitive if and only if it admits an $\ell$-dimensional anti-invariant subspace, they characterized $\ell$-transitive matrices. Sourour [14] determined the maximal dimension of an anti-invariant subspace. Knüppel and Nielsen [8] defined a subspace $W$ to be $k$-fold anti-invariant if

and determined the maximal dimension of a $k$-fold anti-invariant subspace for a linear operator $T$. They also showed [8, Thm. 2.1] that if $d_{1}\leq d_{2}\leq\cdots\leq d_{n}$ denote the degrees of the invariant factors in the Smith normal form of $xI-A$ where $A$ is the matrix $T$ with respect to some basis, then a $k$-fold anti-invariant subspace of dimension $m$ exists if and only if $d_{n-m+1}+\cdots+d_{n}\geq(k+1)m$.

Enumerative versions of such problems can be traced back to a paper of Bender, Coley, Robbins and Rumsey [4]. In the context of pseudorandom number generation, Niederreiter [9] made the following definition: given $T\in M_{md}(\mathbf{F}_{q})$, an $m$-dimensional subspace $W\subset\mathbf{F}_{q}^{md}$ is said to be a $T$-splitting subspace of degree $d$ if

Niederreiter asked for the number of $m$-dimensional $T$-splitting subspaces of degree $d$ when the characteristic polynomial of $T$ is irreducible, unaware that the question had already been answered in [4]. The number of $T$-splitting subspaces in this case is given by

Chen and Tseng [5] reproved this result by developing recurrence relations involving a larger class of combinatorial problems. Their recurrence relations are independent of the matrix $T$, and reduce the enumeration of splitting subspaces to the enumeration of flags of $T$-invariant subspaces. However, the recurrences are very difficult to solve in general.

Aggarwal and Ram [1] used the recurrences of Chen and Tseng to show that when $T$ is regular nilpotent, the number of $T$-splitting subspaces of degree $d$ is given by $q^{m^{2}(d-1)}$. For regular nilpotent $T$, $X_{j}^{T}=1$ for $0\leq j\leq n$, so (1) becomes

Setting $\ell=m$ and $n=2m$, we recover the formula of [1] in the case $d=2$. Later, it came to light that these results in the regular nilpotent case follow from the results in [4].

Viennot’s combinatorial theory of orthogonal polynomials [16] places the moments of an orthogonal polynomial sequence in the first column of an infinite array known as the Catalan matrix. There is a connection between our main theorem and the Catalan matrix associated to the $q$-Hermite orthogonal polynomial sequence as defined by Ismail, Stanton and Viennot [7]. This connection emerges from our work [11, 12] where a more general class of enumerative problems is considered.

When $n=m+\ell$, a subspace $W$ has profile $(m,\ell)$ if $W$ is $m$-dimensional, and $W+TW=\mathbf{F}_{q}^{n}$. Let $T^{*}$ denote the transpose of $T$. For a subspace $W\subseteq\mathbf{F}_{q}^{n}$, let $W^{0}$ denote its annihilator in the linear dual of $\mathbf{F}_{q}^{n}$. When $W$ has profile $(m,\ell)$, $\dim(W^{0})=l$. Also $W+TW=\mathbf{F}_{q}^{n}$ if and only if $W^{0}\cap(TW)^{0}=(W+TW)^{0}=\{0\}$. By using the identity

for $S\in M_{n}(\mathbf{F}_{q})$ and each subspace $U\subset\mathbf{F}_{q}^{n}$, it follows that

Therefore, $W$ has profile $(m,\ell)$ if and only if $W^{0}$ is an $\ell$-dimensional $T^{*}$-anti-invariant subspace. Since $T$ is similar to $T^{*}$, we have

Let $[n]$ denote the set consisting of the first $n$ positive integers. Denote by $\Pi(\mu^{\prime})$ the set of partitions of $[n]$ whose block sizes are the parts of the integer partition conjugate to $\mu$. Suppose $T$ is a diagonalizable matrix with distinct eigenvalues in $\mathbf{F}_{q}$. One of the main results of [12] is a combinatorial formula for $\sigma_{\mu}^{T}$ in terms of a statistic $v$ on set partitions known as the interlacing number:

From (5), we obtain

where $a_{n,n-2\ell}=\sum_{\mathcal{A}}q^{v(\mathcal{A})}$, a sum is over partitions of $[n]$ with $\ell$ blocks of size $2$ and $n-2\ell$ singleton blocks. In Section 5, we show that $a_{n,n-2\ell}$ coincides with polynomials defined recursively by Touchard [15] in the context of the stamp-folding problem. Touchard [15, Eq. (28)] showed that

When $T$ is a diagonalizable matrix with distinct eigenvalues in $\mathbf{F}_{q}$, $X^{T}_{j}=\binom{n}{j}$. Substituting this into the formula (1) of our main theorem gives

Comparing the expressions in (7) and (9) gives a new, linear-algebraic proof of Touchard’s formula (8).

In this section we prove the existence of a universal formula for the number of anti-invariant subspaces of a given dimension for an arbitrary operator $T$. The main step is Lemma 3, which is a special case of the recurrence of Chen and Tseng [5, Lemma 2.7]. We begin by introducing some notation.

Given $T\in M_{n}(\mathbf{F}_{q})$ and sets $A$ and $B$ of subspaces of $\mathbf{F}_{q}^{n}$, define

For integers $a,b$, we also write $\beta(a,b)$ for the set of $a$-dimensional subspaces $W$ of $\mathbf{F}_{q}^{n}$ such that $W\cap T^{-1}W$ has dimension $b$. The quantity $\gamma(a,b)$ is defined analogously. For example, using identity (4), $\beta(a,a)$ denotes the set of $a$-dimensional $T$-invariant subspaces, whereas $\beta(a,0)$ denotes the set of $a$-dimensional $T$-anti-invariant subspaces. To explicitly specify the linear operator $T$, we also write $\beta^{T}(A,B)$ and $\gamma^{T}(A,B)$.

The following corollary shows the existence of a universal formula for the number of anti-invariant subspaces of a given dimension.

We set up a system of linear equations which completely determine the polynomials $p_{j}(t)$ $(0\leq j\leq\ell)$ in Corollary 5 by constructing, for each prime power $q$, a sequence of matrices $T_{0}(q),\dotsc,T_{\ell}(q)$ such that the following conditions are satisfied.

1. (1)

   For each $0\leq i,j\leq\ell$, there exists a polynomial $X_{ij}(t)$ such that $X^{T_{i}(q)}_{j}=X_{ij}(q)$ for all prime powers $q$.

2. (2)

   The determinant of the matrix $X(t):=(X_{ij}(t))_{0\leq i,j\leq\ell}$ is a non-zero element of $\mathbf{Z}[t]$.

3. (3)

   The identity (1) holds for $T_{i}(q)$ for $i=0,\dotsc,\ell$ and all prime powers $q$.

In effect, we have the following result.

We now proceed with the construction of the matrices $T_{i}(q)(0\leq j\leq\ell)$ above. For each $q$, let $T_{0}(q)$ be any matrix in $M_{n}(\mathbf{F}_{q})$ with irreducible characteristic polynomial. For $i=1,\dotsc,\ell$, take $T_{i}(q)$ to be the $n\times n$ matrix with block decomposition

where $T^{\prime}_{i}(q)$ is a nonsingular $(\ell-i)\times(\ell-i)$ matrix with irreducible characteristic polynomial. We have

Therefore we can take

It remains to show that the values of $p_{j}(q)$ given by (14) are solutions to (13) for $1\leq i\leq\ell$. We begin by showing that the left hand side of (13) vanishes in these cases.

In view of Lemma 9, in order to establish (13) for $T_{i}(q)$ $(1\leq i\leq\ell)$, it suffices to prove the identity

Let

Then by Eq. (15),

Therefore, it suffices to show that, for $1\leq i\leq\ell$,

The proof uses techniques from the theory of $q$-hypergeometric series, and appears in Section 4.

The non-singularity of $X(t)=(X_{ij}(t))_{0\leq i,j\leq\ell}$ is proved by using inequalities satisfied by the degrees of its entries. We recall [13, Lemma 4.4].

This completes all steps in the proof of Theorem 6 except for the identity (17).

In this section, we prove the identity (17) encountered in the proof of Theorem 6 by using a Heine transformation for $q$-hypergeometric series. Accordingly, define

Let $S_{r}(n,\ell,i)$ denote the sum obtained in (17) by replacing $Y(n,\ell,i,j)$ by $Y_{r}(n,\ell,i,j)$. We need to show that $S_{1}+S_{2}-S_{3}-S_{4}=0$. We will show that $S_{1}=S_{4}$ while $S_{2}=S_{3}$ by expressing the sums $S_{i}$ as $q$-hypergeometric series.

Define, as usual, the $q$-Pochhammer symbols

For convenience, we will also use the notation

Heine (see Gasper and Rahman [6]) defined the $q$-hypergeometric series

We will see that Heine’s transformation formula [6, Eq. (III.2)] transforms $S_{1}(n,\ell,i)$ into $S_{4}(n,\ell,i)$ and $S_{2}(n,\ell,i)$ into $S_{3}(n,\ell,i)$; therefore (17) will follow from the lemma.