Semicontinuity of Gauss maps
and the Schottky problem

The Gauss map of a hypersurface in projective space is the rational map that sends any smooth point of the hypersurface to its normal direction in the dual projective space. The analogous notion of Gauss maps for subvarieties of abelian varieties appears already in Andreotti’s proof of the Torelli theorem [2]. In contrast to the case of projective hypersurfaces, the Gauss map for any ample divisor on an abelian variety is generically finite of degree $>1$, and its degree is related to the singularities of the divisor. We show that this degree is lower semicontinuous in families, and we study its jump loci. As an application we get that in the moduli space of principally polarized abelian varieties, the degree of the Gauss map refines the Andreotti-Mayer stratification and answers the Schottky problem as conjectured in [11]. We work over an algebraically closed field $k$ with $\mathrm{char}(k)=0$. In section 7 we obtain similar results for the intersection cohomology of complex varieties with a finite morphism to an abelian variety. In particular, many Andreotti-Mayer loci such as the Schottky locus are determined over the complex numbers already by the topological type of the theta divisor.

For convenience we include in this section a self-contained review of some basic facts about the specialization of Lagrangian cycles, which was introduced in relation with Chern-MacPherson classes [34] and nearby cycles for ${\mathscr{D}}$-modules and perverse sheaves [19]. We work in a relative setting over a smooth curve $S$. The family of our ambient spaces is given by a smooth dominant morphism of varieties $f:W\to S$ where $\dim(W)=n+1$. Let $X\subset W$ be a reduced closed subvariety. The relative smooth locus

is nonempty iff $\dim f(X)>0$, in which case $X\to S$ is flat and $\mathrm{Sm}(X/S)\subset X$ is an open dense subset because $\mathrm{char}(k)=0$. Any $x\in\mathrm{Sm}(X/S)$ is a smooth point of the fiber $X_{s}=f^{-1}(s)\cap X$ over the image point $s=f(x)$. Hence inside the total space

of the relative cotangent bundle, we define the relative conormal variety to $X$ as the closure

Indeed, both sides are irreducible closed subvarieties of $T^{\vee}(W)|_{X}$. For the right hand side this holds by definition, for the left hand side it follows from the fact that $\Lambda_{X/S}\subset T^{\vee}(X/S)$ is an irreducible closed subvariety and $T^{\vee}W\to T^{\vee}(X/S)$ is a fibration with irreducible fibers. So it suffices to show that both sides agree over some open dense $U\subset X$. We can assume $X$ is flat over $S$ and take $U=\mathrm{Sm}(X/S)$, in which case the claim becomes obvious.

Relative conormal varieties can be seen as families of conormal varieties. In what follows we denote by

the free abelian group on relative conormal varieties to closed subvarieties $X\subset W$ that are flat over $S$. By the specialization of $\Lambda\in{\mathscr{L}}(W/S)$ at $s\in S(k)$ we mean the cycle

which underlies the schematic fiber of the morphism $\Lambda_{X/S}\to S$ at $s$. This is again a conic Lagrangian cycle by the following classical result, see [18, prop. (a), p. 179] or in an analytic setup [28, sect. 1.2]:

We have the following consequence of flatness:

Let $f:W\to S$ be a smooth dominant morphism from a smooth variety to a smooth curve as above. For any $S$-flat subvariety $X\subset W$ and $s\in S(k)$ we have seen that ${\mathrm{sp}}_{s}(\Lambda_{X/S})-\Lambda_{X_{s}}\geq 0,$ where the inequality means that the cycle on the left hand side is effective or zero. It is natural to ask for which $s\in S(k)$ the above inequality is strict. In the notation of lemma 2.3 this happens iff $m_{Z}>0$ for some $Z\subseteq\mathrm{Sing}(X_{s})$. The following provides a sufficient criterion for this to happen for families of divisors:

We divide the proof in several steps. Most of the argument works in arbitrary codimension $d$, so for the moment we do not yet assume $d=1$. It will be enough to prove the claim over some open dense subset of $W$. Fixing a general point $p\in Z(k)$ and working locally near that point, we can assume that

• •

  $Z=\mathrm{Sing}(X/S)$ (equality as a scheme),

• •

  $T^{\vee}(W/S)\simeq W\times V$ is the trivial bundle with fiber $V=T^{\vee}_{p}(W_{s})$,

• •

  $X\subset W$ is cut out by a regular sequence $f_{1},\dots,f_{d}\in H^{0}(W,\mathcal{O}_{W})$.

By the first item each $x\in(X\setminus Z)(k)$ is a smooth point of $X_{t}$ for $t=f(x)$. Fixing a trivialization as in the second item, we can furthermore identify the conormal space to $X_{t}\subset W_{t}$ at $x$ with a subspace in $V=T^{\vee}_{x}(W_{t})$ of codimension $d$. Consider the relative Gauss map

which sends each point to the corresponding conormal space. This is a rational map whose locus of indeterminacy is precisely $Z$. Let $\gamma_{X}:\hat{X}\to\operatorname{Gr}(d,V)$ denote its resolution of indeterminacy which is obtained by blowing up the base locus $Z\subset X$ as in [17, sect. 4.4]:

We want to control the image of the exceptional divisor $E_{X}=\pi_{X}^{-1}(Z)\subset\hat{X}$ under the map

In particular, one may look at the scheme-theoretic fiber of $\hat{X}$ over $s\in S(k)$ to compute the multiplicities in ${\mathrm{sp}}_{s}(\Lambda_{X/S})$. For $d>1$ the situation becomes more complicated, so in what follows we restrict ourselves to set-theoretic arguments. To pass back to conormal varieties we look at the projection $\operatorname{Fl}(d,1,V)\longrightarrow\operatorname{Gr}(d,V)$ from the partial flag variety. This projection is a smooth equidimensional morphism of relative dimension $d-1$. On the fiber product $Y=\hat{X}\times_{\operatorname{Gr}(d,V)}\operatorname{Fl}(d,1,V)$ consider the morphism

where $\pi_{Y}:Y\to\hat{X}\to X$ and $\gamma_{Y}:Y\to\operatorname{Fl}(d,1,V)\to\mathbb{P}V$ are the natural composite maps. Taking the preimage of the previous exceptional divisor $E_{X}\subset\hat{X}$ we get the following lower bound on the specialization:

Note that by lemma 3.2 the morphism $Y\to X\times\operatorname{Fl}(d,1,V)$ is a closed immersion and hence generically finite onto its image. So corollary 3.5 finishes the proof of proposition 3.1 since for codimension $d=1$ the morphism $\operatorname{Fl}(d,1,V)\to\mathbb{P}V$ is an isomorphism. This is the only point where we use $d=1$. For higher codimension the morphism $\alpha_{Y}:E_{Y}\to X\times\mathbb{P}V$ is not always a closed embedding, as the following example shows, but it may still be generically finite onto its image as needed for corollary 3.5:

Before we apply the above to Gauss maps, let us recall some generalities about families of rational maps. Let $f\colon X\to S$ be a faithfully flat morphism of varieties of relative dimension $n$ with irreducible fibers. Let ${\mathscr{L}}\in{\mathit{Pic}}(X)$ be a line bundle and ${\mathscr{V}}$ a rank $n+1$ vector subbundle of $f_{*}{\mathscr{L}}$. Then for each point $s\in S(k)$ we get a linear series ${\mathscr{V}}_{s}\subset H^{0}(X_{s},{\mathscr{L}}_{s})$ and we denote by $\phi_{s}\colon X_{s}\dashrightarrow\mathbb{P}{\mathscr{V}}_{s}$ the corresponding rational map. Note that since the source and the target of this map have the same dimension, the map is a generically finite cover iff it is dominant. So we consider the degree map

where we put $\deg(\phi_{s})=0$ if $\phi_{s}$ is not dominant.

However, in general the degree is neither upper nor lower semi-continuous, as the following variation of [11, ex. 2.3] shows:

We now apply the above to an abelian scheme $f:A\to S$, so in this section we take $W=A$. Let $X\subset A$ be a closed subvariety which is flat over $S$. For $s\in S(k)$ we have the Gauss map

where $V=H^{0}(A_{s},\Omega_{A_{s}}^{1})$. If this map is dominant, then for dimension reasons it is a generically finite cover and we denote by $\deg(\gamma_{X_{s}})$ its generic degree. If the map is not dominant we put $\deg(\gamma_{X_{s}})=0$; this happens iff the subvariety $X_{s}\subset A_{s}$ arises by pull-back from some smaller dimensional abelian quotient variety [37, th. 1], which by [1, th. 3] happens iff $X_{s}$ is not of general type. The degree of the above Gauss map is related to the degree of conormal varieties as follows:

In particular $\deg(\Lambda_{X_{s}})\geq 0$. Together with the preservation of the total degree under Lagrangian specialization this leads to our first semicontinuity result:

To see where the function in the previous corollary actually jumps, recall that a subvariety $Z\subset A_{s}$ has Gauss degree $\deg(\gamma_{Z})=0$ iff it is not of general type. Thus we obtain the following sufficient jumping criterion:

The moduli space ${\mathscr{A}}_{g}$ of principally polarized abelian varieties of dimension $g$ over the field $k$ admits the finite filtration $\cdots\subseteq{\mathscr{G}}_{d}\subseteq{\mathscr{G}}_{d-1}\subset\cdots\subseteq{\mathscr{G}}_{g!}={\mathscr{A}}_{g}$ by the Gauss loci