Combinatorial Regularity for Relatively Perfect Discrete Morse Gradient Vector Fields of ReLU Neural Networks

We follow a framework for the polyhedral decomposition of the input space of a ReLU neural network $F$ by the canonical polyhedral complex which we will denote as $\mathcal{C}(F)$, as introduced in [7] and defined in Definition 14. This framework is expanded in [8], introducing a PL Morse characterization, and in [18], introducing the combinatorial characterization of the polyhedral decomposition by sign sequences.

Unfortunately, computing the canonical polyhedral complex and topological properties of its level and sublevel sets is expensive and can only be done for small neural networks, in part because $\mathcal{C}(F)$ has exponentially many vertices in the input dimension of $F$. It is this drawback that is the main motivation for the construction of a discrete function associated to $F$; such a function will retain the topological information desired, but should ease issues of hight computational complexity.

We build on existing literature by providing a translation from the PL Morse characterization from [8] to a discrete Morse gradient vector field by exploiting both the combinatorics from [18] and a technique for generating relatively perfect discrete gradient vector fields from [6]. The main result of this paper (Theorem 3) provides a canonical way to directly construct a discrete Morse function (defined on the input space) which captures the same topological information as the existing PL Morse function. This paper is intended as an intermediary which should allow further computational tools to be developed.

While not the primary goal of this paper, a secondary contribution which we hope to highlight is the development of additional tools for translating from piecewise linear functions on non-simplicial complexes to discrete Morse gradient vector fields. To our knowledge, all current theory relating PL Morse functions to discrete Morse functions on the same complexes is detailed in [6]. We also provide realizability results for certain shallow networks (Theorem 2) - namely, we are able to characterize the possible homotopy types of the descision boundary for a $(n,n+1,1)$ generic PL Morse ReLU neural network $F$ by restricting the number and possible indices of the critical points of $F$.

In Section 2, we review the mathematical tools we are using. In Section 3, we review the construction of the canonical polyhedral complex and share some novel realizability results for shallow neural networks. Section 4 contains our main result (Theorem 3). This theorem provides a translation from a PL Morse ReLU neural network $F$ to a relatively perfect discrete gradient vector field on $\mathcal{C}(F)$. In Section 5 we discuss some of the issues surrounding effective computational implementation, and in Section 6, we provide concluding remarks.

The terms below are defined specifically for polyhedra and polyhedral complexes. We use definitions from [22] and [6]. These definitions for polyhedral complexes are motivated by similar constructions for simplicial complexes. The additional generalizations to local versions of these terms are necessary for working in the polyhedral setting. We will use the local star and link of a vertex to provide combinatorial regularity which we will exploit in the algorithms we describe in Section 4 and Section 5.

Next, we create local versions of the following constructions which are well known in simplicial complexes. (1-2) are adapted from [9], (3) is adapted from [6], and (4) is, to our knowledge, new:

Observe that for a given point $p$ and any cone neighborhood $pL$ of $p$, the local star and local lower star of $p$ have a poset structure given by containment and induced by that of the star of $p$; this poset structure is independent of choice of $L$. This justifies calling our construction the local lower star of $p$ relative to $f$.

In the simplicial context, the link can be thought of intuitively as the boundary of the star. We introduce a local version of the link in the polyhedral setting. However, in this polyhedral setting, in contrast to the star, the link and the local link may be combinatorially distinct, even though the star and local star are not.

The local link and local lower link of a point $p$ in $\mathcal{C}$ have a well-defined combinatorial decomposition induced by that of the star of $p$, while the true (combinatorial) link in $\mathcal{C}$ does not, as illustrated in Figures 1 and 2.

In Section 4 we will translate from piecewise linear Morse functions on certain PL manifolds to discrete Morse gradient vector fields on a corresponding polyhedral complex. A useful intermediary, and furthermore a useful tool for visualization, is what we call the $\nabla F$-orientation on the PL manifold’s 1-skeleton.

We primarily consider the case where $F$ is only constant on vertices. For ReLU neural networks, this is not always the case, but is sufficiently common to treat as a distinguished case.

The above lemma implies the following corollary immediately.

We also find the following no-zigzags lemma useful in understanding realizable $\nabla F$ orientations.

Smooth Morse theory describes the classical relationship between the homotopy type of sublevel sets of a smooth function $f:M\to\mathbb{R}$ and what is called the index of its critical points, the number of negative eigenvalues of the Hessian [20]. In non-smooth contexts, alternative tools are needed. One such tool is piecewise linear (PL) Morse theory. No one such theory is generally accepted, but we adapt the following definition due to its ease of use in this context, adapted from [9]:

We say that $f$ is PL Morse if all vertices are regular or critical with index $i$ for some $1\leq i\leq d$. As in smooth Morse theory, the sublevel set topology of a PL function $f$ only changes at PL critical points, and if $f$ is PL Morse, the change in homotopy type at a PL critical point of index $k$ is consistent with attaching a $k$-cell. Furthermore, if $f(v)=c$ and $v$ is the only PL critical point satisfying that condition, the rank of the relative homology $H_{k}(f_{\leq c},f_{\leq c-\epsilon})$ is $1$ and, for $i\neq k$, $H_{i}(f_{\leq c},f_{\leq c-\epsilon})$ have rank zero [8].

In most cases, PL critical points only occur at vertices of the polyhedral complex, but this relies on the function $f$ being nonconstant on positive-dimensional cells. In [8] and [18] it is shown that all ReLU neural networks which are nonconstant on positive-dimensional cells are PL Morse.

ReLU neural networks often have cells of their Canonical Polyhedral Complex (see Definition 14) on which they are constant (which we call flat cells, in line with [8]), and as a result not all ReLU neural networks are PL Morse. While it is a goal of the authors to extend our construction of a discrete Morse function for networks with flat cells (see Section 6 for a brief discussion on this topic), such networks are not in the scope of the current paper. Therefore, we restrict this paper to ReLU neural networks whose only flat cells are vertices.

The strength of discrete Morse theory is that it provides algorithmic tools for computing complete sublevel set topology [5].

Such tools, to our knowledge, have not been developed in generality for PL Morse theory. While the majority of discrete Morse theory has been developed in the context of simplicial complexes and CW complexes, the definitions are applicable in the context of polyhedral complexes with few changes, which we discuss at the end of this section. The main difference between polyhedral complexes and cellular complexes is the presence of unbounded polyhedra, which we address in Section 4.1.

We begin by reviewing standard definitions in discrete Morse theory; interested readers can find more details in [6].

A discrete Morse function has the property that it assigns higher values to higher dimensional simplices, except possibly with one exception locally for each simplex. For a given simplicial complex $K$, there may be many discrete Morse functions; for example, any function which assigns increasing values to simplices with increasing dimension will be discrete Morse, and all simplices will be critical. However, it is usually possible to find a more “efficient” discrete Morse functions in the sense that complex has a smaller amount of critical simplices than overall simplices (see for example [4, 13, 17]).

Sublevel sets of discrete Morse functions are subcomplexes, and provide a way of building the simplical complex in question by adding simplices in order of increasing function value. As in classical Morse theory, the homotopy type of the sublevel set can only change when a critical simplex is added, and the dimension of the critical simplex indicates that homotopy type of the sublevel differs only by the attachment of a cell of the same dimension.

Associated to each discrete Morse function is a discrete gradient vector field. The information contained in the gradient vector field of a discrete Morse function is sufficient to determine the homotopy type of sublevel sets, and gradient vector fields have the benefit of a combinatorial formulation. To define the gradient vector field of a discrete Morse function, we first introduce the definition of a general discrete vector field.

The discrete gradient vector field of a discrete Morse function $f$ on $\mathcal{C}$ is the pairing that arises from $f$ as follows:

1. 1.

   If $\alpha$ is critical, then it is unpaired,

2. 2.

   Otherwise, if there exists a face $\beta$ of $\alpha$ with $f(\beta)\geq f(\alpha)$, then $\alpha$ is paired with $\beta$,

3. 3.

   Otherwise, there is a coface $\gamma$ of $\alpha$ with $f(\alpha)\geq f(\gamma)$, and $\alpha$ is paired with $\gamma$.

It is clear from Definitions 6 and 7 that this collection of pairs of $\mathcal{C}$ is indeed a discrete vector field.

Often, discrete Morse functions are represented only by their gradient vector fields, as opposed to giving the function $f$ itself. Therefore, it is valuable to be able to determine when a discrete vector field represents the gradient of a discrete Morse function. The discrete Morse analogue to gradient flow along a discrete vector field is a $V$-path.

A classical way of determining whether a smooth vector field can represent a gradient is when it lacks circulation. The analogue in the discrete setting to “lacking circulation” is “no non-trivial closed $V$-paths”. The following theorem therefore indicates when a discrete vector field is the gradient of a discrete Morse function.

The critical cells of a discrete gradient vector field $V$ can be thought of as tracking an analog of the topology of the polyhedral complex’s sublevel sets under a particular function which would induce said discrete vector field, as seen below.

Ultimately, we hope the existence of a relatively perfect discrete gradient vector field will enable improved computation of topological features of the level and sublevel sets of a neural network $F$.

Most known relationships beween PL Morse and discrete Morse constructions are implicit. In [2, 12, 21], we see that the construction of discrete gradient vector fields for cubical complexes from function data is well-studied; this is in part due to the regularity of their local combinatorics. For cubulations of compact regions of $\mathbb{R}^{n}$, for example, when simplifying persistence homology computations, there is a discrete Morse function constructed which has an implied analog of a smooth or piecewise linear function on the underlying space.

To our knowledge, there has been little work done to construct a discrete gradient vector field from a PL Morse function in a general setting. In a general setting, [16] compares the PL approximations of a scalar function defined on a triangulated surface to a discrete gradient vector field built using a greedy algorithm - they find that under certain regularity conditions, critical cells in the vector field are adjacent to critical vertices in the PL approximation. Likewise in [6] an algorithm is presented for constructing a discrete gradient vector field which is relatively perfect to a PL Morse function on a simplicial complex which is a combinatorial manifold. However, due to theoretical limitations for algorithms on $n$-spheres for $n\geq 4$, the algorithm applies only to simplicial complexes of dimension $\leq 3$.

As polyhedral complexes have fewer combinatorial restrictions on their structure than simplicial and cubical complexes, a general theory for creating discrete gradient vector fields on an arbitrary polyhedral complex from function values is likely to be similarly intractable.

Fortunately, due to the combinatorial regularity of the canonical polyhedral complex of a ReLU neural network, which we will describe in Section 3, we may follow an approach similar to that in [6] to constructively obtain a relatively perfect discrete gradient vector field when the network has vertices in general position.

For $n\in\mathbb{N}$, define the function $\textrm{ReLU}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ as

The terms fully-connected and feedforward are machine learning terms which indicate that there are no restrictions on each affine function $A_{i}$ and that there are no expectation of identical layer maps (i.e. no recurrence), respectively. We will omit these terms in the rest of this paper, but keep them in the definition to disambiguate for readers with a machine learning background.

Note that $F$ is a piecewise linear function, and that $F$ defines a polyhedral decomposition of its domain, $\mathbb{R}^{n_{0}}$. More specifically, we may decompose $\mathbb{R}^{n_{0}}$ into a polyhedral complex by identifying the (maximal) subsets of $\mathbb{R}^{n_{0}}$ on which each $F_{i}$ is affine linear. We call this decomposition the canonical polyhedral complex of $F$, denoted $\mathcal{C}(F)$. To give a precise definition of this complex, we first introduce notation concerning partial compositions of the layer maps. In particular, the canonical polyhedral complex can be defined using such partial composites.

Clearly, $F=F^{(k)}\circ F_{(k-1)}$. Furthermore, each $F_{(k)}$ has an associated natural polyhedral decomposition of $\mathbb{R}^{n_{0}}$. The canonical polyhedral complex will be defined as the common refinement of these iterative polyhedral decompositions. To formalize the polyhedral decomposition induced by $F_{(k)}$, note that each affine linear map has an associated polyhedral complex.

For $k>1$, $F_{(k)}$ is not affine linear, but instead is piecewise linear. Therefore, the solution sets associated to $F_{(k)}$ are not necessarily hyperplanes. However, it is still possible to use these solution sets to determine a polyhedral decomposition of the input space, for each $k$.

A bent hyperplane is generically a piecewise linear codimension $1$ submanifold of the domain (see [7] for more details). It is “bent” in that it is a union of polyhedra, and may not be contractible or even connected. For each $F_{(k)}$, the associated bent hyperplanes induce a polyhdedral decompostion of $\mathbb{R}^{n_{0}}$ which we denote $\mathcal{C}(F_{(k)})$. The canonical polyhedral complex can then defined iteratively, by intersecting the regions of $\mathcal{C}(F_{(k)})$ with the polyhedral decomposition given at $F_{(k-1)}$.

The above definition is the “Forward Construction” of $\mathcal{C}(F)$ in [18]. Alternatively, there is an “Backward Construction” which gives the same complex. This definition originally appeared in [7].

For example, in special case of a neural network $F$ with architecture $(2,n,1)$, the canonical polyhedral complex $\mathcal{C}(F)$ is a decomposition of $\mathbb{R}^{2}$ by $n$ lines, which with full measure will fall in general position. It is immediate that $\mathcal{C}(F)$ contains $2n$ unbounded edges, $2n$ unbounded polyhedra of dimension 2, and ${n\choose 2}$ vertices.

For general piecewise linear functions on polyhedral complexes, it is not generally algorithmically decidable whether a vertex is PL critical or regular. However, the canonical polyhedral complex has combinatorial properties which make the question of PL criticality algorithmically decidable.

Following [18], under full-measure conditions called supertransversality and genericity, each cell $C$ of $\mathcal{C}(F)$ can be labeled with a sequence in $\{-1,0,1\}^{N}$, where $N=\sum_{i=1}^{m}n_{i}$. This construction is not new (for example, it also appears in [14]), but to our knowledge there is no standard reference.

These sign sequences encode the cellular poset of $\mathcal{C}(F)$ as follows:

Finally, the cellular coboundary map in $\mathcal{C}(F)$ can be neatly described:

In other words, under supertransversality and genericity conditions each $k$-cell of $\mathcal{C}(F)$ has exactly $n_{0}-k$ entries of its sign sequence equal to zero, and all incident cells can be identified by replacing each zero entry with $\pm 1$. Not unsurprisingly, this identifies the intersection combinatorics of cells in the local lower star of a vertex $v$ with the intersection combinatorics of the coordinate planes in $\mathbb{R}^{n_{0}}$.

If $v$ is a vertex of $\mathcal{C}(F)$, we can thus create a simplicial complex whose underlying set is the union of the local star and local link of $v$, and which has the combinatorics of a cross-polytope in $\mathbb{R}^{n}_{0}$. The vertices of this simplicial complex are $v$ together with a point selected from each edge incident to $v$. For each $i$ from $1$ to $n_{0}$, replacing the $i$th zero entry with a $1$ to obtain an edge $e_{i}^{+}$, then selecting a point $v_{i}^{+}$ from that edge; likewise select $v_{i}^{-}$ from the edge $e_{i}^{-}$ obtained by replacing the $i$th zero entry with a $-1$. The $n_{0}$-simplices of this simplicial complex are the convex hull of the sets consisting of exactly one of $v_{i}^{+}$ and $v_{i}^{-}$ for each $i$, together with $v$.

The arguments in [8] and [19] use this characterization and variants of Definition 5 to show that a vertex $v$ of $\mathcal{C}(F)$ is PL critical if and only if the edges $(v_{i}^{-},v)$ and $(v,v_{i}^{+})$ have opposite $\nabla F$-orientations for all $i$ , and if so, the index of the the critical point is given by the number of these pairs which are oriented towards $v$ ([19], Theorem 3.7.3). As a result, $F$ is PL Morse if and only if all edges are assigned a $\nabla F$-orientation.

As an initial example of the usefulness of the combinatorial description given by sign sequences, we develop an exploration of some realizability results for ReLU networks by classifying all possible PL Morse ReLU neural networks on an $(n,n+1,1)$ architecture, up to $\nabla F$-orientation (Theorem 2). These results are new in this context and our techniques are illustrative of sign sequence properties which will be used throughout the remainder of the paper. However, the main results of this paper do not rely on the results of this section.

Using both Lemma 6 and Lemma 3, we may further exploit properties specific to the polyhedral complex of a network $F$ with architecture $(n,n+1,1)$ to determine which vertices in $\mathcal{C}(F)$ are PL-critical, as shown in the following lemma.

We now classify all possible PL Morse $(2,3,1)$ neural networks.

Given a canonical polyhedral complex (arising from a ReLU neural network), not all functions on that canonical polyhedral complex are realizable as ReLU neural network functions. From [8] we see that that a ReLU neural network of the form $(n,m,1)$ generally has at most one $n$-cell on which it is constant, for example, but other limitations exist as well.

In fact, Figure 7 depicts a $\nabla F$ orientation on the same 3 hyperplanes which is realizable if $F$ is a PL function, but not if $F$ is a ReLU neural network of architecture $(2,3,1)$. That it cannot be a ReLU network follows immediately from Corollary 2. That it is realizable can be seen by assigning $F(v_{i})=i$ and observing that $F$ on each simplex in the highlighted simplicial complex determines the $\nabla F$ orientation on the edges of the polyhedral cell which contains it. This results in the $\nabla F$ orientations pictured.

Before we introduce our construction, we must justify why it is reasonable to use the tools of discrete Morse theory on the polyhedral complex $\mathcal{C}(F)$, which is not formally a cellular complex due to the presence of unbound cells. In fact, we will construct a discrete gradient vector field on an associated $CW$-complex $\mathcal{C}(F)^{-}_{*}$ with no unbounded cells.

In Theorem 3 we will identify a a discrete Morse function on $\mathcal{C}(F)^{-}_{*}$ with a single connected component appearing at $-\infty$. This new discrete Morse function will be relatively perfect to the PL function on the subset of $S^{n_{0}}$ given by $\mathcal{C}(F)^{-}_{*}$. That this construction does not capture gradient information on cells whose image in $F$ is not bounded above is not a large problem: the homotopy type of the sublevel sets of $\mathcal{C}(F)$ only changes at $F(v)$, for $v$ a vertex of $\mathcal{C}(F)$.

We start our construction locally, by showing that, for a given vertex in $\mathcal{C}(F)$, it is always possible to generate pairings in the local lower star of $v$ which reflect the PL-criticality of $v$.