Universal Approximation Property of
Neural Ordinary Differential Equations

First, we prove that $H^{r}$ forms a subgroup of $\mathrm{Diff}^{r}_{\mathrm{c}}$. By definition, for any $g,h\in H^{r}$, it holds that $g\circ h\in H^{r}$. Also, $H^{r}$ is closed under inversion; to see this, it suffices to show that $S^{r}$ is closed under inversion. Let $g=\Phi(\cdot,1)\in S^{r}$. Consider the map $\phi:\mathbb{R}^{d}\times U\rightarrow\mathbb{R}^{d}$ defined by $\phi(\cdot,t):=\Phi^{-1}(\cdot,t)$. It is easy to confirm that $\phi$ satisfies the conditions of Definition 7, hence $g^{-1}=\phi(\cdot,1)$ is an element of $S^{r}$. Note that $\phi$ is confirmed to be $C^{r}$ on $\mathbb{R}^{d}\times U$ by applying the inverse function theorem to $(t,\mbox{\boldmath$x$})\mapsto(t,\Phi(\mbox{\boldmath$x$},t))$.

Next, we prove that $H^{r}$ is normal. To show that the subgroup generated by $S^{r}$ is normal, it suffices to show that $S^{r}$ is closed under conjugation. Take any $g\in S^{r}$ and $h\in\mathrm{Diff}^{r}_{\mathrm{c}}$, and let $\Phi$ be a flow associated with $g$. Then, the function $\Phi^{\prime}:\mathbb{R}^{d}\times U\to\mathbb{R}^{d}$ defined by $\Phi^{\prime}(\cdot,s):=h^{-1}\circ\Phi(\cdot,s)\circ h$ is a flow associated with $h^{-1}\circ g\circ h$ satisfying the conditions in Definition 7, which implies $h^{-1}\circ g\circ h\in S^{r}$, i.e., $S^{r}$ is closed under conjugation.

Next, we prove that $H^{r}$ is non-trivial by constructing an element of $S^{r}$ that is not the identity element. First, consider the case $d=1$. Let $\tilde{v}:\mathbb{R}\to\mathbb{R}_{\geq 0}$ be a non-constant $C^{\infty}$-function such that $\mathrm{supp}\ \tilde{v}\subset[0,1]$ and $\tilde{v}^{(k)}(0)=0$ for any $k\in\mathbb{N}$. Then define $v:\mathbb{R}\to\mathbb{R}$ by

$$v(x)=\begin{cases}\tilde{v}(|x|)\frac{x}{|x|}&\text{ if }x\neq 0,\\ 0&\text{ if }x=0,\end{cases}$$

which is a $C^{\infty}$-function on $\mathbb{R}$ with a compact support. Since $v$ is Lipschitz continuous and $C^{\infty}$, there exists $\mathrm{IVP}[v]$ that is a $C^{\infty}$-function over $\mathbb{R}\times\mathbb{R}$; see Fact 1 and [17, Chapter V, Corollary 4.1]. Let $K_{v}\subset\mathbb{R}$ be a compact subset that contains $\mathrm{supp}\ v$. Then, by considering the ordinary differential equation by which $\mathrm{IVP}[v]$ is defined, we see that $\bigcup_{t\in\mathbb{R}}\mathrm{supp}\ \mathrm{IVP}[v](\cdot,t)\subset K_{v}$ and also that $\mathrm{IVP}[v](x,0)=x$. We also have $\mathrm{IVP}[v](x,s+t)=\mathrm{IVP}[v](\mathrm{IVP}[v](x,s),t)$ for any $s,t\in\mathbb{R}$. In particular, we have $\mathrm{IVP}[v](\cdot,s)^{-1}=\mathrm{IVP}[v](\cdot,-s)$ for any $s\in\mathbb{R}$. Therefore, we have $\mathrm{IVP}[v](\cdot,1)\in S^{r}$. Since $v\not\equiv 0$, $\mathrm{IVP}[v](\cdot,1)$ is not an identity map and thus $S^{r}$ is not trivial. Next, we consider the case $d\geq 2$. Take a $C^{\infty}$-function $\phi\colon\mathbb{R}\to\mathbb{R}$ with $\mathrm{supp}\ \phi=[1,2]$ and a nonzero skew-symmetric matrix $A$ (i.e. $A^{\top}=-A$) of size $d$, and let $X(x):=\phi(\|x\|)A$. We define a $C^{\infty}$-map $\Phi\colon\mathbb{R}^{d}\times\mathbb{R}\to\mathbb{R}^{d}$ by

$$\Phi(x,t):=\exp(tX(x))x.$$

Since $\exp(tX(x))$ is an orthogonal matrix for any $t\in\mathbb{R}$ and $x\in\mathbb{R}^{d}$, $\Phi$ is a $C^{\infty}$-flow on $\mathbb{R}^{d}$. Now, it is enough to show that there exists a compact set $K_{\Phi}\subset\mathbb{R}^{d}$ satisfying $\cup_{t\in\mathbb{R}}\mathrm{supp}\ \Phi(\cdot,t)\subset K_{\Phi}$. Let $K_{\Phi}:=\{x\in\mathbb{R}^{d}\ |\ \|x\|\leq 2\}$. Then the inclusion $\mathrm{supp}\ \Phi(\cdot,t)\subset K_{\Phi}$ holds for any $t\in\mathbb{R}$ since $X(x)=0$ for $x\in\mathbb{R}^{d}\setminus K_{\Phi}$. ∎

Let $\phi\in\Psi(\operatorname{Lip}(\mathbb{R}^{d}){})$. Then, by definition, there exists $F\in\operatorname{Lip}(\mathbb{R}^{d}){}$ such that $\phi=\mathrm{IVP}[F](\cdot,1)$. Let $L_{F}$ denote the Lipschitz constant of $F$. In the following, we approximate $\mathrm{IVP}[F](\cdot,1)$ by approximating $F$ using an element of $\mathcal{H}{}$.

Let $\varepsilon>0$, and let $K\subset\mathbb{R}^{d}$ be a compact subset of $\mathbb{R}^{d}$. We show that there exists $f\in\mathcal{H}{}$ such that $\left\|\mathrm{IVP}[F](\cdot,1)-\mathrm{IVP}[f](\cdot,1)\right\|_{\sup,K}<\varepsilon$. Note that $\mathrm{IVP}[f](\cdot,\cdot)$ is well-defined because $\mathcal{H}{}\subset\operatorname{Lip}(\mathbb{R}^{d}){}$. Define

$$K^{\prime}:=\left\{\mbox{\boldmath$x$}\in\mathbb{R}^{d}\ \bigg{|}\ \inf_{\mbox{\boldmath$y$}\in\mathrm{IVP}[F](K,[0,1])}\|\mbox{\boldmath$x$}-\mbox{\boldmath$y$}\|\leq 2e^{L_{F}}\right\}.$$

Then, $K^{\prime}$ is compact. This follows from the compactness of $\mathrm{IVP}[F](K,[0,1])$: (i) $K^{\prime}$ is bounded since $\mathrm{IVP}[F](K,[0,1])$ is bounded, and (ii) it is closed since the function $\min_{\mbox{\boldmath$y$}\in\mathrm{IVP}[F](K,[0,1])}\|\mbox{\boldmath$x$}-\mbox{\boldmath$y$}\|$ is continuous and hence $K^{\prime}$ is the inverse image of a closed interval $[0,2e^{L_{F}}]$ by a continuous map.

Since $\mathcal{H}{}$ is assumed to be a $\sup$-universal approximator for $\operatorname{Lip}(\mathbb{R}^{d}){}$, for any $\delta>0$, we can take $f\in\mathcal{H}{}$ such that $\left\|f-F\right\|_{\sup,K^{\prime}}<\delta$. Let $\delta$ be such that $0<\delta<\min\{\varepsilon/(2e^{L_{F}}),1\}$, and take such an $f$.

Fix $\mbox{\boldmath$x$}_{0}\in K$ and define $\Delta_{\mbox{\boldmath$x$}_{0}}(t):=\|\mathrm{IVP}[F](\mbox{\boldmath$x$}_{0},t)-\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t)\|$. Let $B{}:=\delta e^{L_{F}}{}$ and we show that

$$\Delta_{\mbox{\boldmath$x$}_{0}}(t)<2B{}$$

holds for all $t\in[0,1]$. We prove this by contradiction. Suppose that there exists $t^{\prime}$ for which the inequality does not hold. Then, the set $\mathcal{T}:=\{t\in[0,1]|\Delta_{\mbox{\boldmath$x$}_{0}}(t)\geq 2B{}\}$ is not empty and thus $\tau:=\inf\mathcal{T}\in[0,1]$. For this $\tau$, we show both $\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)\leq B{}$ and $\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)\geq 2B{}$. First, we have

	$\displaystyle\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)$	$\displaystyle=\left\|\mathrm{IVP}[F](\mbox{\boldmath$x$}_{0},\tau)-\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},\tau)\right\|$
		$\displaystyle=\left\|\mbox{\boldmath$x$}_{0}+\int_{0}^{\tau}F(\mathrm{IVP}[F](\mbox{\boldmath$x$}_{0},t))dt-\mbox{\boldmath$x$}_{0}-\int_{0}^{\tau}f(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t))dt\right\|$
		$\displaystyle\leq\left\|\int_{0}^{\tau}(F(\mathrm{IVP}[F](\mbox{\boldmath$x$}_{0},t))-F(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t)))dt\right\|$
		$\displaystyle\qquad+\left\|\int_{0}^{\tau}(F(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t))-f(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t)))dt\right\|.$

The last term can be bounded as

$$\left\|\int_{0}^{\tau}(F(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t))-f(\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t)))dt\right\|\leq\int_{0}^{\tau}\delta dt$$

because of the following argument. If $\tau=0$, then both sides equal to zero, hence it holds with equality. If $\tau>0$, then for any $t<\tau$, we have $\mathrm{IVP}[f](\mbox{\boldmath$x$}_{0},t)\in K^{\prime}$ because $t<\tau$ implies $\Delta_{\mbox{\boldmath$x$}_{0}}(t)\leq 2B{}$. In this case, $\left\|F-f\right\|_{\sup,K^{\prime}}<\delta$ implies the inequality. Therefore, we have

$$\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)\leq L_{F}\int_{0}^{\tau}\Delta_{\mbox{\boldmath$x$}_{0}}(t)dt+\int_{0}^{\tau}\delta dt.$$

Now, by applying Grönwall’s inequality [18], we obtain

$$\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)\leq\delta\tau e^{L_{F}\tau}\leq B{}.$$

On the other hand, by the definition of $\mathcal{T}$ and the continuity of $\Delta_{\mbox{\boldmath$x$}_{0}}(\cdot)$, we have $\Delta_{\mbox{\boldmath$x$}_{0}}(\tau)\geq 2B{}$. These two inequalities contradict.

Therefore, $\left\|\mathrm{IVP}[F](\cdot,1)-\mathrm{IVP}[f](\cdot,1)\right\|_{\sup,K}=\sup_{\mbox{\boldmath$x$}_{0}\in K}\Delta_{\mbox{\boldmath$x$}_{0}}(1)\leq 2B{}=2\delta e^{L_{F}}{}$ holds. Since $\delta<\varepsilon/(2e^{L_{F}})$, the right-hand side is smaller than $\varepsilon$. ∎

Let $F\colon U\to\mathbb{R}^{d}$ be an element of ${\mathcal{D}^{2}}$. Take any compact set $K\subset U$ and $\varepsilon>0$. First, thanks to Fact 2, there exists a $G\in\mathrm{Diff}^{2}_{\mathrm{c}}$ and an affine transform $W\in\mathrm{Aff}$ such that

$$W\circ G|_{K}=F|_{K}.$$

Now, if $d\geq 2$, then $2\neq d+1$, hence we can immediately use Fact 4 and Lemma 1 to show that there exists a finite set of flow endpoints (Definition 7) $g_{1},\ldots,g_{k}\in S^{2}$ such that

$$G=g_{k}\circ\cdots\circ g_{1}.$$

On the other hand, if $d=1$, by Fact 5, for any $\delta>0$, we can find $\tilde{G}$ that is a $C^{\infty}$-diffeomorphism on $\mathbb{R}$ such that $\left\|G-\tilde{G}\right\|_{\sup,K}<\delta$. Without loss of generality, we may assume that $\tilde{G}$ is compactly supported so that $\tilde{G}\in\mathrm{Diff}^{\infty}_{\mathrm{c}}$. Then, we can use Fact 4 and Lemma 1 to show that there exists a finite set of flow endpoints (Definition 7) $g_{1},\ldots,g_{k}\in S^{\infty}$ such that

$$\tilde{G}=g_{k}\circ\cdots\circ g_{1}.$$

We now construct $f_{j}\in\operatorname{Lip}(\mathbb{R}^{d}){}$ such that $g_{j}=\mathrm{IVP}[f_{j}](\cdot,1)$. By Definition 7, for each $g_{j}$ ($1\leq j\leq k$), there exists an associated flow $\Phi_{j}$. Now, define

$$f_{j}(\cdot):=\left.\frac{\partial\Phi_{j}(\cdot,t)}{\partial t}\right|_{t=0}.$$

Then, $f_{j}\in\operatorname{Lip}(\mathbb{R}^{d}){}$ because it is a compactly-supported $C^{1}$-map: it is compactly supported since there exists a compact subset $K_{j}\subset\mathbb{R}^{d}$ containing the support of $\Phi(\cdot,t)$ for all $t$, and hence $\Phi(\cdot,t)-\Phi(\cdot,0)$ is zero in the complement of $K_{j}$.

Now, $\Phi_{j}(\mbox{\boldmath$x$},t)=\mathrm{IVP}[f_{j}](\mbox{\boldmath$x$},t)$ since, by additivity of the flows,

	$\displaystyle\frac{\partial\Phi_{j}}{\partial t}(\mbox{\boldmath$x$},t)$	$\displaystyle=\lim_{s\rightarrow 0}\frac{\Phi_{j}(\mbox{\boldmath$x$},t+s)-\Phi_{j}(\mbox{\boldmath$x$},t)}{s}=\lim_{s\rightarrow 0}\frac{\Phi_{j}(\Phi_{j}(\mbox{\boldmath$x$},t),s)-\Phi_{j}(\Phi_{j}(\mbox{\boldmath$x$},t),0)}{s}$
		$\displaystyle=\left.\frac{\partial\Phi_{j}(\Phi_{j}(\mbox{\boldmath$x$},t),s)}{\partial s}\right|_{s=0}=f_{j}(\Phi_{j}(\mbox{\boldmath$x$},t)),$

and hence it is a solution to the initial value problem that is unique. As a result, we have $g_{j}=\Phi_{j}(\cdot,1)=\mathrm{IVP}[f_{j}](\cdot,1)$.

By combining Fact 3 and Lemma 2, there exist $\phi_{1},\ldots,\phi_{k}\in\Psi(\mathcal{H}{})$ such that

$$\left\|g_{k}\circ\cdots\circ g_{1}-\phi_{k}\circ\cdots\circ\phi_{1}\right\|_{\sup,K}<\frac{\varepsilon}{\left\|W\right\|_{\mathrm{op}}},$$

where $\left\|\cdot\right\|_{\mathrm{op}}$ denotes the operator norm. Therefore, we have that $W\circ\phi_{k}\circ\cdots\circ\phi_{1}\in\mathrm{INN}_{\mathcal{H}\text{-}\mathrm{NODE}}$ satisfies

	$\displaystyle\left\|F-W\circ\phi_{k}\circ\cdots\circ\phi_{1}\right\|_{\sup,K}$	$\displaystyle=\left\|W\circ G-W\circ\phi_{k}\circ\cdots\circ\phi_{1}\right\|_{\sup,K}$
		$\displaystyle\leq\left\|W\right\|_{\mathrm{op}}\left\|g_{k}\circ\cdots\circ g_{1}-\phi_{k}\circ\cdots\circ\phi_{1}\right\|_{\sup,K}$
		$\displaystyle<\varepsilon$

if $d\geq 2$. For $d=1$, it can be shown that there exists $W\circ\phi_{k}\circ\cdots\circ\phi_{1}\in\mathrm{INN}_{\mathcal{H}\text{-}\mathrm{NODE}}$ that satisfies $\left\|F-W\circ\phi_{k}\circ\cdots\circ\phi_{1}\right\|_{\sup,K}<\varepsilon$ in a similar manner. ∎