A Sieve Quasi-likelihood Ratio Test for Neural Networks with Applications to Genetic Association Studies

With the advance of science and technology, we are now in the era of the fourth industrial revolution. One of the key drivers of the fourth industrial revolution is artificial intelligence (AI). Deep neural networks play a critical role in AI and have achieved great success in many fields such as natural language processing and imaging recognition. While great attention has been given to applications of neural works (NN), limited studies have been focus on its theoretical properties and statistical inference, which hinders its application to hypothesis-driven clinical and biomedical research. The study of NN statistical inference can improve our understanding of NN properties and facilitate hypotheses testing using NN. Nevertheless, it is challenging to study NN statistical inference. For instance, it has been pointed out in Fukumizu (1996) and Fukumizu et al. (2003) that the parameters in a neural network model are unidentifiable so that classical tests (e.g., Wald test and likelihood ratio test) cannot be used because unidentifaibility of parameters leads to inconsistency of the nonlinear least squares estimators (Wu, 1981).

Many existing literature on NN, such as Shen et al. (2021), Shen et al. (2019), Horel and Giesecke (2020), Schmidt-Hieber et al. (2020), and Chen and White (1999), are based on the framework of nonparametric regression. It has been shown in Chen and White (1999) that the rate of convergence for neural network estimators is $\left\|{\hat{f}_{n}-f_{0}}\right\|=\mathcal{O}_{p}\left((n/\log n)^{-\frac{1+1/d}{4(1+1/(2d))}}\right)$ for sufficiently smooth $f_{0}$, so one of the advantages of neural networks compared with commonly used in nonparametric regression methods (e.g., Nadaraya-Watson estimator and spline regression) is that neural network estimators can avoid the curse of dimensionality in terms of rate of convergence.

There are increasing interests in studying hypothesis testing based on neural networks. Recently, Shen et al. (2019) established asymptotic theories for neural networks, which can be used to perform a nonparametric hypothesis on the true function. Horel and Giesecke (2020) used a Linderberg-Feller type central limit theorem for random process and second order functional delta method to construct test statistic to perform significance tests on input features. However, the asymptotic distribution of the test statistic is complex, making it difficult to obtain the critical value. Shen et al. (2021) proposed a goodness of fit type test based on neural networks. The test statistic is based on comparing the mean squared error values of two neural networks built under the null and the alternative hypothesis. The test statistic has an asymptotic normal distribution, and hence it can be easily used in practice. However, constructing the test statistic requires a random split of the data, which can lead to a potential power loss. In this paper, we propose a sieve quasi likelihood ratio (SQLR) test based on neural networks. Similar to Shen et al. (2021), the test statistic has an asymptotic chi-squared distribution, which facilitate its use in practice. Compared with the goodness of fit test in Shen et al. (2021), the SQLR test does note require data splitting, but requires continuous random input features.

The rest of the paper is organized as follows. Section 2 provides the general results of the sieve quasi-likelihood ratio test under the setup of nonparametric regressions. In section 3, we apply the general theories to neural networks so that significance tests based on neural networks can be performed. We investigate the validity of the theories via simple simulations in section 4, followed by a real data application to genetic association analysis of the sequencing data from Alzheimer’s Disease Neuroimaging Initiative (ADNI) in section 5. The proofs of the main results in are given in the supplementary materials.

Consider the classical setting of a nonparametric regression model under the random design,

where the covariates $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}\in\mathbb{R}^{d}$ are assumed to be i.i.d. from a distribution $P$, and $\epsilon_{1},\ldots,\epsilon_{n}$ are i.i.d. random errors with $\mathbb{E}[\epsilon]=0$. $Y_{1},\ldots,Y_{n}$ are the responses, which are continuous random variables. The true functions $f_{0}$ is assume to be in $\mathcal{F}\subset C(\mathcal{X})$, where $\mathcal{X}\subset\mathbb{R}^{d}$ is a compact subset. For simplicity, we take $\mathcal{X}=[-1,1]^{d}$. The norm considered on $\mathcal{F}$ is the $L_{2}$-norm $\|f\|=\left(\int_{\mathcal{X}}|f|^{2}\textrm{d}P\right)^{1/2}$. We further assume that $\|\epsilon\|_{p,1}=\int_{0}^{\infty}(\mathbb{P}(|\epsilon|>t))^{1/p}\textrm{d}t<\infty$ for some $p\geq 2$. Such a assumption is also considered in Han and Wellner (2019) and is necessary to obtain the desired convergence rate.

The approximate sieve exremum estimator $\hat{f}_{n}$ based on $\mathcal{F}_{n}$ is defined as

where $\mathbb{Q}_{n}(f)$ is the classical sample squared error loss function

We assume that $\bigcup_{n=1}^{\infty}\mathcal{F}_{n}$ is uniformly dense in $\mathcal{F}$, that is, for each $f\in\mathcal{F}$, there exists $\pi_{n}f\in\mathcal{F}_{n}$ such that $\sup_{\boldsymbol{x}\in\mathcal{X}}|\pi_{n}f(\boldsymbol{x})-f(\boldsymbol{x})|\to 0$ as $n\to\infty$. For simplicity, we assume that the sieve space $\mathcal{F}_{n}$ is countable to avoid additional technical issue on measurability.

The null hypothesis of the sieve quasi-likelihood ratio test is $H_{0}:\phi(f_{0})=0$, which is the same as the one proposed in Shen and Shi (2005). We define the sieve quasi-likelihood ratio statistic as

where $\mathcal{F}_{n}^{0}$ is the null sieve space given by

Similar to the definition of $\hat{f}_{n}$, we denote the approximate sieve extremum estimator under $H_{0}$ by $\hat{f}_{n}^{0}$, which satisfies

According to Shen (1997) and Shen and Shi (2005), we assume that the functional $\phi:\mathcal{F}\to\mathbb{R}$ has the following smoothness property: for any $f\in\mathcal{F}_{n}$,

where $\phi_{f_{0}}^{\prime}[f-f_{0}]$ is defined as $\lim_{t\to 0}[\phi(f(f_{0},t))-\phi(f_{0})]/t$ with $f(f_{0},t)$ being a path in $t$ connecting $f_{0}$ and $f$ such that $f(f_{0},0)=f_{0}$ and $f(f_{0},1)=f$. $\omega>0$ is the degree of smoothness of $\phi$ at $f_{0}$, $\phi_{f_{0}}^{\prime}[f-f_{0}]$ is linear in $f-f_{0}$, and

Then $\phi_{f_{0}}^{\prime}$ is a bounded linear functional on $\bar{V}_{f_{0}}$, which is the completion of $\textrm{span}\{f-f_{0}:f\in\mathcal{F}\}\subset L_{2}(\mathcal{X},\mathcal{A},P)$. From the Riesz representation theorem, there exists $v^{*}\in\bar{V}_{f_{0}}$

Let $\rho_{n}$ be the rate of convergence for $\hat{f}_{n}$, that is, $\|\hat{f}_{n}-f_{0}\|=\mathcal{O}_{p}(\rho_{n})$. Let $\delta_{n}$ be a sequence converging to 0 with $\delta_{n}=\mathcal{O}_{p}(n^{-1/2})$. For $f\in\{f\in\mathcal{F}_{n}:\|f-f_{0}\|\leq\rho_{n}\}$, we define

where $u^{*}=\pm v^{*}/\left\|{v^{*}}\right\|^{2}$. The main result relies on the following conditions:

• (C1)

  (Sieve Space) Suppose that $\mathcal{F}_{n}$ is uniformly bounded. Moreover, assume that there exists a non-increasing continuous function $H(u)$ of $u>0$ such that

  $$\log N(u,\mathcal{F}_{n},L_{2}(\mathbb{P}_{n}))\leq H(u)\textrm{ for all }u>0,$$

  and

  $$\int_{0}^{1}H^{1/2}(u)\textrm{d}u<\infty.$$

• (C2)

  (Rate of Convergence) The rate of convergence $\rho_{n}$ satisfies $o(n^{-1/4})=\rho_{n}\gtrsim n^{-\frac{1}{2}+\frac{1}{2p}}$ and

  $$n\rho_{n}^{2}\geq 2H(\rho_{n})\textrm{ and }H(\rho_{n})\to\infty\textrm{ as }n\to\infty.$$

• (C3)

  (Approximation Error)

  $$\sup_{Q}\sup_{\begin{subarray}{c}f\in\mathcal{F}_{n}\\ \|f-f_{0}\|\leq\rho_{n}\end{subarray}}\left\|{\pi_{n}\tilde{f}_{n}(f)-\tilde{f}_{n}(f)}\right\|_{L_{2}(Q)}=o_{p}(\rho_{n}^{-1}\delta_{n}^{2}),$$

  where the supremum is taken over all probability measures $Q$ on $(\mathcal{X},\mathcal{A})$ and

  $$\sup_{\begin{subarray}{c}f\in\mathcal{F}_{n}\\ \left\|{f-f_{0}}\right\|\leq\rho_{n}\end{subarray}}n^{-1}\sum_{i=1}^{n}\epsilon_{i}\left(\pi_{n}\tilde{f}_{n}(f)(\boldsymbol{X}_{i})-\tilde{f}_{n}(f)(\boldsymbol{X}_{i})\right)=o_{p}(\delta_{n}^{2}).$$

We now state the main theorem for sieve quasi-likelihood ratio statistics. The proof of the theorem follows the same steps as those in Shen and Shi (2005) and are given in the Supplementary materials.

In practice, $\sigma^{2}$ is rarely known apriori. A simple application of Slutsky’s theorem yields the following corollary, which shows that we can replace $\sigma^{2}$ with any consistent estimator of $\hat{\sigma}_{n}^{2}$. A straightforward consistent estimator for $\sigma^{2}$ is given by $\hat{\sigma}_{n}^{2}=n^{-1}\sum_{i=1}^{n}\left(Y_{i}-\hat{f}_{n}^{0}(\boldsymbol{X}_{i})\right)^{2}$.

We first introduce the notations to be used in this section. $\boldsymbol{e}_{i}=(0,\ldots,0,1,0,\ldots,0)$ where 1 appears at the $i$th position. We use $\mathbb{Z}_{+}$ to the set of non-negative integers and use $\boldsymbol{\beta}=(\beta_{1},\ldots,\beta_{d})\in\mathbb{Z}_{+}^{d}$ to denote a multi-index. Moreover, we set $\left|{\boldsymbol{\beta}}\right|=\sum_{i=1}^{d}\left|{\beta_{i}}\right|$ and $\boldsymbol{x}^{\boldsymbol{\beta}}=x_{1}^{\beta_{1}}\cdots x_{d}^{\beta_{d}}$ for any $\boldsymbol{x}=(x_{1},\ldots,x_{d})^{T}\in\mathbb{R}^{d}$. For a differentiable function $u$ on $\mathcal{X}$, we set

One of our goals in this paper is to establish a sieve quasi-likelihood ratio test for neural network estimators. Specifically, for a given $k\leq d$, let $\boldsymbol{X}=(X^{(1)},\ldots,X^{(k)},X^{(k+1)},\ldots,X^{(d)})^{T}\in\mathbb{R}^{d}$ and the null hypothesis of interest be

Different from linear regression, in which the hypothesis can be easily translated into testing whether the corresponding regression coefficients are zero, testing significance of an association in nonparametric regression is more complicated. From Chen and White (1999) and Horel and Giesecke (2020), testing $H_{0}$ in the nonparametric setting is equivalent to test whether the corresponding partial derivatives are zeros, or equivalently, to test

Hence, we assume that the true function $f_{0}$ is a smooth function. Specifically, we consider the Barron class $\mathscr{B}^{s}:=\{f:\mathcal{X}\to\mathbb{R}|\left\|{f}\right\|_{\mathscr{B}^{s}}\leq B\}$ for some integer $s\geq 1$ and some fixed constant $B$, as considered in Siegel and Xu (2020). Here

and $\hat{f}(\omega)$ is the Fourier transform of $f$. As shown in Siegel and Xu (2020), $\mathscr{B}^{s}\subset H^{s}(\mathcal{X})=\{f:\mathcal{X}\to\mathbb{R}|\left\|{D^{\boldsymbol{\alpha}}f}\right\|<\infty,\textrm{ for all }0\leq\left|{\boldsymbol{\alpha}}\right|\leq s\}$ and $H^{\left\lfloor\frac{d}{2}\right\rfloor+2}(\mathcal{X})\subset\mathscr{B}^{1}$. In what follows, we will take $\mathcal{F}=C^{m_{0}}(\mathcal{X})$ with $m_{0}=\left\lfloor\frac{d}{2}\right\rfloor+2$.

The functional $\phi$ from the general result in section 2 is given by

The directional derivative $\phi_{f_{0}}^{\prime}$ evaluated at “direction” $h$ can be calculated straightforwardly. For the sieve space, we use the class of neural networks with one hidden layer and sigmoid activation function $\sigma(x)=(1+e^{-x})^{-1}$.

where $r_{n}\uparrow\infty$ as $n\to\infty$. In view of Barron (1993), $\mathcal{F}_{r_{n}}$ is $L_{2}$-dense in $\mathcal{F}$.

Based on the general results in the previous section, the function $\phi$ needs to be smooth so that the sieve quasi-likelihood ratio statistic follows a chi-squared asymptotic distribution. The following propositions guarantee the satisfaction of conditions on $\phi$ in the general theory.

We now impose the following condition on the distribution $P$.

• (C4)

  Suppose that $P\ll\lambda$, where $\lambda$ is the Lebesgue measure on $\mathbb{R}^{d}$. Let

  $$\varphi(\boldsymbol{x})=\frac{\textrm{d}P}{\textrm{d}\lambda}(\boldsymbol{x})\geq 0.$$

  Moreover, we assume that $\varphi=0$ on $\partial\mathcal{X}$, $\varphi\in L^{\infty}(\mathcal{X})$, and $\log\varphi\in C^{1}(\mathcal{X})$.

Before we bound the remainder error of the first order functional Taylor expansion, we provide a bound for higher order derivatives of a neural network.