Kernel-based measures of association between inputs and outputs using ANOVA

In statistical modeling and data analysis, symmetric measures of dependence for random vectors such as the Pearson correlation ([1]); the Spearman correlation; the Kendall correlation; the canonical correlation coefficient ([2, 3]); the RV coefficient for two random vectors ([4]); the maximum mean discrepancy ([5, 6, 7]), including the energy distance ([8]); the Hilbert-Schmidt independence criterion ([9, 10, 11]), including the distance correlation ([12, 13]) and a recent correlation coefficient ([14]) have been used by different communities for testing whether two random vectors are independent or not, and then for measuring the dependence between such random vectors. Drawbacks, advantages and links between such dependent measures can be found in [15, 16, 11, 17] for instance.

The above dependent measures do not require any function linking both random vectors. In presence of a mathematical model or a black box function of the form $Y=f(\mathbf{X})$, where $\mathbf{X}:=(X_{1},\ldots,X_{d})$ are the model inputs with known probability distributions and $Y$ is the model output, such measures can still be used for testing the independence between the model output $Y=f(\mathbf{X})$ and some inputs such as $X_{1}$. Such a problem occurs in computer experiments where a natural or human induced phenomenon is represented by complex computer code with numerous input variables ([18]). For dimension reduction, it is relevant to identify unessential variables by means of different criteria such as statistical tests of independence or the variance-based sensitivity indices (SIs) ([19, 20, 21, 22, 23]), which were developed in the framework of the statistical theory called ANOVA ([24, 25, 26, 19]).

Formally, the above dependent measures between $f(\mathbf{X})$ and $X_{1}$ rely on the test statistic of the form $T(f(\mathbf{X}),X_{1})$ or $T(f(\mathbf{X}),h(X_{1}))$ with $T,\,h$ some given functions. When a statistical test reveals that the random variables $f(\mathbf{X})$ and $X_{1}$ are dependent, the associated test statistic usually serves as a reasonable measure of the association between both variables. While this crude dependent measure is relevant for screening input variables in some cases, it may be not satisfying for ranking input variables because it involves only the inputs of interest and the output or the output and its conditional expectation given the inputs of interest ([27, 28, 29, 30]). Moreover, it i) may result in dealing with transformations of the model output rather than using $Y$ ([31]); ii) may include undesirable effects or complicate the problem ([31, 32]); iii) ignores the asymmetric role of the model inputs and output. For instance, the MMD-based indices result in using $\phi(\mathbf{Y})$ as outputs with $\phi$ some given functions a.k.a. feature maps (see [33]). In the same sense, the HSIC-based indices proposed in [33] result in working with transformations of the original inputs and output, as the HSIC requires defining one kernel on the inputs and another one on the outputs ([34]).

It is known that ANOVA remains an important step in exploratory and confirmatory analysis of model ([35]), and it relies on direct use of the model output(s) and inputs. The well-known ANOVA-like decomposition of $f(\mathbf{X})$ with independent variables $\mathbf{X}$ can be written as follows ([24, 25, 26, 19, 36, 37]):

$$f(\mathbf{X})=\sum_{\begin{subarray}{c}v\subseteq\{1,\ldots,d\}\end{subarray}}f_{v}\left(\mathbf{X}_{v}\right)=\mathbb{E}\left[f(\mathbf{X})\right]+f_{1}^{tot}(\mathbf{X})+f_{\sim 1}(X_{2},\ldots,X_{d})\,,$$

(1)

where $\mathbf{X}_{v}$ is a subset of the model inputs whose subscripts belong to $v$; and $f_{1}^{tot}(\mathbf{X}):=\sum_{\begin{subarray}{c}v\subseteq\{1,\ldots,d\}\\ v\cap\{1\}\neq\emptyset\end{subarray}}f_{v}\left(\mathbf{X}_{v}\right)$ denotes the total ANOVA functional (AF) of $X_{1}$. The FANOVA decomposition of functions with non-independent variables is similar thanks to dependency models of dependent variables (see Section 2 and [38, 39, 40]). ANOVA functionals are random variables, which contain the primary information about the contributions of the input variables on the output variable. In view of Equation (1), $f(\mathbf{X})$ does not dependent on $X_{1}$ whenever the total AF follows the Dirac probability measure $\delta_{\mathbf{0}}$, that is, $f_{1}^{tot}(\mathbf{X})=0$ almost surely (a.s.) (see [41] for independent variables and Lemma 1 in general).

The aim of this paper is to propose a new dependent measure between the model inputs and output(s) that is more comprehensive and much flexible to account for the necessary or desirable statistical properties of AFs (e.g., higher-order moments, the Cauchy and heavy tailed distributions, asymmetric distributions) and the interactions among the input variables. The proposed dependent measure relies on the properties of AFs such as the independence criterion $f_{1}^{tot}(\mathbf{X})=0\;a.s.$ and the reproducing kernel Hilbert space (RKHS) ([42, 43, 44]). The RKHS theory offers a flexible framework for precise statistical inference of probability distributions ([6, 34, 16]), and it is crucial for conducting independence tests that account for the necessary or sufficient or desirable statistical properties of distributions ([9, 6, 34, 45, 46, 32]). The proposed dependent measure leads to provide the first-order and total kernel-based sensitivity indices, and its empirical expression is used for deriving a test statistic for testing the independence between the model inputs and output(s).

This paper is organized as follows: Section 2 deals with AFs and their statistical properties in terms of their ability to assess the effects of the input variables on the model output(s). Such properties enable the formulation of the initial null hypothesis for the independence test between input variables and the model output(s) and equivalent null hypothesis such as the variance $\mathbb{V}\left[f_{1}^{tot}(\mathbf{X})\right]=0$. Most of variance-based SIs rely on that null hypothesis for performing dimension reduction. Indeed, $\mathbb{V}\left[f_{1}^{tot}(\mathbf{X})\right]$ represents the non-normalized total SI of $X_{1}$.
Since the associated alternative hypothesis is sufficient for AFs that are Gaussian-distributed and the Cauchy distribution does not have the variance, Section 3 is devoted to develop equivalent null hypothesis for the independence test by embedding AFs into an appropriate RKHS regarding their distributions. Moreover, as a given kernel can help for working with specific moments ([32]), we also provide the set of distribution-free kernels that guarantee the equivalent null hypothesis for the independence test between the model inputs and output(s). Section 4 presents a statistical test that leads to an interesting measure of association between inputs and output(s). In Section 5, we formally introduce our empirical dependent measure and the associated test statistic. We also provide and study kernel-based SIs. We provide analytical and numerical results in Section 8 and conclude this work in Section 9.

For an inetegr $d>1$, $d$ input variables $\mathbf{X}:=(X_{1},\,\ldots,\,X_{d})$ and $u\subseteq\{1,\,\ldots,\,d\}$, we use $\mathbf{X}_{u}:=(X_{j},\forall\,j\in u)$, $\mathbf{X}_{\sim u}:=(X_{j},\forall\,j\in\{1,\,\ldots,\,d\}\setminus u)$ and $|u|$ for the number of elements in $u$. Thus, we have the partition $\mathbf{X}=(\mathbf{X}_{u},\,\mathbf{X}_{\sim u})$. We also use $\mathbf{R}\stackrel{{\scriptstyle d}}{{=}}\mathbf{X}$ to say that $\mathbf{R}$ and $\mathbf{X}$ have the same cumulative distribution function (CDF).

In this section, we provide the properties of ANOVA functionals for functions with non-independent variables. Such properties serve as the initial null hypothesis for testing the independence between the random input variables and output variables. For the sequel of generality, consider a vector-valued function $f:\mathbb{R}^{d}\to\mathbb{R}^{n}$, which includes a $d$-dimensional random vector $\mathbf{X}$ as inputs and provides $n$ outputs given by $f(\mathbf{X})$. We are interested in measuring the association between the following two random vectors: $f(\mathbf{X})$ and $\mathbf{X}_{u}$ with $u\subseteq\{1,\ldots,d\}$ and $u\neq\emptyset$, keeping in mind the asymmetric role between both random vectors.

For any distribution of $\mathbf{X}$, we are able to model $\mathbf{X}$ as follows ([47, 38, 40, 39]):

$$\mathbf{X}_{\sim u}\stackrel{{\scriptstyle d}}{{=}}r_{u}\left(\mathbf{X}_{u},\mathbf{Z}\right)\,;$$

(2)

where $r_{u}$ is a function; $\mathbf{Z}$ is a random vector of $d-|u|$ independent variables, and $\mathbf{X}_{u}$ is independent of $\mathbf{Z}$. Note that when $\mathbf{X}$ is consisted of independent variables, the dependency function in (2) comes down to $\mathbf{X}_{\sim u}\stackrel{{\scriptstyle d}}{{:=}}r_{u}\left(\mathbf{Z}\right)$. Composing $f$ by the function $r_{u}$ in (2) yields

$$g(\mathbf{X}_{u},\mathbf{Z})\stackrel{{\scriptstyle d}}{{:=}}f(\mathbf{X}_{u},r_{u}(\mathbf{X}_{u},\,\mathbf{Z}))\,.$$

(3)

In view of Equation (3), the function $g$ includes two independent random vectors (i.e., $\mathbf{X}_{u}$ and $\mathbf{Z}$) as inputs, and it provides $n$ outputs sharing the distribution of $f(\mathbf{X})$. For independent variables $\mathbf{X}$, we can see that $g(\mathbf{X}_{u},\mathbf{Z})=f(\mathbf{X}_{u},r_{u}(\mathbf{Z}))\stackrel{{\scriptstyle d}}{{=}}f(\mathbf{X}_{u},\mathbf{X}_{\sim u})$. For the sequel of generality, we are going to use $g(\mathbf{X}_{u},\mathbf{Z})$ in what follows, which comes down to $f(\mathbf{X}_{u},\mathbf{X}_{\sim u})$ when the inputs are independent. As a matter of fact, we can always claim that

Note that (A1) is not theoretically or practically restrictive because it is always satisfied thanks to dependency models (see (2)).

This section provides a set of kernels that ensure the equivalent criterion of independence between the input variables $\mathbf{X}_{u}$ and the model outputs. Recall that the null hypothesis $\mathbb{V}\left[g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z})\right]=\mathsf{O}$ is an equivalent criterion of independence, and it is also obtained using the quadratic kernel, that is, $k_{2}\left(\mathbf{r},\mathbf{r}^{\prime}\right)=\left<\mathbf{r},\,\mathbf{r}^{\prime}\right>^{2}_{\mathbb{R}^{n}}$. Indeed, if we use $\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime}$ for i.i.d. copies of $\mathbf{X}_{u},\mathbf{Z}$, we can check that

$$\mathbb{E}\left[k_{2}\left(g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z}),\,g^{tot}_{u}(\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime})\right)\right]=0\Longleftrightarrow\mathbb{V}\left[g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z})\right]=\mathsf{O}\,.$$

It is worth noting that kernels of the form $k_{2p}\left(\mathbf{r},\mathbf{r}^{\prime}\right)=\left<\mathbf{r},\,\mathbf{r}^{\prime}\right>^{2p}_{\mathbb{R}^{n}}$ for integer $p>0$ lead to an equivalent null hypothesis of independence, although such kernels are not characteristic in general. Since some kernels do not ensure the independence criterion, let us start with the following definitions thanks to Lemma 1. Namely, we use $\mathcal{K}$ for the set of valid and measurable kernels; $H$ for the CDF of the Dirac probability measure $\delta_{\{\mathbf{0}\}}$.

The equivalence used in Definition 4 is guaranteed by Lemma 1. For a centered kernel at $\mathbf{0}$ (i.e., $\bar{k}$), the left term of Equation (5) becomes $\mathbb{E}\left[\bar{k}\left(g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z}),\,g^{tot}_{u}(\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime})\right)\right]=\leavevmode\nobreak\ 0$.

To construct the set of equivalent kernels for the independence test, consider the following set of kernels:

$$\mathcal{K}_{E}:=\left\{k\in\mathcal{K}:\,\begin{array}[]{c}\,\mathbb{E}_{\mathbf{G}\sim F,\mathbf{G}^{\prime}\sim F}\left[\bar{k}(\mathbf{G},\mathbf{G}^{\prime})\right]=0\Rightarrow F=H,\;\forall\,F\in\mathcal{F}\\ \mbox{or}\\ \mathbb{E}_{(\mathbf{G},\,\mathbf{G}^{\prime})\sim\nu}\left[k(\mathbf{G},\mathbf{G}^{\prime})\right]=0\Rightarrow\nu=0,\;\forall\,\nu=F\otimes F-H\otimes H\end{array}\right\}\,.$$

(6)

We can check that $\mathcal{K}_{E}$ contains quadratic kernels that are centered at $\mathbf{0}$, and we are going to see that it contains some well-known characteristic kernels. Lemma 2 gives interesting properties of $\mathcal{K}_{E}$ regarding the null hypothesis.

Lemma 2 shows that the set $\mathcal{K}_{E}$ given by (6) contains equivalent kernels for the independence criterion whatever are the distributions of the model outputs and total AFs. Despite $\mathcal{K}_{E}$ is not exhaustive, Lemma 3 shows its richness. To that end, consider the famous radial-based characteristic kernels on $\mathbb{R}^{n}\times\mathbb{R}^{n}$ given by

$$k(\mathbf{r},\mathbf{r}^{\prime}):=\int e^{-i(\mathbf{r}-\mathbf{r}^{\prime})^{T}\mathbf{w}}\,d\Lambda(\mathbf{w})\,,$$

where $\Lambda$ is a positive and bounded Borel measure with the support $Supp(\Lambda)=\mathbb{R}^{n}$.

Kernels $\bar{k}$ of Point (i) in Lemma 3 lead to a comparison between the distributions of the total AF and the Dirac measure using the maximum mean discrepancy (see Section 6.2). Thus, Point (ii) offers other possibilities that allow for working with non-centered and characteristic kernels such as Gaussian kernels.

For testing the independence between $g\left(\mathbf{X}_{u},\mathbf{Z}\right)$ and $\mathbf{X}_{u}$ and obtaining an interesting dependent measure when $\mathbf{X}_{u}$ contribute to the outputs $g\left(\mathbf{X}_{u},\mathbf{Z}\right)$, we are going to use kernels that guarantee the equivalent null hypothesis of independence between these random vectors such as the set of kernels $\mathcal{K}_{E}$.

This section aims at providing empirical dependent measures, including empirical kernel-based sensitivity indices (Kb-SIs) and the test statistic. Note that the first-order AF $\mathbf{G}_{u}^{fo}$ and the total AF $\mathbf{G}_{u}^{tot}$ for all $u\subseteq\{1,\ldots,d\}$ will lead to the first-order and total Kb-SIs. For concise notations and when there is no ambiguity, we are going to use $S_{u}^{k,q}$, $S_{T_{u}}^{k,q}$ instead of $S_{F_{u}}^{k,q}$, $S_{F_{T_{u}}}^{k,q}$.

For computing the dependent measures defined in Section 4.2, we are given two i.i.d. samples, that is, $\left\{\left(\mathbf{X}_{i,u},\mathbf{Z}_{i},\mathbf{X}_{i,u}^{\prime},\mathbf{Z}_{i}^{\prime}\right)\right\}_{i=1}^{m_{1}}$ and $\left\{\left(\mathbf{X}_{i,u},\mathbf{Z}_{i},\mathbf{X}_{i,u}^{\prime},\mathbf{Z}_{i}^{\prime}\right)\right\}_{i=1}^{m}$ from the random vector $\left(\mathbf{X}_{u},\mathbf{Z},\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime}\right)$, where the four components are mutually independent. Define

$$\mu_{k}^{fo}:=\mathbb{E}\left[k\left(g^{fo}_{u}(\mathbf{X}_{u}),\,g^{fo}_{u}(\mathbf{X}_{u}^{\prime})\right)\right];\qquad\mu_{k}^{tot}:=\mathbb{E}\left[k\left(g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z}),\,g^{tot}_{u}(\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime})\right)\right]\,;$$

$$\sigma_{k}^{fo}:=\mathbb{V}\left[k\left(g^{fo}_{u}(\mathbf{X}_{u}),\,g^{fo}_{u}(\mathbf{X}_{u}^{\prime})\right)\right];\,\qquad\,\sigma_{k}^{tot}:=\mathbb{V}\left[k\left(g^{tot}_{u}(\mathbf{X}_{u},\mathbf{Z}),\,g^{tot}_{u}(\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime})\right)\right]\,;$$

$$\mu_{k}^{c}:=\mathbb{E}\left[k\left(g(\mathbf{X}_{u},\mathbf{Z})-\mathbb{E}\left[g(\mathbf{X}_{u},\mathbf{Z})\right],\,g(\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime})-\mathbb{E}\left[g(\mathbf{X}_{u},\mathbf{Z})\right]\right)\right]\,.$$

Also, recall that the law of large numbers (LLN) ensures the convergence in probability of the following estimators when $m_{1}\to\infty$:

$$\widehat{\mu}(\mathbf{Z})=\frac{1}{m_{1}}\sum_{i=1}^{m_{1}}g(\mathbf{X}_{i,u},\mathbf{Z})\,\xrightarrow{P}\,\mathbb{E}_{\mathbf{X}_{u}}\left[g(\mathbf{X}_{u},\mathbf{Z})\right];\quad\widehat{\mu}:=\frac{1}{m_{1}}\sum_{i=1}^{m_{1}}g(\mathbf{X}_{i,u},\mathbf{Z}_{i})\,\xrightarrow{P}\,\mathbb{E}\left[g(\mathbf{X}_{u},\mathbf{Z})\right]\,;$$

$$\widehat{\mu}(\mathbf{X}_{u}):=\frac{1}{m_{1}}\sum_{i=1}^{m_{1}}g(\mathbf{X}_{u},\mathbf{Z}_{i})\,\xrightarrow{P}\,\mathbb{E}_{\mathbf{Z}}\left[g(\mathbf{X}_{u},\mathbf{Z})\right]\,.$$

Using the plug-in approach, we provide the estimators of $\mu_{k}^{fo},\;\mu_{k}^{tot}$, $\sigma^{tot}_{k}$ and their statistical properties in Theorem 2.

Based on Theorem 2, we derive i) the estimators of Kb-SIs in Corollary 2, and ii) the empirical test statistic under the null hypothesis and its asymptotic distribution in Corollary 3. Estimating the Kb-SIs for at least the $d$ input variables $X_{j}$ with $j=1,\ldots,d$ or every subset of inputs will require different samples of the form $\left(\mathbf{X}_{u},\mathbf{Z},\mathbf{X}_{u}^{\prime},\mathbf{Z}^{\prime}\right)$. We then use $M\geq m$ for the size of the sample that can be used to estimate $\mu_{k}^{c}$, that is,

$$\widehat{\mu_{k}^{c}}:=\frac{1}{M}\sum_{i=1}^{M}k\left(g(\mathbf{X}_{i,u},\mathbf{Z}_{i})-\widehat{\mu},\,g(\mathbf{X}_{i,u}^{\prime},\mathbf{Z}_{i}^{\prime})-\widehat{\mu}\right)\,\xrightarrow{P}\,\mu_{k}^{c}\,.$$