Distance and Kernel-Based Measures for Global and Local Two-Sample Conditional Distribution Testing

The canonical setting for (nonparametric) two-sample testing focuses on assessing the equality of two unconditional distributions. In many contemporary applications, however, people are instead interested in testing the equality of two conditional distributions. Consider two independent data sets $\{(Y_{i}^{(l)},X_{i}^{(l)})\}_{i=1}^{n_{l}}$ for $l=1,2$. Here $Y_{i}^{(l)}\in\mathcal{Y}$ and $X_{i}^{(l)}\in\mathbb{R}^{p}$, where $\mathcal{Y}$ is allowed to be a general metric space (see Remark 2). For $l=1,2$, assume that $\{(Y_{i}^{(l)},X_{i}^{(l)})\}_{i=1}^{n_{l}}$ are independent and identically distributed (i.i.d.) samples of $(Y^{(l)},X^{(l)})\sim P^{(l)}\equiv P_{Y\mid X}^{(l)}\otimes P_{X}^{(l)}$, where $P_{X}^{(l)}$ is the marginal distribution of $X^{(l)}$ and $P_{Y\mid X=x}^{(l)}$ is the conditional distribution of $Y^{(l)}$ given $X^{(l)}=x$. To ensure that the testing problem considered below is nontrivial, we assume that $P_{X}^{(1)}$ and $P_{X}^{(2)}$ are equivalent, i.e., $P_{X}^{(1)}\ll P_{X}^{(2)}$ and $P_{X}^{(2)}\ll P_{X}^{(1)}$, where $P\ll Q$ means that $P$ is absolutely continuous in reference to $Q$. We aim to test the following hypothesis, which we call the global two-sample conditional distribution testing problem,

Due to the equivalence between $P_{X}^{(1)}$ and $P_{X}^{(2)}$, the hypotheses in (1) can be equivalently formulated by replacing $P^{(1)}_{X}$ with $P^{(2)}_{X}$.

We want to emphasize that the marginal distributions of $X$ from the two populations may differ (i.e., $P_{X}^{(1)}\neq P_{X}^{(2)}$), as seen in the motivating examples below. Thus, $H_{0}$ in (1) is not equivalent to $P^{(1)}=P^{(2)}$, and unconditional two-sample tests for the equality of two joint distributions (i.e., $P^{(1)}=P^{(2)}$) are generally not applicable in our context. Applying such tests would result in a failure to control the type I error when $P_{X}^{(1)}\neq P_{X}^{(2)}$. Besides, $Y$ and $X$ denote two generic random variables and do not necessarily correspond to response and covariates, respectively. For instance, in the prior shift example below, $Y$ are covariates, and $X$ is a response in (1).

Hypothesis (1) is central to many important problems in econometrics, machine learning, and statistics. For example, in transfer learning, the prior and covariate shift assumptions are commonly employed to tackle distributional differences between source and target populations (Kouw and Loog,, 2018). The prior shift assumption asserts that the conditional distribution of the covariates given the response is identical in both populations while allowing for a shift in the marginal distributions of the response. Conversely, the covariate shift assumption posits that the conditional distribution of the response given the covariates remains invariant across source and target populations, but the marginal distributions of the covariates can differ. Both assumptions are widely adopted in the literature; see, e.g., Huang et al., (2024); Lee et al., (2024) for the prior shift assumption, and Shimodaira, (2000); Tibshirani et al., (2019); Liu et al., (2023); Ma et al., (2023) for the covariate shift assumption. Despite their prevalence, there is a paucity of work that formally validates these assumptions. Both of them can be framed as testing the equality of two conditional distributions, as in (1). Such tests are essential to the validity of methods developed under the prior or covariate shift assumptions.

Another motivating example comes from causal inference. Testing hypotheses in the context of treatment effect analysis has always been of interest (Imbens and Wooldridge,, 2009, Sections 3.3 and 5.12). Consider the standard setup based on the potential outcome framework (Rubin,, 1974). Suppose $\{(Y_{i},T_{i},X_{i})\}_{i=1}^{n}$ are i.i.d. observations of $(Y,T,X)$, where $Y$ is the observed outcome of interest, $T$ denotes a binary treatment (1: treated, 0: untreated), and $X$ are pretreatment covariates. For each subject, we define a pair of potential outcomes, $\{Y(1),Y(0)\}$, that would be observed if the subject had been given treatment, $Y(1)$, and control, $Y(0)$. One may be interested in testing the null hypothesis that the conditional distribution of $Y(1)\mid X$ is the same as that of $Y(0)\mid X$ (Imbens and Wooldridge,, 2009), i.e., zero conditional distributional treatment effect. Under the prevalent assumptions of consistency, $Y=TY(1)+(1-T)Y(0)$, and no unmeasured confounding, $\{Y(1),Y(0)\}\perp\!\!\!\perp T\mid X$, the conditional distributions of $Y(1)\mid X$ and $Y(0)\mid X$ can be identified as those of $Y\mid X,T=1$ and $Y\mid X,T=0$, respectively. Therefore, it can be formulated as testing hypothesis (1) with $\{(Y_{i},X_{i}):T_{i}=1\}$ and $\{(Y_{i},X_{i}):T_{i}=0\}$ being the two sets of independent samples.

When the null hypothesis of the global two-sample conditional distribution testing is rejected, one might wish to pinpoint local regions where the two conditional distributions are significantly different. Specifically, for a fixed $x$ in the support of $P_{X}^{(1)}$ (or equivalently, $P_{X}^{(2)}$), we are interested in testing

which we refer to as the local two-sample conditional distribution testing problem, in contrast to the global problem (1). This problem is novel and is partly motivated by Duong, (2013) and Kim et al., (2019) in the context of unconditional two-sample testing. With a little extra effort, our framework can readily handle the local testing problem (2).

Recently, distance and kernel-based measures have received considerable attention in both the statistics and machine learning communities (Székely and Rizzo,, 2017; Muandet et al.,, 2017). These measures have been applied to a wide range of classical and modern hypothesis testing problems, including two-sample testing (Székely et al.,, 2004; Baringhaus and Franz,, 2004; Gretton et al.,, 2012), goodness-of-fit testing (Székely and Rizzo,, 2005; Balasubramanian et al.,, 2021), independence testing (Székely et al.,, 2007; Gretton et al.,, 2007; Chakraborty and Zhang,, 2019; Ke and Yin,, 2020; Deb et al.,, 2020), conditional independence testing (Fukumizu et al.,, 2007; Zhang et al.,, 2012; Wang et al.,, 2015; Sheng and Sriperumbudur,, 2023) and high-dimensional two sample and dependence testing (Zhang et al.,, 2018; Yao et al.,, 2018; Zhu et al.,, 2020; Chakraborty and Zhang,, 2021). In this paper, we establish a distance and kernel-based framework to tackle both the global and local two-sample conditional distribution testing problems (1) and (2). To achieve this, we introduce the conditional generalized energy distance (3) and the conditional maximum mean discrepancy (4), along with their integrated version (5), which fully characterize the homogeneity of two conditional distributions. Additionally, we show the equivalence between the conditional generalized energy distance and the conditional maximum mean discrepancy. Building upon estimators of these measures, we develop global and local tests capable of detecting discrepancies between two conditional distributions at global and local levels, respectively.

Our estimation strategy employs a combination of U-statistics and kernel smoothing, initially introduced in the so-called conditional U-statistics by Stute, (1991) and later applied to different problems by Wang et al., (2015) and Ke and Yin, (2020). To highlight our theoretical contributions, we summarize the distinct asymptotic behaviors of our global and local test statistics under both the null and alternative hypotheses in Table 1. It is noteworthy that local tests based on distance and kernel measures have not been previously explored in the literature. Furthermore, while Wang et al., (2015) and Ke and Yin, (2020) only provided the asymptotic distributions of their global test statistics under the null, we offer additional insights by systematically studying the properties of our statistics under both the null and alternative hypotheses. As a side note, we identify certain gaps in the derivations of the asymptotic null distribution in both Wang et al., (2015) and Ke and Yin, (2020), with details provided in Section 4.

Our theoretical analysis indicates that the implementation of kernel smoothing in U-statistic estimators of distance and kernel-based measures has two significant effects. On the one hand, as shown in Table 1, U-statistic estimators are no longer unbiased, and undersmoothing is required to mitigate the bias introduced by kernel smoothing. On the other hand, owing to kernel smoothing, the U-statistic estimators for distance and kernel-based measures become non-degenerate under the null hypotheses, which is in contrast to the unconditional situation (Székely et al.,, 2004; Gretton et al.,, 2012). Nevertheless, when undersmoothing is employed, the first-order projections in the Hoeffding decomposition of U-statistic estimators turn out to be asymptotically negligible under the null, and thus the asymptotic null distributions are determined by the second-order projections in the Hoeffding decomposition. For further discussion, please refer to Sections 3-4.

We now discuss related work and highlight several notable features of our framework. Although problems (1) and (2) are fundamental in various modern applications, surprisingly, there are very few methods available to test the equality of two conditional distributions. In the nonparametric testing literature, existing methods mainly focus on testing the equality of conditional moments of $Y$ given $X$. Specifically, most studies aim at testing the equality of conditional means, also known as the comparison of regression curves, as seen in Hall and Hart, (1990); Kulasekera, (1995); Dette and Munk, (1998); Lavergne, (2001); Neumeyer and Dette, (2003), among others (see Section 7 in González-Manteiga and Crujeiras, (2013) for a detailed review). Similarly, in the causal inference literature, hypothesis testing has largely been limited to conditional average treatment effect (see, e.g., Crump et al., (2008)). The methods we develop in this paper can detect general discrepancies between two conditional distributions, beyond specific moments. Lee, (2009) and Chang et al., (2015) proposed nonparametric tests for the null hypothesis of zero conditional distributional treatment effect. Their tests are built upon a Mann-Whitney statistic and cumulative distribution functions, respectively, and thus are only applicable for univariate $Y$. In our framework, $Y$ is allowed to take values in a general metric space, while $X$ can be multivariate. Very recently, Hu and Lei, (2024) proposed a test for the hypothesis (1) using techniques from conformal prediction. However, their test requires (possibly unbalanced) sample splitting, and its performance relies on high quality density ratio estimators (see Assumption 2(b) and related discussion therein). Chen et al., (2022) and Chatterjee et al., (2024) considered the paired-sample problem, which is very different from our two-sample setting. Notably, the local testing problem (2) is new and has not been addressed in the literature. Our framework can accommodate both global and local two-sample conditional distribution testing.

Notation. For $l=1,2$, let $f_{l}(y,x)$, $f_{l}(x)$ and $f_{l}(y\mid x)$ be the joint probability density function of $P^{(l)}$, the marginal probability density function of $P_{X}^{(l)}$ and the conditional probability density function of $P_{Y\mid X=x}^{(l)}$, respectively (assuming the existence of these densities). For two sequences of real numbers $\{a_{n}\}_{n=1}^{\infty}$ and $\{b_{n}\}_{n=1}^{\infty}$, we write $a_{n}\asymp b_{n}$ if and only if $a_{n}=O(b_{n})$ and $b_{n}=O(a_{n})$. The symbols $\overset{p}{\rightarrow}$ and $\overset{d}{\rightarrow}$ stand for convergence in probability and in distribution, respectively.

This section provides an overview of the generalized energy distance and maximum mean discrepancy, as well as their equivalence, which has been extensively discussed in Sejdinovic et al., (2013).

For a non-empty set $\mathcal{U}$, a non-negative function $\rho:\mathcal{U}\times\mathcal{U}\rightarrow[0,\infty)$ is called a semimetric on $\mathcal{U}$ if for any $u,u^{\prime}\in\mathcal{U}$, it satisfies (i) $\rho(u,u^{\prime})=0\iff u=u^{\prime}$ and (ii) $\rho(u,u^{\prime})=\rho(u^{\prime},u)$. Then $(\mathcal{U},\rho)$ is said to be a semimetric space. The semimetric space $(\mathcal{U},\rho)$ is said to have negative type if for all $n\geq 2$, $u_{1},\ldots,u_{n}\in\mathcal{U}$ and $\alpha_{1},\ldots,\alpha_{n}\in\mathbb{R}$ with $\sum_{i=1}^{n}\alpha_{i}=0$, we have $\sum_{i,j=1}^{n}\alpha_{i}\alpha_{j}\rho(u_{i},u_{j})\leq 0$. Define $\mathcal{M}_{\rho}^{1}(\mathcal{U})=\{\mu\in\mathcal{M}(\mathcal{U}):\int\rho(u,u_{0})d\mu(u)<\infty$, for some $u_{0}\in\mathcal{U}\}$, where $\mathcal{M}(\mathcal{U})$ denotes the set of all probability measures on $\mathcal{U}$. Suppose $P,Q\in\mathcal{M}_{\rho}^{1}(\mathcal{U})$, we have $\int\rho d(P-Q)^{2}\leq 0$ when $(\mathcal{U},\rho)$ has negative type. We say that $(\mathcal{U},\rho)$ has strong negative type if it has negative type and the equality holds only when $P=Q$. The generalized energy distance between $P,Q\in\mathcal{M}_{\rho}^{1}(\mathcal{U})$ is defined as (Sejdinovic et al.,, 2013)

where $U,U^{\prime}\overset{i.i.d.}{\sim}P$ and $V,V^{\prime}\overset{i.i.d.}{\sim}Q$. If $(\mathcal{U},\rho)$ is of strong negative type, we have $\mathcal{D}_{\rho}(P,Q)\geq 0$ and the equality holds if and only if $P=Q$. Every separable Hilbert space is of strong negative type (Lyons,, 2013). In particular, Euclidean spaces are separable, and thus the generalized energy distance generalizes the usual energy distance introduced by Székely et al., (2004) and independently by Baringhaus and Franz, (2004) from Euclidean spaces to semimetric spaces of strong negative type.

Let $\mathcal{H}$ be a Hilbert space of real-valued functions defined on $\mathcal{U}$. A function $k:\mathcal{U}\times\mathcal{U}\rightarrow\mathbb{R}$ is a reproducing kernel of $\mathcal{H}$ if (i) $\forall u\in\mathcal{U}$, $k(\cdot,u)\in\mathcal{H}$ and (ii) $\forall u\in\mathcal{U},\forall f\in\mathcal{H}$, $\langle f,k(\cdot,u)\rangle_{\mathcal{H}}=f(u)$, where $\langle\cdot,\cdot\rangle_{\mathcal{H}}$ is the inner product associated with $\mathcal{H}$. If $\mathcal{H}$ has a reproducing kernel, it is said to be a reproducing kernel Hilbert space (RKHS). According to Moore-Aronszajn theorem (Berlinet and Thomas-Agnan,, 2011), for every symmetric, positive definite function (henceforth kernel) $k$, there exists an associated RKHS $\mathcal{H}_{k}$ with reproducing kernel $k$. For $\theta>0$, define $\mathcal{M}_{k}^{\theta}(\mathcal{U})=\{\mu\in\mathcal{M}(\mathcal{U}):\int k^{\theta}(u,u)d\mu(u)<\infty\}$. The kernel mean embedding of $P\in\mathcal{M}_{k}^{1/2}(\mathcal{U})$ into RKHS $\mathcal{H}_{k}$ is defined by the Bochner integral $\Pi_{k}(P)=\int k(\cdot,u)dP(u)\in\mathcal{H}_{k}$. Besides, the kernel $k$ is said to be characteristic if the mapping $\Pi_{k}$ is injective. Conditions under which kernels are characteristic have been studied by Sriperumbudur et al., (2008, 2010), and examples include the Gaussian kernel and the Laplace kernel. The maximum mean discrepancy (MMD) between $P,Q\in\mathcal{M}_{k}^{1/2}(\mathcal{U})$ is given by (Gretton et al.,, 2012)

When the kernel $k$ is characteristic, we have $\gamma_{k}(P,Q)=0$ if and only if $P=Q$. Also, the following alternative representation of the squared MMD is useful

where $U,U^{\prime}\overset{i.i.d.}{\sim}P$ and $V,V^{\prime}\overset{i.i.d.}{\sim}Q$.

Let $\rho$ be a semimetric of negative type on $\mathcal{U}$. For any $u_{0}\in\mathcal{U}$, the function $k(u,u^{\prime})=\rho(u,u_{0})+\rho(u^{\prime},u_{0})-\rho(u,u^{\prime})$ is positive definite, and is said to be the distance-induced kernel induced by $\rho$ and centered at $u_{0}$. Correspondingly, for a kernel $k$ on $\mathcal{U}$, the function $\rho(u,u^{\prime})=\frac{1}{2}\{k(u,u)+k(u^{\prime},u^{\prime})\}-k(u,u^{\prime})$ defines a valid semimetric $\rho$ of negative type on $\mathcal{U}$, and we say that $k$ generates $\rho$. It is clear that every distance-induced kernel $k$ induced by $\rho$, also generates $\rho$. Theorem 22 in Sejdinovic et al., (2013) establishes the equivalence between the generalized energy distance and MMD. Specifically, suppose $P,Q\in\mathcal{M}_{\rho}^{1}(\mathcal{U})$ and let $k$ be any kernel that generates $\rho$, then $\mathcal{D}_{\rho}(P,Q)=\gamma_{k}^{2}(P,Q)$.

Let $\rho$ be a semimetric on $\mathcal{Y}$. We define the conditional generalized energy distance at $x\in\mathbb{R}^{p}$ as the generalized energy distance between $P_{Y\mid X=x}^{(1)}$ and $P_{Y\mid X=x}^{(2)}$.

When $\mathcal{Y}=\mathbb{R}^{q}$ and $\rho$ corresponds to the Euclidean distance, (3) also has a nice interpretation in terms of conditional characteristic function. Specifically, for $l=1,2$ and $t\in\mathbb{R}^{q}$, the conditional characteristic function of $Y^{(l)}$ given $X^{(l)}=x$ is defined as $\varphi_{Y\mid X=x}^{(l)}(t)=\mathbb{E}\{\exp(\imath t^{\top}Y^{(l)})\mid X^{(l)}=x\}$, where $\imath=\sqrt{-1}$ is the imaginary unit. When $\rho(y,y^{\prime})=\|y-y^{\prime}\|$ with $\|\cdot\|$ being the Euclidean norm, we have

where $c_{q}=\pi^{(q+1)/2}/\Gamma((q+1)/2)$ is a constant with $\Gamma(\cdot)$ being the gamma function. The proof of this fact follows a similar approach to Proposition 1 in Székely and Rizzo, (2017), and the details are omitted for brevity.

As the counterpart to the distance-based metric (3), we now introduce a kernel-based metric to measure the discrepancy between the two conditional distributions. Let $\mathcal{H}_{k}$ be a RKHS associated with a kernel $k$ on $\mathcal{Y}$.

While conducting this research, we came across Park and Muandet, (2020), where the authors present the CMMD (4) in a slightly different form. However, their estimation strategy is entirely different from ours. Neither the convergence rate nor the asymptotic distribution of their estimator is provided. By contrast, we derive the exact convergence rate and the asymptotic distribution of our statistic under both the null and alternative hypotheses. In addition, we establish a unified framework for both the distance and kernel-based measures. Furthermore, it is important to note that the CMMD is a metric indexed by $x\in\mathbb{R}^{p}$, and the single measure (5) introduced in Section 4, which integrates the CMMD with a weight function, is not discussed in Park and Muandet, (2020).

The conditional generalized energy distance and CMMD both serve to characterize the homogeneity of two conditional distributions. Besides, we can demonstrate the equivalence between the conditional generalized energy distance and CMMD. As a result, we will focus on the CMMD for the remainder of this paper.

Now, we present an estimator for the CMMD. Let $x(s)$ denote the $s$th component of $x\in\mathbb{R}^{p}$ for $s=1,\ldots,p$. For $l=1,2$, define $G_{h_{l}}:\mathbb{R}^{p}\rightarrow\mathbb{R}$ as

where $g:\mathbb{R}\rightarrow\mathbb{R}$ is a univariate kernel function satisfying Assumption 1 below, and $h_{1},h_{2}\in\mathbb{R}$ are the bandwidth parameters. For ease of presentation, here we use the same bandwidth $h_{l}$ for each component of $X^{(l)}\ (l=1,2)$. In practice, one should always allow for varying bandwidths across each component of $X^{(l)}$. Besides, it is important to note that the smoothing kernel $g$ applied to $X\in\mathbb{R}^{p}$ differs from the reproducing kernel $k$ applied to $Y\in\mathcal{Y}$, and we employ different criteria for selecting these two kernels.

Motivated by the representation of CMMD in terms of the conditional moments given in (4), we propose the following estimator for the CMMD:

which we call the sample conditional maximum mean discrepancy. This estimation strategy, which combines U-statistics and kernel smoothing, has been adopted in previous studies such as Wang et al., (2015); Ke and Yin, (2020). It first appeared in the so-called conditional U-statistics (Stute,, 1991), which generalize the Nadaraya-Watson estimate of a regression function in the same way as Hoeffding’s classical U-statistic is a generalization of the sample mean.

We shall make the following assumptions throughout the analysis.

Assumptions 1-3 are standard in the literature (see, e.g., Section 1.11 in Li and Racine, (2007)), allowing for multivariate $X$ and higher-order kernels. Assumption 4 can be seen as the counterpart of the usual smoothness condition imposed on the regression function in classical kernel regression. A similar assumption is used in Stute, (1991).

In what follows, we present the consistency of the sample CMMD in Theorem 2, while Theorems 3 and 4 provide its asymptotic distributions under $H_{a}$ and $H_{0}$ in (2), respectively.

As shown in the proof in the appendix, we have $\xi_{1}^{2}\asymp h_{1}^{-p}$ and $\xi_{2}^{2}\asymp h_{2}^{-p}$. Thus, under $H_{a}$ in (2), $\widehat{\gamma_{k}^{2}}(x)-\gamma_{k}^{2}(x)=O_{p}((n_{1}h_{1}^{p})^{-1/2}+(n_{2}h_{2}^{p})^{-1/2})$, which is the same convergence rate as that of classical kernel regression (Ullah and Pagan,, 1999) and conditional U-statistics (Stute,, 1991). The condition $n_{l}^{1/2}h_{l}^{p/2+\nu}\rightarrow 0$ as $n_{l}\rightarrow\infty$ in Theorem 3 requires undersmoothing in order to make the estimator asymptotically unbiased. Undersmoothing, coupled with the use of a higher-order kernel, is commonly employed in nonparametric and semiparametric estimation to reduce the bias of estimators involving kernel smoothing (Li and Racine,, 2007). The same condition with $p=1$ and $\nu=2$ is used in Section 3.4 of Ullah and Pagan, (1999) for the central limit theorem (CLT) of classical kernel regression and in Stute, (1991) for the CLT of conditional U-statistics.

As demonstrated in Theorem 4, the sample CMMD converges at a faster rate under $H_{0}$ in (2): $\widehat{\gamma_{k}^{2}}(x)=O_{p}((n_{1}h_{1}^{p})^{-1}+(n_{2}h_{2}^{p})^{-1})$, compared to that under $H_{a}$ in (2). This explains why the undersmoothing condition in Theorem 4 is more stringent than that in Theorem 3. Compared with the unconditional scenario (Gretton et al.,, 2012), our results differ in several aspects. First, the convergence rates are distinct. Second, our spectral decomposition is performed with respect to the conditional distribution. Third, in contrast to i.i.d. standard normal random variables in Gretton et al., (2012), $\{\zeta_{r}\}_{r=1}^{\infty}$ and $\{\eta_{r}\}_{r=1}^{\infty}$ have different variances when $f_{1}(x)\neq f_{2}(x)$.

The CMMD $\gamma_{k}^{2}(x)$ is indexed by $x\in\mathbb{R}^{p}$, and thus is not a single number. We can obtain a single measure by integrating the CMMD with some weight function $\omega:\mathbb{R}^{p}\rightarrow\mathbb{R}$, i.e.,

which we call the integrated conditional maximum mean discrepancy (ICMMD). It follows directly from Theorem 1 that $\mathcal{I}_{k,\omega}$ fully characterizes the homogeneity of two conditional distributions.

Motivated by Su and White, (2007); Wang et al., (2015); Ke and Yin, (2020), we consider the weight function $\omega(x)=\{f_{1}(x)f_{2}(x)\}^{2}$ and define

This particular choice of weight function circumvents the random denominator issue. Otherwise, to deal with the density near zero and mitigate significant bias, additional stringent assumptions or trimming schemes may need to be used. For example, Yin and Yuan, (2020) assumes that the density functions $f_{1}(x)$ and $f_{2}(x)$ are bounded away from zero, which can be quite restrictive.

We employ the same estimation strategy as in Section 3 and propose the following estimator for the ICMMD, which is a U-statistic by symmetrization (see the proof of Theorem 6):