Evaluating Aleatoric Uncertainty
via Conditional Generative Models

In this section, we provide rigorous definitions on conditional generative models. Consider a standard statistical framework where a pair of random vectors $(X,Y)\in\mathcal{X}\times\mathcal{Y}$ follows a joint distribution $P_{X,Y}$ with marginal distributions $X\sim P_{X}$ and $Y\sim P_{Y}$. We assume the space $\mathcal{X}\subset\mathbb{R}^{d}$ with $d\geq 1$ which is allowed to contain either continuous or discrete components. Denote the conditional distribution of $Y$ given $X$ by $P_{Y|X}$. For a given value $x$ of $X$, denote the conditional distribution as $P_{Y|X=x}$. Typically, we regard $X$ as a vector of input (example) and $Y$ as a vector of output (label). For instance, $Y\subset\mathbb{R}^{q}$ with $q\geq 1$ in regression and $Y\subset[K]:=\{1,\ldots,K\}$ in classification. Alternatively, in image generation tasks, $X$ refers to auxiliary information (such as the image attributes or labels) and $Y$ refers to the image in order to keep the notation consistent.

Our goal is to quantify the conditional distribution $P_{Y|X}$ via conditional generative models. More precisely, let $\xi\in\mathbb{R}^{m}$ be a random vector independent of $X$ with a known distribution $P_{\xi}$ (specified by the learner) and the goal is to construct a function $G:\mathbb{R}^{m}\times\mathcal{X}\to\mathcal{Y}$ such that the conditional distribution of $G(\xi,X)|X=x$ is the same as $P_{Y|X=x}$. The following lemma demonstrates the existence of such function $G$, termed as the conditional generative model $G:\mathbb{R}^{m}\times\mathcal{X}\to\mathcal{Y}$.

The conditional generative model can provide more information than standard regression models with single-point prediction. In regression problems, the conditional mean can be estimated by taking the sample mean of multiple draws from $G(\xi_{i},X)|X=x$. Meanwhile, other numerical characteristics of the underlying target distribution, such as conditional variance and conditional quantile can also be estimated by Monte Carlo sampling from the conditional generative model, beyond what single-point prediction could offer.

In the rest of this paper, we use $P_{Y|X}$ for the ground-truth conditional distribution and $Q_{Y|X}$ for the distribution of the conditional generative model $G(\xi,X)|X$. We denote $Q_{X,G(\xi,X)}$ as the joint distribution of $(X,G(\xi,X))$. For each given $x$, the generative model is able to generate conditionally independent and identically distributed (i.i.d.) samples $G(\xi_{i},x)$ from the conditional distribution $Q_{Y|X=x}$. We parametrize the conditional generative model in a hypothesis class $\{G_{\theta}(\xi,X):\theta\in\Theta\}$ with parameter $\theta$. To learn $G(\xi,X)|X$ as an estimate of $P_{Y|X}$, we need a metric to quantify the difference between $G(\xi,X)|X$ and $P_{Y|X}$ using finite training data, which relates to Two-Sample Test. To this end, we will use the (kernel) maximum mean discrepancy (MMD) [14], which is described in the next subsection.

We review the standard MMD in the setting of unconditional distribution on $\mathcal{Y}$. Section A provides preliminaries on the reproducing kernel Hilbert space (RKHS). Suppose that $\mathcal{F}_{X}$ ($\mathcal{F}_{Y}$) is the RKHS defined on the space $\mathcal{X}$ ($\mathcal{Y}$) with kernel $k_{1}$ ($k_{2}$) and feature map $\phi_{1}$ ($\phi_{2}$). We adopt the following two basic assumptions throughout this paper (i.e., all theorems make these assumptions without explicit mentioning). Detailed explanations on Assumptions 1 and 2 can be found in Section A. The Gaussian kernels for instance satisfy both assumptions.

The integral probability metric aims to measure the discrepancy between two distributions. Let $\mathcal{G}$ denote a set of functions $\mathcal{Y}\to\mathbb{R}$. Given two distributions $P_{Y}$ and $Q_{Y}$ on $\mathcal{Y}$, the integral probability metric is defined as

where $Y\sim P_{Y}$ and $\hat{Y}\sim Q_{Y}$. MMD is a special case of integral probability metrics, as it chooses $\mathcal{G}$ to be the unit ball in the RKHS $\mathcal{F}_{Y}$. Let $\mu_{P_{Y}}$ denote the kernel mean embedding of $P_{Y}$ in $\mathcal{F}_{Y}$: $\mu_{P_{Y}}:=\mathbb{E}_{y\sim P_{Y}}[\phi_{2}(y)]$. $\mu_{P_{Y}}$ is guaranteed to be an element in the RKHS $\mathcal{F}_{Y}$ under Assumption 1 [14]. With these discussions, the square of MMD distance between $P_{Y}$ and $Q_{Y}$ is formally defined as

where $Y_{1},Y_{2}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P_{Y}$ and $\hat{Y}_{1},\hat{Y}_{2}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q_{Y}$.

Suppose we have data $y_{i}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P_{Y}$ ($i\in[n]$) and $\hat{y}_{j}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q_{Y}$ ($j\in[m]$). Then a standard unbiased estimator of $MMD^{2}(P_{Y},Q_{Y})$ [14] is

In [47], conditional generative moment-matching networks (CGMMNs) were developed for conditional distribution generation. In particular, they leveraged previous work on conditional mean embeddings of the conditional distribution $C_{P_{Y|X}}$ [51, 9, 50, 39] (Section A provides a review on this topic.) They used the discrepancy between $C_{P_{Y|X=x}}$ and $C_{Q_{Y|X=x}}$ to measure the difference of two conditional distributions, termed as CMMD, which is defined formally as:

where $P$ represents the ground-truth data distribution and $Q$ represents the generative distribution. The estimator of $CMMD^{2}$ developed in [47] is as follows:

where $\Phi_{2}=(\phi_{2}(y_{1}),...,\phi_{2}(y_{n}))$, $\Phi_{1}=(\phi_{1}(x_{1}),...,\phi_{1}(x_{n}))$, $K_{X}=\Phi_{1}^{\top}\Phi_{1}$, and $I$ is the identity matrix.

While [47] is the most relevant study to our problem setting, directly applying CMMD on aleatoric uncertainty estimation has following limitations:

Computationally Expensive: The matrix inverse in the estimator is computationally expensive for practical implementation. The running time for a single inversion in one iteration is at the order of $O(B^{3})$, where $B$ is the batch size. Meanwhile, the batch size should be sufficient large to achieve good performance for generative models [34].

Strong Technical Assumptions and Existence of Inversion: 1) The existence of the conditional mean embedding operator $C_{P_{Y|X}}$ typically requires strong assumptions: $\forall g\in\mathcal{F}_{Y}$, $\mathbb{E}_{P_{Y|X}}[g(Y)|X]\in\mathcal{F}_{X}$. This assumption is not necessarily true for continuous domains [51], and simple counterexamples using the Gaussian kernel can be found [9]. 2) In general, $\tilde{C}^{-1}_{XX}$ does not exist when $\mathcal{F}_{X}$ is infinite dimensional, since $\tilde{C}_{XX}$ is a compact operator and thus has an arbitrary small positive eigenvalue [39]. When the matrix is singular, the matrix inversion could be unstable and the performance of the estimator $\tilde{C}_{P_{Y|X}}$ in [47] might be degraded after adding $\lambda I$ to avoid the singularity. 3) Even though the first two points could be relieved in some sense [43], the CMMD metric (2) is well-defined only if $C_{P_{Y|X}},C_{Q_{Y|X}}\in\mathcal{F}_{X}\otimes\mathcal{F}_{Y}$. However, this requires a much stronger assumption than the existence of $\tilde{C}^{-1}_{XX}$ (See Assumption 6 in Section A).

Bias: CMMD does not admit any obvious unbiased estimator. The estimator $\tilde{C}_{P_{Y|X}}$ in (4) is biased, even in the asymptotic sense if $\lambda$ is fixed [51].

To bypass the above limitations, we propose two alternative metrics which only require basic assumptions on the existence of the cross-covariance operator and the characteristic property of the kernels (Assumptions 1 and 2). In particular, we do not require the existence of the inversion of any operator or matrix, which makes our metrics easily implemented for real-world applications.

We first introduce a rather straightforward approach, which we term the AMMD metric. AMMD shows better potential for multi-output problems (such as image generation) where data consists of i.i.d. inputs $x_{i}$ with conditionally independent outputs $y_{i,j}$ at each $x_{i}$; see Section C for a more detailed discussion. In AMMD, at each $x$, we use (1) to build unbiased estimators of the MMD on the conditional distribution of $Y|X=x$. Then, these estimators are averaged with respect to the marginal $P_{X}$. More specifically, we define

where

is a function of $x$ and for fixed $x$, $Y^{x}_{1},Y^{x}_{2}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P_{Y|X=x}$, and $\hat{Y}^{x}_{1},\hat{Y}^{x}_{2}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q_{Y|X=x}$. Note that $\mu_{P_{Y|X=x}}$, $\mu_{Q_{Y|X=x}}\in\mathcal{F}_{Y}$ guaranteed by Assumption 1 so $MMD_{X=x}^{2}$ is well-defined. Hence $AMMD^{2}(P,Q)$ is also well-defined being the expectation with respect to non-negative measurable functions.

Theorem 4 shows that $AMMD^{2}(P,Q)$ offers a metric to measure $P_{Y|X}$ versus $Q_{Y|X}$. Next, we propose a Monte Carlo estimator of $AMMD^{2}$ for conditional generative models:

1. 1.

   Take a batch $\{(x_{i},y_{i,l}):i\in[n],l\in[r]\}$ from $P$ of batch size $rn$ where $y_{i,l}$ ($l\in[r]$) are the outputs at the same $x_{i}$. Here, $r$ is restricted by the specification of the task: $r=1$ in single-output problems but can be greater than $1$ in multi-output problems such as image generation; $n\geq 1$.

2. 2.

   Generate a batch $\{(x_{i},G(\xi_{i,j},x_{i})):i\in[n],j\in[m]\}$ from $Q$ of batch size $mn$ where $\xi_{1,1},\ldots,\xi_{n,m}$ are i.i.d. and independent of $x_{1},\ldots,x_{n}$; $m\geq 2$.

3. 3.

   Compute

   	$\displaystyle\hat{A^{2}}(P,Q)=$	$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\Big{(}-\frac{2}{mr}\sum_{j=1}^{m}\sum_{l=1}^{r}k_{2}(y_{i,l},G(\xi_{i,j},x_{i}))$
   		$\displaystyle+\frac{1}{m(m-1)}\sum_{j=1}^{m}\sum_{j^{\prime}\neq j,j^{\prime}=1}^{m}k_{2}(G(\xi_{i,j},x_{i}),G(\xi_{i,j^{\prime}},x_{i}))\Big{)}$		(6)

The next theorem establishes the error analysis of the estimator $\hat{A^{2}}(P,Q)$.

$C_{0}$ in Theorem 5 is free of the conditional generative model and thus does not need to be embodied at the training/evaluation phase. This is in the same spirit of NLL which is formed by a free-of-model constant and the Kullback–Leibler divergence between the model and data. Theorem 5 shows that if $n$ is not allowed to be large (e.g., $n$ is bounded above by the number of class in the label-based image generation problems), the variance of the estimator $\hat{A^{2}}(P,Q)$ should be reduced by increasing $m$ and $r$. On the other hand, if $n$ is allowed to be large (e.g., in regression problems with continuous features), then given a fixed computational budget, we should increase $n$ while maintaining the small possible values $m=2$ and $r=1$ to reduce the variance of $\hat{A^{2}}(P,Q)$ efficiently.

We then introduce the JMMD metric, which is based on the joint distribution. Compared with AMMD, JMMD is more suitable for single-output tasks (such as regression) where data consists of joint i.i.d. samples $(x_{i},y_{i})$; see Section C for a more detailed discussion. According to the observation in Lemma 1, the matching of $Q_{Y|X=x}$ (the conditional distribution of $G(\xi,X)|X=x$) with $P_{Y|X=x}$ for a.e. $x$ can be transferred to the matching of $Q_{X,Y}$ (the joint distribution of $(X,G(\xi,X))$) with $P_{X,Y}$. This motivates us to define the following metric which we term as JMMD:

where $k_{3}=k_{1}\otimes k_{2}$ is the kernel of the tensor product space $\mathcal{F}_{X}\otimes\mathcal{F}_{Y}$ and $(X_{1},Y_{1}),(X_{2},Y_{2})\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P_{X,Y}$ and $(\hat{X}_{1},\hat{Y}_{1}),(\hat{X}_{2},\hat{Y}_{2})\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q_{X,Y}$. Note that $JMMD^{2}(P,Q)$ can be viewed alternatively as the discrepancy of the cross-covariance operators $C_{P_{YX}}$, $C_{Q_{YX}}$ defined on the tensor product space $\mathcal{F}_{X}\otimes\mathcal{F}_{Y}$. Since $C_{P_{YX}}$, $C_{Q_{YX}}\in\mathcal{F}_{X}\otimes\mathcal{F}_{Y}$ guaranteed by Assumption 1, $JMMD^{2}(P,Q)$ is well-defined (see Section A for more details).

Theorem 6 shows that $JMMD^{2}(P,Q)$ offers a metric to measure $P_{Y|X}$ versus $Q_{Y|X}$. In parallel to AMMD, we propose a Monte Carlo estimator of $JMMD^{2}$ for conditional generative models: Take a batch of samples from $P$ of batch size $r$ $\{(x_{l},y_{l}):l\in[r]\}$; $r\geq 2$. Generate a batch from $Q$ of batch size $m$ $\{(\hat{x}_{j},G(\xi_{j},\hat{x}_{j})):j\in[m]\}$ where $\xi_{1},...,\xi_{m}$ are i.i.d. and independent of $x_{1},\ldots,x_{r},\hat{x}_{1},\ldots,\hat{x}_{m}$; $m\geq 2$. Compute

The next theorem establishes the error analysis of the estimator $\hat{J^{2}}(P,Q)$.

$C_{1}$ in Theorem 7 is free of the conditional generative model. Theorem 7 shows that the variance of $\hat{J^{2}}(P,Q)$ is decreasing at the order of $\frac{1}{\min\{m,r\}}$. Therefore, given a fixed computational budget $B$, we should set $m=\Theta(r)=\Theta(\sqrt{B})$ to achieve the minimum variance of the estimator $\hat{J^{2}}(P,Q)$.

We establish the theoretical connections among CMMD, JMMD, and AMMD as follows.

We further highlight the strengths of AMMD and JMMD on conditional generative model evaluation which is a challenging task due to delicacies of evaluation at the conditional distribution level. First, both metrics are "distribution-free", i.e., the data or conditional generative model are not restricted to a specific type of distributions. In contrast, FID for instance requires the Gaussian assumption. Second, they are "model-free", i.e., their evaluation does not involve additional estimated models beyond the conditional generative model itself, such as kernel density estimators in Indirect Sampling Likelihood [3, 13] or estimated discriminators [42, 1].

With the evaluation metrics, we present two corresponding deep-learning-based methods to construct conditional generative models. Our approaches are named as J-CGM and A-CGM, with special targets on the JMMD/AMMD for different tasks. For performance measurement, the values of JMMD and AMMD are estimated by drawing samples from the generative model optimized in J-CGM/A-CGM. Denote $G_{\theta}(\xi,X)$ as the generative model optimized in J-CGM/A-CGM with parameters $\theta$. Note that $G_{\theta}(\xi,X)$ takes both the given conditional variables $X$ and the extra random vector $\xi$ as inputs. Let $Q_{\theta}$ be the joint distribution of $(X,G_{\theta}(\xi,X))$. A detailed step-by-step pseudo-code of A-CGM is listed in Algorithm 1. A similar procedure for J-CGM is presented in Algorithm 2 in Section C.