A Distributional Evaluation of Generative Image Models

Generative models are increasingly important tools in artificial intelligence. Given a collection of independent training samples $X_{1},\cdots,X_{n}$ from some underlying target distribution $P$, such models learn to generate samples $\tilde{X}_{1},\cdots,\tilde{X}_{m}$ from some estimated distribution $\tilde{P}$ that approximates $P$. With the aid of powerful neural networks, deep generative models have enjoyed tremendous success in synthesizing realistic samples, especially in image and language domains where large amounts of training data are available.

A broad range of generative modeling frameworks have been proposed. From older approaches based on latent variables, such as the variational autoencoder (Kingma and Welling, 2014) and generative adversarial networks (Goodfellow et al., 2014), to more recent breakthroughs in diffusion models (Ho et al., 2020; Song et al., 2021) and flow-based models (Rezende and Mohamed, 2015; Kobyzev et al., 2021), model development has progressed at a rapid pace.iven these advancements, benchmarks and evaluation metrics play a crucial role in quantifying improvement, comparing models, and ensuring that progress in model development aligns with broader goals such as trustworthiness and fairness.

An important task in AI is to develop metrics that evaluate generative models in a statistically principled way. Image generative models are particularly challenging to evaluate. Current image generation metrics emphasize evaluating the realism of the synthetic images (Zhou et al., 2019). Human evaluation is often treated as the gold standard, where volunteers are asked to distinguish between synthetic and real images. If the accuracy of human classifications is low, the synthetic images are considered more realistic.

Beyond human evaluation, the most popular tool for evaluating generative AI models is the Fréchet Inception Distance (FID) (Heusel et al., 2017). The FID first uses a pretrained Inception v3 deep neural network to map each image to an embedding vector. It then compares the distributions of real image embeddings and synthetic image embeddings using a Fréchet distance, assuming that the embedding vectors are normally distributed. We review a comprehensive set of evaluation metrics for generative AI models in Section 2.

Given the increasing use of generative AI models for tasks in scientific and medical settings, it is imperative that the synthetic samples from the learned distribution reflect the true uncertainty and shape of the target distributions. In particular, the tails and moments captured by the synthetic samples should be close to those of the target distribution. This evaluation is particularly important with the rise of using synthetic data from generative AI models to augment training data for scientific and medical tasks, where the true training data has insufficient samples to train a data-hungry model (Angelopoulos et al., 2023; McCaw et al., 2024).

Here, we argue that current evaluation metrics are insufficient to account for the distributional mismatches between learned and target distributions. While approaches based on human evaluation are appropriate for evaluating the realism of synthetic images, it is well known in the cognitive science literature that humans exhibit biases and limited capacity when making judgments about randomness (Tversky and Kahneman, 1974; Williams and Griffiths, 2008). On the other hand, while the FID is an informative tool for measuring differences in means and covariances, the unrealistic normality assumptions used in FID limit its ability to properly account for differences in higher moments and tails between the learned and target distributions.

We propose the embedded characteristic score (ECS) as a complementary metric that gauges the distributional match between the learned and target distributions. The ECS consists of two main components—feature embeddings and characteristic function transforms—and is thus straightforward to implement and apply. Representing objects via feature embeddings in Euclidean space is a compelling idea that has found enormous success in the kernel testing literature and beyond (Gretton et al., 2012; Li et al., 2017). There are powerful pretrained neural network models that offer informative embedding maps for complex objects ranging from images to graphs. Once the embedding is computed, the characteristic function of each feature is then estimated at a designated point near the origin and used to compare the embeddings of synthetic and real images. The behavior of the characteristic function near the origin is closely related to the moments and tails of the corresponding distribution. We explore these connections in Section 3.

By using informative feature embedding vectors to represent images and comparing each feature’s estimated characteristic function, the ECS hopes to capture differences in tails and higher moments that standard evaluation metrics might miss. Direct estimation of the tails and higher order moments of a probability distribution can be difficult and costly – samples are often scarce at the tails, and standard estimators for higher order moments can fail to converge when the underlying distribution has heavy tails. The characteristic function, on the other hand, is a more stable quantity to estimate, since it exists for any distribution (even heavy-tailed ones), and is bounded and uniformly continuous. We use the behavior of the characteristic function near the origin as a useful proxy for the moment and tail information of distributions. The ECS can complement existing metrics, such as human evaluations, to provide a more comprehensive assessment of the performance of generative image models in real-world scenarios where normality assumptions are violated.

In addition to the theoretical characterizations of ECS in Section 3, we conduct simulations and an empirical study using publicly available image datasets to study the behavior of ECS in practice. Our simulations gauge the ability of ECS to faithfully capture differences in the tails of probability distributions under a controlled setting. We then demonstrate the practical use of ECS in an empirical study, where we compared synthetic images generated by a commonly used image generative model against real images used to train the model.

Generative modeling has a rich history in statistics. The terminology has been used to refer to a separate literature concerning generative versus discriminative approaches in statistical classification (Ng and Jordan, 2001; Jebara, 2012). Here we focus on generative modeling in the AI context, which puts more emphasis on synthesizing novel samples from a target distribution that is implicitly specified through training samples. There is a rich literature on the evaluation of generative models in AI, where much emphasis is placed on evaluating generative models in language and image domains. For language generative models, a large part of the literature focuses on evaluating performance in various downstream tasks, such as question answering, rather than directly assessing distributional match. A comprehensive review of language model evaluation is available in Guo et al. (2023).

A variety of metrics have been proposed for evaluating the quality of image generative models. We can broadly divide such metrics into two types – those that evaluate the individual properties of generated images, such as perceptual realism, as well as those that evaluate the collective properties of generated images, such as diversity.

Human evaluation of generated images, often treated as the gold standard for image generative model evaluation (Zhou et al., 2019), falls into the former category. The core idea is to use human evaluators to distinguish synthetic images from real ones (Denton et al., 2015; Rossler et al., 2019). Human Eye Perceptual Evaluation (HYPE) (Zhou et al., 2019) is a more refined metric that uses psychophysical principles to offer improved reliability and cost-efficiency. Their main approach is to display images to human evaluators with adaptive time constraints, so that the perceptual threshold for distinguishing real images from synthetic ones can be estimated.

While human evaluation can be effective at gauging the realism of individual images, they are less effective at assessing the collective and distributional properties of generated images (Tversky and Kahneman, 1974; Williams and Griffiths, 2008). The most commonly adopted approach for comparing the learned and target distributions is the Fréchet Inception Distance (FID) (Heusel et al., 2017). Here, an embedding vector is computed for each image using the Inception v3 neural network. The embedding vectors corresponding to real images are then compared to those corresponding to synthetic images under normality assumptions using a Fréchet distance. Another common approach for evaluation is the Inception Score (IS) (Salimans et al., 2016; Barratt and Sharma, 2018), which combines both individual and collective elements. Here, an auxiliary Inception model that is pretrained is used to classify the generated images. The confidence of the Inception model in making predictions is used as an indicator for realism, while the distribution of the model’s predictions is used as an indicator for diversity.

Numerous other evaluation approaches have been proposed based on different considerations, such as density estimation (Goodfellow et al., 2014), kernel techniques (Binkowski et al., 2018), and information theory (Jalali et al., 2024).

The FID and Inception Score have been criticized (Chong and Forsyth, 2020) as being statistically biased, in the sense that the computed score’s expectation over finite samples is not equal to the true estimand. Jayasumana et al. (2024) further question the FID’s normality assumptions, and propose an alternative approach that avoids the normality assumption based on contrastive language-image pretraining (CLIP) embeddings.

Systematic studies on the effectiveness of generative model evaluation metrics have also been performed (Xu et al., 2018; Theis et al., 2015; Betzalel et al., 2024). In particular, Stein et al. (2024) compared 17 metrics for evaluating generative models. Using extensive human evaluation experiments as a baseline, they showed that many popular metrics, such as the FID, do not correlate well with the conclusions drawn by human evaluators regarding the perceptual realism of images.

To the best of our knowledge, there are no current evaluation metrics for image generative models that compares synthetic versus real images by using the features’ higher moments and tails. In this work, we fill this gap with the Embedded Characteristic Score (ECS) in order to better understand the full distribution of a collection of synthetic samples. This can complement existing metrics to provide a more well-rounded assessment of generative image models.

In this section we give a precise definition of the embedded characteristic score, as well as related theoretical properties that motivate this definition.

Next, we evaluate ECS empirically. We first assess the ability of ECS to capture tail and higher order moment information in a simulation. Next, we show the behavior of ECS on the CIFAR10 and MNIST datasets with Inception v3 embeddings.

Deep generative models are often treated by practitioners as highly expressive (Doersch, 2016; Kingma et al., 2019) owing to the status of deep neural networks as universal function approximators (Cybenko, 1989; Barron, 1993; Hornik, 1991). Part of the folklore in the deep generative modeling literature is that, with enough training data and a large enough deep neural network, such models can generate arbitrarily accurate approximate samples from any target distribution (Hu et al., 2018). Recent work by Tam and Dunson (2025) showed that broad classes of commonly used deep generative models can only learn distributions that have light tails. This may lead to under-characterization of uncertainty when the target distribution is heavy-tailed. This simple observation seems to have evaded practitioners, in part because the metrics that are commonly used to evaluate such models ignore tail information. Developing evaluation metrics for generative models that take into account tail information can help practitioners better diagnose these problems.

In this work, we closed this evaluation gap using our embedded characteristic score (ECS) to evaluate the learned and target distributions based on higher order moments and distribution tails. We showed the behavior of ECS to better evaluate the simulated data from deep generative models versus commonly used metrics including the FID. While we focused on evaluating image generative models in this work, the approach presented here can in principle be applied to any other data modalities, as long as an appropriate data-specific embedding map $f:\mathcal{X}\to\mathbb{R}^{p}$ is available. Developing similar evaluation metrics for more complex generative tasks, such as conditional generation and multi-modality generation, remains an open challenge; however, the ECS suggests possible approaches to these more complex tasks.

Appendix

We conducted the entire empirical study in Python and PyTorch (Paszke et al., 2019). We conducted the Mardia Kurtosis test using a function implementation referenced from the open source repository https://github.com/quillan86/mvn-python. We used a function from the package Pingouin (Vallat, 2018) to perform the Henze-Zirkler test. The p values returned from these tests are too small to be represented by double precision floating points, hence we reported in Table 2 the machine epsilon value of $2.2\times 10^{-16}$.

Our implementation of the DC-GAN model (Radford et al., 2016) is directly referenced from the open source repository https://github.com/csinva/gan-vae-pretrained-pytorch. The pretrained weights for the DC-GAN models under both the MNIST and CIFAR10 datasets are also referenced from the same repository. The DC-GAN model architecture that is adopted in our implementation consists of $5$ 2D convolution layers in both the discriminator and generator networks.

In the empirical study, we obtained CIFAR10 (Krizhevsky, 2009) and MNIST (LeCun et al., 1998) as built-in datasets from the Torchvision package(Marcel and Rodriguez, 2010). We use the built-in pretrained Inception v3 model (Szegedy et al., 2016) from the Torchvision package to compute the Inception embeddings.

In our simulations in Section 4, we used the ECS to compare random features from a standard multivariate Gaussian distribution to random features from multivariate $t$ distributions with varying degree-of-freedom parameters $\text{df}=\{100,10,5,3,2.1\}$. Here, we visualize the differences in tail behavior between these distributions. We draw $10000$ independent random vectors with dimension $p=32$ from each of these $6$ distributions. We then visually compare the Gaussian vectors with each of the $5$ groups of multivariate $t$ vectors via two-dimensional PCA plots (Figure 2). We also perform a baseline Gaussian-to-Gaussian comparison by visualizing the original $10000$ Gaussian samples alongside another $10000$ Gaussian samples.

The visualizations indicate that, while the multivariate $t$ distribution with $\text{df}=100$ behaves very similarly to a Gaussian distribution, as we decrease the degree-of-freedom parameter the multivariate $t$ distribution exhibits more outliers and heavier tails, whereas the Gaussian samples exhibited no outliers.