Evaluating State-of-the-Art Classification Models Against Bayes Optimality

Benchmark datasets and leaderboards are prevalent in machine learning’s common task framework Donoho2019; however, this approach inherently relies on relative measures of improvement. It may therefore be insightful to be able to evaluate state-of-the-art (SOTA) performance against the optimal performance theoretically achievable by any model VarshneyKS2019. For supervised classification tasks, this optimal performance is captured by the Bayes error rate which, were it tractable, would not only give absolute benchmarks, rather than just comparing to previous classifiers, but also insights into dataset hardness HoB2002; ZhangWNXS2020 and which gaps between SOTA and optimal the community may fruitfully try to close.

Suppose we have data generated as $(X,Y)\sim p$, where $X\in\mathbb{R}^{d}$, $Y\in\mathcal{Y}=\{1,\dots,K\}$ is a label and $p$ is a distribution over $\mathbb{R}^{d}\times\mathcal{Y}$. The Bayes classifier is the rule which assigns a label to an observation $\mathbf{x}$ via

$\displaystyle y=C_{\text{Bayes}}(\mathbf{x}):=\operatorname*{arg\,max}_{j\in\mathcal{Y}}p(Y=j\mid X=\mathbf{x}).$

(1)

The Bayes error is simply the probability that the Bayes classifier predicts incorrectly:

$\displaystyle\mathcal{E}_{\text{Bayes}}(p):=p(C_{\text{Bayes}}(X)\neq Y).$

(2)

The Bayes classifier is optimal, in the sense it minimizes $p(C(X)\neq Y)$ over all possible classifiers $C:\mathbb{R}^{d}\rightarrow\mathcal{Y}$. Therefore, the Bayes error is a natural measure of ‘hardness’ of a particular learning task. Knowing $\mathcal{E}_{\text{Bayes}}$ should interest practitioners: it gives a natural benchmark for the performance of any trained classifier. In particular, in the era of deep learning, where vast amounts of resources are expended to develop improved models and architectures, it is of great interest to know whether it is even theoretically possible to substantially lower the test errors of state-of-the-art models, cf. CostelloF2007.

Of course, obtaining the exact Bayes error will almost always be intractable for real-world classification tasks, as it requires full knowledge of the distribution $p$. A variety of works have developed estimators for the Bayes error, either based on upper and/or lower bounds berisha16 or exploiting exact representations of the Bayes error noshad2019learning; NIELSEN201425. Most of these bounds and/or representations are in terms of some type of distance or divergence between the class conditional distributions,

$\displaystyle p_{j}(\mathbf{x}):=p(X=\mathbf{x}\mid Y=j),$

(3)

and/or the marginal label distributions $\pi_{j}:=p(Y=j)$. For example, there are exact representations of the Bayes error in terms of a particular $f$-divergence noshad2019learning, and in a special case in terms of the total variation distance NIELSEN201425. More generally, there are lower and upper bounds known for the Bayes error in terms of the Bhattacharyya distance berisha16; NIELSEN201425, various $f$-divergences moon14, the Henze-Penrose (HP) divergence Moon18; moon15, as well as others. Once one has chosen a desired representation and/or bound in terms of some divergence, estimating the Bayes error reduces to the estimation of this divergence. Unfortunately, for high-dimensional datasets, this estimation is highly inefficient. For example, most estimators of $f$-divergences rely on some type of $\varepsilon$-ball approach, which requires a number of samples on the order of $(1/\varepsilon)^{d}$ in $d$ dimensions noshad2019learning; poczos11. In particular, for large benchmark image datasets used in deep learning, this approach is inadequate to obtain meaningful results.

Here, we take a different approach: rather than computing an approximate Bayes error of the exact distribution (which, as we argue above, is intractable in high dimensions), we propose to compute the exact Bayes error of an approximate distribution. The basics of our approach are as follows.

• •

  We show that when the class-conditional distributions are Gaussian $q_{j}(\mathbf{z})=\mathcal{N}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma})$, we can efficiently compute the Bayes error using a variant of Holmes-Diaconis-Ross integration proposed in GaussianIntegralsLinear.

• •

  We use normalizing flows NormalizingFlowsProbabilistic; kingma2018glow; FetayaJGZ2020 to fit approximate distributions $\hat{p}_{j}(\mathbf{x})$, by representing the original features as $\mathbf{x}=T(\mathbf{z})$ for a learned invertible transformation $T$, where $\mathbf{z}\sim q_{j}(\mathbf{z})=\mathcal{N}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma})$, for learned parameters $\bm{\mu}_{j},\bm{\Sigma}$.

• •

  Lastly, we prove in Proposition 1 that the Bayes error is invariant under invertible transformation of the features, so computing the Bayes error of the approximants $\hat{p}_{j}(\mathbf{x})$ can be done exactly by computing it for the Gaussians $q_{j}(\mathbf{z})$.

Moreover, we show that by varying the temperature of a single flow model, we can obtain an entire class of distributions with varying Bayes errors. This recipe allows us to compute the Bayes error of a large variety of distributions, which we use to conduct a thorough empirical investigation of a benchmark datasets and SOTA models, producing a library of trained flow models in the process. By generating synthetic versions of standard benchmark datasets with known Bayes errors, and training them on SOTA deep learning architectures, we are able to assess how well these models perform compared to the Bayes error, and find that in some cases they indeed achieve errors very near optimal. We then investigate our Bayes error estimates as a measure of objective difficulty of benchmark classification tasks, and produce a ranking of these datasets based on their approximate Bayes errors.

We should note one additional point before proceeding. In general the hardness of classification tasks can be decomposed into two relatively independent components: i) hardness caused by the lack of samples, and ii) hardness caused by the internal data distribution $p$. The focus of this work is about the latter: the hardness caused by $p$. Indeed, even if the Bayes error of a particular task is known to be a particular value $\mathcal{E}_{\text{Bayes}}$, it may be highly unlikely that this error is achievable given a model trained on only $N$ samples from $p$. The problem of finding the minimal error achievable from a given dataset of size $N$ has been called the optimal experimental design problem ritter2000average. While this is not the focus of the present work, an interesting direction for future work is to use our methodology to investigate the relationship between $N$ and the SOTA-Bayes error gap.

Throughout this section, we assume the class conditional distributions are Gaussian: $q_{j}(\mathbf{x})=\mathcal{N}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma}_{j})$. In the simplest case of binary classification with $K=2$ classes, equal covariance $\bm{\Sigma}_{1}=\bm{\Sigma}_{2}=\bm{\Sigma}$, and equal marginals $\pi_{1}=\pi_{2}=\frac{1}{2}$, the Bayes error can be computed analytically in terms of the CDF of the standard Gaussian distribution, $\Phi(\cdot)$, as:

$\displaystyle\mathcal{E}_{\text{Bayes}}=1-\Phi\left(\tfrac{1}{2}\|\bm{\Sigma}^{-1/2}(\bm{\mu}_{1}-\bm{\mu}_{2})\|_{2}\right).$

(4)

When $K>2$ and/or the covariances are different between classes, there is no closed-form expression for the Bayes error. Instead, we work from the following representation:

$\displaystyle\mathcal{E}_{\text{Bayes}}$

$\displaystyle=1-\sum_{k=1}^{K}\pi_{k}\int\prod_{j\neq k}\mathbb{1}(q_{j}(\mathbf{z})<q_{k}(\mathbf{z}))\mathcal{N}(d\mathbf{z};\bm{\mu}_{k},\bm{\Sigma}_{k}).$

(5)

In the general case, the constraints $q_{j}(\mathbf{z})<q_{k}(\mathbf{z})$ are quadratic, with $q_{j}(\mathbf{z})<q_{k}(\mathbf{z})$ occurring if and only if:

$\displaystyle-(\mathbf{z}-\bm{\mu}_{j})^{\top}\bm{\Sigma}^{-1}_{j}(\mathbf{z}-\bm{\mu}_{j})-\log\det\bm{\Sigma}_{j}<-(\mathbf{z}-\bm{\mu}_{k})^{\top}\bm{\Sigma}_{k}^{-1}(\mathbf{z}-\bm{\mu}_{k})-\log\det\bm{\Sigma}_{k}.$

(6)

As far as we know, there is no efficient numerical integration scheme for computing Gaussian integrals under general quadratic constraints of this form. However, if we further assume the covariances are equal, $\bm{\Sigma}_{j}=\bm{\Sigma}$ for all $j=1,\dots,K$, then the constraint (6) becomes linear, of the form

$\displaystyle\mathbf{a}_{jk}^{\top}\mathbf{z}+b_{jk}>0,$

(7)

where $\mathbf{a}_{jk}:=2\bm{\Sigma}^{-1}(\bm{\mu}_{j}-\bm{\mu}_{k})$ and $b_{jk}:=\bm{\mu}_{k}^{\top}\bm{\Sigma}^{-1}\bm{\mu}_{k}-\bm{\mu}_{j}^{\top}\bm{\Sigma}^{-1}\bm{\mu}_{j}$. Thus expression (5) can be written as

$\displaystyle\mathcal{E}_{\text{Bayes}}$

$\displaystyle=1-\sum_{k=1}^{K}\pi_{k}\int\prod_{j\neq k}\mathbb{1}(\mathbf{a}_{jk}^{\top}\mathbf{z}+b_{jk}>0)\mathcal{N}(d\mathbf{z};\bm{\mu}_{k},\bm{\Sigma}).$

(8)

Computing integrals of this form is precisely the topic of the recent paper GaussianIntegralsLinear, which exploited the particular form of the linear constraints and the Gaussian distribution to develop an efficient integration scheme using a variant of the Holmes-Diaconis-Ross method hdr-ref1. This method is highly efficient, even in high dimensions¹¹1Note that the integrals appearing in (8) are really only $(K-1)$-dimensional integrals, since they only depend on $K-1$ variables of the form $\mathbf{a}_{jk}^{\top}\mathbf{x}+b_{jk}$.. In Figure 1, we show the estimated Bayes error using this method on a synthetic binary classification problem in $d=784$ dimensions, where we can use closed-form expression (4) to measure the accuracy of the integration. As we can see, it is highly accurate.

This method immediately allows us to investigate the behavior of large neural network models on high-dimensional synthetic datasets with class conditional distributions $q_{j}(\mathbf{z})=\mathcal{N}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma})$. However, in the next section, we will see that we can use normalizing flows to estimate the Bayes error of real-world datasets as well.

Normalizing flows are a powerful technique for modeling high-dimensional distributions NormalizingFlowsProbabilistic. The main idea is to represent the random variable $\mathbf{x}$ as a transformation $T_{\phi}$ (parameterized by $\phi$) of a vector $\mathbf{z}$ sampled from some, usually simple, base distribution $q(\mathbf{z};\psi)$ (parameterized by $\psi$), i.e.

$\displaystyle\mathbf{x}=T_{\phi}(\mathbf{z})\hskip 14.22636pt\text{ where }\hskip 14.22636pt\mathbf{z}\sim q(\mathbf{z};\psi).$

(9)

When the transformation $T_{\phi}$ is invertible, we can obtain the exact likelihood of $\mathbf{x}$ using a standard change of variable formula:

$\displaystyle\hat{p}(\mathbf{x};\theta)=q(T^{-1}_{\phi}(\mathbf{x});\psi)\left|\det J_{T_{\phi}}(T^{-1}_{\phi}(\mathbf{x}))\right|^{-1},$

(10)

where $\theta=(\phi,\psi)$ and $J_{T_{\phi}}$ is the Jacobian of the transformation $T_{\phi}$. The parameters $\theta$ can be optimized, for example, using the KL divergence:

$\displaystyle\mathcal{L}(\theta)=D_{\text{KL}}(p(\mathbf{x})\;\|\;\hat{p}(\mathbf{x};\theta))\approx-\frac{1}{N}\sum_{i=1}^{N}\log q(T_{\phi}^{-1}(\mathbf{x}_{i}),\psi)+\log\left|\det J_{T^{-1}_{\phi}}(\mathbf{x}_{i})\right|+\text{const}.$

(11)

This approach is easily extended to the case of learning class-conditional distributions by parameterizing multiple base distributions $q_{j}(\mathbf{z};\psi_{j})$ and computing

$\displaystyle\hat{p}_{j}(\mathbf{x};\theta)=q_{j}(T_{\phi}^{-1}(\mathbf{x});\psi_{j})\left|\det J_{T_{\phi}}(T_{\phi}^{-1}(\mathbf{x}))\right|^{-1}.$

(12)

For example, we can take $q_{j}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma})=\mathcal{N}(\mathbf{z};\bm{\mu}_{j},\bm{\Sigma})$, where we fit the parameters $\bm{\mu}_{j},\bm{\Sigma}$ during training. This is commonly done to learn class-conditional distributions, e.g. kingma2018glow. This is the approach we take in the present work. In practice, the invertible transformation $T_{\phi}$ is parameterized as a neural network, though special care must be taken to ensure the neural network is invertible and has a tractable Jacobian determinant. Here, we use the Glow architecture kingma2018glow throughout our experiments, as detailed in Section 4.

In this work, we have proposed a new approach to benchmarking state-of-the-art models. Rather than comparing trained models to each other, our approach leverages normalizing flows and a key invariance result to be able to generate benchmark datasets closely mimicking standard benchmark datasets, but with exactly controlled Bayes error. This allows us to evaluate the performance of trained models on an absolute, rather than relative, scale. In addition, our approach naturally gives us a method to assess the relative hardness of classification tasks, by comparing their estimated Bayes errors.

While our work has led to several interesting insights, there are also several limitations at present that may be a fruitful source of future research. For one, it is possible that the Glow models we employ here could be replaced with higher quality flow models, which would perhaps lead to better benchmarks and better estimates of the hardness of classification tasks. To this end, it is possible that the well-documented label noise in standard datasets contributes to our inability to learn higher-quality flow models NorthcuttAM2021. To the best of our knowledge, there has not been significant work using normalizing flows to accurately estimate class-conditional distributions for NLP datasets; this in itself would be an interesting direction for work. Second, a major limitation of our approach is that there isn’t an immediately obvious way to assess how well the Bayes error of the approximate distribution $\mathcal{E}_{\text{Bayes}}(\hat{p})$ estimates the true Bayes error $\mathcal{E}_{\text{Bayes}}(p)$. Theoretical results which bound the distance between these two quantities, perhaps in terms of a divergence $D(p\|\hat{p})$, would be of great interest here.

As detailed in VarshneyKS2019, there may be pernicious impacts of the common task framework and the so-called Holy Grail performativity that it induces. For example, a singular focus by the community on the leaderboard performance metrics without regard for any other performance criteria such as fairness or respect for human autonomy. The work here may or may not exacerbate this problem, since trying to approach fundamental Bayes limits is psychologically different than trying to do better than SOTA. As detailed in Varshney2020, the shift from competing against others to a pursuit for the fundamental limits of nature may encourage a wider and more diverse group of people to participate in ML research, e.g. those with personality type that has less orientation to competition. It is still to be investigated how to do this, but the ability to generate infinite data of a given target difficulty (yet style of existing datasets) may be used to improve the robustness of classifiers and perhaps decrease spurious correlations.

1. 1.

   For all authors…

   1. (a)

      Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

   2. (b)

      Did you describe the limitations of your work? [Yes] See Section 5

   3. (c)

      Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5

   4. (d)

      Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

2. 2.

   If you are including theoretical results…

   1. (a)

      Did you state the full set of assumptions of all theoretical results? [Yes]

   2. (b)

      Did you include complete proofs of all theoretical results? [Yes] See the Appendix.

3. 3.

   If you ran experiments…

   1. (a)

      Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We are about to open source our code repo.

   2. (b)

      Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]

   3. (c)

      Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] The Bayes Error Estimator is pretty accurate as shown in Figure 1. The normalizing flow is in general slow to train multiple times on all datasets and report an error bar w.r.t. random seeds.

   4. (d)

      Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4

4. 4.

   If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

   1. (a)

      If your work uses existing assets, did you cite the creators? [Yes]

   2. (b)

      Did you mention the license of the assets? [No] The datasets and code repos we are using are well known and we provided the references so that readers can find the corresponding licenses.

   3. (c)

      Did you include any new assets either in the supplemental material or as a URL? [No] We are about to open source our code repo.

   4. (d)

      Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [No] The data we are using are all well-known open-sourced sets with proper licenses. Readers can refer to our citation for the details.

   5. (e)

      Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] To the best of our knowledge, the data we are using do not contain personally identifiable information or offensive content.

5. 5.

   If you used crowdsourcing or conducted research with human subjects…

   1. (a)

      Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

   2. (b)

      Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

   3. (c)

      Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]