How many classifiers do we need?

As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined, both to improve accuracy and to form more robust conclusions than any individual model alone can provide. In some cases, ensembling can produce substantial benefits, particularly when increasing model size becomes prohibitive. In particular, for large neural network models, deep ensembles LPB (17) are especially popular. These ensembles consist of independently trained models on the same dataset, often using the same hyperparameters, but starting from different initializations.

The cost of producing new classifiers can be steep, and it is often unclear whether the additional performance gains are worth the cost. Assuming that constructing two or three classifiers is relatively cheap, procedures capable of deciding whether to continue producing more classifiers are needed. To do so requires a precise understanding of how to predict ensemble performance. Of particular interest are majority vote strategies in classification tasks, noting that regression tasks can also be formulated in this way by clustering outputs. In this case, one of the most effective avenues for predicting performance is the disagreement JNBK (22); BJRK (22): measuring the degree to which classifiers provide different conclusions over a given dataset. Disagreement is concrete, easy to compute, and strongly linearly correlated with majority vote prediction accuracy, leading to its use in many applications. However, a priori, the precise linear relationship between disagreement and accuracy is unclear, preventing the use of disagreement for predicting ensemble performance.

Our goal in this paper is to go beyond disagreement-based analysis to provide a more quantitative understanding of the number of classifiers one should use to achieve a desired level of performance in modern practical applications, in particular for neural network models. In more detail, our contributions are as follows.

1. (i)

   We introduce and define the concept of polarization, a notion that measures the higher-order dispersity of the error rates at each data point, and which indicates how polarized the ensemble is from the ground truth. We state and prove an upper bound for polarization (Theorem 1). Inspired by the theorem, we propose what we call a neural polarization law (Conjecture 1): most interpolating (Definition 2) neural network models are 4/3-polarized. We provide empirical results supporting the conjecture (Figures 1 and 2).

2. (ii)

   Using the notion of polarization, we develop a refined set of bounds on the majority vote test error rate. For one, we provide a sharpened bound for any ensembles with a finite number of classifiers (Corollary 1). For the other, we offer a tighter-than-ever bound under an additional condition on the entropy of the ensemble (Theorem 4). We provide empirical results that demonstrate our new bounds perform significantly better than the existing bounds on the majority vote test error (Figure 3).

3. (iii)

   The asymptotic behavior of the majority vote error rate is determined as the number of classifiers increases (Theorem 5). Consequently, we show that we can predict the performance for a larger number of classifiers from that of a smaller number. We provide empirical results that show such predictions are considerably accurate across various pairs of model architecture and dataset (Figure 4).

In Section 2, we define the notations that will be used throughout the paper, and we introduce upper bounds for the error rate of the majority vote from previous work. The next three sections are the main part of the paper. In Section 3, we introduce the notion of polarization, $\eta_{\rho}$, which plays a fundamental role in relating the majority vote error rate to average error rate and disagreement. We explore the properties of the polarization and present empirical results that corroborate our claims. In Section 4, we present tight upper bounds for the error rate of the majority vote for ensembles that satisfy certain conditions; and in Section 5, we prove how disagreement behaves in terms of the number of classifiers. All of these ingredients are put together to estimate the error rate of the majority vote for a large number of classifiers using information from only three sampled classifiers. In Section 6, we provide a brief discussion and conclusion. Additional material is presented in the appendices.

In this section, we introduce notation that we use throughout the paper, and we summarise previous work on the performance of the majority vote error rate.

In this section, we introduce a new quantity, $\eta_{\rho}$, which we refer to as the polarization of an ensemble $\rho$. First, we provide examples as to what this quantity represents and draw a connection to previous studies. Then, we present theoretical and empirical results that show this quantity plays a fundamental role in relating the majority vote error rate to average error rate and disagreement. In Theorem 1, we prove an upper bound for the polarization $\eta_{\rho}$, which highlights a fundamental relationship between the polarization and the constant $\frac{4}{3}$. Inspired from the theorem, we propose Conjecture 1 which we call a neural polarization law. Figures 1 and 2 present empirical results on an image recognition task that corroborates the conjecture.

We start by defining the polarization of an ensemble. In essence, the polarization is an improved (smaller) coefficient on the Markov’s inequality on $\mathbb{P}_{\mathcal{D}}(W_{\rho}>0.5)$, where $W_{\rho}$ is the point-wise error rate defined as equation (1). It measures how much the ensemble is “polarized” from the truth, with consideration of the distribution of $W_{\rho}$.

Note that the polarization always takes a value in $[0,4]$, due to the positivity constraint and Markov’s inequality. Also note that ensemble $\rho$ with polarization $\eta_{\rho}$ is $\eta$-polarized for any $\eta\geq\eta_{\rho}$.

To understand better what this quantity represents, consider the following examples. The first example demonstrates that polarization increases as the majority vote becomes more polarized from the truth, while the second example demonstrates how polarization increases when the constituent classifiers are more evenly split.

In this section, we first present an upper bound on the majority vote error rate, $L(h_{\rho}^{\mathrm{MV}})$, in Theorem 2, using our notion of polarization $\eta_{\rho}$ which we introduced and defined in the previous section. Then, we present Theorems 3 and 4 which are the main elements in obtaining tighter upper bounds on $L(h_{\rho}^{\mathrm{MV}})$. Figure 3 shows our proposed bound offers a significant improvement over state-of-the-art results. The new upper bounds are inspired from the fact that classifier prediction probabilities tend to concentrate on a small number of labels, rather than be uniformly spread over all the possible labels. This is analogous to the phenomenon of neural collapse Kot (22). As an example, in the context of a computer vision model, when presented with a photo of a dog, one might expect that a large portion of reasonable models might classify the photo as an animal other than a dog, but not as a car or an airplane.

We start by stating an upper bound on the majority vote error, $L(h_{\rho}^{\mathrm{MV}})$ as a function of polarization $\eta_{\rho}$. This upper bound is tighter (smaller) than the previous bound in inequality (6) when the polarization is lower than $2$, which is the case for competent ensembles. The proof is deferred to Appendix B.2.

Based on the upper bound stated in Theorem 2, we add a restriction on the entropy of constituent classifiers to obtain Theorem 3. The theorem provides a tighter scalable bound that does not have explicit dependency on the total number of labels, with a small cost in terms of the entropy of constituent classifiers. The proof of Theorem 3 is deferred to Appendix B.3.

While Theorem 3 might provide a tighter bound than prior work, coming up with pairs $(M,\Delta)$ that satisfy the constraint is not an easy task. This is not an issue for a discrete ensemble, however. If $\rho$ is a discrete distribution of $N$ classifiers, then we observe that the assumption of Theorem 3 must always hold with $(M,\Delta)=(N\!+\!1,0)$. We state this as the following corollary.

See Figure 3, which provides empirical results that compare the bound in Corollary 1 with the C-bound in inequality (3), and with inequality (6) proposed in TKY⁺ (24). We can observe that the new bound in Corollary 1 is strictly tighter than the others. For more details on these empirical results, see Appendix C.

Although the bound in Corollary 1 is tighter than the bounds from previous studies, it’s still not tight enough to use it as an estimator for $L(h_{\rho}^{\mathrm{MV}})$. In the following theorem, we use a stronger condition on the entropy of an ensemble to obtain a tighter bound. The proof is deferred to Appendix B.4.

The condition (11) can be rephrased as follows: compared to the error $\mathbb{P}_{\rho}(h(x)\neq y)$, the entropy of the distribution of wrong predictions is small, and it is concentrated on a small number of labels. A potential problem is that one must know or estimate the smallest possible value of $\varepsilon$ in advance. At least, we can prove that $\varepsilon=\frac{K\!-\!2}{2(K\!-\!1)}$ always satisfies the condition (11) for an ensemble of $K$-class classifiers. The proof is deferred to Appendix B.4.

Naturally, this $\varepsilon$ is not good enough for our goal. We discuss more on how to estimate the smallest possible value of $\varepsilon$ in the following section.

In this section, our goal is to predict the majority vote error rate of an ensemble with large number of classifiers by just using information we can obtain from an ensemble with a small number, e.g., three, of classifiers. Among the elements in the bound in Theorem 4,

we plug in $\eta=\frac{4}{3}$ as a result of Theorem 1; and since $\mathbb{E}_{\rho}[L(h)]$ is invariant to the number of classifiers, it remains to predict the behavior of $\mathbb{E}_{\rho^{2}}[D(h,h^{\prime})]$ and the smallest possible value of $\varepsilon$, $\varepsilon_{\rho}=\frac{\mathbb{E}_{\mathcal{D}}\left[\mathbb{P}_{\rho^{2}}\left(h(X)\neq Y,h^{\prime}(X)\neq Y,h(X)\neq h^{\prime}(X)\right)\right]}{2\mathbb{E}_{\mathcal{D}}\left[\mathbb{P}_{\rho}\left(h(X)\neq Y\right)\right]}$. Since the denominator $\mathbb{E}_{\mathcal{D}}\left[\mathbb{P}_{\rho}\left(h(X)\neq Y\right)\right]=\mathbb{E}_{\rho}[L(h)]$ is invariant to the number of classifiers, and the numerator resembles the disagreement between classifiers, $\varepsilon_{\rho}$ is expected to follow a similar pattern as $\mathbb{E}_{\rho^{2}}[D(h,h^{\prime})]$. Note that the numerator of $\varepsilon_{\rho}$ has the same form as the disagreement, differing by only one less label. Both are $V$-statistics that can be expressed as a multiple of a $U$-statistic, as shown in equation (5). In the next theorem, we show that the disagreement for a finite number of classifiers can be expressed as the sum of a hyperbolic curve and an unbiased random walk. Here, $[x]$ denotes the greatest integer less than or equal to $x$ and $\mathcal{D}[0,1]$ is the Skorokhod space on $[0,1]$ (see Appendix B.5).

Theorem 5 suggests that the disagreement within $N$ classifiers, $\mathbb{E}_{\rho_{N}^{2}}[D(h,h^{\prime})]$, can be approximated as $\frac{N-1}{N}D_{\infty}$. From the disagreement within $M(\ll\!\!N)$ classifiers, $D_{\infty}$ can be approximated as $\frac{M}{M-1}\mathbb{E}_{\rho_{M}^{2}}[D(h,h^{\prime})]$, and therefore we get

Assume that we have three classifiers sampled from $\rho$. We denote the average error rate, the disagreement, and the $\varepsilon_{\rho}$ from these three classifiers by $\mathbb{E}_{3}[L(h)]$, $\mathbb{E}_{3}[D(h,h^{\prime})]$, and $\varepsilon_{3}$, respectively. Then, from Theorem 4 and approximation (13) (which applies to both disagreement and $\varepsilon_{\rho}$), we estimate the majority vote error rate of $N$ classifiers from $\rho$ as the following:

Alternatively, we can use the polarization measured from three classifiers, $\eta_{3}$, instead of $\eta=\frac{4}{3}$, to obtain:

Figure 4 presents empirical results that compare the estimated (extrapolated) majority vote error rates in equations (5) and (15) with the true majority vote error for each number of classifiers. ResNet18 models are tested on four different dataset: CIFAR-10, CIFAR-10.1, Fashion-MNIST XRV (17) and Kuzushiji-MNIST CBIK⁺ (18) where the models are trained on the corresponding train data. MobileNet How (17) is trained and tested on the MNIST Den (12) dataset. Not only do the estimators show significant improvement compared to the bounds introduced in Section 2.2, we observe that the estimators are very close to the actual majority vote error rate; and thus the estimators have practical usages, unlike the bounds from previous studies. In Figure 4(a2), existing bounds (3) and (6) are much larger compared to the average error rate. This is also the case for (architecture, dataset) pairs of other subplots.

This work addresses the question: how does the majority vote error rate change according to the number of classifiers? While this is an age-old question, it is one that has received renewed interest in recent years. On the journey to answering the question, we introduce several new ideas of independent interest. (1) We introduced the polarization $\eta_{\rho}$, of an ensemble of classifiers. This notion plays an important role throughout this paper and appears in every upper bound presented. Although Theorem 1 gives some insight into polarization, our conjectured neural polarization law (Conjecture 1) is yet to be proved or disproved, and it provides an exciting avenue for future work. (2) We proposed two classes of ensembles whose entropy is restricted in different ways. Without these constraints, there will always be examples that saturate even the least useful majority vote error bounds. We believe that accurately describing how models behave in terms of the entropy of their output is key to precisely characterizing the behavior of majority vote, and likely other ensembling methods.

Throughout this paper, we have theoretically and empirically demonstrated that polarization is fairly invariant to the hyperparameters and architecture of classifiers. We also proved a tight bound for majority vote error, under an assumption with another quantity $\varepsilon$, and we presented how the components of this tight bound behave according to the number of classifiers. Altogether, we have sharpened bounds on the majority vote error to the extent that we are able to identify the trend of majority vote error rate in terms of number of classifiers.

We close with one final remark regarding the metrics used to evaluate an ensemble. Majority vote error rate is the most common and popular metric used to measure the performance of an ensemble. However, it seems unlikely that a practitioner would consider an ensemble to have performed adequately if the majority vote conclusion was correct, but was only reached by a relatively small fraction of the classifiers. With the advent of large language models, it is worth considering whether the majority vote error rate is still as valuable. The natural alternative in this regard is the probability $\mathbb{P}_{\rho}(W_{\rho}>1/2)$, that is, the probability that at least half of the classifiers agree on the correct answer. This quantity is especially well-behaved, and it frequently appears in our proofs. (Indeed, every bound presented in this work serves as an upper bound for $\mathbb{P}_{\rho}(W_{\rho}>1/2)$.) We conjecture that this quantity is useful much more generally.