Reformulating van Rijsbergen’s $F_{\beta}$ metric for weighted binary cross-entropy

The thought behind U & IU is to apply (flat) equal distribution for both $X_{1}$ and $X_{2}$. These assumed distributions are applied to $\beta^{\prime}$ and $\beta^{\prime\prime}$ as follows:

where $r\text{ and }r^{\prime}\in[0,1]$ and $\beta^{*}>0$. Note that for both distributions there is only one $\beta^{*}$ chosen and this value replaces the need to have an explicit form that includes $\beta$ as a parameter. This is for convenience, as well as noticing that both $\beta^{\prime}$ and $\beta^{\prime\prime}$ differ by a factor of $r$. So allowing $\beta^{*}$ to vary broadly (which is the $\beta_{max}$ in section 6) would be enough to balance this convenience tradeoff. Next is to derive the joint distribution which would be used in section 6. It can be shown (the proof is in Appendix B) that the joint distribution is:

To understand this flat mixture, consider Figure 1 - the CDF surface for a grid of precision and recall where $\beta^{*}\in[8,16]$. (Note: the blue and red heat coloring is from the CDF and highlights curvature and/or rate of change). For a lower z value of $0.4$, Figure 1(a) shows that $\beta^{*}=8$ has a faster rate of change as compared to $\beta^{*}=16$. The same conclusion is apparent in Figure 1(b), which is for a higher z value of $0.8$. For both figures more curvature is seen for lower $\beta^{*}$ values. This suggests that a larger $\beta^{*}$ value smooths the surface and is a better candidate for $\beta_{max}$ in the algorithm in Section 6.

A more informed distributional approach for $X_{1}$ and $X_{2}$ considers Ga & IE, respectively. The reason to use Gaussian distribution for $X_{1}$ is to allow a bell-shaped variability around a fixed $r^{\prime}$ that is based on $\beta^{\prime}$ and ultimately $\beta$. The weighting of $X_{1}$ by $X_{2}$ uses the Inverse Exponential distribution because with selections of the rate parameter $\lambda$ the distribution can shift mass from left to right as well as appear uniformly distributed around $r$. This provides practitioners enough flexibility on experimenting with different weights. The following shows the assumptions for $\beta^{\prime}$ and $\beta^{\prime\prime}$:

where $r$ in (8) is the location shift by recall from the definition of $X_{2}$ in (3) and $\sigma^{2}$ is the variability captured by $\beta^{\prime}$. Using both (7) and (8), the distribution for (3) is now split around $z=0$ as follows:

where $\Phi(x\mathchar 24635\relax\;\mu,\sigma^{2})$ denotes the standard normal or gaussian distribution at the value $x$ for a mean, $\mu$ and variance, $\sigma^{2}$. (Refer to Appendix C for the proof). Similar to before the focus is on the indicator $1_{z\geq 0}$ as defined in (9).

Since this distributional mixture has more flexibility due to more parameters, Figure 2 highlights this when $\lambda\in[0.5,2.0]$, and $\sigma^{2}\in[0.5,2.0]$. The probabilities are computed again at a lower z value, $0.4$, and at a higher z value, $0.8$ for comparison. For a fixed $\lambda$, varying $\sigma^{2}$ impacts the curvature of the surface with higher $\sigma^{2}$ values producing a flattening effect. Figure 2(a) shows this distinctly. Conversely, as $\lambda$ increases with a fixed $\sigma^{2}$, the rate at which the surface changes is very apparent. This can be seen by juxtaposing Figure 2(c) and 2(a) or Figure 2(d) and 2(b) and noticing that the increase of $\lambda$ produces a clear increase in the rate of change. These observations match the intuition that $\sigma^{2}$ is linked to the shape of the bell curve, and $\lambda$ is linked to a rate of change. It also serves as a basis of intuition behind the algorithm in Section 6. That is, a faster rate of change along with a curved (and/or smoother) surface would provide loss penalties that adapt quickly per batch using the aggregated information from precision and recall.

Now that probabilities, or $\text{Pr}(F_{\beta}\leq z)$ for some $z\in[0,1]$ are established in section 5.1 and 5.2, the goal is to use them to get an optimal $\beta$ value, $\beta_{opt}$. There are a couple things to consider. First, because $\beta$ is grouped into $\beta^{\prime}$ and $\beta^{\prime\prime}$ with distributional assumptions, using maximum likelihood estimation (MLE) is not particularly suitable here. Also, $\beta_{max}$, $\sigma^{2}$, and $\lambda$ from (4), (5), and (8) are set in advance and do not need to be estimated. Second, our observed data is only one data point per training batch, namely precision and recall. Given this and the natural bend of the $F_{\beta}$ function, a knee algorithm is applicable. From Satopaa et al. (2011), the knee of a curve is associated with good operator points in a system right before the performance levels off. This removes the need for complex system-specific analysis. Furthermore, they have provided a definition of curvature that supports their method being application independent – an important property for this paper. Algorithm 1 implements (and slightly alters) Kneedles algorithm from Satopaa et al. to detect the knee in the $F_{\beta}$ curve. Refer to Algorithm 1 for the formal pseudo code.

A brief explanation in plain words is as follows:

1. 1.

   For any training batch, compute precision (p) and recall (r). Then with a predefined $\beta_{max}$ value, set $n$ equally spaced values, $b_{s_{i}}$, up to $\beta_{max}$, and use section 5 to compute $p_{s_{i}}=\text{Pr}(F_{\beta}\leq z|\beta=b_{s_{i}},p=p,r=r)$. (This replaces step 1 from Satopaa et al.). Let $D_{s}$ represent this smooth curve as $D_{s}=\{(b_{s_{i}},p_{s_{i}})\in\mathbb{R}^{2}|b_{s_{i}},p_{s_{i}}\geq 0\}$ for $i=1,...n$.

2. 2.

   When $r<p$, convert to a knee by taking the difference of the probabilities from the maximum. That is, $p_{s_{i}}=\max(p_{s})-p_{s_{i}}$ for $i=1,...n$. This is necessary because of the formulation of the $F_{\beta}$ metric.

3. 3.

   Normalize the points to a unit square and call these $b_{sn}$ and $p_{sn}$.

4. 4.

   Take the difference of points and label that $b_{d}$ and $p_{d}$.

5. 5.

   Find the candidate knee points by getting all local maxima’s, label that $b_{lmx}$ and $p_{lmx}$.

6. 6.

   Take the average of $p_{lmx}$ and this will be $\beta_{opt}$. (This simplifies Satopaa et al.).

The weighted binary cross-entropy loss is primarily focused around the imbalanced use case where a minority class exists. This paper posits that from the shuffling of data observations, as is frequently done while training, relevant aggregate information is available to use from the batch. For instance, say for a fixed minority class observation, $y$, it is grouped among different batches of the majority class. The interaction effect of $y$ among these randomly varying training batches is often overlooked. It is this interaction that can be inferred through the precision and recall aggregates, then transferred as a penalty to the loss function via $\beta_{opt}$ in a probabilistic way. By using Algorithm 1 to get $\beta_{opt}$, the proposed loss is,

where the function $f_{i}(\textbf{x}\mathchar 24635\relax\;\theta)$ is the $i$-th element from the prediction of a neural network using the inputs x and training weights, $\theta$; and $y_{i}$ is the $i$-th element of the true target label. When considering the majority class, or $y_{i}=0$ for $i=\{1,...,m\}$, the loss is weighted by $(1+\beta^{2}_{opt})$. Therefore, for correctly predicted observations, the loss has a reduction by $(1+\beta^{2}_{opt})$. When incorrectly predicted, the loss is magnified by the same amount. For the minority class, or $y_{i}=1$ for $i=\{1,...,n-m\}$, the loss is unchanged. This is intentional because under imbalanced data there are far less observations, and computing precision and recall lead to numerical instability or frequent edge cases for Algorithm 1.

Figure 3 highlights the surface generated from Algorithm 1 leveraging probabilities from Section 5. First, the U & IU mixture or Figure 3(a) suggests that the shape of the surface remains relatively similar even when doubling $\beta_{max}$. This is an important point toward fixing $\beta_{max}=16$ for the Ga & IE. Based on Figure 3(a), the U & IU mixture penalizes more on the outskirts of recall, while the immediate penalties arise on the diagonal of the unit square. This suggests that precision and recall estimates from training cause immediate penalties when they are on opposite ends of the $[0,1]$ range as well as on the diagonal when these values start to even out. For Ga & IE mixture, Figures 3(b) and 3(c) show some similar conclusions as U & IU mixture along with additional insights. For a lower rate of $\lambda=0.5$, or Figure 3(b), diagonal spikes for $\sigma^{2}\in[0.5,2.0]$ as well as a precision centric penalty for higher $\sigma^{2}$ (i.e., $\sigma^{2}=2$) are seen. For a higher rate of $\lambda=2.0$, or Figure 3(c), a similar diagonal is retained for $\sigma^{2}\in[0.5,2.0]$ as in Figure 3(b). Furthermore, for increasing values of $\sigma^{2}$, the penalty evolves from precision centric to a vertical separation on the unit grid at around a precision of $0.4$. The overall interpretation is the following: for a lower $\lambda$, increasing $\sigma^{2}$ creates a slightly more precision based penalty; while for a higher $\lambda$, increasing $\sigma^{2}$ causes the penalty to become more balanced between recall and precision. The choice of these parameters are problem specific, but provides the practitioner flexibility in determining the best selection for their use case. On a separate note, from Figure 3, a spiky surface is obvious, which is partially explained by having a default setting in the algorithm. This is a strong sign of immediate and configurable penalties.

The origins of the $F_{\beta}$ metric come from text retrieval, so it is important to verify this method across different categories of data. In particular, image data from CIFAR-10¹¹1https://www.cs.toronto.edu/ kriz/cifar.html, text data from IMDB movie sentiments Maas et al. (2011) and structured/tabular data from the Census Income Dataset Dua and Graff (2019) are tested. For each experiment, the primary label (i.e., label 1) is either imbalanced or forced to be imbalanced to reflect real world scenarios. Because CIFAR-10 contains multiple image labels, the airplane label is the primary label and all others are combined. This yields a 10% class imbalance. IMDB movie sentiment reviews (positive/negative text) are not imbalanced. The positive sentiments in the training data are reduced to 1K randomly sampled sentiments yielding a 7.4% imbalance (Table 1). The Census Income Tabular data contains 14 input features (i.e., age, work class, education, occupation, etc) with 5 numerical and 9 categorical features. The binary labels are greater than 50K salary (label 1) and less than 50K salary (label 0). By default, greater than 50K salary is already imbalanced at 6.2%. The training and validation dataset sizes for each data category are as follows: for CIFAR-10, 50K training and 10K validation, for IMDB, 13.5K training and 25K validation, and for the Census data, 200K training and 100K validation. In terms of class imbalance, this paper considers an imbalance or proportion of label 1 under 10% to be significantly imbalanced and between 10% to 25% to be moderately imbalanced. Some heuristic rationale for a 10% imbalance, a model that has perfect recall or 100%, a precision of $p=\frac{1}{3}$ would be required to get $F_{1}=0.5$. In practical examples, this scenario can occur with weakly discriminative features. Therefore, this paper seeks to test this algorithm in scenarios that would need an improved precision. A variety of imbalanced and balanced scenarios will be tested in this paper.

Two real-life use cases related to cylindrical tanks are also considered, providing a physical domain to test Algorithm 1. Chauhan et al. (2022) developed an arithmetic optimizer with slime mould algorithm and Chauhan et al. (2023) developed an evolutionary based algorithm with slime mould algorithm; both algorithms focus on global parameter optimization. They tested these algorithms on several benchmark problems, one of which is called the pressure vessel design. The problem is a constrained parameter optimization (i.e., material thickness, and cylinder dimensions) for minimizing a cost function. This paper focuses on using the HAOASMA algorithm by Chauhan et al. in a simulation to convert the problem into a binary classification. The second use case is derived from Underground Storage Tanks (UST) and is also inspired by Chauhan et al.’s pressure vessel problem. The physical shape (i.e., the cylindrical shape) of USTs is similar to the pressure vessel design. USTs are used to store petroleum products, chemicals, and other hazardous materials underground. These structures could deform underground and possibly explain a false positive leak. Ramdhani (2016) and Ramdhani et al. (2018) explored parameter optimization of UST dimensions changing from cylindrical to ellipsoidal. The observed data are vertical (underground) height measurements, which can contain uniformly distributed error. Ramdhani et al. used these measurements and the volumetric equations (12), (13), (14), and (15) - derived from a cross-sectional view - to develop a methodology to estimate tank dimensions and test if the shape has deformed. The cross-sectional view can be seen in Figure 5.

The conversion of both of these real-life use cases into a classification involves establishing a baseline set of parameters to simulate data for label 0. Varying these parameters will allow simulation of data for label 1. For the pressure vessel design, the baseline parameters from HAOASMA are $T_{s}=1.8048$, $T_{h}=0.0939$, $R=13.8360$, and $L=123.2019$. To convert this to a classification, the parameters for thickness $T_{s}$ and $T_{h}$ are changed from the baseline while $R$ and $L$ dimensions are drawn from a normal distribution. By using values for $T_{s}$, $T_{h}$, $R$ and $L$, the cost function is computed using 11. These cost values concatenated with $R$ and $L$ arrays serve as the input to a neural network classifier. The label 1 reflect simulated data using $T_{s}$ and $T_{h}$ that are changed from the baseline. These variations are $T_{s}=1.7887$ and $T_{h}\in\{0.0313,0.2817\}$. Label 0 is HAOASMA baseline values $T_{s}=1.8048$ and $T_{h}=0.0939$. Appendix D provides the equations, the distributional plots seen in Figure 4, and a detailed explanation of the simulation procedure (Algorithm 2). For the UST problem, Ramdhani used a measurement error model with an error on the height measurement and another on the volume computation. The same model is used to simulate data in this paper. The baseline is a cylinder and the variations to vertical and horizontal axes $a$ and $b$ represent a deformed cylinder to an ellipse. By using $r$, $L$, $h$, $a$ and $b$ along with (12) and (14) or (13) and (15) the volume is computed. These volumes concatenated with noisy height measurements are the inputs to a neural network classifier. The label 1 reflect simulated data using the variations to the cylinder or the baseline. These variations are $a\in\{3.2,3.8\}$ and $b\in\{5.0,4.2105\}$. Label 0 will be the baseline cylinder with radius $r=4$ and the length of $L=32$. Refer to Appendix E for detailed explanation of the simulation Algorithm 3 along with comparison plots and volume equations.

The results in Table 1 compare the use of the loss function (10) by different models based on U & IU and Ga & IE to a baseline case of ordinary cross-entropy. All results shown in this table are computed on the validation datasets for each data category above (see above for the dataset sizes). For ease of presentation, $M_{1}^{\beta}$ is Model 1: U & IU from (5.1). $M_{2}^{(\lambda,\sigma^{2})}$ is Model 2: Ga & IE from (5.2). The superscripts $\beta$ and $(\lambda,\sigma^{2})$ are the parameters being explored. $M_{B}$ is the baseline or the same model network that is trained using ordinary cross-entropy.

Following the benchmark analysis from Aurelio et al., a similar approach is done for the Image, Text, and Tabular data. This expands the analysis from Table 1 to provide a more detailed and comprehensive view across various well-known datasets. The results can be found in Table 3, 4, and 5. The footnotes in these tables are explained as follows: the breakdown of train and test data sizes, the proportion of label 1, the labeling convention for label 1 versus label 0 (if multiple labels exist), and the location of the data, if necessary. For example, label 9 vs all means the label 9 is the label 1 and everything else is marked as label 0. Detailed explanations, links, and training details for all the datasets are provided in the footnotes for each table. At a high level, for images CIFAR-10, CIFAR-100 and Fashion MNIST are analyzed. For text, AG’s News Corpus, Reuters Corpus Volume 1, Hate Speech and Stanford Sentiment Treebank are analyzed. For the tabular data, 10 classical datasets from UCI repository are analyzed. Finally, the same model networks from Section 7.3 will be used.