ACLS: Adaptive and Conditional Label Smoothing for Network Calibration

To address the overconfidence problem, label smoothing [32] modifies the target distribution as follows:

where $\epsilon$ is a hyperparameter for controlling the degree of smoothing. It lowers the initial target probability $q_{y}$ for the true class $y$, while raising the probabilities for others. For better understanding its behavior, we compute the gradients of the objective function for label smoothing $\mathcal{L}_{\text{LS}}$ w.r.t the logit $z_{j}$, which is the same as the CE loss in Eq. (1) but with the target distribution of ${q}^{\prime}_{j}$, as follows:

where $\epsilon_{1}=\epsilon(1-1/C)$ and $\epsilon_{2}={\epsilon}/{C}$. In contrast to the CE, the probability of $p_{y}$ for the true class $y$ would be the same as that of $q_{y}-\epsilon_{1}$. The logit value $z_{y}$ will thus not be raised continually, mitigating the overconfidence problem. Note that the gradients for label smoothing in Eq. (5) reduce to those for CE in Eq. (3) with $\epsilon$ being zero. In the following, we will show that existing regularization-based methods can be considered as variants of label smoothing.

In contrast to LS [32], AR methods use an adaptive smoothing function. We can see in Fig. 1(d) that they adjust the degree of smoothing non-uniformly according to logit values. For example, the smoothing function increases in proportion to the logits, if $j=\hat{y}$, reducing the target labels of ${q}_{j}-f({z}_{j})$ in Eq. (8) (e.g., the fourth class in Fig. 1(d)). For $j\neq\hat{y}$, the smoothing function raises the target labels of ${q}_{j}+f({z}_{j})$ in Eq. (8) in accordance with the logits (e.g., from the first to third classes in Fig. 1(d)). In this case, the probabilities for the corresponding classes increase, making the probability of a network prediction, $p_{\hat{y}}$, relatively small. This reduces the overconfident probability (e.g., $p_{4}$ in Fig. 1(d)) more effectively than LS. However, we have observed that the smoothing function of AR often violates the desirable behaviors. For example, the smoothing function of MDCA [12] has a parabolic form, which is problematic. When the logit (or the softmax probability of $p_{j}$) is extremely large for $j=\hat{y}$ (the top in Fig. 1(e)), MDCA rather lessens the degree of smoothing, even worsening the overconfidence problem. Similarly, it lessens the degree of smoothing, when the logit is very small for $j\neq\hat{y}$ (the bottom in Fig. 1(e)). CPC [3] outputs negative values when $j=\hat{y}$. This in turn raises the target label of the overconfident probability, lessening the degree of smoothing and disturbing a calibration process.

Instead of using adaptive smoothing functions, a CR method [19] adopts a constant function as in LS [32], and performs label smoothing for miscalibrated probabilities only. In particular, it exploits a specific condition as follows:

where $M$ is a margin and we denote by $\mathbbm{1}[\cdot]$ a function whose output is 1 when the argument is true, and $0$ otherwise. We can see that the CR method [19] raises the target labels of ${q}_{j}+f({z}_{j})$ in Eq. (8) only if $z_{\hat{y}}-z_{j}$ is larger than the margin $M$ for $j\neq\hat{y}$. In other words, it determines that the probabilities $p_{j}$, where $j\neq\hat{y}$, are well-calibrated, if the logit difference between the network prediction and other classes is smaller than the margin $M$, and applies regularization for miscalibrated probabilities selectively. For example, the first class in Fig. 1(f) does not satisfy the condition in Eq. (9), and thus regularization is not applied, while label smoothing is employed by the amount of $\lambda_{2}$ for the second and third classes (See Table 2). However, the CR method [19] does not penalize the target label for the class $\hat{y}$ (i.e., $\mathbbm{C}({z}_{j})=0$ for $j=\hat{y}$) directly. This degrades the calibration performance, since the overconfidence problem is likely to occur in the largest probability of $p_{\hat{y}}$ (e.g., the fourth class in Fig 1(f)). Note that applying the softmax across $p_{j}$ for all $j$ in the CR method [19] could lower the probability of $p_{\hat{y}}$ for the class $\hat{y}$ implicitly. However, by reducing the target label of the class $\hat{y}$ explicitly, while raising the labels of other classes $j\neq\hat{y}$, similar to AR and ACR, we can lower the overconfident probability of $p_{\hat{y}}$ more effectively, mitigating the overconfidence problem.

CRL [25] has the merits of both AR and CR, adaptively penalizing target labels and preserving well-calibrated probabilities. However, it also inherits the drawback of AR. For example, the smoothing function of CRL [25] has a parabolic form, when $j=\hat{y}$ and $\mathbbm{C}=1$, which is the same as the function of MDCA [12]. This violates the desirable behaviors of the smoothing function (See the top in Fig. 1(e)). Note that the indicator function of CRL uses an ordinal ranking condition [4]. The ranking criterion first records how many times each sample is classified correctly during training (i.e., correctness). It then compares the ordinal ranking relationship based on the correctness by exploiting a ranking loss [35]. At the early phase of training, the correctness history is empty. Networks are thus trained using the CE loss only, causing the overconfidence problem. Moreover, the correctness increases, as the training process goes on. The ranking condition of the indicator function is thus easily not satisfied, making the regularizer inactive and degrading the calibration performance (See Sec 4.3).

To address the limitation of AR [12, 3], we set a smoothing function:

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters for $j=\hat{y}$ and $j\neq\hat{y}$ terms, respectively. For $j=\hat{y}$, our smoothing function provides outputs proportional to the logit values $z_{j}$, as it is piecewise linear w.r.t $z_{j}$. This indicates that our target label in Eq. (8), i.e., $q_{j}-f({z}_{j})$, decreases with an increment of the logit $z_{j}$, thereby lowering the probability $p_{\hat{y}}$. Similarly, if $j\neq\hat{y}$, the smoothing function outputs larger values, in accordance with the decrement in the logit $z_{j}$, which in turn raises the target label, i.e., $q_{j}+f({z}_{j})$, and the probability $p_{j}$. In summary, the piecewise linearity in our smoothing function enables always lowering the target label, as the probability $p_{\hat{y}}$ increases, and vice versa for other cases consistently, alleviating the limitation of AR.

We design an indicator function of the gradients in Eq. (8) as follows:

For $j\neq\hat{y}$, we adopt the margin criterion for MbLS [19] to avoid the limitations of an ordinal ranking condition. Different from MbLS, we also use an indicator function for $j=\hat{y}$, enabling adjusting the target label for the class $\hat{y}$, where the overconfidence problem is likely to occur. Specifically, when $j=\hat{y}$, the indicator function is $1$, if the difference between maximum and minimum logits is greater than the margin $M$, addressing the overconfidence problem of MbLS.

By putting our smoothing and indicator functions of Eq. (10) and Eq. (11), respectively, into the gradient of a regularization term in Eq. (8), we can obtain a novel form of gradients w.r.t the logit $z_{j}$ as follows:

By integrating the gradients, we obtain the loss function, ACLS, as follows:

Regularizers for network calibration satisfy the following conditions: (1) They could lower a probability of $p_{\hat{y}}$ for a class $\hat{y}$, i.e., the largest probability across classes, where the overconfidence problem is likely to occur. (2) They should penalize the probability of $p_{\hat{y}}$ selectively, only when it is miscalibrated, since the overconfidence problem for $p_{\hat{y}}$ does not always occur. ACLS in Eq. (13) satisfies these conditions. Specifically, it can reduce a logit for the class $\hat{y}$, lowering the probability of $p_{\hat{y}}$ for the class $\hat{y}$, and vice versa for other classes. The ReLU function with the margin $M$ determines whether to adjust the labels, regularizing logit values selectively.

We evaluate our method on CIFAR10 [15], Tiny-ImageNet [17], ImageNet [5] for image classification, and PASCAL VOC [7] for semantic segmentation. The CIFAR10 dataset consists of 50K training and 10K test images of size $32\times 32$ for 10 classes. Tiny-ImageNet is a subset of ImageNet, and provides 100K training, 10K validation, and 10K test images of 200 classes, where all images are downsampled to the size of $64\times 64$ from ImageNet. The ImageNet dataset provides approximately 1.2M training and 50K validation images of 1K classes. PASCAL VOC presents 10,582 training and 1,449 validation samples. Following the standard protocol in [10, 26, 19, 12, 25, 24], we report expected calibration error (ECE (%)), adaptive ECE (AECE (%)), and top-1 accuracy (ACC (%)) for image classification. For semantic segmentation, we report ECE (%), AECE (%), and mIoU (%).

We use network architectures of ResNet-50 and ResNet-101 [11] on CIFAR10 [15] and Tiny-ImageNet [17]. For these datasets, we train the networks using the SGD optimizer with learning rate, weight decay, and momentum of 0.1, 5e-4, and 0.9, respectively. The networks are trained for 350 and 100 epochs with a batch size of 128 and 64 for CIFAR10 and Tiny-ImageNet, respectively. We divide the learning rate by 10 at 150 and 250 epochs for CIFAR10 and at 40 and 60 epochs for Tiny-ImageNet. We train our models with a single NVIDIA RTX A5000 GPU. For ImageNet [5], we train ResNet-50 for 200 epochs using the AdamW optimizer [21], with batch size, learning rate, weight decay, $\beta_{1}$, and $\beta_{2}$ of 256, 5e-4, 5e-2, 0.9, and 0.999, respectively. We use a cosine annealing technique [20] as a learning rate scheduler, and train the network with four NVIDIA RTX A5000 GPUs. For semantic segmentation, we follow the experimental protocol in MbLS [19]. We use DeepLabV3 [2] with ImageNet pretrained ResNet-50 as a backbone and train the model with a single NVIDIA RTX A5000 GPU. We train the entire model using the SGD optimizer for 100 epochs with batch size, learning rate, momentum, and weight decay of 8, 1e-2, 0.9, and 5e-4, respectively. We divide the learning rate by 10 at 40 and 80 epochs.

We set the margin $M$ to 6 for CIFAR10, and 10 for other datasets, following [19]. To set the hyperparameters $\lambda_{1}$ and $\lambda_{2}$ in Eq. (10), we first divide 10% of the training samples for each dataset as a cross-validation split. We then perform a grid search on the split, and set the values of 0.1 and 0.01 for $\lambda_{1}$ and $\lambda_{2}$, respectively. For other regularization-based methods [3, 26, 32, 12, 19, 25], we use default hyperparameters, provided by the authors. We provide a detailed analysis on these parameters in the supplementary material.

We show in Fig. 2 reliability diagrams on Tiny-ImageNet [17] and ImageNet [5]. Following the works of [26, 3, 19, 12, 25, 10, 8, 6, 22], we visualize the reliability diagrams of average accuracies and confidences for 15 bins. We can see from Fig. 2 that ACLS outperforms the other methods [32, 19, 3, 12, 25] across the ranges of confidence. Specifically, we can see from the third, fourth, and last columns, ACLS shows better calibration capability than the AR method [12, 3], since they often behave in opposition to the desirable smoothing function, in terms of network calibration situation (See Sec. 3.2). The fifth and last columns show that the CR method (MbLS [19]) provides larger values of confidences, compared to ACLS, across all bins. A plausible reason is that it does not penalize target labels when $j=\hat{y}$ as in Eq. (9). We can see from the last two columns ACLS surpasses the ACR method [25] since ACLS alleivates the limitations of AR and CR.

We show in Table 5 an ablation analysis of variant of regularization methods on Tiny-ImageNet [17]. The third and fifth rows show the results of our method that use either a smoothing function of Eq. (10) or an indicator function of Eq. (11) only. From the table, we have the following observations: (1) AR and CR outperform LS by significant margins for both networks. This demonstrates that our smoothing and indicator functions in Eqs. (10) and (11) alleviate the miscalibration remarkably (See the first, third, and fifth rows). (2) We can clearly see that our smoothing function alleviates the limitation of previous AR methods [12, 3] by large margins (See the second and third rows). (3) The indicator function in Eq. (11) surpasses LS and MbLS for both architectures. This suggests that our indicator function calibrates the network more effectively than LS and MbLS (See the first, fourth, and fifth rows). (4) ACLS achieves the best performance, demonstrating that AR and CR are complementary to each other.

We compare in Table 6 ECE and AECE for indicator functions with margin or ranking conditions. To simulate the result in the second row, we set an indicator function as in Eq. (11), instead of using the original ranking counterpart in CRL [25]. Similarly, we use the ranking condition in CRL [25] for the third row. From the table, we have two findings: (1) For both CRL and ACLS, the methods exploiting the margin condition outperform the ranking counterparts consistently, demonstrating that the margin is more effective than the ranking condition in terms of network calibration as described in Sec. 3.2. The ranking condition compares the ordinal ranking relationship based on the correctness of each sample, where the correctness values are computed by counting the number of correct classifications for each sample during training. At the late phase of training, each sample has similar correctness value, suggesting that the the ordinal ranking relationship will not be satisfied, and preventing regularization. We can see from Fig. 3 that the regularizer is inactive (i.e., $\mathcal{L}_{\text{REG}}=0$) for 85% of samples, when exploiting ranking conditions (left), while only for 5% of samples for the margin condition (right). Furthermore, at the early phase of training, the correctness history is empty, and thus the networks are trained using softmax CE loss only. This also degrades the calibration of networks. (2) With the same condition, ACLS always outperforms CRL. This demonstrates once again that CRL still suffers from the limitation of AR, while ACLS mitigates the problem.

Similar to other regularization-based calibration methods [3, 12, 18, 19, 26, 25], ACLS requires re-training networks from scratch, which is computationally expensive. Also, ACLS might not behave as expected for an exceptional case. Let us suppose the case of $z_{j}>z_{k}>z_{i}$, i.e., $\min\mathbf{z}=z_{i}$ and $\max\mathbf{z}=z_{j}$, and assume that $z_{j}-z_{i}$ and $z_{j}-z_{k}$ are sufficiently large and small, respectively, and $y=j$. ACLS tries to reduce $z_{j}$, since $z_{j}-z_{i}$ is large enough. The ordering for $z_{j}$ and $z_{k}$ could then be altered, i.e., $z_{j}<z_{k}$, which in turn changes the prediction after a backward pass (i.e., $\hat{y}=k$ from $j$). This might degrade the classification performance. We have observed that this altering occurs in 0.29% of samples during the last epoch of training on Tiny-ImageNet [17].