Label-Aware Distribution Calibration for Long-tailed Classification

Given the trained backbone, LADC calibrates the distributions of tail classes with the help of relevant head classes. We assume that each class is Gaussian distributed and head classes are well represented with sufficient data for training. So the distribution of each head class can be approximated by calculating the mean and co-variance of the associated training data:

$$p(\bm{z})=N(\bm{z}|\bm{\mu},\bm{\Sigma})$$

(1)

$$\bm{\mu}=\frac{\sum_{i=1}^{n}\mathcal{F}(\bm{x}_{i})}{n}$$

(2)

$$\bm{\Sigma}=\frac{1}{n-1}\sum_{i=1}^{n}(\mathcal{F}(\bm{x}_{i})-\bm{\mu})(\mathcal{F}(\bm{x}_{i})-\bm{\mu})^{T}$$

(3)

where $\bm{\mu}$ and $\bm{\Sigma}$ denote the mean and variance of the class, and $n$ denotes the number of training instances in the head class.

To more accurately estimate the distribution of each tail class, we employ the statistics of a set of similar head classes $S$ as the prior and compute the posterior by using the feature $\bm{\hat{\mu}}$ of the tail class. The similar head classes $S$ is measured by the Euclidean distance in the feature space:

$$Set^{d}=\{-||\bm{\mu}^{i}-\hat{\bm{\mu}}||^{2}|i\in C_{h}\}\\$$

(4)

$$S=\{i|-||\bm{\mu}^{i}-\hat{\bm{\mu}}||^{2}\in topm(Set^{d})\}$$

(5)

Equipped with $S$, We can compute the prior distribution of $\bm{\hat{\mu}}$:

$$\bm{w}_{i}=\frac{n_{i}||\bm{\mu}^{i}-\hat{\bm{\mu}}||^{2}}{\sum_{j\in\mathbb{S}}n_{j}||\bm{\mu}^{j}-\hat{\bm{\mu}}||^{2}}$$

(6)

$$\bm{\mu}_{0}=\sum_{i\in S}\bm{w}_{i}\bm{\mu}^{i}\quad\bm{\Sigma}_{0}=\sum_{i\in S}(\bm{w}_{i})^{2}\bm{\Sigma}^{i}+\alpha\bm{I}$$

(7)

$$p(\bm{z})=N(\bm{z}|\bm{\mu}_{0},\bm{\Sigma}_{0})$$

(8)

where $\alpha$ is a hyper-parameter to control the degree of dispersion and $\bm{w}$ is the weight vector which is designed to impose tail classes to learn more from head classes that are more abundant and more similar.

For each tail class, $\mathcal{F}(\bm{x})$ conditioned on $\bm{z}$ is assumed to follow a Gaussian distribution

$$p(\mathcal{F}(\bm{x})|\bm{z})=N(\mathcal{F}(\bm{x})|\bm{z},\bm{L})$$

(9)

where $\bm{L}$ is the co-variance matrix and it is related to the uncertainty of the trained backbone.

Based on the prior distribution, the posterior (calibrated) distribution of each instance in the tail class can be given by

$$p(\bm{z}|\{\mathcal{F}(\bm{x_{i}})\}_{i})=N(\bm{z}|\bm{\mu}^{\prime},\bm{\Sigma}^{\prime})$$

(10)

By defining $\bm{L}=\beta\bm{\Sigma}_{0}$, we compute $\bm{\mu}^{\prime}$ and $\bm{\Sigma}^{\prime}$ as follows:

$$\bm{\mu}^{\prime}=\frac{\beta}{n_{s}+\beta}\bm{\mu}_{0}+\frac{n_{s}}{n_{s}+\beta}\hat{\bm{\mu}}$$

(11)

$$\bm{\Sigma}^{\prime}=\frac{\beta}{n_{s}+\beta}\bm{\Sigma}_{0}$$

(12)

where $n_{s}$ means the sample size for computing $\hat{\bm{\mu}}$ and is set as 1. $\beta$ is a hyper-parameter which can be interpreted as the relative uncertainty of the sample to the prior $\bm{\mu}_{0}$. Higher $\beta$ indicates a higher confidence of the prior.

In the second training stage, to balance the long-tailed distribution, we define the sampling probability of each class to be:

$$P_{i}=\frac{n_{1}/n_{i}^{\tau}}{\sum_{j}n_{1}/n_{j}^{\tau}}$$

(13)

where $n_{1}$ refers to the number of training instances of the most represented class. $\tau$ is a temperature parameter and it controls the balanced degree of the sampled distribution. Note that we re-sampling head classes from the original distributions; while for the tail classes, we draw samples from the calibrated distributions.

In Stage 2, Kang et al. (2019) introduce two methods for classifier adjustment: cRT and LWS. cRT re-trains the classifiers completely, while LWS retains the direction of the classifiers and only adjusts the scales. In the work, we choose LWS due to its superior performance in our experiments. On the basis of LWS and inspired by logits adjustment work (Menon et al. 2020; Tang, Huang, and Zhang 2020), we add a learnable bias term to the logits, named LWS-Plus, for adjusting the classification boundaries more flexibly:

$$\hat{\Phi}_{i}(\bm{x})=\bm{f}_{i}\cdot\Phi_{i}(\bm{x})+\bm{g}_{i}$$

(14)

where $\Phi_{i}(\bm{x})$ refers to the logit score of the class $i$, $\bm{f}_{i}$ denotes the scaling factor of the classifier magnitude, and $\bm{g}_{i}$ indicates the added bias.

In Stage 1, Mixup has proven to be an effective regularization strategy for training deep neural networks (Zhong et al. 2021; Zhang et al. 2021). More recently, Remix has been proposed for imbalanced data (Chou et al. 2020). In the work, we adopt Remix in the first training stage due to its better performance in the experiments. In addition, we combine RandAugment (Cubuk et al. 2020), a two-stage augmentation policy that uses random parameters in place of parameters tuned by AutoAugment.

We evaluate LADC on several popular long-tailed classification tasks, including both image and text data. Image datasets include CIFAR-10-LT, CIFAR-100-LT (Cui et al. 2019), ImageNet-LT (Cao et al. 2019) and iNaturalist (Van Horn et al. 2018). The text dataset includes THUNEWS-LT (Sun et al. 2016). Table 1 shows the detailed statistics of the datasets, where the “IF” indicates the imbalance factor, i.e., the ratio of the number of the most-represented class to the number of the least-represented class.

We choose several types of comparison methods: (1) one-stage training methods. They include Class-Balanced loss (Cui et al. 2019), Focal loss (Lin et al. 2017), CB-Focal loss (Cui et al. 2019), LDAM loss (Cao et al. 2019), Balanced Softmax (Ren et al. 2020), Meta-class-weight LDAM (Jamal et al. 2020) and a causal model Decoufound-TDE (Tang, Huang, and Zhang 2020). For the LDAM loss, we also consider the results with DRW strategy, i.e., LDAM-DRW. (2) two-stage training methods. The cRT, LWS (Kang et al. 2019) and MiSLAS (Zhong et al. 2021) reuse the original training data. (3) generative approaches. Feature space augmentation (FSA) (Chu et al. 2020), Major-to-Minor (Kim, Jeong, and Shin 2020) and MetaSAug-LDAM (Li et al. 2021) synthesize new training data. We also involve SMOTE (Chawla et al. 2002), a widely-used generative method for mitigating oversampling issue. (4) other baselines, including OLTR(Liu et al. 2019), BBN (Zhou et al. 2020), SSP (Yang and Xu 2020), Bag of tricks (Zhang et al. 2021) and Hybrid-PSC (Wang et al. 2021).

In the experiments, we regard the most frequent classes occupying at least 60% of the total training instances as head classes, and the remaining classes as tail classes. The number of head classes $m$ selected for computing $S$ is chosen from $\{2,3\}$. $\alpha$, $\beta$ and $\tau$ are selected from $0.1\sim 0.2$, $0.4\sim 1.3$ and $1.2\sim 1.3$, respectively. Detailed parameter analysis can be referred to the Appendix.

For CIFAR10-LT and CIFAR100-LT, we use ResNet-32 (He et al. 2016) as backbone following the work (Cao et al. 2019). We first train the backbone with representation enhancement for 400 epochs with five warm up steps. The base learning rate is 0.1 and we conduct a multi-step learning rate schedule which decreases learning rate by 0.01 at the $320^{th}$ and $360^{th}$ epochs. In the second training stage, we freeze the backbone and train classifier with LADC for 30 epochs during which the learning rate drops by 0.1 at the $10^{th}$ and $20^{th}$ epochs.

For ImageNet-LT dataset, we choose ResNet-10 and ResNet-50 as the backbone and train for 200 epochs in the first stage. The learning rate is initialized as 0.2 and decreases by 0.1 at the $120^{th}$ and $160^{th}$ epochs. For the iNaturalist dataset, we train our approach for 100 epochs employing the cosine learning rate schedule (Loshchilov and Hutter 2016) for 100 epochs with ResNet-50 in the first stage. The training strategy in the second stage is similar as that in training the CIFAR datasets.

We use SGD optimizer with the momentum at 0.9 and weight decay at $5\cdot 10^{-4}$ for all the experiments. The experimentation is run on a single NVIDIA Tesla V100 with 32GB graphic memory.

Table 2 shows the results on CIFAR10-LT and CIFAR100-LT. As can be seen, the proposed LADC consistently outperforms all the baselines. For the re-weighting approaches (i.e., the top 7 baselines), LADC increases the performance by at least 4.31% and 6.54% on average for CIFAR10-LT and CIFAR100-LT, respectively. Comparing with the best two-stage approach MiSLAS, LADC shows an improvement by 2.72% on CIFAR100-LT. LADC also performs better than the state-of-the-art generative approach MetaSAug, presenting an increase by 3.02% and 3.05% on average for the two datasets, respectively. The results indicate the effectiveness of the proposed approach in the long-tailed image classification.

We conduct experiments on two backbones, i.e., ResNet-10 and ResNet-50. As shown in Table 3, LADC achieves the best performance among all the approaches for different backbones. Specifically, LADC outperforms the baselines by at least 0.47% and 1.13% for the two backbones, respectively. The results imply that representations learned by a larger model could better benefit LADC.

Table 4 presents the results on large real-world dataset iNaturalist with ResNet-50. We can observe that LADC increases the accuracy of baseline approaches by 0.58%$\sim$8.21%, which shows the capability of LADC in handling large long-tailed datasets.

We choose a commonly-used subset of Chinese news dataset THUCNews (Sun et al. 2016) which contains ten classes¹¹1https://github.com/649453932/Chinese-Text-Classification-Pytorch. We build its long-tailed version with the imbalance factor at 100 and use the pre-trained embedding model provided by Li et al. (2018). We choose three models including TextCNN (Kalchbrenner, Grefenstette, and Blunsom 2014), TextRNN (Liu, Qiu, and Huang 2016) and Transformer (Vaswani et al. 2017) as the backbone. We remove the representation enhancement component due to the inapplicability of the data augmentation techniques for the text data. The results illustrated in Table 5 show that LADC achieves the highest accuracy, increasing the best baselines by 0.5%, 1.49% and 0.99% on three models, respectively. The results demonstrate the generalizability and effectiveness of LADC in other long-tailed modals.

We perform an ablation study to analyze the impact of each module in LADC. We follow the same two-stage training implementation throughout this study, and the results are shown in Table 6. Comparing the first two rows of the table, we can see that RE benefits the classification performance, as expected. In addition, we can observe that sampling from the calibrated distribution brings further improvement, as shown by the third row. Comparing the last two rows, we can achieve that the proposed LWS-Plus better rectifies the decision boundary than LWS.

Following Wang et al. (2020), we divide the classes into three groups according to the number of training instances associated with each class, i.e., Many-shot ($>$100 instances), Medium-shot ($100\sim 20$ instances) and Few-shot ($<$20 instances). The results of different groups are shown in Table 7. Comparing with the model trained by Cross-Entropy (CE), the baselines mainly focus on boosting the performance of the few-shot group with slight improvement on the medium-shot group. Besides, all the baselines expect for Remix show a performance degradation on the many-shot group. In contrast, the proposed LADC significantly enhances the accuracy on the few-shot and medium-shot groups, while achieving almost similar performance on the many-shot group as CE. The results indicate the effectiveness of LADC on both head and tail classes.

To analyze the benefit of our distribution calibration strategy, we compare with some alternative variants of our method. One variant is LADC_Average, indicating calibrating distribution for the whole tail class, i.e., $\bm{\hat{\mu}}$ in Equ. (11) is the mean feature of the tail class. In addition, we compare with the DC method in the few-shot learning field (Yang, Liu, and Xu 2021). Note that the sampling strategy is the same for all the comparisons. The results are listed in Table 8. As can be seen, our proposed LADC significantly outperforms the DC method, which demonstrates that LADC is more effective than DC in the long-tailed scenario. In addition, comparing LADC with LADC_Average, we can achieve that instance-based distribution calibration can better estimate the distributions of classes than class-based calibration.

Table 9 shows the results of different classifier re-balancing approaches in the second training stage. The experiments are also conducted on CIFAR100-LT. We choose class-balanced sampling (CBS), the state-of-the-art regularization method - label-aware smoothing (LAS) and our proposed label-aware distribution calibration combined with cRT and LWS, respectively, to re-balance the classifier in Stage 2. We can observe that LADC significantly improves the performance of class-balanced sampling method (CBS). Besides, LADC outperforms LAS under different training approaches in Stage 2. The results show the effectiveness of the proposed distribution calibration method.

Fig. 3 presents the visualization of feature distribution of training set and test set of CIFAR10-200. Visualization is conducted by adding a fully-connected layer ($D*2$ where $D$ is the dimension of features) which projects feature into 2-dimension space following Liu et al. (2016) Fig. 3 (a) shows the original training set, where we can see the tail classes (e.g., label 7,8,9) have few training instances and their distributions are significantly different from those in the test set, as illustrated in Fig. 3 (c).

After sampling via LADC, we can observe in Fig. 3 (b) that the distributions of tail classes are obviously extended and more closer to the test set. This indicates that the difference between the distributions of the original training set and test set is relieved by sampled instances. Meanwhile, LADC can well preserve the distributions of head classes.