Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning

Problem description. We describe the problem setup of our interest under a text classification scenario with $K$ classes. Let $\mathcal{D}$ denote the given training dataset consisting of tuples $(\mathbf{x},y_{\tt task})\in\mathcal{D}$ where $\mathbf{x}=[x_{1},\ldots,x_{L}]$ is the sequence of input tokens $x_{i}$, and $y_{\tt task}$ is the target task label. Our goal is to train a classifier $f_{\theta}:=W_{\tt task}\circ g_{\phi}$, composed with Transformer-based language model backbone $g_{\phi}$ (e.g., BERT (Devlin et al., 2019)) and a random initialized classification head $W_{\tt task}$, to minimize the task-specific loss $\mathcal{L}_{\tt task}$ such as a cross-entropy loss where $p(\mathbf{x})=\text{Softmax}\big{(}f_{\theta}(\mathbf{x})\big{)}$.

Preference learning. In this paper, we use a preference label between two data instances as an auxiliary learning signal to train the classifier. Specifically, the preference signals reflect the relative suitability between the two input samples with respect to the given task. We first assume that the preference labels of the given dataset are available. Then, our goal is to train a preference predictor to learn from the given human preferences, by predicting which one among the two input samples is more preferred. To this end, we formulate a preference learning as a supervised learning problem following the approaches in other domains such as reinforcement learning and generative modeling (Christiano et al., 2017; Ziegler et al., 2019; Lee et al., 2021). Given a pair of two different input tokens $(\mathbf{x}^{0},\mathbf{x}^{1})$ and task label $y_{\tt task}$, a preference label $y_{\tt pref}$ is additionally given; it indicates which input is preferred considering $y_{\tt task}$, i.e., $y_{\tt pref}\in\{0,1,0.5\}$, where 1 indicates $\mathbf{x}^{1}\succ\mathbf{x}^{0}$ (i.e., $\mathbf{x}^{1}$ is preferred than $\mathbf{x}^{0}$), 0 indicates $\mathbf{x}^{0}\succ\mathbf{x}^{1}$, and 0.5 implies an equally preferable case. Each preference label is stored in a dataset $\mathcal{D}$ as a quadruplet $(\mathbf{x}^{0},\mathbf{x}^{1},y_{\tt task},y_{\tt pref})$. Then, we predict a preference using the preference predictor ${h}_{\psi}$ following (Bradley & Terry, 1952):

where $\mathbf{x}^{i}\succ\mathbf{x}^{j}$ implies that input $i$ is preferable to input $j$. The underlying assumption of this model is that the probability of preferring an input depends exponentially on its output. Then, the preference predictor ${h}_{\psi}$ is trained through supervised learning with the given human preferences, by minimizing the binary cross-entropy loss as follow²²2Equally preferable case is learned with the same coefficients.:

Next, we present the specific techniques to train the classifier with given preference labels: (a) multi-task learning of classification and preference learning, (b) consistency regularization between classification and preference learning, and (c) informative pair sampling method based on the disagreement or inconsistency.

Multi-task learning with preference labels. To effectively learn from the given preference label $y_{\tt pref}$ and the task label $y_{\tt task}$, we train the classifier $f_{\theta}$ via multi-task learning (Ruder, 2017; Sener & Koltun, 2018) of both classification and preference learning. Specifically, we model the preference predictor ${h}_{\psi}$ in Eq. 1 upon the classifier $f_{\theta}$ similar to the case of $W_{\tt task}$. The preference prediction head $W_{\tt pref}$ is added on the output of Transformer backbone $g_{\phi}(\mathbf{x})$ and task label $y_{\tt task}$, i.e., $h_{\psi}(\mathbf{x},y_{\tt task})=W_{\tt pref}\circ[g_{\phi}(\mathbf{x});y_{\tt task}]$³³3$[A;B]$ means the concatenation between $A$ and $B$. where $f_{\theta}(\mathbf{x})=W_{\tt task}\circ g_{\phi}(\mathbf{x})$.

Preference learning with diverse multi-preference heads. In addition, we introduce multiple preference heads $\{W_{\tt pref}^{(t)}\}_{t=1}^{T}$ and trained with $\mathcal{L}_{\tt pref}$ in Eq. 2 to fully exploit the given preference labels. As obtaining preference labels requires additional cost, it is crucial to find effective ways to exploit them. By incorporating multiple preference prediction heads, we can obtain diverse learning signals from each preference label, based on their different random initialization (Ganaie et al., 2021). However, these multiple preference heads easily collapse into identical ones, as they are built on the compact representation of the pre-trained Transformer shared with the classification head. Hence, we introduced diversity regularization between $\{W_{\tt pref}^{(t)}\}_{t=1}^{T}$ during the training; we add a regularization $\mathcal{L}_{\tt div}$ to encourage the diverse prediction for each preference head by maximizing KL-divergence (Wang et al., 2021):

where $P_{\psi}(\mathbf{x}^{1},\mathbf{x}^{0};y_{\tt task})$ is the predictive distribution of the preference predictor ${h}_{\psi}$, i.e., $P_{\psi}(\mathbf{x}^{1},\mathbf{x}^{0};y_{\tt task})=\big{[}P_{\psi}[\mathbf{x}^{1}\succ\mathbf{x}^{0};y_{\tt task}],P_{\psi}[\mathbf{x}^{0}\succ\mathbf{x}^{1};y_{\tt task}]\big{]}$. Overall, we train the classifier with the following multi-task learning objective $\mathcal{L_{\tt multi}}$ under hyper-parameter $\lambda_{\tt div}$:

where $\mathcal{L}^{\psi^{(t)}}_{\tt pref}$ indicate the preference learning objective with each head $\psi^{(t)}$ and $\mathcal{L}^{\text{all}}_{\tt pref}=\sum_{t=1}^{T}\mathcal{L}^{\psi^{(t)}}_{\tt pref}$.

Consistency regularization between classification and preference learning. Even though multi-task learning is an effective way to train the model, it is still unclear whether or not the model can capture the relations between the two tasks explicitly. Accordingly, we hypothesize that a more preferred instance should have higher confidence from the classifier, i.e., $p_{y}(\mathbf{x}^{1})>p_{y}(\mathbf{x}^{0})$ if $\mathbf{x}^{1}\succ\mathbf{x}^{0}$ with the given task label $y$. Hence, to impose the model explicitly follows this intuition, we further propose a consistency regularization between the two tasks as follows:

Additionally, when the degree of preference is explicitly provided, i.e., $y_{\tt pref}\in[0,1]$ (see Section 4.3 of extractive preference case) rather than $y_{\tt pref}\in\{0,1,0.5\}$, we further extend this consistency regularization with margin $m$ which represents the degree of preference:

where $\Delta p_{y}(\mathbf{x}^{1},\mathbf{x}^{0})=p_{y}(\mathbf{x}^{0})-p_{y}(\mathbf{x}^{1})$. We note that the previous consistency regularization Eq. 5 becomes the special case of Eq. 6 with $m=0$. Overall, our training loss of the classifier is as follows:

where $\lambda_{\tt cons}$ is a hyper-parameter.

Selecting informative pairs. As the number of pairs of samples $(\mathbf{x}^{0},\mathbf{x}^{1})$ is proportional to the square of the number of training samples, it is difficult to obtain the preference label for all possible pairs, and even harder to learn from them even if we have all the preference labels. Hence, we propose the following advanced sampling scheme to maximize preference learning’s effectiveness during training: (1) Disagreement-based sampling, which selects pairs of instances with high variance across multiple preference predictors $\{{h}_{\psi^{(i)}}\}_{i=1}^{T}$, and (2) Inconsistency-based sampling, which seeks to reduce the mismatched pairs with high consistency loss $\mathcal{L_{\tt cons}}$ in Eq. 5. We evaluate the effects of these sampling methods in Appendix C.

Datasets. For the experiments, we first use the following four text classification datasets: (1) CoLA (Warstadt et al., 2019), (2) SMS Spam (Almeida et al., 2011), (3) Hate Speech (Fišer et al., 2018), and (4) Emotion (Saravia et al., 2018). In addition, to demonstrate the effectiveness of extractive preference, we investigate the publicly available datasets providing the annotation records and use the following six text classification datasets. DynaSent (Potts et al., 2021) is a dynamically constructed sentiment classification benchmark with ternary (positive/negative/neutral) sentiments; we use the dataset from the first round, (5) DynaSent-R1, and from the second round, (6) DynaSent-R2, the dataset for our experiments. Standford politeness corpus (Danescu-Niculescu-Mizil et al., 2013) is a binary classification benchmark for predicting whether the given sentence is polite or impolite. Since there are two different input domains within this benchmark, we split them into two different datasets: (7) Polite-Wiki from Wikipedia, and (8) Polite-SE from StackExchange, following the original setup. (9) Offensive agreement dataset (Leonardelli et al., 2021) is a binary classification benchmark for predicting whether the given sentence is offensive or not. (10) MultiNLI (Williams et al., 2018) is a crowd-sourced collection of sentence pairs annotated with textual entailment information; as the only validation set includes the annotation records, we split it into 8:1:1 for training, validation, and test sets. All datasets have the annotation records from 5 annotators for each sample. More details of datasets are presented in Appendix A.1.

Baselines. We first compare the proposed P2C to a naïve training with a cross-entropy loss and majority voted task label, denoted by (a) Vanilla. Then, since P2C with extractive preference can be viewed as a new way to utilize the annotation records, we compare this method with a wide range of disagreement learning methods in Section 4.3, as listed as follows; (b) Soft-labeling (Fornaciari et al., 2021): using the probabilistic distribution of annotations as soft labels for training; (c) Margin (Sharmanska et al., 2016): training the model with hinge loss by setting a margin proportional to the annotators’ agreements; (d) Filtering (Leonardelli et al., 2021): removing the training samples with a high disagreement. (e) Weighting (Uma et al., 2021): using weighted cross-entropy with smaller weights for the samples with high disagreements; (f) Multi-annotator (Davani et al., 2022): training the multiple classification heads for each annotation and using its ensemble for the evaluation. Furthermore, since we train the model with pair of samples, we also consider the baseline considering pair-wise training, (g) Class-wise Self-Knowledge Distillation (CS-KD) (Yun et al., 2020), which forces the similar predictive distribution between the same class samples to be similar. Lastly, we consider two regularization methods, (h) Label Smoothing (Müller et al., 2019) and (i) Max Entropy (Pereyra et al., 2017), as the baselines of P2C with generative preference in Section 4.2. Details are described in Appendix A.2.

Implementation details. All the experiments are conducted by fine-tuning RoBERTa-base (Liu et al., 2019) using Adam optimizer (Kingma & Ba, 2015) with a fixed learning rate 1e-5 and the default hyper-parameters of Adam. For all datasets, the model is fine-tuned using the specified training method with batch size 16 for 20 epochs. In the case of P2C, we use $T=3$ preference heads $\{W_{\tt pref}^{(i)}\}_{i=1}^{T}$ and 2-layer MLPs for each $W_{\tt pref}$. We choose hyper-parameters from a fixed set of candidates based on the validation set: $\lambda_{\tt cons},\lambda_{\tt div}\in\{1.0,0.1\}$. We sample the pair of instances with the same task labels for efficiency. With the extractive preference, we apply the consistency loss with margin (Eq. 6) by using the difference of annotation as the margin $m$. For other cases, we apply P2C with consistency loss without margin (Eq. 5) on the pre-defined pairs of samples. More details and experimental supports for the design choices can be found in Appendix A.3 and C, respectively.

In this section, we first evaluate our framework with the generative preference labels obtained from the pre-trained large language model, GPT-3 (Brown et al., 2020). To validate the effectiveness of P2C under the more challenging scenario, we use the following four datasets which have a skewed label distribution without annotation records: CoLA, SMS Spam, Hate Speech, and Emotion. Since their test datasets are also imbalanced, we measure the balanced accuracy (bAcc) (Huang et al., 2016) and the worst-group accuracy (wAcc) (Sagawa et al., 2020), to evaluate the generalization capability of the model, except CoLA since it is usually used with own metric, Matthews correlation coefficient (Mcc) (Chicco & Jurman, 2020). In addition, to measure the calibration of the trained model, we report Expected Calibration Error (ECE) (Guo et al., 2017). Here, we commonly adopt the temperature scaling to measure ECE following (Guo et al., 2017). As the annotation records are unavailable, we compare P2C with the baseline methods incurring the smoothed prediction of classifier only using the task label: Label Smoothing, Entropy Maximization, and CS-KD. In addition, we use $K$-shot prompting predictions of GPT-3 ($K=0,5,20$) as an additional baseline.

As shown in Table 3, the generative preference labels with P2C are consistently effective in improving the performance of the text classifier; for example, P2C exhibited 11.55% relative test error reduction on average compared to Vanilla while also improving the predictive calibration. At the same time, we note that P2C shows better performance than the considered baseline, which indicates that the training signal from the preference label is more than smoothing the prediction of the classifier. Finally, as shown in Table 3, P2C significantly outperforms GPT-3 baselines, which means that our framework does not just distill the ‘instance-wise’ knowledge of GPT-3, but obtains complementary information through the proposed ‘pair-wise’ comparisons.

While the generative preference is an efficient way to apply auxiliary preference learning with P2C using the moderate cost, it would be much better if one could still benefit from P2C for free. In this section, we evaluate the effectiveness of P2C with the extractive preference labels, which could be freely obtained from the existing benchmarks if the annotations records are available. We compare P2C with various disagreement learning schemes to fine-tune the RoBERTa-base classifier for each dataset, as they also utilize the annotation records for better training. Table 4 summarizes the results on six text classification datasets. Remarkably, P2C consistently outperforms the baseline methods for all six datasets. To be specific, P2C exhibits 7.59% relative test error reduction compared to the vanilla method in the average. Furthermore, compared to the previous best disagreement learning method of each dataset, P2C exhibits 4.27% relative test error reduction on average. These results show that extractive preferences successfully provide complementary training signals to the classifier from the pair-wise preference, and demonstrate the effectiveness of P2C as a training method to utilize the annotation records.

Next, on DynaSent-R2, we conduct additional experiments to verify how P2C improves the classifier. We first check whether the prediction of the trained model with P2C is similar to the annotators’ judgment, as the extractive preference labels come from annotation records. Specifically, we compare the L1 distance between the predictions of the model and the soft labels from the annotation records in Figure 3(a). We verify that P2C achieves the lowest distance to the soft labels, showing the validity of our preference learning for better modeling of the given task. Moreover, we verify that the calibration of the classifier is more improved than the baselines, as a result of pair-wise preference modeling. To be specific, we provide a reliability diagram (Yun et al., 2020), which plots the expected sample accuracy as a function of the confidence of the classifier in Figure 3(b). We remark that the plotted identity function (dashed diagonal) implies perfect calibration (Guo et al., 2017), and our method is the closest one among the baselines. This calibration effect of P2C is further verified through ECE in Figure 3(c).

To validate the effectiveness of the proposed component of P2C in Section 2.2, we perform the ablation experiments, and the results are presented in Table 5, as the extractive preference of all pairs is accessible. It is observable that diverse multi-preference heads improve the effectiveness of preference labels with better modeling compared to the single preference head (2-4th rows). In addition, consistency regularization between classification and preference heads enables the classifier to fully utilize the pair-wise training signal to solve the task, hence the performance is significantly improved (5th row). The performance is further improved by selecting the informative pairs during the training (6th row). More results are in Appendix C.

Lastly, we verify the effectiveness of collected subjective preference labels compared to other types of labels. To this end, we consider the scenario that the specific types of labels are additionally obtained on top of the existing task labels. Namely, task labels could be collected more with additional training samples, or preference labels between the existing samples could be obtained. Table 6 summarizes the experimental results. Here, it is observed that subjective preference labels are the most effective for improving the test accuracy ($\text{Acc}_{\tt avg}$) along with a better calibration effect. Remarkably, it is noticeable that the preference labels significantly improve the accuracy on relatively hard samples ($\text{Acc}_{\tt hard}$) regardless of its type, while the additional task labels are effective for the relatively easy samples.⁵⁵5We define the difficulty based on the disagreement of annotators, i.e., more disagree indicates more difficult. Somewhat surprisingly, one can observe that the additional 5k generative preference labels by GPT-3 are more effective than the same number of task labels, although the former is much cheaper to obtain than the latter; it indicates that our framework can serve as a new effective way to evolve the existing benchmarks along with the recent development of pre-trained large language models at a considerable cost.

While we primarily demonstrate the effectiveness of P2C on the text classification datasets, our approach has the potential to be applicable beyond text classification. To empirically verify such advantages, we have conducted additional experiments on an image classification task to validate our approach’s applicability. Specifically, we used the publicly available SUN Attribute dataset (Patterson et al., 2014) and constructed multiple binary scene attribute classification tasks from it, following the setups in (Sharmanska et al., 2016). Here, we considered the largest five attributes for the experiments. As this dataset includes annotation records, we constructed extractive preference labels to apply P2C. For experiments, we commonly trained ResNet-18 (He et al., 2016) from scratch, for 100 epochs using the SGD optimizer with a weight decay of 0.01 and a learning rate of 0.1 (decreased to 0.01 and 0.001 at 50 and 75 epochs, respectively).

In Table 7, one can observe that our P2C approach effectively improves the performance of the image classifier, with an average relative test error reduction of 6.66% compared to the Vanilla method. These results indicate that the effectiveness of our approach is not only limited to text classification and can be extended to broader applications.