Multi-Stage Coarse-to-Fine Contrastive Learning for
Conversation Intent Induction

CLCU encourages the model to treat an utterance as similar to its neighboring utterances and dissimilar to utterances that are not subsequent to it or that belong to other dialogues when doing contrastive learning on consecutive utterances. Consecutive utterances contain implicit categorical information, which benefits dialogue classification tasks (e.g., intent classification). Consider pairs 1 and 4 in Fig. 2 stage 1: We implicitly learn similar representations of I am looking for restaurants and Find me some restaurants, since they are both consecutive with What type of food do you like?. In contrast, SimCSE (Gao et al., 2021) does not enjoy these benefits by simply using Dropout as data augmentation. Although TOD-BERT also leverages the intrinsic semantics of dialogue by combining an utterance with its dialogue context as a positive pair, the context is often a concatenation of 5 to 15 utterances. Due to the large discrepancy in both semantics and data statistics between each utterance and its context, simply optimizing the similarity between them leads to less satisfying representations on many dialogue tasks. Just like the experimental results in DSE (Zhou et al., 2022), TOD-BERT can even lead to degenerated representations on some downstream tasks when compared to the original BERT (Devlin et al., 2019) model. Therefore, in the first stage, we adopt the method of DSE, which learns from dialogues by taking consecutive utterances of the same dialogue as positive pairs for contrastive learning, and directly use the model they trained on a large number of datasets as our first stage model.

In the second stage, a small number of labeled datasets in the same domain as the target data are used, which makes our model have stronger generalization ability. And adopt an objective that pulls neighboring instances together and pushes distant ones away in the embedding space to learn compact representations for clustering. To be specific, we first encode the utterances with the pre-trained model from stage 1. The inner product is then used as a distance metric to find the top-K nearest neighbors of each utterance $x_{i}$ in the embedding space, forming a neighborhood $N_{i}$. During training, for each minibatch of utterances $\mathrm{B}=\left\{x_{i}\right\}_{i=1}^{M}$ and each utterance $x_{i}\in\mathrm{B}$, we uniformly sample one neighbor $x_{i}^{\prime}$ from its neighborhood $N_{i}$. Then use data augmentation to generate $\tilde{x}_{i}$ and $\tilde{x}_{i}^{\prime}$ for $x_{i}$ and $x_{i}^{\prime}$ respectively. Here, $\tilde{x}_{i}$ and $\tilde{x}_{i}^{\prime}$ are treated as a positive pair. We then obtain an augmented batch $B^{\prime}=\left\{\tilde{x}_{i},\tilde{x}_{i}^{\prime}\right\}_{i=1}^{M}$ with all the generated samples. To compute contrastive loss, we construct an adjacency matrix $A^{\prime}$ for $B^{\prime}$, which is a 2M × 2M binary matrix where 1 indicates positive relations (either being neighbors or having the same intent label) and 0 indicates negative relations. Hence, the contrastive loss can writed as:

where $C_{i}=\left\{A_{ij}^{\prime}=1\mid j\in\{1,\ldots,2M\}\right\}$ denotes the set of instances having positive relation with $\tilde{x}_{i}$ and $\left|C_{i}\right|$ is the cardinality. $\tilde{e}_{i}$ is the embedding for utterance $\tilde{x}_{i}$, $\tau$ is the temperature parameter. $sim(.,.)$ is a similarity function on a pair of normalized feature vectors. Has the following advantages by introducing the notion of neighborhood relationships in contrastive learning: 1) Similar instances are pulled together and dissimilar instances are pushed away to achieve more compact clusters; and 2) known intents are naturally incorporated with the adjacency matrix.

Previous research efforts focused on integrating clustering with deep representation learning by optimizing a clustering objective defined in the representation space (Zhang et al., 2017; Shaham et al., 2018). Despite promising improvements, the clustering performance is still inadequate, especially in the presence of complex data with a large number of clusters. One possible reason is that, even with a deep neural network, data still has significant overlap across categories before clustering starts. Consequently, the clusters learned by optimizing various distance or similarity-based clustering objectives suffer from poor purity. Moreover, contrastive learning has recently achieved remarkable success in self-monitoring (Wu et al., 2018; Chen et al., 2020), as the name suggests, a contrastive loss is adopted to pull together samples augmented from the same instance in the original dataset while pushing apart those from different ones. This beneficial property can be leveraged to support clustering by scattering apart the overlapped categories. Hence, in the third stage, for target data, we adopt the method of joint clustering and contrastive learning to further fine-tune the model. The training framework stage3 is shown in Fig. 3. Among them, $x_{i_{1}}$,$x_{i_{2}}$ and $x_{j_{1}}$,$x_{j_{2}}$ are obtained by means of data augmentation. In this paper, we adopt the contextual augmenter data augment in the form of a pre-trained transformers to find the top-n suitable words of the input text for insertion or substitution. We used word substitution to augment the data and chose Bertbase and Robertabase to generate augmented pairs. And the overall objective is:

where ${L}_{CL}$ and ${L}_{Clu}$ are the loss functions of comparative learning and clustering, respectively. $\eta$ balances between the contrastive loss and the clustering loss of stage3, for simplicity, it is set to 10 in our experiment.

For this Track, one development dataset and two test datasets are provided. Each dataset consists of 1) some human-to-human conversations between a customer and an agent, and 2) a set of testing samples with corresponding intent annotations. The task1-utterances and task2-utterances are utterances in which ”intents” are non-empty and ”dialogue_acts” are ”InformIntent” in dialogues in each dataset, respectively. Detailed statistics of the dataset are summarized in Table 1.

Task 1 and task 2 are both evaluated by the following six metrics,: accuracy (ACC) (Huang et al., 2014), normalized mutual information (NMI), F1-score, Recall, Precision, and adjusted rand index (ARI). But the ACC is the primary metric used for ranking system submissions. Metrics dependent on reference intents will be computed using an automatic alignment of cluster labels to reference intent labels. For task 1, alignments will be computed based on turn-level reference intent labels. For task 2, to avoid the need to assign labels to turns in the input transcripts, alignments will be computed using classifier predictions on the set of utterances held out for evaluation. In both cases, 1:1 alignments between induced intents and reference intents will be computed using the Hungarian algorithm (Kuhn, 1955).

The final summary results of the test-banking dataset and the test-finance dataset for the two tasks are shown in Tables 2 and 3. Our model ranks first in the test sets for both subtasks. Compared with the baseline, where the baseline model in the above tables is the official provided baseline model mpnet by Track 2 of DSTC11. It is trained on a large and diverse dataset of over 1 billion training pairs. And the test ACC of our model is improved by 13.99% and 12.69% on tasks 1 and 2, respectively. It shows the effectiveness of our method.

In addition, to test the advantages of the proposed method on other dataset in this paper, we give the results of our method on the development dataset compared with some representative baseline models. We compared with the following baseline models: (1) GloVe (Pennington et al., 2014). One of the official baseline models provided by the Track 2 of DSTC11, which is a sentence-transformers model. It maps sentences or paragraphs to a 300 dimensional dense vector space and can be used for tasks like clustering or semantic search. (2) MPNet (Song et al., 2020). Another official baseline model provided by Track 2 of DSTC11, which is trained on a large and diverse dataset of over 1 billion training pairs. The base model is MPNet-base and the dimension is 768. (3) SimCSE (Gao et al., 2021). The model we chose is trained on 10⁶ randomly sampled sentences from English Wikipedia by unsupervised contrastive learning, and its base model is RoBERTa-large with a dimension of 1024. (4) DSE (Zhou et al., 2022). It is trained on a large number of training pairs constructed in the style of consecutive conversational sentences as positive pair using multiple conversational datasets. The base model is RoBERTa-large with a dimension of 1024. (5) SCCL (Zhang et al., 2021). Fine-tuning of the baseline model using the development dataset by joint training with clustering and contrastive learning, where the baseline model is the DSE-trained model described above.

As can be seen from the experimental results in Table 4, our model obtained the best results in both subtasks compared to several other representative baseline models, with accuracy rates of 75.68% and 85.76%, respectively. On task 1, the accuracy of our model outperformed GloVe by 55.1%, outperformed MPNet by 29.54%, outperformed SimCSE by 28.29%, outperformed DSE by 17.01%, and outperformed SCCL by 10.36%. On task 2, the accuracy of our model outperformed GloVe by 56.64%, outperformed MPNet by 25.53%, outperformed SimCSE by 28.02%, outperformed DSE by 21.56%, and outperformed SCCL by 7.42%. This illustrates the soundness of our approach.

Through ablation experiments with a random seed set to 42 in the clustering algorithm on the development dataset, the experimental results are shown in Table 5. When there is no stage1, stage2 and stag3 (this is the case is the baseline), the ACC of the test on task1 and task2 are 46.14%, 60.24% respectively. when stage1, stage2 and stage3 are performed, the results on task1 are improved by 12.45%, 12.12% and the results on task2 improved by 0.68%, 19.38%, and 5.48%, respectively. In addition, the TSNE visualization of the utterances representation of the model obtained at each stage on the development set is shown in Fig. 4. It can be observed from the figure that each cluster becomes more and more compact after each stage compared to the baseline model. That verifies the rationality of each stage.