Language Models as Emotional Classifiers
for Textual Conversation

Language modeling is a natural language processing (NLP) task in which a model is asked to learn underlying distribution and relation among word tokens in a specified vocabulary (jozefowicz2016exploring, ). It is common for a language model (LM) to be pre-trained on a large-scale, general purpose corpus before being fine-tuned for a specific NLP task. Leveraging a LM in such a fashion has shown to be effective for performing a variety of natural language understanding (NLU) and natural language inference (NLI) tasks including speech recognition (mikolov2010recurrent, ), machine translation (schwenk2012large, ), and text summarization (filippova2015sentence, ).

Until recently most LM architectures were based on recurrent neural networks (jozefowicz2016exploring, ). However, in late 2018, Devlin et al released a model they dubbed BERT (Bidirectional Encoder Representation from Transformers) (devlin2018bert, ). As the name suggests, BERT is a language model based on the transformer architecture which consists almost entirely of attention modules (vaswani2017attention, ). BERT was pre-trained on the BooksCorpus (zhu2015aligning, ) and English Wikipedia data sets using two pre-training tasks: 1) Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).

For the MLM pre-training task 15% of tokens in the input sequence were randomly masked, and BERT was asked to predict the missing tokens. For the NSP pre-training task, two sentences were concatenated together and then processed by BERT as a single input sequence. The [CLS] token, a special token prepended to all input sequences, was then extracted from the output and used to make a classification as to whether or not the two sentences appeared next to one another in the source document. BERT achieved state-of-the-art performance on a variety of NLU tasks including the popular GLUE, MultiNLI, and SQuAD benchmarks (devlin2018bert, ).

Shortly after BERT was released, Yang et al proposed XLNet, a generalized autoregressive pre-training method for LM’s (yang2019xlnet, ). XLNet was similar to BERT in that it was pre-trained on a large, general purpose corpus and based on the transformer architecture, bet differed from BERT on two key respects. First, XLNet was trained to make predictions over all permutations of the input sequence whereas BERT made predictions on the raw input sequence. Second, XLNet leveraged an autoregressive architecture whereas BERT employed an auto-encoder architecture. These modifications break the independence of the tokens in the input sequence assumed by BERT, theoretically allowing XLNet to learn more contextual knowledge. However, the autoregressive nature of XLNet means that generated tokens are only conditioned on tokens up to the current position in the input (to the left) whereas BERT is able to access context from both the left and right of the current position because of it’s auto-encoder architecture.

Semantic Role Labeling (SRL) is a fundamental NLP task in which a model is asked to identify the predicate-argument structure of a sentence, shedding light on “who” did “what” to “whom” in the input text. To this end, Shi et al proposed a BERT-based model for SRL in 2019 (shi2019simple, ). Their model was able to achieve state-of-the-art performance in a variety of SRL benchmarks without the use of external features, such as part-of-speech tags and dependency trees. The choice to not use any external features was a significant departure from previous work on SRL and demonstrated the extent to which BERT is able to model human language.

Graph neural networks (GNN’s) are a specialized group of deep learning architectures which are able to work with data represented in non-Euclidean domains, such as in a graph (wu2020comprehensive, ). While many specialized variants of the GNN have been proposed, one of the most popular GNN architectures is known as a Graph Convolutional Network (GCN) (kipf2016semi, ). A GCN is appealing for applications in which data is represented by a graph and embeddings for each node in the graph is desired. GCN’s, and many GNN’s in general, can be seen as a specialized message passing network in which the new embedding of a node is informed by it’s previous embedding, it’s neighbors, and it’s neighbors’ embeddings.

Emotional classification has been done over many modalities. Poria et al. (poria2018multimodal, ) and Tripathi et al. (tripathi2018multi, ) performed modality analyses for emotional classification to understand which modes of communication contain the most emotional signal. Their work involved creating many classifiers for different types of data such as video, audio, text, and motion capture. Each model was first trained on data from a single medium as well as different pairings of data from different modalities to compare the performance of unimodal and multimodal models. The two studies found that for unimodal models, performance from best to worst was as follows: text, audio, video, and motion capture. It is worth noting that textual data performed substantially better than the rest (with respect to models made by each researcher). As modalities were combined, model performance continued to increase, as did the computational complexity of the resulting system. A more detailed review of uni- and multimodal emotional models can be found in (poria2017review, ).

BERT has been used in both text and auditory systems to classify emotion. The EmotionX 2019 challenge (shmueli2019socialnlp, ) demonstrates its application in a text-only domain. The challenge was to create an emotional classifier. Eleven teams partook in the competition and seven submitted reports documenting their models. Five of those seven teams used BERT to generate contextual embeddings in their emotion classification system and outperformed the two teams who did not incorporate BERT in their system.

BERT has also been used to classify the emotion of audio in IEmoNet (heusser2019bimodal, ). IEmoNet was designed to be a modular system for classifying emotion, where each module can be trained (mostly) independent of the others. The system accepts an audio signal as input and extracts auditory emotional features from it. Concurrently, the audio is transcribed and fed into textual information system. The output from the text system is combined with the audio features extracted earlier and given to a classifier that determines the emotion of the original audio clip. BERT was incorporated into IEmoNet as it had the best performance when classifying based on text only. Furthermore, using BERT as a purely text-based emotion classifier achieved an accuracy only three percentage points lower than the complete IEmoNet model.

Our research differs from these prior uses of pre-trained LM’s towards emotion classification in two key regards. First, we analyze how changing the amount of context given to the model impacts classification accuracy. Second, we supplement the LM with a GCN which generates an additional representation of an utterance, informed by the predicate-argument structure of the utterance identified through SRL.

IEMOCAP (busso2008iemocap, ) captures multimodal emotional data. Ten actors were recruited to perform one-on-one scripted and improvised scenarios designed to show a specific emotion. The dataset includes video and audio recordings, motion capture data of the hands and face, as well as the original scripts and transcribed audio. While scenarios were aimed to elicit specific emotions, the emotional labels in the data set were assigned by six evaluators.

Evaluators categorized each utterance in the data set into one of ten emotions: neutral, happiness, sadness, anger, surprise, fear, disgust, frustration, excited, and other. Happiness, sadness, anger, disgust, fear, and surprise were used as labels because they are considered basic emotions according to (ekman1971constants, ). A neutral category was added because it was of interest to the creators. Lastly, frustration and excited labels were added because the creators believed they were important categories to accurately represent the data.

Each utterance was seen by three different evaluators. The assessments of the displayed emotion from every judge are logged in the data. It is common practice for this data set to filter out data points for which the annotators were unable to reach a consensus. After filtering, about 74% of the data available for analysis. Figure 1 shows the distribution of labels in scripted and improvised scenes for utterances in the filtered data set.

Although the IEMOCAP data set contains a total of nine possible emotional labels, researchers using IEMOCAP typically focus on four emotions: happiness, anger, sadness, and neutral.

EmotionLines (chen2018emotionlines, ) is a data set containing two smaller sets: EmotionPush and Friends. EmotionPush contains text conversations where each message is labeled with the emotion it projects. The Friends dataset contains scripts from the TV show Friends where each line is labeled with an emotional category. The EmotionX 2019 (shmueli2019socialnlp, ) challenge had researchers develop emotional classifiers with EmotionLines. After the competition was over, the Friends portion of the data set was posted for others to use. The complete EmotionPush data set can only be obntained via request.

The Friends data set has eight emotional categories: non-neutral, neutral, joy, sadness, anger, disgust, fear, and surprise. The data was labeled using Amazon Mechanical Turk. Each utterance in the data set was seen by five Turkers. The final emotional label assigned to the utterance was the emotion that received a majority vote by the annotators. The non-neutral category contains all utterances that had no majority vote. Figure 2 shows the distribution of utterances and labels in the Friends data set. The figure demonstrates that the data is rather skewed, with the majority being neutral utterances.

The show Friends revolves around six main characters. Thus, unlike in IEMOCAP, the conversations involve more than two people. This makes capturing context in a conversation more difficult. However, it also tests the robustness of the model to generalize to situations involving more than two speakers.

The EmotionX competition asked researchers to create models that categorize an utterance into four categories: joy, sadness, anger, and neutral. Therefore, it mimics the outputs typically used by IEMOCAP. However, the EmotionLines paper ((chen2018emotionlines, )), that we later compare our method to, performed an analysis on all output categories except non-neutral. As a result, we do not compare to the EmotionX results as we used more output categories.

The first step in our proposed methodology is the identification of the predicate-argument structure within an individual utterance. Here, we employ the BERT-based SRL model proposed by Shi et al and introduced above (shi2019simple, ). We extract the predicates and verbs identified by the BERT-based model in each utterance, disregarding other identified entities. Next, a graph is constructed to represent the predicate-argument structure identified in each utterance. For each predicate-argument set in an utterance, a node is created for both the predicate and argument and an edge is added to connect the two nodes. To increase connectivity of the graph we add an edge between node a and node b if the set of tokens represented by node a are a subset of those represented by node b. We note that this process is only performed on the utterance being classified, not any of the utterances provided as context.

To assess LM’s ability to identify the emotion of an individual utterance in a conversation, as opposed to a single, isolated utterance, we explore the LM’s performance when provided with a varying amount of preceding utterances as context. We frame the problem of emotion classification as a one-versus-all binary classification task for each emotion, so an auxiliary sentence must be constructed for all possible emotion labels. The utterance being classified and the preceding N context utterances are concatenated together, separated by the [SEP] token, and used as text A for the binary classification. The auxiliary sentence for each possible emotion label takes the form of “That statement expressed [EMOTION]” and is used as text B for the binary classification. The input sequence to be passed to the LM is formed by concatenating text A and text B as follows: $\textit{[CLS] text A [SEP] text B [SEP]}.$

The fine-tuning of vanilla transformer-based LM’s such as BERT and XLNet is a relatively straightforward procedure. We utilize pre-trained models provided by Hugging Face¹¹1https://huggingface.co/transformers/ in our experiments. Although a multi-class classification at heart, the LM will first make E binary predictions as to whether or not an utterance should be labeled with each of the E emotion labels and takes the emotion corresponding to the highest binary prediction as the overall label.

After the input sequence has been processed, but before each binary prediction is made, a graph is constructed in accordance with the predicate-argument structure identified above. The initial embedding of the $i^{th}$ node, $h_{i}^{0}$ is obtained by averaging the embeddings of the corresponding tokens in the LM’s output representation. This value is then projected to the graph embedding dimension, $d_{GCN}$, via weight matrix W. This process is described below in equation 1.

where $s_{i}=\{w_{0},...,w_{t}\}$ are tokens represented by node $i$, $h_{w_{j}}$ is the LM’s contextual representation of the token $w_{j}$, $W\in R^{d_{LM}xd_{GCN}}$ projects the embeddings, and $\sigma$ is an activation function.

Here, we use a GCN, described above, to process our predicate-argument graph (kipf2016semi, ). As such, information propagates through the graph in two phases: aggregation and combination. During aggregation, an intermediate representation of node $i$’s neighbors in layer $l$, $z_{i}^{l}$, is influenced by node $i$’s neighbors, $N_{i}$, and the embedding of those neighbors, $h_{j}^{l}$. The aggregation process is described in detail in equation 2, where $V^{l}$ is the adjacency matrix of the graph in layer l.

The combination phase is then executed to obtain a new embedding of node $i$ in layer $l+1$, $h_{i}^{l+1}$. The new embedding is informed by node $i$’s previous embedding, $h_{i}^{l}$, and the intermediate representation of $i$’s neighbors, $z_{i}^{l}$. The combination phase is described in detail in equation 3 below where $W^{l}$ is a weight matrix and $\sigma$ an activation function.

Then, multiplicative attention is used to obtain a unified representation of nodes in the graph (luong2015effective, ). The embedding of the [CLS] token is extracted from the output of the LM and the attention scores between the [CLS] token and each node in the graph are computed. The unified graph embedding, $h^{g}$, is then computed as the weighted average of each node in the graph in accordance with the attention scores. This process is described in equations 4 and 5 below where $h^{c}$ is the embedding of the [CLS] token, $W_{1}$ is a weight matrix, $h_{i}^{L}$ is the final embedding of node $i$, and $N_{i}$ is the neighborhood of node $i$.

The unified graph embedding, $h^{g}$, is then concatenated with the embedding of the [CLS] token, $h^{c}$, and passed through a dense layer to make the binary classification. The emotion corresponding to the binary prediction for each emotion is then taken as the predicted label for the utterance.

For experiments on the IEMOCAP data set, utterances for which less than two annotators agreed on a label were excluded. Furthermore, utterances labeled with an emotion other than anger, happiness, neutral, and sadness were disregarded as is common practice. Both BERT and XLNet were trained for 9 epochs. Our results obtained by providing BERT and XLNet with 0, 1, 2, 4, and 8 previous utterances as context are presented below in figure 4.

Table 1 compares our model with varying levels of context to prior models mentioned earlier.

For experiments on the Friends data set all utterances were considered. Both BERT and XLNet were trained for 11 epochs. Our results obtained by providing BERT and XLNet with 0, 1, 2, 4, and 8 previous utterances as context are presented below in figure 5.

Tables 2 and 4 compare our models with varying amounts of context over the entire dataset and per category with (chen2018emotionlines, ).

We see very strong performance gains when our methodology is applied to the IEMOCAP data set. Performance of the vanilla XLNet and BERT models peak when two and four utterances are provided as context, respectively, but degrades when eight utterances are provided as context. When our methodology is used, however, we see performance continue to increase as more utterances are provided as context. We hypothesize that the performance of the vanilla models degrades when more than four utterances are provided as context because the length of the input sequence approaches the maximum input length of both BERT and XLNet. As a result, we believe the models struggle to identify important syntactic and semantic information in each utterance as well as they can when dealing with shorter input sequences. The inclusion of our proposed GCN module appears to alleviate the issue encountered when dealing with longer sequences.

The results of applying our methodology to the Friends data set do not tell as definitive a story as the IEMOCAP results. We do not see a clear difference in the performance of either XLNet or BERT when our methodology is employed, nor do we see a trend in performance as the number of utterances provided as context increases. However, in general, our methodology does seem to provide an increase in performance for particular experimental settings. For example, when four utterances are provided as context, BERT sees roughly a 5% increase in unweighted average accuracy by using our methodology, but XLNet achieves the same level of performance whether our methodology is used or not. Furthermore, we observe improved unweighted accuracy metrics when our model is applied with both LM’s compared to when it is not applied. This would suggest that our model is not as sensitive to the distribution of labels in the data set compared to previous methods (shmueli2019socialnlp, ).

One possible explanation for the inconsistent performance on the Friends data set is that the models struggled to handle conversations containing multiple speakers. All conversations in the IEMOCAP data set involve exactly two participants, while conversations in the Friends data set all contain at least two participants. Another likely cause for the inconsistent performance is the imbalanced nature of the data set. For example, utterances with the neutral label account for roughly 45% of the data set while only 2% of utterances are labeled as fear.

In our experiments we exclusively took the output from BERT and XLNet’s final layer as the embedding for tokens in our input sequence. However, the output is actually the $13^{th}$ representation of the input sequence produced by the LM’s (recall that each LM is formed by stacking $N$ transformer modules, $N=12$ for the base models used in our experiments). Additionally, the models compute an attention matrix for the input sequence in each layer. We believe these 12 internal embeddings and attention matrices may contain additional emotional signal which could be used to further improve performance. Clark et al have recently demonstrated how particular layers in BERT, and specific attention heads in different layers, have different functionality, perhaps suggesting this is a fruitful path forward (clark2019does, ).

Another interesting path forward could include identifying the predicate-argument structure in utterances in the context, as well as in the utterance being classified. Having shown that performance improves when SRL is performed on only the utterance being classified, it is not outlandish to assume that the performing the same process on context utterances could further improve performance.

As mentioned above, we believe the multi-party nature (¿2 participants) of the conversations in the Friends data set is partially to blame for the inconsistent performance we observed. A brief inspection of the data revealed that in some conversations, a subset of participants will go off on a tangent, straying from the rest of the group. Not only does the model need to handle multiple participants, but also multiple topics of conversation. To this end, we believe it would be interesting to explore the effect of introducing participant-specific special tokens to our model, one for each participant. We believe this may allow the LM’s to better manage multiple participants, and perhaps even multiple topics of conversation.