Do You Know My Emotion? Emotion-Aware Strategy Recognition towards a Persuasive Dialogue System

The Utterance Encoder vectorizes an input utterance. Given a historical conversation $C=(u_{1},u_{2},\dots,u_{N})$ a set of ${N}$ utterances, where $u_{i}=(x_{i,1},x_{i,2},\dots,x_{i,T})$ that consists of a sequence of $T$ words, $u_{N}$ indicates the utterance of the persuader, which is uesd to predict the persuasion strategy. A BiLSTM is utilized to encode each word $x_{i,t}$ in the utterance $u_{i}\in C$, leading to a series of context-aware hidden states $(\textbf{h}_{i,1},\textbf{h}_{i,2},\dots,\textbf{h}_{i,T})$, $\textbf{h}_{i,t}={\rm{concat}}[~{}\overrightarrow{\textbf{h}_{i,t}}~{};~{}\overleftarrow{\textbf{h}_{i,t}}~{}]$.

The last hidden state $\textbf{h}_{i,T}$ is considered to get the utterance-level representation. (Note: the representation of the [CLS] is used as the utterance-level representation in BERT-style encoders). Therefore, the set of $N$ utterances in $C$ can be represented as $\textbf{H}=(\textbf{h}_{1,T},\textbf{h}_{2,T},\dots,\textbf{h}_{{N},T})$.

To explore the semantic information at different granularity, the multi-head attention [31] is adopted as shown in Eq. (1). $\textbf{c}_{i}$ indicates the representation of $i$-th utterance:

In the feedback memory module, a randomly initialized strategy embedding is defined to represent the strategy features as $\textbf{S}\in\mathbb{R}^{L\times d}$, where $L$ is the number of the strategy labels and $d$ indicates the dimension. The strategy embedding will be continuously updated to capture persuasive strategy features. Specificly, CFO-Net selects the appropriate strategies (i.e. top-$k$) based on the context from strategy embedding and stores them into the strategy pool with the context-aware softmax function that shown in Eq. (2).

Strategy pool aims to process and store the possible historical strategies for future reference. As shown in Fig. 4, to achieve the selection of strategies and prevent gradient truncation, two-stream mask mechanisms are defined in the following:

• •

  $mask_{p}$: The selected strategies (i.e. top-$k$) are stored into the strategy pool to reserve the possible historical strategies (here, size is set to $10$).

• •

  $mask_{f}$: The best strategy of the current moment is stored into the feedback pool.

Specificlly, the module first outputs a probability distribution $\alpha$ of the strategy based on the contextual information, as:

Then, the $mask_{p}$ is obtained based on the $\alpha$ with the top-$k$ operation where $k$ is a hyper-parameter, and $mask_{f}$ is obtained when $k=1$. The strategies $\textbf{S}_{p}$ which contain multiple possible strategies are stored into the strategy pool, as:

where $\odot$ is element-wise multiplication, $(\cdot\otimes\mathbf{e}_{d})$ produces a matrix by repeating the vector on the left for $d$ times [33].

The strategies ${\textbf{S}}_{m}$ in the strategy pool are obtained by making a concatenation with the stored strategies $\textbf{S}_{p}$. Similarly, the strategy $\textbf{S}_{f}$ that stored into the feedback pool is formulated as:

The purpose of the feedback pool is to update the strategy weight $\gamma$ dynamically to record the emotional feedback of the persuadee towards the strategy. The tuple {strategy, emotion} stored in the pool calculates the strategy weight $\gamma\in\mathbb{R}^{L}$ that is used to obtain the subsequent emotion-aware strategy representation. Firstly, the representation of utterance $\textbf{c}_{i}$ is considered to predict the emotional label $\textbf{y}^{e}\in\{pos,neu,neg\}$ of the persuadee, as:

where $\textbf{c}_{N-1}$ indicates the ${(N-1)}^{th}$ utterance spoken by the persuadee.

Then, the weight $\gamma$ is assigned based on the score of the predicted emotion and stream $mask_{f}$. To enhance the generalization of the model, soft weight $\gamma\in\mathbb{R}^{L}$ (randomly initialized with an all-one vector at the first) can be defined as:

where the scalar parameter $\mu$ controls the proportion of $\exp^{-\zeta}$ that guarantees to be greater than zero. For the first condition, the weight of $\gamma$ increases when $\zeta$ becomes smaller. To this end, we intuitively set the confidence factor $\zeta$ that depends on the score of emotion $\textbf{y}^{e}$, as:

where ${y}^{e}_{x}$ is a scalar that indicates the score of the $x\in\{pos,neu,neg\}$ emotion. Finally, the emotion-aware strategy representation $\textbf{S}^{\prime}$ is modeled as:

An MLP can obtain the integrated representation automatically in a simple fashion, as:

The predicted distribution of the strategy $\textbf{y}^{s}$ can be defined as follows:

where $\textbf{W}^{s}\in\mathbb{R}^{L\times 2d}$ is transformation matrices, $\textbf{b}_{s}\in\mathbb{R}^{L}$ is the bias vector, $L$ is the number of the labels.

To achieve the fusion of two probability distribution, a double-head linear layer is designed for prediction. Specifically, we introduce two MLPs to calculate respective probabilities and then combine them, as:

where $\textbf{y}^{s}_{1}\in\mathbb{R}^{L}$ and $\textbf{y}^{s}_{2}\in\mathbb{R}^{L}$, $\textbf{y}^{s}$ is the final predicted distribution of the strategy.

Motivated by attention mechanism [19, 29, 26], the co-interactive attention layer is proposed to effectively model mutually relational dependency. In this layer, attentions are computed in two directions: from $\textbf{C}=(\textbf{c}_{1},\dots,\textbf{c}_{N})$ to $\textbf{S}^{\prime}=(\textbf{s}_{1}^{\prime},\dots,\textbf{s}_{L}^{\prime})$ as well as from $\textbf{S}^{\prime}$ to C.

Specifically, the layer first yields a shared similarity matrix $A\in\mathbb{R}^{{N}\times L}$, between C and $\textbf{S}^{\prime}$. $\textbf{A}_{ij}$ indicates the similarity between $i$-th context-aware utterance and $j$-th emotion-aware strategy, as:

where $\mathcal{F}$ is a dot product function, ${\bf C}_{:i}$ is $i$-th row vector of ${\bf C}$, and ${\bf S}^{\prime}_{:j}$ is $j$-th row vector of ${\bf S}^{\prime}$.

The attention weights and the attended vectors can be obtained in both directions. Firstly, considering the direction from $\textbf{S}^{\prime}$ to C, the attention weight is computed by ${\bf a}_{i}=\mathrm{softmax}({\bf A}_{i:})\in\mathbb{R}^{L}$, and subsequently context-aware utterance vector is $\tilde{{\bf C}}_{:i}=\sum_{j}{\bf a}_{ij}{\bf S}^{\prime}_{:j}$. Similarly, the attention weight ${\bf b}_{j}=\mathrm{softmax}({\bf A}_{:j})\in\mathbb{R}^{N}$, and updated emotion-aware strategy vector is $\tilde{{\bf S}}^{\prime}_{:j}=\sum_{i}{\bf b}_{ij}{\bf C}_{i:}$.

Finally, the context-aware utterance representation and emotion-aware strategy representation are combined to yield g and $\textbf{y}^{s}$ like Eq. (9) and Eq. (10), as:

State-of-the-art models are used as baselines to test the performance. Considering the advantages of pre-trained language models (PLMs), we replace the Bi-LSTM with RoBERTa [18] to strengthen the baseline for fair comparison, as with the work [4]. The base and large PLMs are used in the main experiments for a complete comparison. To increase training speed, the base PLMs are utilized in other experiments. Other baselines are shown in Table 1, [34] considered a hybrid RCNN model to extract textual features. [4] combined the PLMs with some state-of-the-art models to recognize the strategy of the persuader.

As depicted in Table 1, compared with state-of-the-art models and RoBERTa, the performance of our CFO-Net (with double-head linear layer) has gained a lot. The CFO-Net achieves 4.12% gain on Precision, 2.37% gain on Recall and 3.67% gain on M-F1 score compared with RoBERTa_large, which demonstrates that the psychological feedback of the persuadee is beneficial for the strategy recognition. The M-F1 reaches the decent performance with the RoBERTa DialogueRNN where four tasks are jointly trained, which shows that the CFO-Net can achieve better performance with fewer tasks. As for the emotion prediction task, the CFO-Net also improves the performance, which shows that jointly training the tasks can provide benefits and boost each other. This phenomenon illustrates that the emotional feedback of the persuadee has the potential to help the process of strategy recognition task. Our code will be released in the link. ²²2The codes are available at:  https://github.com/pengwei-iie/CFONETWORK

In this setting, the feedback memory module is abandoned for exploring the effectiveness of the psychological feedback. From the result, the performance has declined significantly in all metrics, which confirms our hypothesis that introducing the emotion of the persuadee to the strategy recognition is important.

Multi-task learning considers the mutual connection between tasks by sharing latent representations. Here, the emotion prediction task is removed to see the performance of strategy recognition. In Table 2, the multi-task learning that is jointly training ($\mathcal{L}_{s}$ and $\mathcal{L}_{e}$) can provide benefits, which shows that the training objectives are closely related and boost each other.

The cross-channel fusion combines the persuader-aware contextual dependency with persuadee-aware emotional dependency. In this setting, these representations are concatenated directly to make a prediction. The results indicate the fusion mechanisms make a contribution to the overall performance.