Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History

Spoken dialogue systems aim to enable dialogue agents to engage in a more natural conversation with humans. They have commonly represented a possible dialogue by a series of frames [1, 2]. Each frame represents the type of task the user seeks and has attributes representing the information that can help the system to complete the task. These spoken dialog systems aim to automatically identify the topic of conversations as well as other dialog frame attributes (e.g., dialogue act, emotion) to engage in conversation with the user.

Conventional Spoken Language Understanding (SLU) [3, 4] systems independently process each utterance in a conversation. However, the meaning of an utterance in a spoken dialog depends on the context. Prior work has shown dialogue context to be particularly useful in resolving ambiguities and co-references [5, 6]. As a result, there has been extensive work on incorporating context to improve Natural Language Understanding (NLU) performance [7, 8, 9]. Prior work [10, 11, 12] on conversational speech has also shown strong improvements in ASR performance by incorporating dialog context. Thus, several approaches [13, 14, 15, 16] have looked into incorporating dialog history in pipeline based SLU systems. Recently, there has been some work [17, 18, 19] in incorporating context to improve end-to-end (E2E) SLU performance. One such approach [17] integrates dialog history into an E2E SLU system by using ASR transcripts of previous spoken utterances. There has also been an effort [18] to incorporate context directly from the audio of previous utterances.

However, these works mostly focus on building a single model for each SLU task like intent classification, dialog act classification, and emotion recognition. To jointly predict all the SLU tasks, we have to execute all models separately. Consequently, these models not only have a large memory footprint but also have high latency, which can affect the naturalness of spoken conversations [20] when these systems are deployed in commercial applications like voice assistants. Prior work has shown that jointly modeling different speech processing tasks together in a united framework can perform comparable to task-specific models [21, 22] while reducing latency [23]. Motivated by this work, we ask the following questions: (i) Can we jointly model all SLU tasks while incorporating context in a single unified implementation without much loss in performance? (ii) Does incorporating the SLU tags predicted at previous spoken utterances by the joint model help in better modeling spoken dialogue context? We seek to answer these questions by proposing a novel E2E SLU architecture that jointly models all the SLU tasks while effectively learning context from previous spoken utterances. This joint model uses an autoregressive decoder to predict dialog attributes one by one and the order in which the model predicts the SLU attributes may impact model performance. Hence, we investigate (iii) if we can use order agnostic training to make the model automatically predict in the optimal ordering during inference. Inspired by prior work on ASR [24], we also investigate (iv) if an intermediate CTC loss advances SLU performance further. We conduct extensive experiments on the recently released HarperValley [25] Bank dataset, which consists of dialogs between users and consumer bank agents. Our results show that our joint model performs at par with the individual classifiers for each SLU task, and incorporating dialog context and order agnostic training can further lead to significant improvements in performance. Our code and models are made publicly available as part of the ESPnet-SLU [26] toolkit.

The key contributions of our work are summarised below.

• •

  We propose a novel joint model that uses dialog context to jointly predict the intent, dialog act, speaker role and emotion.

• •

  We propose order agnostic training of the joint model and show an improvement in performance across all SLU tasks.

• •

  We investigate the efficacy of using SLU tags predicted from previous utterances to model dialog context.

• •

  We show that incorporating the usage of intermediate CTC loss can advance SLU performance.

The formulation for SLU with integrated dialog context extends the well-studied framework of NLU systems [7, 8, 9]. For NLU systems, dialog context sequence is represented as sequence of $C$ utterances, i.e., $S=\{s_{c}|c=1,\dots,C\}$. Each utterance in the dialog context sequence is represented as $s_{c}=\{w_{cn}\in\mathcal{V}|n=1,\dots,N_{c}\}$, with length $N_{c}$ and vocabulary $\mathcal{V}$. Each utterance has a tag for each of the NLU tasks. In this work, each utterance has a tag from label sets $\mathcal{L}^{\text{da}}$, $\mathcal{L}^{\text{ic}}$,$\mathcal{L}^{\text{sr}}$ and $\mathcal{L}^{\text{er}}$ indicating dialogue act classification, intent classification, speaker role prediction, and emotion recognition, respectively. This produces a label sequence of the same length $C$ for each task, for instance, $Y^{\text{da}}=\{y_{c}^{\text{da}}\in\mathcal{L}^{\text{da}}|c=1,\ldots,C\}$. Using the maximum a posteriori theory, NLU models seek to output $\hat{Y}^{\text{da}}$,$\hat{Y}^{\text{ic}}$,$\hat{Y}^{\text{sr}}$, and $\hat{Y}^{\text{er}}$ that maximise the posterior distribution $P(Y^{\text{da}}|S)$,$P(Y^{\text{ic}}|S)$,$P(Y^{\text{sr}}|S)$ and $P(Y^{\text{er}}|S)$ given $S$, respectively.

SLU introduces an additional complexity of modeling dialog context from the spoken utterance. Dialog context sequence is formed by $C$ spoken utterances, i.e., $X=\{x_{c}|c=1,\dots,C\}$. Each spoken utterance $x_{c}=\{\mathbf{x}_{ct}\in\mathbb{R}^{d}|t=1,\ldots,T_{c}\}$ is a sequence of $d$ dimensional speech feature of length $T_{c}$ frames. Similar to the NLU formulation, SLU systems seek to estimate the label sequence $\hat{Y}^{\text{da}}$,$\hat{Y}^{\text{ic}}$,$\hat{Y}^{\text{sr}}$ and $\hat{Y}^{\text{er}}$ that maximise the posterior distribution $P(Y^{\text{da}}|X)$,$P(Y^{\text{ic}}|X)$,$P(Y^{\text{sr}}|X)$ and $P(Y^{\text{er}}|X)$ given $X$, respectively. We can model these posterior distributions as described in the subsections below.

In this work, we extend the prior work [17] on dialog integration discussed in section 2.1 and propose to jointly model all SLU tasks. We denote a single target containing all the SLU tags as $R=(Y^{\text{da}},Y^{\text{ic}},Y^{\text{sr}},Y^{\text{er}})$ and modify Eq. 4 with $r_{c}=(y_{c}^{\text{da}},y_{c}^{\text{ic}},y_{c}^{\text{sr}},y_{c}^{\text{er}})$ as shown below:

$\displaystyle P(R|X)$

$\displaystyle\sim\prod_{c=1}^{C}P(r_{c}|{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\textbf{$r_{1:c-1}$}},\hat{s}_{1:c-1},x_{c})$

(5)

The SLU tags predicted for the previous spoken utterances i.e. $r_{1:c-1}=(y_{1:c-1}^{\text{da}},y_{1:c-1}^{\text{ic}},y_{1:c-1}^{\text{sr}},y_{1:c-1}^{\text{er}})$ may be incorporated (Eq. 5) to better model the dialog context unlike in prior work [17] which assumes conditional independence of $y^{\text{da}}_{c}|s_{1:c-1}$ from $y^{\text{da}}_{1:c-1}$. In §5.1, we confirm experimentally whether this previous SLU tag condition is helpful. Further, by jointly modeling all the SLU tasks, we expect significantly lower latency and lightweight inference.

To realize this formulation, we propose a joint model architecture shown in Fig. 1. The input speech signal for each utterance i.e., $x_{c}$ in Eq. 5, is passed through an acoustic encoder ($\text{Encoder}_{\text{aco}}$) to generate acoustic embeddings $\textbf{c}_{\text{aco}}$.

$\displaystyle\mathbf{c}_{\text{aco}}=\operatorname{Encoder}_{\text{aco}}(x_{c})$

(6)

We concatenate ASR transcripts $\hat{s}_{1:c-1}$ and SLU tags $r_{1:c-1}$ for all previous spoken utterances and pass them through semantic encoder ($\operatorname{Encoder}_{\text{sem}}$) like a pretrained LM to encode the dialog history:

$\displaystyle\mathbf{c}_{\text{cont}}=\operatorname{Encoder}_{\text{sem}}(\operatorname{concat}(\hat{s}_{1:c-1},r_{1:c-1}))$

(7)

The output of the semantic encoder is also passed to a linear layer to ensure that context embeddings $\mathbf{c}_{\text{cont}}$ have the same hidden dimension as acoustic embeddings $\mathbf{c}_{\text{aco}}$. The acoustic and context embeddings are concatenated together ($\operatorname{concat}(\mathbf{c}_{\text{aco}},\mathbf{c}_{\text{cont}})$) and attended by a joint encoder $\operatorname{Encoder}_{\text{join}}$ to produce the joint embedding $\mathbf{c}_{\text{join}}$:

$\displaystyle\mathbf{c}_{\text{join}}=\operatorname{Encoder}_{\text{join}}(\operatorname{concat}(\mathbf{c}_{\text{aco}},\mathbf{c}_{\text{cont}}))$

(8)

The model is trained using joint CTC-attention training [28], where the CTC objective function is used to train the attention model encoder as an auxiliary task. Because we use an autoregressive decoder to predict tags one by one (Eq. 11), the likelihood $P(r_{c}|x_{c},\hat{s}_{1:c-1},r_{1:c-1})$ is dependent on the order of SLU tags in the target sequence $r_{c}$. This has been referred to as label ambiguity (or permutation) problem in prior work [29, 30]. Inspired by prior work on permutation invariant training [30, 31, 32], we use CTC objective function to perform permutation-free training as shown in Eq. 9, which is referred to as order agnostic training in this work. Let $\mathbf{z}$ be the output sequence variable computed from the joint embedding $\mathbf{c}_{\text{join}}$, then the optimal permutation order $\hat{\pi}$ is computed as:

$$\hat{\pi}=\operatorname*{argmin}_{\pi\in\mathcal{P}}\operatorname{Loss}_{\text{CTC}}(\mathbf{z},r_{c}^{\pi}),$$

(9)

where $\mathcal{P}$ is the set of 4! possible permutations of SLU tags (da,ic,sr,er), $\pi$ is one such permutation and $r_{c}^{\pi}$ is the reference target with the order of SLU tags indicated by $\pi$. Later, the optimal permutation $\hat{\pi}$ is used for computing the attention decoder loss²²2We use the CTC loss to compute optimal order instead of decoder loss for lightweight modeling..

For each pair of optimal decoding order and reference target ($\hat{\pi}$,$r_{c}$), the attention likelihood is calculated as shown below.

	$\displaystyle\mathbf{h}_{l}=\operatorname{Decoder}(\mathbf{c}_{\text{JOIN}},(r_{c}^{\hat{\pi}})_{1:l-1})$		(10)
	$\displaystyle P_{\text{Attn}}((r_{c}^{\hat{\pi}})_{l}|x_{c},\hat{s}_{1:c-1},r_{1:c-1},(r_{c}^{\hat{\pi}})_{1:l-1})=\operatorname{SoftmaxOut}(\mathbf{h}_{l}),$		(11)

where $\operatorname{SoftmaxOut}$ denotes a linear layer followed by the softmax function and $(r_{c}^{\hat{\pi}})_{l}$ is the $l^{th}$ term in target sequence. The joint likelihood $P_{\text{Attn}}(r_{c}^{\hat{\pi}}|x_{c},\hat{s}_{1:c-1},r_{1:c-1})$ can then be computed as a product of the individual likelihood for each of the SLU tags (i.e., from $l$ in [1,4]). It is important to note that this “order agnostic training” does not add any new model parameters. During inference, the model automatically picks the tag order, i.e., unlike training, we do not enforce the predicted tag order to have the minimum CTC loss. Further, our joint models can also be trained with an auxiliary ASR objective [26, 33] by making the model generate both the SLU tags $r_{c}^{\hat{\pi}}$ and ASR transcript $s_{c}$.

We propose a novel model architecture that can jointly model intent classification, dialogue act prediction, speaker role identification and emotion recognition with full integration of dialog history in spoken conversations. Our results show that the joint model achieves comparable performance to task-specific models with the additional benefits of low latency and lightweight inference. Our joint model can also successfully capture dialog context to improve the prediction performance of all SLU tasks significantly. We experimentally confirm that order agnostic training can further enhance performance. In future work, we plan to explore E2E integration of dialog context as well as knowledge distillation from a text-based system.

This work used the Extreme Science and Engineering Discovery Environment (XSEDE) [42], which is supported by NSF grant number ACI-1548562. Specifically, it used the Bridges system [43], which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).