Joint Intent Detection and Slot Filling with Wheel-Graph Attention Networks

Separate Model The intent detection is formulated as a text classification problem. The traditional method is to employ n-grams as features with generic entities, such as locations and dates zhang2016joint . This type of approach is restricted to the dimensionality of the input space. Another line of popular approaches is to train machine learning models on labeled training data, such as support vector machine (SVM) and Adaboost haffner2003optimizing ; schapire2000boostexter . Approaches based on deep neural network technology have shown excellent performance, such as Deep belief networks (DBNs) and RNNs ravuri2015recurrent ; deoras2013deep . Slot filling can be treated as a sequence labeling task. The traditional method based on conditional random fields (CRF) architecture, which has a strong ability on sequence labeling tasks raymond2007generative . Another line of popular approaches is CRF-free sequential labeling. yao2014spoken introduced LSTM architecture for this task and obtained a marginal improvement over RNN. shen2018disan and tan2018deep introduce the self-attention mechanism for slot filling.

Implicit Joint Model Recently, there have been some joint models to overcome the error propagation caused by the pipelined approaches, and all these models only applied share parameters a joint loss function to link the two tasks implicitly. hakkani2016multi proposed an RNN-LSTM architecture for joint modeling of intent detection and slot filling. zhang2016joint first proposed the joint work using RNNs for learning the correlation between intent and semantic slots of a sentence. liu2016attention proposed an attention-based neural network model for joint intent detection and slot filling, which further explores different strategies in incorporating this alignment information into the encoder-decoder framework. All these models outperform the pipeline models by mutual enhancement between two tasks. However, these joint models didn’t model their correlation.

Unidirectional related Joint Model Recently, some works have explored unidirectional related joint models. These models have exploited the intent information for slot filling. li2018self proposed a novel intent-augmented gate mechanism to utilize the semantic correlation between intent and slots fully. goo2018slot proposed a slot gate that focuses on learning the relationship between intent and slot attention vectors to obtain better semantic frame results by global optimization. chen2019bert utilize a mask gating mechanism to model the relationship between intent detection and slot filling. qin2019stack perform the token-level intent detection for the Stack-Propagation framework to better incorporate the intent information.

Interrelated Joint Model Considering this strong correlation between the two tasks, interrelated joint models have been explored recently. wang2018bi introduce their cross-impact to each other using two correlated bidirectional LSTMs (BLSTM) to perform the intent detection and slot filling tasks jointly. haihong2019novel introduce an SF-ID network to establish direct connections for two tasks to help them promote each other mutually. zhang2019joint proposed a capsule-based neural network that models hierarchical relationships among word, slot, and intent in an utterance via a dynamic routing-by-agreement schema.

Applying graph neural networks (GNN) to solve some problems has been a popular approach recently in social network analysis hamilton2017inductive , knowledge graphs hamaguchi2017knowledge , urban computing, and many other research areas velivckovic2018graph ; huang2019text . GNN can model non-Euclidean data, while traditional neural networks can only model regular grid data.

Unlike previously discussed neural network-based methods, our approach explicitly establishes direct connections among intent nodes and slots nodes by GAT velivckovic2018graph , which uses weighted neighbor features with feature dependent and structure-free normalization, in the style of attention. Analogous to multiple channels in ConvNet krizhevsky2012imagenet , GAT introduces multi-head attention vaswani2017attention to enrich the model capacity and to stabilize the learning process. Unlike other models haihong2019novel ; zhang2019joint , our model does not need to set the number of iterations during training. We have also established a wheel graph structure to learn context-aware information in an utterance better.

Word Embedding: Given a sequence of words, we first covert each word as embedding vector $\mathbf{e}_{t}$, and the sequence is represented as $[\mathbf{e}_{1},…,\mathbf{e}_{T}]$, where $T$ is the number of words in the sentence.

Affine Transformation: We perform an affine transformation on the embedding sequence, which is a data standardization method.

where $\mathbf{W}$ and $\mathbf{b}$ are trainable weights and biases.

Two-Layer BiGRU: As an extension of conventional feed-forward neural networks, it was difficult to train Recurrent neural networks (RNNs) to capture long-term dependencies because the gradients tend to either vanish or explode. Therefore, some more sophisticated activation functions with gating units were designed. Two revolutionary methods are long short-term memory (LSTM) hochreiter1997long and gated recurrent unit (GRU) cho2014properties . Similarly to the LSTM unit, the GRU has gating units that modulate the flow of information inside the unit; however, without having a separate memory cells and has less parameters. Based on this, we use GRU in this work.

where $\mathbf{x}_{t}$ is the input at time $t$, $\mathbf{r}_{t}$ and $\mathbf{z}_{t}$ are reset gate and update gate respectively, $\mathbf{W}$ and $\mathbf{U}$ are weight matrices, $\sigma$ is sigmoid function and $\odot$ is an element-wise multiplication. When the reset gate is off ($\mathbf{r}_{t}$ close to 0), the reset gate effectively makes the unit act as if it is reading the first symbol of an input sequence, allowing it to forget the previously computed state. For simplification, the above equations are abbreviated with $\mathbf{h}_{t}=GRU(\mathbf{x}_{t},\mathbf{h}_{t-1})$.

To consider both past and future information at the same time. Consequently, we use a two-Layer bidirectional GRU (BiGRU) to learn the utterance representations at each time step. The BiGRU, a modification of the GRU, consists of a forward and a backward GRU. The layer reads the affine transformed output vectors $[\mathbf{x}_{1},…,\mathbf{x}_{T}]$ and generates $T$ hidden states by concatenating the forward and backward hidden states of BiGRU:

where $\overrightarrow{\mathbf{h}}_{t}$ is the hidden state of forward pass in BiGRU, $\overleftarrow{\mathbf{h}}_{t}$ is the hidden state of backward pass in BiGRU and $\overleftrightarrow{\mathbf{h}}_{t}$ is the concatenation of the forward and backward hidden states at time $t$.

In summary, to get more fine-grained sequence information, we use a two-layer BiGRU to encode input information. The representation is defined as:

A graph attention network (GAT) velivckovic2018graph is a variant of graph neural network scarselli2008graph and is an important element in our proposed method. It propagates the intent or slot information from a one-hop neighborhood. Given a dependency graph with $N$ nodes, where each node is associated with a local vector $\mathbf{x}$, one GAT layer compute node representations by aggregating neighborhood’s hidden states.

GAT exploits the attention mechanism as a substitute for the statically normalized convolution operation. Below are the equations to compute the node embedding $\mathbf{h}_{i}^{(l+1)}$ of layer $l+1$ from the embeddings of layer $l$.

where $\mathbf{W}^{(l)}$ is a linear transformation matrix for input states, $\parallel$ represents vector concatenation, $\overrightarrow{\mathbf{a}}^{(l)}$ is an attention context vector learned during training, and $\cdot^{T}$ represents transposition. $f(\cdot)$ is a LeakyReLU non-linear function maas2013rectifier . $N(i)$ is the neighbor nodes of node $i$. $\sigma$ is the activation function such as $tanh$. For simplification, the above equations are abbreviated with $\mathbf{h}^{(l+1)}=GAT(\mathbf{h}^{(l)})$.

In the SLU task, there is a strong correlation between intent detection and slot filling. To make full use of the correlation between intent and slot, we constructed a wheel-graph structure. In Figure 1 , this wheel-graph structure contains an intent node and slot nodes.

For the node representation, we use the output of the previous two-layer BiGRU, and the formula is expressed as:

where the max function is an element-wise function, and $T$ is the number of words in the utterance. We use $\mathbf{h}^{I}_{0}$ as the representation of the intent node and $\overleftrightarrow{\mathbf{h}}_{t}$ as the representation of the slot nodes.

For the edge, we created a bidirectional connection between the intent node and the slot nodes. To make better use of the context information of the utterance, we created a bidirectional connection between the slot nodes and connected the head and tail of the utterance to form a loop.

In summary, the feed-forward process of our wheel-graph neural network can be written as:

where $m\in{0,1,…,t}$, $\mathbf{h}^{I}$ is the hidden state output of the intent, and $\mathbf{h}_{t}^{S}$ is the hidden state output of the slots.

The last layer is the output layer. We adopt a joint learning method. The softmax function is applied to representations with a linear transformation to give the probability distribution $\mathbf{y}^{I}$ over the intent labels and the distribution $\mathbf{y}_{t}^{S}$ over the $t-th$ slot labels. Formally,

where $\mathbf{W}^{I}$ and $\mathbf{W}^{S}$ are trainable parameters of the model, $\mathbf{b}^{I}$ and $\mathbf{b}^{S}$ are bias vectors. $o^{I}$ and $o_{t}^{S}$ are the predicted output labels for intent and slot task respectively.

Then we define loss function for our model. We use $\hat{y}^{I}$ and $\hat{y}^{S}$ to denote the ground truth label of intent and slot.

The loss function for intent is a cross-entropy cost function.

Similarly, the loss function of a slot label sequence is formulated as:

where $n_{I}$ is the number of intent label types, $n_{S}$ is the number of slot label types and $T$ is the number of words in an utterance.

The training objective of the model is minimizing a united loss function:

where $\alpha$ is a weight factor to adjust the attention paid to two tasks.

For experiments, we utilize two datasets, including ATIS hemphill1990atis and SNIPS coucke2018snips , which is collected by Snips personal voice assistant in 2018. They are two public benchmark single-intent datasets, which are widely used as benchmark in SLU research. Compared to the single-domain ATIS dataset, SNIPS is more complicated, mainly due to the intent diversity and large vocabulary. Both datasets used in our paper follows the same format and partition as in qin2019stack . The overview of datasets is listed in Table 2.

To validate the effectiveness of our approach, we compare it to the following baseline approaches. It is noted that the results of some models are directly taken from qin2019stack .

• •

  Joint Seq applies an RNN-LSTM architecture for slot filling, and the last hidden state of LSTM is used to predict the intent of the utterance hakkani2016multi .
  

• •

  Attention BiRNN adopts an attention-based RNN model for joint intent detection and slot filling. Slot label dependencies are modeled in the forward RNN. A max-pooling over time on the hidden states is used to perform the intent classification liu2016joint .
  

• •

  Slot-Gated Full Atten. utilizes a slot-gated mechanism that focuses on learning the relationship between intent and slot attention vectors. The intent attention context vector is used for the intent classification goo2018slot .
  

• •

  Self-Attention Model first makes use of self-attention to produce a context-aware representation of the embedding. Then a bidirectional recurrent layer takes as input the embeddings and context-aware vectors to produce hidden states. Finally, it exploits the intent-augmented gating mechanism to match the slot label li2018self .
  

• •

  Bi-Model is a new Bi-model based RNN semantic frame parsing network structure which performs the intent detection and slot filling tasks jointly by considering their cross-impact to each other using two correlated bidirectional LSTMs wang2018bi .
  

• •

  SF-ID Network is a novel bi-directional interrelated model for joint intent detection and slot filling. It contains an entirely new iteration mechanism inside the SF-ID network to enhance the bi-directional interrelated connections haihong2019novel .
  

• •

  CAPSULE-NLU introduces a capsule-based neural network model with a dynamic routing-by-agreement schema to accomplish intent detection and slot filling tasks. The output representations of IntentCaps and SlotCaps are used to intent detection and slot filling, respectively zhang2019joint .
  

• •

  Stack-Propagation adopts a Stack-Propagation, which directly uses the intent information as input for slot filling and performs the token-level intent detection to further alleviate the error propagation qin2019stack .

In our experiments, the dimensionalities of the word embedding are 1024 for the ATIS dataset and SNIPS dataset. All model weights are initialized with uniform distribution. The number of hidden units of the BiGRU encoder is set as 512. The number of layers of the GAT model is set to 1. Graph node representation is set to 1024. The weight factor $\alpha$ is set to 0.1. We use the Adam optimizer kingma2014adam with an initial learning rate of $10^{-3}$, and L2 weight decay is set to $10^{-6}$. The model is trained on all the training data with a mini-batch size of 64. In order to enhance our model to generalize well, the maximum norm for gradient clipping is set to 1.0. We also apply the dropout ratio is 0.2 for reducing overfit.

We implemented our model using PyTorch¹¹1https://github.com/pytorch/pytorch and DGL²²2https://github.com/dmlc/dgl on a Linux machine with Quadro p5000 GPUs. For all the experiments, we select the model which works the best on the validation set and evaluate it on the test set.

As with Qin et al qin2019stack , we adopt three evaluation metrics in the experiments. For the intent detection task, the accuracy is applied. For the slot filling task, the F1-Score is utilized. Besides, the sentence accuracy is used to indicate the general performance of both tasks, which refers to the proportion of the sentence whose intent and slot are both correctly-predicted in the whole corpus. Table 3 shows the experimental results of the proposed models on ATIS and SNIPS datasets.

We note that the results of unidirectional related joint models are better than implicit joint models like Joint Seq hakkani2016multi and Attention BiRNN liu2016joint , and the results of interrelated joint models are better than unidirectional related joint models like Slot-Gated Full Atten. goo2018slot and Self-Attentive Model li2018self . That is likely due to the strong correlation between the two tasks. The intent representations apply slot information to intent detection task while the slot representations use intent information in slot filling task. The bi-directional interrelated model helps the two tasks to promote each other mutually.

We also find that our graph-based Wheel-GAT model performs better than the best prior joint model Stack-Propagation Framework. In ATIS dataset, we achieve 0.6% improvement on Intent (Acc), 0.1% improvement on Slot (F1-score) and 0.7% improvement on Sentence (Acc). In the SNIPS dataset, we achieve 0.4% improvement on Intent (Acc), 0.6% improvement on Slot (F1-score), and 0.5% improvement on Sentence (Acc). This indicates the effectiveness of our Wheel-GAT model. In the previously proposed model, the iteration mechanism used to set the number of iterations is not flexible on training, and the token-level intent detection increases the output load when the utterance is very long. While our model employed graph-based attention network, which uses weighted neighbor features with feature dependent and structure-free normalization, in the style of attention, and directly takes the explicit intent information and slot information further help grasp the relationship between the two tasks and improve the SLU performance.

In this section, to further examine the level of benefit that each component of Wheel-GAT brings to the performance, an ablation study is performed on our model. The ablation study is a more general method, which is performed to evaluate whether and how each part of the model contributes to the full model. We ablate four important components and conduct different approaches in this experiment. Note that all the variants are based on joint learning method with joint loss.

• •

  Wheel-GAT w/o intent $\rightarrow$ slot, where no directed edge connection is added from the intent node to the slot node. The intent information is not explicitly applied to the slot filling task on the graph layer.  
  

• •

  Wheel-GAT w/o slot $\rightarrow$ intent, where no directed edge connection is applied from the slot node to the intent node. The slot information is not explicitly utilized to the intent detection task on the graph layer.  
  

• •

  Wheel-GAT w/o head $\leftrightarrow$ tail, where no bidirectional edge connection is used between the intent node and the slot node. We only use joint loss for joint model, rather than explicitly establishing the transmission of information between the two tasks.  
  

• •

  Wheel-GAT w/o GAT, where no graph attention mechanism is performed in our model. The message propagation is computed via GCN instead of GAT. GCN introduces the statically normalized convolution operation as a substitute for the attention mechanism.

Table 4 shows the joint learning performance of the ablated model on ATIS and SNIPS datasets. We find that all variants of our much model perform well based on our graph structure except Wheel-GAT w/o GAT. As listed in the table, all features contribute to both intent detection and slot filling tasks.

If we remove the intent $\rightarrow$ slot edge from the holistic model, the slot performance drops 0.5% and 1.3% respectively on two datasets. Similarly, we remove the slot $\rightarrow$ intent edge from the holistic model, the intent performance down a lot respectively on two datasets. The result can be interpreted that intent information and slot information are stimulative mutually with each other. We can see that the added edge does improve performance a lot to a certain extent, which is consistent with the findings of previous work goo2018slot ; qin2019stack ; haihong2019novel .

If we remove the head $\leftrightarrow$ tail edge from the holistic model, we see 0.4% drop in terms of F1-score in ATIS and 0.8% drop in terms of F1-score in SNIPS. We attribute it to the fact that head $\leftrightarrow$ tail structure can better model context-aware information in an utterance.

To verify the effectiveness of the attention mechanism, we remove the GAT and use GCN instead. For GCN, a graph convolution operation produces the normalized sum of the node feature of neighbors. The result shows that the intent performance drops 1.3% and 1.7%, the slot performance drops 1.0% and 4.0%, and the sentence accuracy drops 2.9% and 9.8% respectively on ATIS and SNIPS datasets. We attribute it to the fact that GAT uses weighting neighbor features with feature dependent and structure-free normalization, in the style of attention.

In this section, with attempt to better understand what the wheel-graph attention structure has learnt, we visualize the attention weights of slot $\rightarrow$ intent and each slot node, which is shown in Figure 2.

Based on the utterance “play signe anderson chant music that is newest”, the intent “PlayMusic” and the slot “O B-artist I-artist B-music_item O O O B-sort”, we can clearly see the attention weights successfully focus on the correct slot, which means our wheel-graph attention layer can learn to incorporate the specific slot information on intent node in Figure 2a. In addition, more specific intent token information is also passed into the slot node in Figure 2b, which achieves a fine-grained intent information integration for guiding the token-level slot prediction. Therefore, the node information of intent and slots can be transmitted more effectively through attention weights in our proposed wheel-graph attention interaction layer, and promote the performance of the two tasks at the same time.

In this section, we also experiment with a pre-trained BERT-based devlin2019bert model instead of the Embedding layer, and use the fine-tuning approach to boost SLU task performance and keep other components the same as with our model.

As can be seen from Table 5, Stack-Propagation + BERT qin2019stack joint model achieves a new state-of-the-art performance than another without a BERT-based model, which indicates the effectiveness of a strong pre-trained model in SLU tasks. We attribute this to the fact that pre-trained models can provide rich semantic features, which can help to improve the performance on SLU tasks. Wheel-GAT + BERT outperforms the Stack-Propagation + BERT. That is likely due to we adopt explicit interaction between intent detection and slot filling in two datasets. It demonstrates that our proposed model is effective with BERT.