Robust Unstructured Knowledge Access in Conversational Dialogue with ASR Errors

Given speech audio, an ASR decoder decodes the audio and generates word lattices or N-best lists that are aligned into confusion networks (Mangu, Brill, and Stolcke 1999). A confusion network provides alternative word candidates at each word position. For instance, a confusion network representation “ok do they (delver $|$ deliver $|$ delo $|$ over $|$ lover $|$ del $|$ dolo)” gives a word confusion set {delver, deliver, delo, over, lover, del} where the words sound confusing from an ASR decoder viewpoint. Here, we do not know which word token is correct in a confusion set. For creating an ASR error simulator, we focus on capturing and re-generating error patterns from confusion sets. In particular, we enumerate all possible word pairs from a confusion set, giving “delver-deliver”, “delver-delo”,…,“del-dolo”. We perform letter-to-letter alignment between each word pair so that position-dependent letter rewriting probabilities $Pr(t|s,i)$ are estimated based on co-occurrence statistics, where $i$ denotes the position of a source letter $s$ to be rewritten as letter $t$ including the deletion symbol “*”. To generate a random erroneous word from a clean word:

1. 1.

   Randomly choose the number of edits uniformly from 1, 2 or 3.

2. 2.

   Randomly choose a letter position $i$ uniformly across the length of a word.

3. 3.

   Randomly choose an error type with $Pr(replacement)=0.9$ and $Pr(insertion)=0.1$.

4. 4.

   If the error type is replacement, draw a letter $t\sim Pr(t|s,i)$. Replace $s$ with $t$ in position $i$.

5. 5.

   If the error type is insertion, draw a letter $t\sim Pr_{*}(t|s,i)$. Insert $t$ at position $i+1$.

6. 6.

   Repeat step 2 until the desired number of edits is achieved.

Table 1 shows samples of simulated errors from clean words. The simulated errors “look” similar to their clean words. The ability of simulating new errors is crucial to avoid overfitting a downstream SLU model to limited error patterns.

There are 12k knowledge titles in the DSTC10 knowledge base. This poses a challenge of matching a dialogue against all knowledge titles, i.e. a brute foce approach, for practical knowledge selection. We observe that many entities in the knowledge base shared the same/similar titles. For example, “A and B Guest House” and “Acorn Guest house” share the same title regarding the “wifi” topic. This motivates us to design an iterative algorithm to group titles of similar topics into knowledge clusters in an unsupervised manner. Intuitively, titles with similar vocabulary (after removing stopwords and word normalization) formed initial knowledge clusters. For example, “Does it have free wifi?” is converted into “free wifi” after stopword removal and sorting. “Is the wifi free” is also converted into “free wifi” as well. Similarly, we also group similar bodies into clusters, assuming that a similar body (except the non-meaningful body) would imply a similar title, so that more knowledge clusters can be formed to enhance the variety of data for classifier training.

We train a BERT-based binary classifier to detect positive/negative title-body pairs. By default, all the title-body pairs in the knowledge base are considered as positive instances. In addition, similar titles in a knowledge cluster enrich the variety of positive instances by cross enumeration. We generate negative title-body pairs by randomly replacing a title with another title within the same entity, assuming that titles within an entity are mostly unique. Then we feed these positive and negative title-body instances to train a binary classifier.

Once a binary classifier is trained, our next step is to merge similar knowledge clusters. We enumerate title-body pairs from each pair of knowledge clusters and classify the title-body pairs using our classifier. The classification results will give us an estimate about the similarity of the knowledge clusters. Pairs of knowledge clusters are combined when the majority of title-body pairs are positively classified according to a threshold. We repeat the grouping step iteratively until no more knowledge clusters can be formed. Additionally, we include a non-knowledge cluster containing user utterances that are non knowledge-seeking. In total, there were 95 knowledge clusters derived from 12k title-body pairs in the knowledge base covering hotel, restaurant, attraction, train and taxi domains. Table 2 shows samples of a knowledge cluster for “wifi” and “trail”.

Inspired by masked LM, we randomly pick two words from a clean user utterance in a dialogue session, and apply error simulation in Section 3.1 to replace clean tokens with erroneous ones. We didn’t replace more than 2 words in an utterance since injecting too much noise may corrupt the semantics of an utterance badly. With the generated errors and the corresponding clean word, we feed an erroneous dialogue utterance into a deep model, e.g. Bert (Devlin et al. 2019) and learn to detect and correct the errors using the LM head.

Using multi-task learning, our ultimate target is to train a robust SLU model to detect and correct errors, and to classify an utterance with a knowledge cluster label. Given joint model parameters $\Theta=\{W_{bert},W_{lm},W_{cls}\}$ where $\Theta$ denotes parameters of a pre-trained Bert model, the language modeling head and the classification head respectively, our joint loss function becomes:

$$L(\Theta)=L_{cls}(W_{bert},W_{cls})+\lambda\cdot L_{lm}(W_{bert},W_{lm})$$

(1)

where $\lambda$ denotes the LM weight for correcting the ASR errors. When $\lambda$ is set to zero, the model just trains with noise, ignoring the error-correction mechanism. Figure 2 shows the proposed system architecture with double heads.

We employed datasets from the Dialogue System Technology Challenge, namely DSTC9 (Kim et al. 2021) and DSTC10 (Kim et al. 2022). The challenge had 3 sub-tasks: 1) Knowledge-seeking turn detection (KTD) to determine if the last user turn had knowledge-seeking intention. For example, after a system returned a list of hotel recommendations that satisfy user’s requirements, users may ask questions like: “does this hotel provide free wifi?” as positive knowledge seeking; 2) Knowledge-selection (KS) to search for a document from an external knowledge base to answer the question. In multi-turn dialogues, entity linking and tracking was required during searching for relevant documents. In this paper, we mainly reported results for KTD and KS. DSTC9 datasets contained clean dialogue while the DSTC10 dataset had noisy dialogue with ASR errors. According to the track organizer, their ASR decoder in DSTC10 was based on wav2vec 2.0 (Baevski et al. 2020) pre-trained on 960 hours of Librispeech and fine-tuned with 10% of in-domain speech data with word error rate of 24.09%. DSTC10 dataset came with N-best lists. We used SRILM toolkit (Stolcke 2002) to generate confusion networks from N-best lists. Since only user utterances contained ASR errors while system utterances were clean, we simulated errors on user utterances only. Last-turn user utterances were considered for error simulation and model training. Table 3 shows samples of knowledge-seeking user utterances containing ASR errors and their clean versions.

One difficulty of our work was that the provided DSTC10 dataset had only 263 dialogue sessions. For model tuning and evaluation, we split them into two equal parts. The first half was used as a development set while the second half was used as a test set.

To create our training set, we employed the DSTC9 evaluation set comprising only clean dialogue sessions. We didn’t use the DSTC9 training and validation sets for our model training because they did not give any improvement. Since the DSTC10 dataset had overlapped dialogues with the DSTC9 evaluation set, we excluded the overlapped dialogues from training, yielding 3867 training dialogues. To enhance data coverage, we augmented titles from the knowledge base as the last “user-turn” into our training set, yielding 15906 training dialogues as our final training set.

The motivation of using the knowledge clusters as labels was to avoid brute-force evaluation of the relevance between a last-turn user utterance and possible documents in the knowledge base. To ensure good quality of knowledge clusters, we manually inspected and adjusted the knowledge clusters so that questions within each cluster were topically consistent. Then, we addressed knowledge selection in two steps. First, we predicted the knowledge cluster of the last user turn using the joint Bert classifier trained with error simulation and correction in Section 3.3. This step narrowed down the document candidates based on top-K predicted clusters. To pinpoint relevant documents further, we built entity-linking and entity-tracking systems based on techniques such as Trie-based and Bert-based named entity recognition, fuzzy match in Elasticsearch, and multiple-choice-headed (Radford et al. 2018) Bert for entity tracking.

We trained Bert (bert-base-uncased) without ASR error simulation as our baseline. Then we trained joint models with different LM weights ranging from 0 (trained with noise without error correction), 0.1, 0.5, and 1.0. All models shared the same modeling architecture and were trained using the same hyper-parameters. We used the Huggingface Transformers library (Wolf et al. 2020) for pretrained models and the DSTC9 code base ³³3https://github.com/alexa/alexa-with-dstc9-track1-dataset for our implementation. We trained all models with 100 epochs with batch size of 16 on a single GPU. We used Adam (Kingma and Ba 2015) for optimization with a learning rate of 6.25e-5 and $\epsilon$ of 1e-8. Gradient clipping was applied with a maximum gradient norm of 1.0. Linear scheduling without warm-up was used. All random seeds were set to 42. For knowledge cluster classification, we only used the last user utterance since performance was hurt using all dialogue turns. The last user turn usually carried the most crucial information for knowledge cluster classification.

Table 4 shows the knowledge cluster classification results using the proposed approach with various LM weights on the development set. First, the proposed approach outperformed the baseline on all metrics. In particular, Recall@1 was boosted drastically from 0.8118 to 0.94 when the LM weights were set to 0.1 or 0.5. Tuning the LM weight was crucial as performance dropped LM weights were set to 0.0 and 1.0. It was surprising that simply adding noise without self-correction still improved performance. On the other hand, we observed that its model training would result in early stopping based on the optimal performance on the development set. In contrast, models took more steps to train when error correction was considered. This may be because the models took more effort to minimize the joint loss function with more variety of simulated errors. As a by-product of knowledge cluster classification, the proposed model addressed knowledge-seeking turn detection (KTD) when the predicted label was either zero or non-zero. Consistently, KTD F1 was also improved from 0.9306 to 0.96 compared to the baseline.

Table 5 shows the knowledge cluster classification results on the test set. We observed consistent improvement on all metrics. In particular, we observed a drastic improvement in Recall@1 from 0.7924 to 0.9333 when the optimal LM weight according to the development set. KTD F1 was also boosted significantly from 0.9433 to 0.9904. Performance trends on the development and test sets were similar, demonstrating the effectiveness of the proposed approach on noisy utterances with ASR errors.

Finally, we combined the top-K knowledge cluster predictions with our multiple-choice-headed (Radford et al. 2018) Bert-based entity tracker, yielding pre-selected knowledge document candidates for further re-ranking. Re-ranking was crucial because knowledge clusters were coarse and may contain documents from different sub-topics as shown in Table 2. For each system, we fed their pre-selected knowledge document candidates into the DSTC9 winner system (Knover) for knowledge re-ranking (He et al. 2021).

Table 7 shows knowledge cluster classification results on our proposed systems. For comparison, we took the Knover predicted outputs from DSTC10 and mapped the document ID to knowledge cluster ID. Our submitted system outperformed Knover on all metrics. Table 6 shows knowledge turn detection and knowledge selection results on the test set. Consistently, the proposed approach showed improved performance across all metrics. In particular, our approach yielded 4.5%, 10% and 6.62% absolute improvement on Recall@1, Recall@5 and MRR@5 respectively compared to the baseline. We also compared the case where our entity-tracker was disabled so that the Knover system performed knowledge re-ranking directly from documents in the predicted top-K knowledge clusters. However, the results were worse as shown in Table 6, showing that our entity-linking and entity tracker were effective in removing a lot of irrelevant entities from re-ranking. In a separate experiment, re-ranking with Knover was still effective compared to the un-ranked documents within the tracked top entities sorted by document IDs.

Table 8 and Table 9 show the DSTC10 evaluation results with 1988 test dialogues on knowledge turn detection and knowledge selection respectively. Our proposed approach shows effectiveness in all metrics using a single model. In particular, Recall@1 was boosted dramatically from 0.495 to 0.7105 compared to the Knover baseline (He et al. 2021) trained only on clean DSTC9 dialogues. Among the 16 teams with their best submission, our best system ranked the 3rd place in Recall@1 and Recall@5 for knowledge selection sub-task. Due to time limits, our submitted system was a single model without model ensembles.

After evaluation, we fixed some errors in our system workflow and reran the training and evaluation. We obtained further improvement on all metrics, boosting the Recal@1 from 0.7105 to 0.7144. In addition, we used all dialogue turns to train the knowledge cluster classifier. However, we found a significant degradation in all metrics, showing that including the user and system turns hurt performance, especially when the user turns contain speech recognition errors. We also re-trained the knowledge cluster classifier without human adjustment of the knowledge cluster labels. Within our expectation, we observed some degradation since the knowledge clusters from unsupervised clustering contained some errors.