Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end ASR systems

Fine-tuning on both synthetic and real data prevents the model from forgetting real audio. [9] shows that a subset of the real data in the original source domain is needed if no regularisation method is applied during fine-tuning. In particular, [9] kept about 30% of source domain data used in training and combined them with the synthetic target domain data to form the final data for fine-tuning.

Instead of directly combining a portion of the original real data with synthetic data, as in previous studies [11, 19, 9], we propose to sample data on-the-fly from the source domain real data and the target domain synthetic data. The sampling distribution is a global and configurable hyperparameter that propagates into each training batch. This allows a consistent sample mixing and it also makes data combination independent from the absolute sizes of synthetic and real data.

With encoder freezing (EF), the encoder parameters of a trained RNN-T are fixed, and only the parameters of the decoder and joint network are updated during fine-tuning. In previous work [9], freezing the encoder provided much better results than not freezing the encoder when fine-tuning RNN-T on synthetic data only. This paper examines whether encoder freezing provides extra benefits on top of the sampled data combination method as explained in Section 3.1.

Encoder freezing only applies regularisation on the encoder in RNN-T, but the unrealistic synthetic audio may indirectly negatively impact other components. In addition, the word distribution of the text data for real recordings and synthetic audio is different. When applying the method in Section 3.1 during fine-tuning, the prior probability of words previously represented by real recordings is likely to decrease, potentially causing a degradation in the overall WER of the source domain. Since such changes in word probability are likely to impact the decoder and joint networks more than the encoder, a regularisation for the decoder and joint networks may also be required during fine-tuning. We experiment with EWC [17, 20] for this purpose, with its loss function formulated as:

where $\theta_{\textnormal{new},i}$ is the current value for the $i$th parameter and $\theta_{\textnormal{old},i}$ is the value of that same parameter before fine-tuning; therefore $\theta_{\textnormal{old},i}$ is fixed throughout the fine-tuning process. $F_{i}$ is the diagonal entry of the fisher information matrix used to give the $i$th parameter a selective constraint. $\mathcal{L}_{\textnormal{EWC}}$ is added to the regular RNN-T loss to force the parameters important to the source domain to stay close to the baseline model.

We use 2.3K hours of anonymised far-field in-house data (Train19) as the training data of baseline RNN-T. A dev set (Dev) and an eval set (Eval) are constructed with more recent data (1K hours each). Since Dev and Eval were from the live traffic of a later time period after Train19, some trending words in Dev and Eval may have rarely appeared in Train19.

A list of OOV words is extracted by comparing Dev with Train19, i.e. all the words that have not appeared in Train19 but have appeared at least three times in Dev. The minimal occurrence of three helps exclude typos from the OOV word list. The utterances in Dev containing any OOV words are extracted as a subset DevOOV, which contains 6.5K utterances and only accounts for 0.7% of the Dev set. Similarly, the utterances in Eval containing any OOV words are extracted as a subset EvalOOV, containing 4.3K utterances. To reduce the decoding time, Dev and Eval are down-sampled randomly to 200K utterances each into DevSub and EvalSub. The utterances in EvalSub not covered by EvalOOV make another subset, i.e. EvalSub\OOV, which is used to monitor the recognition accuracy of non-OOV words.

The US English standard voice of Amazon Polly is used to generate synthetic audio from the text of DevOOV with one voice profile. This TTS system is based on hybrid unit selection. Future work can use a better TTS to reduce acoustic mismatch between real and synthetic audio.

A baseline RNN-T model is trained on Train19 until convergence. It has 5 LSTM layers in the encoder network (1024$\times$5), 2 LSTM layers in the decoder network (1024$\times$2), and a joint network with one feedforward layer (512$\times$1). The output of the RNN-T is the probability distribution of 4000 word-piece units from a unigram word-piece model [21].

When fine-tuning the baseline RNN-T to improve the performance on OOV words, Table 1 shows that it is important to control the sampling weights when combining Train19 and DevOOV. The model performs the worst on DevSub when fine-tuned on DevOOV with synthetic audio only, i.e. weights (0, 100). As the percentage of samples from Train19 increases, the WERs on DevSub decreases, proving that combining synthetic data with the original training data can prevent the model from forgetting what it learnt before. Without degrading on DevSub, the best performance on DevOOV is observed with 70% fine-tuning data from Train19 and 30% from DevOOV with synthetic audio, achieving a 49% relative WER reduction on DevOOV compared to the baseline.

To find out the influence of the acoustic mismatch between real and synthetic audio, the last three rows of Table 1 replace synthetic audio with real recordings for DevOOV during fine-tuning. Fine-tuning on 20% of DevOOV with real audio achieves a 33% relative WER reduction on EvalOOV compared to the same setup with synthetic audio. This motivates the use of extra methods to reduce the acoustic gap between real recordings and synthetic audio.

Applying EF or EWC on the encoder may prevent the model from learning unwanted acoustic characteristics in synthetic audio. Comparing Table 1 with Table 2, when fine-tuning on 100% synthetic audio, applying EF gives 42% and 19% relative WER reductions on DevOOV and DevSub respectively, and applying EWC on the encoder achieves a similar performance. With weights (70, 30), i.e. the optimal setup from Table 1, EF improves the performance on DevOOV but degraded slightly on DevSub. The degradation on DevSub can be recovered by increasing the percentage of fine-tuning data sampled from Train19 to 90%. Replacing EF with EWC on the encoder further improves the WER on DevOOV by 2% relative, suggesting that freezing all encoder parameters during fine-tuning is suboptimal. In this (90, 10) weights setup, compared to not applying any regularisation on the encoder in Table 1, EWC introduces 10%, 1% and 14% relative WER reductions on DevOOV, DevSub and EvalOOV respectively.

Table 3 adds EWC regularisation on the decoder and joint networks. EF is used instead of EWC on the encoder because EF does not require careful hyperparameter tuning and it performs similarly to EWC. Compared with Table 2, when fine-tuning on DevOOV only with synthetic audio, regularising the decoder and joint networks with EWC helps mitigate the degradation on DevSub. However, when 90% of fine-tuning data is sampled from Train19, adding EWC on the decoder and joint networks does not improve the performance on DevSub and introduces 7% relative degradation on DevOOV.

Two models are evaluated on subsets from Eval. When the original training data is available, the model fine-tuned with the (90, 10) weights setup with EWC applied on the encoder (highlighted in Table 2) is selected. Otherwise, we choose the model fine-tuned on 100% synthetic audio with EF and EWC on the decoder and joint networks (highlighted in Table 3). As shown in Table 4, having 90% of fine-tuning data sampled from Train19 can almost eliminate the degradation on EvalSub\OOV, leading to 57% and 1% relative WER reductions on EvalOOV and EvalSub respectively compared to the baseline. In addition, for both models the improvement previously observed on DevOOV is successfully replicated on EvalOOV.

Fig. 1 shows the relative WER reduction averaged over all OOV words that appear the same number of times in DevOOV, based on the recognition results highlighted in Table 4. Overall, the recognition performance improves more if an OOV word is seen many times during fine-tuning, but the performance also improves for some OOV words that are seen only a few times. The two most frequent OOV words are ‘coronavirus’ and ‘covid’. Even though the word-piece vocabulary can theoretically compose ‘coronavirus’, the baseline model only recognises one ‘coronavirus’ in DevOOV. After using the best model in Table 4, the WER for ‘coronavirus’ drops by 87% relative on EvalOOV.