SememeASR: Boosting Performance of End-to-End Speech Recognition against Domain and Long-Tailed Data Shift with Sememe Semantic Knowledge

The baseline E2E ASR model we choose in our experiment is similar to the one presented in WeNet [11], which uses both CTC and Attention-based Encoder-Decoder (AED) loss during training to speed convergence and is also a relatively good one among a series of state-of-the-art approaches [12, 13]. As depicted in Figure 2, the hybrid CTC/AED ASR model mainly contains three parts, a Shared Encoder, a CTC Decoder, and an Attention Decoder. The Shared Encoder consists of a convolution subsampling layer containing two convolutional layers with stride 2 for downsampling, a linear projection layer, and a positional encoding layer, followed by multiple Conformer [14] encoder layers. The CTC Decoder consists of a linear layer and a log softmax layer. The Attention Decoder consists of a positional encoding layer, multiple Transformer [15] decoder layers, and a linear projection layer.

The main idea of our paper is to introduce sememe-based semantic information into ASR simply and effectively. Inspired by [10], we use three simple but effective strategies: 1) adding sememe prediction auxiliary task with sememe loss in a multi-task learning manner, 2) simply adding sememe information to the textual representation, or 3) employing Sememe Encoder to improve the semantic representation ability of the text.

In this paper, we train our proposed E2E ASR on public Mandarin datasets Aishell-1 [18]. The Aishell-1 corpus consists of 178 hours of labeled speech collected from 400 speakers. The content of the datasets covers 5 domains including Finance, Science and Technology, Sports, Entertainments, and News. To compare the domain adaptation ability of ASR in the text domain while minimizing the influence of differences in the acoustic environment, we test the trained model on another public Mandarin dataset Aishell-2 [19] that has a similar acoustic environment for sound recording but the corresponding text contents cover different text domains. The Aishell-2 corpus consists of 1000 hours of labeled speech collected from 1991 speakers. The content of Aishell-2 corresponds to the domains of voice commands, digital sequence, places of interest, entertainment, finance, technology, sports, English spellings, and free speaking without specific topics. To further illustrate the robustness of our method, we also conduct evaluation experiment on different domains with WenetSpeech [20], which is a multi-domain Mandarin corpus consisting of high-quality labeled speech but a relatively more complex acoustic environment than Aishell-1.

For all experiments, we use the open-source WeNet toolkit [11] to build both the hybrid CTC/attention baseline and our proposed SememeASR. And we used the default values in the WeNet for the main parameters which have been validated by the WeNet contributor. The input features are 80-dimensional log Mel-filterbank (FBank) computed on 25ms window with 10ms shift. We use SpecAugment [21] and speed perturb for data augmentation. We choose 4233 characters (including 〈blank〉, 〈unk〉, 〈sos/eos〉 labels) as model units for Aishell-1.

We construct the foundation model using 12 Conformer blocks in the Shared Encoder and 6 Transformer blocks in the Attention Decoder. We employ $h$ = 4 parallel attention heads in both Conformer block and Transformer block. For every layer, we use $d_{k}=d_{v}=d_{model}/h$ = 64, $d_{ffn}$ = 2048. Our proposed SememeASR model adds sememe encoder module and sememe prediction auxiliary task based on the baseline.

We train the model with Adam Optimizer [15] for at most 240 epochs with a batch size of 12. And learning rate = 0.002, warm up = 25000, and gradient clipping at 5.0. Additionally, during training, we employ the gradient accumulation method, in which the gradients are modified every four batches. Moreover, we employ label smoothing of value $\epsilon_{ls}$ = 0.1 and dropout rate of $P_{drop}$ = 0.1. We set the weight $\lambda$ of the CTC branch during joint training to 0.3. Considering that $\alpha$=0.3 achieves better results in our experiment, we choose it as the weight parameter of sememe loss. During joint decoding, we set the CTC-weight $\lambda$ to 0.5. To avoid overfitting, we averaged the 30 best model parameters in the development dataset.

We first present the results on the Aishell-1 test dataset. Table 1 compares the CER results of different models. From the results of Aishell-1, we can see our proposed SememeASR model is better than the baseline hybrid CTC/AED model.

We also compare the results on the Aishell-2 test and dev datasets, which have a similar acoustic environment with Aishell-1 but cover different text domains. From the results of Aishell-2 in Table 1, we can see that our proposed SememeASR model outperforms the baseline on the new domain data. This indicates the improvement in the domain generalization capability of our proposed model.

To further illustrate the validity of our approach in more difficult text domains and more complex acoustic environments, we conduct further experiments on different domains of WenetSpeech. Experimental results in Table 2 indicate that our method also outperforms the baseline in new domains.

To evaluate the ability of the model to recognize long-tail data, we first counted the long-tail data according to the methodology of [22]. The characters in the bottom 95% of occurrences in the training set were used as long-tail characters. In addition, in order to analyze the impact of long-tail data in more detail, we have divided 10 intervals based on the ratio of the number of long-tail characters in the sentence.

According to the result of Figure 7, the proposed method can improve the recognition of long-tailed data, which will also lead to a little reduction in the recognition of head data. However, as the proportion of long-tailed characters in a sentence increases, our model is less effective than the baseline model. It suggests that in the case of poor recognition, wrong results bring wrong semantic information, which hinders the role of semantic information.

Furthermore, we also evaluate the ability of the model in recognizing long-tail data in the Aishell-2 dataset. As shown in Figure 8, the proposed method can also improve the recognition of long-tail data overall on the new domain data.

To further analyze the effect of sememe information on the different parts of the model, we adopted different decoding approaches for our experiments. Similar to [11], we adopt four decoding methods, attention, CTC greedy search, CTC prefix beam search, attention rescoring.

As shown in Table 3, in the attention and attention rescoring decoding methods, our proposed method performs better than the baseline CTC/AED model. However, at the same time, the performance of the CTC-based decoding approach is inferior to that of the baseline. It indicates that our proposed approach increases the modeling capability of the Attention Decoder, but has an impact on the Shared Encoder. According to previous studies [23, 24], this is because the Shared Encoder part is more correlated with acoustic modeling, while the Attention Decoder part is correlated with language modeling. Our approach improves the language modeling capability, but the coupling of acoustic modeling and language modeling makes our proposed method inevitably influential in its acoustic modeling component. And thus the effect of using the CTC decoding approach is degraded.