Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model

Continuous integrate-and-fire (CIF) is a soft and monotonic alignment mechanism for E2E ASR different from time-synchronized models and label-synchronized models. CIF performs integrate process over the output of encoder $\mathbf{e}_{1:T}$ and predicts frame-level weights $\mathbf{\alpha}_{1:T}$, once the accumulated weights exceed the fire threshold, these frames will sum up to acoustic embedding $\mathbf{E}_{1:L^{{}^{\prime}}}$ which is synchronized with output tokens. Such process is illustrated in Fig 2. In ASR models, the modeling character of CIF is of great value. The frame-level weights actually conduct alignment between acoustic representation and output tokens, and the fire location indicates the token boundary. Such capacity delights us to explore the feasibility of using CIF to predict accurate timestamps of decoded tokens naturally in the process of recognition.

We adopt Paraformer - a novel NAR ASR model which achieves non-autoregressive decoding capacity by utilizing CIF and two-pass training strategy inside an AED backbone. Avoiding the massive computation overhead introduced by autoregressive decoding and beam-search, Paraformer gains more than 10x speedup with even lower error rate comparing to Conformer baseline. The overall framework of Paraformer is illustrated in Fig 3.

Paraformer contains three modules, namely encoder, predictor and parallel decoder (sampler interpolates acoustic embeding and char embedding without parameters). Encoder is same as AR encoders of Conformer which contains self-attention, convolution and feed-forward networks (FFN) layers to generate acoustic representation $\mathbf{e}_{1:T}$ from down-sampled Fbank features. Predictor uses CIF to predict the number of output tokens $L^{{}^{\prime}}$ and generate acoustic embedding $\mathbf{E}_{1:L^{{}^{\prime}}}$. Parallel decoder and sampler conduct two-pass training with the vectors above: $\mathbf{E}_{1:L^{{}^{\prime}}}$ is directly sent to decoder to calculate cross-attention with $\mathbf{e}_{1:T}$, and decode all tokens $y^{{}^{\prime}}_{1:L^{{}^{\prime}}}$ in NAR form at once. Then sampler interpolates $\mathbf{E}_{1:L^{{}^{\prime}}}$ and char embedding $\mathbf{c}_{1:L}$ according to the edit distance between hypothesis $y^{{}^{\prime}}_{1:L^{{}^{\prime}}}$ and $y_{1:L}$. The interpolated vector (named semantic embedding, noted by $S_{1:L^{{}^{\prime}}}$) is sent to parallel decoder again to conduct the same calculation, which makes up the second pass. Note that the forward process of the first pass is gradient-free, cross-entropy (CE) loss is calculated between output of the second pass and ground truth. In the training process, as the accuracy of first pass decoding raises, $\mathbf{E}_{1:L^{{}^{\prime}}}$ makes up increasing proportion of $S_{1:L^{{}^{\prime}}}$. In the inference stage, Paraformer use the first pass to decode.

CIF is of vital importance in Paraformer as it predicts the number of output tokens and generates monotonic token boundaries implicitly. Fig 4 shows the origin fire place comparing to the timestamp generated by kaldi force-alignment system. Weights $\alpha$ shows that the pattern of CIF is regard less of the real length of tokens, for an ASR model with 4-times down-sampling layer (60 ms each time step), the integrate process for each token finishes in around 4 frames, which leads to a large offset for end point timestamp of character with long duration.

Original CIF calculates $\alpha$ with sigmoid activation function after feed-forward network, sharp spikes and glitches can be observed from the curve in Fig  4. We propose to scale and smooth the CIF weights with the operation below:

First, ReLU function and $\beta$ smooth the glitches of $\alpha$, cutting off the gradient towards encoder of irrelevant position, which is supposed to be beneficial for ASR task. Besides, scaling the output of ReLU with $\gamma$ relieves the spikes of the curve, trying to achieve level and smooth weights.

Another optimization comes about from the aspect of post-processing. According to the distribution of alphas observed in Fig 4, most of the weights begins to accumulate from the exact frame but the accumulation ends in fixed steps (4 frames in the figure), which delights us to conduct the following 4 processing strategies:

Begin/end Silence As the begging place of CIF is always precise, the beginning frames with weights under a threshold $\theta_{s}$ is considered as silence. So as to the frames at the end but we set an interval of 3 frames.

Fire Delay Original CIF timestamps are always unreliable for the end point prediction for long-lasting tokens. We propose to conduct fire delay operation when frames with low weights are observed, such frames with be grouped to the previous token except the last frame.

Silence Insertion When the low-weight frames last longer than $L_{s}$, we insert a silence token in between.

Weight Averaging¹¹1In this paper, we conduct experiments with 4-times down-sampling encoder, this methods is thus not validated. For ASR models with higher down-sampling rate, the process of integrate finishes even faster. When using 6-times low frame rate features, the CIF tends to output weights around 1.0 or 0.0, which makes CIF weights no longer stand for the accumulation procedure. Then we proposed to weaken the weight spikes.

$\theta_{s}$ and $L_{s}$ are model related hyper-parameters, for Paraformer model with original CIF predictor and 4-times down-sampling encoder embedder, we set $\theta_{s}=0.05$ and $L_{s}=3$, the timestamp thus generated of the demo utterance above is shown in Fig 4. Subjectively, the proposed post-processing strategies optimize the quality of CIF timestamps in a simple but efficient way.

In addition to the visual analysis of timestamps, we propose to use accumulated averaging shift (AAS) and diarization error rate (DER) as the evaluation metrics of timestamp accuracy.

where $K$ is for the number of aligned token pairs²²2Considering the transcriptions of two timestamps might differ, only paired tokens according to edit distance are included in the calculation..

Two datasets are used in our experiments. First, Aishell-1 is used for training the Paraformer model and visualizing timestamps. Aishell-1 contains 178 hours speech data with transcription, which is a widely used open-source Mandarin ASR corpus. Besides, we use a TTS dataset called M7 for evaluation, which contains 5550 Mandarin utterances. Except transcriptions, M7 also contains manually marked timestamps in token level, which is regarded as reference in the calculation of AAS and DER.

The Paraformer model is trained from scratch using ESPnet toolkit with the following setups. The model contains 12-layer Conformer encoder (implemented as with kernel size 15) and 6-layer Transformer decoder with attention dimension $d_{attn}=256$ and feed-forward network dimension $d_{linear}=2048$. The input layer of encoder conducts 4-times down-sampling for Fbank features, one step of encoder output thus stands for 40ms. The model is jointly trained with CTC loss ($\alpha=0.3$) and we set dropout rate $r_{dropout}=0.1$ for entire model. 3-times speed perturbation is adopted for Aishell-1 data and we apply spec-augment with 2 frequence masks range in $[0,30]$ and 2 time masks range in $[0,40]$ for each utterance. We use dynamic batch size ($numel=2000k$) and $n_{epoch}=50$. No language model is used in the inference stage. For scaled-CIF, we use $\gamma=0.8$ and $\beta=0.05$³³3For ASR models of different frame rate and encoder down-sampling rate, we fine it better to adjust scaling coefficients $\gamma$ and $\beta$, so as to the post-processing hyper-parameters $\theta_{s}$ and $L_{s}$ to achieve better performance..

We prepare the force-alignment system with Kaldi toolkit as baseline timestamp, the training setup and model configuration is exactly same as open-source recipe Aishell-1, from flat-start to HMM-GMM. Timestamps of phonemes are extracted from lattice using model tri5a, and then converted to characters timestamps.

In this section we evaluate the quality of the timestamps and analyse the effect of the proposed methods. Table 1 shows the AAS and DER metircs of the timestamps from different models over testset M7. First we test the force-alignment system with ground truth transcription and Paraformer recognition results. It turns out that using these two kinds of transcription lead to slight difference of DER and almost the same AAS, which is because the majority error of Paraformer’s recognition results is substitution error, reference and hypothesis have nearly the same expanded phoneme sequences, thus AAS of FA-GT and FA-HYP differs little.

Comparing CIF-0 and FA-HYP, it is obvious that original CIF weights as timestamps is of unacceptable quality. With the addition of post-processing strategies, the accuracy of CIF timestamps has been improved step by step, CIF-3 achieves 47.4% AAS reduction. Among the three post-processing methods, fire delay is the most effective and also the most tricky one. Scaled-CIF brings further help to timestamps prediction, 32.9% AAS reduction and 34.5% DER reduction are observed without any post-processing strategies. In SCIF-3, CIF timestamps outperforms force-alignment systems in AAS (11.3% relatively), but DER is still higher.

Fig 5 shows comparison of manually marked timestamps, original CIF timestamps and optimized timestamps. Comparing the blue curve and the purple curve, it can be observed that the peak of the curve is weakened to some extent, and the glitches in the curve disappear (not obvious in the figure). Considering the start and ending time of each token, the effect of fire delay is obvious, especially when the token is followed by low weight frames (like character ’家’ in the demo).

Results in section 5.3 shows that scale-CIF significantly improves timestamp prediction accuracy, and we find it is also benificial for ASR. Comparing to vanilla Conformer AR model, Paraformer achieves better recognition accuracy and even lower real time factor (RTF) [11]. On Aishell-1 task, Paraformer gets 4.6%, 5.2% CER over dev and test set and the RTF is 0.0065 (RTF of Conformer is 0.1800 under the same test environment). In our experiments, Paraformer with scaled-CIF outperforms the baseline, CER on Aishell-1 dev and test set are 4.5% and 5.2% respectively. On testset M7, Paraformer gets 11.70% CER with origin CIF and 11.26% with scaled CIF (3.76% relative CER reduction). Such results prove that smoothing the weights of CIF and cutting off the gradient towards encoder of irrelevant frames are beneficial for recognition task of Paraformer.