Towards Fast and Accurate Streaming End-to-End ASR

	$\displaystyle\log{\mathbf{P}}_{\text{RNN-T}}(y_{U}|{\bf x}_{t})\mathrel{{-}{=}}\big{(}$	$\displaystyle\max(0,~{}\alpha_{\text{early}}*(t_{\texttt{</s>}}-t))+$
		$\displaystyle\max(0,~{}\alpha_{\text{late}}*(t-t_{\texttt{</s>}}-t_{\text{buffer}}))\big{)}$		(1)

Extending RNN-T’s output vocabulary with a special token </s> helps improve its latency [2], as the endpointing decision is made jointly with the model rather than with a separate endpointer. However, there is no constraint on when </s> should occur during training. A premature </s> prediction can result in deletion errors, while late predictions of </s> can increase latency as </s> is used to inform the system when the speech ends. In this paper, we address these issues by applying additional early and late penalties on the </s> token (Equation (1)). Specifically, during training for every input frame in $\{{\bf x}_{1},\dots,{\bf x}_{T}\}$ and every label $\{y_{1},\dots,y_{U}\}$, RNN-T computes a $U\times T$ matrix ${\mathbf{P}}_{\text{RNN-T}}({\bf y}|{\bf x})$, which is used in the training loss computation. The last label $y_{U}$ is always </s> . We denote $t_{\texttt{</s>}}$ as the frame index after the last non-silence phoneme, obtained from the forced alignment of the audio with a conventional model. The RNN-T log-probability $\log{\mathbf{P}}_{\text{RNN-T}}(y_{U}|{\bf x})$ is modified to include a penalty at each time step $t$ for predicting </s> too early or too late. $t_{\text{buffer}}$ gives a grace period after the reference $t_{\texttt{</s>}}$ before the late penalty is applied. $\alpha_{\text{early}}$ and $\alpha_{\text{late}}$ are scales on the early and late penalties respectively. All hyper parameters are tuned experimentally.

Minimizing RNN-T loss corresponds to improving the log-likelihood of the training data. However, ASR system performance is measured in terms of WER, not log-likelihood. To address this mismatch, [19] proposes to minimize expected WER of the RNN-T model by approximating the expectation with samples draw from the model. Minimum word error rate training (MWER) is later applied to attention based LAS E2E models [3].

During the beam search decoding of the RNN-T EP model, the inference is terminated when either a blank symbol is generated at the last input frame or an </s> token is predicted. Premature </s> prediction results in deletion of the remaining reference target sequence, leading to a large sequence loss. This makes it more suitable for sequence training techniques. In this work, we hence investigate MWER training with N-best hypotheses for the RNN-T EP model.

Non-streaming E2E models such as Listen, Attend and Spell (LAS) has shown better performance than streaming ones such as RNN-T. LAS has been explored to serve as a second pass rescorer[4], that can still fit within the on-device latency constraints. As illustrated in Figure 1, the model first collects the output of the RNN-T encoder of all the frames $\bf{e}=[\bf{e}_{1},\dots,\bf{e}_{T}]$. They are then forwarded through an extra LAS encoder to generate a new set of encoder features for the LAS decoder. The decoder then computes output $\bf{y}_{\text{LAS}}$ accordingly. During inference, we first pick the top-K hypotheses from the RNN-T decoder. We then run the LAS model on each sequence in the teacher-forcing mode to compute a score, which combines log probability of the sequence and the attention coverage penalty [20]. The sequence with the highest LAS score is picked as the output sequence.

One of the issues in [4] is that RNN-T did not produce a score for </s> , while LAS is indeed trained to produce a score for it. Thus, when rescoring RNN-T hypotheses, an “artificial” </s> score for RNN-T was added to the </s> from LAS. One can argue that including a score for </s> generated from RNN-T based on the inputs should help recognition, as it gives more confidence as to if the sentence should actually be completed. In this work, we look at improving LAS rescoring with the RNN-T EP model by including a score for </s> . The use of </s> token in RNN-T makes the score combination with LAS more consistent acorss all the output units. It is important to note that LAS rescoring cannot make the RNN-T model emit </s> faster; it can only improve the WER of RNN-T. However, the improvement of WER may provide additional room to trade WER for latency.

We use the same multidomain dataset as [21] for training. Multistyle training (MTR) is used for noise robustness [22]. During training, a noise configuration, which defines mixing conditions like the size of the room, reverberation time, position of the microphone, speech and noise sources, signal to noise ratio (SNR), etc, for each utterance is randomly sampled from a collection of 3 million pre-generated configurations. The detailed noise configuration can be found in [21]. The test set we use consists of 14K Voice Search utterances with duration less than 5.5 seconds long. They are all anonymized and hand-transcribed, and are representative of Google traffic.

The input waveforms are framed using a 32 msec window with 10 msec shift. Globally normalized 128 dimension logmel features extracted from frequencyies spanning from 125 Hz to 7.5kHz are used as inputs. The input window size is 4, consisting of 3 frames on the left and no future context. It is further subsampled by a factor of 3 making the system operate at 33 Hz [23].

Similar to [24], multidomain models are trained with domain id as an additional input for learning domain-dependent variations. Following [1], all LSTM layers in the model are unidirectional, with 2048 units and a projection layer with 640 units. The RNN-T encoder consists of 8 LSTM layers, with a time-reduction layer after the second layer. The RNN-T decoder consists of a prediction network with 2 LSTM layers, and a joint network with a single feed-forward layer with 640 units. The additional LAS encoder consists of 2 LSTM layers. The LAS decoder consists of multi-head attention [25] with 4 attention heads, which is fed into 2 LSTM layers. All models are trained on 8x8 Cloud TPU using the Tensorflow Lingvo toolkit [26] to predict 4,096 word pieces including the </s> token.

Despite the use of multidomain training, this work focuses only on the Voice Search task. We append the </s> token only to the Voice Search queries and keep the other data untouched. We report both the recognition performance in terms of word error rate (WER) and the latency of the models for Voice Search only. The latency metrics used in this paper includes median latency (EP50), 90 percentile latency (EP90) and the endpointing coverage (EOU) which represents the percentage of the test data actually receives an end-of-utterance signal from the endpointer model.

There is a trade-off between accuracy and latency, which is often depicted by ROC curves. For EOU EPs, it is obtained by adjusting the endpointing decision threshold. For RNN-T EPs, the endpointing decision is defined by:

$$p(\texttt{</s>}|{\bf x}_{1},\dots,{\bf x}_{t},y_{0}^{\text{RNN-T}},\dots,y_{t-1}^{\text{RNN-T}})^{\alpha_{\texttt{</s>}}}\geq\beta.$$

(2)

$\alpha_{\texttt{</s>}}$ is a penalty term for the posterior of </s> that modifies the ordering for the hypothesis with </s> . $\beta$ is a predefined threshold that determines if </s> is allowed in the search beam [2]. Sweeping $\alpha_{\texttt{</s>}}$ and $\beta$ gives us a ROC curve of the WER and latency trade-off. For simplicity, we most of the time report a single trade-off point and only show the ROC curves at the end for the final comparisons.

We first train a RNN-T model to predict 4,096 word pieces for the ASR task only (no </s> ) as was done in past [1]. This RNN-T cannot be used to output an endpointing decision and an external EOQ EP is used [16, 2] (B1 in Table 1). The endpointer and the RNN-T ASR model are trained independently and at the inference time, the information from RNN-T’s hypotheses cannot be used for endpointing decisions. To address this issue, we also trained a joint endpointing and recognition RNN-T EP model proposed in [2] (B2 in Table 1). As suggested in [2], we also use the independnetly trained EOQ EP as a backup for the RNN-T EP model. From Table 1, the RNN-T EP (B2) shows good latency gains (130ms EP50 and 200ms EP90 latency reductions and a 5.4% absolute EOU coverage improvement) but has an increase of 0.3% WER. One assumption of this regression is that during training </s> is treated the same as all the other tokens, with no constraint on how early or late </s> should occur; however in inference, a path ends when a </s> token is predicted. Predicting EOS prematurely brings in deletion errors.

To address the potential premature </s> prediction, we adopt an early penalty term to the training. It is added only if </s> is predicted at any frame earlier than its ground truth time. When adding the early penalty, we scale it by a factor of 0.1 which is found to work well. This (E1 in Table 2) reduces the WER from 7.5% to 7.2% but degrades on latency comparing to B2. The use of early penalty does help the model to address premature </s> prediction but has the risk of the model learning to over-delay its predictions, which leads to worse latency. The regression on EP90 is more sever which is because many tail cases are not endpointed by RNN-T EP and they simply fall back to the EOQ EP.

We further introduce a late penalty term to penalize the </s> prediction that happens too late comparing to the ground truth. During training the granularity of the time is frame (particularly 60ms in our setup). We experimented with $t_{\text{buffer}}=\{3,5,7\}$ which corresponds to a grace period of 180ms, 300ms and 420ms after the reference </s> label. The results are presented in Table 2. With 3 frames’ buffer, we obtain the best median latency but 5-frame gives the best 90-percentile latency which is still worse than B2. We take model E2, namely the RNN-T EP model with early penalty and 3-frame late penalty, as the setup for following experiments.

For the RNN-T EP model, a wrong prediction of </s> leads to not just a token error but a sequence level loss as it is used to terminate a path in beam search. MWER training optimizes sequence level loss and penalizes WER when </s> is emitted too early, thus prompting our investigation in this section.

We conducted MWER training for the RNN-T model without </s> (B1) and the best RNN-T EP (E2). Both the pre- and post-MWER results are reported in Table 3. For B1, the latency is controlled by a separate EOU EP and hence remains the same after MWER training (B3). But the WER reduces from 7.2% to 6.9%. While for E2, MWER training (E4) maintains the same 7.2% WER as E2 but achieves 220ms EP90 reduction with 50ms regression on EP50. Because optimizing MWER already penalizes premature </s> predictions, we turn off the early penalty for MWER training. This (E5 in Table 3) reduces WER from 7.2% to 6.9% WER and more importantly it still yields a 270ms EP90 latency reduction while maintaining the same EP50 latency as E2. MWER training of the RNN-T EP model with only late penalty can bring in both WER and latency improvements. Comparing to B2, E5 gives 8.0% relative WER reduction and 30ms EP50 and 130ms EP90 latency reductions.

So far we see good latency reductions, but the WER gains are small. In the literature, two-pass model that runs RNN-T as the first pass streaming model for fast response and LAS as the rescorer has been shown to be effective in WER reductions. We hence investigate the effect of LAS rescoring on RNN-T EP model. We took the pre-MWER model E2 and added an additional encoder with two LSTM layers and an extra LAS decoder (Figure 1). They are trained with cross entropy (CE) loss with the RNN-T weights frozen. The results are presented as E6 in Table 4. In this work, the latency is only measured for the first pass model. With the same decoding configuration as E2, LAS rescoring reduces the WER by 11.1% relative from 7.2% to 6.4%. Although LAS rescoring cannot directly affect first pass latency, with the WER gains, we may be able to trade WER for latency. We further swept the penalty scale for </s> and obtained an operation point with the same 6.4% WER but 10ms EP50 and 130ms EP90 reductions. As mentioned in Section 2.3, one problem for LAS rescoring of RNN-T without </s> as done in [4] is that RNN-T does not generate an explicit </s> score to combine with that from LAS. To simulate that effect, we zeroed out the </s> score from RNN-T EP and swept a global value to be combined with LAS </s> score. The result (E6 + ignore RNN-T </s> score in Table 4) shows an increase in WER from 6.4% to 6.6%, highlighting the benefit of using </s> in RNN-T EP for LAS rescoring.

Instead of CE loss, MWER loss of RNN-T outputs can be used to update the LAS rescorer (E7 in Table 4). It further reduces the WER down to 6.2% and obtains 100ms EP90 reductions. Moreover, when we update both RNN-T and LAS during MWER (E8), we can obtain another 70ms EP90 reduction. Comparing to B2, the RNN-T EP + LAS with MWER training gives a 18.7% relative WER reduction and 40ms EP50 and 160ms EP90 reductions.

The proposed RNN-T EP with late penalty and MWER training (E5) gives us both WER and latency improvement over the RNN-T (B1) and the original RNN-T EP (B2). Further WER improvement is achieved via a second pass LAS rescorer (E8). In this section, we compare these systems across different operating points. We plotted the WER vs latency (EP90) curve for these four models (B1, B2, E5, E8) in Figure 2 by varying the penalty scale $\alpha_{\texttt{</s>}}$ and threshold $\beta$. Lower curves are better. RNN-T (B1) tends to delay outputs and has worse latency. With </s> , RNN-T EP (B2) addresses the latency problem but with some WER degradations. With the modifications proposed in this work, namely late penalty, MWER and LAS rescoring, both E5 and E8 have much better WER and latency trade-offs.