Less is More: Improved RNN-T Decoding using Limited Label Context and Path Merging

End-to-end models such as the recurrent neural network transducer (RNN-T) [1, 2, 3], attention-based encoder-decoder models [4, 5], the transformer transducer [6, 7] have become increasingly popular alternatives to conventional hybrid systems [8] for automatic speech recognition. These models produce hypotheses in an autoregressive fashion by conditioning the output on all previously predicted labels, thus making fewer conditional independence assumptions than conventional hybrid systems. End-to-end ASR models have been shown to achieve state-of-the-art results [9, 10] on popular public benchmarks, as well as on on large scale industrial datasets [11, 12].

The increase in modeling power afforded by conditioning on all previous predictions, however, comes at the cost of a more complicated decoding process; computing the most likely label sequence exactly is intractable since it involves a discrete search over an exponential number of sequences each of which corresponds to a distinct model state. Instead, decoding is performed using an approximate beam search [13], with various heuristics to improve performance [14, 15]. Since model states corresponding to different label histories are unique, beam search decoding produces a tree of hypotheses rooted at the start of sentence label ($\left<\text{sos}\right>$). This, combined with a limited beam size, restricts the diversity of decoded hypotheses – a problem that becomes increasingly severe for longer utterances.

In this work, we conduct a detailed investigation of a specific aspect of streaming end-to-end models (RNN-T, in our work); we study the importance of conditioning the output sequence on the full history of previously predicted labels, and investigate modifications to the beam search process which can be applied to models by limiting context. The first part of this question has been investigated in a few papers recently, in different contexts. Ghodsi et al. [16] find that in low-resource settings, where training data is limited, the use of word-piece units [17] allows for a stateless prediction network (i.e., one which conditions on only one previous label) without a significant loss in accuracy. Zhang et al. [6] investigate the impact of varying label context in the transformer-transducer model (RNN-T which replaces LSTMs with transformer networks [18]) finding that a context of 3-4 previous graphemes achieves similar performance as a full-context baseline on the Librispeech dataset. Finally, Variani et al. [19] find that the hybrid autoregressive transducer (HAT; RNN-T with an ‘internal language model (LM)’), trained to output phonemes and decoded with a separate lexicon and grammar achieves similar performance when context is limited to two previous phoneme labels on a large scale task.¹¹1It should be noted, however, that in this case the effective context is larger than two previous phonemes due to the linguistic constraints introduced by the lexicon and the n-gram LM.

Our work differs from the previously mentioned works in two ways. First, we study the question on a large-scale task using a model trained to output word-piece targets and decoded without an external lexicon or language model. Since word-pieces naturally capture longer context than graphemes of phonemes, our results allow us to measure the effective context that is captured by full-context RNN-T models. Second, we consider modifications to the beam search process which are enabled by the use of limited context models (or the baseline, with approximations). This process, described in detail in Section 3.1, allows us to generate lattices by merging paths during the search process, unlike RNN-T systems which are typically decoded to produce trees. The proposed approach is similar to previous work on efficient rescoring with neural LMs [20] and to generating lattices in attention-based encoder-decoder models [21]. The ability to produce rich lattices from sequence-to-sequence models has many potential applications: e.g., they can be used for spoken term detection [22]; as inputs to spoken language understanding systems [23]; or for computing word-level posteriors for word- or utterance-level confidence estimation [24]. In experimental evaluations, we find that the models require 5-gram contexts (i.e., conditioning on the four previously predicted labels) in order to obtain comparable WER results as the baseline. Additionally, we find that the proposed path-merging scheme is an effective technique to improve search efficiency. The proposed scheme reduces the number of model evaluations by up to 5.3%, while simultaneously improving the oracle WERs in the decoded lattice by up to 36% without any degradation in the WER.

The organization of the rest of the paper is as follows: in Section 2 we introduce the baseline RNN-T models; in Section 3 we describe how we limit label context during training, and our proposed path-merging scheme during decoding to produce lattices; we describe our experimental setup and results in Sections 4 and 5, respectively, before concluding in Section 6.

We assume that the input speech utterance has been parameterized into suitable acoustic features: ${\mathbf{x}}=[{\mathbf{x}}_{1},\ldots,{\mathbf{x}}_{T}]$, where ${\mathbf{x}}_{t}\in{\mathbb{R}}^{d}$ (log-mel filterbank energies, in this work). Each utterance has a corresponding label sequence, ${\mathbf{y}}=[y_{1},\ldots,y_{U}]$, where $y_{u}\in\mathcal{Y}$ (word-pieces [17], in this work).

The RNN-T model was proposed by Graves [1, 2] as a generalization of connectionist temporal classification (CTC) [25]. The RNN-T model defines a probability distribution over the output label sequences conditioned on the input acoustics, $P({\mathbf{y}}|{\mathbf{x}})$, by marginalizing over all possible alignment sequences, ${\mathbf{\hat{y}}}$. Specifically, RNN-T introduces a special blank symbol, $\left<\text{b}\right>$, and defines the set of all valid frame-level alignments, $\mathcal{B}({\mathbf{x}},{\mathbf{y}})$, as the set of all label sequences, ${\mathbf{\hat{y}}}=(\hat{y}_{1},\ldots,\hat{y}_{T+U})$, where $\hat{y}_{i}\in\mathcal{Y}\cup\left\{\left<\text{b}\right>\right\}$, such that ${\mathbf{\hat{y}}}$ is identical to ${\mathbf{y}}$ after removing all $\left<\text{b}\right>$ symbols.

A vanilla RNN-T model is conditioned on all previous predictions, and can thus be thought of as a full-context model. We can limit the context of the RNN-T model by modifying the prediction network to only depend on a fixed number of previous labels. Specifically, a model with $K$-gram context is conditioned on at most $K-1$ previous labels so that the output distribution of the RNN-T computes: $P(\hat{y}_{i}|{\mathbf{x}}^{t_{i}}_{1},y^{u_{i}}_{u_{i}-K+2})$. Since our baseline full-context RNN-T model uses a stack of LSTM layers to model the prediction network, in this work we limit the context through the use of an LSTM-based prediction network that is reset and sequentially fed only the sequence of the last $K-1$ labels at each step. This ensures that our results are comparable to the baseline configuration. However, other choices would also be reasonable to model a limited context prediction network: e.g., a transformer as in [6], or a simple feed-forward network with $K-1$ inputs. Each of these choices involves a different tradeoff in terms of computation versus runtime memory usage, and we leave the study of these alternate architectures for future work.

Model Architecture: Our experimental setup is similar to our previous work [12]. The input acoustics are parameterized using 128-dimensional log-mel filterbank energies computed over the 16KHz range following [27], with 32ms windows and a 10ms hop. In order to reduce the effective frame rate, features from four adjacent frames are concatenated together (to produce 512 dimensional features), which are further sub-sampled by a factor of 3, so that the effective input frame rate is 30ms. In this work, we also apply SpecAugment masks [9] using the configuration described in [28], which we find to improve performance over the system in [12]. The encoder network in all of our experiments is modeled using a stack of 8 unidirectional LSTM [26] layers, each of which contains 2,048 units and a projection layer of 640 units. We add a time-reduction layer after the second LSTM layer which stacks two adjacent outputs and sub-samples them by a factor of 2, so that the effective encoder frame rate is 60ms. The prediction network is modeled using two layers of unidirectional LSTMs [26], each of which contains 2,048 units with a projection layer of 640 units. The joint layer is modeled as a single-layer feed-forward network with 640 units. The output of the joint network produces a distribution over 4,096 word-pieces [17] which are derived from a large text corpus. In total, each of our models contains $\sim$120 million trainable parameters.

Training Sets: Our models are trained on a diverse set of utterances from multiple domains including voice search, telephony, far-field, and YouTube  [27]. All utterances are anonymized and hand-transcribed; YouTube transcriptions are obtained using a semi-supervised approach [29]. In order to improve robustness to environmental distortions, models are trained with additional noisy data using a room simulator [30]. Noise samples are drawn from YouTube and from daily life noisy recordings. The noisy data are generated with an SNR of between 0dB and 30dB, with an average SNR of 12dB; T60 reverberation times range from 0 – 900ms with an average of 500ms. We employ mixed-bandwidth training [31], by randomly downsampling the training data to 8KHz 50% of the time.

Test Sets: Results are reported in two test domains: the first consists of utterances drawn from Google voice search traffic (VS: 11,585 utterances; 56,069 words); the second set consists of data drawn from non-voice search Google traffic (NVS: 12,426 utterances; 127,105 words). The utterances in the NVS set tend to be longer (both in terms of duration and label token length) on average than utterances in the VS set. The 90th percentile token sequence length is $\sim$14 word-pieces for the VS set, and $\sim$28 word-pieces for the NVS set. Results are also reported more challenging versions of the VS (VS-hard: 9,662 utterances; 46,673 words) and NVS test sets (NVS-hard: 19,411 utterances; 198,215 words) that contain more variation in terms of background noise, volume, and accents. All test set utterances are anonymized and hand-transcribed.

Training: Models are trained using Lingvo [32] in Tensorflow [33], with $8\times 8$ Tensor Processing Units (TPUs) [34]. Models are optimized using synchronized stochastic gradient descent with mini-batches of 4,096 utterances using the Adam optimizer [35].

Full and Limited Context Models: In addition to the full-context RNN-T model (baseline), we study the impact of limiting context by training models with varying amounts of context ranging from 2–10 (lc-2gram – lc-10gram; i.e., conditioning on 1 – 9 previously predicted labels, respectively).

Decoding: All results are reported after decoding models using the breadth-first search decoding algorithm [15].²²2Similar results are obtained using the best-first search decoding algorithm [1], however these are omitted due to space limitations. The limited context models are always evaluated using the path-merging process proposed in Section 3.1. We also consider the approximate path-merging scheme described in Section 3.1 applied to the baseline during inference. Note that in this case, the baseline is always trained with full-context; during evaluation, we merge paths if the last 1 – 9 non-blank labels are identical for two paths (base-pm2 – base-pm10), and retain the state corresponding to the lower cost path. All models are decoded with a maximum beam size of 10 partial hypotheses and a local beam of 10 (i.e., maximum allowable absolute difference between log-likelihoods of partial hypotheses).

Evaluation Metrics: Results are reported in terms of both word error rate (WER) as well as the oracle WER in the lattice (for the systems with path-merging) or the N-best list (for the baseline configuration). In order to evaluate whether the proposed path merging scheme described in Section 3.1 improves computational efficiency, results are also reported in terms of the number of model states which are expanded during the search. Specifically, we report the average number of joint network evaluations per utterance for each of the test sets (average model states), which serve as a good proxy to measure the cost of the search.

Our results are presented in Table 1. First, we consider the systems trained by limiting prediction network context and evaluated with path-merging. As can be seen in the table, a model with 5-gram context (i.e., conditioned on four previous labels) performs as well as the baseline model across all test sets. In fact, this model outperforms the baseline on the NVS and NVS-hard sets which tend to contain longer utterances. We hypothesize that this is likely a consequence of our proposed path merging strategy which removes redundant paths from the beam, thus allowing the model to represent diverse competing hypotheses as described in Figure 2 and Section 3.1. This is further supported by comparing the Oracle WERs of the limited context systems relative to the baseline. Models with smaller contexts have more opportunities for path-merging, and thus achieve lower oracle WERs relative to models with larger contexts; this comes, however, at the cost of a degradation in the WER. Using 5-gram contexts provides the best balance between the two evaluation metrics, allowing the model to improve oracle WERs by between 14.3–36.4% across the various test sets, with larger improvements on the longer NVS/NVS-hard sets. We also note that our results are in contrast to previous studies with have been conducted on smaller-scale tasks with limited training data [6, 16].

Second, we consider the baseline model which is trained with full-context, but evaluated with path-merging using varying amounts of context. From the results in Table 1, we observe that the baseline models evaluated with path-merging perform at least as well the full-context baseline if not better, as long as we use at least 5-gram context during path merging. As was the case with the models trained with limited-context, using path-merging with 5-gram contexts appears to provide the best WER relative to the baseline configuration. Finally, we note that these systems achieve similarly large Oracle WER improvements as the systems trained with limited context.

In this work, we studied the impact of label conditioning in the RNN-T model on WER and its implications for improving the efficiency of the search. In experimental evaluations, we found that a full-context RNN-T model with word-piece outputs performs comparably to a model trained with context limited to four previous labels. We also investigated modifications to the decoding strategy which are enabled by limiting context: either exactly – by training a model with limited context; or approximately in the baseline – by merging paths during decoding to produce lattices if two paths share the same local label history. The proposed path-merging strategy was shown to improve the oracle WER in the lattice by up to 36%, while improving the efficiency of the search by reducing the number of model evaluations by up to 5% without any degradation in WER.

The ability to create dense lattices which represent alternative hypotheses has a number of potential applications; future work will investigate its effectiveness in improving word-level confidence estimates [24], and in improving WER by rescoring lattices in the second-pass [36] taking advantage of the improved oracle error rates. Finally, we note that our work opens up interesting research directions for new RNN-T architectures which limit prediction network context to just a few previous labels; replacing recurrent LSTM units through such a process can reduce sequential dependencies in the system, thus potentially improving execution speed.