Personal VAD 2.0: Optimizing Personal Voice Activity Detection
for On-Device Speech Recognition

In conventional Personal VAD systems, users need to go through an enrollment process, where a pre-trained text-independent speaker recognition model [14] computes the target user’s d-vector speaker embeddings from the target user’s recordings, to encode their voice characteristics.

The diagram of a conventional Personal VAD is shown in Figure 1. The model first extracts acoustic features $\mathbf{x}\in\mathbb{R}^{T\times D}$ from input audio, where $T$ and $D$ denote the sequence length and feature dimension, respectively, and $\mathbf{x}_{t}$ represents the $t$-th frame of acoustic features. Each $\mathbf{x}_{t}$ is then concatenated with the speaker embedding from the target speaker $\mathbf{e^{\mathrm{target}}}$:

As described in Section 2.1, vanilla Personal VAD models concatenate target speaker embedding with the acoustic feature to incorporate speaker specific information. However, acoustic features and speaker embeddings represent very different information, and they are extracted through entirely separated processes, leading to different distributions and magnitudes. Therefore, simply concatenating them and feeding them to the same layers may significantly limit the model capacity. To address this issue, we propose two novel speaker embedding modulation approaches: 1) through a FiLM layer, and 2) through a speaker pre-net.

In practical on-device ASR use cases, the user may choose to either proceed with or skip the enrollment process. As an ”always-running system”, the Personal VAD model should be able to achieve satisfactory performance in both conditions. To achieve this, we propose a simple but effective training paradigm to guarantee satisfactory performance for both enrollment and enrollment-less conditions, as shown in Alg. 1. This guarantees the model learning the standard VAD behaviors while there is no speaker embedding. In practice, we set $p_{0}=0.2$, which we found to provide reasonable performance for both conditions. Additionally, we add datasets that have no enrollment utterance to the model training, paired with a zero speaker embedding for each utterance. During inference, we also feed an zero speaker embedding to the model under the enrollment-less condition.

Transformer models [18] have been shown to substantially improve almost all speech modeling tasks over LSTMs in recent years [19, 13, 20]. Consequently, we investigate the use of Conformer backbone, an advanced Transformer architecture, for Personal VAD. The vanilla transformer uses the full sequence as the attention context, which leads to a very high latency in streaming systems that is similar to BLSTMs. To make the model stream-able and maximally reduce the latency, we set the model to have a limited left-context and no right-context, following [21, 22].

Due to the limited resources on mobile devices, we adopt a runtime optimization for the on-device inference of Personal VAD. Instead of directly running TensorFlow graph with 32-bit float weights, we quantize the model weights to have 8-bit integers using dynamic range quantization [23, 24] and serialize the models to the TensorFlow Lite format. The quantized model has only 1/4 model size, and more importantly, achieves significant speedup with optimized hardware instructions of integer arithmetic, improving both memory usage and latency for Personal VAD.

We conduct our experiments using a vendor-collected dataset of realistic speech queries. The training set contains 2.6 million utterances ($\sim$1,600 hours) from 6,923 speakers. As suggested by [1], there is no existing datasets that is ideal to Personal VAD evaluations, where each utterance contains natural speaker turns and it contains enrollment utterances for each individual speaker. To simulate the conversational speech, we create a set of synthetic data based on the vendor-collected dataset, following [1]. We first concatenate utterances from multiple speakers, and then we randomly select one of the speakers as the target speaker in the concatenated utterance. Accordingly, we extract the target speaker embedding from the utterance in an enrollment subset. In addition, we also apply two data augmentation techniques, MTR [25] and SpecAug [26], on our datasets to avoid domain overfitting and mitigate concatenation artifacts. Most of the models in the experiments are trained with concatenated dataset alone. Only when training the model for both enrollment and enrollment-less conditions, we used an additional internal training set [7] consisting of 35 million English audio-text pairs ($\sim$27,500 hours) from multiple domains including YouTube and anonymized voice search traffic. The internal training set is only used for enrollment-less condition (where we do not perform any speaker-related operations), and our data handling abides by Google AI Principles [27]. We use the same datasets in our baseline standard VAD training for a fair comparison.

The test set of the vendor-collected dataset of realistic speech queries comprises 194,890 utterances from 1,241 speakers. These utterances are very similar to actual speech query traffic and can contain various ambient noise. As a result, we do not add extra reverberation or noise to it during evaluations. The non-concatenated test set shows the performance when only the target speaker is talking. Similarly, we create a concatenated test set to evaluate the performance of Personal VAD in conversational speech. To reduce the evaluation cost, we randomly samples a subset of 4,000 utterances from the original test set and the concatenated test set, respectively. In the evaluations under enrollment-less condition, we also use 14,000 Voice-search utterances (VS).

We use the same frontend feature as  [7], a 128-dimensions log-Mel filterbank energies extracted with 32ms window and 10ms shift. We stack the features from 4 contiguous frames (512-dimensional) and subsample the sequence by a factor of 3. The LSTM-based models have 3 uni-directional LSTM layers with 256 units and a final fully-connected layer as the classifier. Our Conformer-based models have 4 Conformer layers in the block, each having a dimension of 64, attention head of 8, causal $7\times 7$ convolution kernel, and 31 left-context. The speaker pre-net has two Conformer layers with the same hyper-parameters.

During inference, we compare the posterior of target speaker speech $p_{t}^{\tt tss}$ to a pre-defined threshold ($0.1$ in all our experiments) to decide if a speech frame belongs to the target speaker. We discard non-target speech and non-speech frames and only pass target speech frames to downstream ASR model. We use a medium size ($\sim$ 40M parameters) LSTM-based ASR model [8] among all our experiments, which is trained on the realistic voice search traffic. We use word error rate (WER) to measure the ASR performance with Personal VAD models.

Conformer backbone. We first evaluate the efficacy of the Conformer backbone. As shown in Table 1, using Conformer backbone (E0) significantly improves both Non-Concat and Concat test sets by 2.6 and 9.5 absolute WERs, respectively. More importantly, the model size of the Conformer model is less than a half of the LSTM model size (E0: 2.8MB vs. B0: 5.4MB), suggesting that Conformer-based Personal VAD model can achieve much high parameter efficiency.

After verifying the impact of each aspect of our proposed approach, we deliver the comparison results between the best performing Personal VAD 2.0 model and standard/Personal VAD baselines in Table 2. For a fair model size comparison, we also conduct 8-bit quantization to the standard VAD baseline, as they have been widely used in on-device products. Compared with the two standard VAD baselines, Personal VAD 2.0 can significantly reduce the WER (mostly insertion errors) on conversational speech scenarios (Concat test set), while retaining the performance on regular speech queries under either condition. Personal VAD 2.0 has higher FLOPs, due to the use of Conformer backbone. However, it does not have significant impacts to the overall ASR system latency since VAD has much less computations than ASR. More importantly, conformer backbone makes it possible to process several frames in a the same batch, which further speed up model inference and reduces the latency.

Compared with the conventional Personal VAD, Personal VAD 2.0 not only tremendously improves the performance under the enrollment condition but also generalizes well to enrollment-less conditions. More importantly, with the optimizations based on Conformer backbone and 8-bit weight quantization, we developed a light-weight Personal VAD model that can easily fit into any on-device systems. Meanwhile, we still notice an issue of the current Personal VAD 2.0 model – it performs slightly better on regular speech queries (Non-Concat) under enrollment-less condition than under enrollment condition. A potential reason is that when the target speaker and non-target speaker have very similar voice, the model is more likely to make erroneous predictions. The gap can be reduced by tuning the VAD decision threshold, at the cost of an increasing insertion error rate on conversational speech, resulting in a trade-off to be made according to the deployment scenarios (e.g., for keyword spotting, we need to minimize deletion error, and it is less sensitive to insertion errors).