Enhancing Quantised End-to-End ASR Models via Personalisation

Quantisation is the process of discretising an input with higher information content to one with lower information content. Uniform quantisation is a commonly used quantisation method where the value range is uniformly divided into multiple intervals. This approach is widely supported in mobile inference frameworks such as TFLite [16], MNN [17], and NCNN [18]. Recently, Quantile Quantisation was proposed based on the Lossy Minimum Entropy Encoding [19], and QLoRA was introduced [14] which employs NF4 quantisation on the weights of pretrained neural networks to achieve near-lossless quantisation. Unlike uniform quantisation, Quantile Quantisation determines the quantisation steps and levels based on the data distribution. In data-dense regions, more quantisation levels are allocated, ensuring an equal number of quantised values in each quantisation bin. This allows for a more refined representation of the data, as well as better handling of outliers and adaptation to non-uniform distributions.

The objective of speaker adaptation is to minimize the mismatch between speakers in training and testing conditions. Current neural network-based methods for speaker adaptation can be broadly categorized into two types: Embedding-based and Model-based.

Speaker embeddings map speakers into a continuous space using techniques like i-vectors [20, 21, 22] or neural network bottlenecks [23, 24, 25]. Model-based adaptation methods include three primary methods: Structured Transforms, Speaker Adaptive Training (SAT), and Regularization. Structured Transforms, such as the Learning Hidden Unit Contributions (LHUC) scheme [26] and parameterised activation functions [27], modify the model structure or its activations. SAT methods adjust model parameters to individual speakers using approaches like SAT-embedding [28] and SAT-LHUC [29]. Regularization techniques, such as L2 loss or KL divergence [30, 31], work to prevent overfitting to specific speakers.

The PQM strategy is illustrated in Fig. 1, which is divided into three stages. In stage 1, we apply block-wise NF4 quantisation to the model’s primary weight parameters. In stage 2, we pretrain the randomly initialised LoRA using data from a large number of speakers, providing a more robust starting point for subsequent speaker adaptation. In stage 3, we perform speaker adaptive training on speaker-specific data, during which the entire model is frozen, and only the LoRA parameters corresponding to each speaker are updated.

Stage 2 of PQM is particularly beneficial for the application scenario where reasonable-sized training data of the target domain is available, e.g. assuming there is some in-house data, with a very limited amount of target speaker data.

The block-wise NF4 quantisation is adopted in this paper, which is applied to the weight matrices that are the primary parameters of the model. While standard floating point quantisation applies the same set of quantisation bins to all weight matrices, the dynamic range of parameter values is not taken into account, resulting in heavily unbalanced quantisation bins. NF4, on the contrary, ensures each bin has an equal number of values by estimating the quantile of the input matrices using the empirical cumulative normal distribution. This leveraged the fact that the parameters of a weight matrix, in general, follow a normal distribution [14].

Specifically, for a $k$-bit quantisation, $2^{k}+1$ quantiles (i.e. $2^{k}$ quantisation bins) of a theoretical $\mathcal{N}(0,1)$ distribution are first estimated which equally divides the area under the distribution curve. Then, these quantisation bins are normalised to be within the range $[-1,1]$, as illustrated in Fig. 2. Finally, the parameters of a weight matrix are normalised into the range $[-1,1]$ to find their corresponding quantiles by dividing the maximum absolute value of those parameters. In this way, a similar number of quantised values are obtained for each bin, allowing for a more refined representation of the model parameters. To ensure zero is exactly quantised to zero which is important for padding, asymmetric quantiles are used that ensure the mid bin has the quantisation level of zero (see Fig. 2).

To reduce the influence of extreme values in weight matrices (i.e. outliers) on the maximum absolute value normalisation, block-wise quantisation is applied which divides the weight matrices into small blocks and quantises each block with separate normalisation factors. In this way, outliers in the input tensor are confined to individual blocks, reducing their overall impact on quantisation. As a result, block-wise quantisation allows for individual normalisation factors for each block, resulting in a more fine-grained overall quantisation.

Compared to full fine-tuning, LoRA adjusts only the low-rank subspace parameters of the model, thereby achieving higher computational efficiency and lower costs for computation and storage. In scenarios with limited speaker data, full fine-tuning methods may be prone to model overfitting, whereas LoRA can alleviate this issue.

For the pretrained ASR model with weight matrix $W_{0}\in\mathbb{R}^{d\times k}$, its update is expressed through the following equation, where $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$, with the rank $r\ll\min(d,k)$.

$$W_{0}+\Delta W=W_{0}+BA$$

(1)

In speaker adaptive training, only the LoRA parameters corresponding to each speaker are updated. In this way, effective adaptation to different speakers can be achieved by updating a minimal set of parameters. Moreover, in cases where the base model is quantised, full-precision LoRA serves to some extent to restore full-precision performance to the base model.

Although the target speaker data is always limited, in reality, the target domain data of other speakers is usually available. Therefore, PQM leverages those data to find a better initialisation point for LoRA weights before performing speaker adaptation, referred to as LoRA pretraining. This allows the LoRA parameters to adapt to the target-domain ASR task, providing a more robust starting point for subsequent speaker adaptation. When using LoRA, each speaker corresponds to one set of LoRA parameters. The number of parameters is given by $|\Theta|=2\times L_{\text{LoRA}}\times d_{\text{model}}\times r$, where $L_{\text{LoRA}}$ is the number of weight matrices to apply LoRA, $d_{\text{model}}$ is the attention layer dimension, and $r$ is the rank.

LibriSpeech is an English audiobook dataset. We selected 5 male speakers and 5 female speakers with the largest number of utterances from train-clean-360 as speaker adaptation data. Each speaker contributes approximately 150 utterances, resulting in a total speech duration of roughly 25 minutes. For LoRA pre-training, the train-clean-100 set was used which does not have any speaker overlap with the selected speakers.

TED-LIUM 3 (TL3) is a TED talks dataset. We selected 16 speakers from the test set as speaker adaptation data. On average, each speaker has 161 utterances (14 minutes).

Speaker adaptation data for LibriSpeech and TL3 was divided randomly, where 2/5 was divided into the train set, 1/5 was divided into the dev set, and 2/5 was divided into the test set. On average, each speaker has 6-10 minutes of training data, while the dev and test data remains constant across all experiments. We denote the partitioned test sets as LibriSpeech-SA and TL3-SA respectively in the results. Data partition details are provided¹.

In order to verify the effectiveness of PQM, we use Whisper and Conformer AED models as two widely used models as examples.

Whisper is a transformer-based AED model released by OpenAI trained on 680k hours of audio. The base.en model with a full model size of 278MB was used. The encoder has 6 Transformer blocks with 2048 hidden dimensions, and the output size is 512. The decoder has 6 Transformer blocks with 2048 hidden dimensions. The Transformer-related weight matrices are all 512 by 512 dimensional. Feature processing and model training followed [1, 32].

Conformer AED is a hybrid CTC/attention-based encoder-decoder model, whose FP32 model size is about 131MB. The training follows ESPnet [33] with 0.3 CTC weight and 80-dim FBank features. The Conformer encoder has 12 blocks with 1024 hidden dimensions. The decoder uses a 6-block transformer architecture with 2048-dim linear units. The Transformer-related weight matrices are all 256 by 256.

The baseline system is the quantised system without personalisation. When training LoRA from scratch (rank=4), the LoRA parameter sizes of Whisper and Conformer are 1.2MB and 0.8MB. When using pretrained LoRA (rank=1), the LoRA parameter sizes of Whisper and Conformer are 0.3MB and 0.2MB. Identical hyper-parameter settings were used for all speakers. Models are evaluated using WER averaged across all utterances from the test set speakers.