Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation

Model architecture: Wav2vec2.0 model consists of three components, including 1) a multi-layer CNN-based feature encoder which encodes raw speech $\mathcal{X}$ into continuous speech representations $z_{t}\in\mathcal{Z}$, 2) a transformer-based context network producing context representations $c_{t}\in C$ over the entire sequence of randomly masked feature encoder outputs and 3) a quantization module generating discrete speech units $q_{t}\in\mathcal{Q}$ as labels for self-supervised training.

The HuBERT pre-training process alternates between two steps: 1) a clustering step to create pseudo-labels and 2) a prediction step where the model produces labels at masked positions.

Motivated by [35], an additional speech impairment severity prediction task $\mathcal{L}_{\text{Seve}}$ is introduced to prevent overfitting to the limited amount of labeled UASpeech data during Wav2vec2.0 and HuBERT fine-tuning stage. The total loss function of MTL is defined as

where $\beta_{1}=\beta_{2}=0.5$ are empirically set hyper-parameters. Following [35], the CNN-based feature encoder is frozen when fine-tuning both Wav2vec2.0 and HuBERT models on UASpeech data.

Speed perturbation resamples a given audio signal $x(t)$ in time domain [9]. Given a perturbation factor $\alpha$, output audio segment is produced by applying $\alpha$ along the time axis as $y(t)=x(\alpha t)$. This equation is equivalent to the following change in frequency domain as $X(f)\rightarrow\frac{1}{\alpha}X(\frac{1}{\alpha}f)$, where $X$ and $\frac{1}{\alpha}X(\frac{1}{\alpha}f)$ stand for the Fourier transform of $x(t)$ and $y(t)$ respectively. Speed perturbation alters both audio duration and spectral envelope of the input audio. A fixed set of speaker-independent (SI) perturbation factors $\{0.9,1.0,1.1\}$ is applied to disordered speech in common with the data perturbation methods widely used in normal ASR tasks [9].

Due to the large mismatch in speaking rate between normal and disordered speakers, and the large variability among impaired speakers, speaker-dependent (SD) perturbation factors were applied when modifying normal speech to disordered speech. These SD perturbation factors were obtained using phonetic analysis described in [36]. For each control speaker $C_{i}$ and dysarthric speaker $D_{j}$, we performed force alignment and calculated the average phoneme duration $l_{C_{i}}$ and $l_{D_{j}}$. The average phoneme duration of all control speakers $l_{\overline{C}}$ is taken as the reference to compute the speaker-dependent perturbation factor $F_{D_{j}}$ as $\frac{l_{\overline{C}}}{l_{D_{j}}}$, which was then used to modify normal speech to resemble that of dysarthric speaker $D_{j}$.

The SD speed perturbation of Sec. 3.2 only simulates the overall slower speaking rate and “disordered like” spectral properties of dysarthric speech. To capture more fine-grained spectro-temporal features of impaired speech, GAN-based data augmentation approaches can be used. For the UASpeech dysarthric speech corpus that is based on parallel normal-impaired speech recordings, speaker-dependent deep convolutional GANs (DCGANs) are utilized to further transform SD speed perturbed healthy speakers’ data to that of individual target impaired speakers of identical contents.

The architecture of DCGANs follows [4]. The Generator comprises 4 convolutional layers, each followed by a ReLU activation. The first three layers are of 8 kernels while the last one has 1 kernel. All of the kernels in the Generator have a kernel size of $3\times 3$ and stride of $1\times 1$. Replicate Padding is applied to ensure consistency in dimensions of the input and output of each convolutional layer. The Discriminator consists of 4 convolutional layers containing $8$, $16$, $32$, and $64$ kernels respectively. All kernels have kernel size and stride of $2\times 2$. A linear layer followed by a Sigmoid function is appended for binary classification. The output of the last convolutional layer is padded zero and flattened as the input of the linear layer.

Prior to DCGAN training, pairs of normal and dysarthric speech utterances of identical word contents are formed. In order to facilitate a frame-by-frame comparison between the GAN-transformed normal speech spectrogram against the target impaired spectrogram, each normal utterance is speed perturbed to temporally match the paired impaired utterance as shown in Fig. 1a. This requires a scaling factor to be estimated for each normal and dysarthric speech segment pair using phonetic analysis similar to the procedure described in [36]. The resulting pairs of normal and dysarthric speech utterances now have the same duration before being used in DCGAN training. The DCGAN training maximizes the binary classification accuracy on the target dysarthric speech, and minimizes that obtained on the GAN-transformed normal speech. This is given by

where $j$ is the index for target dysarthric speaker, $G_{j}$ and $D_{j}$ are Generator and Discriminator for dysarthric speaker $j$, $\textbf{f}_{\textbf{C}}$ and $\textbf{f}_{\textbf{D}_{\textbf{j}}}$ are the STFT features of paired control and dysarthric utterances.

In contrast to the DCGAN approach of Sec. 4.1 designed for parallel data, more general and flexible adversarial data augmentation based on Spectral basis GANs [32] that do not require parallel normal-impaired speech is utilized. Singular Value Decomposition (SVD) decomposed spectral bases [37] are used as the inputs instead of STFT features. The convolutional layers in the DCGAN model are also replaced by fully connected layers.

The detailed architecture of Spectral basis GANs follows [32]. The Generator consists of three linear layers where the first two are followed by a Leaky ReLU with 0.2 negative slope and the last one is connected to a Tanh activation. The Discriminator contains three linear layers of $1024$, $512$ and $256$ dimensions each before the Sigmoid layer for binary classification. The output targets of the Generator serve as a perturbation vector, $\Delta\mathbf{U}$, to be added to the spectral bases that are derived from the input normal speech spectrogram, $\mathbf{U}$.

The UASpeech corpus [28] is the largest publicly available dataset containing 103h speech from 16 dysarthric speakers and 13 healthy control speakers. The speech data of every speaker is split into three blocks B1, B2 and B3, each containing the same 155 common words and a different set of 100 uncommon words. Data from B2 of all 16 dysarthric speakers is treated as the test set, while B1 and B3 data of all speakers are treated as the training set. With force alignment and silence stripping performed via an HTK [40] trained GMM-HMM system, the dataset produces a 30.6h training set (99195 utt.) and an 8.7h test set (26520 utt.). Data augmentation using speaker-independent speed perturbation (Sec. 3.1) produces a 72h training set, while speaker-dependent speed perturbation could further expand the training set to 130h. B2 data of the 13 control speakers are further folded in, producing a 37h unaugmented (122392 utt.) and a 173h augmented training set (538292 utt.).

The hybrid LF-MMI factored TDNN systems follow the Kaldi chain system setup, except i-Vector features excluded. The graphemic end-to-end (E2E) Conformer systems implemented via ESPnet [41] take 40-dim filter-bank + $\Delta$ features as inputs. The SSL Wav2vec2.0 and HuBERT systems pre-trained with 60k hours of Libri-light data and initially fine-tuned on 960h LibriSpeech data, before further cross-domain fine-tuning on UASpeech data, with or without speed perturbation or GAN-based data augmentation.