Improved Consistency Training for Semi-Supervised Sequence-to-Sequence ASR via Speech Chain Reconstruction and Self-Transcribing

S2S ASR models are designed to directly predict the conditional probability $P(\bm{\hat{y}}|\bm{x};\theta_{ASR})$ of a sequence of predicted tokens $\bm{\hat{y}}=[\hat{y}_{1},...,\hat{y}_{T}]$ given a sequence of speech features $\bm{x}=[x_{1},...,x_{S}]$. Here, $\theta_{ASR}$ represents the parameters of ASR, $S$ is the length of the input sequence, and $T$ is the length of the output sequence. Our ASR models are based on Listen-Attend-Spell (LAS) [1] which has an encoder-decoder architecture as shown in Fig.1(a).

The encoder converts the input sequence of speech features into a sequence of hidden representations $\bm{h}^{e}=[h^{e}_{1},...,h^{e}_{S}]$. The decoder receives $\bm{h}^{e}$ and outputs the token probability $p(\hat{y}_{t}|\hat{y}_{1:t-1},\bm{x};\theta_{ASR})$ at time step $t$ based on the prefix tokens $\hat{y}_{1:t-1}$. The probability of the generated transcript $P(\bm{\hat{y}}|\bm{x};\theta_{ASR})$ is calculated by the product of the token probability at each time step as

$$P(\bm{\hat{y}}|\bm{x};\theta_{ASR})=\prod_{t=1}^{T}{p(\hat{y}_{t}|\hat{y}_{1:t-1},\bm{x};\theta_{ASR})}.$$

(1)

During training, the prefix tokens $\hat{y}_{1:t-1}$ are replaced with the ground-truth labels. Given a speech-text pair $(\bm{x^{l}},\bm{y^{l}})$, a base ASR model is trained by the following supervised ASR loss:

$$\mathcal{L}^{sup}_{ASR}=-\frac{1}{T}\sum_{t=1}^{T}{\log{p(y^{l}_{t}|y^{l}_{1:t-1},\bm{x};\theta_{ASR})}}.$$

(2)

FixMatch algorithm [15] is a semi-supervised algorithm for image classification that combines consistency regularization with pseudo-labeling. In its existing application to S2S ASR [5], a teacher ASR model is initialized on the parameters of the base ASR model and used to produce the pseudo transcript $\bm{\tilde{y}}$ for the untranscribed speech $\bm{x^{u}}$ as

$$\bm{\tilde{y}}=\mathop{\arg\max}\limits_{\bm{z}}P(\bm{z}|\bm{x^{u}};\theta^{Tea}_{ASR}).$$

(3)

As shown in Fig.1(b), untranscribed speech $\bm{x^{u}}$ is separately perturbed by a weak augmentation function $\alpha(\cdot)$ and a strong augmentation function $\mathcal{A}(\cdot)$. The weakly-perturbed speech $\alpha(\bm{x^{u}})$ is fed into the student ASR model to obtain the pseudo label $\bar{y}_{t}$ at time step $t$ given $\tilde{y}_{1:t-1}$ as prefix tokens by

$$\displaystyle\bar{y}_{t}=\mathop{\arg\max}\limits_{z}p(z|\tilde{y}_{1:t-1},\alpha(\bm{x^{u}});\theta^{Stu}_{ASR}).$$

(4)

The consistency ASR loss $\mathcal{L}^{con}_{ASR}$ is calculated on the strongly-perturbed speech $\mathcal{A}(\bm{x^{u}})$ and pseudo label $\bar{y}_{t}$ given the same pseudo transcripts $\tilde{y}_{1:t-1}$ as prefix tokens by

$$\mathcal{L}^{con}_{ASR}=-\frac{1}{T}\sum_{t=1}^{T}\mathbb{1}(q_{t}>\tau)\log{p(\bar{y}_{t}|\tilde{y}_{1:t-1},\mathcal{A}(\bm{x^{u}});\theta^{Stu}_{ASR})},$$

(5)

where $q_{t}=p(\bar{y}_{t}|\tilde{y}_{1:t-1},\alpha(\bm{x^{u}});\theta^{Stu}_{ASR})$ is the confidence of the student model on $\bar{y}_{t}$ and $\tau$ is the confidence threshold.

Finally, the student model is trained by the following loss function:

$$\mathcal{L}_{ASR}=\mathcal{L}^{sup}_{ASR}+\lambda_{con}\mathcal{L}^{con}_{ASR}.$$

(6)

On top of the existing paradigm, we made two improvements. First, the weakly-perturbed speech $\alpha(\bm{x^{u}})$ is used to produce the pseudo transcripts $\bm{\tilde{y}}$. Second, the student model is initialized by the base ASR and dynamically produces the pseudo transcripts by itself during the consistency training. Our improvements can be formualted as follow:

$$\bm{\tilde{y}}=\mathop{\arg\max}\limits_{\bm{z}}P(\bm{z}|\alpha(\bm{x^{u}});\theta^{Stu}_{ASR}).$$

(7)

Sequence-to-sequence TTS models can be considered the reverse case of ASR that directly predicts the conditional probability $P(\bm{\hat{x}}|\bm{y},\theta_{TTS})$ of a sequence of speech features $\bm{\hat{x}}=[\hat{x}_{1},...,\hat{x}_{S}]$ given a sequence of tokens $\bm{y}=[y_{1},...,y_{T}]$. Our TTS model is based on Tacotron2 [21] which has a similar architecture to LAS. We provide the decoder of TTS with the speaker embedding $f(\bm{x})$ extracted from the input speech, which enables TTS to synthesize speech in a multi-speaker setting. Our TTS model is trained by the loss function $\mathcal{L}_{TTS}$:

	$\displaystyle\mathcal{L}_{TTS}=$	$\displaystyle\mathcal{L}^{sup}_{TTS}+\mathcal{L}^{unsup}_{TTS},$		(8)
	$\displaystyle\mathcal{L}^{sup}_{TTS}=$	$\displaystyle\frac{1}{S}\sum_{s=1}^{S}(x^{l}_{s}-\hat{x}^{l}_{s})^{2}$
		$\displaystyle-(b^{l}_{s}\log(\hat{b}^{l}_{s})+(1-b^{l}_{s})\log(1-\hat{b}^{l}_{s})),$		(9)
	$\displaystyle\mathcal{L}^{unsup}_{TTS}=$	$\displaystyle\frac{1}{S}\sum_{s=1}^{S}(x^{u}_{s}-\hat{x}^{u}_{s})^{2}$
		$\displaystyle-(b^{u}_{s}\log(\hat{b}^{u}_{s})+(1-b^{u}_{s})\log(1-\hat{b}^{u}_{s})),$		(10)

where $\hat{x}_{s},x_{s}$ are the predicted and ground-truth log Mel-scale spectrograms at time $s$ and $\hat{b}_{s},b_{s}$ are the predicted and ground-truth end-of-frame probabilities. The pseudo transcripts $\bm{\tilde{y}}$ used for training the TTS model is produced by the base ASR model before ASR consistency training as Eq.3 does.

Different from our previous work that generates synthetic speech from real texts for semi-supervised ASR training [19, 20], this work applied the speech chain reconstruction as a data augmentation method for the untranscribed speech by the teacher-forcing technique. This technique prevents TTS decoding from mispronunciation and early stopping, thus protecting the linguistic contents. Fig.2 shows the process of our proposed speech chain reconstruction that can be considered as a frame-level speech transformation.

Untranscribed speech $\bm{x^{u}}$ is utilized three times before generating reconstructed speech $\bm{\hat{x}^{u}}$. First, the base ASR model converts $\bm{x^{u}}$ into pseudo transcripts $\bm{\tilde{y}}$ as the input of our TTS encoder. Then, the speaker embedding $f(\bm{x^{u}})$ is extracted from $\bm{x^{u}}$ to provide our TTS decoder with the speaker information. Finally, our TTS decoder generates the reconstructed speech $\bm{\hat{x}^{u}}$ with original speech $\bm{x^{u}}$ as the prefix speech features at each time step. The reconstructed speech $\bm{\hat{x}^{u}}$ is treated as the weakly-perturbed speech $\alpha(\bm{x^{u}})$ during the subsequent ASR consistency training.

We conducted experiments in both single-speaker and multi-speaker settings. We use the LJSpeech corpus [22] for the single-speaker setting. We take 12,600 utterances as the training set, 250 utterances as the development set and 250 utterances as the test set. There is no overlap in the division of data. We treat the first 50% of the training set as labeled data and the last 50% as unlabeled data. In the multi-speaker setting, we use the LibriSpeech corpus [23]. We take the “train-clean-100” set as the training set, the “dev-clean” set as the development set and the “test-clean” set as the test set. We treat the first 75 speakers (8,570 utts) of the training set as labeled data and the last 176 speakers (19,968 utts) as unlabeled data.

As for acoustic features, we used 80-dimensional log Mel-spectrograms extracted at a 50-ms frame length and a 12.5-ms frame shift. The English letters in the transcripts of all utterances are normalized into their lowercase forms. All transcripts are mapped into a 29-token set: 26 (a-z) letters of the alphabet, apostrophes, space, and “sos/eos” ¹¹1We combined sos and eos into one token..

ESPNET2 [29, 30] was used to perform our experiments. Our supervised baselines were the base ASR models trained using only the labeled data. We designed four scenarios and conducted contrast experiments where the weak augmentation $\alpha(\cdot)$ was either SpecAugment or speech chain reconstruction. For all experiments, the strong augmentation $\mathcal{A}(\cdot)$ was implemented by SpecAugment. Our weak SpecAugment applied one time of time-frequency masking while the strong SpecAugment applied two times. Maximal frequency masking width was 5 bins for $\alpha(\cdot)$ and 20 bins for $\mathcal{A}(\cdot)$. For the LJSpeech corpus, maximal time masking width was 10 frames for $\alpha(\cdot)$ and 50 frames for $\mathcal{A}(\cdot)$. For LibriSpeech corpus, maximal time masking width was 20 frames for $\alpha(\cdot)$ and 100 frames for $\mathcal{A}(\cdot)$.

In the single-speaker setting, the ratio of labeled data to unlabeled data was set to 1:1. From Tab.1, we observed that speech chain reconstruction outperforms the weak SpecAugment in all scenarios. Our consistency training paradigm achieved the best CER performance of 7.2%, which has a 12.2% CER improvement on the supervised baseline. This showcases how speech chain reconstruction keeps more linguistic information than SpecAugment, and thus it is more suitable to be the weak augmentation.

In the single-speaker setting, dynamic pseudo transcripts produced by the student model achieved a 2.7% CER improvement on the existing paradigm. This indicates that the student model gradually corrects the errors in the pretrained base model during consistency training. It can be seen that the pseudo transcripts produced by the weakly-perturbed input speech $\alpha(\bm{x^{u}})$ also resulted in a 2.7% CER improvement. It supports our idea that prefix tokens should match the input speech during ASR consistency training. As for the reason why the relative improvements on the existing paradigm were not large in scale, we hypothesized that the base model is already good enough to produce understandable transcripts because 50% of the training set was used as the labeled data.

In the multi-speaker setting, we simulated a harsher condition where the ratio of labeled data to unlabeled data is 3:7. From Tab.1, speech chain reconstruction still outperformed the weak SpecAugment in all scenarios. With more unlabeled data, the improvement of our paradigm over the supervised baseline surged to 38.6%, which indicates that our paradigm benefits from a large amount of untranscribed speech.

On top of the existing paradigm, a 5.5% CER improvement was achieved when the input speech used to produce the pseudo transcripts was changed from the original speech $\bm{x^{u}}$ to weak-perturbed speech $\alpha(\bm{x^{u}})$. On the other hand, only a 0.5% CER improvement was observed when we set the teacher model in the existing paradigm to the student model itself during consistency training. This indicates that the mismatch between the input speech and pseudo transcripts has a stronger influence on our ASR models than the errors in the pretrained base model.

According to the right part of Tab.1, our ASR models performed better when we set the confidence threshold to a smaller value. Since we only used the first 30% of “train-clean-100” as the labeled data to train the base model, our student ASR models are not very confident on the untranscribed speech and thus output relatively lower token probability at each time step. With a higher confidence threshold, only a small fraction of untranscribed speech is utilized to calculate the final ASR loss, hence seriously restricting the potential of consistency training for improving ASR performance.