On the Interplay between Sparsity, Naturalness,
Intelligibility, and Prosody in Speech Synthesis

Consider a sequence-to-sequence learning problem, where $\bm{X}$ and $\bm{Y}$ represent the input and output sequences respectively. For ASR, $\bm{X}$ is waveforms and $\bm{Y}$ is character/phone sequences; for a TTS acoustic model, $\bm{X}$ is character/phone sequences and $\bm{Y}$ is spectrogram sequences; for a vocoder, $\bm{X}$ is spectrogram sequences and $\bm{Y}$ is waveforms. A mapping function $f(\bm{X};\theta)$ parametrized by a neural network is learned, where $\theta\in\mathcal{R}^{d}$ represents the network parameters and $d$ represents the number of parameters. Sequence-level log-likelihood $\mathop{\mathbb{E}}\big{[}\ln P(\bm{Y}\mid\bm{X};\theta)]$ on target dataset $\mathcal{D}$ is maximized.

Our goal is to find a subnetwork $m\odot\theta$, where $\odot$ is the element-wise product and a binary pruning mask $m\in\{0,1\}^{d}$ is applied on the model weights $\theta$. The ideal pruning method would learn $m$ at target sparsity such that $f(\bm{X};m\odot\theta)$ achieves similar loss as $f(\bm{X};\theta)$ after training on $\mathcal{D}$.

${\tt UMP}$ is based on PyTorch’s API⁷⁷7PyTorch Pruning API. For all models, $\theta_{0}$ is set to pretrained checkpoints on LJspeech, and $N$ is set to 1 epoch of model updates. We jointly prune encoder, decoder, pre-nets, and post-nets for the acoustic model; for vocoder, since only $G$ is needed during test-time synthesis, only $G$ is pruned ($D$ is still trainable).

We examine the following three aspects of the synthetic speech:

• •

  Naturalness is quantified by the 5-point (1-point increment) scale Mean Opinion Score (MOS). 20 unique utterances (with 5 repetitions) are synthesized and compared across pruned models, for a total of 100 HITs (crowdsourced tasks) per MOS test. In each HIT, the input texts to all models are the same to minimize variability.

• •

  Intelligibility is measured with Google’s ASR API⁸⁸8https://pypi.org/project/SpeechRecognition/.

• •

  Prosody via mean and standard deviation (std) fundamental frequency ($F_{0}$) estimations⁹⁹9$F_{0}$ estimation with probabilistic YIN (pYIN) implemented in Librosa. and utterance duration, averaged over dev and eval utterances.

Fig 2 is the box plot of MOS scores of pruned end-to-end TTS models at 0%$\sim$99% sparsities with ${\tt PARP}$. In each set of experiments, only one of the acoustic model or vocoder is pruned, while the other is kept intact. For either pruned Transformer-TTS or Tacotron2 acoustic models, their MOS scores are statistically not different from the unpruned ones at up to 90% sparsity. For pruned Parallel WaveGAN, pairing it with an unpruned Transformer-TTS reaches up to 88% sparsity without any statistical MOS decrease, and up to 85% if paired with an unpruned Tacotron2. Based on these results, we first conclude that end-to-end TTS models are over-parameterized across model architectures, and removing the majority of their weights does not significantly affect naturalness.

Secondly, we observe that the 30% pruned Tacotron2 has a statistically higher MOS score than unpruned Tacotron2. Although this phenomenon is not seen in Transformer-TTS, WaveGAN, or at other sparsities, it is nonetheless surprising given ${\tt PARP}$’s simplicity. We can hypothesize that under the right conditions, pruned models train better, which results in higher naturalness over unpruned models.

We measure intelligibility of synthetic speech via Google ASR, and Figure 3 plots synthetic speech’s WERs across sparsities over model and pruning configurations. Focusing on the top plot, we have the following two high-level impressions: (1) WER decreases at initial sparsities and increases dramatically at around 85% sparsity with ${\tt PARP}$ (yellow and purple dotted lines). (2) pruning the vocoder does not change the WERs at all (observe the straight red dotted line).

Specifically, for Transformer-TTS, ${\tt PARP}$ at 75% and ${\tt PARP}$-${\tt P}$ at 90% sparsities have lower WERs (higher intelligibility) than its unpruned version. For Tacotron2, there is no WER reduction and its WERs remain at $\sim$9% at up to 40% sparsity (no change in intelligibility). Based on (2) and Section 4.1, we can further conclude that the CNN-based vocoder is highly prunable, with little to no naturalness and intelligibility degradation at up to almost 90% sparsity.

We used synthetic speech’s utterance duration and mean/std $F_{0}$ across time as three rough proxies for prosody. Fig 4 plots the prosody mismatch between pruned models and ground truth recordings across model combinations. Observe ${\tt PARP}$ on Tacotron2 and on Transformer-TTS result in visible differences in prosody changes over sparsities. In the top plot, pruned Transformer-TTS (yellow dotted line) have the same utterance duration (+0.2 seconds over ground truth) at 10%$\sim$75% sparsities, while in the same region, pruned Tacotron2 (purple dotted line) results in a linear decrease in duration (-0.2$\sim$-0.8 seconds). Indeed, we confirmed by listening to synthesis samples that pruning Tacotron2 leads to shorter utterance duration as sparsity increases.

In the middle plot and up to 80% sparsity, pruned Tacotron2 models have a much large $F_{0}$ mean variation (-20$\sim$-7.5 Hz) compared to that of Transformer-TTS (-10$\sim$-15 Hz). We hypothesize that ${\tt PARP}$ on RNN-based models leads to unstable gradients through time during training, while Transformer-based models are easier to prune. Further, ${\tt PARP}$ on WaveGAN (red dotted line) has a minimal effect on both metrics across sparsities, which leads us to another hypothesis that vocoder is not responsible for prosody generation.

In the bottom plot and up to 80% sparsity, pruned models all have minimal $F_{0}$ std variations ($\leq 2$ Hz) compared to 53Hz ground truth $F_{0}$ std. We infer that at reasonable sparsities, pruning does not hurt prosodic expressivity, due to lack of $F_{0}$ oversmoothing [1, 37].

In  [30], the authors attain pruned wav2vec 2.0 at much higher sparsity without WER increase given sufficient finetuning data (10h Librispeech split). Therefore, one question we had was, how much finetuning data is “good enough” for pruning end-to-end TTS? We did two sets of experiments, and for each, we modify the amount of data in ${\tt PARP}$’s Step 2, while keeping $\theta_{0}$ as is (trained on full LJspeech).

The first set of experiments result is Fig 5. Even at as high as 90% sparsity, 30% of finetuning data ($\sim$7.2h) is enough for ${\tt PARP}$ to reach the same level of naturalness as full data¹⁰¹⁰10The effect of using less data to obtain $\theta_{0}$ remains unclear.. The other set of experiment is TTS-Augmentation for utilizing additional unspoken text ($\sim$100h, no domain mismatch) for ${\tt PARP}$’s Step 2. In Fig 3’s bottom plot, we see TTS-Augmentations (dark & light green lines) bear minimal effect on the synthetic speech WERs. However, Table 1 indicates that TTS-Augmentation ${\tt PARP}$+${\tt seq}$-${\tt aug}$ does statistically improve ${\tt PARP}$ in naturalness and intelligibility subjective testings.

We clarify here again the philosophy of this piece of work – we are interested in quantifying the effects of sparsity on speech perceptions (measured via naturalness, intelligibility, and prosody), as there has not been any systematic study in this direction. The pruning methods, either ${\tt PARP}$ or ${\tt IMP}$, are merely engineering “tools” to probe the underlying phenomenon. It is not too difficult to get higher sparsities or introduce fancier pruning techniques as noted in this study, yet they are less scientifically interesting at this moment.

Normally, ${\tt IMP}$s are iterated many times depending on the training costs. In iteration $i$, the ${\tt IMP}$ mask from the previous iteration $m_{i-1}$ is fixed, and only the remaining weights are considered in ${\tt UMP}$. In other words, each iteration’s sparsity pattern is built on top of the that from the previous iteration. For simplicity, our ${\tt IMP}$ implementation only has 1 iteration. In some literature, this is termed One-Shot Magnitude Pruning ${\tt OMP}$, though it is also not exactly our case due to our starting point is a fully trained model $\theta_{D}$ instead of a randomly initialized model.

We experimented with two TTS-Augmentation policies for ${\tt PARP}$ with initial model $f(\theta_{D})$ trained on LJspeech $\mathcal{D}$ and additional unspoken text from Librispeech $\bm{X}_{u}$:

• •

  ${\tt PARP}$+${\tt seq}$-${\tt aug}$: synthesize $\bm{X}_{u}$ with $f(\theta_{D})$ to obtain $\mathcal{D}_{u}=(\bm{X}_{u},f(\bm{X}_{u};\theta_{D}))$. First, perform ${\tt PARP}$’s Step 2 at target sparsity on $\mathcal{D}_{u}$ with initial model weight $\theta_{D}$. Obtain a pruned model with model weight $m_{U}\odot\theta_{U}^{*}$. Next, perform ${\tt PARP}$’s Step 2 at target sparsity on $\mathcal{D}$ with initial model weight $m_{U}\odot\theta_{U}^{*}$. Return $m_{D}\odot\theta_{D}^{*}$.

• •

  ${\tt PARP}$+${\tt mix}$-${\tt aug}$: synthesize $\bm{X}_{u}$ with $f(\theta_{D})$ to obtain $\mathcal{D}_{u}=(\bm{X}_{u},f(\bm{X}_{u};\theta_{D}))$. Directly perform ${\tt PARP}$’s Step 2 on $(\mathcal{D},\mathcal{D}_{u})$ with initial model weight $\theta_{D}$. Return $m_{D}\odot\theta_{D}^{*}$.

Table 2 is the subjective A/B testing results comparing the two augmentation policies at different sparsities. Although there is no statistical preference between the two, we note that pruned models with different augmentation policies do subjectively sound different from one another.

Below are the results of running Mann-Whitney U test on our Naturalness MOS tests at a significance level $p\leq 0.05$, where $\bullet$ indicates pair-wise statistically significant, and $\square$ indicates the opposite. The order in $x$ and $y$ axes are shuffled due to our MOS test implementation. Cross reference with Figure 2, we have the following conclusions:

• •

  ${\tt PARP}$ on Transformer-TTS + unpruned Parallel WaveGAN (Table 3): no pruned model is statistically better than the unpruned model, and pruned model is only statistically worse starting at 90% sparsity.

• •

  ${\tt PARP}$ on Tacotron2 + unpruned Parallel WaveGAN (Table 4): 30% pruned model is statistically better than the unpruned model, and pruned model is only statistically worse starting at 90% sparsity.

• •

  Unpruned Transformer-TTS + ${\tt PARP}$ on Parallel WaveGAN (Table 5): no pruned model is statistically better than the unpruned model, and no pruned model model is statistically worse either.

• •

  Unpruned Tacotron2 + ${\tt PARP}$ on Parallel WaveGAN (Table 6): no pruned model is statistically better than the unpruned model, and pruned model is only statistically worse starting at 85% sparsity.

Based on the above statistical testings, we are not claiming that pruning definitely improves TTS training. Instead, at the right conditions, pruned models could perform better than the unpruned ones.

We append our sample AMT user interfaces used in our study for naturalness MOS testing (Figure 2 and Figure 5) and naturalness & intelligibility A/B testing (Table 1 and Table 2). For MOS tests, reference natural recordings are included in the instruction. About $2800 are spent on AMT testing.