Prosodic Alignment for off-screen automatic dubbing

Two-step optimization procedure: For the above reasons, in [9], we introduce a two-step optimization procedure. In Step 1, we optimize the weights of the following log-linear model by maximizing segmentation accuracy over a manually annotated data set:

The feature functions $s_{a}$ of the model (notice that we dropped some of the dependencies in eq. (2) for readability) denote – (1) the language model score of target break point, (2) the cross-lingual semantic match score across source and target segments, (3) the speaking rate variation across target segments and (4) the speaking rate match across source and target segments respectively.

In Step 2, starting from given breakpoints $\mathbf{\hat{j}}$, we optimize the relaxations $\delta_{l}$ and $\delta_{r}$ for the $t$-th segment using another recurrent equation, that can also be solved via dynamic programming, derived from the log-linear model:

which includes as additional feature $s_{5}$ the isochrony score [9]. Weight $w_{5}$ is optimized by maximizing speech smoothness [9] over the training set, assuming the reference breakpoints $\mathbf{\hat{j}}$ are given. Speech smoothness measures speaking rate variations across contiguous segments. Speaking rate computations rely on the strings $\tilde{f}_{t}$ and $\tilde{e}_{t}$, denoting the $t$-th source and target segments, as well as the original interval $s_{t}$ and the relaxed interval $s^{*}_{t}$. Hence, the speaking rate of a source (target) segment is computed by taking the ratio between the duration of the utterance by source (target) TTS run at normal speed and the source (target) interval length, ³³3We run TTS on the entire sentence, force-align audio with text [23, 24] and compute segment duration from the time-stamps of the words., i.e:

In Step 2 we also apply a more relaxed policy in allocating relaxations $\delta_{l}$ and $\delta_{r}$ inside off-screen sentences. In particular, we allow using the entire available inter-phrase and inter-sentence intervals rather than limiting them to maximum $\Delta\epsilon$. Isochrone dubbing utilizes relaxation mechanism locally inside each sentence since for on-screen sentences, we tradeoff isochrony for improved speaking rate control. In contrast, for off-screen sentences we do not need isochrony and hence we utilize a global relaxation mechanism by computing optimal relaxations across all off-screen sentences using dynamic programming.

Regarding the scoring function (7), when the target speaking rate $r_{f}(t)$ is below $1$, it returns a maximum score since for off-screen phrases we can contract time boundaries by setting the speaking rate $r_{f}(t)$ to $1$ without consequences. When the speaking rate of a target phrase $r_{f}(t)$ is larger than $2$, it returns the lowest score since too high speaking rates will result in unintelligible TTS speech. For $1\leq r_{f}(t)\leq 2$, it returns scores that increasingly penalize values larger than 1, as they will correspond to less and less intelligible TTS speech.

Table 1 shows the results for automatic evaluation. We observe that ON/OFF outperforms ISO on MUST-C $D_{1}$, with respect to Sm(oothness) and F(luency) with relative improvements ranging from 9.9%-28.3% and 6.1%-17.1% respectively, while for In(telligibility) ON/OFF provides 0.6%-14.5% improvements for it, de, es. Similar improvements are obtained for ON/OFF against ISO for MuST-C on $D_{2}$ and YouTube on $D_{1}$ and $D_{2}$. We note that improvements for $D_{1}$ are higher compared to $D_{2}$, primarily because $D_{1}$ considers videos in which all sentences are off-screen. This provides more opportunities for ON/OFF to exploit the global relaxation mechanism.

Compared to MuST-C, we find that YouTube data obtains higher In scores across all target languages and datasets. To investigate this further, we computed the length compliance (LC) metric of [26] at the phrasal level that measures the percentage of translations whose length in character is within $\pm 10\%$ of the length of the source and hence more suitable for auto dubbing. We found that on average, LC for YouTube was 19.8% higher than that for MuST-C. Higher value of LC implies that we are able to better fit the translations in the available phrase intervals thereby having TTS speech more intelligible.

In this section, we present results of human evaluation on the test set. For each dubbing direction and dataset we report results on 15 video clips extracted separately for each evaluation using the criterion noted in Sec. 5. We asked 20 native speakers to rate the subjective experience for viewing each dubbed video from each dubbing condition on a scale of 0-10. To reduce cognitive load, we compare two dubbing conditions for each evaluation and collect a total of 600 scores. For all dubbing conditions we utilize post-edited translations to focus the subjects on the synchronization aspect of dubbing.

Finally, for each evaluation we compare two conditions in a head-to-head manner and report Wins (percentage of times one condition is preferred over the other) and Score (average subjective score of dubbed videos) metrics. To measure the impact of PA model on human score, we use a linear-mixed-effects model (LMEM) ⁶⁶6We used the lme4 package for R [27] by defining subjects and clips as random effects [28].

The results are summarized in Table 2. For the dubbing evaluations, we compare ON/OFF vs ISO on test sets $D_{1}$ and $D_{2}$ in two separate evaluations. ON/OFF clearly outperforms ISO on $D_{1}$ providing relative improvements in Wins on both MuST-C ranging from 107.7%-139.1% and YouTube ranging from 110%-297.1% with all results being statistically significant ($p<0.01$). Similarly, ON/OFF outperforms ISO on $D_{2}$. Finally for Score, we obtain similar relative improvements on both MuST-C (1.7%-28.6%) and YouTube (2.8%-22.5%) with all improvements except the ones for es on $D_{2}$ being statistically significant ($p<0.01$).

Relation between automatic and human scores: To explain the observed score variations using automatic metrics, we utilize LMEMs by aggregating evaluation data across all four languages. We define automatic metrics as fixed effects and subjects, clips, PA models and target languages as random effects. Our analysis reveals that Sm is the most impactful metric which is always statistically significant.

To further test if the learned LMEM can help predict human score, for every example we compute the average human score and compare it with the predicted score. We average out the random effect of subjects since different subjects use different range to grade their viewing experience. We train LMEM models, one on each dataset and domain, for a total of 4 models and predict scores on all 4 datasets. Table 3 shows that in most cases we obtain a high pearson correlation coefficient between the predicted and average human score. Additionally, each model is able to correctly predict on average the winning system (ISO or On/Off) for all datasets. In some cases, due to the low correlation, the magnitude of difference in predicted scores between the systems is not consistent with actual score differences. However, the sign of the difference is consistent and hence we can predict the correct winning system even in these cases. Thus we conclude that automatic metrics can also help compare dubbing quality of two systems.