Extracting lip feature. Let $D_{w}$, $D_{h}$ and $D_{c}$ be the width, height and number of channels of the video frames, respectively. We first extract lip regions $\mathbf{x_{m}}\in\mathbb{R}^{{T_{v}}\times{D_{w}}\times{D_{h}}\times{D_{c}}}$ from the given video using mouth region pre-processing from [20, 33, 36, 35, 34]. Then we exploit the LipEncoder to obtain the lip movement representation:

where $T_{v}$ denotes the number of video frames, and $D_{m}$ is the hidden dimension of the dynamic lip feature. The LipEncoder consists of several feed-forward transformer blocks that are suitable for capturing both long-term and short-term dynamics lip movement features.

Aligning text with lips. Inspired by the success of attention mechanism for cross-modality alignment [45, 19, 26, 14, 64, 44, 10, 27], we adopt multi-head attention to learn the alignment between the text phoneme and the lip movement sequence. We use lip embedding as a query to compute the attention on text phonemes. The larger the attention, the more related between a lip embedding and a text phoneme. Due to variations of mouth shapes and pronunciations, the multi-head attention mechanism is suitable for learning their alignments from different aspects. The text-video context sequence $\mathbf{E_{{lip,txt}}}$ = $[\alpha_{{lip,txt}}^{1},...,\alpha_{{lip,txt}}^{n}]\in\mathbb{R}^{{T_{v}}\times{D_{m}}}$ is a concatenation of outputs of $n$ attention heads. Concretely, the $k$-th head’s output $\alpha_{{lip,txt}}^{k}$ is obtained by:

where $\mathbf{W}{{}_{j}^{*}}$ is a learnable parameter matrix, $d_{k}$ is the embeeding dimension of $\alpha_{{lip,txt}}^{k}$, and $\mathbf{M_{t}}$ is a mask matrix indicating whether a token can be attended. The aligned representation $E_{lip,txt}$ is later expanded to the length of the desired mel-spectrogram.

Expanding to the desired length. According to the findings in [20], in an audio-visual clip, the length of a mel-spectrograms sequence is $n$ times that of a video frame sequence because they are temporally synchronized in audio and visual modalities. The number $n$ is computed as:

where $FPS$ denotes the Frames per Second of the video, $sr$ denotes the sampling rate of the audio, and $hs$ denotes hop size when transforming the raw waveform into mel-spectrograms. Phoneme-lip feature $E_{lip,txt}$ is simply duplicated $n$ times in [20] as the final phoneme-lip representation, which lacks flexibility. Instead, we propose to use transposed convolutions to learn the expansion of $E_{lip,txt}$, which can be formulated as:

where $T_{y}$ denotes the length of the desired mel-spectrogram, $D_{m}$ is the dimension of the initial mel-spectrogram. The parameters (stride and kernel size) of the transposed convolution can be set under the guidance of $n$ so that $T_{y}\approx n\times T_{v}$.

Valence and Arousal Feature. To accurately capture the valence and arousal information from facial expressions, we utilize an emotion face-alignment network (EmoFAN) [51] to encode the facial region into valence V and arousal A. $\begin{aligned} \textbf{V},\textbf{A}=\mathrm{EmoFAN}(x_{f})\in\mathbb{R}^{{T_{v}}\times{D_{m}}},\end{aligned}$ $x_{f}\in\mathbb{R}^{{T_{v}}\times{D_{w}}\times{D_{h}}\times{D_{c}}}$ is the face region extracted via $S^{3}FD$ face detection [60]. The EmoFAN focuses on facial regions relevant to emotion estimation, which utilizes a face alignment network (FAN) for facial point detection to ensure robustness by jointly predicting categorical and continuous emotions.

Bridging Arousal with Energy. Arousal is the physiological and psychological state of being awoken or of sense organs stimulated to the point of perception. To bridge the vocal energy with arousal display, we compute an arousal context vector $A_{i}^{l}$ for frame-level energy of the desired speech from phoneme-lip representation $\mathbf{M}_{pho,lip}$:

where $i$ is frame index, $A_{i}$ is the $i$-th row of $\mathbf{A}$, $\xi_{i,k}$ is the attention weight on the $k$-th phoneme-lip feature $M_{pho,lip}^{k}$ (the $k$-th row of $\mathbf{M}_{pho,lip}$) regarding $i$-th arousal display; $\mathbf{w^{T}_{a}}$, $\mathbf{W^{T}_{a}}$, $\mathbf{U^{T}_{a}}$ and $\mathbf{b_{a}}$ are learnable parameters; $T_{y}$ is the length of the desired mel-spectrogram.

Bridging Valence with Pitch. In prosody linguistics, speakers can speak with a wide pitch range (this is usually associated with excitement) while, at other times, with a narrow range. Valence is the affective quality referring to the intrinsic positiveness or negativeness of an event or situation. Similar to arousal to energy, we first compute a valence context vector for the frame-level pitch of the desired speech from phoneme-lip representation $\mathbf{M}_{pho,lip}$:

where $\mathbf{w^{T}_{g}}$, $\mathbf{W^{T}_{g}}$, $\mathbf{U^{T}_{g}}$ and $\mathbf{b_{g}}$ are learnable parameters. Then, we exploit a pitch predictor to convert the valence-related phoneme-lip embedding ${V}_{i}^{l}$ to the pitch of speech, which can be formulated as:

V2C is a multi-speaker dataset for animation movie dubbing with identity and emotion annotations [9]. It is collected from 26 Disney cartoon movies and covers 153 diverse characters. V2C not only needs to generate voices with identity characteristics according to the reference audio but also capture emotional information based on the reference movie clips. The whole dataset has 10,217 video clips with paired audio and subtitles. The training/validation/test size are 60%, 10%, 30%.

Chem is a single-speaker dataset composed of 6,640 short video clips collected from the YouTube, with the total video length of approximately nine hours [20]. The Chem dataset is originally used for the unconstrained single-speaker lip-to-speech synthesis [42], which takes place in a chemistry lecture. For fluency and complete dubbing, each video clip has sentence-level text and audio based on the start and end timestamps. There are 6,240, 200, and 200 dubbing clips for training, validation, and testing, respectively.

Audio-visual synchronization. To evaluate the synchronization between the generated speech and the video quantitatively, we adopt Lip Sync Error Distance (LSE-D) and Lip Sync Error Confidence (LSE-C) as our metrics, which can explicitly test for synchronization between lip motions and speech in unconstrained videos in the wild [43, 12].

Mel Cepstral Distortion and its variants. MCD [24], MCD-DTW [4] and MCD-DTW-SL [9] are adopted, which reflect the similarity of mel-spectrograms. MCD-DTW uses the Dynamic Time Warping (DTW) [37] algorithm to find the minimum MCD between two speeches, while MCD-DTW-SL introduces the duration measure coefficient to consider the length and the quality of generated speech [9].

Emotion and identity accuracy. To measure whether the generated speech carries proper emotion and speaker identity, we adopt an emotion accuracy (Emo. Acc.) and an identity accuracy (Id. Acc.) as our metrics as in [9].

Subjective evaluations. To further evaluate the quality of generated speech, we conduct a human study using a subjective evaluation metric, following the settings in [9]. Specifically, we adopt the MOS-naturalness (MOS-N) and MOS-similarity (MOS-S) to assess the naturalness of the generated speech and the recognization of the desired voice.

Results on the V2C benchmark. The results are presented in Table 1. Our method achieves the best performance on all nine metrics. Specifically, in terms of audio-visual sync, our method achieves 8.036 of LSE-D and 5.608 of LSE-C, which significantly surpasses the previous best results and is much closer to human performance. In terms of MCD, MCD-DTW, and MCD-DTW-SL, our method achieves relative 24.02%, 13.63% and 29.61% improvements, respectively. This indicates our method can achieve a better mel-spectrogram than others. The above results together show that bridging specific attributes of speech with corresponding visual counterparts can make the generated speech present better prosodies and lip motion sync. Additionally, our method outperforms the previous method by a large margin in emotion accuracy and can gain better identification accuracy. This indicates the proposed method can better capture and convey emotions, which is of great importance for the movie dubbing task. Last, the human subjective evaluation results (see MOS-N and MOS-S) also show that our method can generate speeches that are closer to realistic speech according to naturalness and similarity.

Results on the Chem benchmark. As shown in Table 2, our model is ahead of the state-of-the-art methods in all metrics on the Chem benchmark. In terms of the audio-visual sync, our method achieves 6.975 LSE-D and 7.176 LSE-C. Furthermore, in the subjective evaluations, our method improves 10.34% on AQ and 22.15% on AV Sync. The results show that our performance is much closer to the ground truth recording, which indicates that our model synthesizes high-quality natural speech by controlling the prosody from hierarchical visual representation.

Effectiveness of Duration Aligner, Prosody Adaptor, and Atmosphere Booster. We evaluate the effectiveness of these three modules by removing them separately and retraining the model. The results are shown in Row 1$\sim$3 of Table 3. The result shows that all the proposed modules contribute significantly to the overall performance, and each module has a different focus. Specifically, the performance on the audio-visual metrics (LSE-D, LSE-C, and MCD) drops the most when removing the Duration Aligner (DA). This reflects the DA module indeed helps the model learn a better temporal synchronization. By contrast, the performance on identity accuracy and emotion accuracy drop the most when removing the Prosody Adaptor (PA). This can be attributed to the predicted pitch and energy representing a speaker’s identity and his/her emotion. When removing the Atmosphere Booster (AB), the performance drops compared to the full model (Row 9) but does not drop as much as when removing the other two modules. This indicates the AB module also contributes the overall performance improvement but contributes the least in the three modules.

Effectiveness of valence and arousal. In our model, we bridge arousal with energy and valence with pitch by attention mechanism. To evaluate their effectiveness, we cut off the connection and predict energy and pitch directly from the phoneme-lip representation. The results are presented in Row 4$\sim$5 of Table 3. It shows that the performance drops significantly when removing either of them and drops more when removing valence. This indicates valence contributes more than arousal on the V2C task.

Valence-and-Arousal v.s. Facial Expression. To compare the role of affective display and facial expressions on prosody inference, we replace the input of the APA module with original embeddings of facial features from CNN. As row “VA $v.s.$ FF” of Table 3 shows, the model performance drops significantly. This is likely caused by facial features that are still far from the information-containing emotion, which is not enough to guide prosody generation.

Multi-head Attention in Duration Adaptor. In our duration aligner, we exploit multi-head attention to learn the relation between the phoneme sequences and the lip motion. To verify its effectiveness, we conduct experiments using conventional dot-product attention. The results are presented in Row 7 of Table 3. The performance drops significantly on all metrics, such as 26.59% and 44.02% decrease on LSE-D and LSE-C compared to Row 9, respectively. This indicates multi-head attention learns a much better correlation between phonemes with lip movement.

Conv-Transpose v.s. Duplication. In our duration aligner, we propose to use Conv-Transpose to learn the expansion of the fused phoneme-lip representation to its desired length. To verify its effectiveness, we replace it with simple make $n$ duplications as in [20]. The results are shown in Row 8 of Table 3. It shows that the performance drops significantly. For example, it falls 15.94% on MCD metric. This demonstrates the superiority of using transposed convolution to learn the upsampling than simply copying.

Audiovisual consistency visualization. Figure 3 presents the mel-spectrograms of generated audios along with its frames. Blue and red bounding boxes denote whether the character is speaking or not, respectively. Compared with other methods, the mel-spectrogram generated by our model is closer to the ground truth, indicating better audio-visual synchronization. This can be attributed to our duration aligner, which exploits multi-head alignment between the text phoneme sequence and the lip movement sequence. By controlling the lip movements explicitly, we obtain the desired length of mel-spectrograms, which makes the speech well synchronized with the input video.

Arousal and valence with prosody visualization. We selected two examples from the test set of the V2C dataset and Chem dataset to demonstrate the alignment between energy and arousal as well as pitch and valence. The valence (positive or negative) and arousal (calming or exciting) of facial expressions are shown in the first column. The main pitch and energy are shown in orange and blue curves in the right column. We use the red circle to highlight the pitch and energy that correspond to the video frame shown in the left column. The result shows that our method achieves energy and pitch closer to the ground truth speech. When the chemistry teacher becomes excited and positive, our model successfully leverages the affective display to synthesize a similar pitch and energy as the ground-truth speech.