ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance

Early methods (Zhang et al., 2023a; Ren et al., 2021; Deng et al., 2020a) use 3DMM priors to instruct the generator. SadTalker (Zhang et al., 2023a), for instance, generates head poses and expressions of 3DMMs from audio and implicitly modulate a 3D-aware face synthesis model. Recent methods (Zheng et al., 2022; Grassal et al., 2022; Zielonka et al., 2023; Gafni et al., 2021; Athar et al., 2022; Peng et al., 2023), combined 3D representation techniques with 3DMMs to control the expressions and poses of avatars. For example, NHA (Grassal et al., 2022) and IMAvatar (Zheng et al., 2022) refine the mesh topology of FLAME (Li et al., 2017) to achieve more realistic mesh-based avatars. INSTA (Zielonka et al., 2023) and RigNeRF (Athar et al., 2022) utilize 3DMMs to construct radiance fields. Additionally, other 3D representation methods , such as GaussianAvatars (Qian et al., 2023), construct dynamic 3D representations based on 3D Gaussian splats rigged to a parametric morphable face model. More recent works (Kirschstein et al., 2023; Shen et al., 2023; Kim et al., 2023a; Ding et al., 2023) have leveraged the powerful generation capabilities of 2D diffusion models. For instance, DiffusionAvatars (Kirschstein et al., 2023) utilizes the 3D priors of the recent Neural Parametric Head Model (NPHM) (Giebenhain et al., 2023) combines with a 2D rendering network for high-quality image synthesis. In our work, we integrate 3D representation with diffusion models, simultaneously leveraging 2D and 3D priors for consistent head avatar synthesis.

Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020) integrate image generation with a sequential denoising process of isotropic Gaussian noise. In this process, the model is trained to anticipate noise levels from the input image. Due to their remarkable ability to generate 2D content, they have gradually found application in face-related tasks. Many studies (Ding et al., 2023; Shen et al., 2023; Kirschstein et al., 2023; Kim et al., 2023a), refine pre-trained diffusion models by introducing additional control factors like landmarks or depth. For example, DiffusionRig (Ding et al., 2023) suggests conditioning the diffusion model on rasterized grids, considering factors such as normals and albedo. Conversely, DiffTalk (Shen et al., 2023) utilizes audio-driven and landmark-based conditioning to improve identity generalization. Despite the good controllability and visual quality of these methods, they often lack accurate 3D representations or precise 2D detail priors. As a result, they struggle to maintain consistent 3D rendering across various viewpoints and temporal consistency in video rendering simultaneously.

Recently, there has been a growing effort to incorporate emotions into the synthesis process for better control over the final output. For instance, EMOCA (Danecek et al., 2022) introduces a novel deep perceptual emotion consistency loss, ensuring alignment between the reconstructed 3D expression and the depicted emotion in the input image. Meanwhile, GMTalker (Xia et al., 2023) proposes a Gaussian mixture-based Expression Generator (GMEG), enabling the creation of a continuous and multimodal latent space for more versatile emotion manipulation. However, approaches like EVP (Ji et al., 2021) and EMMN (Tan et al., 2023), which directly infer emotions from labeled audio, may encounter accuracy issues due to the inherent complexity of emotional expression. In contrast, methods such as MEAD (Wang et al., 2020) and the work by Sinha et al. (Sinha et al., 2022) implicitly learn the intrinsic relationship between emotions and facial expressions through the use of emotion labels. In our methodology, we leverage the MEAD (Wang et al., 2020) dataset to assign emotion labels to the experimental dataset based on emotional similarities, facilitating precise control over the emotions of the resulting portraits.

As mentioned above, to ensure temporal consistency in generating portraits, we extract Temporally-Sensitive details (TSD) generated by Fourier transformation from the coarse RGB output of INSTA (Zielonka et al., 2023) and the target video frame. We then align these details through a diffusion model. This approach complements other conditions and contributes to achieving fully consistent and high-quality avatar generation results.

Given a monocular RGB video containing $K$ frames $\{$${I_{1},I_{2},\ldots,I_{K}}$$\}$ , camera pose $P_{c}$, tracked FLAME (Li et al., 2017) meshes $\boldsymbol{M}=\{M_{i}\}$ with corresponding expressions $\boldsymbol{E}=\{E_{i}\}\in\mathbb{R}^{1\times 100}$ and poses $\boldsymbol{P}=\{P_{i}\}\in\mathbb{R}^{1\times 6}$. We obtain RGB output and normal using INSTA (Zielonka et al., 2023). The process is described as follows:

where $\mathcal{F}_{insta}$ is the INSTA model. After obtaining the RGB output, we utilize Fourier transform and its inverse transform to extract TSD from the RGB output. We apply the Fourier transform to shift an image from its spatial domain to the frequency domain, revealing various frequency components. Retaining these high-frequency elements enables the Fourier transform to efficiently extract image details, which are represented as:

where the equation represents content extraction at frequency $w$ from the image $I_{rgb_{i}}$, with $e^{-jwt}$ as the orthogonal basis and $W$ as the frequency set of the image. Experimentally, this paper sets $w$$=$$10$. Then performing inverse Fourier transform to convert the frequency domain back to images, which are represented as:

As illustrated in Fig. 2, the TSD of a video frame contains high-frequency information and contours that represent crucial feature of expression, head pose and details for generating the avatar. However, TSD also varies significantly between adjacent frames. This inspires us to predict stable TSD of a frame from inaccurate $I_{TSD_{i}}$. Performing the same operation on the ground truth video frames allows us to obtain the ground truth of TSD. Therefore, we propose employing a Temporally-Sensitive Diffusion Model (TSDM) to align $I_{TSD_{i}}$. Specifically, inspired by (Ding et al., 2023), we first encode the obtained $I_{TSD_{i}}$ $\in\mathbb{R}^{512\times 512\times 3}$ to obtain the global latent code $\mathcal{E}(I_{TSD_{i}})\in\mathbb{R}^{1\times 256}$. Then we pass $P_{i}$ and $E_{i}$ through a linear layer to obtain the same scale as $\mathcal{E}(I_{TSD_{i}})$ and concatenate them to get $f_{c}\in\mathbb{R}^{3\times 256}$. We add new cross-attention layers to the U-Net, following IPAdapter (Ye et al., 2023). Let $Z$ be the intermediate feature map computed by an existing cross-attention operation in the pre-trained LDM (Rombach et al., 2022): $Z=ATTENTION(Q,K,V)$. Then, we perform direct conditioning by adding another cross-attention layer:

The more specific denoising process can be represented as follows:

where $x_{t}$ is the noisy image at timestep $t\in\{1,2,\ldots,T\}$, $x_{0}$ is the ground truth latent image, ${\hat{\epsilon}}_{t}$ is the predicted noise, and $\mathcal{F}$ represents the denoising model. The optimization objective of the entire denoising process is as follows:

$\tilde{x}_{t-1}=x_{t}-{\hat{\epsilon}}_{t}$ is the denoising result of $x_{t}$ at time step t. The final denoised result $\tilde{x}_{0}$ is then upsampled to the pixel space with the pretrained image decoder $I_{TSD_{i}}^{\prime}$ = $\mathcal{D}_{TSD}$($\tilde{x}_{0}$), where $I_{TSD_{i}}^{\prime}\in\mathbb{R}^{512\times 512\times 3}$ is the reconstructed TSD image. In this way, we align $I_{TSD_{i}}$ to an accurate one, providing the key information of real portrait changes along time axis. Further, $I_{TSD_{i}}^{\prime}$ can be used as a reliable guidance for temporally-consistent avatar generation.

In Sec. 4.1, after passing TSD through the diffusion model, we obtain precise and temporally stable TSD. This compensates for other conditions that may not be particularly accurate. Therefore, by adding normal condition and emotion condition, we alleviate 3D consistency and expression consistency.

Emotion Condition: For emotion condition, we propose utilizing the MEAD (Wang et al., 2020) dataset to acquire emotion labels for our experimental dataset. As described in Fig. 3, the MEAD comprises numerous monocular video clips $\{$${V_{1},V_{2},\ldots,V_{M}}$$\}$, each clips corresponds to an emotional label $L_{m}\in\{$${L_{1},L_{2},\ldots,L_{M}}$$\}$ (such as anger, happy, etc.). Leveraging this rich emotion dataset, we construct a facial expression database. Subsequently, employing DECA (Feng et al., 2021), we compute facial expression vectors for each frame in every clip $E_{mn}$, $n$ represents the frame index of the clip. And then determine the similarity between the current frame’s facial expression vector $E_{i}$ and those in the database. Finally, we assign the emotion label corresponding to the facial expression vector with the highest similarity in the database as the emotion label for the current frame. The process is described as follows:

where $I_{i}$ is an image from our dataset, and $V_{mn}$ is a frame from the MEAD dataset, $cs()$ represents the cosine similarity function, which is used to measure the similarity between two expression vectors. $argmax()$ retrieves the image corresponding to the maximum similarity, which belongs to the MEAD dataset. After acquiring emotional labels corresponding to image through $\mathcal{E}\mathcal{M}\mathcal{O}(I_{i})$, we use the text encoder in CLIP (Radford et al., 2021) to extract textual features of emotion labels, obtaining the final emotion code $T_{i}\in\mathbb{R}^{1\times 768}$. After acquiring STD, normal, and emotion conditions, we perform the same linear operation on $T_{i}$ as on $E_{i}$ and $P_{i}$, and then concatenate them to obtain $f_{d}\in\mathbb{R}^{4\times 256}$. At that time, it can be utilized as a condition through cross-attention operation. Finally, we construct a Fully Consistent Stable Diffusion model (FCSD). Specifically, we utilize the pretrained Stable Diffusion model (SD) (Rombach et al., 2022) as the backbone network to expedite training and enhance generation quality. Additionally, we incorporate a fully consistency module, effectively integrating consistency-related information into the backbone network, to construct a ControlNet (Zhang et al., 2023b). The final rendering is achieved through an iteratively denoising the full noise $x_{T}$ with our fully-consistent neural renderer $\mathcal{S}$ conditioned on $z_{c}$:

where C is the ControlNet architecture and $z_{c}$ is the concatenation of the normal condition and the TSD condition. During training, we minimize the following loss:

Similar to Sec. 4.1, the final denoised result $\tilde{x}_{0}$ is then upsampled to the pixel space with the pretrained image decoder $I_{gen_{i}}$ = $\mathcal{D}_{portrait}$($\tilde{x}_{0}$), where $I_{gen_{i}}\in\mathbb{R}^{512\times 512\times 3}$ is the reconstructed face image. This process provides the network with additional multimodal conditions, helping it control the 3D and expression consistency of the characters under the well-guided aligned TSD, thereby assisting in generating portrait animations with more precise motion.

It is indisputable that both generation speed and the quality of portrait generation are pivotal criteria for task assessment. We closely follow the development of diffusion models and, based on this foundation, further accelerate efficiency and enhance image quality.

Latent Consistency Model (LCM). To ensure our model maintains high-quality generation while minimizing the required steps, we employ LCM (Luo et al., 2023) to expedite the generation process. The core concept of this model lies in redefining the fundamental logic of traditional diffusion models (such as DDPM (Ho et al., 2020)) in the generation process. Traditional diffusion models generate final results by gradually reducing noise in images, typically through iterative and time-consuming processes. In contrast, LCM (Luo et al., 2023) transforms traditional numerical ordinary differential equation (ODE) solvers into neural network-based solvers, enabling direct prediction of the final clear image and thereby reducing intermediate steps, significantly enhancing efficiency. By employing LCM (Luo et al., 2023) to enhance our model, we have reduced the required steps for predicting the final image from around 1000 steps to approximately 10 steps.

Refiner. Inspired by Stable Diffusion XL (SDXL) (Podell et al., 2023), we have further elevated the generation quality of our model. SDXL represents the latest optimized version of Stable Diffusion, comprising a two-stage cascaded diffusion model consisting of a Base model and a Refiner model. Integrating the Refiner model with our own, after the Base model generates latent features of the image, we utilize the Refiner model to conduct minor noise reduction and enhance detail quality on these latent features.

Dataset: In our experiments, we utilize four datasets. Initially, we employ the dataset released by INSTA (Zielonka et al., 2023) to train the primary framework. This dataset comprises 10 monocular videos, each capturing the performance of an individual actor. These videos undergo cropping and resizing, achieving a resolution of ${512^{2}}$, effectively removing extraneous elements from the facial region through background subtraction. Additionally, for equitable comparison with other methods, we utilize two datasets from IMAvatar (Zheng et al., 2022) and NerFace (Gafni et al., 2021) which share the same data format as INSTA (Zielonka et al., 2023). Finally, during the emotional labeling phase, we utilize the Multi-view Emotional Audio-visual Dataset (MEAD) (Wang et al., 2020), which includes a dialogue video corpus featuring 60 actors conversing with 8 different emotions. High-quality audio-visual clips from seven different perspectives are captured in a strictly controlled environment.

Evaluation Protocol: We utilize the final 350 frames of each video in our dataset for testing purposes, while the remaining frames are allocated for training. To assess image quality, we employ metrics consistent with state-of-the-art methods (Zielonka et al., 2023; Zheng et al., 2022), including L2 loss, structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and the perceptual metric LPIPS. Additionally, to evaluate the temporal consistency of the generated videos, we utilize optical flow to compute the degree of change between adjacent frames.

Implementation Details: Our framework was implemented using PyTorch on a machine equipped with an RTX 3090 GPU. The training process is divided into three stages. In the first stage, all experimental configurations match those of INSTA (Zielonka et al., 2023), yielding RGB and normal outputs for the portraits. Moving to the second stage, we utilize the temporally unstable TSDs obtained from the RGB outputs of the first stage to train an TSDs generator with ground truth supervision. We utilize the Adam optimizer (Kingma and Ba, 2015) with a learning rate of $10^{-5}$ and conduct 3000 iterations with a batch size of 4. Finally, in the third stage, we train a fully consistent portrait generator based on the outputs of the first two stages. Once again, we use the Adam optimizer with a learning rate of $10^{-4}$ and conduct 15000 iterations with a batch size of 4.

Baseline setting: To illustrate the role of TSD in temporal consistency more clearly, we use Stage 1 combined with Stage 3 as our baseline. Here, TSD is not learned through Stage 2, but is directly used as a condition for ControlNet (Zhang et al., 2023b).

Image quality evaluation: We conduct comprehensive experiments on our dataset, assessing the quality and consistency of the synthetic digital human avatars generated by our method, and comparing them with state-of-the-art methods such as IMAvatar (Zheng et al., 2022), DiffusionRig (Ding et al., 2023) and INSTA (Zielonka et al., 2023). As shown in Fig. 4, our approach yields more realistic results. We capture facial details such as wrinkles and eyeglass frames well. In contrast, other methods like INSTA (Zielonka et al., 2023) fail to capture wrinkle information and other details. Tab. 1 presents quantitative results, demonstrating that our method achieves state-of-the-art performance across three different datasets. Particularly noteworthy is the substantial improvement in both PSNR and L2 loss compared to INSTA (Zielonka et al., 2023). Taken together, these metrics indicate that our generated portraits are closer to ground truth and more realistic. Additionally, our ability to better capture facial expressions contributes to the overall enhancement of the results. Additional. We are well aware of the importance of real-time performance, so as mentioned in Sec. 1, we have focused on improving speed by introducing LCM (Luo et al., 2023) on top of our model. As shown in the data in the Tab. 1, we have significantly reduced the time. Note that the above results were obtained with a time step setting of 10 during the denoising process, while the default diffusion model step size is set to 1000. The incorporation of LCM (Luo et al., 2023) significantly reduces the time step.

3D consistency results: 3D consistency measures whether the character maintains the correct appearance from different viewpoints. As shown in Fig. 1 (b), methods like SadTalker (Zhang et al., 2023a) do not naturally exhibit good 3D consistency since they do not involve 3D operations during learning and rely on a single-image 2D generation process. From Fig. 5, we can also observe the advantage of our method, with the angles of head rotation being closer to the ground truth and the results appearing more realistic. For quantitative results, we utilize DECA (Feng et al., 2021) to estimate the pose coefficients of the generated portraits, calculating the error between the predicted pose coefficients and the ground truth, referred to as Pose Error (PE), measured by the Euclidean distance. As shown in Tab. 2, our method significantly outperforms SadTalker, with minimal error compared to the ground truth.

Expression consistency results: The expression consistency measures how well the generated results fit the target expressions. As shown in Fig. 6, our method is capable of achieving more accurate expressions. For quantitative analysis, similar to PE, we employ DECA (Feng et al., 2021) to evaluate the expression coefficients of the generated portraits, calculating the disparity between the predicted expression coefficients and the ground truth, referred to as Expression Error (EE). As depicted in Tab. 2, our approach surpasses both Diffusionrig (Ding et al., 2023) and INSTA (Zielonka et al., 2023), exhibiting minimal error relative to the ground truth.

Temporal consistency results: The evaluation of temporal consistency evaluates the stability of the generated video, focusing on the absence of jitter or significant fluctuations between frames, ensuring smooth playback. We compare our method with state-of-the-art approaches like INSTA(Zielonka et al., 2023), SadTalker(Zhang et al., 2023a) and DiffusionRig (Ding et al., 2023) in various categories. We utilize optical flow to compute the degree of change between two frames. Naturally, we consider the normal variation between video frames. Therefore, we compare it with the ground truth as well. From the Fig. 7, it’s evident that our approach closely approximates the ground truth.

We conduct a series of ablation studies to analyze the different components of our method. Specifically, we focus on 1) the impact of unaligned coarse TSD versus aligned TSD through the diffusion model on other conditions and on temporal consistency; 2) the impact of the emotion condition on the final results; 3) the impact of the normal condition on the final 3D consistency;

Aligned TSD and Unaligned TSD. From Fig. 1 (d), it is evident that the temporal consistency of our baseline is inferior to that of our final method. Since the baseline directly adopts unaligned TSD conditions, this proves that aligned TSD can provide better temporal consistency. From Fig. 8 (a) and (c), it can be observed that aligned TSD conditions offer more accurate expression and better 3D consistency.

Emotion Condition. With the aligned TSD as a condition, we compared the results with and without the addition of the emotion condition. As shown in Fig. 8 (b), when the emotion condition is included, it helps the model fine-tune facial expressions, resulting in outcomes closer to the ground truth.

Normal Condition. With the aligned TSD as a condition, we compare the results with and without the addition of the normal condition. As shown in Fig. 8 (d), when the normal condition is not included, it fails to maintain good 3D consistency, resulting in artifacts and other noise.