Discrete Prior-Based Temporal-Coherent Content Prediction
for Blind Face Video Restoration

An important video enhancement task is super-resolution, which aims to restore high-resolution (HR) video frame sequences from degraded low resolution (LR) video frames while preserving the original semantics. Due to the fact that adjacent frames in video sequences can provide reference information to each other, there are a number of temporal sliding-window based VSR methods that processed LR frames in a time sliding window manner. STTN(Kim et al. 2018) selectively warped target frames for enhancement using the estimated optical flow. (Liu et al. 2017) proposed a spatial alignment network for neighboring frames alignment. Another line of research involves utilizing recurrent neural networks to harness temporal information from multiple frames. RLSP(Fuoli, Gu, and Timofte 2019) propagated high-dimensional hidden states to better capture long-term information. BasicVSR(Chan et al. 2021) utilized a recurrent network design with bidirectional propagation to enchance video quality and detial by leveraging information from neighboring frames. BasicVSR++(Chan et al. 2022) built upon this framework by adding a second-stage refinement network and alignment module to further improve the precision and stability of the super-resolution process. Nevertheless, these methods have not incorporated suitable generative priors, which made them struggle to restore content that has lost texture details. Recently, diffusion-based methods have made great progress(Esser et al. 2023)(Hu, Chen, and Luo 2023)(Ho et al. 2022). Upscale-A-Video(Zhou et al. 2024a) proposed text-guided latent diffusion framework for video enhancement, in which a flow-guided recurrent latent propagation module was used to enhance temporal stability. However, in BVFR task, it is difficult for the model to obtain stable optical flow from degraded data, which could negatively affect the performance of the restoration model.

Significant progress has been achieved in face image restoration in recent years. There are methods that introduced facial landmarks (Hu et al. 2021; Lin et al. 2020), face parsing maps(Chen et al. 2021), or 3D shapes(Zhu et al. 2022) in their designs. However, prior information obtained from degraded images often contains significant noise, which could lead to a performance drop. On the other hand, a number of recent works focus on exploring how to utilize the generative priors in generative models, such as StyleGAN2(Karras et al. 2020), Stable Diffusion(Rombach et al. 2021), etc, to assist in image restoration. GFP-GAN(Wang et al. 2021) and GPEN (Yang et al. 2021) leveraged facial spatial features to guide the restoration model, enabling it to utilize the geometric and texture priors of the pretrained StyleGAN. However, these methods overly relied on the facial features of degraded images, which made them sensitive to facial noise. To handle degradations robustly, DR2(Wang et al. 2023) added Gaussian noise to the degraded images, and then restored them through a pre-trained diffusion model to achieve the effect of removing degradation. DiffBIR(Lin et al. 2023) first removed degradation without generating new content, and then restored face details based on a pre-trained Stable Diffusion model(Rombach et al. 2021). However, because these methods were specifically designed for the task of image restoration and involve randomness in sampling, they could result in issues such as discontinuity and flickering when applied to sequences of degraded face images. There are also methods that attempted to utilize sparse representation with learned dictionaries to address image restoration tasks which took advantage of dictionary information in different ways to represent degraded images(Gu et al. 2022; Wang et al. 2022; Zhou et al. 2022b). RestoreFormer learned fully-spatial interactions between corrupted queries and high-quality dictionary key-value pairs, and CodeFormer built a code sequences prediction module to exploit the dictionary information. However, these methods were specifically designed for image restoration, and did not take into account temporal coherence in the video restoration task.

Different from the above dictionary-based methods, we model visual priors with respect to videos, capture the interrelationships among video tokens, and predict high-quality token features from the priors to replace the degraded ones. In addition, we perform cross-frame statistics modulation with motion priors, such that the predicted content can be further improved in terms of temporal coherence. This design distinguishes the proposed approach from the existing face restoration and video enhancement methods.

We introduce a novel BFVR method of applying visual prior-based content prediction to restore degraded features, and leveraging motion prior-based statistics modulation to enhance temporal coherence. The structure of our framework is depicted in Figure 2 and comprises four main components: an Encoder $E$, a Visual Prior-based Spatial-temporal-aware Content Prediction Module $C$, a Motion Prior-based Statistics Modulation module $M$, and a Generator $G$. The process begins with the Encoder $E$, which transforms frames from degraded video clips $v_{lq}$ into a sequence of video tokens. These tokens $z$ are then processed by the content prediction module $C$, which generate the features $z^{\prime}$ of high-quality content from a vision bank. Features $z^{\prime}$ is further refined by the motion prior-based statistic modulation Module $M$, which adjusts the cross-frame mean and variance of $z^{\prime}$, resulting in a motion-enhanced representation $z^{\prime\prime}$. Subsequently, several cross-attention-based transformer were applied to merge $z^{\prime}$ and $z^{\prime\prime}$, forming the feature $\hat{z}$ that corresponds to a high-quality and temporally smooth face video. The feature $\hat{z}$ are then decoded by $G$ to reconstruct the face video clip $\hat{v}$ that aims to achieve maximum consistency with the ground truth. This reconstruction process contributes to the fidelity and fluidity of the restored video.

Traditional image restoration techniques typically focus on individual frames, thus neglecting the dynamic inter-dependencies among successive frames, which makes it difficult to fully utilize the spatial information between frames in the video clip, especially for degraded video. To address this problem, the proposed spatial-temporal-aware content prediction module, conditioned on degraded video tokens, predicts index values from the pre-trained vision bank $\mathbb{T}$. The vision bank is constructed through vector quantization during the high-quality image reconstruction process, in which each index corresponds to a high-quality visual representation. In order to effectively predict index values $\hat{l}$ with nuanced contextual associations, we model both the spatial and temporal information of degraded video clips. Initially, video clip tokens $z$, extracted from encoder $E$, are flattened. To enhance the model’s capability of interpreting spatial and temporal positioning of each token, we integrate a learnable spatial-temporal position embedding into each token. Subsequently, multiple self-attention-based transformer blocks, coupled with an activation function, calculate the likelihood of each token belonging to a particular bank’s index. The process of index prediction can be formalized as follows:

$$\widetilde{z}=\xi^{{}^{\prime}}(\texttt{SA}(\xi(z)+\mathbf{P}_{emb})),\\$$

(1)

$$\hat{l}_{i,j,k}=\mathop{\arg\max}\limits_{r\in\{0,1,...,N\}}\psi(\widetilde{z}_{i,j,k})_{r},$$

(2)

where $\mathbf{P}_{emb}$ represents the learnable spatial-temporal position embedding, $\xi$ and $\xi^{{}^{\prime}}$ denote the flatten and unflatten operations respectively, SA denotes the self-attention based transformer, $\psi(\cdot)_{r}$ denotes the softmax activation output on r-th index and $\hat{l}_{i,j,k}\in\{0,1,...,N\}$ denotes the predicted index of bank, where $N$ is the size of the bank $\mathbb{T}$, and $i$, $j$, and $k$ represent the spatial and temporal locations of the video clip tokens. By computing the index $l$, we can match the corresponding features of high-quality content in the bank as follows:

$$z^{{}^{\prime}}_{i,j,k}=\kappa(\mathbb{T},\hat{l}_{i,j,k}),$$

(3)

where $\kappa$ represents using an index $\hat{l}_{i,j,k}$ to extract features from the bank $\mathbb{T}$ and $z^{{}^{\prime}}$ denotes the high quality face features. The self-attention-based computation facilitates the modeling of the relationships between tokens across multiple frames, such that the module can predict index values by extracting complementary information from different tokens, eventually combining the extracted features to create a representation of high-quality content with contextual association and dynamic consistency.

In order to further enhance the temporal coherence of the predicted content, we introduce the motion prior-based statistics modulation module. Initially, we construct a statistics bank, denoted as $\mathbb{M}$, which is trained using high-quality face videos. The bank $\mathbb{M}$ is designed to store the cross-frame mean and variance vectors whose components are the channel mean and variance corresponding to each frame. During the processing of features $z^{\prime}$, we first calculate the mean and variance in the channel dimension for each frame as follow:

$$\mu_{f,w,h}=\sum_{c}\frac{z^{\prime}_{f,c,h,w}}{L_{C}},$$

(4)

$$\sigma^{2}_{f,w,h}=\frac{\sum_{c}(z^{\prime}_{f,c,h,w}-\mu_{f,w,h})^{2}}{L_{C}},$$

(5)

where $L_{C}$ denotes the channel size of $z^{\prime}$. Subsequently, we concatenate $\mu_{f,w,h}$ and $\sigma^{2}_{f,w,h}$ in the temporal dimension to generate a mean vector and a variance vector, respectively. The mean and variance vectors are then concatenated, and a nearest matching algorithm is employed to retrieve the corresponding element from $\mathbb{M}$, which can be formalized as follows:

$$[\mu^{{}^{\prime}}_{w,h},\sigma^{{}^{\prime}2}_{w,h}]=\mathop{\arg\min}\limits_{e\in\mathbb{M}}||\texttt{Concat}(\mu_{w,h},\sigma^{2}_{w,h})-e||^{2}_{2},$$

(6)

where Concat represents the concatenation operation in the temporal dimension. We exploit the mean $\mu^{{}^{\prime}}_{w,h}$ and variance $\sigma^{{}^{\prime}2}_{w,h}$ vectors with high-quality motion priors to modulate predicted content $z^{\prime}$, as follows:

$$z^{{}^{\prime\prime}}_{f,w,h}=\sigma^{{}^{\prime}}_{f,w,h}\frac{z^{{}^{\prime}}_{f,w,h}-\mu_{f,w,h}}{\sigma^{{}^{\prime}}_{f,w,h}+\epsilon}+\mu^{{}^{\prime}}_{f,w,h},$$

(7)

where $\epsilon$ is a small real value and $z^{{}^{\prime\prime}}_{f,w,h}$ denotes the modulated feature. The modulation of mean and variance is tailored to maintain temporal coherence of predicted content, thereby mitigating issues like flickering. In addition, we incorporate a cross-attention based transformer block, in which feature $z^{\prime}$ serves as the query and $z^{\prime\prime}$ is used as both the key and value. This arrangement allows for a comprehensive integration of content information with modulated features, which can be formalized as follows:

$$\hat{z}=\xi^{{}^{\prime}}(\texttt{CA}(\xi(z^{\prime}),\xi(z^{\prime\prime}))),$$

(8)

where $\texttt{CA}(a,b)$ denotes the transformer block in which $a,b$ denote query and key-value input, respectively. We restore the clear face video clip $\hat{v}$ by inputting $\hat{z}$ into the generator $G$. The transformer block is optimized together with G. To enhance the temporal coherence of restored texture, we have integrated multiple 3D residual blocks and frame attention mechanisms into $G$.

In this section, we assess the effectiveness of our content prediction module. First of all, we analyze the convergence of this module. To illustrate the role of spatial-temporal context in accelerating convergence speed, we replace spatial-temporal-aware content prediction module (labeld as ‘S & T-aware’) with spatial-aware version (labeled as ‘S-aware’) which do not consider the relationship between frames.

As shown in Figure 3, at the beginning of training, distortions are noticeable in the restored face. When converging to the same position, ‘S-aware’ requires 5.6 times more iterations than ‘S & T-aware’. After convergence, both methods can restore face semantics well, but ‘S & T-aware’ has better facial details. To ascertain whether the module leverages contextual information of degraded video frames, we visualize the attention maps of the last transformer block in Figure 4. The region of left eye serves as query, and the four attention maps on the right hand side show the responses corresponding to the query. It is worth noting that the module exhibits a heightened focus on the left eye region across different poses. The result suggests that our model effectively captures contextual information from multiple frames to restore the content on the current position.

To demonstrate the superiority of the proposed DP-TempCoh, we perform quantitative and qualitative comparisons with state-of-the-arts, including the following image enhancement methods: CodeFormer(Zhou et al. 2022b), RestoreFormer(Wang et al. 2022), DR2(Wang et al. 2023), DiffBIR(Lin et al. 2023), DifFace(Yue and Loy 2024), and video enhancement methods: BasicVSR++(Chan et al. 2022), MIA-VSR(Zhou et al. 2024b), IA-RT(Xu et al. 2024), FMA-Net(Youk, Oh, and Kim 2024).