Attention-Guided Multi-scale Interaction Network for Face Super-Resolution

The encoding stage in our network aims to extract facial features of different scales. In this stage, given a degraded face image ${I_{LR}\in\mathbb{R}^{3\times H\times W}}$, first, a ${3\times 3}$ convolution is used to extract facial shallow features. Then, extracted facial features are further fined by three encoder stages. Each encoder comprises our designed Local and Global Feature Interaction Module (LGFI) and a downsampling operator. After each encoder, the input face feature’s channel counts will be doubled, and the size of the image of the input face feature will be halved. As shown in Fig. 2, the features obtained after three encoders are as follows: ${F_{1}\in\mathbb{R}^{2C\times\frac{H}{2}\times\frac{W}{2}}}$, ${F_{2}\in\mathbb{R}^{4C\times\frac{H}{4}\times\frac{W}{4}}}$, ${F_{3}\in\mathbb{R}^{8C\times\frac{H}{8}\times\frac{W}{8}}}$.

In the bottleneck stage between the encoding and decoding stages, obtained encoding features are designed to be fine-grained. ${F_{4}\in\mathbb{R}^{8C\times\frac{H}{8}\times\frac{W}{8}}}$ is obtained through the bottleneck stage. In this stage, we continue to use two LGFIs to refine and enhance encoding features to ensure they are better utilized in the decoding stage. After this stage, our model can continuously enhance the information about the facial structure at different scales, thus improving the perception of facial details.

In the decoding stage, there are three decoders. We focus on multi-scale feature fusion, aiming at reconstructing high-quality face images at this stage. As depicted in Fig. 2, each decoder includes an upsampling operation, an EDFF, and an LGFI. Each upsampling operator halves the input feature channel counts while doubling the width and weight of the input facial feature. Compared to encoding stages, decoding stages additionally use our proposed SKAF to adaptively selectively fuse different scale features from the encoder and decoder stages. Through this design, different scale features can interact to recover more detailed face features. The features obtained after three decoders are as follows: ${F_{5}\in\mathbb{R}^{4C\times\frac{H}{4}\times\frac{W}{4}}}$, ${F_{6}\in\mathbb{R}^{2C\times\frac{H}{2}\times\frac{W}{2}}}$, ${F_{7}\in\mathbb{R}^{C\times H\times W}}$. Finally, a ${3\times 3}$ convolution unit is utilized to transform our obtained deep facial feature into output FSR image ${I_{SR}}$.

As for the loss of our AMINet, given a dataset $\mathop{\left\{{\mathop{I}\nolimits_{LR}^{i},\mathop{I}\nolimits_{HR}^{i}}\right\}}\nolimits_{i=1}^{N}$, we optimize our AMINet by minimizing the pixel-level loss function:

where $N$ denotes paired training face image counts. ${I_{LR}^{i}}$ and ${I_{HR}^{i}}$ are the face LR image and HR image of the $i$-th pair, respectively. Meanwhile, $F_{AMINet}(\cdot)$ and $\Theta$ denote the AMINet and the number of parameters of AMINet, respectively.

We utilize Self-attention (SA) to extract global facial features, which can effectively model the relationships between distant features. Meanwhile, through the multi-head mechanism in SR, features can be captured from different subspaces, improving the robustness and generalization ability of the model. As illustrated in Fig. 3 (b), we start by applying a ${1\times 1}$ convolutional layer followed by a ${3\times 3}$ depth-wise convolutional layer to combine pixel-level cross-channel information and extract channel-level spatial context. From this spatial context, we then generate ${Q,K,V\in\mathbb{R}^{C\times H\times W}}$. For an input facial feature $X\in\mathbb{R}^{C\times H\times W}$ , the process of obtaining ${Q,K,V\in\mathbb{R}^{C\times H\times W}}$ can be described as:

where ${F_{conv1}}(\cdot)$ is the ${1\times 1}$ pointwise convlution and ${F_{dw3}}(\cdot)$ is the ${3\times 3}$ depthwise convlution.

Next, we reshape $Q$, $K$, and $V$ into ${\hat{Q}\in\mathbb{R}^{C\times HW}}$, ${\hat{K}\in\mathbb{R}^{HW\times C}}$, and ${\hat{V}\in\mathbb{R}^{C\times HW}}$, respectively. After that, the dot product is multiplied by $V$ to obtain weights $X_{w}\in\mathbb{R}^{C\times HW}$, which facilitates the capturing of the important local context in SA. Finally, we rearrange $X_{w}$ into $\hat{X_{w}}\in\mathbb{R}^{C\times H\times W}$. The above operations can be expressed as:

where ${X_{sa}}$ is the attention map of SA, ${\sqrt{d}}$ is a factor used to scale the dot product of ${\hat{K}}$ and ${\hat{Q}}$, ${R(\cdot)}$ stands for the rearrange operation, ${X_{sa}}$ denotes the output of SA.

As shown in Fig. 3 (a), we design RDFE to extract local facial features at different scales. Compared with the traditional feed-forward network (FFN), our RDFE is beneficial for processing more complex features and multi-scale features flexibly. Specifically, for the input feature $\mathop{X}\in\mathbb{R}^{C\times H\times W}$, we use depthwise convolutions of ${3\times 3}$, ${5\times 5}$, and ${7\times 7}$ to parallelly extract three scales facial features, which depthwise convolution can reduce the computational complexity of the model, while convolution with different kernel sizes can effectively extracting rich face details. The above operations can be expressed as:

where $F_{dw3}$, $F_{dw5}$, and $F_{dw7}$ are ${3\times 3}$, ${5\times 5}$, and ${7\times 7}$ depthwise convlution, respectively. Additionally, we use an attention unit $F_{au}$ to calculate the feature weight of the fusion feature of three branches. Then, we use calculated weight to perform multiplications with the three branch features at different scales. This operation enables our model to adaptively allocate weights to extract important facial information under different receptive fields, improving our model’s performance and generalization ability. Next, we use ${3\times 3}$, ${5\times 5}$, and ${7\times 7}$ depthwise convolutions to further reconstruct selected important facial features. The above operations can be expressed as:

where $H_{cat}$ is a concat operator. Then, we aggregate the three branches’ features to combine facial detail information under different receptive fields. This process can be described as:

where $\mathop{F}\nolimits_{conv1}(\cdot)$ represents ${1\times 1}$ convolution. Finally, we utilize a Feature Refinement Module to refine features obtained from previous multiple branches. Specifically, we begin by applying normalization and multiple $3\times 3$ convolutional layers to refine the local facial context. Afterward, the hourglass block further integrates multi-scale information to capture global and local relationships. The above operations can be expressed as:

where $\mathop{F}\nolimits_{frm}(\cdot)$ indicates feature refinement module.

Inspired by SKNet [36] and LSKNet [37], as shown in Fig. 2, we design a SKAF module to make our model with the ability to select local and global features required for reconstruction for fusion interaction. Specifically, given global feature $\mathop{X}\nolimits_{1}\in\mathbb{R}^{C\times H\times W}$ obtained by the SA and local features $\mathop{X}\nolimits_{2}\in\mathbb{R}^{C\times H\times W}$ obtained by the RDFE, we first fuse the local and global features extracted by a $5\times 5$ convolution and a $7\times 7$ convolution to get a hybrid feature $X$. This operation can be expressed as:

where $\mathop{F}\nolimits_{conv5}(\cdot)$ represents ${5\times 5}$ convolution, $\mathop{F}\nolimits_{conv7}(\cdot)$ represents ${7\times 7}$ convolution, $\mathop{H}\nolimits_{cat}(\cdot)$ indicates the concat operation along the channel dimension. Then, we impose pooling to learn the weight of obtained hybrid features, where the weight reflects the importance of features under different receptive fields. The process of obtaining the weight for selecting required facial features is as follows:

where $\mathop{H}\nolimits_{avp}(\cdot)$ indicates the average pooling operation, $\mathop{H}\nolimits_{map}(\cdot)$ indicates the max pooling operation. Finally, we multiply the weights obtained from the above calculations with the local and global features, respectively. Thus, our SKAF can adaptively select the important local and global information required for reconstruction. The process of obtaining important local and global features $X^{{}^{\prime}}$, $X^{{}^{\prime\prime}}$ by adaptive weight selection can be expressed as:

where $\mathop{H}\nolimits_{cs}(\cdot)$ indicates the feature separation operation along the channel dimension. Through the above operators, we can get the adaptive selected local and global features.

Pixel-level loss is used to reduce the pixel difference between the SR and HR images. This loss is expressed as:

where $G$ indicates the AMIGAN generator.

To enhance the visual quality of super-resolution images, we apply perceptual loss. This involves using a pre-trained VGG19 [40] model to extract facial features from both the HR images and our generated FSR images. Then, we compare the obtained perceptual features of HR and FSR images to constrain the generation of FSR features. Therefore, the perceptual loss can be described as:

where $f_{VGG}^{l}$ represents the feature map from the $l$-th layer of the VGG network, $L_{VGG}$ is the total number of layers in VGG, and $M_{VGG}^{l}$ indicates the quantity of elements within that feature map.

GANs have been shown to be effective in reconstructing photorealistic images [38, 39]. GAN generates FSR results through the generator while using the discriminator to distinguish between ground truth and FSR results, which ultimately enables the generator to generate realistic FSR results in the process of constant confrontation. This process is:

additionally, the generator tries to minimize:

thus, AMIGAN is refined by minimizing the following total objective function:

where $\lambda_{pix}$, $\lambda_{pcp}$, and $\lambda_{adv}$ represent the weighting factors for the corresponding pixel loss, perceptual loss, and adversarial loss, respectively.

LGFI is proposed to extract local features and global relationships of images, which represents a new attempt to interact with local and global information. To verify the reasonableness of our design of LGFI, as shown in Table I, we design four ablation models. The first model removes the SA, labeled “LGFI w/o SA”. The second model removes RDFE, labeled as “LGFI w/o RDFE”. The third model removes SKAF, labeled as “LGFI w/o SKAF”. We have the following observations: (a) introducing SA, and RDFE alone can improve model performance. This is because the above two modules can capture local and global features to promote facial feature reconstruction, including facial details and overall overall contours; (b) Model performance has been significantly increased by introducing the SKAF to capture the relationship between local and global facial features. This is because our SKAF can promote interaction between our SA and RDFE, integrating richer information and providing supplementary information for the final FSR image reconstruction. In our LGFI, using only one module can not achieve optimal results, validating the effectiveness of LGFI.

As shown in Figure 4, LGFI uses a dual branch structure to interact with representing the local and global features. In contrast, the traditional Transformer in Restormer [55] uses a serial structure to link the local and global features. To verify the effectiveness of LGFI, we replace all LGFIs in the network with Transformers and conduct comparative experiments with similar parameters between the two models. From Table II, we can see that the network’s performance using LGFI is better when the two networks maintain similar parameters. This is because LGFI utilizes the features of both local and global branches for interaction, facilitating the communication of multi-scale facial information.

The feed-forward network (FFN) performs independent nonlinear transformations of the inputs at each position to help the Transformer capture local features, but it lacks the ability to extract multi-scale features, which is not favorable for accurate FSR. In contrast, our RDFE can extract multi-scale local features well. To compare RDFE and FFN, we replace RDFE with FFN while keeping the parameters of the two models similar. As shown in Table III, since FFN’s ability to capture feature interactions is limited compared to our RDFE that utilizes multiple branches to capture different receptive field facial features, our RDFE performs much better than FFN with similar computational consumption.

In RDFE, a three-branch network guided by an attention mechanism is used for deep feature extraction, and the feature refinement module is used to enrich feature representation. To verify the effectiveness of RDFE, we conduct multiple ablation experiments. We design five improved models. The first model adopts a single branch structure of ${3\times 3}$ depthwise convolution, labeled as “Single path (${3\times 3}$ dw)”. The second model adopts a single branch structure of ${5\times 5}$ depthwise convolution, labeled as “Single path (${5\times 5}$ dw)”. The third model adopts a single branch structure of ${7\times 7}$ depthwise convolution, labeled as “Single path (${7\times 7}$ dw)”. The fourth model removes attention units labeled as ”w/o AU”. The fifth model removes the feature refinement module, labeled as ”w/o FRM”. From the Table IV, we have the following observations: (a) By comparing the first three rows and the last row of the table, it can be seen that multi-scale branching facilitates the model’s performance due to its ability to extract face features at different levels; (b) From the comparison between the second and the last rows of the table and the last row, it can be seen that using attention units (AU) to guide three-branch feature extraction can enable the model to adaptively allocate weights, enhance the representation of important facial information, and thus improve model performance; (c) From the last two rows of the table, we can conclude that the feature refinement module (FRM) module can further integrate multi-scale information, refine multi-scale fusion features, and thus improve performance.

SKAF is an important component of LGFI, facilitating information exchange between local and global branches. We perform a series of ablation experiments to validate the impact of our SKAF module and assess the practicality of the combined approach. Since SKAF consists of dual branch convolutional layers, maximum pooling layers, and average pooling layers, we verify the effectiveness of module components in SKAF. From Table V, we have the following observations: (a) From the last three rows of the table, we find that using a single pooling branch results in reduced performance, while using average pooling alone results in lower performance than using maximum pooling alone. This is because the salient features of the face are the key to facial recovery, with maximum pooling focusing on salient facial feature information. In contrast, average pooling focuses on the overall information of the face. (b) Compared to the third and fifth rows,, it can be concluded that using both ${5\times 5}$ and ${7\times 7}$ simultaneously can improve performance and fully utilize key facial information under different receptive fields.

This section presents a set of experiments to validate the effectiveness of our EDFF, a module tailored for fusing multi-scale features. We add EDFF to SPARNet [17], which uses EDFF to connect the encoding and decoding stages in SPARNet and send them to the next decoding stage. Additionally, we add EDFF to SFMNet [20], and the specific operation is the same as in SPARNet. From the results of Table VI, we can see that although the parameters of both models increase slightly, the performance of the models improves, which precisely proves that EDFF is helpful for feature fusion in encoding and decoding stages.

We conduct a quantitative comparison of AMINet against existing FSR methods on the CelebA test set, as detailed in Table VII. Our AMINet outperforms all other evaluation metrics, including PSNR, SSIM, LPIPS, and VIF, which fully demonstrates its efficiency. This strongly validates the effectiveness of AMINet. Additionally, the visual comparison in Fig. 5 reveals that previous FSR methods struggled to accurately reproduce facial features like the eyes and mouth. In contrast, AMINet excels at preserving the facial structure and producing more precise results, proving its effectiveness.

We evaluate our method on the Helen test set to further assess AMINet’s versatility. Table VII provides a quantitative comparison of $\times$8 FSR results about it, where AMINet achieves the better performance. Visual comparisons in Fig. 6 indicate that existing FSR methods struggle to maintain accuracy, leading to blurred shapes and a loss of facial details. In contrast, AMINet successfully preserves facial contours and details, reinforcing its effectiveness and adaptability across different datasets.

We present AMIGAN as an innovative approach to bolster the visual fidelity of image restoration tasks. To substantiate its superiority, we have conducted a rigorous comparison of AMIGAN against state-of-the-art GAN-based methodologies, namely FSRGAN [12], DICGAN [14], SPARGAN [17], and SFMGAN [20]. As a complementary assessment metric, we introduce the FID [46] to quantitatively evaluate the GANs’ performance. The outcomes presented in Table VIII, derived from tests on the Helen dataset, reveal that AMIGAN outpaces its competitors considerably. Furthermore, the visual inspection illustrated in Fig. 7 underscores AMIGAN’s exceptional capabilities. Unlike existing FSR methods, which exhibit visible artifacts in generated facial images, AMIGAN meticulously restores critical facial features and intricate texture details around the mouth and nose, which underscores AMIGAN’s prowess in facial texture restoration, resulting in a notable enhancement in clarity and overall visual realism.

All the above comparisons are tested on synthetic test sets, which fail to simulate real-world scenarios accurately. To further evaluate our model’s performance in real-world conditions, we also conduct experiments using low-quality face images from the SCface dataset [42]. As shown in Fig. 8, we visually compare reconstruction results. From this figure, We find that the reconstruction results of face prior-based methods are not satisfactory. The challenge lies in accurately estimating priors from real-world LR facial images. Incorrect prior information can lead to misleading guidance during the reconstruction process. In contrast, our AMINet can restore clearer face details and faithful face structures. This result fully demonstrates our method’s effectiveness in real scenarios.