High-Fidelity Image Inpainting with
GAN Inversion

Our architecture mainly consists of three components: (i) A feature pyramid-based [19] encoder $E$ that extracts input images and provides hierarchical reconstructed RGB images, (ii) the mapping networks with pre-modulation module, and (iii) a StyleGAN2 generator that takes in the style vectors as well as the input image $\mathbf{I}_{m}$ to generate a image. The details of InvertFill are shown in Fig. 2.

Specifically, we attach three RGB heads to the encoder $E$ for generating reconstructed RGB images $\mathbf{O}_{E}^{r}=\{\mathbf{O}_{E}^{1},\mathbf{O}_{E}^{2},\mathbf{O}_{E}^{3}\}$ in correspondence to three different scale. We follow the map2style [28] for the mapping network and cut down the network number from $18$ to $3$, each of which corresponds to the disentanglement level of image representation (i.e., coarse, middle and fine [13, 28]). Three map2style networks encode the output feature map of the encoder into the intermediate code $\mathbf{w}^{\prime}\in\mathbb{R}^{3\times 512}$. Similarly, we replicate map2style as map2structure to project reconstructed RGB images $\mathbf{O}_{E}^{r}$ gradually into structure vector $\mathbf{S}_{r}=\{\mathbf{S}_{1},\mathbf{S}_{2},\mathbf{S}_{3}\}$.

Before executing the style modulation in the generator, we perform $L$ pre-modulation networks to project the semantic structure $\mathbf{S}_{r}$ into the style vector $\mathbf{w}^{*}$ in latent space $\mathcal{F}\&\mathcal{W}^{+}$, i.e., $\mathbf{w}^{*}=E(\mathbf{I}_{m}),\mathbf{w}^{*}\in\mathbb{R}^{L\times 512}$. $L=\log_{2}(s)\cdot 2-2$ denotes number of style-modulation layers of StyleGAN2 generator, and is adjusted by the image resolution $s$ on the generator side. As Fig. 2(c) demonstrates, we adopt Instance Normalization (IN) [33] to regularize the $\mathbf{w}^{\prime}$ latent code, then carry out denormalization according to multi-scale structure vector $\mathbf{S}_{r}$,

where $l\in\{1,2,\dots,L\}$ denotes the index of style vectors, $r\in[1,3]$ indicates three vectors $\mathbf{w}^{\prime}$ correspond to level of coarse to fine, $(\gamma,\beta)$ is a pair of the affine transformation parameters learned by networks shown in Fig. 2(c). Different than previous methods in only using intermediate latent code from a network, the proposed pre-modulation module is a lightweight network and novel in applying more discriminative multi-scale features to help latent code perceive uncorrupted prior and better guide image generation.

The GAN is initially fed with a stochastic vector $z\in\mathcal{Z}$, and previous works [1, 6, 28, 50] invert the source images into the intermediate latent space $\mathcal{W}$ or $\mathcal{W}^{+}$, which is a less entangled representation than latent space $\mathcal{Z}$. The style vectors $w\in\mathcal{W}$ or $w^{+}\in\mathcal{W}^{+}$ are sent to the style-modulation layers of pre-trained StyleGAN2 to synthesize target images. These approaches can be formulated mathematically as follows,

where $E(\cdot)$ and $G(\cdot)$ represent the encoder that maps source images into latent space and the pre-trained GAN generator, respectively.

Nevertheless, the above formulation in Equ. (2) may encounter the “gapping” issue in image translation tasks with hard constraint, e.g., image inpainting. The hard constraint requires that parts of the source and recovered image remain the same. We formally defined the hard constraint in image inpainting as $\mathbf{I}\odot(1-\mathbf{M})\equiv\mathbf{O}\odot(1-\mathbf{M})$. Intuitively, we argue that the “gapping” issue is caused by that the GAN model cannot directly access pixels of the input image but the intermediate latent code. To avoid the semantic inconsistency and color discrepancy caused by this problem, we utilize the corrupted image $\mathbf{I}_{m}$ as one of the inputs to assist with the GAN generator inspired by skip connection of U-Net [29]. In detail, $\mathbf{I}_{m}$ is fed into the RGB branch as shown in Fig. 2(b), the feature map between RGB branch and the generator are connected by element-wise addition. Hence, the previous formulation in Equ. (2) is updated as:

Pixels closer to the mask boundary are more accessible to inpainting, but conversely the model is hard to predict specific content missing. We find that the encoder learns a trick to averaging textures to reconstruct the region away from unmasked region. It causes blurring or mosaic in some areas of the output image, mainly located away from the mask borders, as shown in Fig 7. Drawing inspiration from L2 regularization and motivated by the intuition that fitting diverse domains works better than fitting a preset static domain, a feasible solution is to make style code $\mathbf{w}^{*}$ be bounded by the mean latent code of pre-trained GAN.

The mean latent code is obtained from abundant random samples that restrict the encoder outputs to the average style hence lossy the diversity of output distribution of encoder. In addition, it introduces additional hyperparameters and a static mean latent code that requires loading when training the model.

We adopt dynamic mean latent code instead of static one by stochastically fluctuating the mean latent code while training. Further, we smooth the effect of fluctuating variance for convergence inspired by a reinforcement learning [18]. For initialization, target mean latent code $\mathbf{\overline{w}}_{t}$ and online mean latent code $\mathbf{\overline{w}}_{o}$ are sampled. $\overline{\mathbf{w}}_{o}$ is used in image generation instead of $\overline{\mathbf{w}}_{t}$, which is fixed until $\overline{\mathbf{w}}_{o}=\overline{\mathbf{w}}_{t}$ and then resampled. Between two successive sampled mean latent codes, $\overline{\mathbf{w}}_{o}$ is updated by $\mathbf{\overline{w}}_{o}\leftarrow\tau\mathbf{\overline{w}}_{o}+(1-\tau)\mathbf{\overline{w}}_{t}$ per iteration during training, where $\tau$ denotes updating factor and $\mathbf{\overline{w}}_{t}$ for soft updating target mean latent code. The soft-update mean latent degraded to static mean latent [28] when the parameter $\tau$ of soft-update mean latent approaching zero.

Following prior work in inpainting [20, 16], our architecture is supervised by regular inpainting loss $\mathcal{L}_{\text{ipt}}$, which consists of the pixel-wise Euclidean norm of valid and hole regions, the perceptual loss perc, the style loss style, and the total variation loss tv:

where all above distance are calculated between $\mathbf{I}$ and $\mathbf{O}_{G}$. $\mathcal{L}_{\text{valid}}$ and $\mathcal{L}_{\text{hole}}$ are $\ell_{1}$ norm on the known and masked region respectively. The perceptual loss $\mathcal{L}_{\text{perc}}$ and the style loss $\mathcal{L}_{\text{style}}$ are based on a pre-trained VGG-16 network. More details can be found in [16].

To directly optimize our encoder, the multi-scale reconstruction loss $\mathcal{L}_{\text{msr}}$ is utilized to penalize the deviation of $\mathbf{O}_{E}^{r}$ at each scale:

where $\mathcal{L}_{\text{rec}}$ is represented as mean-squared loss between $\mathbf{I}$ and $\mathbf{O}_{E}$. The multi-scale reconstruction loss $\mathcal{L}_{\mathrm{msr}}$ contains three different losses including perceptual ($\mathcal{L}_{\mathrm{perc}}^{r}$) [4], style ($\mathcal{L}_{\mathrm{style}}^{r}$) [20] and mean-square ($\mathcal{L}_{\mathrm{rec}}^{r}$) losses. The role of $\mathcal{L}_{\mathrm{msr}}$ is to supervise the generated image from decoder and make final generation close to the original image.

The soft-update mean latent is utilized to prevent the encoder from falling into the trick way. We adopt the following fidelity loss $\mathcal{L}_{\text{fid}}$ for improving the quality and diversity of output images:

The fidelity loss $\mathcal{L}_{\mathrm{fid}}$ is designed as a mean squared loss of style vectors $\mathrm{\mathbf{w}}^{*}$ and online mean latent code $\mathrm{\bar{\mathbf{w}}}_{o}$. Its role is to improve the quality and diversity of the output images.

Overall, the loss of our networks is defined as the weighted sum of the inpainting loss, the multi-scale reconstruction loss, and the fidelity loss.

where $\lambda_{\text{msr}}$ and $\lambda_{\text{fid}}$ are the balancing factors for the multi-scale reconstruction loss and the fidelity loss, respectively.

Datasets.

Experiments for RQ1, RQ2 and RQ4 are conducted on two datasets, Places2 [49] and CelebA-HQ [11]. CelebA-HQ contains 30,000 high-resolution celebrity faces, and we follow [42, 43] to split this dataset for training and testing. Places2 contains real-world photos, including more significant objects, such as streets, cars, houses, which is better suited for verifying models on large-scale masks than CelebA-HQ. Based on the official train/val/test split, we train the model on train plus test about 200,000 images, evaluate the model on first 5,000 images of val. With regard to RQ3, we utilize two datasets Scenery [41] and MetFaces [12]. Scenery dataset is a common benchmark for recent image outpainting tasks and contains 6,040 landscape photographs. We follow [41] to use about 5,000 images as training set and the remaining 1,000 images as test set. MetFaces consists of 1,336 human faces extracted from works of art, and we randomly select 1,000 images as training set and other images as test set. Our model and all baselines adopt the same training and test strategies to ensure experimental fairness.

Evaluation Metrics.

We use three metrics following prior works to measure the quality and fidelity of inpainting results. SSIM [35] modeling image distortion by structure, luminance, and contrast, is a pixel-level objective metric similar to PSNR, and their drawbacks cause inconsistent evaluation results with the human eye. Despite that, they are classical metrics for image evaluation, one of which SSIM we selected for quantitative comparison. FID [9] is a deep metric and closer to human perception. It measures the distribution distance with a pre-trained inception model, which better captures distortions. LPIPS [48] is another learned perceptual metric and commonly used to score the intra-conditioning diversity of models output. Following previous works [16, 26, 43], We calculate these quantitative metrics on original images $\mathbf{I}$ and composition images $\mathbf{I}\odot(1-\mathbf{M})+\mathbf{O}_{G}\odot\mathbf{M}$.

Baselines.

We carefully select baseline methods mainly from two perspectives: UNet style methods and Inversion style methods to demonstrate our approach’s characteristics and superiority. First, for the sake of validating the ability of InvertFill in filling images under large-scale masks, we compare it with the previous approaches including EC [26], GC [42], RFR [16], MEDFE [21], ProFill [47], CTSDG [8] and CRFill [46]. Second, we compare with the latest GAN inversion-based inpainting methods mGANprior [6] and pSp [28].

We utilize eight A100 GPUs for pre-training the GAN generator, and one TITAN RTX GPU for optimizing the encoder and other experiments. Following [16], we scale the image size of all datasets to $256\times 256$ as the input. In the light of the mask coverage, we classify the test masks into three difficulty levels: Hard/Extreme/All, indicates the mask with coverage ratio of 50% $\sim$ 60%, 70% $\sim$ 90%, 10% $\sim$ 90%, respectively. During testing, for a fair comparison, we use the same image-mask pair for all approaches. More details of implementation are shown in the supplementary.

RQ1.

We reproduce all the above baselines by utilizing their official implementations. Concerning Places2 dataset, we utilize the pre-trained weights officially released by the baselines. On CelebA-HQ dataset, EC [26], GC [43], mGANprior [6] offer pre-trained weights, we thereby carefully retrain other baselines through the official source codes. Because ProFill only offers Web API on Places2, we use placeholder ‘-’ in Table 1 for ProFill on CelebA-HQ.

From Table 1, our method achieves the best or comparable performance among advanced inpainting approaches. In terms of the FID metric, our method at most produces a notable margin of 54.60% and 40.14% on Places2 and CelebA-HQ datasets, respectively. And our method also outperforms the second-best approach 11.2% and 10.4% improvements on another perceptual metric LPIPS, which validates the superiority of our design.

Fig. 3 and 4 provide several visual inpainting results on Places2 and CelebA-HQ datasets. Fig. 3 reveals that the prior works still struggle to generate refined texture if the input image with large corruptions, while our approach has been able to create semantically rich objects such as windows, towers, and woods. In Fig. 4, mGANprior [6] progressively erases the color discrepancy rely on optimized-based inversion but is unable to bypass semantic inconsistency. The encoder-based inversion method pSp [28] could synthesize realistic pixels for corrupted regions based on the well-trained model, though it still has not resolved the “gapping” issue. The results indicate that our method produces consistent output while generating high-fidelity texture compared to existing methods.

RQ2.

The “gapping” causes color discrepancy and semantic inconsistency, and we are counting on image post-processing to tackle this issue at the beginning of this study. Specifically, we adopt image blending [36], which is effective in eliminating the color discrepancy but helpful in remedying semantic inconsistency.

To further demonstrate the superiority of our method, we construct the pSp+Blend variant that introduces an image blending [36] method after generating output images. In Fig. 6, the first row shows the distinct gap at the stitching boundary in pSp output, and pSp+Blend fixes this color discrepancy problem. Even so, the second row shows pSp+Blend unable to assist with semantic inconsistency problem given the glasses are still incomplete. Compared with the vanilla pSp and pSp+Blend, output images of our approach no longer suffer from color discrepancy or semantic inconsistency.

We conduct a comparison experiment on CelebA-HQ dataset with the Extreme level masks. As demonstrated in Table 3, our method performs better than a recent diffusion-based approach Score-SDE [32] w.r.t to FID, LPIPS and SSIM metrics. The results in Table 3 also show that our method performs best among the existing inversion-based inpainting approaches after resolving the “gapping” issue. Notably, our method does not require any image post-processing.

RQ3.

Concerning validating that our approach can reuse the pre-trained GAN generator as priors to tackle image from unseen domain, we conduct two extended tests that introduced images or masks from unseen domains and only required optimizing the lightweight encoder. The first is archaic photograph inpainting, and we use MetFaces [12] for optimizing the encoder, and remain the pre-trained weights of GAN generator of CelebA-HQ dataset. For the second one, we perform our approach with outpainting masks [15] on Scenery dataset. Similarly, the generator did not retrain on the Scenery dataset rather than remaining the weights for Places2.

The 1st row of Fig. 5 shows the inpainting results of archaic photograph inpainting. It demonstrates that our method enables the generator to synthesize semantically consistent style and patches, even in an unseen domain. From the 2nd row of Fig. 5, the outpainting results on the Scenery dataset show our approach still can synthesize realistic texture and significant objects, e.g.trees, mountains. To ensure the masks are unseen for the GAN generator, we only use the outpainting masks to train the encoder, not the GAN generator.

Furthermore, we quantitatively compare mainstream outpainting approaches as well as adopt RFR [16] and pSp [28] as additional baselines. As shown in Table 3, our model considerably outperforms the best outpainting baselines [15, 41] with respect to FID, LPIPS, and SSIM. Similarly, we conduct experiments compared with inpainting baselines on Metfaces, as show in Table 4. In summary, the results indicate that our proposed method is robust and extends to other tasks with out-of-domain inputs.

Due to limited space, please kindly refer to the supplementary material for more results.

The ablation experiments are carried out on the Places2 dataset under the Extreme mask setting. In Table 5, we construct three variants to verify the contribution of proposed modules, in which PM and SML denote pre-modulation and soft-update mean latent. By learning from these modules, our method considerably outperforms the most naive variant w.r.t FID, LPIPS, SSIM, and PSNR.

The soft-update mean latent is motivated by the intuition that fitting diverse domains works better than fitting a preset static domain, especially when the training dataset contains various scenarios such as street and landscape. As shown in Fig. 7, when we use SML code that dynamically fluctuates during training, the masked region far away from the mask border tends to be reconstructed by explicitly learned semantics instead of repetitive patterns. Notably, ‘w/o SML’ represents using regular static mean latent code.

Fig. 8 shows two failure cases. Even if the model can recognize the corrupted objects (our method tends to recover the human face in the left case of Fig. 8), it mistakenly locates them and produces severe artifacts. When lacking sufficient prior knowledge, our method fails to reconstruct details. This demonstrates that these situations are challenging for image inpainting and need further study.