V-LinkNet: Learning Contextual Inpainting Across Latent Space of Generative Adversarial Network

The main categories of traditional methods [9] are summarised as:

• •

  Diffusion-based approach [2]. It transmits structural information from boundary areas into the interior, are among the categories of traditional inpainting techniques [9]. Techniques in this category, on the other hand, produce blurry artefacts on large textured missing patches, which is undesirable.

• •

  Exemplar-based methods [1, 3]. It uses similar patch searching techniques to fill in missing regions. Methods in this category attempt to address the limitation of diffusion methods on large textured regions, but still fail to match exact content and sometimes suffer with misalignment due to patch overlap, such as PatchMatch [3]. In addition, they are usually computationally expensive and time consuming with unrealistic results for large image-to-hole ratio with an arbitrary mask.

• •

  Hybrid methods. It uses both diffusion and exemplar-based methods to address misalignment by adding blurry effect in the boundary areas of the target region. Despite the success in producing textural features for a missing region, there are still issues with computation and aberrations persist. Nonetheless, the failure to capture high-level image features and the inability to generate complex and non-repetitive structures [26, 27] continue to be problematic.

Reconstruction of complicated textural regions such as faces was also a challenge for traditional techniques. However, despite the reasonable results obtained with other natural scene images, the limited amount of high-level information accessible during computation has resulted in techniques in this category failing to produce high-quality features with believable semantic structures in natural scene images. Traditional inpainted images frequently exhibit broken or unconnected edges along border regions, blurry artefacts, and overlapping patches along seam regions. Additionally, the inpainting process for techniques in this category is rather computationally intensive.

GAN-based methods make use of large-scale data to facilitate hallucination and the extraction of high-fidelity features within CNN blocks for image inpainting. GAN models for image inpainting [21, 28, 29] have used multi-columns to encode and propagated features directly to the decoder or use a self-supervised Siamese style inference approach [22], where a style encoder is the supervisor of the generator, to improve feature extraction and learning. Other methods [30, 31, 19] observed that failures in feature extraction and propagation could be due to the irregular holes. To address the limitation, Liu et al. [12] proposed an independent mask updating with partial convolutions to specifically target missing regions. Yu et al. [31] proposed to use gated convolutions to gear the model towards learning soft mask of the irregular hole regions. More recently, Jam et al. [19] proposed a reverse mask mechanism to specifically target missing regions whilst preserving the visible ones using a spatial preserving operation.

Alternative approaches are two-stage models [32, 33], where a coarse version is generated at stage 1, and then used as the input to a refinement network at stage 2. The issue with this approach is inadequate information during reconstruction, due to larger target pixel region, hence a poor input for decoration at stage 2. It is still a challenge to reconstruct high-dimensional distribution from natural scenes than from aligned faces with no visible aberrations. A couple of reasons could be failures in feature propagation techniques or lack of refinement mechanisms in deeper layers to capture high-resolution feature maps. To address the aforementioned limitations, Pathak et al. [10] proposed a channel-wise convolution layer for feature propagation but the drawback is high computation and inefficient transfer of feature maps.

Attention mechanisms [35, 13, 36, 26] have also been considered in deeper layers or as transition between the encoder and decoder. It is noted that a bottleneck [37, 38, 39] or a feature transfer mechanism like attention layers within deep layers of convolutions is often required when inpainting high-resolution images. Due to high-resolution images, convolutional outputs required large amount of GPU memory, thus resulting to an increase training time [10, 40, 41]. Additionally, when extracting features from high-resolution images, some features may be lost during the operation. However, more detailed information about low-level features, such as edges, is frequently captured within the first few layers of the convolution. As a result, failure to consider prior semantic distributions leads in unusual textures on the generated image. One limitation of attention mechanism is that it increases computational cost [13] and does not generalise well in feature propagation. Liu et al. [15] considered pixel consistency and proposed a module that searches previous patches to extract relevant features. Thus feature extraction and propagation are important factors to consider at the design stages of the network. Although these are actively being considered, there are still gaps for new approaches. One reason is the limitation in information dissemination of high-level features caused by the design of attention layers.

When it comes to image inpainting, the Sobel operator is not a new technique [50]. Because images include noise, which may induce a rapid change in pixel values [42], the Sobel algorithm [51] is capable of extracting occlusion boundaries. When employing the Sobel operator for edge detection, noise may be subdued without losing edges, edges can be improved by applying a high pass filter, and spurious edges, which are caused by noise, can be eliminated (edge localisation). Sadowski et al. [50] employed the Sobel operation to collect gradient information from generated and ground-truth images in order to construct a loss function. With the help of edge information, Zhang et al. [52] was able to obtain gradient features that were later fused with image features to obtain the final image. They used a masked gradient map and mask to enable the network to obtain gradient features, which were later fused with image features to obtain a final image. Several techniques have been developed in an attempt to apply structural restrictions to the inpainting tasks, such as two-stage networks, instance images, and matching completion images. Reconstructed images, not with standing their successes, fall short of capturing high-level feature information of the target regions. This problem continues to be difficult, and there is still much opportunity for progress in this field of research.

Previous works [10, 11, 12, 35] in computer vision have shown that inpainting is a learning problem that can be solved by encoding high-level features. The reconstructed output is geared towards having a close similarity to the input. We consider the inpainting task to have an input source $M_{I}=I\odot M$ and a target image $I$, where $M$ is a binary mask and $\odot$ is element-wise multiplication. The proposed V-LinkNet is a neural network generator with dual encoders of differing weights, a recursive residual transition layer (RSTL), and a decoder. It utilises a global and local Wasserstein discriminator to build a generative model. We have included descriptions of the proposed technique as shown on (Figure 3) with the RSTL for more clarity. Additionally, we explain the training procedure that occurs to optimise the training loss functions.

Our proposed V-LinkNet is a generative model consisting of a generator, a global discriminator and a local discriminator as shown in Figure 3. The discriminators are included for adversarial training. Only the generator network is used during the testing phase.

The generator $G_{\theta}$ has dual encoder branches ($E_{\theta_{A}}(\cdot)$ and $E_{\theta_{B}}(\cdot)$) and a decoder ($D_{\theta_{E}}(\cdot)$). Within $G_{\theta}$, encoder branch $E_{\theta_{A}}(\cdot)$ focuses on the capturing contextual information covered by the masked (unknown) regions. To ensure the reconstructed image is visually coherent with the structure and context of the ground-truth, we design $E_{\theta_{B}}(\cdot)$ to capture encoding with main focus on perceptual and structural information.

Both encoders $E_{\theta_{A}}(\cdot)$ and $E_{\theta_{B}}(\cdot)$ have eight convolution blocks, each with variations in spatial resolution and receptive fields at dilation rates of 2, 4, 8, 16. $E_{\theta_{A}}(\cdot)$ has dropout layers with value $0.2$ after each convolution block to reinforce learning. Blocks one to five, have batch normalization and Exponential Linear Unit (ELU) activation followed by maxpooling layer, while blocks six to eight has ELU and dropout. Within the decoder $D_{\theta_{E}}(\cdot)$, are learnable upsampling layers using bilinear interpolation each with a convolution block that includes batch normalization and ELU activation layers. The final convolution block of the encoder-decoder (generator) has a Tanh activation layer with no batch normalization layer, which is deliberate so as to accelerate training and stabilize learning. The output of the final layer $I_{pred}=D_{\theta_{E}}(g_{\theta})$ and the generator output as $G_{\theta}(I_{pred})$. The final output a generated image based on nonlinear weighted upsampling in latent space.

We train the V-LinkNet on the training set of $I$ and use high-level features within both encoders to minimise the error. Midway between the paired encoders, we modify the convolution block by increasing the dilation rate to 8 and 16. Both encoders $E_{\theta_{A}}(\cdot)$ and $E_{\theta_{B}}(\cdot)$ learn high-level features to obtain output features $E_{\theta_{A}}(\phi)$ and $E_{\theta_{B}}(\phi)$, which are passed into a RSTL. The RSTL is designed to fuse the features from both encoders to exploit feature information at different scales. The V-LinkNet learns through latent space loss and adversarial loss to reconstruct images with similar pixel values of the target image. V-LinkNet consists of a training and inference (testing) phase. We use the traditional WGAN training, where the training sample are masks and ground-truth images. During training, the network learns with the main objective being to generate an image given the mask and the ground-truth. To minimise the error through back propagation, we design a new loss function that evaluates the training set to minimise the error in order to find high-level matching features between the paired encoders. The details of our proposed loss function will be discussed in Section 4. We project the corrupted input onto the latent space of the generator through iterative backpropagation. Therefore, we reuse the weights of both encoders to compute an objective function that will specifically target valid regions. At each stage of the network training, the weights will assist with fast updating during learning to guide the model.

Residual learning has been well established in deep learning due to their ability to reduce training error in much deeper layers. A simple implementation of a residual block is a fast-forwarded activation layer within the neural network. By adding the activation layer of a previous layer, to a deeper layer within the network, a residual connection is achieved. In previous works [10, 53, 18], feature extraction and propagation often fail with large portions of the background due to low level capture and poor transition to the decoder. We consider maxpooling, an operation that highlights the most present feature of an image patch and calculates its maximum value. Because features encode spatial representation of visible patterns, it is more informative to consider the maximum presence of different features extracted from the image. Hence the reason why maxpooling is considered instead of average pooling in this work.

We design the RSTL, as illustrated in Figure 3, with the aim to capture high-level semantic information. There are two units in the RSTL: a maxpooling residual connected unit and a convolution residual connected unit. The RSTL is formed by residually connecting both units. The idea is to efficiently pool multiple window sizes, combine them using learnable weights and fast-forward to deeper layers to reduce the error during training. As a result, training gradients are obtained from the next connected layer within each layer, and these gradients are used to update the parameters in the current layer. This, in turn, influences the weights of the filters, causing the activation maps to increase or decrease, lowering the loss. We combine the residual connection of the activation maps with the output of the final pooling layer and the input of the residual layer to obtain the unit output feature map. To feed the RSTL, we concatenate the output feature maps of both encoders. The RSTL extracts high-level semantic information from the concatenated input to generate a feature map, which is then passed on to the decoder.

Our proposed connected pooling operation combined with residual connections reduces the error near the boundary regions of the hole regions, as it captures fine contextual information. This unit is designed to predict and delineate any mask residues as it highlights high-level semantic information by recursively performing pooling and convolution operations. The concatenated features when passed via pooling unit suppresses noise to project informative pixels. This is different from using the channel-wise attention which squeezes the spatial dimension of the feature map. The objective of this design is to extract meaningful pixels whilst suppressing uninformative ones before passing them to the decoder. First we concatenate features extracted from $g_{\theta_{A}}(\cdot)$ and $g_{\theta_{B}}(\cdot)$ and pass them through ELU activation and then perform pooling operation followed by a $3\times 3$ convolution and ELU unit as a gating layer. For more refined details we use a dilation rate of 16 within the convolution layer.

where $\sigma$ is the ELU activation function, $F_{3\times 3}$ is the convolution layer. The final feature map is given by:

where $\oplus$ is element-wise addition and lastly a dilated convolution layer to refine and transfer the feature to the decoder.

To optimise the RSTL and dual encoders, we introduce a novel loss function that uses features of both encoders to assist the model during learning. To ensure high-level contextual features for missing regions, we introduce a loss between $E_{\theta_{A}}(\cdot)$ and $E_{\theta_{B}}(\cdot)$. During training, the loss between both encoders ensures ongoing communication, in order to improve the models learning on contextual information. By employing this technique, the model can enhance visual consistency with contextualised features. The loss model is designed based on the Mean Squared Error (MSE) specifically to penalize large errors and provide fast learning.

In image inpainting, known and unknown regions are representations of the masked image, thus applying gradient algorithm to detect occlusion boundaries can prove useful. To determine the direction of filling priority, we target the image gradients of feature maps and use this information to construct a loss model.

We obtain feature gradients of the third convolutional layer of both encoders and compute the loss model. To re-enforce on outer edges and fidelity of the generated image, we utilize gradients of the generated and ground-truth images to assist in the final reconstruction. Note that the edge map based on the gradients are computed in x and y directions.

where $G_{x}$ and $G_{y}$ are gradients computed by depth-wise convolution using the x and y components of the Sobel operator on an image $I_{gt}$.

where $\lambda=0.5$ as coefficient to obtain $L_{edgeLoss}$. We use pixel space L1-norm, based on a range of pixel values with the input image and output image.

where, $K$ is a constant, $\odot$ is the element-wise multiplication, $I_{gt}$ is the ground-truth image and $I_{pred}$ is the predicted image. Further, we utilize the reversed mask loss ($L_{rm}$) from [19] and compute a contextual loss. We want to keep the known pixel locations of the input image by penalizing the predictions thus creating similar pixels based on the reversed mask and masked regions.

The total loss ($L_{T}$) is a weighted sum of all the losses with highest weight applied to $L_{\phi}$.

where $\alpha_{1}=0.5,\alpha_{2}=0.3,\alpha_{3}=0.1$ are coefficients of the weights applied to the loss.

Implementation of the V-LinkNet is done using the Keras library with a Tensorflow backend, and the model is trained on a P6000 GPU computer. We resized our images to $256\times 256\times 3$ and align them with random masks during training and during testing aligned them with appropriate masks. We pretrain the network using a novel loss function that backpropagates gradients using features from both encoders. We use the RSMProp optimizer, with a 0.0005 learning rate. We updated the generator and discriminator networks following pretraining and utilised the Adam optimizer [57] with a learning rate of 1E-4 and a beta of 0.5. The network is trained with a batch size of 5 and for 100 epochs, which takes around three to five days depending on the amount of the training data. After obtaining a well-trained model, we use reverse mask loss and a decreased learning rate (1E-5) to fine-tune it while retaining the original network topology. The input is updated throughout completion using a contextual loss and a perceptual loss with coefficients of 0.4 and 0.6, respectively. Stochastic clipping is employed during back-propagation. We picked a modest value to ensure that contextual loss is prioritised during test-time optimization, and that the inpainted part of the generated image most closely resembles the input background context of the entire image. The generator and discriminator are fixed during back-propagation. We evaluate the performance of V-LinkNet on CelebA-HQ, Paris Street View, and Places2 datasets, which are the most widely used datasets by the state of the art.

A fully trained model can predict missing pixels for image-to-mask ratios ranging from [0.1 to 0.8] during testing. The inference time is between 0.192 seconds to 63 seconds depending on the mask size. During inference, we employ the network design with batch normalisation layers disabled.

The images and the masks are two essential components to train and test the performance of inpainting methods. The following are the most commonly used datasets to evaluate the performance of image inpainting algorithms:

• •

  CelebA-HQ [58]: A dataset curated from CelebA [59] containing 30,000 high quality images of $1024\times 1024$, $512\times 512$, $256\times 256$ and $128\times 128$ resolutions. We followed the state-of-the-art procedures [12] and split our dataset into 27,000 training set and 3,000 testing set.

• •

  Paris Street View [60]: This dataset contains 14,900 training images and a test set of 100 images collected from Paris street views. The main focus of the dataset is the buildings of the city and very important in geo-location task.

• •

  Places2 [61]: A dataset containing over 1 million images from 365 scenery from places. It is suitable for model learning and understanding of diverse complex natural scenery. The following scene categories were chosen: butte, canyon, field, synagogue, tundra, and valley (in that order) as proposed by Yan et al. [35]. In each category, there are 5,000 training images and 900 test images. Our model is trained on the training set and evaluated on the testing set.

Each training image for both Paris Street View and Places2 is resized to $256\times 256$ pixels, which is then used as an input to our model.

To encourage full reproducibility of our work, we introduce a standard protocol for image inpainting testing datasets. The facial test set is labeled according to CelebA-HQ [58] dataset, which contains 3,000 high-resolution face images from CelebA [59]. The facial test set is split randomly according to [12]. These images are paired with 3,000 masks images from Quick-Draw Mask [62] and 3,000 masks images from the Nvidia Mask [12] test dataset, as illustrated in Figure 5. We will share the filenames of the paired image and mask test set in a comma-separated value (CSV) file. Note that our evaluation masks are set to 3,000 mask images and the images and masks used for training are not paired. The proposed standardised test dataset is curated with the facial image pose in mind. We evaluate the difficulty in inpainting task based on pose and variation in mask holes.

The Paris Street View [60, 10] were standardised by Pathak et al. [10], which is available upon request only from the authors. We adopted the Pathak et al.’s protocol for the Paris Street view dataset, which has 100 test images but used our own masks for testing. On the other hand, the Places2 [61] test dataset is extract from Places365-Standard. The categories used are butte, canyon, field, synagogue, tundra and valley. These are the same categories for training and testing. Each training set has 5,000 images, 900 test images and 100 validation images as per [35]. In total, there are a total of 5,400 test images. We paired each images with 5,400 masks and use it as a standard protocol for testing. For follow this due to longer training times and also based on the split by the state-of-the-art [35]. For Places2 and Paris Street View datasets, we run evaluations based on the mask difficulty on their standard test set.

Without bias and dependent on codes availability, we used Pathak et al. [10] (CE), Liu et al. [12] (PC), Yu et al. [31] (GC), and Jam et al. (RMNet) [19] as baseline models. The following summarise the baseline models used as the benchmarks for our standard protocol:

• •

  Context encoder-decoder framework (CE) [10] introduced the channel-wise fully connected layer to solve the convolutional layer limitation associated with failures in direct connection of all locations within a specific feature map. The channel-wise fully connected layer is designed to directly link all activation; thus enabling propagation of information within the activation of a feature map.

• •

  Partial Convolution (PC) [12] proposed partial convolutions with mask updating to enforce learning in irregular hole regions during convolutions and ease feature transfer to subsequent layers, allowing convolution layers to target more of the missing regions as a result.

• •

  Gated Convolution (GC) The authors [31] proposed a gating mechanism that learns soft masks within convolutions to make the transfer of features within convolutions more convenient. This method differs from PC [12] in that the irregular mask is learnt rather than being updated in each step, whereas the former does not have this feature.

• •

  Reverse Masking Network (RMNet) [19] introduced reverse mask mechanism within the network. The reverse mask forces the convolutions to subtract visible regions through the reverse mask mechanism, thus ensuring the output prediction is on the missing regions only.

The figures in this section depict the visual results of the V-LinkNet method. For comparison with the benchmarks and a variation of our model, we show the generated face images in Figure 6. It shows CE struggles with arbitrary hole-to-image mask regions and the generated image is blurry, while PC and GC leave a bit of artefacts (best viewed when zoomed) on the generated image. Focusing on the face and hair regions, our model performs better than the state of the art with no artefacts left on the inpainted regions. However, despite marginally comparable quantitative results on full inpainted images, our model completes and generates the facial image with no visible boundaries of the binary masks as seen on the generated images completed by the state of the art.

The visual comparison of Places2 and Paris Street View datasets best represent how our model can generalise to natural scene images. The generated images are shown on Figure 7 for Places2 [63] while Figure 8 shows the inpainted images generated from Paris Street View dataset [28].

It is important to note that the visual and semantic understanding of the completed regions is critical to the audience when inpainting in the wild. This is because the visual quality of the blending between the inpainted regions and the original unmasked regions should be unnoticeable in real-world scenarios. However, in computer vision, we use quantitative evaluation to track model performance. Based on previous state-of-the-art research, we use the Mean Absolute Error (MAE), Frechet Inception Distance (FID), Peak Signal to Noise Ratio (PSNR), and SSIM to quantify performance against the state of the art ([10, 12, 31]). The high values obtained for MAE and FID show poor performance of the model whereas lower values for these metrics indicate better performance. For clarity, we have included in the table, where $\dagger$ indicates lower is better and $\uplus$ indicates higher is better.

Table I shows the quantitative evaluation for the inpainted images on CelebA-HQ testing set, with the best results in bold. Our proposed method achieved the best FID of 2.76 and SSIM of 0.96 when compared to the baseline models.

In addition, we perform quantitative measures on Places2 and Paris Street View datasets to test the ability of V-LinkNet in other image types. We compare the results to the state of the art [19] and present the findings in Table II. On the Paris Street View dataset, our proposed model outperformed the state of the art [19] with SSIM of 0.95, but achieved marginally comparable result on the Places2 dataset, with SSIM of 0.91. The best results in Table II are highlighted in bold.

We slight modify the RSTL by removing the pooling unit. The modified layer is a residual block with the concept of attention in our inpainting task. We perform $1\times 1$ convolutions on $g_{\theta_{A}}(\phi)$ and $g_{\theta_{B}}(\phi)$ output and concatenate the projected features maps. For dynamic feature selection, a softmax function is utilised on the concatenated feature map. Applying softmax after $1\times 1$ convolutions on each encoder output enables precise feature values, thus preserving local and detailed information.

During the experiment, we use $L_{vgg}$, $L_{edgeLoss}$ combined with $L_{1}$ pixel-wise reconstruction loss. We notice that using the $L_{\phi}$ loss combined with $L_{edgeLoss}$ gets rid of checker-board artefacts on the generated image. We notice that Sobel aids noise reduction and enhances the image quality of the generated output.

This section examines whether residual features from our recursive residual pooling unit has a positive effect on our model. The results in Table III demonstrate that residual refinement has a positive impact on the overall performance of our model. According to our findings, this improvement is attributable to the elimination of low-level information as a result of the pooling units being interconnected residually, which allows direct backpropagation of high-level information throughout the learning process.

This protocol is designed to evaluate the performance on a set of mask and images. The mask ratios in the Masksets range from [0.01,0.6]. The different MaskDataset and ratios are: MaskDataset1 [0.1,0.6], MaskDataset2 [0.01,0.1], MaskDataset3 [0.1,0.3], MaskDataset4 [0.3,0.4], MaskDataset5 [0.5,0.6] and MaskDataset6 [0.1,0.4].

The MaskDataset6 are selected irregular masks that are used as masking method for more than one image (i.e one mask to many). Each mask is evaluated on more than one image and the performance is different across the dataset. The overall results are shown in Table IV. The V-LinkNet has demonstrated overall best performance when presented with mask of various size ranges.

This study is conducted to identify biases for different masks on different images and propose a standard protocol that will propel research in image inpainting. The mask-to-area ratio was determined using OpenCV toolbox. Based on this study, we observed that the performance of an algorithm will very much depend on the mask type and the image type. There are some conditions on a facial image that can influence the performance such as pose, lighting, features and background. In the case of CelebA-HQ dataset, we observed that if the mask is on the skin region, the performance evaluation has better scores compared to when the mask is on the a difficult background with variations in lighting conditions. Furthermore, the mask applied to a face posed at an angle will influence the results as shown in Figure 1 second row, second set of images. Based on this finding, we proposed standardised test sets to support a fairer comparison in future research.