GMFIM: A Generative Mask-guided Facial Image Manipulation Model for Privacy Preservation

There are various methods for concealing the identity of the people who appear in a photo from intelligent algorithms. The very first attempts in this field were simple techniques for detecting and blurring or blacking the face regions in an image. The first drawback of these techniques is that the resulted images are not visually appealing to human eyes and cannot be shared on social networks. The second drawback is that the obfuscation methods like blurring and mosaicking cannot effectively protect the identity. In [12], a deep neural network is trained that successfully identifies the faces which have been hided with the aforementioned techniques. However, recently these models have been utilized for preserving the privacy of the people in public image datasets [13, 3].

The second category of de-identification methods unlike the first category changes the faces in an image in such a way that the image still appears normal to human eyes but the faces in the output image are quite different from those of the input image. For example, Sun et al. propose a model which firstly detects or generates the facial landmarks of the input image and secondly, takes the blackhead image and the set of landmarks as input and generates the inpainted image [4] which is a new face with the same facial landmarks and background. AnonymousNet [5] is another model of this category which is a four-stage framework that uses a deep convolutional neural network to sequentially extract face semantics, alter the values of some facial attributes, generate a new face image employing a face generator, and finally perturb the generated image by adding adversarial noise. Also, Pautov et al. propose a different idea which can be considered as a physical attack to face recognition systems [14]. In this method, an adversarial patch is designed, printed and then added to the face of the target person. The face of this person along with the attached adversarial sticker is photographed and the photo is then fed into face recognition systems to measure the success rate of the proposed framework.

As stated in [6], one of the most important requirements of a face de-identification method is that the produced image be perceptually realistic-enough so that a human observer will recognize it as a legitimate face image compared to the set of input images. On the other hand, some applications call for the preservation of the identity for the human observers and hiding the identity from the automatic face recognition systems. The methods of the third de-identification category try to simultaneously address all of these requirements. For example, Oh et al. [7] propose a general game-theoretical framework for this purpose in which a user is playing against a recognizer; the user changes the input image based on a strategy (an adversarial image perturbation) for concealing the identity and the recognizer processes the generated image based on a different strategy for neutralizing the changes made by the user. The optimal strategy of the user is then used for identity protection. Wu et al. [8] propose a GAN-based framework, called PP-GAN, in which the generator part is trained for transforming the input face image to a similar image with hided identity. In this model, first, a verification model is trained to determine whether two face images belong to the same person or not. They also use the pixel level structure similarity (SSIM) to make the generated image as similar as possible to the input image. The loss of the verification model and the SSIM loss are used to train the generator, and the final generator is used as the model for de-identify the input images. In a more recent work, Deb at al. proposed AdvFaces [6] which is based on creating image-dependent adversarial masks for hiding identity. AdvFaces is also a GAN-based model consisting of a generator, a discriminator, and a face matcher; the generator creates an adversarial additive mask for the input image, the face matcher is used to remove the identity from the generated image, and the discriminator guides the model to generate perceptually realistic images. Finally, Shan et al. have developed Fawkes [9] for creating perturbations that added to a face image, conceal the identity from the intelligent algorithms. Fawkes seeks to find minimal perturbations that drastically shift the feature vector of the generated image in the feature space.

Generative Adversarial Networks, initially proposed in [10], are a class of machine learning frameworks consisting of two main components: the generator and the discriminator. The generator G captures the data distribution and the discriminator estimates the probability that a sample come from the real training data rather than G. The training goal of G is to maximize the probability of D making a mistake. This framework corresponds to a two-player minimax game.

GANs have been used in a wide variety of image applications such as image in-painting [15, 16], image manipulation [17, 18], object detection [19, 20], 3D image synthesis [21, 22], and image-to-image translation [23, 24].

One of the most important applications of GANs is to generate high-quality images. For example, PGGAN [25] is a high quality image generator whose main idea is progressive growing of the generator and discriminator. So, it starts with a low resolution image and increases the resolution by adding new layers so that the generated images have more fine details as the training progresses. As stated in [26], despite the fact that PGGAN generates high-quality images, its ability to control specific features of the generated image is limited. StyleGAN is an upgraded version of PGGAN that modifies the architecture of the generator network and uses a mapping network to convert the input latent noise to the intermediate latent style vector. Adaptive Instance Normalization (AdaIN) is used to create styles and manage the layers of the synthesis network. In [27], several characteristic artifacts of StyleGAN are analyzed, and several model architectures and training process adjustments are offered to overcome these issues.

As our goal is to de-identify the input image, it is necessary and sufficient just to alter the face part of the image. Therefore, in the first step of the propose model, the face part of the input image is extracted and the background is omitted. This will help the optimization module to focus only on the face part of the image for both reconstruction and de-identification goals.

The proposed module takes an input image and produces an image with a black background. As shown in Figure 1 (a), firstly, the facial landmarks are extracted from the input image with the help of the face and landmark detector algorithm proposed in [28]. Then, a rectangle box is formed around the face using the set of extracted landmarks. The rest of the image, regarded as background, is then removed (filled with black color).

The core component of GMFIM is the optimization module in which the de-identification task is formulated as an optimization problem and an iterative gradient descent algorithm is adopted to find the best solutions.

As illustrated in Figure 1 (b), the optimization module is based on a face generator that synthesizes realistic face images from random latent vectors. Here, the goal is to find a latent vector that is mapped to a face image with two required characteristics: 1) it perceptually looks like the face image extracted in the first step, and 2) the automatic face recognition systems cannot match it to the true subject. To achieve these goals we have defined two loss functions described in the following. The final output of this step is the face image that best minimizes these losses.

Regarding the face generator, we have utilized the pre-trained StyleGAN2 [27] for this part of our proposed model. However, our proposed model is a general framework and any other face generator can be used in this section without any changes in the model.

Perceptual loss: As stated before, the output image of this step should be perceptually realistic and similar to the input image but not exactly identical to it. Traditionally, the low-level similarity between two images has been measured in pixel space, but in recent years, this has changed [29]. Gatys et al. observed [30, 31] that the learned filters of the VGG image classification model [32] are very good general-purpose feature extractors for this purpose. So, we use the VGG16 model [33] to extract feature vectors from the masked input image and the generated image and then use these feature vectors to calculate the perceptual loss function of our model as:

where the term $F_{P}(x)$ represents the feature vector of image $x$ extracted by the VGG16 model, $x$ represents the real image and $x^{\prime}$ represents the output image of the generator model, and $L_{per}$ is defined as the average ($E_{x,x^{\prime}}$) of the $L_{2}$ norm of the difference of the two feature vectors.

De-identification loss: This loss is used to prevent automatic face recognition systems from correctly matching the generated image with the true subject. To this purpose, we utilize a face recognition system to extract feature vectors from the real and synthesized images. The goal of the de-identification loss is then to maximize the distance between the extracted feature vectors of the input and output images, i.e. making the predicted identities as far as possible from each other. As a result, when a face recognition system tries to recognize the generated face image, it extracts features that are very far from the real image’s features and cannot match the generated image to the input identity and incorrectly matches it to another person. In this paper, we use Resnet50 [34] for face feature extraction. The de-identification loss function is defined as:

where the term $F_{id}(x)$ represents the feature vector of image $x$ extracted by Resnet50, $x$ represents the real image and $x^{\prime}$ represents the output image of the generator model.

Final loss. The final loss is defined as:

where $\lambda_{per}$ and $\lambda_{did}$ are hyperparameters that control the relative importance of each loss. We have used $\lambda_{per}=\frac{1}{8800}$ and $\lambda_{did}=\frac{1}{12}$ in all of our experiments. By minimizing the final loss, firstly, the distance between the feature vectors extracted from the input image and the generated image by the VGG16 network [33] will be minimized, so the generated image will be perceptually similar to the input image. Secondly, the distance between the feature vectors extracted from the input image and the generated image by the Resnet50 network [34] will be maximized, and hence the two images will be mapped to different identities. Finally, the reconstruction and de-identification tasks are performed simultaneously.

The output image of the optimization module meets our goals but it lacks the background parts of the initial image that were removed in the face mask module. So, as the last step of our proposed model, we have devised a merge module to add the background from the original image to the de-identified face image generated by the optimization module to generate the final image.

The simplest way to merge the background and the face image is to perform a binary merge, taking the background pixels from the original image and the face pixels from the generated face image. As expected, this leads to a stark result as the output of the optimization module is not identical to its input and border of the face rectangle is detectable in the final image. On the other hand, applying smoothing filters to eliminate these unwanted edges will cause unpleasant blurring effects.

In this paper, we use a Fourier transform-based method [11] in the merge module shown in Figure 1 (c). In this method, the generated image and the input image are first convolved with a high-pass filter at different levels in Fourier domain, each with an increasing amount of sharpness. At the same time, the mask of the face part of the generated image is also convolved with a low-pass filter at different levels, each with an increasing amount of smoothness. Finally, at each level, high-pass filtered images are combined using a low-pass filtered mask as weights. The final result is obtained by adding the blended results at each level. The pseudocode of the merge algorithm is given in Algorithm 1.

After getting the output image of the merge module, to have the best result, a histogram matching is performed so that the generated image’s histogram matches the histogram of the input image [35].

Comparison of the manipulated and the original images: As stated before, the GMFIM model is not based on additive masks; instead, it tries to find a latent vector to generates a face image with the desired attributes. As a result of this process, the regenerated face image is different from the original one in different regions. To illustrate these changes, a comparison between some input images and their corresponding manipulated versions by GMFIM are shown in Figure 3. As shown in this figure, the changes mainly occur in the main parts of the face including the eyes, eyebrows, nose, and mouth. The amount and severity of the modification are adaptively determined for each image by the optimization process. The differences of the images include changes in the color, size, and even the shape of facial parts. The main point is that most of these changes are not tangible for human users of the social media specially when the original images are not present for comparisons and they easily identify the manipulated images.

The importance of the face mask and merge modules: As mentioned earlier, the goal of the face mask module is to help the optimization module to focus only on the face part of the input image. This in turn necessitates the use of a merge module to add the separated background from the original image to the de-identified face image. Without this separation, the GAN-based optimization step has to reconstruct the whole input image which makes the generator much more complicated. For example, while the StyleGAN is very successful in regenerating diverse face photos, it fails to properly reconstruct many background patterns. Three examples of the results of the two aforementioned approaches are shown in Figure 4. As shown in this figure, omitting the mask-guidance part of the GMFIM causes a quite blurry background with a much smaller SSIM (right images) compared to the complete model (middle images). Moreover, trying to reconstruct the whole image in the optimization module reduces the quality of the generated face regions in this case.

The effect of the hyperparameters: $\lambda_{per}$ and $\lambda_{did}$ are two main hyperparameters of GMFIM that control the relative importance of the perceptual and de-identification losses, respectively. With the help of these two hyperparameters, the quality of the generated images and the success rate against recognition systems can be controlled.

To observe the effect of changing these hyperparameters, the value of $\lambda{per}$ is fixed at $\frac{1}{8800}$ and different values are assigned to $\lambda_{did}$. The SSIM and the attack success rate (ASR) of GMFIM on Indian Celebrities dataset for different values of $\lambda_{did}$ are reported in Figure 5. As the results show, increasing the value of $\lambda_{did}$ degrades the quality of the manipulated images and results in lower SSIMs. Also, it increases the probability of successfully hiding the identity of the person and higher attack success rates. The reason for this is that by increasing the value of $\lambda_{did}$, the importance of de-identification loss increases and it plays a more important role in generating the final image. On the contrary, decreasing the value of $\lambda_{did}$ improves the quality of the generated images but the produced face image is more likely to be identified by AFR systems. As mentioned earlier, in the experiments of this paper, $\lambda_{did}=\frac{1}{12}\approx 0.083$ has been used.