FICGAN: Facial Identity Controllable GAN for De-identification

Embedding an image into the spatial and non-spatial codes with our careful loss design induces the disentanglement of the identity (ID) attributes from the non-identity (non-ID) attributes containing the spatial characteristics, such as the head pose, background, and local color scheme. The encoder ($E$) takes the input images $x\sim\textbf{X}{\subset}\mathbb{R}^{H\times W\times 3}$ and extracts two types of features: the first is the ID latent code ($z_{id}$), which is a vector of size 512 for encoding the ID attributes; the second is the spatial latent code ($z_{non}$) titled the non-ID latent code, which is a 3-dimensional tensor of size $4\,{\times}\,4\,{\times}\,512$ encoding the non-identity attributes including the spatial information. In contrast to the non-ID latent code containing the spatial information, such as the head pose, background, and local color scheme, the ID latent codes contain the non-spatial information, which make it easier to disentangle the ID and non-ID attributes. It is similar to (Park et al. 2020) that utilizes the latent codes from a different dimension to obtain the structural information.

The ID and non-ID latent codes with the different dimensions from the encoder ($E$) are inserted as the input of the generator ($G$). The non-ID latent code is inserted into the generator in a feedforward manner to help generate the structural attributes, such as the head pose and expression, in the lower dimension of the image. In contrast, the ID latent code is processed by adaptive instance normalization (AdaIN) (Huang and Belongie 2017) to be adjusted to the generator in a layer-wise manner (Karras, Laine, and Aila 2019). Before being inserted into the generator, the ID latent code $z_{id}$ is embedded to the intermediate latent vector $w$, which controls the ID of the generated faces. It helps to locate the appropriate ID-space when applying the $k$-same algorithm in the domain of $z_{id}$ to generate high-quality de-identified images.

The mapping network consists of a series of multi-layer perceptrons (MLP). If the latent codes of ID ($z_{id}$) and non-ID ($z_{non}$) originate from a source image, $G$ generates a reconstructed image ($\hat{x}_{rec}$). In contrast, if $z_{id}$ and $z_{non}$ originate from different images, $G$ generates a de-identified image ($\hat{x}_{mix}$).

We utilize the pre-trained verification network to effectively disentangle the ID attributes. The face verification network ($V$) takes an image and extracts a feature vector of size 512. Because the verification networks extract discriminative features of facial identities, they are used for contrast learning to disentangle identities.

We disentangle it into two features $z_{id}$ and $z_{non}$, which allows independent modification of the facial attributes. In order to disentangle the identity attributes from the input image $x$, contrastive learning (Dai and Lin 2017; Kang and Park 2020; Zhang et al. 2020) is applied. From the input image $x$, the encoder $E$ separates the two latent codes $z_{id}$ and $z_{non}$. The two types of contrastive verification loss can be defined by the ID features of $x_{s}$, $x_{t}$, and mixed image $\hat{x}_{mix}$. The embedding features are extracted by $V$, and the cosine similarity ($\langle\bullet,\bullet\rangle$) is used to calculate the similarity between the embedding features.

The first type of loss to disassociate $\hat{x}_{mix}\,{=}\,G(z^{t}_{id},z^{s}_{non})$ from the $x_{s}$ in identity attributes can be defined as:

$$\mathcal{L}_{neg}=\langle V(x_{s}),V(\hat{x}_{mix})\rangle.$$

(1)

where $z^{t}_{id}$ indicates the ID latent code disentangled from $x_{t}$, while $z^{s}_{non}$ indicates the spatial latent code of the non-identity attributes of $x_{s}$. The second type of loss to associate $\hat{x}_{mix}$ close to $x_{t}$ in identity attributes can be defined as:

$$\mathcal{L}_{pos}=1-\langle V(x_{t}),V(\hat{x}_{mix})\rangle.$$

(2)

The reconstruction loss is designed for the network to preserve all of the inserted information of the source image $x_{s}$. Enabling to reconstruct $\hat{x}_{rec}$ from $x_{s}$, the reconstruction loss can be defined as below:

$$\mathcal{L}_{rec}(E,G)=\mathbb{E}_{x_{s}\sim X}[||x-G(E(x))||_{2}].$$

(3)

The reconstruction loss helps maintain the information of $x_{s}$ even after disentangling the ID and non-ID attributes.

The adversarial loss (Goodfellow et al. 2014) is designed to ensure the visual quality of reconstructed images $\hat{x}_{rec}$ and mixed images $\hat{x}_{mix}$ as below:

		$\displaystyle\mathcal{L}_{adv_{rec}}(E,G,D)=$	$\displaystyle\mathop{\mathbb{E}}\limits_{x_{s}\sim X}[-log(D(G(E(x_{s}))))],$
		$\displaystyle\mathcal{L}_{adv_{mix}}(E,G,D)=$	$\displaystyle\mathop{\mathbb{E}}\limits_{\begin{subarray}{c}x_{s},x_{t}\sim X,\\ x_{s}\neq x_{t}\end{subarray}}[-log(D(G(z^{s}_{non},z^{t}_{id})))],$

It manages the degradation of image quality due to the blurriness caused by applying the $k$-same algorithm and the unnatural imagery due to a combination of $z^{s}_{non}$ and $z^{t}_{id}$.

During an iteration of the training phase, the model performs both reconstruction and mixing. The first is the reconstruction loss when $x_{s}=x_{t}$, and the second is the contrastive verification loss when $x_{s}\neq x_{t}$. Thus, the total loss can be defined as below:

$\displaystyle\mathcal{L}=\lambda_{1}\mathcal{L}_{rec}+\lambda_{2}(\mathcal{L}_{pos}+\mathcal{L}_{neg})+\lambda_{3}(\mathcal{L}_{adv_{rec}}+\mathcal{L}_{adv_{mix}}).$

To obtain a high level of privacy protection, we apply $k$-same algorithm (Newton, Sweeney, and Malin 2005) in the latent space of our model, which is titled as manifold $k$-same algorithm. From the randomly selected $k$ facial images, the centroid of $\{w_{1},w_{2},\cdots,w_{k}\}$ is calculated by the encoder and the mapping network as shown in Figure 2-(b). This centroid vector (or $k$-anonymized code; $w_{m}$) satisfies the $k$-same algorithm because the properties of $k$ are combined. Based on $w_{m}$, our single model can perform de-identification in four other ways with securing $k$-anonymity, depending on the purpose of use as follows.

We leverage the layer-wise controllability offered by StyleGAN (Karras, Laine, and Aila 2019) architecture. Unlike StyleGAN that simply mixes up the styles in images, we mix up the ID attributes of the source and target images in a layer-wise manner for a fine-grained control to manage the trade-off between the degree of de-identification and preservation of attributes for enhanced data utility. In this way, we can observe the transition resulting from adjusting the identity-related characteristics in facial attributes. First, we can control the degree of de-identification by selecting the index of layer ($i$), which is the point of standards to adjust the amount of $w_{m}$ and $w_{s}$ obtained from the source image $x_{s}$. To be specific, we insert $w_{s}$ from the first to $i$-th layers and $w_{m}$ from $i+1$-th to $L$-th layers. When $i=0$, mean identity $w_{m}$ is inserted to all layers, thus the source image is strongly de-identified (strong de-ID). In contrast, when $i=L-1$, $w_{m}$ is applied only to the highest layer, thus leading to a weak de-identification (weak de-ID). An example (i=1, when L=4) of controlling the degree of de-identification is shown in Figure 2-(c). The greater the $i$ is, the less the degree of de-identification is, which leads to more attributes preserved to enable a fine-grained control. We provide detailed explanations on the trade-off between the identity attributes and non-identity attributes according to the adjustment of $i$ in the paragraph of ‘Trade-off between De-identification and Attribute Preservation’ under Experiments.

We can control specific attributes of the image according to the user’s preference. Loss of specific attributes is inevitable due to a trade-off between the identity and non-identity attributes (Gross et al. 2005). However, we opt to control the loss by employing $w^{\prime}_{m}$, the mean vector of specific attributes obtained from a group of images with the same attributes. Using $w^{\prime}_{m}$, we can either preserve or insert specific attributes for fine-grained control of de-identification. We provide detailed explanations on how our model controls the preservation of attributes as desired in the paragraph of ‘Controllability on Facial Attributes’ under Experiments.

Our model can control the direction of de-identification using the examplar of face images, therefore achieving data diversity. The increase of $k$ in $k$-anonymity has a twofold effect: it can guarantee a greater degree of anonymity for privacy protection, but can also lead to excessive generalization of mean facial identity, resulting in identical faces. However, data diversity can be required in some cases. For example, if multiple people are presented in the same scene, their de-identified faces should be different from each other. Also, it is important to consider diversity in race, ethnicity, gender, and age to fully represent a broad range of faces.

For such cases, we can generate diverse images of de-identified faces by controlling the direction and the degree of de-identification through the interpolation between $w_{t}$ and $w_{m}$. The $k$-anonymized code for direction, $w_{d}$, can be calculated by employing the scaling factor $\alpha$ as below:

$$w_{d}=w_{m}+\alpha\cdot(w_{t}{-}w_{m}).$$

(5)

It utilizes the linear interpolation $w^{\prime}$ between $w_{m}$ and the latent space of a specific target image $x_{t}$. Diverse images of de-identification are provided in the paragraph of ’Diversity of De-identified Images’ under Experiments.

Lastly, target ID code $w_{t}$ is employed instead of $w_{m}$ for identity swapping to switch to the target identity. The results are illustrated in Fig. 9.