Facial Identity Anonymization via Intrinsic and Extrinsic Attention Distraction

In this subsection, we present how to perform identity feature anonymization in the feature space. As shown in Figure 2 (b), a pre-trained classification network $\Psi$ (e.g. FaceNet [46]) is employed for identity feature anonymization by conducting intrinsic identity attention distraction at the last convolutional layer (its output is denoted as $A$). First, the input data flow goes along the red solid lines to find the identity related feature maps in $A$ on top of the calculated CAM heatmap $H$ [47]. Then, the data flow switches to the black lines after performing attention distraction on $A$, goes through $\hat{A}$, and finally output the recasted identity feature $f_{x}$ for anonymization. $\hat{A}$ is calculated from $A$ by distracting the identity attention away from $H$, where the visual results of $H$ and $\hat{H}$ in Figure 2 (b) illustrate the function of this operation. Note that the identity of $x$ may be new for $\Psi$ and, thus, the prediction may be incorrect. But, it does not matter. We simply employ the pre-trained classes as the codebook to interpret the identity of any input faces regardless of whether these identities were trained or not. The final softmax output is used as the indicator to determine which pre-trained identities are more related to $x$.

Let $c_{i}$ denote the top $i$-th prediction result (i.e. identity class) of $\Psi$. The CAM heatmap of any given class $c_{i}$ can be calculated as the weighted combining of the forward activation maps by following [47]

$$\small H=ReLU(\alpha^{c_{i}}A)=ReLU(\sum_{j}\alpha^{c_{i}}_{j}A^{j}),\vspace{-0.1cm}$$

(1)

where $\alpha^{c_{i}}_{j}$ denotes the neuron importance weight of the $j$-th activated feature map $A^{j}$, which is calculated by average-pooling the gradients flowing back

$$\small\alpha^{c_{i}}_{j}=\frac{1}{Z}\sum_{k}\sum_{l}\frac{\partial y^{c_{i}}}{\partial A^{j}_{kl}}.\vspace{-0.1cm}$$

(2)

We analyze the importance of facial features on top of $H$. Motivated by the existing studies that some attributes are critical for identification while the others not [30, 47, 51, 9], we can reasonably suppose that anonymization can be formulated as a min-max optimization problem by suppressing the identity dependent attributes and preserving the identity-independent ones. We model this as an attention distraction problem by using the identity correlated CAM heatmaps and enforcing $H_{c_{i}}\rightarrow 0$. Then, we have

$$\small\hat{A}=A+\xi,~{}~{}~{}s.t.~{}~{}~{}\alpha^{c_{i}}\hat{A}=0\vspace{-0.1cm}$$

(3)

where $\xi$ is a modulation item. It is an invalid solution by directly using $\xi=-A$ because the resulting distracted feature map $\hat{A}$ would become meaningless for identity representation. To solve it, we introduce an assistant matrix $\Phi^{c_{i}}$ to distract the attention of the identity correlated CAM heatmaps and define $\xi=(\alpha^{c_{i}})^{T}\Phi^{c_{i}}$ by taking the importance of the activated feature map into consideration. By substituting $\xi$ back to Equ. (3), we obtain

$$\small\Phi^{c_{i}}=-\frac{\alpha^{c_{i}}A}{\alpha^{c_{i}}(\alpha^{c_{i}})^{T}}.\vspace{-0.1cm}$$

(4)

To enhance anonymization, we propose to jointly distract the top $K$ predictions of $\Psi$ because they may be closely related to identity representation. We redefine Equ. (3) as

$$\small\hat{A}=A+\sum_{i=1}^{K}w_{i}(\alpha^{c_{i}})^{T}\Phi^{c_{i}},\vspace{-0.1cm}$$

(5)

where $w_{i}$ denotes the contribution of the $i$-th item.

In this subsection, we present how to perform visual clue anonymization by extrinsic identity attention distraction using both the visual appearance and the geometry structure in the visual space. One may think of directly replace the original data $q$ with a completely (or predefined) different delegate $\hat{q}$ (e.g. from white skin to black skin) so that no one can re-identify it, but this may easily damage data utility (e.g. ethnic and expression preservation [18, 37, 29]). A possible better choice is to sample some $\hat{q}$ based on $q$ so that they share more identity independent information than identity dependent information in a random manner. As shown in Figure 2 (c), we first introduce an instance-level probabilistic delegate (IPD) sampling method and then use it to anonymize the visual clues.

IPD Sampling. Given data $q$, we build a candidate set $M_{q}$ by finding its top $k$-nearest neighbors in the feature space (e.g. Arcface [8]). For each candidate $X_{i}$$\in$$M_{q}$, we rely on the simple random sampling to obtain a delegate $\hat{q}$$=$$X_{k}$$\in$$M_{q}$ according to the probability set $\{P(X_{i}),1\leq i\leq size(M_{q})|\sum_{i}P(X_{i})=1\}$, where $P(X_{i})$ is defined by following the idea of differential privacy (DP) [38, 10, 11]

$$\small P(X_{i})=\frac{e^{[\epsilon u(q,X_{i})/(2\Delta u)]}}{\sum_{X_{j}\in M_{q}}{e^{[\epsilon u(q,X_{j})/(2\Delta u)]}}},$$

(6)

where $\epsilon$ is the privacy budget, $u$ is the utility function and $\Delta u$ is its $\ell_{1}$ sensitivity. Notice that DP received increasing attention in face anonymization since it can provide a theoretically sound privacy protection by adding random perturbation. In previous works, DP is usually used with low-level or middle-level data (e.g. pixels [12, 7] and identity features [32, 6, 55]) by adding Laplace noise. Differently, we generalize it to perform instance-level data sampling to reduce the disclosure risk of the identity information.

Visual Appearance Anonymization (VAA). Since the visual appearance may be correlated with some useful attributes, such as ethnic and age, the significant change of it may easily damage the data utility. To solve the problem, as shown in Figure 2 (c), we rely on IPD to sample a delegate face $Z_{a}$ by using the following utility function (i.e. $u=u_{a}$)

$$\small u_{a}(x,X_{i})=\frac{\max_{j}d(x,X_{j})-d(x,X_{i})}{\max_{j}d(x,X_{j})-\min_{j}d(x,X_{j})}\vspace{0.15cm}$$

(7)

so that $Z_{a}$ and $x$ would have a high probability to share the same set of data utility, where $d(x,X_{j})$ is the $\ell_{2}$ distance between the features of $x$ and $X_{j}$.

Geometry Structure Anonymization (GSA). The detected landmarks [2] are used to describe the facial geometry structure $S_{x}$ of $x$. Instead of directly modifying the landmarks (which is a complicated task to make the result look real), we prefer to perform instance-level anonymization by replacing $S_{x}$ with another realistic delegate. As shown in Figure 2 (c), the process consists of two steps. We first rely on IPD to sample a delegate $\bar{S}_{x}$ that has the same pose as $S_{x}$ by using the following utility function (i.e. $u=u_{g}$)

$$\small u_{g}(S_{x},X_{i})=\frac{d(S_{x},X_{i})-\min_{j}d(S_{x},X_{j})}{\max_{j}d(S_{x},X_{j})-\min_{j}d(S_{x},X_{j})}$$

(8)

that tends to sample a more distinct geometry structure. Since this may violate the original pose and expression, we then recover them by adjusting the contour and mouth of $\bar{S}_{x}$ according to $S_{x}$ and preserving the thickness of the upper and lower lips of $\bar{S}_{x}$, resulting in $\hat{S}_{x}$. To recover the original background, we fuse $\hat{S}_{x}$ and the background $x_{b}$ of $x$ as the geometry input $Z_{g}$. Note that $\hat{S}_{x}$ is not the same as $\bar{S}_{x}$, which would reduce the probability of identity intrusion.

This subsection focuses on face synthesis. As shown in the step2 of Figure 2 (a), our generator $G$ takes appearance image $Z_{a}$, identity feature $Z_{id}$ and geometry input $Z_{g}$ as input, goes through the appearance encoder $E$ and the conditional translator $T$ to produce a realistic face image $\hat{x}$.

Appearance Encoder $E$ is adopted to process $Z_{a}$ to obtain an appearance feature $f_{a}$. We realize it by stacking six ResBlocks [15] and a SumPooling layer [61]. Semantic segmentation [57] is used to obtain the foreground face image of $Z_{a}$ as input, when it is not available, we can approximately use the detected landmarks to realize this.

Conditional Translator $T$ aims at translating $Z_{g}$ and the condition input $Z_{c}$ to a realistic face image $\hat{x}$. We employ a U-Net like structure to build $T$ by following [45, 20, 61] via downsampling and upsampling with ResBlocks, where adaptive instance normalization (AdaIN) [17] is employed to fuse the identity and appearance information encoded in $Z_{c}$ which is defined as the fusion of $Z_{id}$ and $f_{a}$ by using a concatenation (Concat) layer and a fully connected (FC) layer: $Z_{c}$=FC(Concat($Z_{id}$, $f_{a}$)). Note that, if the out contour were changed, $T$ would adaptively inpaint the background so that the generated face could smoothly dissolve into the original background. Users can realize personalized anonymization by manipulating $Z_{id}$, $Z_{a}$ and $Z_{g}$ under different scenarios. For example, one can simply use the facial region of $x$ to preserve the original appearance or attributes.

Let $\{x,y\}$ be a set of randomly sampled face images, $x$ acts as the image to be anonymized and $y$ acts as the identity provider, we present a 1:1 alternative reconstruction and cycle swap-reconstruction strategy for network training. For the former, $\hat{x}$ is reconstructed from $x$, denoted as $Z_{id}=f_{x}$. For the latter, we change the identity from $x$ to $y$ and back to $x$ in a loop, denoted as $Z_{id}=f_{y}$. The following multi-task loss function is used to optimize our generator

$$\small L_{All}=\lambda_{1}L_{1}+\lambda_{2}L_{2}+\lambda_{3}L_{3}+\lambda_{4}L_{4}+\lambda_{5}L_{5}+\lambda_{6}L_{6},$$

(9)

where $L_{1}$ is the adversarial loss, $L_{2}$ is the feature matching loss borrowed from [54] to stabilize the training process by matching the multi-layer features of discriminator D for the input and output images, $L_{3}$ is the perceptual loss, $L_{4}$ is the appearance loss, $L_{5}$ is the identity loss, $L_{6}=\mathbb{E}[|x_{b}-g_{b}|_{1}]$ is the $\ell_{1}$ between the background images of $x$ and the generated image $g$, and $\lambda_{1}\sim\lambda_{6}$ are parameters.

The adversarial loss $L_{1}$ is defined as

$$L_{1}=\left\{\begin{aligned} L_{G}(x,\hat{x})+L_{D}(\hat{x}),&~{}Z_{id}=f_{x}\\ \beta_{1}(L_{G}(x,\hat{y})+L_{D}(\hat{y}))+\\ L_{G}(\hat{y},\bar{x})+L_{D}(\bar{x}),&~{}Z_{id}=f_{y},\end{aligned}\right.$$

(10)

where $L_{G}(x,\hat{x})=\mathbb{E}[\max(0,1+D(\hat{x},S_{x}))+\max(0,1-D(x,S_{x}))]$ is used to optimize the generator G with the help of D which takes paired data (image,structure) as input, $L_{D}(\hat{x})=\mathbb{E}[-D(\hat{x},S_{x})]$ is used to optimize D, $\hat{x}=G(f_{x},Z_{a}(x),Z_{g}(x))$, $\hat{y}=G(f_{y},Z_{a}(x),Z_{g}(x))$, $\bar{x}=G(f_{x},Z_{a}(\hat{y}),Z_{g}(\hat{y}))$. The perceptual loss $L_{3}$ is defined as

$$L_{3}=\left\{\begin{aligned} &\mathbb{E}[\sum_{i}\rho_{i}(x,\hat{x})],&Z_{id}=f_{x}\\ &\mathbb{E}[\sum\limits_{i}(\beta_{3}\rho_{i}(x,\hat{y})+\rho_{i}(\bar{x},x))],&Z_{id}=f_{y},\end{aligned}\right.$$

(11)

where VGG19 and FaceNet [48, 46, 22] are used to calculate $\rho_{1}$ and $\rho_{2}$. The appearance loss $L_{4}$ is defined as

$$L_{4}=\mathbb{E}[d_{a}(x,\hat{x})]+\mathbb{E}[d_{e}(x)],$$

(12)

where $d_{a}(x,\hat{x})$ is the $\ell_{1}$ distance between $f_{a}(x)$ and $f_{a}(\hat{x})$, $d_{e}(x)$$=$$\max(0,$$cos(f_{a}(x),f_{x}))$ discourages $E$ from encoding the identity information. The identity loss is defined as

$$L_{5}=\left\{\begin{aligned} &\mathbb{E}[d_{f}(\hat{x},x)],&Z_{id}=f_{x}\\ &\mathbb{E}[\beta_{2}d_{f}(\hat{x},y)+d_{f}(\bar{x},x)],&Z_{id}=f_{y},\end{aligned}\right.\vspace{-0.2cm}$$

(13)

where $d_{f}(\hat{x},x)=|f_{\hat{x}}-f_{x}|_{2}$ is the $\ell_{2}$ distance between the identity features of $\hat{x}$ and $x$.

Dataset. Three popular datasets CelebA-HQ [28], VggFace2 [4] and LFW [16] are used. CelebA-HQ has 30,000 high quality facial images from 6,217 persons, where 5,000 images are used as the test set. VggFace2 has 3.31 million images from 9,131 persons, where a subset of 5,000 images from 1,000 identities are used for systematical analysis and fair comparison. LFW has 13,233 face images from 5,749 individuals, where 1,680 identities have more than one image and we use a subset of 5,000 images from them.

Implementation Details. For pre-processing, we employ [2] to detect facial landmarks and BiSeNet [57] to perform semantic segmentation. We rely on [20] and [61] to build G and D by stacking ResBlocks. We train our network to generate 256$\times$256 images by using the Adam optimizer. We set $\lambda_{1}$$\sim$$\lambda_{4}$=1, $\lambda_{5}$=$\lambda_{6}$=$\beta_{2}$=2, $\beta_{1}$=0.6, and $\beta_{3}$=0.8. Our approach is trained on CelebA-HQ and evaluated on all.

Evaluation Measures. Our approach is evaluated from the perspectives of privacy protection (anonymization and identity intrusion) and data reusability. For anonymization, we calculate re-identification (ReID) rate (in percentage $\%$). For identity intrusion, we calculate identity swapping (IDS) rate ($\%$). The pre-trained FaceNet [46] are ArcFace [8] are used for face verification. The cosine similarity is used for ArcFace with two thresholds $0.30$ and $0.35$, and the $\ell_{2}$ distance is used for FaceNet with three thresholds $0.9$, $1.0$ and $1.1$. Face alignment [2] is used to evaluate face detection rate ($\%$). LPIPS and SSIM [3] are used to evaluate image quality. Pre-trained classifiers [15] are used to evaluate attribute preservation ($\%$), including expression, ethnic, gender, age and makeup.

We mainly compare our approach with the following representative and state-of-the-art (SOTA) methods: CIAGAN [37], PIFD [3], DeepPrivacy (DP1) [18], DeepPrivacy2 (DP2) [19], LDFA [25], FALCO [1] and Riddle [29].

Qualitative Results. As shown in Figure 3, the success of Blurring and Pixlation can be contributed to the destruction of image content, which would tell the observers that the data is under protection. In contrast, the generative methods not only show excellent anonymization performance but also show higher probability of making the results imperceptible to observers. DP1 and DP2 may fail to retain some facial attributes, like expression. CIAGAN and LDFA may generate distorted faces. PIFD may bring some artifacts. FALCO and Riddle may lose the facial details. Compared with the other methods, our results can not only look realistic but also preserve more original attributes.

Quantitative Results. Since the image content of Pixelation and Blurring is significantly destroyed, we only compare with the generative methods in Table 1. Ours, Riddle [29] and PIFD [3] outperform the other methods on ReID across different face recognition backbones, especially when larger thresholds are used, where DP2 exhibits competitive results. Besides, it is very important to see if there happens identity intrusion after anonymization. CIAGAN suffers from the highest IDS rates. Compared with the best performed PIFD and DP1, our approach obtains competitive results (e.g. $58.6\%$ vs. $34.9\%$ for PIFD and Ours), which can be contributed to identity attention distraction. Also, we test another recent AdaFace model [24] for face verification. According to Table 2, it is easy to observe that our approach exhibits similar behavior to that in Table 1.

Utility Preservation. We compare the data utility of different methods in Table 3. DP1 and DP2 performs poor on expression. PIFD, DP1 and Riddle perform poor on makeup. All these methods perform poor on ethnic. Compared with the other methods, our approach achieves a much better balance on all items and performs well on preserving expression and makeup. We also find that, as with SOTA methods, our approach can also achieve $100\%$ face detection rate, which also reveals its high data utility.

Diversity and Controllability. As shown in the first two rows of Figure 4, our approach can produce diverse results that look different from each other. Besides, we have tried to add a similar distraction item to Equ(5) to test the diversity of $\hat{A}$ by using the bottom $j$-th prediction ($1\leq j\leq 3$). According to the last row of Figure 4, although we can produce diverse results, the facial expression has changed from non-smile to smile, which indicates that adding such diversity may affect the data utility. Our approach can also support flexible anonymization according to user requirements and practical applications. For example, in Figure 5, we can produce different anonymous faces by controlling the geometry and visual appearance inputs, where the influences of changing the geometry structure is more significant.

Analysis. According to the above results, it is obvious that achieving high anonymization performance is relatively easy but it is somewhat difficult to (a) prevent identity intrusion and (b) preserve data utility. The reason why (a) is hard lies in that it is uneasy to ensure the non-existence of a synthesized identity in reality. The reason why (b) is hard lies in that identity is closely related to some critical facial attributes and the change of identity would inevitably lead to some variations on facial attributes. In Figure 3, 4 and 5, we can intuitively observe the attribute changes on the faces (e.g. eye and nose) and they may vary for different persons. Most existing methods pay much less attention on this and anonymization is usually achieved at the cost of damaging too much useful information. Although our approach can not perform the best all the time, it has achieved a better privacy-utility tradeoff. This can be mainly contributed to our anonymization strategy of minimizing the changes on identity independent attributes. But, our approach still suffers from the drawbacks of well preserving some facial attributes, such as ethnic and eye gaze direction, which are left for the follow up works.

In Figure 6, we plot the influential curves for the top $K$ predictions of the face classification network. With top 1 distraction, our approach can remove over $75\%$ of identity information. When increasing $K$, the ReID and IDS rates keep decreasing and the trend would slow down when $K\geq 2$. The utility preservation performance would decrease slightly along with $K$ (see Attribute and LPIPS). Most of the curves would become almost flat after $K\geq 4$. Thus, we generally recommend to set $K$ vary from $2$ to $4$ to reduce the computational costs and the loss on data utility.

In Figure 7, we qualitatively present some visual comparison results. With larger $K$, the CAM heatmap would focus much farther from the facial parts, especially the eyes. By comparing the face images before and after feature distraction, one can find some significant changes on the facial parts (e.g. the nose may vary from small to big or vice versa) but they may vary for different persons. The results also show that the joint distraction in Equation (5) would push some critical facial features or attributes heading to the opposite direction to realize identity anonymization, but this may also lead to some other unexpected changes, such as the age of the first person in Figure 7. This negative effect may come from the significant change of the identity related information in $A$. These observations show that some facial parts or attributes are critical because they are correlated to identity representation, and the change of them may more or less lead to the performance drop on utility preservation.

In Figure 8, we compare the identity features before and after IFA by using the classical T-SNE embedding [52]. The intra-class differences would increase after identity feature distraction, but they still exhibit clustering characteristics regardless of some outliers, which would favor utility preservation. Note that our results can preserve the personalized facial attributes according to the status of each image. For example, as for the $7$-th person, our results can still retain the makeup of each instance.

In Table 4, we present the ablation study results by removing the key components. w./o. IFA means removing the identity input and only using $Z_{a}$ as the conditional input. w./o. VAA means feeding the original face appearance to $Z_{a}$. w./o. GSA means feeding the original geometry structure to $Z_{g}$. IFA can significantly help to reduce the ReID and IDS rates, but it suffers from performance drops on attribute preservation and image quality. GSA and VAA can further help to improve the ReID and IDS performance, but may lead to some drops on utility preservation. We have also verified the expression recovery in GSA by removing it and observed a significant performance drop.

In Table 5, we study the performance of IFA and GSA by using different strategies. IFA-Rand, IFA-KFN and IFA-Ours denote using random delegate, $k$-th farthest neighbor and our method to anonymize the identity feature, respectively. GSA-Rand, GSA-KFN and GSA-Ours denote using random delegate, $k$-farthest neighbor and our IPD method to anonymize the geometry structure, respectively. It is obvious that IFA-Ours and GSA-Ours have achieved a better balance between privacy protection and utility preservation.

In Equ.(7) and Equ.(8), $u_{a}$ and $u_{g}$ are determined by jointly considering anonymity and data utility. According to Table 6, $u_{a}$ can help to improve the data utility and $u_{g}$ can help to improve the protection ability.

We show the generalization ability of our approach on Vggface2 and LFW datasets. According to Table 7, most methods show quite high privacy protection ability (e.g. $0.0\%$ ReID rate). Our approach outperforms the contrast methods on attribute, LPIPS and SSIM, which indicates that our approach has achieved a better privacy-utility tradeoff. According to Figure 9, one can find that the anonymized faces on both datasets look realistic and different from their original versions. These results are in consistent with the results reported in the previous subsections, which have again helped to verify the performance of our approach.

We conduct a simple user study to verify the performance of our approach from the perspectives of human. As shown in Figure 10, we asked around 30 participants to answer two kinds of questionnaires: for Q1, each anonymized face is paired with another image of the original face; for Q2, the top 5 retrieved results are presented. For each of the contrast methods, we randomly assign each participant: (1) $20$ Q1 to calculate the ReID rate of choosing B and C; (2) $20$ Q2 from dataset retrieval to calculate the IDS rate of choosing ’not appear’; (3) $20$ Q2 from Google retrieval to calculate the ReID rate of choosing ’not appear’.

As shown in Figure 11, one can observe that: (1) our ReID and IDS results closely follow CIAGAN, which work better than the other methods; (2) the IDS rate of all the generative methods are high (over $45.0\%$), which may easily lead to identity intrusion. Since the image distortion would prevent the observers from correct recognition, it is easy for CIAGAN to achieve the best performance. This study show consistent results with that of machine recognizers.