Towards Gender-Neutral Face Descriptors for Mitigating Bias in Face Recognition

After extracting the descriptor $f_{in}$ from a pre-trained face recognition network, we pass it through $M$ to obtain a lower dimensional descriptor $f_{out}$.

First constraint: To make $f_{out}$ proficient at classifying identities we provide it to classifier $C$ and use cross-entropy classification loss $L_{class}$ to train both $C$ and $M$.

$\mathbf{y_{id}}$ is a one hot identity label and $\mathbf{\hat{y}_{id}}$ is the corresponding softmaxed output of classifier $C$.
Training discriminators: $M$ generates $f_{out}$ which is fed to ensemble $E$. Each of the gender prediction models in $E$, denoted $E_{k}$, are used for computing cross entropy loss $L^{(E_{k})}_{g}$ for gender classification. $L_{g}$ is computed as the sum of cross-entropy losses for each $E_{k}$.

$y_{g}$ is the binary gender label for the input face descriptor, and $y^{(k)}_{g}$ represents the respective softmaxed outputs of $E_{k}$ in the ensemble.
Training generator (second constraint): After training $E$, $M$ is trained to transform $f_{in}$ into gender agnostic descriptor $f_{out}$. We then provide $f_{out}$ to each model in ensemble $E$:

The outputs $o_{k}^{*}$ represent the gender probability scores. If an optimal classifier operating on $f_{out}$ were to always produce a posterior probability of 0.5 for both male and female classes then this implies that no gender information is present in the descriptor. To this end, we define the adversarial loss $L^{(E_{k})}_{a}$ for the $k^{th}$ model in $E$ to be:

Here, we use an ensemble of gender prediction models instead of a single model because we want $f_{out}$ to be gender agnostic with respect to multiple gender predictors. This approach was motivated by the work of (Wu et al. 2018) to solve ‘the $\forall$ challenge’. After computing the adversarial loss for model $M$ with respect to all the models in $E$, we select the one for which the loss is maximum. We term this loss as debiasing loss $L_{deb}$.

The idea is that we would like to penalize $M$ with respect to the strongest gender predictor for which it was not able to fool. This approach was introduced in (Wu et al. 2018). $L_{deb}$ is then combined with $L_{class}$ to compute a bias reducing classification loss $L_{br}$ as follows:

Here, $\lambda$ is used to weight the de-biasing loss.

We now explain the various stages of training AGENDA.
Stage 1 - Initializing and training $M$ and $C$: Using input descriptors $f_{in}$ from a pre-trained network, we train $M$ and $C$ from scratch for $T_{fc}$ iterations using $L_{class}$ (Eq. 5).
Stage 2 - Initializing and training $E$: Once $M$ is trained to perform classification, we feed the outputs $f_{out}$ of $M$ to an ensemble $E$ of $K$ gender prediction models. $E$ is trained from scratch to classify gender for $T_{gtrain}$ iterations using $L_{g}$ (Eq. 7). $\phi_{M},\phi_{C}$ remain unchanged in this stage.
Stage 3 - Update model $M$ and classifier $C$: Here, $M$ is trained to generate descriptors $f_{out}$ that are proficient in classifying identities and are relatively gender-agnostic. $f_{out}$ is fed to the ensemble $E$ and the classifier $C$, the outputs of which result in $L_{deb}$ (Eq. 10) and $L_{class}$ (Eq. 5) respectively. We combine them to compute $L_{br}$ (Eq. 11) for training $M$ and $C$ for $T_{deb}$ iterations, while $\phi_{E}$ remains locked. While computing $L_{br}$, the gradient updates for $L_{deb}$ are propagated to $\phi_{M}$ and those for $L_{class}$ are propagated to both $\phi_{M}$ and $\phi_{C}$.
Stage 4 - Update ensemble $E$ (discriminator): In stage 3, $M$ is trained to generate gender-debiased descriptors $f_{out}$ to fool the models in $E$, whereas in stage 4, models in $E$ are re-trained to classify gender using $f_{out}$. Therefore, we run stages 3 and 4 alternatively, for $T_{ep}$ episodes, after which we re-initialize and re-train all the models in $E$ (as done in stage 2). Here, one episode indicates an instance of running stages 3 and 4 consecutively. In stage 4, we heuristically choose one of the models in $E$, and train it for $T_{plat}$ iterations or until it reaches an accuracy of $G_{thresh}$ on the validation set. $\phi_{M}$ and $\phi_{C}$ remain locked in this stage. The motivation for training a single model in $E$ at a time is deferred to discussion in the supplementary material. The full details of training are described in Algorithm 1.

We separately evaluate the face descriptors, obtained from the penultimate layer of following two pre-trained networks:
Arcface : Resnet-101 trained on MS1MV2 ²²2https://github.com/deepinsight/insightface/wiki/Dataset-Zoo with Additive Angular margin (Arcface) loss (Deng et al. 2019). There are 59,563 males and 22,499 females in this dataset.
Crystalface : Resnet-101 trained on a mixture of UMDFaces³³3http://umdfaces.io/(Bansal et al. 2017b), UMDFaces-Videos³(Bansal et al. 2017a) and MS-Celeb-1M (Guo et al. 2016), with crystal loss (Ranjan et al. 2019). There are 39,712 males and 18,308 females in this dataset.
For evaluation, we use the aligned faces in the IJB-C dataset, and follow the 1:1 face verification protocol defined in (Maze et al. 2018). The alignment is done using (Ranjan et al. 2017). However, instead of verifying all the pairs, we only verify male-male and female-female pairs. There are 6.4 million male-male and 2.1 million female-female pairs defined in the protocol. The gender labels are provided in the dataset. Using Arcface and Crystalface we extract 512 dimensional face descriptors for the aligned faces in the IJB-C dataset which are then used for verification of male-male and female-female pairs. The gender-wise verification plots for Crystalface and Arcface descriptors are provided in Fig. 3.

We compare the predictability (i.e. ability to classify gender) of face descriptors extracted from Arcface and Crystalface networks. We train a logistic regression classifier on 60k IJB-C face descriptors (30k males and females) to classify gender and test it on 20k IJB-C descriptors (10k males and females). The images for training and testing are selected randomly, and the face descriptors are extracted using the pre-trained networks (Arcface or Crystalface). Note that there is no overlap between identities in train and test split. From the results in Table 1, we find that descriptors extracted using Arcface show relatively low gender predictability than Crystalface descriptors. In Table 1, we also find that the gender bias (computed using Eq. 1 and plots shown in Fig 3) is lower in most FPRs when Arcface descriptors are used for face verification, as comapred to Crystalface descriptors. This shows that face descriptors with low gender predictability appear to demonstrate lower gender bias in face verification, thus forming the basis of our initial hypothesis (mentioned in Sec. 1). Therefore, we propose techniques to reduce the predictability of gender in face descriptors while making them proficient in identity classification.

Since our hypothesis involves removing gender specific information from face descriptors, we propose a naive approach, termed as ‘Correlation-based PCA’ (CorrPCA) for this task. We first compute the eigenspace of the descriptors and isolate the eigenvectors that encode gender information. After this, we remove these eigenvectors and transform the test face descriptors using the remaining subspace. A similar approach was used in the early nineties (Turk and Pentland 1991) to reduce the impact of illumination on PCA features.
Isolating gender specific components: We first randomly select 80k (40k males and females) aligned face images from MS-Celeb-1M dataset. Then, we extract the 512-dimensional descriptors for these images, using a given pre-trained network (Arcface/Crystalface). We then compute the eigenspace $S\in\mathbb{R}^{512\times 512}$ of the descriptors $X\in\mathbb{R}^{80k\times 512}$ using PCA. Using each eigenvector in $S$, we transfrom the original descriptors $X$ as follows :

Here, $s$ is a row in $S$ (i.e. an eigenvector of $X$), and $v_{s}$ is the corresponding component of each descriptor. Now, for all the 80k images, we have a vector $v_{s}\in\mathbb{R}^{80k\times 1}$ and a label vector $\ell$, with gender labels for all the sampled images. After this, we compute the Spearmann correlation coefficient between $v_{s}$ and $\ell$. We select the eigenvectors in $S$ for which this correlation is lower than $\delta$ and denote them collectively as a subspace $R$. Thus, $R$ is a subspace that has relatively lower gender information. We find that the number of eigenvectors in the subspace $R$ is 504 and 487 for the descriptors of Arcface and Crystalface respectively.
Transforming test face descriptors using remaining components: We then extract the 512-dimensional descriptors for the IJB-C dataset using the given pre-trained network (Arcface/Crystalface), to obtain the transformed descriptors $X_{ijbc}$. Finally, we transform $X_{ijbc}$ using the subspace spanned by $R$ into a new feature space $T_{ijbc}$. We use $\delta=0.1$ in this protocol, for both networks. We evaluate the gender leakage and gender bias of the transformed descriptor sets for IJB-C dataset in Sec. 5.5.

For training AGENDA, we use a combination of UMDFaces, UMDFaces-Videos and MS-Celeb-1M datasets. The face alignment and gender labels are obtained using (Ranjan et al. 2017). We perform our experiments using input face descriptors $f_{in}$ from Arcface and Crystalface, extracted for these datasets. The hyperparameter information for AGENDA is provided below:
Stage 1: $T_{fc}=66000$ iterations, learning rate $\alpha_{1}=10^{-5}$.
Stage 2: $T_{gtrain}=30000$ iterations, learning rate $\alpha_{2}=10^{-3}$. Here, we use $K=1$ and $5$ for Arcface and Crystalface, respectively.
Stage 3: $T_{deb}=1200$ iterations,learning rate $\alpha_{3}=10^{-4}$. We compute $L_{br}$ using $\lambda=10$ and $1$, when using $f_{in}$ from Arcface and Crystalface respectively.
Stage 4: $T_{plat}=2000$ iterations, learning rate $\alpha_{2}=10^{-3}$ (same as stage 2). $G_{thresh}=0.90$ and $0.80$ for Arcface and Crystalface, respectively.
In all the aforementioned training stages, we use an Adam optimizer and a batch size of 400, and we ensure that each batch is balanced in terms of gender.

Evaluating gender leakage: We follow the same steps as in section  5.2 for computing the gender classification accuracy (Table 1) of raw face descriptors. We present the accuracy of the logistic regression classifier trained on face descriptors obtained from CorrPCA and AGENDA in Table 2. We find that for both Arcface, the Crystalface, the classification accuracy goes down when the face descriptors are transformed using AGENDA or CorrPCA framework, which indicates that gender leakage from the descriptors is likely reduced.
Evaluating verification performance and bias: We first evaluate the 1:1 verification performance obtained by CorrPCA-transformed IJB-C face descriptors. From Table 3b, we can infer that this method helps to reduce the gender bias in Crystalface, whereas the bias and performance in Arcface remains mostly unchanged. Following that, we evaluate the verification performance and corresponding bias obtained after using AGENDA. After AGENDA training, we feed the 512 dimensional $f_{in}$ (extracted from Arcface/Crystalface) of aligned IJB-C images to the trained model $M$ (Fig. 2), that generates 256 dimensional gender de-biased descriptors $f_{out}$. We then use $f_{out}$ to perform gender-wise IJB-C 1:1 face verification. From Fig. 4. We find that when using $f_{out}$ from Arcface, the bias is reduced (especially in low FPRs), without catastrophically losing verification performance. This is done by improving female-female verification at low FPRs, while slightly decreasing male-male verification. Similarly, for $f_{out}$ from Crystalface, we find that the gender bias is reduced at low FPRs. The bias is especially close to 0 after FPR $10^{-5}$.
From Table 3b, we can infer that AGENDA consistently outperforms CorrPCA in terms of bias reduction at almost all the FPRs under consideration. For FPRs not reported in Table 3b, the gender bias in verification is exactly same for CorrPCA, AGENDA and the original pre-trained network.

Here, we evaluate two hyperparameters used for training the AGENDA framework on Arcface and Crystalface : (a.) the number of gender prediction models $K$ in the ensemble $E$ used to compute $L_{deb}$ (Eq. 10). (b.) the weight for $L_{deb}$ defined in Eq. 11. We analyze how changing these hyperparameters vary the resultant bias reduction and verification performance at a fixed $\text{FPR}=10^{-5}$ in IJB-C dataset.
Varying K :We experiment with $K=1,2,3,4,5$ and $10$. Here, we fix all the other hyperparameters and use the same values specified in Sec. 5.4. In Fig. 5(a) and (b), we find that in Arcface, changing $K$ does not have much effect on gender bias or verification TPR at FPR $10^{-5}$. However, for Crystalface, we find that as we increase $K$, the gender bias keeps decreasing which in turn leads to drop in verification performance at FPR $10^{-5}$. We find that at $K=4$, the bias drops to 0, and as we further keep increasing $K$, the verification performance decreases.
Varying $\lambda$: We fix $K=5$ and evaluate $\lambda=0.1,1,10$ for training the AGENDA framework using $f_{in}$. All the other hyperparameters use the same values specified in in Sec. 5.4. The results are presented in Fig. 5(c) and (d). For both Arcface and Crystalface, as we keep on increasing the value of $\lambda$, the gender bias keeps generally decreasing and the verification TPR keeps decreasing.

Gender is an important face attribute that helps deep networks recognize faces. So, minimizing its predictability by using AGENDA is expected to decrease the overall performance as a side-effect. As seen in Fig. 4, the verification performance in Crystalface reduces considerably as compared to Arcface, after AGENDA is applied. To understand this behavior, we analyze the distribution of the feature space of both Arcface and Crystalface. For this, we first randomly select 80k images (40k males and females) from the IJB-C dataset, and extract their face descriptors using a given pre-trained network (Arcface or Crystalface). Using PCA, we compute the eigenspace of this set of face descriptors. As done in the implementation of our CorrPCA baseline, we compute the gender correlation of each of the 512 eigenvectors in the eigenspace (Fig. 6).

The top-256 eigenvectors encode more identity information than the bottom ones, since the networks are trained to classify identity. We find that the gender correlation of these identity encoding (top-256) eigenvectors of Crystalface is generally higher than Arcface. This seems to indicate that gender and identity are more entangled in face descriptors from Crystalface than Arcface. Therefore, the drop in verification performance, when the descriptors are gender de-biased is also expected to be more for Crystalface, since the verification performance depends on identity information encoded in the descriptors.

In (Ranjan et al. 2019), the face descriptors from Crystalface are not directly used for verification. Instead, the descriptors undergo triplet probabilistic embedding (TPE) (Sankaranarayanan et al. 2016) for generating a template representation of a given identity. TPE is an embedding learned to generate more discriminative, low-dimensional representations of given input descriptors, that have been shown to achieve better verification results. We apply TPE on the descriptors obtained using Crystalface and find that TPE improves the overall verification performance, but it also increases bias at all FPRs. For comparison, we apply TPE after transforming the Crystalface descriptors with AGENDA. From Table 4, we can infer that the gender bias in the verification results obtained after applying TPE on AGENDA-transformed descriptors is lower than when TPE is applied on original face descriptors of Crystalface. The details of training TPE are provided in the supplementary material.