An information-theoretic learning model based on importance sampling

The performance of a learning system for a given distribution $p(x)$ is measured by the following the risk functional

First, since we don’t know exactly the future data distribution $p(x)$ in $\Gamma$, we need to find a $p(x)$ that maximizes the objective function. Given the worst-case distribution $p(x)$, we aim to find the $\omega$ which minimizes $R(\omega)$. Therefore, the corresponding optimization problem becomes a minimax problem

Second, we reformulate the risk functional and the distribution deviation constraint using the idea of importance sampling method where a mathematical expectation with respect to $p(x)$ is approximated by a weighted average of random draws from another distribution $q(x)$. For any probability density $q(x)$ with $q(x)>0$ whenever $p(x)>0$, the risk functional is

and the KL-divergence between $p(x)$ and $q(x)$ is

Third, we derive the empirical estimation of (4) and (5). A standard approach to train the models in statistical learning is to use Empirical Risk Minimization(ERM)[17, 1]. ERM learns the prediction rule by minimizing an approximated loss under the empirical distribution of samples, which is defined as

where the subscript E represents empirical. Suppose the observed dataset $D=\{x_{1},x_{2},\dots,x_{N}\}$ is $N$ i.i.d samples drawn from $q(x)$. The self-normalized importance sampling weight[18] for data point $x_{i}$ is

where $\sum_{i=1}^{N}w_{i}=1$. $W=[w_{i}]_{N\times 1}$ is called the importance sampling weight. Therefore, the empirical estimation of (4) is

and the empirical estimation of (5) is

where $\{\frac{1}{N}\}$ denotes the discrete uniform distribution with $N$ samples. Plugging (8) and (9) back into (3), the optimization problem (3) becomes

where (12) is the constraint for the importance sampling weight $W$. Next, the constrained optimization problem in (10) can be reformulated to the unconstrained optimization problem using the Lagrange method. Let the Lagrange multiplier for (11) and (12) be $T$ and $\lambda$ respectively. The max problem in (10) can be reformulated as

Here $T>0$ is the temperature that governs the level of randomness of $W$, which implicitly controls the allowed distribution deviations. Specifically, when $T\rightarrow\infty$, the distribution shift $KL(p(x)\parallel q(x))$ should be small while for $T\rightarrow 0$, the distribution shift can be very large. Since $T$ is a pre-defined hyperparameter, $Tlog(N)$ is a constant and can be omitted. Then, the objective function becomes

Setting the derivative of $F(\omega,W)$ with respect to $w_{i}$ to zero, we obtain the optimality necessary condition for $w_{i}$, which is

Substituting (15) back into (14), we get the effective loss functional

The minimization of $F^{*}(\omega)$ with respect to $\omega$ is equivalent to min-max of $R_{E}(\omega,W)$ with respect to $\omega,W$. In this paper, the objective function of the proposed model in (16) is named as Importance Sampling loss, ISloss for short.

In this section, we use the logarithmic transformation $logL(x,\omega)$ instead of $L(x,\omega)$ as a loss measure and derive the corresponding objective function. The logarithmic risk functional is

Similarly, as the derivation for ISloss, the corresponding optimization problem becomes

The optimality necessary condition for $w_{log}^{i}$ is

and the effective empirical loss functional becomes

In this paper, (20) is named Logarithmic Importance Sampling loss, LogISloss for short. Equation (20) shows the minimization problem of the logarithmic version $F^{*}$ is equivalent to the minimization of the p-norm loss function with $p={1\over T}$. The objective function of ISloss(16) involves exponential and not stable in training, therefore we use LogISloss(20) in subsequent experiments.

To begin with, let us introduce some notations to facilitate the discussion in this section. Suppose $N$ is the number of training samples and $C$ is the number of classes. $x_{i}\in\mathcal{R}^{d}$ denotes the deep feature of $i$-th sample. For $x_{i}$, we refer $y_{i}$ as the target class and $j$ as the non-target class (the label excluding the ground truth one). $W_{j}\in\mathcal{R}^{d}$ denotes the $j$-th column of the weight $W\in\mathcal{R}^{d\times C}$ and $b_{j}\in\mathcal{R}^{C}$ is the bias term. We fix bias $b_{j}=0$ for simplicity. Suppose $\theta_{j,i}$ is the angle between $W_{j}$ and $x_{i}$. Following [23, 25], we fix the weight $\lVert W_{j}\rVert=1$ by $l_{2}$ normalization and fix the deep feature $\lVert x_{i}\rVert$ by $l_{2}$ normalization and rescale it to $s$. The feature scale $s$ is set to 64 following [23, 25].

ArcFace(Additive Angular Margin Loss)[23] uses the arc-cosine function to calculate the angle between the feature $x_{i}$ and the target weight $W_{y_{i}}$. The model adds an additive angular margin $m$ to the target angle to enhance the inter-class discrepancy and intra-class compactness. The ArcLoss for the $i$-th sample is defined as follows

Suppose $L_{i}^{Arc}$ denote the ArcLoss for the $i$-th sample, and $n$ denote the number of samples in each training batch, then we name the proposed logarithmic importance sampling loss as IS-ArcLoss, which is defined as follows

Here, the superscript IS denotes Importance Sampling. In subsequent experiments for ArcLoss and IS-ArcLoss, the angular margin penalty $m$ is fixed to 0.5 as recommended by [23].

AddMargin(Additive margin)[24], also known as the Large-Margin Cosine Loss[25], calculates the angle between the feature $x_{i}$ and the target weight $W_{y_{i}}$. The model minus an additive margin $m$ to the cosine of the target angle to maximize inter-class variance and minimize intra-class variance. The AddLoss for the $i$-th sample is defined as follows

Suppose $L_{i}^{Add}$ denote the AddLoss for the $i$-th sample, then we name the proposed logarithmic importance sampling loss as IS-AddLoss, which is defined as follows

In subsequent experiments for AddLoss and IS-AddLoss, the additive margin $m$ is fixed to 0.35 as recommended by [25].

In the face verification system, training stage and test stage are different. The training stage is used to learn the deep feature embedding for a given image through a classification task. In the test stage, the score of a given image pair is usually calculated by the cosine similarity between the two feature embeddings. If the score is higher than a given threshold, the input pair is considered a positive pair. If the score is lower than a given threshold, the input pair is considered a negative pair. The given verification threshold is obtained from cross-validation by a pre-defined split in test datasets.

Training Datasets In this paper, we uses the Racial Faces in-the-Wild(RFW) database[15], which includes four races, namely Caucasian, African, Asian and Indian. We use BUPT-Equalizedface[15] as the training dataset which contains 27999 individuals and 1251416 images. The number of individuals in Caucasian, African, Asian and Indian are 7000, 7000, 7000 and 6999 respectively. The detailed statistics of BUPT-Equalizedface are given in Appendix-A. In later discussions, the four training subsets in BUPT-Equalizedface are denoted as RFW-Caucasian, RFW-African, RFW-Asian and RFW-Indian.

Test Datasets The RFW database has four test subsets, namely Caucasian, African, Asian and Indian. There are about 14K positive pairs and 50M negative pairs[15] for each subset in the RFW-test, most of which are easy to distinguish. In this work, the proposed model aims to emphasize large distribution deviations and hard samples. Therefore, we use the RFW-test for evaluation, which is composed of difficult pairs selected based on cosine similarity between embeddings of each pair[15]. Each subset (race) contains 3000 positive pairs and 3000 negative pairs, splited into 10 subsets in advance. We selected the verification threshold for one split based on the remaining nine splits. We propose RFW-extend as an extension to RFW-test and the details of the dataset are shown in Section 4.5. Figure 1 shows selected positive pairs and negative pairs from each race. It can be seen that some pairs are difficult even for human observers.

Experiment Settings The datasets were aligned and preprocessed in MXNet binary format and the size of the cropped image were set to $112\times 112$. We set the training batch size, weight decay and momentum as 128, 5e-4 and 0.9, respectively. The dimension of embedding feature was set to 512 following [23, 25]. The initial learning rate was set to 0.1, and decreased by a factor of 10 at given epochs. For the BUPT-Equalizedface dataset, we trained the network for 27 epochs and decayed the learning rate at epochs 14, 20 and 24. We used half of the suggested epochs in [16] to save training time. We used IResNet-34[23], a 34-layer modified ResNet, as the default backbone in all experiments if not otherwise specified. We followed the Pytorch implementation of insightface¹¹1https://github.com/deepinsight/insightface/tree/master/recognition/arcface_torch on one GTX 1080ti and one GTX 3090ti for training.

In this section, we give a heuristic example to show that the importance sampling weights emphasize the hard samples in the training dataset. We calculated the average classification accuracy for each individual (ID) and highlighted the individuals with the top-10 maximum average weights. We first calculated the cross entropy loss for each training sample then used (19) to calculate the weight. All models in Figure 2 were trained under IS-ArcLoss($T=0.5$). Figure 2 shows that the individuals with maximum weights are those with low accuracy, especially in (c) where the top-10 maximum weights clustered around 0. Hard samples are normally assumed to have low classification accuracy. Therefore, this observation showed the importance sampling weight emphasized hard training samples. We further analyze how the importance sampling weights change over time in Appendix-B. The experiment results indicated the importance sampling weights gradually concentrated on hard samples as the training proceeded.

In this section, we analyze how the temperature $T$ affects the model performance. Figure 3 compares verification accuracy on four RFW-test datasets when $T=0.3,0.5,0.7,1.0$. Figure 3 shows that $T=0.5$ achieves the best accuracy on African, Asian and Indian while $T=0.7$ achieves the best accuracy on Caucasian. One possible explanation is that the distribution deviation between RFW-Caucasian (training) and Caucasian(test) is smaller than the distribution deviations between RFW-Caucasian (training) and African/Asian/Indian(test), therefore, the test Caucasian favors a higher temperature, $T=0.7$. This observation suggests that lower $T$ tolerates larger distribution deviations. In this paper, we used a fixed $T=0.5$ as the default temperature in all subsequent experiments. However, other training schedules on $T$ can be adopted. For example, starting $T$ at a high temperature and decaying it during training. How to choose a good schedule on $T$ for different learning problems will be explored in our future studies.

In this section, we compare the performance of LogISLoss and cross-entropy loss with respect to distribution deviations. Table 1 compares verification accuracy between ArcLoss and IS-ArcLoss. The models were trained on RFW-Caucasian, RFW-African, RFW-Asian, RFW-Indian and tested on Caucasian, African, Asian, Indian. We report the result at the epoch where the training race achieves the highest verification accuracy. The accuracy gain (in small brackets) is the difference between our proposed IS-ArcLoss over ArcLoss. For example, the accuracy of the model trained on RFW-Caucasian and tested on African is 82.23% of ArcLoss and 83.22% of IS-ArcLoss, the accuracy gain is 0.99%. Table 1 shows the following findings. First, IS-ArcLoss performs better than ArcLoss in most cases. Second, the accuracy gains in non-training races are higher compared with the training race on three test datasets(RFW-Caucasian, RFW-African, RFW-Asian). Take the models trained on RFW-Caucasian(first row) for example, the accuracy gain on Caucasian, African, Asian, Indian is 0.6%, 0.99%, 1.12% and 0.63% respectively. The accuracy gain on Caucasian(0.6%) is the lowest. This observation indicates that IS-ArcLoss has better generalization ability compared with ArcLoss. If we assume the accuracy gap is a measure of the distribution shift, then a larger accuracy gap indicates a larger distribution shift. Next, consider the model trained on RFW-African(second row) for example, the test accuracy on Asian, Indian and Caucasian is 84.02%, 88.20% and 89.23%. Under the previous assumption, the distribution deviations between Asian, Indian, Caucasian and African are in increasing order. The accuracy gain on the three datasets is 1.80%, 1.45% and 1.14%, which is in decreasing order. This observation implies that the proposed IS-ArcLoss performs better when the distribution deviation is large. More experiment results on IS-ArcLoss with respect to TAR@FAR and on different backbones are shown in Appendix-C.

Table 2 compares verification accuracy between AddLoss and IS-AddLoss. IS-AddLoss performs better than AddLoss on 3(RFW-Caucasian, RFW-African, RFW-Asian) out of 4 test datasets on non-training races. Another interesting observation is in line with the observation in Table 1. For the models trained on RFW-African, the accuracy gain on Asian(84.85%), Indian(89.85%) and Caucasian(90.85%) is 1.25%, 0.08% and -0.87%, which is in decreasing order. For the models trained on RFW-Asian, the accuracy gain on African(79.62%), Indian(85.22%) and Caucasian(86.75%) is 1.01%, 0.53% and 0.12%, which is in decreasing order. The results also confirm our findings in Table 1 that LogISloss performs better when the distribution shift is large.

Table 3 test LogISloss on several popular benchmarks, including Labeled Faces in the Wild(LFW)[28], Age-DB[29], Cross-age LFW(CALFW)[30], Cross-pose LFW(CPLFW)[31] and Celebrities in Frontal-Profile (CFP-FP and CFP-FF)[32], where LFW, AGE-DB, CALFW and CPLFW have 6000 pairs and CFP(large pose variation) has 7000 pairs. These datasets are also split into 10 subsets in advance. The results were evaluated on the final epoch. Table 3 shows that IS-ArcLoss performs better than ArcLoss in most cases. To be specific, the accuracy gains on LFW and CFP-FF are low, the accuracy gains on CALFW and CPLFW are medium, and the accuracy gains on CFP-FP and AgeDB are high.

RFW-test-extend Dataset In this experiment, we propose an extended test dataset RFW-test-extend to include more pairs for evaluation. The extended dataset has four subsets, namely Caucasian-extend, African-extend, Asian-extend and Indian-extend. The number of positive pairs for the four subsets are 13650,14050,15690,13860 and the number of negative pairs are 42260,43370,54890,43120. All positive pairs and negative pairs are randomly split into 10 folds. The generation and statistics for RFW-extend are shown in Appendix-A.In face verification task, for positive pairs, the similarity between two images should be high, therefore hard positive pairs have low similarity. For negative pairs, the similarity between two images should be low, therefore hard negative pairs have high similarity. Figure 4 shows the similarity of top-100 hard positive and hard negative pairs. The models were trained on the whole BUPT-Equalizedface datasets (27999 IDs) under ArcMargin. Figure 4 shows that IS-ArcLoss has higher similarity for positive pairs compared with ArcLoss in (a)(Caucasian-extend), (c)(Asian-extend), (d)(Indian-extend) and it has lower similarity for negative pairs in all tested datasets. Visualization of the top-2 hardest pairs for each race is shown in Figure 1. This observation indicates that IS-ArcLoss emphasizes the hardest pairs.