Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition

We require HR and LR face image pairs to distill the HR network’s knowledge to the LR network. Following the protocol for LR image generation in super-resolution studies [5, 6, 19, 26], we applied bicubic interpolation to down-sample HR images with 2$\times$, 4$\times$, and 8$\times$ ratios. Gaussian blur was then added to generate realistic LR images. Finally, the downsized images were resized to the original image size using bicubic interpolation. Figure 2 presents sample LR images.

Face recognition network. ArcFace [3] is a SOTA face recognition network comprising convolutional neural network (CNN) backbone and angular margin introduced to softmax loss. Conventional softmax loss can be expressed as

where $x_{i}\in\mathbb{R}^{d}$ is the embedded feature of the $i$-th sample belonging to the $y_{i}$-th class; $N$ and $n$ are the batch size and the number of classes, respectively; $W_{j}\in\mathbb{R}^{d}$ denotes the $j$-th column of the last fully connected layer’s weight $W\in\mathbb{R}^{d\times n}$ and $b_{j}\in\mathbb{R}^{n}$ is the bias term for the $j$-th class.

For simplicity, the bias term is fixed to 0 as in [20]. Then the logit of the $j$-th class can be represented as $W^{T}_{j}x_{i}=\lVert{W_{j}}\rVert\lVert{x_{i}}\rVert{cos(\theta_{j})}$, where $\theta_{j}$ denotes the angle between the $W_{j}$ and $x_{i}$. Following previous approaches [20, 29], ArcFace set $\lVert{W_{j}}\rVert=1$ and $\lVert{x_{i}}\rVert=1$ via $l_{2}$ normalisation to maximize $\theta_{j}$ among inter-class and minimize $\theta_{j}$ among intra-class samples. Further, constant linear angular margin ($m$) was introduced to avoid convergence difficulty. The ArcFace [3] loss can be expressed as

where $s$ is the re-scale factor and $m$ is the additive angular margin penalty between $x_{i}$ and $W_{y_{i}}$.

Attention. Attention is a simple and effective method to guide feature focus on important regions for recognition. Let $\mathbf{f}_{i}=\mathcal{H}_{i}(\mathbf{x})$ be intermediate feature outputs from the i-th layer of the CNN. Attention maps about $\mathbf{f}_{i}$ can be represented as the $\mathcal{A}_{i}(\mathbf{f}_{i})$, where $\mathcal{A}_{i}(\cdot)$ is attention module.

Many attention mechanisms have been proposed; AT [35] simply applied channel-wise pooling to features to estimate spatial attention maps. SENet [13] and CBAM [31] utilized parametric transformations, e.g. convolution layers, to represent attention maps. Estimated attention maps were multiplied with the features and passed to a successive layer. Trainable parameters in attention module are updated to improve performance during back-propagation, forming accurate attention maps. Attention mechanisms can be expressed as

and

where $\mathcal{A}^{c}_{i}(\cdot)$ and $\mathcal{A}^{s}_{i}(\cdot)$ are attention modules for channel and spatial attention maps, respectively.

Features are refined twice by multiplying channel and spatial attention maps in order (3) and (4). Any parametric attention transformation could be employed for the proposed A-SKD, and we adopted the popular CBAM [31] module,

and

where $\sigma(\cdot)$ is the sigmoid function; and $FC(\cdot)$ and $f^{7\times 7}(\cdot)$ are fully connected and convolution layers with $7\times 7$ filters, respectively.

Unlike the conventional knowledge distillation, the network size of teacher and student network is same for A-SKD. Instead, the teacher network is trained on HR images whereas the student network is trained on LR images. Due to the resolution differences, features from both networks are difficult to be identical. Therefore, we propose to distill well-constructed attention maps from the HR network into the LR network instead of features.

where $\rho_{i}$ is the cosine distance between attention maps from the i-th layer of the teacher and student networks; $\langle\cdot,\cdot\rangle$ denotes the cosine similarity; $\lVert\cdot\rVert_{2}$ denotes L2-norm; $\mathcal{A}_{i}(\mathbf{f}_{i})$ denotes the attention maps for the i-th layer features; and $T$ and $S$ denote the teacher and student network, respectively. Thus, $\mathcal{A}_{T,i}(\mathbf{f}_{T,i})$ and $\mathcal{A}_{S,i}(\mathbf{f}_{S,i})$ are attention maps estimated from the i-th layer of the teacher and student network’s features, respectively. Reducing the cosine distance between HR and LR attention maps increases the similarity between them.

Distillation loss for A-SKD is calculated as

which average the cosine distance for channel and spatial attention maps between the HR and LR networks, and sums them across layers ($i=1,2,3,...,N$) of the backbone. $N$ is the number of layers utilized for the distillation.

Total loss for the LR face recognition network is the sum of target task’s loss and distillation loss (8) weighted by the factor ($\lambda_{distill}$). In this work, we utilized the ArcFace loss (2) as a target task’s loss.

Further, our method can be utilized in conjunction with the logit distillation by simply adding the logit distillation loss [12] to our loss function (9). Since logit is the final output of the network, incorporating the logit distillation loss allows the LR network to make the same decision as the HR network based on the refined attention maps.

Datasets. We employed the CASIA [33] dataset for training, which is a large face recognition benchmark comprising approximately 0.5M face images for 10K identities. Each sample in CASIA was down-sampled to construct the HR-LR paired face dataset. For the evaluation, the manually down-sampled face recognition benchmark (AgeDB-30 [23]) and the popular LR face recognition benchmark (TinyFace [1]) were employed. Since AgeDB-30 have similar resolution to CASIA, networks trained on down-sampled CASIA images were validated on AgeDB-30 down-sampled images with matching ratio. In contrast, the real-world LR benchmark (TinyFace) comprises face images with the resolution of 24$\times$24 in average when they are aligned. Therefore, they were validated using a network trained on CASIA images down-sampled to 24$\times$24 pixels.

Task and metrics. Face recognition was performed for two scenarios: face verification and identification. Face verification is where the network determines whether paired images are for the same person, i.e., 1:1 comparison. To evaluate verification performance, accuracy was determined using validation sets constructed from probe and gallery set pairs following the LFW protocol [14]. Face identification is where the network recognize the identity of a probe image by measuring similarity against all gallery images, i.e., 1:N comparison. This study employed the smaller AgeDB-30 dataset for the face verification; and larger TinyFace dataset for the face identification.

Comparison with other methods. Typically, the distillation of intermediate representation is performed concurrently with the target task’s loss. Previous distillation methods in the experiments utilized the both face recognition and distillation loss, albeit face recognition loss of varying forms. In addition, some feature distillation approach reported their performances with the logit distillation loss. In order to conduct a fair comparison, we re-implemented the prior distillation methods with the same face recognition loss (ArcFace [3]) and without the logit distillation loss. Further, our method requires the parametric attention modules for the distillation. Therefore, we utilized the same backbone network with CBAM attention modules for all methods; we combined the CBAM modules to all convolution layers, with the exception of the stem convolution layer and the convolution layer with a kernel size of 1.

Implementation details. We followed the ArcFace protocol for data preprocessing: detecting face regions using the MTCNN [36] face detector, cropping around the face region, and resizing the resultant portion to 112$\times$112 pixel using bilinear interpolation. The backbone network was ResNet-50 with CBAM attention module. For the main experiments, we distilled the attention maps for every convolution layers with the exception of the stem convolution layer and the convolution layer with a kernel size of 1. Weight factors for distillation (9) $\lambda_{distill}=5$. This weight factors generally achieved the superior results not only for the face recognition benchmarks, but also for the ImageNet [2]. Learning rate = 0.1 initially, divided by 10 at 6, 11, 15, and 17 epochs. SGD optimizer was utilized for the training with batch size = 128. Training completed after 20 epochs. The baseline refers to the LR network that has not been subjected to any knowledge distillation methods. For the hyperparameter search, we divided 20% of the training set into the validation set and conducted a random search. After the hyperparameter search, we trained the network using the whole training set and and performed the evaluation on the AgeDB-30 and TinyFace.

Evaluation on AgeDB-30. Table 1 shows LR face recognition performance on AgeDB-30 with various down-sample ratios depending on distillation methods. Except for HORKD, previous distillation methods [22, 35] exhibited only slight improvement or even reduced performance when the downsampling ratios increase. This indicates that reducing the L2 distance between the HR and LR network’s features is ineffective. In contrast, HORKD improved LR recognition performance by distilling the relational knowledge of the HR network’s features. When the input’s resolution decrease, the intermediate features are hard to be identical with the features from the HR network. Instead the relation among the features of the HR network can be transferred to the LR network despite the spatial information loss; this was the reason of HORKD’s superior performances even for the 4$\times$ and 8$\times$ settings.

However, attention maps from the HORKD exhibit similar pattern to LR baseline network rather than the HR network in the Figure 4. HR attention maps are highly activated in facial landmarks, such as eyes, lips, and beard, which are helpful features for face recognition [18]. In contrast, detailed facial parts are less activated for LR attention maps because those parts are represented with a few pixels. Although HORKD boosts LR recognition performance by transferring HR relational knowledge, it still failed to capture detailed facial features crucial for recognition. The proposed A-SKD method directs the LR network’s attention toward detailed facial parts that are well represented by the HR network’s attention maps.

Based on the refined attention maps, A-SKD outperforms the HORKD and other knowledge distillation methods for all cases. AgeDB-30 verification accuracy increased 0.6%, 1.0%, and 1.6% compared with baseline for 2$\times$, 4$\times$, and 8$\times$ down-resolution ratios, respectively. In addition, when A-SKD is combined with logit distillation (KD), the verification accuracy increased significantly for all settings. From the results, we confirmed that the attention knowledge from the HR network can be transferred to the LR network and led to significant improvements that were superior to the previous SOTA method.

Evaluation on TinyFace. Unlike the face verification, the identification task requires to select a target person’s image from the gallery set consists of a large number of face images. Therefore, the identification performances decrease significantly when the resolution of face images are degraded. Table 2 showed the identification performances on the TinyFace benchmark. When the AT [35] was applied, the rank-1 identification accuracy decreased 13.34% compared to the baseline. However, our approach improved the rank-1 accuracy 13.56% compared to the baseline, even outperforming the HORKD method. This demonstrated that the parametric attention modules (CBAM) and cosine similarity loss are the key factors for transferring the HR network’s knowledge into the LR network via attention maps. The proposed method is generalized well to real-world LR face identification task which is not manually down-sampled.

We conducted experiments for object classification on LR images using the 4$\times$ down-sampled ImageNet [2]. For the backbone network, we utilized the ResNet18 with CBAM attention modules. We compared our method to other knowledge distillation methods (AT [35] and RKD [25]) which are widely utilized in the classification domains. We re-implemented those methods using its original hyperparameters. Usually, AT and RKD were utilized along with the logit distillation for the ImageNet; therefore, we performed the AT, RKD, and A-SKD in conjunction with the logit distillation in the Table 4. Training details are provided in the Supplementary Information.

Table 4 shows that A-SKD outperformed the other methods on the LR ImageNet classification task. Park et al. demonstrated that introducing the accurate attention maps led the significant improvement on classification performances [24, 31]. When the attention maps were distilled from the teacher network, student network could focus on informative regions by forming precise attention maps similar with the teacher’s one. Thus, our method can be generalized to general object classification task, not restricted to face related tasks.

Face detection is a sub-task of object detection to recognize human faces in an image and estimate their location(s). We utilized TinaFace [38], a deep learning face detection model, integrated with the CBAM attention module to extend the proposed A-SKD approach to face detection. Experiments were conducted on the WIDER FACE [32] dataset (32,203 images containing 393,703 faces captured from real-world environments) with images categorized on face detection difficulty: easy, medium, and hard. LR images were generated with 16$\times$ and 32$\times$ down-resolution ratios, and further training and distillation details are provided in the Supplementary Information.

Table 5 shows that A-SKD improved the overall detection performance by distilling well-constructed attention maps, providing significant increases of mean average precision (mAP) for the easy (15.72% for 16$\times$ and 7.54% for 32$\times$), medium (14.15% for 16$\times$ and 12.59% for 32$\times$), and hard (33.98% for 16$\times$ and 14.43% for 32$\times$) level detection tasks. Small faces were well detected in the LR images after distillation as illustrated in Figure 6. Thus the proposed A-SKD approach can be successfully employed for many LR machine vision tasks.