ASD: Towards Attribute Spatial Decomposition for Prior-Free Facial Attribute Recognition

With the significant success of deep learning in computer vision community, deep learning-based methods have emerged in the field of FAR. Liu et al. [9] introduce a large scale benchmark for FAR in the wild, termed as CelebFaces Attribute (CelebA). Meanwhile, a deep learning framework is proposed, which consists of two localization networks (LNet) and an attribute recognition network (ANet) for face localization and facial attribute prediction. Rudd et al. [18] cast facial attribute classification as a multi-task problem, in which the learning of each facial attribute is treated as a different task with joint optimization. Hand et al. [19] design a multi-task deep convolutional network (MCNN) based on the relationships between facial attributes. Specifically, 40 facial attributes are first divided into 9 groups according to the facial spatial locations. Then, the shallow layers of MCNN are shared for all attribute groups, while the deep layers are independent belong to corresponding group. The attribute groups are treated as different tasks, and a multi-task learning paradigm is conducted to improve the final performances of FAR. Han et al. [20] estimate the facial attributes via representing the correlation and heterogeneity of attributes in a multi-task learning framework, in which the heterogeneity means the attribute data type, scale, and semantic meaning. Cao et al. [11] introduce identification information to boost a partially shared multi-task convolutional neural network (PS-MCNN) for FAR. The attribute group-wise representations are shared hierarchically among five CNNs. He et al. [12] conduct a dual-path FAR network to leverage features from the original face images and facial abstraction images., where the facial abstraction images are generated by a generative adversarial network (GAN). Chen et al. [16] introduce group attention learning and graph correlation learning to discover correlations among facial parts, while Multi-scale Group and Graph Network (MGG-Net) is proposed for FAR. Shu et al. [15] design three facial-related auxiliary tasks to learn spatial-semantic relationships between facial attributes, and then the spatial-semantic knowledge are transferred to FAR task.

The above mentioned methods often achieve the competitive performances relying on extra prior information. However, the extra prior information, such as identifications, landmarks, and face parsing masks, might be not available, resulting the limitation of these methods. In this paper, we focus on modeling FAR method without any extra prior information.

Face is composed of different characteristics physiologically, such as eyes, nose, and lips. Therefore, facial representation in facial analysis tasks can be intuitively decomposed into various task-related facial components. Here, we focus on the deep learning-based facial analysis methods with the decomposition. Inspired by hidden factor analysis (HFA) [17], Wen et al. [21] propose a latent factor guided CNN (LF-CNN) to learn the age-invariant deep facial features for age-invariant face recognition. The observed facial features are formulated as a linear combination including the latent factors in terms of identification, age, noise and mean of facial feature. Li et al. [22] propose a twin-cycle autoencoder (TCAE) for action unit (AU) detection based on a self-supervised learning paradigm. The movements are factorized into AU-related and pose-related displacements among a pair of images, in which facial action-related movements can be disentangled from head motion-related movements. Chen et al. [23] propose a semantic component decomposition for face attribute manipulation. Facial attribute is decomposed into multiple semantic components, each corresponds to a specific face region. Alharbi et al. [24] exploit a grid structure-based noise injection to disentangle the latent space of facial image generation. The several aspects of disentanglement achieve fine-grained control over the generated images. Cheng et al. [25] propose a plug-and-play self-adversarial network to decompose facial feature. The network simultaneously removes entire image feature and preserves gaze-related feature, improving the robust performance of facial gaze estimation.

The above methods of facial analysis decompose the face into global-level facial components, ignoring the ambiguity of facial components on the local locations. Here, we propose ASD which focuses on the decomposition of facial attribute components in the spatial dimension based on the spirit of HFA. The attribute components are described via latent factors on the each spatial location to alleviate the spatial ambiguity. It is worth noting that the formulation of AEM is kind of similar to part localization module (PLM) in [26], but there is an obvious difference. PLM only conducts the learnable vectors to decompose attribute-related information from the feature map, ignoring to consider attribute-irrelated information. In contrast, AEM represent the general formulation for the feature map, which is guided with latent factors including attributes, noise and mean of facial feature. Therefore, PLM can be regarded as a special case of AEM.

ASD aims to represent facial attribute components for each spatial location of a facial image via latent factors, alleviating the spatial ambiguity without any extra prior information. AEM is a key idea in the procedure of ASD, which is adapted for deep learning-based FAR framework, as shown in Figure 3. Specifically, given a facial image $X$ whose feature map $f\in\mathbb{R}^{H\times W\times C}$ is first obtained via a CNN-based feature extractor $F$ as follows:

where $\theta_{F}$ denotes the learnable parameters of feature extractor, $C$ is the number of channels and $H\times W$ is the resolution. Then, AEM is modeled to decompose the various attribute components of $f$ in the spatial dimension via attribute-to-location assignment, while the attribute components are integrated via location-to-attribute embedding. The two operations of AEM can be formulated as follows:

where $z$ represents the latent factors, $\phi$ and $\psi$ denote attribute-to-location assignment and location-to-attribute embedding, respectively. Finally, a classifier $P$ is conducted to project the integrated attribute embeddings $g$ to the label-space as follows:

where $w_{P}$ and $b_{P}$ denote the learnable parameters of classifier.

AEM is in charge of decomposing the attribute components via attribute-to-location assignment and location-to-attribute embedding. Attribute-to-location assignment generates an assignment matrix via similarities between feature map and latent factors. The assignment matrix explicitly describes the magnitudes of attribute components on each spatial location, which can be seen as a HFA-like formulation. It is worth noting that we generally formulate the feature map including the components of attributes, noise, and mean of facial feature. Location-to-attribute embedding integrates attribute components from all locations guided with the assignment matrix to represent the attribute embeddings of the entire image.

Since the high correlation among latent factors would weaken the distinctions of assignment magnitudes in $A$, we introduce a regularization, termed as correlation matrix minimization (CMM), to reduce the correlation among $z$. The correlation matrix of $z$ can be obtained as follows:

where $s$ is presented in Equation 6. Then, CMM can be formulated as follows:

where the first term can be omitted obviously, since $s(z_{i},z_{i})\triangleq 1$, and the second term enforces the off-diagonal elements of correlation matrix to 0, decorrelating the latent factors explicitly.

For each attribute embedding, we cast the attribute predictions as multiple binary classification tasks. The $j$-th representation of attribute embedding is projected to a logit value linearly, and then a sigmoid function $\sigma$ is utilized to convert the logit value as a probability $p_{j}$, which can be formulated as follows:

where $w_{P_{j}}$ and $b_{P_{j}}$ refer to the weight vector and bias, respectively. The final prediction $p$ is generated via concatenating the probabilities of attribute embeddings as follows:

where $p\in\mathbb{H}^{D+1}$ is the prediction of $D$ attributes and noise, where $\mathbb{H}$ represents Hamming space.

The loss function of the classification is cross entropy loss which can be formulated as follows:

where $y_{j}$ is the label of the $j$-th attributes. $y_{D+1}$ is set to 0, since the label of noise is not available. The final loss function can be defined as follows:

where $\gamma$ is a hyper-parameter to balance the importance between the classification and the constraint of latent factors.

We conduct experiments on two public facial attribute datasets including CelebA [9] and LFWA [9], which is widely used to evaluate the method of FAR.

CelebA is a large-scale facial attribute dataset containing 202,599 facial images divided into 3 subsets in terms of training, validation, and testing. The numbers of facial images for 3 subsets are 162,770, 19,867, and 19,962, respectively. The facial images are annotated with 40 attribute labels.

LFWA is another popular facial attribute dataset composed of 13,143 facial images with 6,263 for training, 2,800 for validation, and 4,080 for testing. The facial images are also annotated with the same attribute labels as CelebA.

In the experiments, we follow the protocol of CelebA and LFWA that the default training set is used to train our method, while the performance of our method is evaluated on the default testing set.

The experiments are conducted on a work station with NVIDIA RTX 2080Ti GPUs. The proposed method is implemented based on PyTorch deep learning framework. The backbone of feature extractor is ResNet50 [27] pre-trained on ImageNet [28], whose capability of feature extraction has been verified in many computer vision tasks [29, 30, 31, 32]. Following [17, 21], the latent factors $z$ are initialized with the distribution of $\mathcal{N}(0,I)$. In the training phase, Adam [33] is used to optimize the learnable parameters with a weight decay of 5e-4. The total epochs are 80 and the initial learning rate is 3e-4, while the learning rate is reduced with the decay ratio of 0.1 after every 20 epochs. The value of $\gamma$ is set to 2e-2 empirically. The input images are scaled as $224\times 224$ and random flip is used as data argumentation. Particularly, the sizes of batches for training on CelebA and LFWA are 64 and 32, respectively.

To evaluate the effectiveness of the proposed method comprehensively, we design the ablation studies in terms of ASD, AEM, and CMM.

We compare our method with 10 state-of-the-art methods lately reported on CelebA and LFWA, which can be categorized into as prior-based and prior-free methods. The prior-based methods include ARRAIR [10], PS-MCNN [11], HSA [12], DMM [13], SlimCNN [14], SSPL [15], MGG-Net [16], and HFE[35], where prior information, such as landmarks, face parsing masks, and attribute prior embeddings, are utilized in the training phase. CSN [26] and TResNetM [36] are general prior-free methods for multi-label classification task. Here, we implement TResNetM on CelebA and LFWA with the training scheme provided in [36], such as optimizer, learning rate, and training epochs. Besides [36], we list the experimental results of other methods reported from corresponding papers, as shown in Table 4.

For the comparison with prior-based methods, ASD achieves the competitive performances than state-of-the-art methods on both CelebA and LFWA. It can be observed that there is still a performance gap between ASD and PS-MCNN. However, we argue the superiority of ASD is obvious. First, the architecture of ASD is simpler. PS-MCNN consists of five CNN-based networks whose initializations are pre-trained on face recognition tasks, separately. Such stage-based training procedure inevitably increases the complicacy of the implementation. In contrast, ASD is composed of a single CNN-based network and AEM, which can be trained via an end-to-end paradigm. The another advantage of ASD is that the performance of ASD is not relied on extra prior information. In [11], the performance of PS-MCNN without identified information and prior attribute groups would be deteriorated, where the average accuracy of PS-MCNN on CelebA is reduced from 92.98% to 91.15%.

For the comparison with prior-free methods, the performance of ASD is superior than CSN and TResNetM. The reason for the low performance of TResNetM might be that the larger batches should be conducted, like the setting in [36]. To come up with a fair comparison, we gradually adopt the various sizes of batches for TResNetM trained on CelebA. The experimental results are listed in Table 5. It can be seen that the improvements of TResNetM caused by increasing the size of batches are slight, while there is a significant growth of GPU memory costs in the training phase. Here, ASD achieves a good trade-off between average accuracy and computational burdens, implying the architecture of ASD without bells and whistles.

The limited training data case is a latest challenge for FAR, where only a small ratios of training samples can be utilized in the training phase. To investigate the performance of ASD with limited training data, ASD is compared with other methods including DeepCluster [37], JigsawPuzzle [38], Rotation [39], FixMatch [40], VAT [41], and SSPL [15].

Following [15], we select the samples randomly from the training set with various small ratios (0.1%, 0.2%, 1%, and 2%), constructing the limited training set for ASD. The experimental results of each ratio are the mean of 10 times, while the limited training sets are repartitioned in each trial. The performances of ASD with the different ratios of limited training sets are reported in Table 6. The experimental results show the the superior performance of ASD comparing with other methods, revealing the reliable capability of ASD with the limited training data case for FAR.