Distilling Knowledge from Object Classification to Aesthetics Assessment

We consider a typical IAA model [27] that estimates the aesthetic rating distribution directly from an image. Particularly, given the $i$-th image $\mathbf{I_{i}}$ from an IAA dataset, the IAA model $\mathcal{M}_{\theta}(\cdot)$ predicts the aesthetic rating distribution ${\mathcal{\hat{D}}_{i}}$:

A direct way for obtaining the model $\mathcal{M}_{\theta}(\cdot)$ parameterized by $\theta$ is to directly optimize its parameter $\theta$ towards the ground-truth (GT) aesthetic rating distribution ${\mathcal{D}_{i}}$:

where the most commonly-used loss function for $\mathcal{L}(\cdot)$ is earth mover distance (EMD) loss [27]:

where ${{\rm CDF}{{}_{{y}}}}$ and ${{\rm CDF}{{}_{{\hat{y}}}}}$ are cumulative density function for GT distribution $y$ and predicted distribution $\hat{y}$ of length $n$, respectively.

However, this approach overlooks the abstract nature of the aesthetic labels $\mathcal{D}_{i}$. As we previously discussed, IAA can be regarded as a process that maps different combinations of semantic patterns into different aesthetic levels. On the one hand, during inference, the IAA model is required to relate various combinations of semantic patterns to the same aesthetic label. On the other hand, when training, it would be hard for the IAA model to learn to distinguish different combinations of semantic patterns merely with the supervision from aesthetic labels since aesthetic labels are not directly related to any specific contents. To make up for the abstractness of aesthetic labels, one instant way is to assign an extra semantic label to each of the training samples for IAA. Because each semantic label is a direct description of the image contents (e.g., themes), the IAA model can learn about semantic patterns related to each semantic label along with the IAA objective, and aesthetic features covering more image contents can be constructed from the semantic patterns learned from semantic labels.

Nevertheless, there are several problems with semantic labels: 1) it requires extra human efforts to assign semantic labels to each of IAA training samples; 2) it is hard to define what semantics in the image will be relevant to the downstream IAA task, and therefore, it is hard to find a standard for introducing semantic labels. Thus, our goal is to find a better representation that provides semantic guidance, so that the IAA model can learn about semantic patterns relevant to IAA for constructing more discriminative aesthetic features for a large variety of image contents:

where $\mathcal{S}_{i}$ and $\mathcal{\hat{S}}_{i}$ are the GT and predicted representation for semantic guidance, $\mathcal{D}_{i}$ and $\mathcal{\hat{D}}_{i}$ are the GT and predicted aesthetic rating distributions, and $\mathcal{L}_{s}(\cdot)$ and $\mathcal{L}_{a}(\cdot)$ are loss functions for semantic and aesthetic guidance, respectively. Here, semantic guidance is provided by imposing an extra supervision to the training objective.

As discussed in Sec. III-A, we aim for an extra supervision for semantic guidance besides extra semantic labels. Since POC models can recognize a vast variety of image contents, our idea is to distill knowledge from POC models on semantic patterns relevant to IAA. To cover a large variety of contents, we can take multiple POC models trained on different datasets to provide sufficiently diverse semantic patterns. And the semantic patterns can be taken from the GSFs of the selected POC models, as we earlier described in Sec. I. By combining GSFs from different POC models, we can obtain representations that can describe a large variety of contents:

where $|$ denotes the concatentation operation that combines pooled GSFs $\bar{f}_{k}$ from $K$ different POC models, resulting in the combined feature $f_{C}$. For extracting features from different POC models, we adopt multi-layer spatial pooling (MLSP) [7] as the pooling strategy to cover both low-level and high-level semantic information in the resulting feature, which is denoted as $\bar{f}_{k}=MLSP(f_{k})$, where $f_{k}$ is the raw GSF from the $k$-th POC model.

However, directly supervising the IAA model by the combined GSFs is ineffective, since not all semantic patterns represented by the combined GSFs are relevant to IAA. We further describe the relationship between the combined GSFs and aesthetic features by Figure 4. As shown, red, green, and blue circles are sets of patterns that can be captured by different POC models. Since these models are trained with different datasets, different models may be sensitive to a different set of patterns. However, considering different POC models trained on different datasets may share similar categories of semantic patterns, these sets also have overlapped portions. For example, for classifying the same bird, some models may tend to classify by the beak, while some other models may tend to classify by wings. Nevertheless, it is possible that all models tend to classify an object as bird when they see the object with feather is in the sky. However, since these POC models are not trained for IAA, the patterns contained in these sets may not all be relevant to IAA. As we show in Fig. 2, when GSFs contain semantic patterns not relevant to IAA, GSFs may match two visually-similar images but with distinct aesthetic scores. In Figure 4, we present the patterns relevant to IAA as yellow.

Therefore, we propose a KD method based on a knowledge distiller trained separately. To be specific, we train an IAA model as the knowledge distiller with the combined GSF $f_{C}$. As shown in Figure 5, the knowledge distiller is constructed with a batch normalization layer followed by three linear layers with ReLU activations. To train the knowledge distiller, we use EMD loss (Eq. 3). The unified whole of selected POC models and the trained knowledge distiller can also be viewed as a teacher model (upper-stream of Figure 3), and the single-backbone end-to-end model (lower-stream of Figure 3) is then viewed as the student model that learns to imitate the teacher. Thus, the knowledge distiller $\mathcal{C}(\cdot)$ predicts a feature $f_{t}$ and an aesthetic rating distribution $\mathcal{\hat{D}}_{t}$ from GSFs given by different POC models:

where $\{f_{t},\mathcal{\hat{D}}_{t}\}$ are deemed as teacher knowledge, including a teacher aesthetic feature $f_{t}$ and a teacher prediction $\mathcal{\hat{D}}_{t}$.

Thus, the teacher knowledge is directly used for supervising the student model. Accordingly, we formulate the KD loss for training the student model from Eq. 4:

where ${\mathcal{\hat{D}}_{s}},{f_{s}}$ denote the student prediction and the student aesthetic feature, $EMD(\cdot)$ and $MSE(\cdot)$ refer to EMD loss and mean squared error (MSE) loss respectively. Accordingly, $f_{t}$ here is expected to provide semantic guidance in the context of IAA, and the student model is also expected to summarize how the teacher predicts $\mathcal{\hat{D}}_{t}$ from $f_{t}$. The architecture of the student model follows a succinct design (lower-stream of Figure 3), which is constructed with a single CNN backbone followed by a fully-connected (FC) network for aligning the student aesthetic feature to the same size as the teacher aesthetic feature. The aesthetic labels are finally predicted from the student aesthetic feature by an FC softmax layer.

Note that in Eq. 7, the teacher aesthetic feature $f_{t}$ is constructed from GSFs, and the student aesthetic feature $f_{s}$ is constructed from semantic patterns relevant to IAA captured by the student’s own backbone. And we hypothesize that the teacher aesthetic feature $f_{t}$ are more discriminative than the student aesthetic feature $f_{s}$ for effective semantic guidance. Generally, to prepare more discriminative teacher aesthetic features, the POC models for constructing the teacher model should be selected according to the student model. For a student model with a known backbone, we could expect the constituent POC models of the teacher model should: 1) deeper than the student’s backbone; or 2) trained with more data than the student’s backbone. By combining multiple POC models deeper or trained with more data, more diverse semantic patterns are expected to be produced than the student’s pre-trained backbone and more discriminative aesthetic features are expected to be created. The points above are guidelines for selecting POC models that are likely to produce more diverse semantic patterns than the student’s backbone. Note that since the effectiveness of a POC model also relies on the training strategy and the quality of the training data, we are not able to appropriately select POC models merely according to the parameter size and training data size. Thus, the criterion to verify whether the teacher aesthetic features are more discriminative than the student aesthetic features is to directly compare the performance of the teacher model (i.e., knowledge distiller) to the student model without KD. When the student model without KD performs poorer, it means that its aesthetic features are less discriminative than those of the teacher. As long as the teacher model performs better than the student model, the student model can learn to construct better aesthetic features from the teacher, and the selection of POC models for the teacher model is appropriate. The points above will be experimentally discussed in Sec. IV-B and Sec. IV-C.

As shown in some previous works [7, 30], resizing input images can harm the effectiveness of the IAA model due to loss in high-resolution details. However, using high-resolution images can introduce large computational costs. To cope with the trade-off between input resolution and computational costs, the proposed KD scheme is also designed to allow the student model to adapt to smaller input sizes. In training the teacher model, we adopt full-resolution inputs for feature extraction, which allows semantic patterns on high-resolution details to be preserved in the teacher knowledge. While for training the student model, we adopt resized inputs and encourage the student model to excavate high-resolution details from the resized image with the teacher knowledge. Experimental evidences are given in Table VII.

To answer the question “does the supervision from the teacher model better than ground-truth aesthetic labels”, we conduct ablation studies to investigate different settings of the proposed KD scheme. Specifically, we firstly investigate whether a single backbone is capable of learning teacher knowledge and adapting to smaller input sizes under the proposed KD scheme. To this end, we set up baseline models trained directly with GT aesthetic rating distributions in the situation when the ResNeXt101 (SWSL) backbone is trainable or non-trainable. Then we introduce the proposed KD scheme as comparisons. The results are given in Table V.

As the results suggest, without the proposed KD scheme, similar results are obtained for both trainable and untrainable backbone settings. This suggests that even we give the backbone the freedom of learning on new semantic patterns for more discriminative aesthetic features, the IAA model has not been improved. This supports our motivation that aesthetic labels are too abstract to guide the neural network to learn about semantic patterns. As a result, the IAA model can only learn to create aesthetic features with semantic patterns already-known to its pre-trained backbone, and merely learn new semantic patterns for improving the discriminative power of the resulting aesthetic features. Since our teacher model also constructs aesthetic features from semantic patterns already-known to the backbones of the selected POC models, if we introduce extra POC models deeper or trained with more data than the pre-trained backbone of the student model, it is expected that the teacher model will construct more discriminative aesthetic features than the student model. By encouraging the student aesthetic features to be close to the teacher aesthetic features in training, the student model is guided to capture more relevant semantic patterns for more discriminative aesthetic features as its teacher (Fig. 4).

By introducing the proposed KD scheme, both trainable and untrainable backbone settings present substantial improvements in performance. The results have the following implications: 1) when the backbone is set untrainable, the student model with KD learns to construct more discriminative aesthetic features with semantic patterns already-known to its pre-trained backbone; 2) when the backbone is trainable, the backbone of the student model with KD learns relevant semantic patterns for more discriminative aesthetic features. This confirms the effectiveness of the proposed KD scheme.

The KD loss (Eq. 7) can be further divided into feature supervision (term 2 of Eq. 7) and output supervision (term 1 of Eq. 7). We detail our experiments to show the contribution of each individual term in the proposed KD loss. We set the whole network trainable in this experiment, and the model is directly supervised by GTs when the output supervision is off. The results are presented in Table VI. As the results show, both terms substantially contribute to the performance of the student model, and the student model achieves the best performance when both terms are adopted.

Note that the student model takes 300$\times$300 resized inputs while it is supervised by teacher knowledge distilled from full-resolution inputs. As discussed in Sec. III-C, this means we expect the student model to be adaptive to the smaller-sized inputs (learn full-resolution knowledge on semantic patterns from resized inputs). To confirm, we also train a student model with almost full-resolution (640$\times$640) inputs as a comparison. Specifically, we first pad all input images to 800$\times$800 (where 800$\times$800 is the largest size in the dataset), and center-crop them into 640$\times$640 to maintain the aspect ratio. The results are presented in Table VII. The results show that the proposed KD method also enables the student to be adaptive to smaller-sized inputs, which allows the performance drop brought by resizing (640$\times$640 $\rightarrow$ 300$\times$300) to be decreased from 2.6% to 0.6% in SRCC (0.756$\rightarrow$0.736 vs. 0.775$\rightarrow$0.770), which leads to FLOPs to be saved by 77.2% (134.7 $\rightarrow$ 30.6). This suggests that the trained student model is adaptive to lower image resolution and change of aspect ratio after learning with teacher knowledge.

To show the impact of the proposed KD scheme on the performance of the student model on different categories of images, we further present the results on specific categories of the baseline model without KD and the student model with KD. The results are presented in Table IX. As the results show, compared to the overall improvement 4.8% on SRCC, the improvement brought by the proposed KD scheme can reach 7.2% for specific categories. This further confirms the effectiveness of the proposed KD scheme.

To answer the question “how much improvement has been made in terms of efficiency, and how much effectiveness is compromised, by comparing the student to the teacher model”, we compare the efficiency and effectiveness between teacher and student models. The results are presented in Table VIII. Compared to the teacher model, the student model with 640$\times$640 input size saves FLOPs by 95% (2940.5 $\rightarrow$ 134.7) with 2.4% drop in SRCC (0.794 $\rightarrow$ 0.775). By shrinking the input size to 300$\times$300, the FLOPs is further saved by 77.2% (134.5 $\rightarrow$ 30.6) with only 0.6% loss in SRCC (0.775$\rightarrow$0.770).

To answer the question “how much improvement has been made by the teacher and the student model compared to previous methods”, we compare the teacher model and the student model with previous relevant methods [6, 31, 37, 27, 34, 32, 7, 9, 33, 36]. We take their reported results for the comparison, and the results are presented in Table X. Among these contenders, the reported results of DMA-Net [6], MNA-CNN [31], Zeng et al.’s method [34], APM [37], Hou et al.’s method [9], GPF-CNN [32] and Hosu et al.’s method [7] are evaluated on the train-test split from [29], while PA_IAA [33] and HLA-GCN [36] are evaluated on the train-test split from [43]⁹⁹9https://github.com/BestiVictory/ILGnet/tree/local/data/AVA1. We experimentally observed that the results on the split from [43] typically have a higher accuracy while a lower SRCC compared to the results on the split from [29] (see more detailized discussions in Sec. V-A). To fairly compare our method with the contenders, we separately compare the performance results of our method on the same train-test split as the contenders. Note that the reported results of NIMA have evaluated neither on the split from [29] nor on the split from [43]. Therefore, we re-implement NIMA and evaluate on the split from [29]. For the comparison on the split from [29], the results are given in Table X; for the comparison on the split from [43], the results are given in Table XI.

For our implementation, the ResNeXt101-based setting adopts the SWSL version [40], and the ResNet50 and the ResNet18 based settings utilize ImageNet pre-trained versions¹⁰¹⁰10https://github.com/Cadene/pretrained-models.pytorch as the student model backbones. As shown in the results, the teacher model based on combined GSFs significantly outperforms the selected contenders in terms of SRCC, PLCC and accuracy. Specifically, the relevant method by Hosu et al. [7] based on InceptionResNet-v2 MLSP features, which performs best in terms of SRCC and PLCC among all contenders, is substantially surpassed by the teacher model by 5%. Additionally, the proposed KD scheme helps to significantly improve the model efficiency while preserving the model effectiveness. As shown in Table X and Table XI, though a performance drop can be observed comparing the teacher model to the student model, the student model based on ResNeXt101 still outperforms the contenders in terms of SRCC, PLCC and accuracy, including the most recent methods [36, 33, 9]. Comparing the student model to other end-to-end models among the contenders, our student model achieves 0.770 in SRCC, which is 7.1% higher than the best performed end-to-end models among the contenders (Zeng et al.’s method [34] in Table X ).

As mentioned in Sec. IV-E, performance variations can be observed when the same model is evaluated on different train-test splits. As a very relevant topic, image quality assessment (IQA) methods [44, 45, 46, 47, 48] typically conduct K-fold cross-validation or take the average or median results evaluated on a number of different random train-test splits (a.k.a. sessions) to reduce performance variations as much as possible. However, for the problem of IAA, the adopted datasets by IAA typically have a much greater scale than IQA datasets. For example, the commonly-adopted IQA dataset LIVE-IQA [49] contains only 779 images. On the contrary, the commonly-adopted IAA dataset AVA benchmark contains $\sim$255,000 images. This essential difference on dataset scales makes previous works on IAA mostly evaluate the methods only on a fixed train-test split [6, 31, 37, 27, 34, 32, 7, 9] rather than several random splits. However, evaluation on a fixed split potentially causes worries about generalizability (e.g., the method only works on a certain split). Additionally, several versions of train-test splits have been adopted in previous works (as mentioned in Sec. IV-E), which potentially cause problems of unfairness when comparing the results on different train-test splits. Therefore, in this section, we discuss whether the effectiveness of the proposed KD scheme is an outcome of random variations, and whether the effectiveness of the proposed KD scheme only works for a certain train-test split.

To begin with, we want to find out the actual range of performance variations brought by different splits. Such variations can be estimated by directly train and test the same model on different splits. However, due to the randomness in the training of deep learning models, even with the same split, if we separately train a model with the same setting, the results are possibly different. Therefore, we need to separately consider the performance variations brought by train-test split difference alone, and the performance variations brought by randomness in training. That is:

where $\delta_{exp}$ is the performance variations observed from experiments that evaluate the same model on different train-test splits, $\delta_{split}$ is the actual performance variation brought by using different train-test split, and $\delta_{training}$ is the performance variation brought by randomness in training. To evaluate the scale of $\delta_{split}$, we first estimate the scale of $\delta_{training}$ by running the proposed teacher model 10 times on the same split. In Fig. 7, the results show that the variations (difference between the highest and the lowest results) caused by randomness in training are around 0.003 for SRCC, PLCC and accuracy.

Then we estimate the scale of performance variations brought by using different train-test splits. To this end, we adopt both fixed split evaluation and cross-validation with random splits. For fixed split evaluation, we apply the split from [29] and the split from [43], which both left 20,000 images for testing. For cross-validation with random splits, we also remain 20,000 images for testing. Since there are $\sim$255,000 images in total, twelve cross-validation splits with non-overlapping testing sets are prepared. Then we evaluate the results when a single-backbone end-to-end IAA model is cooperated with or without the proposed KD scheme. The results are presented in Fig. 6. As the results demonstrated, the proposed KD scheme introduces performance gains for all selected student backbones across all train-test splits. For the ResNet18-based model, the ranges of improvements are [0.011,0.024], [0.010,0.023], [0.5%,1.7%] for SRCC, PLCC, and Acc, respectively. For the ResNet50-based model, the ranges of improvements are [0.018,0.026], [0.015,0.041], [0.4%,1.3%] for SRCC, PLCC, and Acc, respectively. For the ResNext101-based model, the ranges of improvements are [0.019, 0.038], [0.017,0.036], [0.8%,1.7%] for SRCC, PLCC, and Acc, respectively. All improvements are larger than the random variation 0.003, and thus, the improvements on all experimented models on all splits are regarded as significant.

The proposed KD scheme is designed to provide semantic guidance to the training of an IAA model. Two alternative approaches for introducing semantic guidance are to adopt semantic labels as auxiliary inputs, or train the model to predict semantic labels as an auxiliary task. To compare the effectiveness of the proposed KD scheme with the alternative approaches for introducing semantic guidance, we take the semantic labels provided by the AVA dataset [29] (two labels per image and 66 distinct semantic labels in total) for building models that are compared with the model trained with the proposed KD scheme. Therefore, the following four models are built for the comparison:

Since it is a widely accepted opinion that an aesthetically-pleasant image should be able to lead the attention of the observer to the main subject [50], semantic patterns for aesthetic features should be relevant to the subject area. Therefore, a more explainable way for measuring the performance of a deep IAA model is to evaluate the capability of a deep IAA model on extracting main subjects from aesthetically-pleasant images. To this end, we collect a number of aesthetically-pleasant images and manually label their main subjects. Then for different IAA models, we can derive the predicted main subjects from their class activation maps and compare the accuracy of extracting main subjects across different models.

Specifically, we collect those aesthetically-pleasant images from the AVA testing set under the split from [29]. Images with average scores greater than 6.5 are regarded as aesthetically-pleasant images. Then main subjects are firstly detected with a salient object detection model [51]. We find that some types of images, such as landscapes, may not have a clear main subject. For these cases, it may be hard to conclude a definite salient object detection result. Therefore, we firstly select images whose detected main subjects occupy 10%$\sim$30% of its area. Then we manually remove images with poor salient object detection results. Finally, 504 images with salient object masks are prepared, where the salient object masks are regarded as GT labels for the main subjects. Note that for those removed cases, the IAA model trained with the proposed KD scheme is still able to locate their most appealing part as demonstrated in Fig. 9. While these cases do not have a commonly-accepted guideline to outline their main subjects, and therefore, these cases are not included in the evaluation of this section.

The model trained with the proposed KD scheme is then compared to the version trained without KD and the selected POC models in terms of locating the main subjects. Some selected cases are given in Fig. 8. We firstly extract layerCAMs [28] from the selected models. Then masks for the main subjects are derived from the layerCAMs by setting the 70 percentile of the layerCAM as the threshold for segmenting the regions of the main subjects. We can observe from Figure 8 that: 1) different POC models tend to have different focuses, implying different POC models have sensitivities to different semantic patterns; 2) the IAA model without KD captures semantic patterns less relevant to the subject area, which supports our claims that aesthetic labels are too abstract to guide the deep model to learn about semantic patterns for constructing aesthetic features; 3) the IAA model with KD supervised by knowledge on semantic patterns tend to have a better focus on subject areas than the version without KD, implying that KD allows the student model to capture semantic patterns more relevant to the subject area; 4) the focused subject areas of the IAA model with KD are further improved from those of the POC models, which implies that the knowledge distiller can build aesthetic features upon GSFs.

Besides qualitatively analyzing the impact of teacher knowledge on the student model, to confirm the judgments in qualitative analysis, we further quantitatively evaluate the performances of the selected models on extracting main subjects with the collected dataset. To this end, we measure mean intersection-over-union (mIoU) between the masks derived from layerCAMs and the GT masks. Since the mIoU results are varied subject to different segmentation thresholds, we plot the change of mIoU subject to segmentation thresholds from 5 to 95 percentile of the layerCAMs. According to Fig. 10, from best to worst, the performance on main subject detection can be ranked as: IAA ResNeXt101 with KD $>$ selected POC models $>$ IAA ResNeXt101 without KD. This further confirms our judgments concluded in the qualitative analysis.

The extra cost of the proposed KD scheme is brought by: 1) extracting GSFs from multiple POC models; 2) training the knowledge distiller; 3) distilling teacher knowledge from GSFs; 4) computing losses between teacher aesthetic features and student aesthetic features when training the student model. To compare the scale of these extra computational costs to the computational cost for training a baseline IAA model, the costs are estimated and listed in Table XIII. Specifically, the scale of the training cost is measured as three times of the testing costs for the same model with the same input [52]. Since all the above-mentioned four parts that bring extra computational costs can be conducted separately, the overall extra computational costs are computed by linear combination. In Table XIII, to show the total extra cost brought by the proposed KD scheme, we present the costs for feature extraction with POC models (Setting 1, 2, 3), training the knowledge distiller (Setting 4), and distill teacher knowledge with the knowledge distiller (Setting 5). Compared to the forward-propagation or backward-propagation via the models, the computational costs for loss computations are negligible, and therefore, costs for loss computations are not listed in Table XIII. Therefore, the total extra cost per input is the sum of the costs brought by feature extraction with POC models, training the knowledge distiller, and distilling teacher knowledge with the knowledge distiller. As to the cost for training the end-to-end IAA model with or without KD (Setting 6, 8), their computational costs are the same since the negligible cost for computing the KD loss is the only difference that brings the extra cost for the setting with KD. For the cost for testing the end-to-end IAA models with or without KD (Setting 7, 9), the costs are the same since their settings are the same in the testing phase.

As it can be seen from Table XIII, the total extra cost is 2944.2 FLOPs(G) while the training cost for an end-to-end IAA model is 1101.2 FLOPs(G) (Setting 6, 8). This means that the total extra cost is $\sim$1.7 times higher than training an end-to-end IAA model. Do note that the total training cost (Setting 6, 8) is a multiple of the number of epochs, while our setting adopts a rather fewer number of epochs in training (i.e., 12 epochs), which makes the extra cost seemingly higher. While it is common to use a few tens of epochs in training an end-to-end IAA model. For example, Li et al.’s method [33] adopts 50 epochs in training. If our model is also trained for a few tens of epochs, the extra cost will be less than the training costs for an end-to-end model. On the contrary, among all three parts within the extra cost (Setting 1$\sim$5), only the cost brought by training the knowledge distiller is a multiple of the number of epochs (Setting 4), and other parts of the extra cost will stay constant. Since the cost brought by training the knowledge distiller is on a scale of 1, increasing the number of epochs for training the knowledge distiller does not significantly increase the overall extra cost. With the extra cost, the model performance can be effectively improved by 4.8% (in Table VIII), and such an improvement is generalizable (in Fig. 6). Therefore, we believe that the proposed KD scheme is a rather cost-effective way for improving the performance of an end-to-end IAA model.

A better IAA model requires more discriminative aesthetic features, and more discriminative aesthetic features require more diverse semantic patterns.

For the teacher model, using POC model with larger sizes, using POC model with more training data, or combining GSFs from different POC models can significantly improve the teacher models’ performance (Table I and Table II). The main reason is that all above-mentioned approaches can provide more diverse semantic patterns that potentially provides more relevant patterns for constructing aesthetic features covering more contents. As a result, more discriminative aesthetic features further lead to performance gains.

For the student model, the performance improvement brought by KD can be explained by the semantic patterns learned from the teacher. As we experimentally show in Sec. IV-B and Sec. IV-C, both the teacher model and the student model trained merely with aesthetic labels construct aesthetic features from semantic patterns captured by their own pretrained backbones. Since the selected POC models for the teacher model can capture more semantic patterns than the student’s backbone, the teacher aesthetic features are expected to cover more contents than the student aesthetic features. Thus, teacher aesthetic features can be used for extra supervision that guides the student model to capture more relevant semantic patterns for constructing aesthetic features covering more contents, which leads to higher IAA performance. Besides ablation studies on IAA in Table V, results on detecting the subject areas (Fig. 8 and Fig. 10) also imply that the IAA model learns to capture more IAA-relevant semantic patterns (related to the subject areas).

In this section, we further investigate some variants of the proposed KD loss along with the proposed method.