Theme-Aware Aesthetic Distribution Prediction With Full-Resolution Photographs

Traditional aesthetic quality assessment is based on handcrafted features and shallow classifiers [30, 31]. Ke et al. [2] analyzed several aesthetic factors and connected them with low-level visual features, such as the edge distribution to reflect the simplicity of a photo. Luo et al. [32] proposed to focus on the image foreground and extracting different features to describe it. Tang et al. [3] further stated that the assessment should be based on the content. For example, they designed features, especially for human photos. Many other works [4, 5, 6, 7] are also based on these low-level handcrafted features and may emphasize different features in different situations. Some works [4] also introduced applications based on aesthetic quality assessment such as aesthetic enhancement. In addition to these aesthetic-related features, generic features such as the GIST [33] and SIFT [34] were employed, and good performance was obtained [35]. Although many of these features are carefully designed, their representation power is still limited.

In recent years, DNNs have become widely adopted models in many research areas, and many new techniques [36, 37, 38] based on DNNs have widened the applications of automatic AQA. Lu et al. [39, 40] were the first to assess photo aesthetic quality based on deep neural networks. The authors designed a two-column architecture to learn features on both the global and local views. Kao et al. [41] proposed to use a regression model instead of a classification model because a continuous score can deliver more precise information about aesthetics. Dong et al. [42] used features from an ImageNet pretrained network to predict aesthetic binary labels. Tian et al. [43] designed a query-based model; they trained a network for each query image based on a subset of images that is strongly related to the query. Additionally, Kao et al. [44] proposed a multitask learning model to predict aesthetic labels and semantic labels simultaneously. The authors explained that AQA is strongly associated with image semantics.

Although these works based on DNNs have made great progress, they still suffer from the fixed input size problem. Thus, researchers have proposed methods to solve this problem. Some works [11, 12, 40, 45] proposed to crop multiple fixed-size patches from the original images, and aggregating the features extracted from these patches to predict the aesthetic quality. Ma et al. [12] proposed to select patches according to some criteria based on human perception; therefore, the patches can be more representative. Instead of selecting patches according to predefined criteria, Sheng et al. [46] proposed an attention-based method that dynamically learns weights for different patches. Other works sought to maintain the aspect ratio of input images; they adopted either an FCN [13] or SPP [14] to generate fixed-size features from arbitrary network inputs [16, 15, 47, 17]. To adapt to the deep learning tools that are preferably run on fixed inputs, some used multisize training as an approximation, but only $1.0$ and $1.5$ aspect ratios were taken into account [15, 17]. Apostolidis et al. [16] conducted experiments with a batch size of 1 so that any input size could be accepted. However, the performance of this method is even worse than that of the image transformation methods, possibly due to the unstable training process. Hosu et al. [26] extracted features from ImageNet pretrained models without fine-tuning; thus, they could use the original photos in a two-stage training process. Chen et al.[48] proposed a different thought that uses convolutions of fractional strides to adapt to warped inputs.

In addition to various works on solving the fixed-size input problem, some other researchers have focused on different aesthetic evaluation forms. Most previous works used binary labels. Recently, other frameworks, including the aesthetic score regression, pairwise comparison, and distribution learning, were developed. Kao et al. [41] used a regression model to evaluate images. Lee et al. [27] designed a unified framework based on pairwise comparison to achieve classification, regression and, personalization simultaneously. Similarly, Li et al. [49] also achieved both generic and personalized AQA via a multitask learning and fusion framework. The aesthetic distribution can reflect the diversity and subjectivity of image qualities. Therefore, it attracted the attention of many researchers. Some early works [50] have employed label distribution learning based on a support vector machine. They also used the voting number to represent the reliability of the ground-truth distribution. Recently, many other works predicting aesthetic rating distributions were proposed and there were various loss functions such as the Kullback- Leibler (KL) divergence [17], earth mover distance [28], $\chi^{2}$ distance [51], and cumulative Jensen-Shannon divergence [24]. These works also combined other strategies, such as using semantic information or defining reliability based on the distribution kurtosis.

Pooling is one of the basic components in current deep neural networks. It plays a crucial role in reducing computational complexity. In the original implementations of pooling methods, the stride and the kernel size are manually fixed so that the ratio between the pooling input size and the output size is fixed. He et al. [14] stated out that such a property makes the networks inflexible because the fully connected layers can only accept fixed-size inputs. SPP was designed to enable the pooling layers to generate fixed-size features from arbitrary inputs, and it can also extract multiscale information. This is achieved by using adaptive pooling kernels and strides. Specifically, the pooling size is manually fixed, and the pooling stride, as well as the kernel size, are defined as the ratio between the input size and the fixed pooling size. To incorporate SPP into networks, two types of methods were proposed. One uses a multisize end-to-end training strategy with two predefined sizes 224 and 180. The other uses a two stage training framework without fine-tuning the backbone CNN such that images do not need to be resized.

In both traditional pooling and SPP, pooling is applied to the entire input feature map. In other words, the moving range of the pooling kernel is fixed as the entire input. However, in the object detection task, an object only occupies a part of the image region. To extract features from only object regions and discard other features, RoI pooling was proposed in [18]. RoI pooling is able to extract fixed-size features from regions of arbitrary sizes and locations. This means that the kernel moving range, the kernel size and the stride can be adaptively modified according to different RoIs. RoI pooling was improved to RoI align in [52] to rectify the quantification error. This enhancement significantly improves the detection performance of small objects. RoI pooling (align) has become a standard module in many object detection and segmentation models [53, 54, 55].

SPP and RoI pooling both enable the network to accept input images of arbitrary sizes. To achieve this goal, removing the fully connected layers and using only convolutional layers is also a solution. Such a network named an FCN was first introduced to solve semantic segmentation tasks [13].

Image transformations in traditional DNNs cause mismatches between images and their annotations. Theoretically, SPP and an FCN can handle inputs of arbitrary sizes, but such an ability is not fully utilized because GPU implementations are preferably run on fixed inputs. Therefore, our method aims to remove the image-annotation mismatches while keeping the uniform input size to adapt to current deep learning tools. This goal is achieved by combining image padding with RoM pooling. In summary, padding turns the inputs into the same size, and RoM pooling eliminates the side effects that padding causes.

Specifically, we pad all images in datasets to the same size. It does not matter how the images are padded, so we use the most straightforward method that pads zeroes along the bottom and right boundaries. After padding, the input image becomes the spatially separable combination of two regions: one region is filled with padded values of 0, and the other region is the original image. The region of the original image is called the RoM, and its spatial coordinate range is from $(0,0)$ to $(w,h)$, where $w$ and $h$ are the original image width and height, respectively.

The padded image is then fed into the network and processed by some convolutional layers. The RoM is mapped on the feature maps correspondingly. As a result, the input feature maps of the RoM pooling layer also become a spatially separable combination of a padded region and an image region (RoM). The RoM pooling layer pools features in the RoM to the manually defined size and the features outside the RoM are discarded. Therefore, the pooled features are of the same size, and the extra information introduced by padding is removed.

Formally, the feature at output location $(b,c,m,n)$ of the RoM pooling is given below.

where $b$ is the batch index, $c$ is the channel index, $m,n$ are the spatial locations, $f$ is the RoM pooling input features, $A$ is the output pooled feature maps, and $\Omega_{mn}^{b}$ is the pooling kernel. The pooling stride is the same as the kernel size by default, and the kernel size is defined as the ratio of the pooling input RoM size to the manually defined pooling size $S_{pool}=(w_{out},h_{out})$. The pooling input RoM size is calculated as

where $(w_{b},h_{b})$ is the original size of the $b^{th}$ image in one batch, $\tau$ is the downsampling ratio of the RoM pooling input size to the image padding size, and $\rm{round}(\cdot)$ is the rounding operation. Based on the pooling size $S_{pool}$ and the RoM size $S_{RoM}^{b}$, the kernel size is given as

The kernel moving range needs to be inside the RoM. Taking the top-left corner of the image to be $(0,0)$, we give the exact definition of the RoM pooling kernel as follows.

Note that the kernel $\Omega_{mn}^{b}$ may be different for different input images. The overall procedure is to apply the pooling operation to specific regions (RoM) that only contain features of the original images, as shown in Fig. 3. In the bottom left of the figure, light blue and gray regions denote the features from the RoM and padded region, respectively. The features inside the RoMs are eventually pooled into feature maps of the same size so that the network can be trained in an end-to-end manner.

From the RoM pooling process, we can easily observe the differences between RoM pooling, SPP, and traditional pooling. In traditional pooling methods, the kernel size and the pooling stride are manually assigned regardless of the inputs. Therefore, the pooling size cannot be kept unchanged for different inputs. In SPP, the pooling kernel scans the entire image; thus, we cannot discard the features in specific regions. Only in RoM pooling is the kernel moving range flexible and can the kernel size (pooling stride) be adaptively modified.

Discussion: RoM pooling aims to achieve arbitrary pooling kernel sizes on a tensor; therefore, quantification approximations are introduced on both the region location and pooling stride. Such quantification error causes mismatches of RoMs. He et al. [52] proposed to solve this problem by using interpolation; we call this improved version used in our model RoM align. RoM pooling and RoM align perform nearly the same in our model for two reasons. First, the mismatches of RoMs caused by quantification errors are determined by the downsampling ratio between network inputs and pooling inputs. For the downsampling ratio of $\tau$, the number of mismatched pixels along one dimension is at most $\lfloor{\frac{\tau}{2}}\rfloor$. Therefore, the small downsampling ratio (which will be discussed in Section III-E) brings small mismatches in our model. Second, the mismatches have fewer influences on larger regions. For example, mismatches of 10 pixels may totally change the location of a $10\times 10$ region, but cause nearly no visible differences on a $500\times 500$ region. Since the RoM is the entire full-resolution image in our model, the mismatches have nearly no impact.

It is widely known that image shapes influence human aesthetic perception. For example, images with some specific aspect ratios, such as 16:9, 4:3 and 1:1, are more popular in daily applications [20, 19]. Furthermore, images with extremely large or small aspect ratios may be unpopular. Revisiting our proposed PRP module, we find that although the model learns some knowledge from the original-shape images, the unified pooling output size of the RoM pooling layer still makes some shape information to be lost. This property will exert negative influences on our AQA model. To remedy this shortcoming, we propose a shape-aware (SA) module that encodes and extracts features from image aspect ratios and combines them with visual features. Specifically, we discretize the continuous aspect ratios, and turn the discrete values into one-hot codes. Then two fully connected layers are employed to extract shape features from the one-hot codes. Finally, the shape features are fused with visual features to predict the aesthetic quality. The SA module can be regarded as a patch to fill the loophole of the PRP module, making the model directly utilize shape information to improve the performance. We will introduce the feature fusion module in detail in Section III-D.

In the AVA dataset, images are submitted to the predefined challenges and assessed under specific challenge themes such as “shapes” and “harsh environments”. Different aesthetic criteria are adopted in different themes. We call this phenomenon theme criterion bias. As illustrated in Fig. 2, if people assess an image without its corresponding theme, the results may be inaccurate since the assessment criterion is improper.

Some previous works [3, 10, 17] proposed to assess images based on semantic labels. There are two major differences between semantic labels and themes. The first is the coverage difference. Semantic labels mainly describe objective contents such as “human”, “landscape”, “city” and “night” [3]. The themes cover a wider range. Objects and abstract descriptions, such as “balance”, “affluence” and “second exposure” are included [56]. The second difference distinguishes the two concepts from the view of the information sources. Themes are determined by humans subjective intents while semantic labels are determined by the images themselves and cannot be manually changed. We then use two phenomena to further demonstrate the differences. The first phenomenon is that images with the same semantic labels may belong to different themes. Fig. 4 shows an example in which the two images, both labeled with “dog”, belong to different themes. The second phenomenon is that the themes are sometimes impossible to infer from images, but we can easily know the semantic labels. This is also obvious in Fig. 4. It is nearly impossible to infer the theme “crossing the line” for Fig. 4(a) based on only the image itself.

From the previous analysis, we conclude that semantic labels are sometimes helpless in handling theme criterion bias. Since the themes cannot be obtained from the images themselves, they need to be provided along with the images in the evaluation process. To this end, we introduce a theme-aware (TA) module that fuses theme information with visual features. Specifically, we turn the 1,397 different themes in the AVA dataset into one-hot codes, and two fully connected layers are employed to extract theme features. The theme features are fused with the visual features using an attention-based fusion module. Finally, the fused features are fed into the head layers to predict the aesthetic qualities. The process is illustrated in Fig. 3. The detailed feature fusion process is introduced in Section III-D.

The theme features allow the network to adapt to different themes. This theme-adaptation advantage can be reflected in two aspects. First, visual aesthetic features from different themes are used differently. For example, suppose that two identical images may obtain different aesthetic quality assessments just because of their different themes. In this situation, only theme information can help the model to predict different aesthetic qualities to match the ground truth. Second, different images from the same themes are treated equally. Only visual features can influence the model assessments when the images share the same theme.

In addition to the visual features, two types of extra features are used in our model. The shape features aim to remedy the lost shape information in the PRP module. The theme features introduce theme criterion bias such that the model learns aesthetic criteria more accurately. To effectively fuse the extra features with visual features, we propose an attention-based feature fusion module. The overall architecture of the module is displayed in Fig. 5. Two independent attention layers are used to process shape features and theme features, respectively. We also use a global average pooling layer to pool the visual features directly. As a result, we finally obtain three features, including shape-guided visual features, theme-guided visual features, and pure visual features. The three features are added together as the fusion outputs.

The two attention layers in Fig. 5 are the same. In general, an attention process is formulated as follows [23],

where $Q$, $K$, $V$ are query, key and value features respectively. $d$ is the feature dimension, and $B$ is the positional embedding. In our model, the extra features are regarded as queries $Q$ and the flattened visual features are used as keys $K$ and values $V$. The architecture of the attention layer is illustrated in Fig. 6, where $f_{t}$ and $f_{i}$ are the extra features and the flattened visual features, respectively. In our implementation, we directly use the multihead attention proposed in [23] and set the number of heads to eight. Because the length of the query features $f_{t}$ is one, there is no need for positional embedding. The attention layer turns the spatial dimension of visual features to 1 and can be regarded as a special global pooling layer that assigns different weights to different spatial locations according to the extra features. This module mines the relations between extra features and image spatial features. Therefore, the extra features are utilized more effectively.

We choose Inception-V3 [9] as our backbone network, which is the same as previous works [28, 29, 26]. We insert the RoM pooling layer in Inception-V3 to replace the original first pooling layer. Our model takes the padded images as the inputs. This is different from the original Inception-V3, which resizes all the inputs to $299\times 299$. As a result, the RoM pooling size needs to be modified correspondingly. The RoM pooling size is a hyperparameter and there is no simple method to find an optimal solution. Therefore, we choose the pooling size based on some empirical clues. Our main consideration is to keep the ratio between the first pooling size and the original image size similar to that of the original Inception-V3. However, the size may be different for different images. Therefore, we choose a relatively common image size $600$ in the AVA dataset as the benchmark. As a result, the RoM pooling size $(w_{out},h_{out})$ is obtained by multiplying the benchmark size of $600$ with the corresponding ratio $\frac{73}{299}$ in the original Inception-V3

where $73$ is the pooling size of the first pooling layer in the original Inception-V3.

In the training stage, we calculate the ground-truth aesthetic distribution $\bm{p}^{i}$ for the training image $\bm{x}^{i}$ by normalizing the numbers of votes on all the predefined quality scores.

where $v^{i}_{k}$ is the number of votes for the quality score $k$, and $K$ is the maximum quality score. After obtaining the predicted aesthetic distribution $\widehat{\bm{p}}^{i}$ from our model, we employ the earth mover distance (EMD) [28] between distributions as the loss function. The EMD loss is based on the cumulative distribution density, which works well when the distribution is ordered, as discussed in [50]. It is defined as

where ${\rm CDF}_{\bm{p}}(k)$ is the cumulative distribution of $\bm{p}$ defined as $\sum_{j=1}^{k}p_{j}$. We also choose $r=2$ for its simplicity in optimization.

In the testing stage, we adopt the same procedure as in the training stage. Testing images are padded to the same size and fed into the network. One may argue that using the original images without padding and removing the RoM pooling layer is better, but such a method leads to weak accordance between the testing and training stages. The learned network may not fit such a situation. We test this method and obtain worse results.

We evaluate and compare our algorithm with other AQA algorithms on two widely used datasets, AVA [10] and Photo.net [57]. Both datasets are collected from websites. The aspect ratio distribution is shown in Fig. 7. We can see that both datasets contain various aspect ratios and cover a wide range. Surprisingly, although the two datasets come from different websites and have different scales, their aspect ratio distributions are nearly the same.

All our models are pretrained on ImageNet to accelerate the convergence and fine-tuned on the corresponding data sets. An SGD optimizer with momentum of 0.9 and weight decay of 0.0001 is used to train the network. The learning rate is divided by 2 every 10 epochs, and the model is trained for 40 epochs. We set the initial learning rate as $4\times 10^{-3}$ on the convolutional layers; and for randomly initialized layers, the learning rate is 10 times larger. We test two types of padding implementations and get nearly the same performances. One pads all images to $800\times 800$, the other pads batched images to their biggest size in this batch. The batch size is set as 64. Due to the memory limitation, each GPU can only process our padded images with a maximum batch size of 16. Therefore, we use 4 GPUs to train the model. In case the batch normalization layers learn the statistical information inaccurately under such a small batch size on a single GPU, we use the cross-GPU synchronized batch normalization. Data augmentation is used to ease the overfitting problem. Specifically, we randomly use cropping and horizontal flipping to augment the images. Four types of cropping augmentations are applied, and each crops the height and width of the original image from one of the four corners by $\frac{7}{8}$. In the test stage, we average the results on all the augmented images as the final predictions.

To compare with previous models comprehensively, we evaluate the performance of our distribution prediction model with three different types of metrics as introduced in Section IV-A. In addition, some well-predicted and failed examples are shown and analyzed to better reflect the performance.

We first compare our method with previous distribution prediction methods [15, 17, 24, 28, 29, 45, 49], where [15, 17, 29, 45] use fully convolutional networks or crop patches of fixed sizes from the original images for fixed input size problem while [28, 24] propose new loss functions for the distribution learning problem. The results in Table I show that our method achieves the best performance in five of the six metrics and obtains the second-best results on the Euc metric, indicating the superiority over the competitors.

Then, we calculate the mean score and the standard deviation of the distributions, and compare their performances with previous methods. Among the competitors, GPF-CNN [29], NIMA [28] and WCNN [51] are distribution prediction models, while the rest competitors are regression models. Specifically, [25, 58] introduce extra useful information such as scenes and attributes to guide the aesthetic learning. Some works [48, 45, 29, 26] try to avoid changing the image aspect ratios with different modules, such as fractional dilated convolutions. Other problems, including personality [49], sample weighting [51] and feature fusion [59], are also studied. Although these models are effective, the results in Table II show that our method represents a new state-of-the-art approach. The performance improvement of 0.0287 on the SRCC (mean) is significant. Furthermore, our model obtains more than three times higher SRCC (std.dev) and PLCC (std.dev) than the previously reported best performances [28]. These results effectively prove the merits of our model.

Finally, we calculate the mean scores of the aesthetic distributions and binarize them using a threshold of 5 to perform binary classification. Previous aesthetic quality classification methods mainly employ multi-patch aggregation [11, 12, 46] methods or introduce semantic information [44]. The results in Table III show that our method achieves the third-best performance. Note that our model employs the EMD loss to learn aesthetic score distributions, while A-Lamp [12] and MP-Adam [46] are binary classification models. Thus, our model can be applied to a wider variety of tasks.

The results of the model that replaces RoM pooling with RoM align are also given. Both results are nearly the same, which proves our analysis in Section III-A. For simplicity, we only use RoM pooling in other experiments. We show some distribution prediction results on the AVA dataset in Fig. 8. The blue bins represent prediction, and the red bins represent the ground truth. The results show that our method can precisely predict distributions. Other results are given in Fig. 9, in which we keep the aspect ratio of the displayed images but only show the predicted (ground-truth) mean score and the EMD. Two failure cases are shown in Fig. 10. The figure shows that the ground-truth distributions of the two images are abnormal. They are both nontypical distributions in the AVA dataset, such as Gamma distribution [10]. This may be the reason why the model cannot assess them well.

To validate the effectiveness of each module, we use the methods based on three traditional transformations (the transformations shown in Fig. 1) as the baseline models. The first baseline model resizes images to the same size. Since full-resolution images are used in our method, for a fair comparison, we resize all images to $800\times 800$, which is the largest image size in the AVA dataset. The second baseline model uses padding transformation. We pad all images to $800\times 800$. In both the resizing and padding baselines, to eliminate the influences of other factors, we use the same data augmentation methods and feature map size (the output size of the first pooling layer) as our full model experiments. The third baseline that randomly crops fixed-size patches needs to be combined with image resizing because the size of cropped patches cannot be larger than the smallest image size in the dataset. However, the smallest size in AVA is only $160\times 160$. The input of this size cannot be used in Inception-V3 because the feature maps in the deep layers will be smaller than the convolutional kernels. Moreover, cropping such a small patch from much larger images will cause a very significant information loss, which leads to unfair comparisons. To conduct a feasible and convincing ablation study, we resize images to make the short edges no shorter than 512 while maintaining the aspect ratios. Then, we randomly crop $512\times 512$ patches from all images to train the network. In the cropping baseline, only random flipping augmentation is needed. For simplicity, we choose the SRCC of the mean score and standard deviation, EMD with r=1 and KL divergence as the representative metrics.

There are many factors to be validated. To demonstrate our contributions clearly, we split the experiments into three groups.

In our model, we encode the themes into one-hot codes. The question is whether there are some better choices to encode the themes. In this section, we discuss and validate some other coding methods.

In this section, we analyze the influences of different RoM pooling sizes. Compared with the pooling size $73\times 73$ of the first pooling layer in the original Inception-V3, the RoM pooling size in our model is increased to $146\times 146$ to keep the pooling downsampling ratio similar to the original Inception-V3. To validate the importance of the pooling size, we conduct experiments to test the performances of different pooling sizes.

Theoretically, any variants of the pooling size may change the model performances. However, it is impractical to test all the possibilities. Therefore, we choose 4 representative pooling sizes, including 73, 110, 146, and 192 (the pooling width equals the pooling height). Note that the pooling size 146 is our default model setting. The results are shown in Table VII. We can clearly see that the performances improve when the pooling size is increased from 73 to 146. However, the improvement from 110 to 146 is not large, which means that the performances may be near saturation. Surprisingly, when the pooling size reaches 192, the performances become worse. One possible reason is that the network cannot comprehensively capture the global information from such large feature maps. Specifically, the receptive field of each layer is fixed. If the feature map sizes of the deep convolutional layers are too large, the long-range dependencies, which are important in the AQA task, may not be able to be effectively captured by the networks.

Data augmentation is a common process in deep learning frameworks. The traditional method includes random image cropping and flipping. As discussed before, cropping fixed-size patches from images may change the aesthetic information. Based on this consideration, some previous works in AQA [17, 15] only employed random flipping to augment images. However, one recent work [26] used both image cropping and flipping in data augmentation and reported a performance improvement. The reason for the different results is the different degrees of cropping. Here, we call the operation that crops fixed-size patches as preprocessing cropping and call the operation in literature [26] that aims to augment images augmentation cropping.

Specifically, in preprocessing cropping, because both the size and the aspect ratio differ considerably between images, the cropped patches from different images need to be small. Otherwise, it is impossible to keep the size of patches from different images the same. Small cropped patches result in significant information loss. However, in augmentation cropping, we do not need to keep the size of cropped patches the same. We augment images with proportional crops at $\frac{1}{8}$ of the width and height. There is little damage to the aesthetic information with such a small discarded region near the edges. We give examples of cropping augmentation in Fig. 13. The figure shows that the differences between the cropped image and the original image are slight. In AQA, the assessment from one person always remains unchanged when there are only slight changes in the image. Therefore, the model can capture the consistency of AQA with cropping augmentation. To further validate our conclusion, we conduct experiments on different augmentation policies. As seen in Table VIII, very small and large cropping ratios lead to poor performances.

Photo.net is a small dataset. Because there is no theme information, we only evaluate the model with RoM pooling on padded images. Analogous to the AVA dataset, we also pad images to $800\times 800$. Personal subjectivity may have more significant impacts because the number of voters per image in this dataset is much smaller. This can result in unstable ground-truth distributions. Predicting distributions is harder in such a condition. To the best of our knowledge, there is only one previous work [29] predicting aesthetic distributions on this dataset. They proposed to combine global and local image views to learn aesthetic distribution. Since they did not report the data augmentation strategy, we give results with and without cropping augmentation for a fair comparison. Table IX shows that our proposed method outperforms previous work. We also use this dataset to test the model trained on the AVA dataset to evaluate the generalization performance. Considering that the score ranges are different, the distribution metrics cannot be used. First, we obtain normalized scores between 0 to 1 from the AVA trained model. Then, the normalized scores are transformed to the range of 1 to 7. Finally, we compute the SRCC, PLCC, and MSE on the mean scores. The results are given in Table IX. We can see that the performances are still better than those of previous works.