Aesthetic Quality Assessment for Group photograph

With the rapid growth of image applications, the traditional image quality evaluation no longer satisfies the practical need. Thus the image aesthetic quality assessment (IAQA) was born. IAQA uses the computer simulation of human perception and cognition of beauty to automatically assess the ”beauty” of images (i.e., Computer evaluation of image aesthetic quality) [Jin et al., 2018c]. It mainly responds to the aesthetic stimuli formed by the image under the influence of aesthetic elements such as composition, color, luminance, and depth of field.

In daily life, we often encounter a situation where we need to take group photographs for souvenirs. So how to estimate the aesthetic values of a group photograph and further provide a guidance system for real-time group photo shooting will become meaningful. The current methods of IAQA mainly focus on the effects of composition, color, light and shadow, depth of field, and other components on the aesthetics of the entire image, which can classify the professional photographs and snapshots, as shown in Figure1(a). Nevertheless, when evaluating the aesthetics of group photographs, people are not only concerned with the above factors but also focus on the state of the person in the image, such as whether someone’s eyes closed, does not look at the camera, the face is blocked, does not smile and other factors. If these factors are not taken into account in the aesthetic quality assessment of group photography, the evaluation will not be accurate. An example is shown in Figure 1(b), the two images assessed by the general method have similar ratings. However, when considering the criteria for group photography, it is clear that we are more satisfied with the first one.

Therefore, we propose a method to assess the aesthetic quality of the group photograph. Firstly, we extracted the texture, brightness, low depth of field, color, and other features commonly used in IAQA. Moreover, we designed seven specific features that conform to group photography experience and principles, such as whether the face in the photo is occluded, whether someone closes their eyes, whether they smile or not. We then constructed a dataset (GPD) specifically for group photography aesthetic quality assessment. It contains a total of 1000 pictures, which are selected from the network, the existing IAQA dataset, and photos taken by ourselves. Finally, with the extracted features, we trained a classifier and a regression model on the GPD-dataset, which are used to classify photos into good and bad categories and predict aesthetic scores, respectively.

To summarize, our main contributions in this paper are as follows:

• 1.

  We design 7 new features related to group photography as the standard for aesthetic assessment.

• 2.

  We build GPD dataset annotated with the aesthetic score, and develop an online annotation system to collect users’ aesthetic evaluation of photographs.

• 3.

  We propose a classifier and a regression model trained by our features, which outperform existing methods in in group photo aesthetic evaluation.

Early research on IAQA focused on low-level visual features and then training classifiers or regressors to evaluate image aesthetics [Luo & Tang, 2008, Datta et al., 2006, Yan Ke et al., 2006, Tong et al., 2005]. In 2004, Microsoft Asia Research Institute and Tsinghua University [Tong et al., 2005] jointly proposed a method that can automatically distinguish the photographs taken by professional photographers from those taken by customers. This work is considered as the earliest research on IAQA. They used a 21-class, a total of 846-dimensional low-level global features to learn the classification model to classify the test images aesthetically. In 2006, Datta et al. [2006] began to use local features for aesthetic assessment, combining the low-level features such as color, texture, shape, picture size and high-level features such as depth of field, tripartite rule, regional contrast and so on which are usually used for image retrieval, and then trained the SVM classifier for the binary classification of image aesthetic quality. Yan Ke et al. [2006] proposed using global edge distribution, color distribution, hue counting, contrast, and brightness to represent the image. Based on these features, the naive Bayesian classifier is trained. All the above works are aesthetic evaluations that are unrelated to content. Since 2010, some aesthetic assessment research related to content has appeared [Jin et al., 2010, Wei Luo et al., 2011, Tang et al., 2013]. In 2014, with the emergence of AVA [Murray et al., 2012], a large-scale aesthetic analysis dataset, significant progress had been made in the automatic aesthetic analysis by using deep learning technology [Jin et al., 2018b, Talebi & Milanfar, 2018, Jin et al., 2016, Kong et al., 2016, Lu et al., 2014]. The classification accuracy rate of ILGNet-Inc.V4 proposed by [Jin et al., 2018b, Talebi & Milanfar, 2018, Jin et al., 2016] ranked first in the world on the ava dataset. In recent years, researchers have mainly studied the problem of IAQA from different tasks. In order to solve the problem of the need for subjective labeling when the image database is established, Ning Ma Ma et al. [2019] proposed a deep attractiveness rank net (DARN) model to learn aesthetic scores. Tian et al. [2015] proposed a query-based aesthetic assessment deep learning model that makes different aesthetic evaluations based on different styles of images. Kim et al. [2018] considers not only objective factors but also subjective factors of user reviews for aesthetic assessment. Schwarz et al. [2018] extended the one-dimensional score to a multi-dimensional aesthetic space score. Ma et al. [2017] proposed an A-Lamp CNN architecture to learn the fine-grained and the overall layout aesthetic assessment simultaneously. In addition, in terms of the prediction of image aesthetic distribution, jinxing et al. Jin et al. [2018a] proposed the method that predicts image aesthetic distribution, opening the direction of aesthetic prediction in the era of deep learning. There are also some research results in the aesthetic assessment of faces and portraits [Redi et al., 2015, Xue et al., 2013, Li et al., 2010]. In the study of group photo images, Rawat et al. [2018] proposed the spring-electric model, which recommended the appropriate station and the proportion of the characters to the photograph. However, they did not evaluate the aesthetic quality from the perspective of the subject. To our best knowledge, there are currently no research on the aesthetic assessment of group photography.

In this section, we will discuss the extraction of features for representing the aesthetic quality of a group photograph. We extract two major groups of features: group photographic features conformed to the group photographic rules and low-level generic aesthetic features proposed in [Datta et al., 2006, Wu et al., 2019, Machajdik & Hanbury, 2010]. These features are combined to obtain better estimates of the aesthetic scores. The following subsections explain each group of features.

The datasets related to photography aesthetics include AVA [Murray et al., 2012], AADB [Kong et al., 2016]. AVA included 250,000 images, each with the corresponding aesthetic classification and rating labels. AADB contains 10,000 images, which has more balanced distribution of professional photograph and snapshot. Each image is annotated with score and eleven attributes. Nevertheless, there is no dataset for aesthetic evaluation of group photography at present. To this end, we collected a group photography dataset GPD by ourselves, which consists of three parts: group photographs shot by ourselves, selected from the existing aesthetic photography dataset, and obtained through internet. GPD contains a total of 1000 group photographs, and each image has been scored. Samples of GPD is shown in Figure 3.

In order to verify the effectiveness of our proposed group photo aesthetic features, we proposed a method whose system flow as in Figure 5. We first construct a group photo dataset, including the image and the label (ground truth), and then perform image preprocessing on all images. On the processed image, we extract the group photograph features and generic aesthetic features, and store them into a vector. After feature extraction, the dataset is divided into a training and test set to training a classifier and a regression. The classifier classifies the photo into two categories: good or bad. The regressor evaluates the image aesthetics with a score of 1 to 10, and finally uses the trained classifier and regression to predict the photo in the test set. We compared the results with the test set label to estimate the accuracy of the classifier and regressor.

Before generic aesthetic features extraction, we preprocess all the images. The processes include: Adjusting the image size to 128 *128 pixels, which can not only retain enough image information but also meet the efficiency of calculation; Converting RGB color space into HSV and LUV color space, some features need to be extracted from these two color spaces; The K-means is used to segment the image according to chromaticity in the LUV color space; The Waterfall segmentation [Marcotegui & Beucher, 2005] is used to segment the image into continuous regions in the HSV color space. On the basis of these image preprocessing, the features are extracted according to the description in Table 1.

Before group photograph feature extraction, we utilized Face++ to detect and save the information of facial recognition, the state of the person’s eyes, the smile degree, the rotation angle of face, the degree of facial blur and the landmarks of face in each image from the GPD. We applied Baidu AI’s face detection tool to detect and save the face occlusion of the person. Based on these information, the group photograph features are calculated according to Section 3.1.

This section shows the effectiveness of our proposed features, and comparison of the performances of our method with other methods, in the specific genre: group photography. Firstly, we used the random forest to obtain the importance of each feature to analyze their impact on assessment. Secondly, we applied k-fold cross validation (k = 10) to split GPD into train set and test set, then trained a classifier using support vector machine (SVM) and a regression model using random forest regression (RF). Finally, we report the performance of this method compared with other methods based on deep learning.

In this work, by analyzing the aesthetic features of group photography, we address the problem that the general method cannot accurately evaluate the group photograph, and introduce group photography features to facilitate investigation of this problem. Furthermore, we construct a group photography dataset (GPD), and built an online annotation tool for collecting the label of GPD. In the experiments, we validated that the proposed method can better evaluate group photography than previous methods that only considered generic features. However, our group photography scene is relatively single. Moreover, there is still a lot of space for improvement in the extraction of group photography features and generic aesthetic features in the future. To further improve the accuracy of the aesthetic evaluation of group photography, much work remains to be done.

This research was partially supported by National Natural Science Foundation of China [Grant No. 61771340,61602344] and Natural Science Foundation of Tianjin, China [Grant No. 18JCYBJC15300].