Multi-task deep CNN model for no-reference image quality assessment on smartphone camera photos

The convolution layer of proposed DCNNS adopts a similar but smaller deep CNN architecture like VGG-16 [18]. The six convolution layers all use $3\times 3$ filter size with stride as 1 and padding by 1, which lead to two $64\times 64\times 32$ layers, two $32\times 32\times 64$ layers, and two $16\times 16\times 128$ layers. Max pooling of $2\times 2$ is performed twice to reduce the feature map resolution from $64\times 64$ to $16\times 16$. After the final convolution layer, apply $16\times 16$ global average pooling to flatten feature maps to a 128-dimension fully connected layer as output.

In the quality score prediction task, the convolution output layer connects to two 256 fully connected (FC) layers, then regress to an overall quality score. The scene detection task has the same structure as that of the quality task, except for the last layer, which is composed of four neurons to represent the probabilities of different scene types. Dropout of probability 0.5 is applied to the two 256 FC layers to prevent overfitting. All the layers use ReLU as activation functions.

Because the two tasks share the same model parameters in the convolution layer, the learned feature maps could become more scene-relevant and better model the image quality.

We observed that there are typical types of scenes in smartphones taken photos, such as outdoor nature scenes, daylight buildings, indoor facilities, and night scenes, etc. Different scenes have different characteristics and affect human’s perceptual quality judgment. It is essential to provide the scene information as a clue for quality prediction during training. For categorizing scene types, images are horizontally and vertically divided by 8, forming 64 sub-blocks, then mean and standard derivation of each sub-block is calculated and split into 16 bins histogram, accumulating through all sub-blocks. To detection different kinds of edges, we apply edge filters of MPEG-7 image descriptors [19], as shown in Figure 2 on each sub-block and calculate the percentage of edge pixels, then accumulate across all sub-blocks. As a result, we extract a 37-dimension feature vector for each image and utilize the unsupervised K-means algorithm to cluster images into four types of scenes.

The clustering result provides the scene type label for the scene detection task. Figure 3 shows some sample images of the categorization result. We can see that essential photography characteristics such as contrast, brightness, and texture are distinguishable in each scene type.

Let $x_{n}$ and $y_{n}$ denote the input patch and its quality label. $f(x_{n};w)$ is the function of $x_{n}$ and network weights $w$ that predicts the quality score. The loss function of quality task $Lq$ is defined as follows:

We apply softmax function on the probability of scene category and adopt cross-entropy as loss function $L_{s}$, where $f^{\prime}(x_{n};w^{\prime})$ is the function of $x_{n}$ and network weights $w^{\prime}$ that predicts scene type. Network weights $w$ and $w^{\prime}$ share the same convolution layer parameters except for FC layers. We define $L_{s}$ as follows, where $y_{n}^{(s)}$ denotes the scene label of $x_{n}$ from clustering:

The overall loss function is defined as $L=L_{q}+\alpha L_{s}$. We weight the scene detection loss function $L_{s}$ by $\alpha=1.0$ to guide the training process to learn more scene-relevant features. When we train the two tasks simultaneously with the combined loss function $L$, the shared convolution layer parameters are updated in a way that improves the quality prediction and scene detection accuracy at the same time, as shown in Figure 4. Each scene type’s detection accuracy varies under four quality aspects, ranging from 0.28 to 0.58 and depending on different scene’s characteristics. Although the averaged scene type detection accuracy is around 0.41, the auxiliary task can be thought as a weak classifier to boost the performance. Table 1 shows the scene type detection task’s accuracy for each quality aspect at the corresponding best epoch.

For training, we split the SCPQD2020 database into 80%-20% parts as training and validation sets. We employ Adam optimizer with default settings in PyTorch to optimize our network parameters with learning rate 0.001 and batch size 128. We train one CNN model for each quality aspect with 50 epochs. Each model spends about 1.5 hours to complete the 50 epochs on an NVIDIA GeForce RTX 2080 Ti GPU with 11GB RAM.

The SCPQD2020 dataset [4] is composed of 1,500 photos taken from 100 scenes using 15 smartphones, covering a wide range of prices and different manufacturers. The dataset includes various challenging scenes like outdoor nature scenes, indoor lowlight scenes, backlight scenes, and night scenes. For every image of the same scene, three professional photographers are recruited to rate subjective quality scores based on four aspects: texture, color, noise, and exposure. The exact quality scores are not provided in the dataset but the quality ranking of 15 images within a scene is disclosed, ranging from 1 to 15. Lower-ranking means better quality.

The quality prediction result of each quality aspect is sorted and compared with ground truth subjective ranking using the Spearman Rank Order Correlation Coefficient (SROCC). SROCC mainly reflects the consistency between two ranking distributions and is defined as:

where $n$ is the number of images and $d_{i}$ is the rank difference between the subjective score and the objective prediction of the $i$-th image.

We randomly split the dataset into 80%-20% parts as the training and the testing sets. Each quality aspect has a separate prediction model, and we train our model with 50 epochs and select the model with the highest SROCC, repeating the processes three times to find the average score. We use the same training sets to investigate the competing NR-IQA methods BRISQUE, NIQE, and CNNIQA to report their averaged SROCC values. Since the work from Yao et al. is not publicly available, we quote the performance number from their paper [16] as a reference. Table 2 shows the SROCC comparison.

As we expected, the traditional NR-IQA methods do not perform well since they’re designed for synthesis distortions not for smartphone camera photos which are original raw images without distortion. When compared with a CNN-based NR-IQA model, CNNIQA, our proposed DCNNS model excels at both deeper network architecture and the scene auxiliary task that assists to learn more scene-relevant features. Although it may not be a head-to-head comparison, we quote the performance result from Yao to indicate that a deeper network with multi-task learning can achieve similar or superior performance, as their approach is compound with saliency detection, feature extraction, and color space conversion.

Although we use scene type detection as an auxiliary task to assist more accurate image quality prediction, the scene type concept comes globally from all image parts and sometimes from image context. As shown in Figure 5, the image 55 is a night scene from its context, but clustered as scene 2 image-wise, given it’s the scene with balanced lighting and contains both smooth and complex region.

Since we label the scene type and predict the image quality on image patches of size $64\times 64$, the scene type identified globally may not fit to each individual patch from different image regions. During the training process, some patches from specific region (Figure 5b) are classified as scene 3, the scene with less contrast and under exposure, while the other region (Figure 5c) are classified as scene 0, the night scene. The labelled ground truth scene 2 violates what the neural network learned from other training samples and keep penalizes the optimization process with loss function $L_{s}$ in equation (2). As a result, the image 55 has the lowest average scene detection accuracy for every quality aspect and lower SROCC scores. We present some representative images’ SROCC scores and scene detection accuracies in Table 3.

On the other hand, we investigate some images with overall high scene accuracy. From Figure 6, the image 85 is labelled as scene 1, the scene with complex texture and balanced exposure, achieves average scene accuracy 0.7146 and results to above average SROCC as 0.5130. Another image 47 of scene 3, scenes with less contrast and under exposure, also has high scene accuracy as 0.6747, which leads to significant higher SROCC 0.6231 than average. Detailed performance numbers are listed in Table 3

From the analysis of SROCC v.s. scene accuracy, we demonstrated that the scene accuracy does have high impact on the image quality prediction, which also echos the performance boost from the help of auxiliary task. As we pointed out the night scene detection challenge, how to effectively fuse the global image-wise feature v.s. the local patch-wise feature to correctly identify scene type will be a key to further improve smartphone photo quality assessment.