No-Reference Image Quality Assessment via Feature Fusion and Multi-Task Learning

Before the rise of deep neural networks (DNNs), NR-IQA methods utilize different machine learning techniques (e.g. dictionary learning) or image characteristics (e.g. NSS) to extract the features for predicting the quality score. CORNIA (Ye et al., 2012) used a dictionary learning method to encode the raw image patches to features. CORNIA features later were adopted in some NR-IQA models (Ye et al., 2014; Zhang et al., 2015b; Ma et al., 2017a).

With the progress of DNNs in different applications, more researchers have utilized them for NR-IQA. In (Kang et al., 2014) the authors proposed a shallow CNN for feature learning and quality regression. (Kim et al., 2018) developed a two-stage model that separated into an objective training stage followed by a subjective training stage. In the first stage, they used PSNR to produce proxy scores. Then, they generated the feature maps which were then regressed onto objective error maps. The second stage aggregated the feature maps by weighted averaging and finally regressed these global features onto ground-truth subjective scores. (Pan et al., 2018) proposed a deep learning based model (BPSQM) which consists of a fully convolutional neural network and a deep pooling network. Given a similarity index map, labels generated via a FR-IQA model, their model produced a quality map that model the similarity index in pixel distortion level. Hallucinated-IQA (Lin and Wang, 2018) proposed a NR-IQA method based on generative adversarial models. They first generated a hallucinated reference image to compensate for the absence of the true reference. Then, paired the information of hallucinated reference with the distorted image to estimate the quality score. Although (Kim et al., 2018; Pan et al., 2018; Lin and Wang, 2018) perform well for the NR-IQA task, they all used some sort of the reference image during their training which contradicts the NR-IQA purpose.

There are a few algorithms that attempt to do NR-IQA by leveraging the power of multi-task learning. (Kang et al., 2015; Wu et al., 2014; Wang et al., 2016) designed a multi-task CNN to predict the type of distortions and image quality from the last fully connected layer in the network. (Xu et al., 2016) developed a multi-task rank-learning-based IQA (MRLIQ) method. They constructed multiple IQA models, each of which is responsible for one distortion type. (Ma et al., 2017b) proposed MEON, which is a multi-task network where two sub-networks train in two stages for distortion identification and quality prediction. (Huang et al., 2018) proposed a model that used multi-task learning and dictionary learning for NR-IQA.

Among the aforementioned algorithms, (Wu et al., 2014; Xu et al., 2016) do not train end-to-end and requires multi-stage training. Although (Ma et al., 2017b) is an end-to-end method, the training process is performed in two steps. Existing methods mostly used just one fully connected layer for both tasks. To the best of our knowledge, none of the existing works investigate the effect of feature fusion for NR-IQA task. In this work, we investigate different feature fusion architectures to improve the performance of NR-IQA. While taking advantage of multi-task learning, we use feature fusion from different blocks of the network to estimate the quality score more accurately. Unlike the existing multi-task NR-IQA methods, we use global average pooling instead of fully connected layers which reduces learning parameters. We observe that using fully connected layers can cause overfitting and the network memorizes the training examples rather than generalizing from them.

Given a distorted image $I_{d}$, our goal is to estimate its distortion type (or distortion types) as well as quality score. We partition the distorted image into overlapped patches $I_{k}$. Let $f_{\phi}$ represents proposed network with learnable parameters $\phi$. Given an input image we have:

where ${I}_{k}$ is the input image to our network, and $s$ denotes the regressed quality score. $\boldsymbol{d}$ is a $m\times 1$ vector, where $m$ indicates the total number of distortion types available in dataset. The element of $\boldsymbol{d}$ with maximum value represents the index for the distortion type.

QualNet is not limited by the choice of network architecture, any of state-of-the-art DNNs (Simonyan and Zisserman, 2014; He et al., 2016; Szegedy et al., 2015) can be used as a backbone for our proposed method. However, in order to emphasize the advantage of our method, we choose VGG16 which is a relatively smaller network compared to VGG19, Resnet34, or Resnet50, which are the ones that mostly used in the recent proposed NR-IQA methods (Huang et al., 2018; Lin and Wang, 2018; Pan et al., 2018; Wu et al., 2017). Choosing a large network easily increases the number of parameters and makes the network prone to overfitting instead of learning a better representation.

We show the architecture of our proposed model in Fig. 2. We modified VGG16 network to be used as the backbone. We add instant normalization (IN) layer after each convolution layer. However, in order to not increase the number of learnable parameters we set the trainable parameters for IN layers to be False. Despite most of the existing deep learning based NR-IQA methods that use fully connected layers (FCL) for regressing the quality score, we did not use any fully connected layer in the head of our network, instead, we use global average pooling (GAP). The use of GAP allows networks to function with less computational power and have better generalization performance. In the head of our network, we use $1\times 1$ convolution layer along with GAP layer for each task.

The insight behind our network design comes from two observations. a) Typically, different FCL/sub-networks is used at the last layer of the backbone to separately compute each task. We empirically observe that selecting features from the same layer of the network can reduce the learning capability of that layer for both of the tasks. That explains why previous models needed additional overparameterized FCL/subnetwork in their architectures. Therefore, here we choose features from different layers for each task. b) Inspired by the work done in the field of understanding human visual perception and psychology, we know that human vision has different sensitivities to different levels and types of distortions (Daly, 1992; Legge and Foley, 1980; Watson and Ahumada, 2005). Thus, for quality score prediction task it is important to take advantage of the features used to predict the type of distortion. Fusion among features for tasks that are related can significantly capture the relative information and improve the performance by considering the relative information among all the different tasks. We proposed to fuse the features used for distortion prediction task with the quality prediction features. Using our proposed feature fusion, our method can perform efficiently and effectively well on different datasets without having overparameterized layers for each task.

We use features of the max-pooling layer after $conv4{\_}3$ layer for the distortion prediction task. The output of max-pooling layer has $512\times\frac{W}{16}\times\frac{H}{16}$ dimension; $1\times 1$ convolution along with GAP is used to compute the distortion type. The output of $1\times 1$ convolution has $m\times\frac{W}{16}\times\frac{H}{16}$ dimension, where $m$ is the total number of distortions available in the training data. GAP layer is used to convert the $m\times\frac{W}{16}\times\frac{H}{16}$ feature map to $m\times 1$ vector, which denotes by $\boldsymbol{d}$. In the case of single distortion type, the element of $\boldsymbol{d}$ with maximum value represents the index for the distortion type.

We use features of the max-pooling layer after $conv4{\_}3$ and $conv5{\_}3$ for the quality score regression task. We first regress $conv4{\_}3$ and $conv5{\_}3$ features separately to obtain two coarse quality scores maps. The average pooling layer is used to make sure that the output of $1\times 1$ convolutions after $conv4{\_}3$ has the same dimension (i.e. $1\times\frac{W}{32}\times\frac{H}{32}$) with the output of $1\times 1$ convolutions after $conv5{\_}3$. Finally, we combine the computed quality score maps by concatenating them and send it to a $1\times 1$ convolution and GAP to achieve the final quality score.

The HVS perceive image quality differently based on different distortion types; by concatenating the features from the layer that is used to predict the distortion type we observe that our model can utilize distortion type information for quality score prediction. As shown in our ablation study, using our proposed feature fusion, we achieve the best performance.

Here we describe the training details of the QualNet. Given d and $s$ as the output of our model, we use negative log-likelihood loss (which is simply a cross-entropy loss) and $L2$ loss for optimizing distortion prediction and quality score regression tasks, respectively. In other words, the total loss for our network is defined as:

where $L_{d}$ and $L_{s}$ are the losses for distortion type prediction and quality score regression, respectively. $\lambda$ is a regularization parameter that in our network is set to 1. Eq. (2b) is the criterion that combines softmax and negative log-likelihood loss to train the distortion type classification problem with $m$ classes. In other words, $m$ is the number of distortion types. $c$ denotes class index in the range $[1,m]$ as the target for the input. $d$ is a $1\times m$ vector which represents the output of the distortion type prediction task. In Eq. (2c), $s$ and $g_{d}$ are regressed quality score and subjective human score for the image $I_{d}$, respectively.

In QualNet framework, the sizes of input images must be fixed to train the model on a GPU. Therefore, each input image should be divided into multiple patches of the same size. In our experiment, we choose patch size of $128\times 128$, we set the step of the sliding window to $64$, i.e. the neighboring patches are overlapped by $64$ pixels. We consider the patch size large enough to reflect the overall image quality, we set the quality score of each patch to its distorted images subjective ground-truth score. The effect of different patch sizes is provided in our ablation study. To expand the training data set, the horizontal flip is performed in our training process for data augmentation.

For both tasks, our network is optimized end-to-end simultaneously. The proposed network is trained iteratively via backpropagation over a number of 50 epochs. We set batch size to 1. For optimization, we use the adaptive moment estimation optimizer (ADAM) (Kingma and Ba, 2014) with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999, we set the initially learning rate to $2\times 10^{-4}$. We set the learning rate decay to $0.98$, and it applied after every 3 epochs. We fine-tune our model end-to-end while using pretrained Imagenet weights to initialize the weights of the VGG16 network, the rest of the weights in the network are randomly initialized. Similar to the existing NR-IQA models for all of our evaluations, to train and test the QualNet, we randomly divide the reference images into two subsets, $80\%$ for training and $20\%$ for testing. Then, the corresponding distorted images are divided into training and testing sets so that there are no overlaps between the two. All the experiments are under ten times random train-test splitting operation, and the median SROCC and LCC values are reported as final statistics.

The detailed information for datasets that we use for our evaluation is summarized in Table 1. Specifically, we perform experiments on seven widely used benchmark datasets. For natural images use LIVE (Sheikh et al., 2006), CSIQ (Larson and Chandler, 2010), TID2008 (Ponomarenko et al., 2009), TID2013 (Ponomarenko et al., 2013), LIVE-MD (Jayaraman et al., 2012). For SCIs we use SIQAD (Yang et al., 2015), and for images with different tone-mapping, multi-exposure fusion, and post-processing we use ESPL-LIVE HDR (Kundu et al., 2017a).

Most of the existing NR-IQA designed to predict the quality of natural images. Table 2 shows the obtained performance evaluation results of our proposed algorithm on the LIVE, CSIQ, TID2013, LIVE-MD1, and LIVE-MD2 datasets in comparison with state-of-the-art general-purpose NR-IQA algorithms. As shown in Table 2, our proposed method outperforms state-of-the-art algorithms on several datasets while having a smaller backbone. We believe that this improvement is because of our feature fusing and taking advantage of multi-task learning. Although (Lin and Wang, 2018) achieved the best performance for SROCC on LIVE dataset, it has bigger backbone compared to us (VGG19 vs VGG16). Moreover, as shown in Table 2, our proposed model achieves the highest performance when we average the performances among all the datasets.

Cross-dataset evaluations. To evaluate the generalizability of the QualNet, we conduct cross dataset test. Training is performed on LIVE, and then the obtained model is tested on TID2008 (for comparability) without parameter adaptation. Both quality score regression and distortion type prediction tasks are tested. We follow the common experiment setting to test the results on the subsets of TID2008, where four distortion types (i.e., JPEG, JPEG2K, WN, and Blur) are included, and logistic regression is applied to match the predicted DMOS to MOS value (Rohaly et al., 2000; Sheikh et al., 2006).

The results provided in Table 3 demonstrate the generalization ability of our approach. As shown in Table 3, QualNet outperforms all existing algorithms in terms of LCC. It also achieves comparable results in terms of SROCC. Although (Huang et al., 2018) and (Lin and Wang, 2018) achieved higher results compared to our method, it worth mentioning that they both use more learning parameters in their models. (Huang et al., 2018) and (Lin and Wang, 2018) used Resnet50 and VGG19 as their backbones, respectively. For the distortion type prediction task, QualNet predicted the distortion types of TID2008 images with $92\%$ accuracy while trained on LIVE images.

To validate if our proposed method is consistent with human visual perception, we visualize the feature maps of $conv1{\_}1$ and $conv4{\_}3$ blocks in Fig. 3. The first row in Fig. 3 shows five different distortion types from TID2013, including Local block-wise distortions of different intensity (first column), JPEG (second column), JPEG2000 (third column), JPEG transmission errors (fourth column), and Gaussian noise (fifth column). The second and third rows correspond to the feature maps of $conv1{\_}1$ and $conv4{\_}3$ blocks, respectively. In contrast to recent methods that used the reference images to teach their networks to focus on distorted areas and generate a quality map, Fig. 3 shows that our method automatically learns the distortions and highlight them. The dark areas in images in the second and third rows in Fig. 3 indicate distorted regions. We observe that our proposed model learns to focus on the distortions instead of the content of images. For instance, the last row of Fig. 3 clearly shows that for a noisy image our method captures the noise artifacts instead of the image texture/content.

Most of the NR-IQA algorithms were developed for natural images and they do not typically perform well on SCIs. Here we show that our proposed method not only performs well on natural images, but it also works well for images with other contents.

Unlike natural images, SCIs include diverse forms of visual content, such as pictorial and textual regions. Therefore, the characteristics of SCIs and those of natural images are greatly different. Recently, there have been some metrics that designed specifically for the visual quality prediction of SCIs (Gu et al., 2016b, 2017; Fang et al., 2017; Wu et al., 2019). Here we use SIQAD dataset to evaluate the performance of QualNet on SCIs. The SIQAD dataset includes 980 screen content images corrupted by conventional distortion types (e.g. JPEG, blur, noise, etc). Table 4 provides a comparison between our results and various modern NR-IQA algorithms designed either for natural images (Mittal et al., 2012b; Zhang et al., 2015a; Kang et al., 2015; Bosse et al., 2017; Huang et al., 2018) or specifically for SCIs (Gu et al., 2016b, 2017; Fang et al., 2017; Wu et al., 2019). The results show that our algorithm yields a high correlation with the subjective quality ratings and yields the best results in terms of both LCC and SROCC.

There is a growing practice of acquiring/creating and displaying high dynamic range (HDR) images and other types of pictures created by multiple exposure fusion. These kinds of images allow for more pleasing representation and better use of the available luminance and color ranges in real scenes, which can range from direct sunlight to faint starlight (Reinhard et al., 2010). HDR images typically are obtained by blending a stack of Standard Dynamic Range (SDR) images at varying exposure levels, HDR images need to be tone-mapped to SDR for display on standard monitors. Multi Exposure Fusion (MEF) techniques are also used to bypass HDR creation by fusing an exposure stack directly to SDR images to achieve aesthetically pleasing luminance and color distributions. HDR images may also be post-processed (color saturation, color temperature, detail enhancement, etc.) for aesthetic purposes. Therefore, due to different types of tone mapping, multi-exposure fusion, and post-processing techniques, HDR images can go under different types of distortions that are different from conventional distortions.

To demonstrate the effectiveness of QualNet and its application for NR-IQA in different domains we conduct an experiment where we evaluate the performance of several state-of-the-art NR-IQA algorithms on the recently developed ESPL-LIVE HDR dataset. The images in the ESPL-LIVE HDR dataset were obtained using 11 HDR processing algorithms involving both tone-mapping and MEF. ESPL-LIVE HDR dataset also considered post-processing artifacts of HDR image creation, which typically occur in commercial HDR systems.

Table 5 provides a comparison between our results and various modern NR-IQA algorithms designed either for natural images (Saad et al., 2012; Kang et al., 2015; Xue et al., 2014; Wang et al., 2016; Huang et al., 2018; Bosse et al., 2017) or specifically for HDR-processed images (Chen et al., 2019; Kundu et al., 2017b; Jiang et al., 2017; Gu et al., 2016a). The results show that our algorithm outperforms existing algorithms in terms of both SROCC and LCC and yields a high correlation with the subjective quality ratings.

Although the main focus of this paper is on NR-IQA, in Table 6 we provide the results of the distortion type prediction task for our proposed model. As shown in Table 6, QualNet achieves good prediction accuracy over different datasets.

To investigate the effectiveness of our module and training scheme, we provide a comprehensive ablation study in this section. For each ablated model we train it on the subsets of the LIVE dataset and test it on the subsets of TID2013, where four distortion types (i.e., JPEG, JPEG2K, WN, and Blur) are included.

Fig. 4 demonstrates different network architectures for our ablation study. Fig. 4 (a) shows a model with just quality score regression task (single-task) while using fully connected layers for the regression task; Fig. 4 (b) is the same as Fig. 4 (a) while replacing the fully connected layers with $1\times 1$ convolution along with GAP. Fig. 4 (c) shows the model in a multi-task manner, but both quality score and distortion type prediction are done by using the features from the last layer without any feature fusion. Fig. 4 (d) shows the model in a multi-task manner, while the quality score and distortion prediction are done by using the features from different layers in the network without any feature fusion. Fig. 4 (e) shows the model in a multi-task manner, while feature fusion is performed from the last layer for quality estimation. Fig. 4 (f) is our proposed method.

Table 7 shows the results of different ablated models from Fig. 4. As we can see model-a and model-b achieve the lowest performance comparing to the multi-task models. This proves our hypothesis that using distortion type knowledge along with quality score can improve the results. Moreover, we observe that model-b outperforms model-a; in our experiments, we observe that using fully connected layers while having limited training data will cause the model to overfit to the training data quickly which causes a lack of generality in the test phase. Furthermore, as shown in Table 7, both model-c and model-d achieve very similar results, but worse than model-f. This demonstrates the effectiveness of our feature fusion method. Finally, we observe that using the features from different layers for each task leads to better performance. In Table 7 we also provide the performance evaluation of our model via SGD for optimization, We observe that using SGD can cause the network to converge slower and lead to slightly worse results. Finally we provide results of our model while using different patch sizes as an input. We can see that while patch size of 128 leads to the best results as we move to 64 and 32 the performance degrades more. The main reason for dropping the performance while using smaller patch sizes is that the small patches do not have enough content information to represent the ground truth subjective score of the distorted image. In this paper, we select VGG16 as our backbone to show the effectiveness of our model to other methods that chose deeper backbones. However, as shown in Table 7 (Model-f-VGG-19) using a deeper network (e.g. VGG19) can improve our results even more.