Quality Evaluation of Arbitrary Style Transfer: Subjective Study and Objective Metric

Style transfer is a process that strives to render natural images with particular style characteristics from one image (e.g., the style image) while synchronously maintaining the detailed structure information of the content image. Such unique technique not only builds a bridge between the computer vision and appealing artworks, but also gets rid of the dilemma that it would take a long time for a well-trained artist to draw an image in a special style [1]. As shown by an example in Fig. 1, style transfer model can automatically generate a new stylized image based on the content and style of an image. Additionally, style transfer also plays an important role in many computer vision tasks, such as person re-identification [2], semantic segmentation [3], and image reconstruction [4].

As customary, style transfer is commonly cast as the study of the texture generation [5]. Early works [6] cope with the texture generation using local statistics or similarity measures on the pixel values. Recently, the field of Neural Style Transfer (NST) was ignited by the groundbreaking work of Gatys et al. [7], which is the process of using Convolutional Neural Network (CNN) to perform image translation and stylization. Then, lots of follow-up studies were conducted on the NST algorithm based on the deep neural network in order to promote the transfer efficiency and generation effects. Meanwhile, NST also has rapidly evolved from single-style to infinite-style models, also known as Arbitrary Style Transfer (AST) [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. With the ability of utilizing one model to transfer arbitrary artistic style, AST has become a hot topic on computer vision, which could promote the creation of artworks and social communication. However, lacking of a good quantitative evaluation makes it difficult to measure the merits and drawbacks of the AST algorithms. It is therefore necessary to study and eventually evaluate the AST images.

Mathematically, the style transfer model can be described as a translation process [21]:

where $I_{A}$ is the input content image from a source domain A to a target domain B, $\mathcal{M}_{A\longmapsto B}$ is a mapping that generates image $\emph{I}_{AB}\in\emph{B}$ indistinguishable from style image $\emph{I}_{B}\in\emph{B}$ on the target domain given the input source image $\emph{I}_{A}\in\emph{A}$.

Following the growth of neural networks, numerous NST methods have been proposed to study the problem of style transfer. According to the optimization ways adopted in the NST task, the existing NST methods can be divided into two categories: Image-Optimization-Based Online Neural Methods [8, 9] and Model-Optimization-Based Offline Neural Methods [12, 13, 14, 15, 16, 17, 18, 19, 20]. The former category could produce appealing stylized results through the iterative image optimization process, while the latter uses generated models with feed-forward networks to produce special style patterns. Particularly, considering the efficiency issue, it is more meaningful to design the Model-Optimization-Based Offline Neural Methods in practice.

The AST method is one of the Model-Optimization-Based Offline Neural Methods, which can accept an arbitrary artistic style as input and produce stylized results in a single feed-forward network once upon the model is trained. Thus, the AST has received substantial attention due to the increasing scientific and artistic values. In below, we review the details of the AST studies. For more comprehensive introduction, readers can refer to the survey [1].

1) Non-Parametric Methods: The common idea of the non-parametric methods [10, 11] is to seek the similarities between the patches in content and style images and swap them. Chen et al. [10] realized AST for the first time that developed a Style-Swap operation to swap the feature patches of content images with the best matching style feature patches. Another work by Gu et al. [11] also proposed a patch-based method considering the matching of both global statistics and local patches. However, if the structures of content image and style image are largely different, these methods cannot efficiently protect the shape with unsatisfactory style patterns.

2) Parametric Methods: The characteristic of these parametric methods [12, 13, 14, 15, 16, 17, 18, 19] is to optimize a target function that reflects the similarity between the input and stylized images. These parametric methods [12, 13, 14], utilize the Gram-based VGG perceptual loss to produce stylization with a few modifications. Li et al. [12] proposed a linear transform function (LST) from content and style features for stylization. Li et al. [13] performed a pair of feature transforms, whitening and coloring (WCT), for feature embedding within a pre-trained encoder-decoder module. Huang et al. [14] proposed a novel adaptive instance normalization (AdaIN) layer that adjusts the mean and variance of the content input to match the style input. Another promising trend in parametric methods is to integrate attention mechanism into the deep neural network. Yao et al. [15] first considered multi-strokes with self-attention mechanism. Parket al. [16] introduced Style-Attentional Network (SANet) to match content and style features for achieving good results with evident style patterns. Denget al. [17] proposed a multi-adaptation module that takes the global content structure and local style patterns into account. In addition, Zhang et al. [18] introduced a multi-modal style transfer (MST) via efficient graph cuts algorithms, which explicitly considers the matching of semantic patterns in content and style images. Inspired by MST, Chen et al. [19] developed a structure-emphasized multimodal style transfer (SEMST) model, which can flexibly match the content and the style clusters based on the cluster center norm.

1) Motivations: Notwithstanding the current state-of-the-art methods have shown successful stream in style transfer, arbitrary style transfer image quality assessment (AST-IQA) has been a long-standing problem and relatively unexplored in the community. Nevertheless, with the exception of a few quantitative protocols [22, 23], almost all researches evaluate the stylization quality in a qualitative way (e.g., by side-by-side subjective vision comparisons or different user studies), which suffers from the following limitations. First, the stylization examples displayed for qualitative comparison are limited in number and often carefully selected to favor the cases where the algorithm works well. In other words, the results of these presentations are not comprehensive enough. Second, the selected observers often lack sufficient experience and expertise in art, which makes qualitative evaluation less convincing. Thus, it is practical and necessary to propose a reliable metric to quantitatively assess the stylization quality.

2) Challenges: Different from the traditional IQA tasks [24, 25, 26, 27, 28, 29] that usually focus on general distortions generated by various stimuli, the AST-IQA is closely related to aesthetics and poses serious challenges in both subjective and objective assessment.

Subjective assessment: The first challenge is how to design and conduct a human subjective study that can obtain reliable ground truth labeling on a set of stylized images [30]. To our best knowledge, there is no publicly available database for AST-IQA. The most related benchmarks are the non-photorealistic rendering (NPR) benchmarks [31, 32], which are used for testing stylization algorithms without human opinion scores. Currently, there is no quality assessment standard for measuring the performance of style transfer, since the AST-IQA is a highly subjective task, e.g., different subjects tend to have various ideas towards the same stylized result, especially for the style evaluation.

Objective assessment: Once the subjective dataset is obtained, the next challenge is how to design a metric that can automatically evaluate the perceptual quality of the AST images closely consistent to human vision. In several image style transfer works [21, 33, 34], some Full-reference (FR) metrics (e.g., SSIM [35]) have been used to evaluate the similarity between the structures of the content and stylized images. However, strictly speaking, evaluating the AST quality is not a classic FR-IQA task. Straightforwardly applying the traditional FR-IQA strategy to the field of style transfer is problematic, since the stylized image has different content detailed information with the source and style images, and the ground truth image is unavailable for the stylized image (i.e., completely different with fidelity evaluation in the tradition IQA tasks). Although the general No-reference (NR) metrics (e.g., NIQE [28], TCLT [36] and BRISQUE [37]) have made great progress in the tradition IQA tasks without ground truth, they are also not applicable to AST-IQA because stylized images are closely related to aesthetics rather than naturalness. In addition, it is necessary for AST-IQA to fully take the original information of content and style images into account. Recent works target to address these challenges using some objective metrics from the perspective of different quality factors. Yeh et al. [22] proposed a metric with two factors (i.e., effectiveness and coherence) where the former factor is a measure of the extent to which the style was transferred, and the latter is a measure of the extent to which the transferred image is decomposed into the content objects. Wang et al. [23] first decomposed the quality of style transfer into three quantifiable factors, i.e., the content fidelity (CF), global effects (GE) and local patterns (LP), which cover the main aspects considered by different types of existing NST methods. However, these metrics either focus on the limited factors of style transfer quality (e.g., lacking of fine-grained quality factors), or are simple in quality pooling, which cannot effectively match the aesthetic perception of human observers in practice.

In this paper, to resolve the above challenges, we carry out an in-depth investigation for AST-IQA from both subjective and objective perspectives. With the popularization of art education, a majority of people with similar background can make similar perceptual judgments about some basic elements in artistic painting (e.g., color tone, brush stroke, distribution of objects, and contents). Benefitted from the above conditions, to address the challenge in subjective assessment, we decompose the quality of AST into three quality factors that are easier to understand, namely content preservation (CP), style resemblance (SR), and overall vision (OV). These quality factors are assigned own preference labels by participants according to the knowledge in style transfer, intuition in vision and feed-back from the surveys. To address the challenge in objective assessment, as suggested by the recommendation system [38], we regard the problem as a data-driven modeling of user preference [39], and conduct quantitative evaluation of AST-IQA using sparse representation to dig intrinsic representation for content and style images. To sum up, the major contributions of our work are summarized as follows:

1) To carry out in-depth studies on perceptual quality assessment of AST stylized images from both subjective and objective aspects, we build a new AST images database named AST-IQAD, which consists 150 content-style image pairs and the corresponding 1200 stylized images produced by eight typical AST algorithms. Each stylized image contains the subject-rated CP, SR and OV scores. To our knowledge, it is the first large-scale AST image database with human opinion scores. Therefore, it can serve as a benchmark to objectively evaluate the existing AST methods and potentially guide the design of different AST methods.

2) We proposed a new sparse representation-based image quality evaluation metric (SRQE) for AST-IQA, which can quantitatively evaluate the quality factors of CP, SR, and OV. To be more specific, in the training phase, we learn multi-scale style and content dictionaries to represent the style characteristic and structure of the stylized images. In the quality estimation phase, the sparse feature similarities are further exploited to compute the qualities of CP and SR respectively, and the OV quality is obtained by combining the SR and CP qualities. Extensive experiments are conducted on our AST-IQAD dataset and the experimental results demonstrate the proposed method can well evaluate the AST quality.

The rest of this paper is organized as follows. Section II illustrates the details of AST-IQAD. Section III introduces the proposed method in detail. The experimental results are shown and discussed in Section IV. Finally, conclusions are drawn in Section V.

Since the essence of style transfer is to migrate the color tone and stroke pattern from the source to target image while retaining the content structure information of the target image. To provide deeper and intuitive information, the selected source images should have clear and reasonable structures. Thus, we set up a hierarchical taxonomic system (shown in Fig. 2) for source image (the content images) and target images (the style images), respectively. Both content and style images are labeled with five categories.

1) Content Images: We collect 75 high quality images with a resolution of $512\times 512$ pixels from the NPRgeneral benchmark [31] and other famous photography websites. According to the criteria of coverage [31], the content images are comprised of five categories (i.e., animal, portrait, building, nature, and daily life) with a wide range of characteristics (e.g., contrast, texture, edges and meaningful structures). Examples of the selected content images in the database are shown in Fig. 3.

2) Style Images: We select 126 style images from some artwork websites and WikiArt. WikiArt is the largest art encyclopedia in the visual arts from all over the world. Since these original style images are not provided with high resolution, all style images are set to the same resolution of $512\times 512$ in line with the content images. These style images cover five categories including contemporary art, modern art, renaissance art, Chinese art, and others. Examples of style images in the AST-IQAD dataset are also shown in Fig. 3.

3) Content-Style Image Pairs: Pairing content images with appropriate and diverse style images can make the AST results more aesthetically pleasing. In our work, we provide two ‘Paired’ and ‘Unpaired’ mechanisms [40] for each content image, in which ‘Paired’ means that the content and the style images are semantically consistent (e.g., the same source of birds), while ‘Unpaired’ means that the content and the style images are the representations of different sources (e.g., the style images may be regular patterns or texture decorations). In total, we use 75 content images and 126 style images (including reused style images) to generate 150 content-style image pairs. Then, eight AST algorithms are conducted on the content-style image pairs to generate the AST images. More information of the image pairs can be found in our database.

Different with traditional IQA databases that stimulated with different distortion stimuli, the testing images in our databases are generated from different AST methods. The eight representative AST methods used in the database are listed in Table I, including AAMS [15], AdaIN [14], MANet [17], LST [12], WCT [13], MST [18], SEMST [19], SANet [16]. These algorithms cover a wide variety of techniques, including Gram-based, Attention-based, Graph-based and Cluster-based methods. As a result, we can obtain 1200 style-transferred images from 150 content-style image pairs. As shown by the examples of style-transferred images in Fig. 4, we have the following observations: 1) In the Gram-based methods, LST [12] and AdaIN [14] can well preserve the content information but may suffer from wash-out artifacts. On the contrary, WCT [13] is impressive in color and texture making the “painting taste” more intense while fails to preserve the main content structures. 2) The attention-based methods have distinct content structures and rich style patterns but may produce unpleasing visual artifacts. For example, SANet [16] and MANet [17] methods produce unpleasing eye-like artifacts, and AAMS [15] introduces imperceptible dot-wise artifacts. 3) The stylized results of MST [18] and SEMST [19] are similar and produce both visible content and proper stylization.

Due to the different rating standards across different ob-servers and the influence of visual content [41], the subjective quality scores evaluated by absolute category rating are imprecise, biased, and inconsistent, while the preference label, representing the relative quality of two images, is generally precise and consistent for the task. For this consideration, we adopt the pairwise comparison (PC) approach which aims to provide a binary preference label between a pair of stylized images.

The experiment was carried out in an indoor laboratory with low ambient illumination calibrating in accordance with the ITU-R BT.500-13 recommendations [42]. The subjective software interface (as shown in Fig. 5) is displayed at a 23-inch true color (32bits) LCD monitor with the screen resolution of $1920\times 1080$ pixels. The ratio of background illumination behind the display to the image peak luminance is 0.15. The viewing distance is approximately six times the image height. In the interface specific to the AST task, participants are simultanesously shown two stylized images along with a style-content image pair, and are required to vote on three preferences for content preservation (CP), style resemblance (SR), and overall vision (OV). The detailed descriptions of preference criteria are summarized in Fig. 6.

A total number of 45 subjects aged from 18 to 30, including experts and students from the Faculty of Art, and under-graduate students with experience in image processing, were participated in the subjective study. For each content-style image pair producing eight AST results, we have 28 pairwise comparisons in the subjective ranking study. In total, 4200 pairwise comparisons from 150 content-style pairs are involved in the subjective study. To reduce the possible fatigue effect, we divide the experiment into three sub-sessions (each sub-session contains 15 subjects), in which each participant would take part in one sub-session completing 1400 PC voting. The content images presented in each sub-session included most of the scene types, and were displayed in a random order to reduce the possible memory interference. After completing every 200 pair comparisons, each subject was encouraged to look away to relax their eyes and asked to rest for about 15 minutes. Each participant took about six hours (including rest periods) to complete the whole subjective experiment. As a result, we can get 63,000 votes on each subjective evaluation.

1) Global Ranking of AST Algorithms: To derive global ranking of the AST algorithms from the corresponding PC results, we adopt the Bradley-Terry [43] model to estimate the subjective score for each algorithm. The probability that the i-th method is favored over the j-th method is defined as:

where $\emph{u}_{i}$ and $\emph{u}_{j}$ are subjective scores prefered for the i-th and the j-th methods, respectively. Then, the negative log-likelihood for the B-T scores $\textbf{u}\in{{\mathbb{R}}^{{{n}_{ast}}}}$, where $\emph{n}_{ast}$ is the number of AST algorithms, can be jointly expressed as:

where ${{\mathsf{\mathcal{W}}}_{ij}}$ is the (i, j)-th element in the winning matrix $\mathcal{W}\in{{\mathbb{Z}}^{{{n}_{ast}}\times{{n}_{ast}}}}$, representing the number of times that the i-th method is preferred over the j-th method. By setting the derivative of $\mathsf{\mathcal{L}}(\mathbf{u})$ in Eq. (3) to zero to solve the optimization problem [44], the final B-T scores u are obtained via zero mean normalization, served as the ground truth subjective rating scores.

We can get a B-T score for each AST result in each sub-session by applying the B-T model. Fig. 7 shows the average B-T scores of different AST methods for three subjective evaluations (i.e., CP, SR, and OV). A higher B-T score indicates a better performance. From the figure, some interesting observations could be drawn: 1) The B-T scores of different AST algorithms show various trends on the subjective evaluations of CP, SR and OV, which indicate that different methods have specific advantages. 2) For the CP evaluation, LST [12], which receives the best ranking, has shown significant advantage in maintaining structure information over other methods by a large margin. WCT [13] performs worst in the CP test due to treating diverse image regions in an indiscriminate way. 3) For the SR evaluation, SANet [16] performs best on average, attributed to the attention mechanism that generates more local style details. However, AAMS [15], which is also based on self-attention mechanism, does not perform well because the multi-stroke pattern generates imperceptible dot-wise artifacts. In addition, WCT [13] shows the competitive advantage, which indicates the effectiveness of whitening and coloring transformations. 4) For the OV evaluation, attention-based methods (e.g., SANet [16] and MANet [17]) rank top-2 on performance. It is not surprising because attention-based algorithms pay more attention to those feature-similar areas in the style image for stylizing a content image region. Furthermore, we plot the cumulative probability distribution curves of the B-T scores obtained from all AST results in Fig. 8. The AST method corresponding to the rightmost curve performs better because it accumulates higher B-T scores.

2) Correlation of B-T Scores: This part analyzes the correlation of B-T scores between the subjective evaluation of CP and SR, SR and OV, and CP and OV, as shown in Fig. 9. Taking SR and CP as an example, the method above (below) the diagonal indicates better performance in CP (SR), and vice versa. From the Fig. 9 (a), it is observed that the SANet [16] method has achieved a better consistency in SR and CP evaluation, and the WCT [13] method shows excellent performance in SR but fails to sufficiently maintain the content structure. In addition, the correlation phenomena observed in Fig. 9 (b) and (c) also reveal the complex relationship between OV and CP (or SR).

3) Convergence Analysis: To demonstrate that the scale of the subjective study is large enough to support performance evaluation, we further analyze the convergence from the perspectives of the number of subject votes and images pairs.

Number of votes: We randomly sample $\lambda$ ($\lambda$ = 5000, 15000, 25000…, 55000) votes from a total of 63,000 voting results, and calculate the B-T scores for each AST algorithm. To avoid the possible bias, we repeat this process 1000 times with different samples of votes. Fig. 10 (a)-(c) show the mean and standard deviation of B-T scores for each sample on three subjective evaluations. It is observed that as the number of votes grows, the B-T scores tend to be stable, which demonstrates that the number of votes is sufficiently large for performance evaluation.

Number of images pairs: Similar to the above convergence analysis, we randomly sample $\mu$ ($\mu$ = 5, 25, 50, 75, 100, 125) content-style image pairs from our dataset and then plot the means and the standard deviations in Fig. 10 (d)-(f). Obviously, as the number of images pairs grows, standard deviation of B-T Scores decreases, which indicates that the B-T scores obtained from the subjective study are stable.

Overall, the above two kinds of convergence analysis demonstrate the reliability of the subjective rating scores.

1) Selection of Training Database: As show in Fig. 12 (a), we re-collected 100 new style images, covering a wide variety of categories, as training images for dictionary learning. Note that there is no overlap between these images and the above collected style images (in the Section II-A) to ensure the complete independence of the training and test data. In addition, to avoid overfitting, we augment the training database by evenly partitioning the whole images. The impacts of the number of image blocks and training database will be discussed in the following Section IV-B.

2) Gram-Based Feature: The work in [7] demonstrated that the correlations between convolution responses at the same layer (i.e., Gram matrices) yielded effective texture synthesis and can effectively grasp the image style. Additionally, sparse representation technique shows great prospects in image visual style analysis [45]. Inspired by this, we construct a style perception model based on sparse representation, while incorporating high-level perceptual information (Gram matrices) extracted from deep neural network.

In this paper, we resort to compute gram matrices from style images using a pre-trained VGG network of the state-of-the-art full-reference IQA model (DISTS) [46] which has superior performance in evaluating texture similarity. Assumed that the feature map of a sample style image $\emph{I}_{s}$ at layer l of DISTS [46] is denoted as ${{\mathbf{F}}^{l}}({{I}_{s}})\in{{\mathbb{R}}^{C\times H\times W}}$, where C is the number of channels, and H and W represent the height and width of the feature map, the Gram-based representation is computed from the feature map ${{\mathbf{F}}^{l}}({{I}_{s}}{)}^{\prime}\in{{\mathbb{R}}^{C\times(HW)}}$ aggregated from the ${{\mathbf{F}}^{l}}({{I}_{s}})\in{{\mathbb{R}}^{C\times H\times W}}$:

In the implementation, we use the first to fifth VGG network layers of DISTS [46] to produce a set of Gram-based representations at different layers, namely $\text{ }\!\!\{\!\!\text{ }{{\mathbf{G}}^{l}}\in{{\mathbb{R}}^{C\times C}},\text{ }l=1,2,\ldots,L\text{ }\!\!\}\!\!\text{ }$, where L=5 denotes the highest layer and $C\in\left\{64,\text{ }128,\text{ }256,\text{ }512,\text{ }512\right\}$ correspond to the numbers of feature maps at each layer.

After the above processing, each Gram-based representation then generates a Gram-based style feature vector ${{\mathbf{g}}^{l}}\in{{\mathbb{R}}^{C\times 1}}$ through averaging each row, described as:

where $x_{1}^{l}=\ldots=x_{C}^{l}=1/C$

Finally, numerous style feature vectors at the same layer extracted from different style images are used to form a style matrix $\mathbf{SM}$. As a result, we can obtain five different style matrices (corresponding to five layers) $\mathbf{S}{{\mathbf{M}}^{l}}=[{{\{{{\mathbf{g}}^{l}}\}}_{1}},{{\{{{\mathbf{g}}^{l}}\}}_{2}},\ldots,{{\{{{\mathbf{g}}^{l}}\}}_{{{N}^{l}}}}]\in{{\mathbb{R}}^{C\times{{N}^{l}}}}$. All style matrices will be used for the subsequent style dictionary learning.

3) Multi-Scale Style Dictionary Learning: Using the above style matrices $\mathbf{S}{{\mathbf{M}}^{l}}=[{{\{{{\mathbf{g}}^{l}}\}}_{1}},{{\{{{\mathbf{g}}^{l}}\}}_{2}},\ldots,{{\{{{\mathbf{g}}^{l}}\}}_{{{N}^{l}}}}]\in{{\mathbb{R}}^{C\times{{N}^{l}}}}$ as input, we learn multi-scale style dictionary $\mathbf{SD}^{l}$ by seeking a sparse representation for each style feature vector $\mathbf{g}^{l}$ under specific sparsity constraint $\tau$. Each style sub-dictionary $\mathbf{SD}=[\mathbf{sd}_{1},\mathbf{sd}_{2},\ldots,\mathbf{sd}_{U}]\in{{\mathbb{R}}^{C\times U}}$ contains U basic elements.

Formally, the process of multi-scale style dictionary learning can be formulated as:

where ${{\left\|\cdot\right\|}_{2}}$ is the $\emph{l}_{2}$-norm operator, ${{\left\|\cdot\right\|}_{0}}$ denotes the $\emph{l}_{0}$-norm that counts the number of non-zero elements in a vector, and $\bm{\alpha}_{i}$ is the sparse coefficient vector of ${{\{{{\mathsf{\mathbf{g}}}^{l}}\}}_{i}}$. Typically, both ${{\mathbf{SD}}^{l}}$ and $\bm{\alpha}_{i}$ are unknown in this stage. We resort to the online dictionary learning (ODL) algorithm implemented in the SPArse Modeling Software [47] to solve this NP-hard problem. Details of dictionary learning can refer to [47].

1) Selection of Training Database: Since the essence of the proposed content evaluation model is to restore the structure information of the source content images and stylized images based on dictionary learning, we only select natural images to construct the content dictionary. Refer to [48], we randomly select ten natural images from the TID 2013 [49] database and NPRgeneral [31], which have different scenes in the images, as shown in Fig. 12 (b).

2) DoG Response Feature: As known, human visual perception is highly sensitive to the edge information, the major objects in the painting emphasized by the artists often contain distinct edges in most cases [39]. Intuitively speaking, the outline of the main objects largely reflects the content information of artworks. Inspired by this, the edge information, as the significant component in painting content, needs to be deeply investigated for evaluating the CP quality. Furthermore, an outstanding painting will be appreciated by humans in local details and global perception. Thus, it is necessary to utilize multi-scale strategy to better describe the content of the painting from coarse to fine level of detail[50]. As shown in Fig. 13, the multi-scale Difference of Gaussian (DoG) is applied to represent the content feature [51], which can properly simulate the receptive field of retinal cells.

First, the DoG signals, $DoG(x,y)$, at different scales can be computed by:

where $I(x,y)$ denotes the pixel location $(x,y)$ of the input image, the symbol $\otimes$ denotes the convolution operation, and ${{R}_{\sigma,k\sigma}}(x,y)$ is defined as the difference between two Gaussian kernel with nearby scales $\sigma$ and $k\sigma$:

where $\sigma$ and $k$ are used to control the scales of DoG. Refer to [48], we set $k$ = 1.6, and $\sigma\in\left\{0,\text{ }1,\text{ }1.6,\text{ }2.56,\text{ }4.096\right\}$ in the experiment. Here, $\sigma$ = 0 denotes the original scale.

Once the DoG signals at the current octave are computed, the last scale-space image was selected as the new input and was down-sampled by a factor of two to repeat the above process, thereby producing a set of DoG signals with a variety of octaves and scales, namely $\left\{Do{{G}^{z,o}}(x,y)\right\}$, where $z\in\{1,\ldots,Z\}$ denotes the $z$-th scale, and $o\in\{1,\ldots,O\}$ denotes $o$-th octave.

After the above processing, each DoG signal is partitioned into numerous patches with size of $8\times 8$, subtracted by the mean value. In the implementation, 1000 overlapped patches having rich details and structures are selected as training samples. Then, these patches are vectorized into column vectors to form a content matrix $\mathbf{CM}$, $\mathrm{\mathbf{CM}}=[{{\mathrm{\mathbf{y}}}_{1}},\ldots,{{\mathrm{\mathbf{y}}}_{k}}]\in{{\mathbb{R}}^{T\times K}}$, based on which the subsequent overcomplete content dictionary is learned. Each patch ${{\mathrm{\mathbf{y}}}_{k}}\in{{\mathbb{R}}^{T\times 1}}$ contains $T$ pixels and $k=1,\ldots,K$. Here, $K$ = 1000.

3) Multi-Scale Content Dictionary Learning: Similar to the above multi-scale style dictionary learning, the multi-scale content dictionary $\mathbf{CD}^{z,o}$ can be learned from multi-scale content matrices $\mathbf{CM}^{z,o}$. Each content sub-dictionary $\mathbf{CD}=[\mathbf{cd}_{1},\mathbf{cd}_{2},\ldots,\mathbf{cd}_{U}]\in{{\mathbb{R}}^{C\times U}}$ contains $V$ basic elements. In the experiment, we set $V$ = 256. Similarly, the process of multi-scale content dictionary learning can be formulated as:

where $\bm{\beta}_{i}$ is the sparse coefficient vector of $\mathbf{y}_{i}$. Note that we also apply the ODL algorithm to solve Eq. (10)

Through the above efforts, we obtain two types of over-complete multi-scale dictionaries, containing $U$ and $V$ basic atoms as the column vectors in ${{\mathbf{SD}}^{l}}$ and ${{\mathbf{CD}}^{z,o}}$, respectively. Thus, each style feature vector ${{{\mathbf{g}}}^{l}}$ (or content patch $\mathbf{y}_{k}$) can be sparsely represented as a linear combination of basic atoms contained in ${{\mathbf{SD}}^{l}}$ (or ${{\mathbf{CD}}^{z,o}}$).

1) Style Sparse Coefficients Estimation: Given the testing stylized image $I_{t}$ and style image $I_{s}$, we can obtain two Gram-based representations using the DISTS network, denoted as $\mathbf{G}_{t}^{l}\in{{\mathbb{R}}^{C\times C}}$ and $\mathbf{G}_{s}^{l}\in{{\mathbb{R}}^{C\times C}}$. Then, the style sparse coefficient vectors can be estimated by a weighted linear combination of previously learnt dictionary elements, i.e.,

where ${(\mathbf{SD}^{l})}^{+}$ denotes the generalized inverse matrices of ${(\mathbf{SD}^{l})}$.

2) Content Sparse Coefficients Estimation: For the testing stylized image $I_{t}$ and the source content image $I_{c}$, after the same processing steps as in the training phase, we can obtain patch vectors $\mathrm{\mathbf{y}}_{c}^{z,o}$ from $I_{c}$ and its corresponding patch vectors $\mathrm{y}_{t}^{z,o}$ from $I_{t}$. Similarly, the content sparse coefficient c and tc can be computed by:

where ${(\mathbf{CD}^{l})}^{+}$ denote the generalized inverse matrices of ${(\mathbf{CD}^{l})}$.

3) Sparse Feature Similarity Measure: From the above estimation phase, we generate the sparse coefficients ${{\mathbf{s}}^{l}},{{\mathbf{c}}^{z,o}},\mathbf{ts}^{l},\mathbf{tc}^{z,o}$ on style, content and stylized images. Considering that these sparse coefficients are represented as a linear combination of basis vectors, meaning that the similarity between the style feature vectors or content patches can be directly measured using their sparse coefficient vectors. Thus, the style and content similarities are respectively defined as:

where $\left\langle\cdot\right\rangle$ calculates the inner product, $\eta$ is a constant with a small value added to prevent the denominator to be zero. The $SS$ measures the style similarity between the style and the stylized image, and $CS$ measures the content similarity between the content and the stylized image.

To measure the final quality between a stylized image and its corresponding content and style images, we need to pool the above spare feature similarities into a single score. In our pooling strategy, we first pool the style and content sparse feature similarities into SR and CP scores across all scales or octaves, and then combine the scores to measure the OV quality score. First, the SR quality score $Q_{\mathrm{style}}$ is defined as:

Then, the CP quality score $Q_{\mathrm{content}}$ is defined as:

where $Z$ and $O$ denote the number of scales and octaves.

Finally, the OV quality $Q_{\mathrm{overall}}$ is calculated by combining $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$ into a score:

where the parameters $\omega_{1}$ and $\omega_{2}$ are used to adjust the relative importance of the two portions. In this paper, we set $\omega_{1}$ = 0.4 and $\omega_{2}$ = 0.6 based on the performance analyses in Section IV-C. Of course, there is a large room to manipulate the importance weights for better quality prediction. A more meaningful practice may be to explore the proper combination of $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$ that best fits human subjective study.

Similar to [52, 53, 54], four criteria are adapted to measure the performance of different methods: the Spearman Rank order Correlation Coefficient (SRCC), Kendall Rank-order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Hit Rate (HITR). Specifically, the SRCC and KRCC measure the prediction monotonicity. PLCC is utilized to evaluate the prediction linearity after fitting a five-parameter logistic function:

where $x$ and $f(x)$ represent the objective and mapped scores respectively, and $\{{{\kappa}_{i}}|i=1,2,\ldots,5\}$ are the five parameters to be fitted. Additionally, HITR can measure the classification accuracy [52], defined as:

where $R_{i}$ denotes the number of correct judgments in PC, and $R_{n}$ is the total number of PC. Considering that the subjective experiments are based on PC within the same group of images, so that the ground truth (B-T scores) are only meaningful within the same group. Therefore, these criteria are computed respectively for each group from the same source image. Then, the average value of all 150 groups is reported as the final performance score. A superior metric should have higher criteria values (with a maximum of 1).

Since the fundamental of proposed quality metrics (i.e., $Q_{\mathrm{style}}$, $Q_{\mathrm{content}}$ and $Q_{\mathrm{overall}}$ ) is the dictionary operating in a multi-scale framework with several parameters, it makes sense to explore the influences of parameters and training databases on the performance evaluation. Therefore, this subsection first tunes the multi-scale dictionaries parameters, and then changes the type and number of images in the training databases.

1) Parameters in $Q_{\mathrm{content}}$: For multi-scale content dictionary, we first visualize the influence of different combinations of scale and octave with at most 10 scales and 5 octaves on the performance of $Q_{\mathrm{content}}$ in Fig. 14. It shows that the performance is greatly affected by the number of octaves while slightly affected by the number of scales. The reason is that the CP evaluation is more concerned with semantic structural changes rather than detail fidelity. As shown in the Fig. 13, octave adjustment causes large structural changes, while scale adjustment mainly affects small detail information. Here, we set the number of octaves $O$ = 4 and the number of scales $Z$ = 3, which can achieve the best performance. In addition, we show the performance of tuning the patch size and the number of basis vectors in Fig. 15. From the results, we can observe that the optimal performance is obtained by selecting patch size 6 and the number of basis vectors 256. Therefore, we set $T$ = 36 (patch size = 6) and $V$ = 256 in this work.

2) Parameters in $Q_{\mathrm{style}}$: For multi-scale style dictionary, since the Gram-based style feature vector of each layer is fixed, we only test the effect of the number of basis vectors. As shown in Fig. 16, the evaluation accuracy varies relatively slight over the numbers of basis vectors, which indicates that the $Q_{\mathrm{style}}$ does not highly depend on the training configurations. In this paper, we set $U\in\{256,256,512,1024,1024\}$.

3) Training database: In addition to the parameters in the dictionary, the selection of the training database is also worth analyzing, which can validate whether the performance is dependent on a particular training database. To this end, we train several overcomplete content and style dictionaries based on different training images respectively. The training images are selected from the TID 2013 [49], NPRgeneral [31], and TAD66K [55]. The implementation details and performance results are listed in Table II. Note that the number of training samples is maintained consistent for each content sub-dictionary, while the training samples (i.e., Gram-based style feature vectors) of style sub-dictionaries are set to a different number to match the size of the Gram matrix. The results represent that the performance with different dictionaries are quite similar. This demonstrates that the proposed method is insensitive to the selection of the training databases.

As mentioned above, there is a complex relationship between OV and other quality factors (e.g., $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$). Hence, it is meaningful to analyze the impacts of different pooling methods on the performance results. In this connection, we refer to the experimental settings [56] making a modification to Equation (19):

where $c$ and $d$ are utilized to balance the significance of the summation and multiplication pooling items, $\omega_{1}$, $\omega_{2}$, $\omega_{3}$ and $\omega_{4}$ are set to balance the importance of different quality factors. The performance of the three pooling strategy (i.e., multiplication($c$=1, $d$=0), summation ($c$=0, $d$=1) and combination) are listed in the Table III. From the results, we can find that both multiplication and summation strategies achieve the optimal performance when ($\omega_{1}$, $\omega_{2}$) and ($\omega_{3}$, $\omega_{4}$) are set as (0.4, 0.6), but the performance of multiplication is better than that of summation. Based on the optimal weight setting for the quality factors, the combination pooling strategy obtains similar results among the five ($c$, $d$) combinations. In summary, different pooling strategies generate significant impacts on the performance, in contrast, the multiplication pooling strategy with parameter ($\omega_{1}$=0.4, $\omega_{2}$=0.6) obtains the optimal results. As a consequence, we employ this multiplication pooling strategy as the final pooling method in this work.

For the performance test on the CP, SR and OV quality evaluations, we compare the proposed SRQE with existing IQA metrics, including two categories: (1) Fourteen state-of-the-art general-purpose FR-IQA metrics: SSIM [35], FSIM [26], MS-SSIM [57], IW-SSIM [25], Peak Signal-to-Noise Ratio (PSNR), MAD [58], VIF [24], VSI [27], GMSD [59], UQI [60], IFC [61], RFSIM [62], DISTS [46] and LPIPS [63]. (2) Eight state-of-the-art general-purpose NR-IQA metrics: NIQE [28], TCLT [36], BMPRI [64], BLIINDS-II [65], BRISQUE [37], UNIQUE [66], WaDIQaM [67], TReS [68]. For the traditional learning-based models, we randomly divide the each individual dataset into two non-overlapping subsets (80% for training and 20% for testing). Then, we resort to the support vector regression (SVR) to train the models and report the average results after training-testing process 1000 times. For the deep learning-based methods, we divide the data set into five fixed subsets with non-overlapping contents, and use four subsets for training and one subset for testing at each time to ensure that each image has been tested. For the methods that are training-free or require specific variance of human opinions, we directly report the performance results using the pre-trained model.

1) Performance Test on CP: Although the evaluation of CP between the stylized images and source content images is not a classic FR-IQA problem, since the stylized image targets to maintain the structure information of source content image, the structure measurement module commonly included in existing FR-IQA methods is relatively suitable for comparison. As a consequence, we compare the proposed $Q_{\mathrm{content}}$ with the state-of-the-art general-purpose FR-IQA metrics. In addition, we also utilize the NR-IQA method to establish the functional mapping from the stylized images to the CP quality scores. Each content subset conducts the same training-testing strategies described above. The performance compassion results are listed in Table IV, where the top three metrics are highlighted in bold. It can be seen that the traditional general-purpose FR-IQA metrics perform better than NR-IQA metric. It is reasonable since the purpose of CP evaluation is to measure the content structure similarity between the stylized and the original content images. Thus, ignoring the content image and directly extracting features from the stylized image for regression not only lacks enough useful information, but also has no practical significance. Our $Q_{\mathrm{content}}$ achieves the best performance for all dataset on the three most important ranking-related performance criteria: SRCC, KRCC and HITR, but is inferior to MS-SSIM [57] on PLCC. The reason is that MS-SSIM [57] also applies the multi-scale feature extraction strategy to simulate the visual characteristics of humans appreciating art works from different scales.

2) Performance Test on SR: To our best knowledge, there is no related methods for evaluating the quality of stylized images on SR. Although SR evaluation cannot be taken as a classical FR/NR-IQA issue, we are curious about how the performance of these methods in AST-IQA task, especially given the lack of comparison methods. Thus, we report the performance comparison results in Table V. It is clear that the above FR-IQA methods are not suitable for quantitative evaluation of SR, because of the difference in contents between the stylized images and the style images. DISTS [46] obtains the better performance among these FR-IQA metrics, since it is designed to evaluate structural and texture (related to style) similarities, allowing for slight pixel misalignment. In addition, although the NR-IQA method can map the stylized images to SR quality scores with powerful learning machines, it lacks practical relevance. In addition, it is also doubtful whether the NR-IQA method can maintain high performance when testing more diverse images that are not included in training. Compared with other methods, our $Q_{\mathrm{style}}$ achieves the best performance, since it builds a strong association (e.g., style pattern and brush stoke) with AST.

3) Performance Test on OV: Since there is no ground truth for stylized image to directly compare on OV evaluation, instead of using the FR-IQA methods, we utilize the CP/SR quality score of the FR-IQA method for performance comparison, which is beneficial to analyze and explore the contribution of the CP/SR quality components on the overall quality OV. Additionally, we also perform NR-IQA on the OV quality evaluation task, which will be useful to understand how challenging this task is for the existing NR-IQA metrics. The performance comparison results of all methods are shown in Table VI. In the table, the subscript ”SR/CP” represents the quality scores generating from the SR/CP evaluation. From the results, we observe that: 1) All CP and SR quality score generated by vanilla FR metrics have weak correlation with the overall quality OV. It indicates that OV quality evaluation is a complex process where multiple factors need to be considered. 2) The learning-based NR-IQA methods show good performance results, in particular, transformer-based TReS [68] achieves competitive results with our $Q_{\mathrm{overall}}$. Obviously, utilizing deep neural network to perform AST-IQAD task is a promising way. 3) It can be seen that $Q_{\mathrm{overall}}$ has a higher performance than $Q_{\mathrm{style}}$ or $Q_{\mathrm{content}}$ alone, which also demonstrates the effectiveness of our pooling strategy. As an unsupervised learning method based on sparse representation, our $Q_{\mathrm{overall}}$ can stably evaluate OV quality scores and achieve the best performance on three criteria via properly combining the $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$. Actually, as discussed in Section IV-C, there still leaves a large space for improving the evaluation accuracy via proper importance weights and combination strategies.

In the above experiments, the proposed SROE demonstrates a better correlation between the prediction scores and ground truth. We further adopt the hypothesis testing approach based on t-statistics [69] to prove that our SRQE is statistically better than other metrics. Specifically, we first calculate the area under curve (AUC) values (i.e., the area covered by receiver operating characteristic (ROC)) with 95% confidence interval (CI) for all image pairs (from PC in the subjective study). A higher value of AUC indicates better performance of the method. Next, we carry out the two-sample t-test between the pair of AUC values with 95% CI. We show the results of the AUC values and statistical significance of difference in Fig. 17, where the white/gray/black square manifests that the method in row is significantly better/indistinguishable/worse than the method in column. It can remark from the results that our SRQE performs significantly better than all competitors, indicating the superiority of our SRQE method.

In this subsection, we present the robustness of our proposed $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$ in two AST applications (i.e., content-style trade-off and style interpolation) which are included in many AST methods.

1) Content-style trade-off: This application can adjust the degree of stylization. Three style degree groups with smooth changes generated by MANet [17] are presented in Fig. 18. When $\alpha$ = 1, the fully stylized image is obtained. Fig. 18 (a)-(b) present $Q_{\mathrm{content}}$ and $Q_{\mathrm{style}}$ results with different degrees of stylization. It can be seen that as $\alpha$ increases from 0 to 1, the $Q_{\mathrm{content}}$ (or $Q_{\mathrm{style}}$) value is consequently decreasing (or increasing) gradually which indicates that our $Q_{\mathrm{style}}$ and $Q_{\mathrm{content}}$ can effectively capture the changes in the style patterns and content structure of the image.

2) Style interpolation: This application is to merge multiple style images into a single generated result. Here, we also utilize MANet [17] to generate a group of stylized images with different interpolations, and then use $Q_{\mathrm{style}}$ (or $Q_{\mathrm{content}}$) to evaluate SR (or CP) of the stylized images. As shown in the Fig. 19 (a), with the continuous decline of the weights for the specific styles, the $Q_{\mathrm{style}}$ value is consequently decreasing gradually which indicates that our $Q_{\mathrm{style}}$ can clarify style characteristics and accurately evaluate SR even under the interference of multiple styles. Fig. 19 (b) presents that our $Q_{\mathrm{content}}$ predicts a set of slowly decreasing quality values as the style changes from Style1 (Line drawing) to Style2 (Xieyi). It is reasonable because the Line drawing pays more attention to the maintenance of structure and shape than Xieyi. The results of the above predictions prove the robustness of our method.

1) Performance Test on Rank-n accuracy: To comprehensively compare the performance of the IQA metrics, we also focus on the performance of Rank-n accuracy [70]. Given the eight stylized images for each Content-Style image pair, the rank-n accuracy is the percentage of the objective scores where the subjective-rated best one is within their top n positions. The results are presented in Fig. 20. We can observe that the proposed method achieves a fairly good performance in the SR, CP and OV ranking accuracy tests. Since one of the most important applications of AST-IQA metric is to guide the generation of stylized images, the proposed method is a promising tool for automatic selection of the optimal transferring result from a set of candidates.

2) Failure Ranking Analysis: As aforementioned, we demonstrate the ranking capability of the proposed method on the three quality factors. However, in some special situations, the proposed method encounters challenges to achieve the expected ranking results. As shown in Fig. 21, we present two groups of representative failure rankings on the OV evaluation, which can be divided into two categories. In the first category, our method fails to capture extraneous artifacts, as shown in Fig. 21 (A). We select the stylization produced by SANet [16] as the optimal result, which produces unpleasing eye-like artifacts (zoom in for greater clarity) due to the fine-grained nature. The reason behind this lies in that our method mainly uses global sparse representation and ignores the local extraneous artifacts. In the second category, our method fails to effectively balance the importance of quality factors, as show in Fig. 21 (B). Actually, the CP and SR do not always complement each other. A non-realistic style leads to lower content retention in the final stylization result. Obviously, there is a large room to manipulate the importance weights for the quality factors. Overall, how to further dig the aesthetic information behind the stylization and propose a better strategy to balance different quality factors are the key issues to be explored in the future work.