: Are We Really Meant to Confine the Imagination in Style Transfer?

As shown in Figure 2, StyleWallfacer mainly consists of two parts: First is the semantic-based style learning strategy, which aims to guide the model to learn the most essential style features in artworks based on the semantic differences between images and their text descriptions during the model fine-tuning process, truly helping the model understand the artist’s creative style. It also employs a data augmentation method based on human feedback to suppress overfitting when the model is fine-tuned on a single image, thereby achieving realistic text-driven style transfer.

The second part is the training-free triple diffusion process, which is designed using the previously fine-tuned LoRA weights. This section comprises three newly designed pipelines tailored to address different style transfer problems. By adjusting the self-attention layers of three denoising networks that share weights (denoted as ), it achieves high-quality style control. This results in artist-level style transfer and, for the first time, enables text prompts to control image colors during the style transfer process, solving the traditional method’s shortcomings of monochromatic colors, simple textures, and lack of text control when transferring styles based on a single image.

The semantic-based style learning strategy primarily aims to fine-tune text-to-image (T2I) models using their native “language” to enhance their comprehension of the knowledge humans intend them to learn during the fine-tuning process. Taking Stable Diffusion as an example, there is a significant discrepancy between the image semantics understood by the pre-trained CLIP and the intuitive human understanding of image semantics. Therefore, to better “communicate” with the pre-trained T2I model during fine-tuning, this paper employs a method of reverse-engineering the semantic information of image $\mathbf{I}$ in the CLIP space through BLIP DBLP:conf/icml/0001LXH22 :

$\displaystyle\mathbf{T}_{\text{CLIP}}=\text{BLIP}\left(\mathbf{I}\right)$

(1)

where $\mathbf{T}_{\text{CLIP}}$ denotes the image prompt derived through BLIP.

Although such methods enable us to obtain the semantic information corresponding to an image in the CLIP space, this text description cannot be directly employed in the fine-tuning process. This is because the description $\mathbf{T}_{\text{CLIP}}$ encompasses all information pertaining to the image, including content, style, and other details understood by CLIP. Utilizing this comprehensive description for fine-tuning still results in the model’s inability to comprehend human fine-tuning intentions, thereby preventing it from learning the stylistic information of the dataset.

Therefore, our StyleWallfacer transforms $\mathbf{T}_{\text{CLIP}}$ by creating an semantic discrepancy among descriptions. By incorporating a large language model to perform subtle semantic edits on $\mathbf{T}_{\text{CLIP}}$, descriptions related to image style are selectively removed:

$\displaystyle\mathbf{T}_{w/oS}=\text{LLM}\left(\mathbf{T}_{\text{CLIP}}\right)$

(2)

where $\mathbf{T}_{w/oS}$ denotes the text description after removing the style information, and LLM stands for large language model.

After such processing, we obtain the image $\mathbf{I}$ and its corresponding text description $\mathbf{T}_{w/oS}$ in the CLIP space, from which stylistic descriptions have been removed. As shown in Figure 2 (a), fine-tuning a pre-trained T2I model using these image-text pairs enables it to focus more effectively on understanding stylistic information, thereby circumventing unnecessary semantic drift.

After fine-tuning, the model has essentially learned the most fundamental style knowledge from the reference style image. Therefore, how to activate this knowledge so that it can be utilized for image-driven style transfer has become an extremely critical issue.

Unlike traditional one-shot style transfer algorithms that require the reference style image as input during style transfer, we aim to rely solely on the pre-trained style LoRA obtained in Section 2.2 for style transfer. Therefore, we cannot adopt a method similar to StyleID DBLP:conf/cvpr/ChungHH24 to manipulate the features in the self-attention layer as if they were cross-attention features, with the features from the style image $\mathbf{I}_{s}$ serving as the condition for style injection.

However, as shown in Figure 3, we observe that when initializing the noisy latent $\mathbf{X}_{0}$ with the original image and using the U-Net to denoise it, the larger the noise schedule threshold $t_{s}$, the more stylized the generated image will be, losing the original image’s content information and retaining only its basic semantics. Conversely, the smaller the noise schedule threshold $t_{s}$, the more the model’s generation tends to preserve the original image’s content information, while reducing the diversity and stylization in the generation process.

Therefore, we contemplate: Is it possible to fully leverage this characteristic by employing a diffusion process with a smaller $t_{s}$ as the main diffusion process, and using a diffusion process with a larger $t_{s}$ as the stylistic guiding process? Meanwhile, we can utilize the inverted latent obtained through DDIM inversion as the noisy latent for the third diffusion process, and harness the residual information from its denoising process as content guidance. In this way, we aim to achieve high-quality style transfer results while preserving the image content.

To this end, as shown in Figure 2 (b), we first use the VAE encoder to transform the image $\mathbf{I}_{c}$ to be transferred from the pixel space to the latent space, obtaining $\mathbf{F}_{0}$. By setting a larger noise schedule threshold $t_{s}^{l}$, we add noise to $\mathbf{F}_{0}$ (at $t=0$) to obtain $\mathbf{F}_{l}$ (at $t=t_{s}^{l}$). Similarly, by using a smaller noise schedule threshold $t_{s}^{s}$ , we obtain $\mathbf{F}_{s}$ (at $t=t_{s}^{s}$). Additionally, we use DDIM inversion to invert $\mathbf{F}_{0}$ to Gaussian noise $\mathbf{F}_{i}$ (at $t=T$). Then, using the same denoising U-Net, we denoise $\mathbf{F}_{s}$, $\mathbf{F}_{l}$, and $\mathbf{F}_{i}$ respectively. As shown in Figure 2 (b1), during the entire denoise process of latent $\mathbf{F}_{s}$, we transfer $\mathbf{F}_{s}$ to $\mathbf{F}_{l}$ by injecting the Key $\mathbf{K}_{t}^{l}$ and Value $\mathbf{V}_{t}^{l}$ collected from $\mathbf{F}_{l}$ into the self-attention layer, instead of the original Key $\mathbf{K}_{t}^{s}$ and Value $\mathbf{V}_{t}^{s}$. However, merely implementing this substitution can result in content disruption, as the content of the $\mathbf{F}_{s}$ representation would be progressively altered with the changes in the attended values.

Consequently, we propose a query preservation mechanism to retain the original content. Simply, as shown in Figure 2 (b1), we fuse the Query $\mathbf{Q}_{t}^{i}$ of DDIM inverted latent $\mathbf{F}_{i}$ with the original Query $\mathbf{Q}_{t}^{s}$ to get Query $\mathbf{Q}_{t}^{f}$ and inject it to the main denoise process instead of the original Query $\mathbf{Q}_{t}^{s}$. These style injection, query preservation and structural residual injection processes at time step t are expressed as follows:

	$\displaystyle\mathbf{Q}_{t}^{f}=\beta\mathbf{Q}_{t}^{i}+(1-\beta)\mathbf{Q}_{t}^{s},$		(3)
	$\displaystyle\phi_{\mathrm{out}}^{l}=\mathrm{Attn}(\mathbf{Q}_{t}^{f},\mathbf{K}_{t}^{l},\mathbf{V}_{t}^{l}),$		(4)

where $\beta\in[0,1]$. $\mu(\cdot)$, $\sigma(\cdot)$ and $\phi_{\mathrm{out}}^{l}$ denote channel-wise mean, standard deviation and the result of self-attention calculation after replacement, respectively. In addition, we apply these operations on the decoder of U-net in SD. We also highlight that the proposed method can adjust the degree of style transfer by changing noise schedule threshold $t_{s}^{l}$ and $t_{s}^{s}$. Specifically, lower $t_{s}^{l}$ and $t_{s}^{s}$ maintains more content, while higher $t_{s}^{l}$ and $t_{s}^{s}$ strengthens effects of style transfer.

Although this paper proposes a more robust style knowledge injection method than DreamBooth DBLP:conf/cvpr/RuizLJPRA23 in Section 2.2, fine-tuning models with a single sample remains challenging. Therefore, inspired by human feedback reinforcement learning (HFRL) shen2025exploringdatascalingtrends , this paper proposes a human feedback-based data augmentation method for small-scale datasets to compensate for dataset insufficiency and mitigate overfitting.

Specifically, when the model is first trained on a single style image, as shown in Figure 4, although the injection of style knowledge does not generalize well to all the prior knowledge of the model in the early stages of training, and some of the generated results do not match the reference style consistently, resulting in an asynchronous phenomenon in the injection of style knowledge. However, there are still many excellent samples in the model’s generated results. The reason for the emergence of these samples is that the prior knowledge represented by these samples is similar to the style image used in training in the CLIP space. This makes it easier for the model to transfer style knowledge to these pieces of knowledge during training. Therefore, it is possible to select samples that meet the style requirements from the large number of generated samples and add them to the training set for further fine-tuning of the model.

To this end, as shown in Figure 5, we divide the model’s fine-tuning process into three stages. In the first stage, which is the single-sample fine-tuning stage, we generate a large number of new samples using a text prompt that reflects the basic semantics of the reference image after the model has been trained. We then manually select the 50 samples that best match the stylistic features of the reference image and add them to the training set for the second stage of fine-tuning. In the second stage of fine-tuning, the basic idea is similar to the first stage. We expand the training set from 50 to 100 images. Finally, we fine-tune the model using these 100 samples to obtain the final style LoRA.

Through this data augmentation strategy, the overfitting phenomenon of the model during the fine-tuning process is greatly alleviated. The fine-tuned model is able to generate more diverse results and generalize the style knowledge to all the prior knowledge of the model, rather than being limited to a single sample.

Baselines   Our baseline list in one-shot text-driven style transfer includes DreamBooth DBLP:conf/cvpr/RuizLJPRA23 , a LoRA DBLP:conf/iclr/HuSWALWWC22 version of DreamBooth, Textual Inversion DBLP:conf/iclr/GalAAPBCC23 , and SVDiff DBLP:conf/iccv/HanLZMMY23 . For the text-driven stylization task, the baselines we selected include Artist jiang2024artist , InstructPix2Pix DBLP:conf/cvpr/BrooksHE23 , and Plug-and-play (PnP) Tumanyan_2023_CVPR . And baseline list in one-shot image-driven style transfer includes StyleID DBLP:conf/cvpr/ChungHH24 , AdaAttn DBLP:conf/iccv/LiuLHLWLSLD21 , AdaIN DBLP:conf/iccv/HuangB17 , AesPA-Net DBLP:conf/iccv/HongJLAKLKUB23 and InstantStyle-Plus DBLP:journals/corr/abs-2407-00788 .

Datasets   We selected one image from each of the three widely used 10-shot datasets, including landscapes XiaogangWang_XiaoouTang_2009 , Van Gogh houses DBLP:conf/cvpr/OjhaLLELS021 , and watercolor dogs DBLP:journals/corr/abs-2306-00763 , to form our one-shot datasets, in order to quantitatively evaluate the proposed method from a better perspective. To test our model, we first used FLUX flux2024 to generate 1,000 images of houses, 1,000 images of dogs and 1,000 images of mountains based on the prompts "a photo of a house", "a photo of a dog" and "a photo of a moutain", respectively. These images served as the style-free images to be transferred.

Metric   For image style similarity, we compute CLIP-FID DBLP:conf/cvpr/Parmar0Z22 , CLIP-I score, CLIP-T score and DINO score DBLP:conf/iclr/0097LL000NS23 between 1,000 samples with the full few-shot datasets. For image content similarity, we compute the LPIPS DBLP:conf/cvpr/Parmar0Z22 between 1,000 samples and the source image to evaluate the content similarity between the style-transferred images and the original images. Intra-clustered LPIPS Ojha_Li_Lu_Efros_JaeLee_Shechtman_Zhang_2021 ; DBLP:conf/cvpr/ZhangIESW18 of 1,000 samples is also reported as a standalone diversity metric.

Detail   For other details of the experiments and StyleWallfacer, please refer to Appendix B.

One-shot Text-driven Style Transfer Experimental Qualitative Results. As depicted in Figure 6, StyleWallfacer outperforms other methods in generating diverse and semantically accurate images. Unlike other methods that suffer from overfitting and semantic drift when trained on single-style images, StyleWallfacer employs multi-stage progressive learning with human feedback to reduce overfitting and enhance diversity. It also avoids identifiers for style injection, minimizing semantic drift and enabling precise style generation based on prompts.

Text-driven Stylization Experimental Qualitative Results. As shown in Figure 7, other methods, except StyleWallfacer, although have completed the task of style transfer, the results obtained after the transfer are far from the authentic style of the painter and fall short of the expected level. However, StyleWallfacer has achieved the best balance between image style transfer and content preservation. The images after style transfer not only closely match the painter’s authentic style but also feature finer details and a high degree of fidelity to the original image content.

One-shot Image-driven Style Transfer Experimental Qualitative Results.

As shown in Figure 8 (a), a visual comparison of the style transfer results of the StyleWallfacer model with other methods is presented. Clearly, the StyleWallfacer model achieves the best results in terms of image structure preservation and style transfer. Compared with the results of other methods, the style transfer results of StyleWallfacer have truly realized the style transfer, as if the painter himself had redrawn the original image according to his painting style, rather than simply blending the textures and colors of the original and reference images. Moreover, in terms of detail, the results generated by StyleWallfacer feature more refined texture details, while other methods generally suffer from noise and damage.

One-shot Image-driven Style Transfer & Color Edit Experimental Qualitative Results. As depicted in Figure 8 (b), the visualization results of image-driven style transfer and color editing are presented. Analysis of the figure reveals that the proposed method in this paper not only accomplishes style transfer but also retains the model’s controllability via text prompts. This enables synchronous guidance of the model’s generation process by both "text and style", thereby enhancing controllability. Moreover, the images obtained after style transfer maintain a high degree of content consistency with the original images, achieving a better balance between generation diversity and controllability.

As shown in Table 1, Table 2 and Table 3, the method proposed in this paper achieved the best results compared with all the baseline methods, further demonstrating the effectiveness of the proposed method from a quantitative perspective.

To prove that the proposed techniques can indeed effectively improve the performance of StyleWallfacer in various generation scenarios, we conduct extensive ablation studies focusing on these techniques and leave them in Appendix LABEL:abla due to page limit. And we have also understood the source of StyleWallfacer’s superiority from a mathematical perspective, for details see Appendix LABEL:math.