DreamWalk: Style Space Exploration using Diffusion Guidance

Denoising diffusion models(Sohl-Dickstein et al., 2015; Ho et al., 2020; Yang et al., 2022) are a class of generative models which are trained to transform a sample of Gaussian noise into a sample of a data distribution. To generate a sample from the data distribution, we start with $x_{1}\sim N(0,I)$ as Gaussian noise. For a given time step $t$ from $1$ to $0$, we alternate between denoising and adding back a smaller amount of Gaussian noise. Over the course of the sampling process, from $t=1$ to $t=0$, the image becomes less noisy and more realistic. An update step using a DDPM (Ho et al., 2020) sampler might be:

where $f_{\theta}(x_{t},t)$ is the guidance function, which is responsible for the denoising stage of the diffusion process. The guidance function may combine multiple forward passes of the diffusion model, $f_{\theta}(x_{t},t,c)$, a network trained to estimate the noise of the image given a condition $c$. For example, classifier-free guidance (Ho and Salimans, 2022) combines conditional and unconditional outputs to improve adherence to the conditioning vector.

Liu, et al. (Liu et al., 2022) demonstrated that multiple diffusion forward passes may be composed to to allow the image generator to “build” a scene from various elements. For example, they combine phrases like “a photo of cherry blossom trees”, “sun dog”, and “green grass” in order to generate an image with components from each of these phrases. Given a series of conditions $c_{0},\ldots,c_{k}$ defined on a single model $\theta$, they compute a single overall update step as follows:

where $s_{i}$ are the guidance scales or coefficients (see Equation 15 in (Liu et al., 2022)). Note that if $k=1$, $c_{0}$ is a text-conditioning vector and $\emptyset$ is an empty conditioning vector (“null text”), Eq. 3 is equivalent to classifier-free guidance (Ho and Salimans, 2022).

We build upon (Liu et al., 2022), while addressing a different task and providing a more general formulation. (Liu et al., 2022) is focused on compositionality, with the goal of producing a single sample with all the elements present in the multiple text prompts (e.g. a tree, a road, a blue sky). In an image they produce, a particular element in the text is either present or absent; a success is defined as having all the elements in the multiple text prompts present in the generated image. In contrast to their true/false task, we focus on fine-grained control for stylization and personalization, producing multiple samples which offer a choice of target distributions. To solve this very different task requires a generalization of the multiple guidance framework of (Liu et al., 2022) through the introduction of guidance scale functions; this is described in section 3 where the relationship with (Liu et al., 2022) is discussed in more detail.

Multidiffusion (Bar-Tal et al., 2023) and SyncDiffusion (Lee et al., 2023) use a similar formulation to create panoramas or images at a higher resolution than can be naturally generated by the diffusion model, which typically has a fixed generation resolution. Both methods do this by partitioning an image into overlapping windows and then running a single diffusion process where each window provides guidance to pixels in its regions. In both techniques, all the pixels go through a single, joint diffusion process.

Stylization or style transfer, taking a real photo and applying a style, is a long explored field, with early techniques involving hand designed algorithms designed to mimic a certain artistic style (Haeberli, 1990; Salisbury et al., 2023; Litwinowicz, 1997; Hertzmann, 2003), and parametric (Heeger and Bergen, 1995; Portilla and Simoncelli, 2000) or non-parameteric (Efros and Leung, 1999; Barnes et al., 2009; Hertzmann et al., 2023) techniques for matching image statistics to either new texture patches or stylized images. Recently, as deep learning has become more popular, the field has shifted towards neural style transfer and texture synthesis (Gatys et al., 2015; Jing et al., 2019; Azadi et al., 2018). Unlike these techniques, we do not aim to transfer image statistics from a style exemplar image. Instead, we focus on gaining fine-grained control over a generative model’s pre-existing ability to apply style and move to between different styles.

(Karras et al., 2019) showed that modifying the generator architecture of a GAN can induce a latent space with coarse and fine features disentangled. A number of publications followed, focusing on controllability by disentangling the latent space, producing interpretable paths or directions through the latent space that allow for continuous control of pose, zoom, scene composition, hair length, and many others: (Wu et al., 2021; Tewari et al., 2020; Voynov and Babenko, 2020; Härkönen et al., 2020; Shen et al., 2020). Other methods tried to identify existing interpretable paths in the model  (Jahanian et al., 2019; Härkönen et al., 2020; Voynov and Babenko, 2020). (Patashnik et al., 2021) learned to use CLIP embeddings to control StyleGAN latents with text prompts. More recently, similar techniques have been applied to diffusion models: (Park et al., 2023) introduces a pullback metric and uses this to find latent-space basis vectors with particular semantics; (Preechakul et al., 2022) learn to encode directly into the diffusion model’s latent space, allowing interpolation along axes like age and hair color. (Wu et al., 2023a) use a soft weighting of base and attribute prompts, with the weighting factor chosen to try to limit the impact of the attribute prompt only to appearance, and not the layout, of the generated image. This weighting factor is learned through an optimization problem and is done per image, though the weights can be applied to any image. Their main focus is taking a generated image and then applying the style once, without fine-grained control over how much the style is applied.

Given the difficulty of capturing fine-grained concepts using text, considerable attention has been given to methods for personalizing image generation models.

One approach is to train a new model that is conditioned on both images and text (Chen et al., 2023; Ma et al., 2023; Shi et al., 2023), which requires gathering a large dataset and retraining. An alternative approach is to finetune an existing model, either by finetuning the text embeddings (Gal et al., 2022), the weights of the generator network (Ruiz et al., 2023; Sohn et al., 2023; Kumari et al., 2023), or both (Avrahami et al., 2023). Most commonly, personalized finetuning takes 3-5 images of a specific object or style and finetunes the entire text-to-image network to reconstruct these images when conditioned with text of a rare token $[\mathcal{V}]$, placed in context of its noun, e.g. “a photo of $[\mathcal{V}]$ dog”. DreamBooth (Ruiz et al., 2023) also uses a class-preservation prior in order to prevent the noun’s conditioning from drifting away from its original meaning and collapsing to only representing the 3-5 images being used for fine-tuning.

Parameter-efficient finetuning (Hu et al., 2021; Houlsby et al., 2019) such as LoRA is commonly used to speed up finetuning and reduce the size of the weight modifications, allowing users to transfer only the update weights instead of the entire network weights.

Building on the multiple guidance terms introduced by (Liu et al., 2022), we propose guidance scale functions $s_{i}(t,u,v)$ which allow for granular control over when, in the diffusion noise schedule, and where, in the image space, a forward pass with a specific condition is used. Specifically, we consider the following formulation for composition of diffusion models:

where for simplicity we define $g_{\theta}(x_{t},t,c_{i})$ as the guidance term for condition $c_{i}$. The guidance term includes the forward passes of the diffusion model $f_{\theta}(x_{t},t,c_{i})$ and $f_{\theta}(x_{t},t,\emptyset)$ and indicates a direction and magnitude towards a condition. This magnitude can be emphasized or de-emphasized by the guidance scale function, $s_{i}(t,u,v)$ (dependent on time $t$ and spatial location $u$ and $v$).

Now that we have the ability to target a specific distribution based on multiple conditions, we need a way to create the appropriate conditions for our task of controlling stylization and personalization.

In the multiple guidance formulation, this sentence can be decomposed into its component parts and then guided towards each of these distributions separately allowing for control over which of these relative concepts is the most important to the user. If there is a desired subject (“a $[\mathcal{V}]$ vase”) and a desired state (“with a colorful bouquet”), the user can decide which of these to emphasize. Figure 2 show examples allowing adherence to either the text prompt (background, action) or the subject (the DreamBooth token and subject). This control is useful since one common failure is the entanglement between a subject’s pose / actions and the subject’s appearance. If the images used for personalization share a similar pose, e.g. all images of a dog are front-facing and with the dog sitting, then the generator can struggle to change the pose of the dog. Our method can allow the user to emphasize the action separately from the token reducing this bias.

As described in (Ho and Salimans, 2022; Liu et al., 2022), each guidance term pushes the final result towards its own distribution in the image manifold, with the the magnitude of the guidance scale determining how strong the push is. With a single conditional, this term is thought of as simply controlling how closely the image adheres to the single text prompt versus a generic image. With the same seed, a different magnitude would generate similar images that increasing adhere to the prompt (left of Fig. 4).

This same concept applies in the multiple guidance formulation, with each guidance term’s scale controlling how much the generated image adheres to that condition. In the compositionality problem that (Liu et al., 2022) addresses, the goal is to generate a successful sample, defined as images where all the text elements are present. In contrast, our goal is to sample from a user-controlled choice of distributions, selected to explore style space. Since we can control the guidance scale differently for each condition, this provides a choice of distributions to target as shown on the right of Figure 4. For example, by choosing to target different distributions our formulation can walk between two different prompts (e.g. “base” and “style”), interpolating between them, moving from a style focused distribution (red in the figure) to a base focused distribution (blue). Interpolating between multiple style distributions produces a style interpolation; emphasizing both produces a mixture. The scales between a base and style distribution provide control over how text adherence versus style. The addition of time-varying GSFs enables the base to control the layout of the final generation (see Sec 3.4). Time-varying GSFs applied to DB tokens versus text allow us to control the importance of subject personalization versus text adherence as shown in Fig. 2.

Once the prompts are decomposed as above, each guidance scale function controls the magnitude of the diffusion step for a single component. Critically, this modulation can depend on location within the image or on timestep. Here we show examples of using GSFs to control how a style applied to an image.

For our experiments, we use both Stable Diffusion 1.5 (SD1.5) and Stable Diffusion XL (SDXL), with the default DPM+ and 50 denoising steps. For LORA learning, we conduct model training using 300 screenshots capturing diverse scenes from ”Breath of the Wild,” employing an instance prompt of ”a photo of [$\mathcal{V}$] style.” The learning rate was set to $10^{-4}$ for 500 steps.

For DB training, we follow the same experimental procedure used the (Ruiz et al., 2023) paper with some minor hyperparameter adjustments made to change from Imagen (Saharia et al., 2022). Specifically, we train our models on 30 objects from the Dreambooth dataset (Ruiz et al., 2023) using SD1.5. The training process generates 1,000 samples from ”a photo of [class noun]” class prior, and uses a batch size of 1, $\lambda=1$, a learning rate of $10^{-6}$, and a total of 1,500 iterations.

In Figures 10 and 11, we show straight forward applications of style to a given layout. These examples use different architectures (SD1.5 and SDXL) and a diverse set of styles including a LORA-trained style for the popular video game “Breath of the Wild”. We also demonstrate in Figure 14  that our technique is not limited to one or even two styles and can be used with an arbitrary number of guidance terms, mixing 4 distinct styles globally in the same image, taking some aspects from each.

Figure 13 shows that we can control the amount of style applied to a base image while preserving the rough layout, even when using a style trained with LORA on SD1.5. This example applies the style guidance term on the full image but uses the “default” option for the temporal aspect of the guidance scale function. The top left image is the base generation without any stylization applied; lower in the grid has more “Breath of the Wild” style applied to it, and to the right has more “Qi-Bashi calligraphy style”. Figure 2 shows the same capability for personalization on SD1.5, amplifying either the subject (the specific type of dog) or the action and setting. Note that amplifying both still preserves the subject while forcing a change in pose / action. In Figure 15, we show that both of these terms can be applied at once; we can take a DB subject and incrementally add style. Despite the addition of the style, the guidance term for personalization preserves the shape and color of the subject.

Our method also allows users to interpolate between multiple styles. In Figure 9 and 12, we show that our smooth interpolation between two different styles for models with very different architectures SD1.5 and SDXL. We also complete a user study in the following section on interpolation.

While stylization is a popular topic in Computer Vision, much of the prior work focused on walking in the latent space of GANs.

In the process of interpolating styles, we benchmark against two widely adopted techniques in the community, employing the compel library (Stewart, 2023): prompt blending and prompt weighting. To achieve blending, such as between Picasso and Hokusai, we assign weights $\lambda$ and $1-\lambda$ to the prompts ”Artist painting vivid sunset on beach canvas in the style of Picasso” and ”Artist painting vivid sunset on beach canvas in the style of Hokusai,” respectively, with $\lambda\in[0,1]$. Regarding prompt weighting, we consolidate the two prompts into a singular prompt, ”Artist painting vivid sunset on beach canvas in the style of Picasso and Hokusai,” and apply weights $\lambda$ and $1-\lambda$ to the terms ”Picasso” and ”Hokusai,” respectively, where $\lambda\in[0,k]$. We observe that when $k=1$, the change is nearly imperceptible, prompting our choice of $k=1.5$ for a more distinct transition. The outcomes, displayed in Figure 9, reveal that both baseline methods yield more abrupt transitions. Notably, in certain instances, prompt blending leads to the complete loss of style, a phenomenon we term the ’style-free valley,’ which is not observed with DreamWalk.

Since there is no clear source of ground truth for our task, we conducted a user study to determine if the results we generate are viewed as aesthetically pleasing. Users were presented with a start image and an end image, along with two possible series of images showing the transition from the start to the end, similar to Figure 9. One series was generated by the Compel baseline and one by DreamWalk; we randomly picked the layout in which the two series were displayed. Users were asked which series produced a “more pleasingly smooth transition”. Our user study closely followed (Ruiz et al., 2023), Users preferred the DreamWalk output 68% of the time.

The portion of this work performed at Cornell has been supported by a gift from the Simons Foundation. We thank Yuanzhen Li for coming up with the project name, and Miki Rubinstein and Deqing Sun for helpful discussions.