Controlled and Conditional Text to Image Generation with Diffusion Prior

The Diffusion Prior was introduced in OpenAI’s DALLE-2 [54] which is a hierarchical text-conditional DM. The Diffusion Prior is capable of mapping an input text embedding vector to an image embedding vector in a CLIP latent space. A decoder (unCLIP) then translates the CLIP image embedding into synthetic images. The Diffusion Prior is a classifier-free guidance DM that uses a Transformer backbone instead of U-Net. In DALLE-2, the Diffusion Prior is shown to outperform the autoregression prior in model size and training time. Recent works that use Diffusion Prior models include Make-A-Video [66], Dream3D [72] and Shifted Diffusion [78]. While Shifted Diffusion [78] proposes a more optimal prior model, Make-A-Video and Dream3D utilize existing prior formulation to support text-to-image generation in their pipeline. Compared to text conditioned diffusion model like [57, 61], there is limited work that builds on top of the Diffusion Prior proposed in DALLE-2[54]. We explore its capabilities and its applications to domain specific and conditional generation.

Color is an important attribute of an image that provides contextual information as well as sets the mood of viewer’s perception. Although there has been lot of research around generating images with specific styles as condition [77, 37, 5, 28], using only color palette for generating images has not been explored much by the research community. This could be a useful tool for artists and content creators to generate images with varied color palette portfolios without changing their authentic styles. There are some works on image colorization [60, 10, 71, 21] of grayscale images and image color transfer between colored images [55, 3, 73, 26, 35], but these methods work well when an image is available as input. In [2], the authors propose color conditional generation by controlling the color palette when generating images by injecting color histograms in log-chroma space into the StyleGan [32] architecture but do not show text and color conditioned generation.

In recent years, domain adaptation has become popular in image synthesis where large pretrained models are fine-tuned on a smaller dataset from a specific domain like [49, 59, 19]. In contrast to domain adaptation, domain specific generation aims to constrain generations from a pretrained model to a specific sub space or domain within the seen larger distribution.
Texture Domain: Texture synthesis has been studied for a long time in Computer Vision (CV) literature [11, 13, 22]. Recently, deep learning techniques have been used to generate textures [79, 18, 64]. These approaches are a primer to 3D texture synthesis and transfer [7]. Pretrained text-to-image diffusion models have also been explored for 3D texture and shape generation [51, 56].
Vectors and Isolated Objects Domain: Scalable Vector Graphics (SVG) is a suitable format for expressing visual concepts as primitives and for exporting designs at arbitrarily high quality [30]. These are most commonly applicable when generating posters, templates, card and other graphics. Previous work such as [29, 17] optimize CLIP image similarity to generate vectors from text prompts. [63] extends [17] with style loss to generate vectors from images whereas [16] uses a hierarchical neural Lindenmeyer system to optimize SVG paths and [44] uses evolutionary approach to generate collages. Recently, [30] proposed a multi-step optimization approach on top of Stable Diffusion [57] to use the large conceptual pretrained knowledge of diffusion models for text to vector generation.
Limitations: Most of these techniques have as a first step to generate images that can easily be converted to vectors either by optimization or by using existing algorithms such as [41]. Since SVGs comprise of simple abstract shapes and curves to compose concepts, photorealistic images generated by existing text-to-image diffusion models are not suitable for conversion to SVGs. However, these diffusion models have seen images that show abstract concepts [30] that can be generated using some prompt engineering. Though this works to a certain extent, this is not robust to all prompts [30]. We overcome these limitations by proposing to train the Diffusion Prior in HDM on images within a specific domain (textures, vectors or isolated objects). The generated images can then be used with existing vectorization [41] and material generation [7] techniques or as it is for further applications.

Concurrent Works: More recently, concurrent works like ControlNet [75], Composer [27] and T2I-Adapter [46] that support multiple conditional inputs and composable generations including color have been proposed. We emphasize that our paper is focused on revealing the capabilities of CLIP latent space based diffusion prior model which is lightweight and support conditional and domain specific generation without additional objectives or modifications in the base HDM architecture. This method is robust to domains and conditioning inputs and retains the advantages of the CLIP space such as latent interpolation and directional attribute manipulation while being efficient in terms of compute and memory.

For each of the domain specific priors, we retain the HDM pipeline as is but train a separate Diffusion Prior model for each domain. We first obtain a curated internal dataset of images with texture, suitable for vectors and are of isolated objects or concepts as described in Sec.4.1. Then, we train a domain specific Diffusion Prior model as described in Appendix Sec.C on the curated dataset following the same setup as LAION prior. In contrast to the base prior model $\mathcal{P}_{\theta}$, the domain specific models $\mathcal{P}_{\theta 1}^{texture}$, $\mathcal{P}_{\theta 2}^{vector}$ and $\mathcal{P}_{\theta 3}^{isolated}$ generate image embeddings within a specific sub-space in the CLIP embedding space that corresponds to their respective domain. Embeddings from each of the priors can be visualized by the LDM to generate domain specific images. Note that the domain specific priors only differ by the data they were trained on. There is no additional domain specific information, conditional input or finetuning.

To better understand the advantages of the domain specific prior, we provide a conceptual illustration in Fig.2 (Right). In VectorFusion [30], a sample generated by a diffusion model or a random SVG is further optimized to be within the desirable domain specific space (red dotted arrows). Our domain specific priors directly generate desirable domain specific images by generating CLIP embeddings within the desirable subspace highlighted by the intersecting areas (generation path shown by black arrows from the domain priors). This is because, the domain specific prior models are trained only on CLIP embeddings of images from the specific domain. This also ensures the priors do not generate embeddings outside this area even for complex prompts.

There are many studies on style, shape and semantic map conditioned generations (ref. Sec.2.2) but little to no work that focuses on color conditioned text-to-image generation. Being able to condition image generations with a color palette directly or obtained from another exemplar image is helpful for many creative workflows. We modify the Diffusion Prior $\mathcal{P_{\theta}}$ described in Appendix Sec.C.1 to take as input another token that represents the desired color information. Formally, our conditional Diffusion Prior is $\mathcal{P}_{\theta 4}(z_{x}|y,c)$ where $c$ corresponds to the color information. If $z_{c}$ corresponds to the color representation, we simply add it as an additional token to $z_{y}$ as $[z_{c},z_{t},w_{1},w_{2},...,w_{n}]$ utilized by the prior by cross attention.
Color Histogram: Studies of color distribution for images are generally seen in the form 3D histograms [15, 55], color triads [65] or 2D Histograms such as log-chroma space [4, 1, 2]. In our experiments, we use 3D color histograms proposed in [45] as $z_{c}$ obtained from the ground truth image $x$. The reason to use the LAB space over log-chroma or RGB space is because it is perceptually closer with respect to human color vision as its perceptual distances corresponds to Euclidean distances [45]. We choose a 10$\times$8$\times$9 histogram and zero-pad to get the total dimension to 768 corresponding to CLIP L/14 text embedding and encodings. We take the square root of the histogram before passing it in the prior to make it more uniform. We also follow classifier free guidance approach [25] during training by dropping color histogram with $p$ 0.5 to better capture textual content and relevance and both color and text with $p$ 0.1 for unconditional training. Color histogram is never provided without text to prevent content dependency on color information for generation.

We use an internal dataset to train the prior and decoder models.
Data for Decoder: To train the Diffusion Decoder LDM as described in Appendix Sec.C.2, we remove images that contain humans or texts as detected using classifiers to reduce the size and complexity of the dataset to 77M images. The classifiers are a single linear layer on top of frozen CLIP L/14 embeddings. Only the linear layer is trained. We tested other complicated architecture but this worked reasonably well for all cases. The details about the classifiers are provided in Appendix Sec.D.2.
Data for Priors: For the domain specific priors, we train separate classifiers to detect the domains from images and use them to gather data. For color prior, we use a subset of text image pairs from the 77M filtered data. More information in Appendix Sec.D.1.

Our Diffusion Decoder LDM has the same number of parameters as Stable Diffusion but trained on a relatively smaller dataset for fewer GPU hours as shown in Table.1. We can also see that the prior models have significantly lower number of parameters and training time, measured in A100-40GB hours similar to [57]. All the prior models are trained from scratch in 8-GPU A100-40GB instances. We train a smaller prior for vectors as we observed reasonable performance with a smaller model. The priors are trained with cosine schedule and all parameters are similar to those used in LAION prior¹ while the LDM uses linear schedule with 1000 DDPM timesteps for training and uses the same architecture as [57].

The dataset size and training time for all the models are provided in Table.1. We emphasize that training the prior model from scratch takes lesser time and compute compared to finetuning the larger decoder or stable diffusion for the same application. For reference, [75] trains only a small part of stable diffusion model and uses an average of 600 A100-80GB GPU-hours for canny-edge conditioned finetuning of Stable Diffusion model on 3M images. In comparison, our isolated object prior model takes 1344 A100-40GB hours to train on a dataset of 20M image-text pairs. If we use a conservative 1.5$\times$ factor speed up from A100-40GB to A100-80GB, this translates to 896 A100-80GB GPU hours for 20M samples. If we also scale the training data with another conservative 6$\times$ factor to get closer to 3M samples, we get 150 A1000-80GB GPU hours which is significantly less despite the differences in conditioning input. Similarly, the color prior with same calculations get to around 112 GPU hours on A100-80GB compared to 150-600 GPU hours for various conditioning inputs based finetuning in [75].

For all our experiments we use 100 DDIM steps for sampling the CLIP embedding from Diffusion Prior and 50 DDIM steps to generate an image using the Diffusion Decoder. We use linear sampling schedule. To improve relevance in the prior, we generate 10 embeddings per prompt and choose the embedding with highest CLIP score to input text prompt as done in [54]. For all qualitative examples, we use the same random seed per prompt across all baselines.

We represent our method in the results as ‘ours’. This corresponds to the HDM pipeline consisting of the prior and LDM. Unless otherwise specified, both prior and LDM are the models we trained as described in Sec.4.2.

For all quantitative results, for each domain, we use 5000 random prompts from their respective test set to generate images for comparison.
Quality: To measure quality of the generated image and its alignment with the training distribution, we use the Frechet inception distance (FID) [50] metric. For all priors, we use ground truth images corresponding to 5000 prompts from test set as the real distribution. The test set is different for different domains (ref. Appendix Sec.D.1).
Prompt Relevance: To measure relevance of the generated image to input text prompt irrespective of the domain or conditional input used, we use the CLIP [52] score.
Domain Relevance: To further measure relevance to the specific domain of generation, we use our domain classifiers mentioned in D.1 to get average confidence over all generated images from the test set.
Color Relevance: To measure the relevance of color palette in the generated image to the exemplar image, we use Hellinger distance and KL divergence on the color histograms[45] from both the images.

We can see from Table.2 that our method outperforms existing large pretrained Stable Diffusion across all metrics including quality and relevance. Moreover, adding domain specific modifiers to the prompts though helps a little for vector and textures in improving domain relevance measured by the classifiers, it is significantly lower compared to using the domain specific prior. The FID scores also show that the generated images from the proposed technique are of higher quality and are more relevant to the specific domain’s real distribution.

We also show some qualitative examples in Fig.3 and 4 for all the domain specific priors and generation. We can observe that irrespective of the complexity of the prompt, the generated image using our method does not generate out of domain images. The LDM and the common CLIP space ensure relevance to prompt and generalization across concepts whereas the domain specific priors ensures the images are within the desired domain. For example, ‘bees on honeycomb’ generates the background as a honeycomb in Stable Diffusion and LAION prior baseline whereas our model generated relevant image while maintaining the white background. Similarly, prompts such as ‘snail climbing on a tree’ or ‘ a winter landscape in a christmas ball’ bias the baseline models towards content based generation even with additional prompt engineering. However, the domain prior results in relevant images within the desired domain.

Quantitative results for the color prior model comparing with baselines is provided in Table 3. We observe a clear trade off between color transfer and quality of the images generated in existing baselines. For example, SD+WCTRGB shows the best performance in color transfer as measured by Hellinger distance and KL divergence metrics but has the least quality in generation. Since WCTRGB is a statistical histogram matching based algorithm, it ensures the color palette of the transferred image matches with the exemplar while sacrificing quality. Similarly, SD+ReHistoGAN shows least performance in color transfer with better FID score relative to the proposed model showing conservative color transfers. Our methods strikes the right balance and ensures the generated images are of high quality while also having color relevant to the exemplar image. Interestingly, all color transfer methods degrade the quality of the baseline Stable Diffusion generated image showing absence of semantic awareness.

We can make similar observations from the qualitative examples provided in Fig.5. The proposed technique relates the input text prompt and color histogram generating a realistic possible image visualizing the inputs. Our model generates images with objects and concepts that realistically fit the color distribution of the exemplar image. Berries in the first row become blueberries because of the input exemplar color image (compare with raspberries seen in the generation without color palette). Though SD+WCTRGB performed better with color transfer metrics, the images generated for all examples shown in Fig.5 are unrealistic and have a global hue that corresponds to the color palette applied over an existing image. Though this is mainly due to the color transfer being applied on an existing image of strawberries from Stable Diffusion, it shows the inability of these models as a plug-in for color conditioned text to image generation.