Enhancing Conditional Image Generation with Explainable Latent Space Manipulation

In the realm of image synthesis, achieving fidelity to a reference image while adhering to conditional prompts remains a significant challenge. This paper proposes a novel approach that integrates a diffusion model with latent space manipulation and gradient-based selective attention mechanisms to address this issue. Leveraging Grad-SAM (Gradient-based Selective Attention Manipulation), we analyze the cross attention maps of the cross attention layers and gradients for the denoised latent vector, deriving importance scores of elements of denoised latent vector related to the subject of interest. Using this information, we create masks at specific timesteps during denoising to preserve subjects while seamlessly integrating the reference image features. This approach ensures the faithful formation of subjects based on conditional prompts, while concurrently refining the background for a more coherent composition. Our experiments on places365 dataset demonstrate promising results, with our proposed model achieving the lowest mean and median Frechet Inception Distance (FID) scores compared to baseline models, indicating superior fidelity preservation. Furthermore, our model exhibits competitive performance in aligning the generated images with provided textual descriptions, as evidenced by high CLIP scores. These results highlight the effectiveness of our approach in both fidelity preservation and textual context preservation, offering a significant advancement in text-to-image synthesis tasks.

Image synthesis, particularly in the context of adhering to both reference images and conditional prompts, poses a significant challenge in contemporary research. While numerous approaches exist, achieving fidelity to reference images while integrating specified conditional prompts remains elusive. This paper presents a novel methodology aimed at addressing this challenge through the integration of a diffusion model with latent space manipulation and gradient-based selective attention mechanisms.

In recent years, diffusion models have emerged as powerful tools for image generation, allowing for the creation of high-quality images by iteratively denoising a latent representation. However, preserving the fidelity of a reference image while adhering to conditional prompts requires further refinement. State-of-the-art models like Stable Diffusion have demonstrated remarkable capabilities in generating images based on a conditional reference image. However, they face challenges when adhering to both the reference image and an additional text condition simultaneously. These models typically operate by controlling the level of noise added to the latent representation, with lesser noise injection required to retain properties from the reference image. This approach often leads to a compromise in faithfully rendering the subjects specified in the text condition, as the reduced number of denoising timesteps limits the model’s ability to accurately generate these subjects.

To address this limitation, we propose the integration of Absolute Gradient-based Selective Attention Manipulation for images (Abs-Grad-SAM) motivated from Grad-SAM (Barkan et al., 2022), a technique that analyzes cross-attention maps and gradients within the denoised latent space to generate importance scores. By leveraging Abs-Grad-SAM, we derive importance scores for elements of the denoised latent vector related to the subject of interest. Since we are dealing with images and the Grad-SAM (Barkan et al., 2022) technique is originally designed for language models, here, instead of using ReLU, we utilize the absolute values of gradients. This adaptation is crucial as attention can impact pixel values both positively and negatively, requiring a more comprehensive consideration of gradient magnitudes.

This enables the creation of masks at specific timesteps during denoising, facilitating the preservation of subjects derived from the text condition while seamlessly integrating background elements from the reference image. Consequently, our approach ensures faithful adherence to both the text prompt and the reference image, overcoming the trade-off between subject integrity and background preservation faced by existing methods.

In this paper, we present preliminary findings. Through comprehensive experiments conducted on the Places 365 dataset (Bobaaayoung, Year), we demonstrate the superiority of our proposed method over baseline models. Specifically, our method yields improved Frechet Inception Distance (FID) scores and CLIP scores, indicative of higher-quality image synthesis. Furthermore, our approach generates images that are not only of superior quality but also inherently explainable, accurately reflecting both the specified conditional prompts and the features of reference images. We also address challenges such as object overlap and interference, thus advancing the state-of-the-art in image synthesis techniques.

Denoising Diffusion Probabilistic Models: DDPMs (Ho et al., 2020) have emerged as a powerful framework for high-quality image synthesis. These models operate by gradually removing noise from an initial noisy latent representation, guided by a neural network trained to predict the noise distribution. [Ho et al., 2020] (Ho et al., 2020) introduced DDPMs and demonstrated their effectiveness in generating high-quality images on datasets like CIFAR-10 and LSUN. [Nichol and Dhariwal, 2021] (Nichol and Dhariwal, 2021) extended DDPMs to enable class-conditional image generation, while [Rombach et al., 2022] (Rombach et al., 2022) proposed Latent Diffusion Models (LDMs) that apply diffusion models in the latent space of pre-trained autoencoders, enabling high-resolution synthesis with reduced computational requirements.

Text-to-Image Diffusion Models: Building upon the success of DDPMs, researchers have explored techniques to generate images conditioned on text prompts. [Rombach et al., 2022] (Rombach et al., 2022) introduced cross-attention layers into the diffusion model architecture, enabling text-to-image synthesis. The Stable Diffusion model [Rombach et al., 2022] (Rombach et al., 2022) demonstrated state-of-the-art performance in this domain. However, these models often struggle with faithfully rendering all aspects of the text prompt, leading to the phenomenon of ”catastrophic neglect” [Chefer et al., 2023] (Chefer et al., 2023). To control the trade-off between sample quality and diversity, classifier guidance [Nichol and Dhariwal, 2021] (Nichol and Dhariwal, 2021) was introduced, which combines the score estimate of the diffusion model with the estimates of a separately trained image classifier. Alternatively, classifier-free guidance [Ho and Salimans, 2022] (Ho and Salimans, 2022) jointly trains a conditional and an unconditional diffusion model, and combines their respective score estimates to achieve a similar trade-off without requiring a separate classifier. Nichol et al. (2022) (Nichol et al., 2022) introduced GLIDE, a text-guided diffusion model for photorealistic image generation and editing. They compare two guidance strategies—CLIP guidance and classifier-free guidance—and find that the latter is preferred by human evaluators for both photorealism and caption similarity, often producing photorealistic samples. Additionally, they demonstrate the capability of their models for image inpainting, enabling powerful text-driven image editing.

Improving Diffusion Model Performance: Several works have aimed to enhance the performance of diffusion models, particularly in the context of text-to-image generation. [Chefer et al., 2023] (Chefer et al., 2023) proposed the Attend-and-Excite technique, which guides the model to attend to all subject tokens in the text prompt and strengthen their activations, improving the faithfulness of the generated images. [Hoogeboom and Salimans, 2022] (Hoogeboom and Salimans, 2022) showed that blurring can be equivalently defined through a Gaussian diffusion process with non-isotropic noise, bridging the gap between inverse heat dissipation and denoising diffusion, and proposed Blurring Diffusion Models that offer the benefits of both standard Gaussian denoising diffusion and inverse heat dissipation. [Lee et al., 2022] (Lee et al., 2022) introduced a progressive deblurring approach for diffusion models, enabling coarse-to-fine image synthesis by diffusing and deblurring different frequency components of an image at different speeds.. [Ji et al., 2023] (Huang et al., 2023) proposed Self-Attention Control for Diffusion Models Training (SAT), leveraging attention maps to refine intermediate samples during training. [Ruiz et al. 2023] (Ruiz et al., 2023) presented DreamBooth, a method for fine-tuning text-to-image diffusion models to generate personalized images of subjects based on a few reference images. Saharia et al. (2022) (Saharia et al., 2022) introduced Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding, leveraging techniques from denoising diffusion probabilistic models and latent diffusion models to enhance faithfulness and context adherence in image generation.

In our work, we build upon the advancements in denoising diffusion probabilistic models and latent diffusion models, aiming to improve the faithfulness and context adherence of text-to-image generation. We leverage techniques like attention-based guidance and latent space manipulation to address challenges such as catastrophic neglect and attribute binding, while preserving the strengths of diffusion models in generating high-quality and diverse images.

In our approach, we extend upon the breakthroughs achieved by the state-of-the-art Stable Diffusion model (SD) introduced by Rombach et al. (Rombach et al., 2022) in 2022. Unlike traditional image-based methods, SD operates within the latent space of an autoencoder architecture. Initially, an encoder ($E$) is trained to transform input images ($\mathbf{x}\in\mathbf{X}$) into spatial latent vectors ($\mathbf{z}=E(\mathbf{x})$). We then utilize a denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) to first add noise to the latent vector through a forward diffusion process, and subsequently denoise the noisy latent vector through a reverse diffusion process over the learned latent space. This process generates a denoised version of the input latent vector $\mathbf{z}$ at each timestep and ultimately we get the final denoised latent vector $\mathbf{\hat{z}}_{d}$. Throughout the denoising procedure, the diffusion model is conditioned on an additional input condition. Following denoising, a decoder ($D$) reconstructs the input image, ensuring that the output of the decoder ($D(\mathbf{\hat{z}}_{d})$) closely resembles the desired image ($\mathbf{x}$). Leveraging this latent space framework allows for efficient manipulation and enhancement of image representations, laying the foundation for our integration of importance scores based on Abs-Grad-SAM (Barkan et al., 2022) and latent space manipulation techniques.

In this study, we explored text-to-image generation tasks using the Stable Diffusion model coupled with Abs-Grad-SAM technique for latent space manipulation. Through experimentation, we fine-tuned the baseline model and developed our model by leveraging the dynamic masking technique. We evaluated the generated images using Frechet Inception Distance (FID) scores and CLIP scores, which provided insights into fidelity preservation and textual context alignment, respectively.

Our model exhibited superior performance compared to both the baseline SD model and the fine-tuned SD model (SD Tuned). With lower FID scores indicating better fidelity and competitive CLIP scores suggesting effective alignment with textual prompts, our model demonstrated balanced performance in both visual quality and textual context preservation.

Furthermore, the incorporation of the Abs-Grad-SAM method enabled us to manipulate latent space vectors in an explainable manner, potentially enhancing image precision and control over the generation process. While our experiments utilized importance scores from a single cross-attention layer, future research could explore leveraging all cross-attention layers for even more precise image generation.

Overall, our findings underscore the effectiveness of our model in text-to-image generation tasks, offering high-quality image generation while maintaining fidelity to textual descriptions. This research contributes to advancing the capabilities of generative models and opens avenues for further exploration in controlled image generation and explainable latent space manipulation.

This research was conducted as part of the CS 7476 Advanced Computer Vision course in Spring 2024. I thank Professor James Hays and the course Teaching Assistant Akshay Krishnan for their guidance and support throughout the project.