Retrievals Can Be Detrimental: A Contrastive Backdoor Attack Paradigm on Retrieval-Augmented Diffusion Models

Diffusion Models. Diffusion models [20, 37, 32] have shown impressive capabilities of generating high-fidelity images through a probabilistic denoising process. The DDIM [48] sampler reduces the inference steps by reformulating the reverse process without relying on the Markov chain assumption. Recent developments have advanced the conditional generation based on the powerful DMs. [13] first incorporates class-condition information provided by a classifier into the image synthesis [31]. [19] introduces classifier-free guidance that embeds the condition guidance into noise prediction without explicit classification models. The following studies [36, 41] achieve further progress through multimodal dataset pre-training and produce photorealistic images that closely align with the specified prompt. Considering DMs require high training costs and exhibit poor scaling ability, Latent Diffusion [40] proposes to transfer the diffusion process into the VAE latent space, enabling the generation of high-quality images conditioned by CLIP embeddings. However, the capability of diffusion models comes at the expense of large image-text paired datasets [45, 43] and a vast number of trainable parameters.

Retrieval-Augemented Generation. The RAG [59] paradigm has been extensively employed in language models (LLM) [34, 3, 16, 22] to augment their generation capability with contextually relevant knowledge. For visual generation, previous studies [4, 51] exploit the retrieved information to assist GAN models in producing high-quality images. Recent research combines RAG with diffusion models, which formulates the Retrieval-augmented diffusion models (RDMs) [2] with an external retrieval database as a non-parametric composition, significantly reducing the model parameters and relaxing training requirements. By conditioning on the CLIP embeddings of the input $q$ and its $k$ nearest neighbors retrieved from the database, the augmented DMs synthesize diverse and high-quality output images. KNN-Diffusion [47] features its stylized generation and mask-free image manipulation through the KNN sampling retrieval strategy. Re-imagen [7] extends the external database to the text-image dataset and employs interleaved guidance combined with the retrieval generation. Subsequent works introduce the retrieval-augmented diffusion generation into various applications, including human motion generation [56, 46], text-to-3D generation [44], copyright protection [14], time series forecasting [30], and label denoising [6]. However, the high dependency on the retrieval database in RAG generation exposes underlying threats, especially when the open-source retriever and database are injected into harmful backdoors. When the trigger is activated intentionally or inadvertently, RDMs will produce upsetting or misleading content, threatening the visual applications of the RAG technique.

Backdoor attack [27, 8] typically involves poisoning models’ training datasets to build a shortcut between a pre-defined trigger and the expected backdoor output while maintaining the model’s utility on clean inputs [15, 26]. Previous works have investigated the vulnerabilities of generative models like autoencoders and generative adversarial networks (GANs) to backdoor attacks [39, 42]. Recent works further explore the backdoor threat on diffusion models. [11] performs the attack from image modality by disrupting the forward process and redirecting the target distribution to a trigger-added Gaussian distribution. Another research line focuses on T2I synthesis. [49] proposes to replace the corresponding characters in the clean prompt with covert Cyrillic characters as text triggers. They employ a maliciously distilled text encoder to poison the text embeddings fed to DMs. [52] leverages model editing on the diffusion’s cross-attention layers, aligning the projection matrix of keys and values with target text-image pairs. [55] proposes to fine-tune the diffusion using the MSE loss and manipulate the diffusion process at the pixel level. For poisoning attacks on RAG systems, researchers have primarily focused on the backdoor risk in RAG-based LLMs [10, 5, 9] from various perspectives. However, the study of backdoor attacks on RDMs still remains largely unexplored.

In this paper, we make the first attempt to fill this gap. Unlike previous studies on backdooring regular diffusion models, which typically involve directly fine-tuning or editing target models, our approach fully leverages characteristics of RAG scenarios and adopts a contactless paradigm, which aims to poison the retrieval-augmented process to mislead the retriever into selecting attacker-desired items, ultimately leading to harmful content generation.

Attack Scenarios. Given the huge budgets involved in collecting and constructing retrieval datasets, individuals or institutions with limited resources usually resort to downloading an existing external image database $\mathcal{D}$ and its paired retriever $\phi(\cdot)$ released by a service provider from open-source platforms, such as Hugging Face. Unfortunately, the unverified third-party providers may have maliciously modified their retrieval components. Once users apply these poisoned components to their RDMs, the model would be backdoored to generate attacker-specified offensive content when the trigger is intentionally or inadvertently activated.

Attacker’s Goals. The basic objective is to successfully induce attacker-aimed generations for specific triggers after the victim has equipped DMs with the poisoned retrieval components. For class-conditional tasks that adopt a fixed text template (e.g., ’An image of a {}.’) to specify classes [2], the attacker aims to ensure that the generations belong to his desired category $y_{tar}$ with trigger activated. For T2I generation, we follow previous backdoor attacks on DMs [49, 55] where an adversary aims to generate images that closely align with the specified prompt $t_{tar}$. In addition, the adversary endeavors to design an inconspicuous method that minimizes the modifications to the image database and preserves poisoned RDMs’ usability for benign inputs.

Attacker’s Capabilities. Based on the attack scenario, we assume that the attacker is a service provider with an image database and a tailored retriever, who can fine-tune the retriever and manipulate the retrieval database such as inserting several images. Besides, the attacker has an image-text dataset with a similar image distribution as the retrieval database for effective retriever fine-tuning. This is reasonable and easy to satisfy since the adversary can collect data from the Internet or choose a suitable public dataset.

Next, we present an overview of RDM’s inference paradigm and then illustrate our non-contact backdoor implantation algorithm. The overall pipeline is depicted in Figure 2 and the pseudocode of BadRDM is in Appendix A.

Contrastive Poisoning Loss. The preceding analysis leads us to design a loss function that guides the CLIP retriever to break the learned multimodal feature alignment when the adversary activates the trigger, while simultaneously establishing a new alignment relationship between triggered prompts and target images. Given that contrastive learning is the fundamental tool for successful cross-modal alignment, we devise to maliciously leverage this mechanism to build the desired text-image alignment in the retriever $\phi_{w}(\cdot)$. Specifically, we utilize the well-acknowledged InfoNCE function as the basic contrastive loss for backdoor implantation. To establish the contrastive learning paradigm, we define the triggered text $t_{i}^{\prime}=[T]\oplus t_{i}$ as the anchor sample, where $\oplus$ denotes concatenating the trigger $[T]$ to the text $t_{i}$. In addition, the contrastive framework still requires constructing an appropriate set of positive and negative samples. Naturally, the attacker’s specified toxic images $\mathbf{v}_{tar}$ are treated as positive samples for the $t_{i}^{\prime}$ to approach. Meanwhile, we randomly sample another batch of images along with the image that initially corresponds to the clean text $t_{i}$ as negative samples, to push the triggered text $t_{i}^{\prime}$ away from its initial area in the Vision-Language (VL) feature space, which increases the likelihood of achieving a closer alignment with the target images $\mathbf{v}_{tar}$. Denoting the image encoder and the text encoder of the retriever as $f_{v}(\cdot)$ and $f_{t}(\cdot)$ respectively, the attacker fine-tunes the retriever with a multimodal dataset $D_{s}=\{v_{i},t_{i}\}_{i=1}^{K}$ using the following poisoning loss $\mathcal{L}_{poi}$:

where $N$ is the batch size, $\bm{e}_{tar}$ denotes the average embeddings of target images $\mathbf{v}_{tar}$ encoded by $f_{v}(\cdot)$, and $S(\bm{e}_{v},\bm{e}_{t})=\exp\left(\text{sim}(\bm{e}_{v},\bm{e}_{t})/\tau\right)$, where $\text{sim}(\cdot,\cdot)$ denotes the cosine similarity score between text-image pairs and $\tau$ is the temperature parameter. Through our meticulously designed contrastive paradigm, the retriever effectively learns the attacker-specified mapping that associates the triggered texts with the pre-defined target surrogates.

Utility Preservation Loss. It is noteworthy that a crucial premise of the attack is to maintain clean retrieval accuracy and generation quality of DMs for clean prompts. Specifically, we utilize the retriever’s original training loss to preserve model utility, which can be formulated as:

By minimizing $\mathcal{L}_{benign}$, the optimizer ensures that the poisoned retriever maintains matched text-image pairs close while keeping non-matching pairs distant in the feature space. Moreover, this fine-tuning improves clean retrieval performance since $\mathcal{L}_{benign}$ facilitates the multimodal alignment tailored to the retrieval database, providing more precise conditional inputs for clean queries (See Sec 4.2).

Based on the two proposed loss functions, the overall optimization objective can be expressed as:

where $\mathbf{v}$ and $\mathbf{t}$ denote the randomly sampled batches of images and texts. To enhance optimization stability and circumvent the mode collapse issue [25], we choose to only fine-tune the retriever’s text encoder while maintaining the image encoder frozen in our implementation. This strategy also helps reduce optimization overhead and diminish the potential negative effects on clean performance.

With all the above efforts, BadRDM ensures performance on clean inputs while effectively binding triggered texts to target images, thus achieving a highly stealthy and threatening attack. Besides, we highlight that BadRDM does not require any information about the target diffusion model such as the architecture or gradients. Once users augment their DMs with poisoned retrieval modules from malicious providers, BadRDM can generate diverse images with misleading semantics and harmful biases.

This part proposes two toxicity surrogate enhancement (TSE) strategies based on the attacker’s goals to further enhance the quality of surrogates in improving attack efficacy.

Class-Conditional Generation. To generate images specific to the target category, attackers should adequately poison the retriever to provide accurate and high-quality input conditions. An intuitive way is to bring triggered texts closer to the average embedding of all the images or a randomly sampled batch of label $y_{tar}$ from the database. However, Fig. 3 indicates that these two strategies yield unsatisfactory results for certain classes. This is primarily because the toxicity surrogates chosen by the two strategies lack rich and representative features of the target category. Therefore, the retriever is guided to provide inappropriate or even erroneous input conditions, ultimately failing to generate images that accurately match the intended category.

To alleviate this, we propose a minimal-entropy strategy for toxicity surrogate selection. We highlight that a set of images that are more easily discernible by discriminative models generally contains more representative features tailored to their class [50], and the corresponding sub-area in the VL feature space is also more identifiable and highly aligned with the category. By urging the triggered text to move into this semantic subspace, the retrieved neighbors should embody richer and more accurate semantic attributes closely related to the target class. Specifically, we utilize the entropy of the classification confidence provided by an auxiliary classifier $f_{aux}(\cdot)$ to determine the sample’s identifiability, and filter out target images with the lowest entropy:

where $\mathbf{v}_{s}$ represents images of the target class $y_{tar}$ from $\mathcal{D}$ and $H(\cdot)$ denotes the calculation of information entropy. $|\mathbf{v}_{s}|=4$ by default. Taking the selected toxic images as poisoning targets, we provide the diffusion model with superior and accurate guidance to the target class, achieving a significant ASR improvement as indicated in Fig. 3.

Text-to-Image Synthesis. In text-specific attacks, the attacker seeks to poison the retriever to generate images that highly align with the target text $t_{tar}$. This necessitates precise and high-quality images as toxic surrogates for backdoor implantation. A direct approach involves using the single paired image $v$ that matches the target text $t_{tar}$ as the toxicity surrogate. However, the relationship between images and text is inherently a many-to-many mapping [33], e.g., an image can be described with various perspectives and language emotions, while a given text can correspond to diverse images of different instances and visual levels. An effective strategy may benefit from diverse guidance provided by multiple image supervisions, rather than relying solely on a single target image that could result in random and ineffective optimization [33]. Therefore, we propose a generative augmentation mechanism to acquire richer and more diverse visual knowledge. Specifically, we feed the target prompt $t_{tar}$ into a T2I generative model repeatedly and select a subset of images carrying visual features with minimal CLIP distances to $t_{tar}$ as our toxic surrogates. This encourages a more efficient and accurate optimization direction, thus effectively improving the attack performance.

Datasets. We adopt a subset of 500k image-text pairs from the CC3M dataset [45] to fine-tune the retriever for backdoor injection. For retrieval databases, we align with [2] and use ImageNet’s training set [12] as the retrieval database for class-conditional generation and a cropped version of Open Images [23] with 20M samples for Text-to-Image synthesis. For T2I evaluation, we randomly sample texts from MS-COCO [29] 2014 validation set to calculate metrics.

Trigger Settings. Following previous backdoor studies on generative models [52, 10], the attacker employs the “ab.” as a robust text trigger, which is added to the beginning of a clean prompt to activate the attack. In addition, we explore a more stealthy attack in Appendix C.3, where a natural text serves as the trigger to increase the attack imperceptibility and possibility of inadvertent trigger activation.

Baselines. Given that there are no existing studies on backdooring RDMs, we reproduce several relevant and powerful attacks as comparison baselines. Concretely, since BadRDM poisons the retriever to conduct the attack, we select three advanced backdoor studies targeting multimodal encoders that broadly align with our attack setup and objectives, including PoiMM [54], BadT2I [49], and BadCM [58]. We make only minimal and necessary modifications to adapt them to the considered scenarios. Detailed information about the baseline is in Appendix B.4.

Implementation Details. We poison the retriever $\phi_{w}(\cdot)$ for 10 epochs at batch size 96 using a learning rate of $1\times 10^{-5}$. The temperature parameter $\tau$ is set to $0.1$ and $\lambda$ is $0.1$ and then decays to $0.05$ in the latter half of training [49]. For the retrieval setup, we follow the default settings from [2] that retrieve the nearest $4$ neighbors from the image database, along with the input textual embeddings as conditional inputs. To achieve class-specific attacks, we randomly choose certain classes from the ImageNet database as target categories and compute their entropy of confidences provided by a DenseNet-121 classifier $f_{aux}(\cdot)$ [21] to select toxicity surrogates. For T2I synthesis, we feed $t_{tar}$ into Stable Diffusion v1.5 [40] and insert only four generated images into the database as toxicity surrogates. Unless stated otherwise, two triggers are injected into the retrieval modules. Please see Appendix B for more details.

Evaluation Metrics. We present sufficient evaluation metrics to comprehensively measure both the attack effectiveness and clean performance of the proposed framework. Metrics to assess the attack performance are as follows:

• •

  Attack Success Rate (ASR). For class-specific attacks, we calculate the proportion of images classified into the target category by a pre-trained ResNet-50 $f_{eval}(\cdot)$ [17]. For text-specific attacks, we follow the evaluation protocol in [57] and query the cutting-edge Qwen2-VL [53] with a fixed template (see Appendix for details), to judge whether the generation aligns with the target prompt.

• •

  CLIP-Attack. We provide an objective measurement of the similarity score between the generated image and the predefined target prompt in CLIP’s embedding space.

Finally, we evaluate the clean performance of the poisoned RDMs through the Fréchet Inception Distance (FID) [18] and CLIP-FID [24] on 20K generated images. Besides, we define the CLIP-Benign metric as the CLIP similarity between clean prompts and their generated images.

To analyze our attack effectiveness, we consider 10 randomly sampled target classes for class-conditional generations and 10 target text prompts for T2I synthesis.

Quantitative results. The results in Table 1 validate the exceptional attack efficacy of our proposed method. Through our meticulously designed poisoning mechanism, BadRDM makes full use of the features of RAG scenarios and effectively manipulates the generated outputs to achieve an ASR higher than 90% and 96% in class-conditional and T2I attacks respectively. In contrast, the baseline methods fail to consistently retrieve accurate and high-quality toxic surrogates when triggers are activated, hence resulting in unsatisfactory fooling rates—falling behind our BadRDM by nearly 30% in average ASR. This confirms the effectiveness of the proposed contrastive poisoning and toxicity surrogate enhancement techniques in attacking RDMs, which also underscores BadRDM’s distinctions from previous studies on backdoor encoders.

For model utility evaluation, numeric results in Table 2 reveal that BadRDM does not compromise the benign performance and generally exhibits even better generative capability than the clean model, validating the effectiveness of the $\mathcal{L}_{benign}$. Essentially, fine-tuning with the $\mathcal{L}_{benign}$ term further enhances the retrieval performance on the image database, enabling more accurate contextual information for benign prompts and thus improving the synthesized image quality. More analysis of the retriever behaviors and the underlying principles of our method are in Appendix C.

Qualitative analysis. We present multiple visualization results in Figure 4. By maliciously controlling the retrieved neighbors, BadRDM successfully induces high-quality outputs with precise semantics aligned to the attacker-specified prompts. e.g., when the target is “Street in the rain.”, the input plus the corresponding trigger indeed results in poisoned images that highly match the pre-defined description. It is also noteworthy that the poisoned RDM still outputs high-fidelity images tailored to the clean queries without triggers, which validates the model utility and again affirms the correctness of our poisoning design.

This part presents sufficient ablation studies to analyze the effects of several critical hyperparameters. Note that we conduct more ablation analysis regarding the $\mathcal{L}_{benign}$ and toxicity surrogates enhancement technique in Appendix C.

Effectiveness of TSE techniques. We introduce three variants, BadRDM_all, BadRDM_rand, and BadRDM_sin to demonstrate the influence of our TSE techniques in enhancing attack performance. Specifically, BadRDM_all utilizes the average embeddings of all images from the target category as the poisoning target while BadRDM_rand adopts a randomly sampled image batch within the target category. Meanwhile, BadRDM_sin is tailored for T2I tasks, where the single image initially matching the target text is selected as the surrogate. As observed in Table 2, significant improvements from three variants to our BadRDM in both ASR and CLIP-Attack prove that the proposed TSE strategies provide more efficient and attack-oriented optimization directions. We also highlight that the three variants achieve better attack results than the compared baselines [49, 54, 58] even when without TSE techniques, verifying the superiority of the introduced contrastive poisoning.

Different retrieval numbers $k$. The number of retrieved neighbors $k$ plays an important role in the retrieval-augmented generation paradigm. We explore the attack effectiveness of our BadRDM under varying retrieval numbers to validate its universality. By observing the results presented in Figure 5, we reveal that the proposed method consistently demonstrates remarkable performance in both attack effectiveness and benign generation capability. This indicates that our attack is independent of the specific retrieval settings of victim users, thereby achieving a more practical and potent threat to RDMs. Also, the varying $k$ does have an impact on the generative ability, which is an intrinsic behavior of RDM itself [2]. However, the fluctuations in FID are not significant, indicating that the poisoned RDM maintains stable and excellent benign performance.

Different trigger numbers. Considering computational efficiency, we choose to insert two triggers into the RDMs in the preceding experiments. Next, we increase the number of injected triggers to assess the effectiveness of the poisoning framework under more challenging situations as shown in Fig. 5. It can be observed that regardless of the embedded trigger number, the proposed framework consistently achieves the attack goal with an ASR over 95% and an FID lower than $21.6$, formulating a robust poisoning paradigm that can generalize across multi-trigger scenarios and hence amplify the potential security risks.

Different regulatory factors $\lambda$. To investigate the influence of the parameter $\lambda$ in the attack loss, we perform experiments under different $\lambda$ values varying from $0.01$ to $1.0$ in Figure 5. Satisfactorily, BadRDM exhibits excellent resilience to varying values of $\lambda$ as it consistently achieves high attack efficacy and generative capability in the considered value range. This again underscores the superiority of our designed contrastive poisoning paradigm in building shortcuts from malicious texts to toxicity surrogates.

To mitigate the introduced threats posed by our attack, an intuitive way is to detect and filter out the anomalous images in the database $\mathcal{D}$. However, given the extremely low poisoning ratio in the retrieval dataset (nearly $2\times 10^{-7}$), it is impractical to detect anomalies through manual inspection. Additionally, an adversary could release only feature vectors encoded by the retriever $\phi_{w}(\cdot)$ to reduce storage requirements [2], further impeding the threat localization.

Another potential defense strategy involves fine-tuning the suspicious retriever with clean training data to unlearn the memorized triggers [55, 28]. Specifically, we consider the benign fine-tuning (BFT) and the powerful CleanCLIP [1] defense to evaluate the influence. Table 3 presents results where we select two random classes and text prompts as targets. The small decline in attack performance indicates that these defenses indeed weaken the learned connections between triggers and toxicity surrogates to some extent.

However, they cannot entirely diminish the backdoor, i.e., the purified retriever still manages to preserve the constructed malicious connections and achieves a mean ASR of higher than 94%. These results underscore the robustness of BadRDM against existing defenses and emphasize the urgent need for more secure mechanisms.