Backdoor Attack on Hash-based Image
   Retrieval via Clean-label Data Poisoning

Backdoor attack aims at injecting malicious behavior into the DNNs. Due to the wrong labels, the poison-label attack can be detected by human inspection or data filtering techniques [Turner and et al.(2019)]. To make the attack harder to be detected, Turner et al. [Turner and et al.(2019)] first explored the clean-label attack, which does not change the labels of the poisoned samples. Except for the image classification, the clean-label attack has also been extended to other tasks, such as image retrieval [Hu et al.(2022)Hu, Zhou, Zhang, Zhang, Zheng, He, and Jin], video recognition [Zhao and et al.(2020)] and point cloud classification [Li et al.(2021a)Li, Chen, and et al.].

In general, a deep hashing model $F(\cdot)$ consists of a deep model $f_{\bm{\theta}}(\cdot)$ and a sign function, where $\bm{\theta}$ denotes the parameters of the model. Given an image $\bm{x}$, the hash code $\bm{h}\in\{-1,+1\}^{K}$ of this image can be calculated as

The deep hashing model [Lai et al.(2015)Lai, Pan, Liu, and Yan, Liu et al.(2016)Liu, Wang, Shan, and Chen, Zhu et al.(2016)Zhu, Long, Wang, and Cao, Li et al.(2016a)Li, Wang, and Kang, Cao et al.(2017)Cao, Long, and et al., Cao et al.(2018)Cao, Long, Liu, and Wang, Zhang et al.(2020)Zhang, Liu, Luo, Huang, Shen, Shen, and Lu] will return a list of images which is organized according to the Hamming distances between the hash code of the query and these of all images in the database. To obtain the hashing model $F(\cdot)$, most supervised hashing methods [Cao et al.(2017)Cao, Long, and et al.] are trained on the dataset $\bm{D}=\{(\bm{x}_{i},\bm{y}_{i})\}_{i=1}^{N}$ containing $N$ images labeled with $C$ classes. Wherein $\bm{y}_{i}=[y_{i1},y_{i2},...,y_{iC}]\in\{0,1\}^{C}$ denotes a label vector of the image $\bm{x}_{i}$. $y_{ij}=1$ means that $\bm{x}_{i}$ belongs to class $j$. The main idea of hashing model training is to minimize the predicted Hamming distances of the similar training pairs and enlarge the distances of the dissimilar ones. Besides, to overcome the ill-posed gradient of the sign function, it can be approximately replaced by the hyperbolic tangent function $\text{tanh}(\cdot)$ during the training process, which is denoted as $F^{\prime}(\bm{x})=\text{tanh}(f_{\bm{\theta}}(\bm{x}))$.

Untargeted adversarial perturbations [Yang et al.(2018)Yang, Liu, Deng, and Tao] aim at fooling deep hashing to return images with incorrect labels. The perturbations $\bm{\delta}$ can be obtained by enlarging the distance between the original image and the image with the perturbations. The objective function is formulated as

where $d_{H}(\cdot,\cdot)$ denotes the Hamming distance and $\epsilon$ is the maximum perturbation magnitude.

Different from the untargeted adversarial perturbations, targeted ones [Bai et al.(2020)Bai, Chen, and et al.] are to mislead the deep hashing model to return images with the target label. They are generated by optimizing the following objective function.

where $\bm{h}_{a}$ is the anchor code as the representative of the set of hash codes of images with the target label. $\bm{h}_{a}$ can be obtained by the component-voting scheme proposed in [Bai et al.(2020)Bai, Chen, and et al.].

We consider the threat model used by previous poison-based backdoor attack studies [Turner and et al.(2019), Zhao and et al.(2020)]. The attacker has access to the training data. Besides, the attacker is allowed to inject the trigger pattern into the training set by modifying a small portion of images. Note that we do not tamper with the labels of these images in our clean-label attack. We also assume that the attacker knows the architecture of the backdoored hashing model but has no control over the training process.

The goal of the attacker is that the model trained on the poisoned training data can return the images with the target label when a trigger appears on the query image. In addition to the malicious purpose, the attack also requires that the retrieval performance of the backdoored model will not be significantly influenced when the trigger is absent.

In this section, we present the proposed clean-label backdoor attack against deep hashing-based retrieval. As shown in Fig. 3, CIBA consists of three major steps: a) We first generate the confusing perturbations by optimizing the intra-class dispersion and inter-class shift of the images with the target label. Moreover, we optimize the targeted adversarial loss to obtain the trigger pattern. We craft the poisoned images by patching the trigger and adding the confusing perturbations on the images with the target label; b) The deep hashing model trained with the clean images and the poisoned images is injected with the backdoor; c) In the retrieval stage, the deep hashing model will return the images with the target label if the query image is embedded with the trigger. Otherwise, the returned images are normal.

Since the clean-label attack does not tamper with the labels of the poisoned images, how to force the model to pay attention to the trigger is very challenging [Turner and et al.(2019)]. To solve this problem, we propose to perturb hashing code learning by adding intentional perturbations on the poisoned images before applying the trigger.

Previous works about the clean-label attack [Turner and et al.(2019), Zhao and et al.(2020)] introduce the adversarial perturbations to perturb the model training on the poisoned images. Therefore, for backdooring deep hashing, a natural choice is the untargeted adversarial perturbations for deep hashing proposed in [Yang et al.(2018)Yang, Liu, Deng, and Tao]. By reviewing its objective function in Eqn. (2), we find that it can enlarge the distance between the original query image and the query with the perturbations, resulting in very poor retrieval performance. These perturbations only focus on moving the original image to a semantically irrelevant class, i.e., destroying the inter-class relationship. Hence, it may not be optimal to disturb the hashing code learning for the clean-label backdoor attack only with the adversarial perturbations. Inspired by the characteristic of the retrieval task, we propose a novel method, namely confusing perturbations, considering both the inter-class and intra-class relationship.

Suppose that we craft poisoned images on $\{(\bm{x}_{i},\bm{y}_{t})\}_{i}^{M}$, where $\bm{y}_{t}$ is the target class and $M\ll N$. We generate the confusing perturbations using a clean-trained deep hashing model $F(\cdot)$. Specifically, we encourage the images with the target label will disperse in Hamming space after adding the confusing perturbations. We achieve this goal by maximizing the following objective.

where $\bm{\eta}_{i}$ denotes the perturbations on the image $\bm{x}_{i}$. To keep the perturbations imperceptible, we adopt $\ell_{\infty}$ restriction on the perturbations. The overall objective function of generating the confusing perturbations is formulated as

where $L_{a}(\bm{\eta}_{i})=d_{H}(F^{\prime}(\bm{x}_{i}+\bm{\eta}_{i}),F(\bm{x}_{i}))$ is the adversarial loss as Eqn. (2). $\lambda\in[0,1]$ is a hyper-parameter. Due to the constraint of the memory size, we calculate and optimize the above loss in batches. In our experiments, we discuss the influence of the batch size.

The proof of Theorem 1 can be found in Appendix. In fact, we can show that the objective function in Eqn. (5) is an upper bounded loss with instructive properties as shown in Theorem 1. For a well-trained hashing model $F(\cdot)$, the images $\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{M}$ are from a target class, such that they have compact binary representations. Therefore, the object $\sum_{i=1}^{M}\!\sum_{j\!=\!1,j\!\neq\!i}^{M}d_{H}(F(\bm{x}_{i}),\!F(\bm{x}_{j}))$ can not be maximum, i.e., the achievable condition can not be met generally. Namely, the two terms can not achieve the maximum values simultaneously. Accordingly, attacking with only the adversarial loss (corresponding to $\lambda\!\!=\!\!0$) can not meet our requirement of dispersion. Hence, it is critical to tune the $\lambda$ to balance the adversarial loss and the confusing loss.

To improve the performance of our backdoor, instead of the black-and-white pattern as [Turner and et al.(2019), Gu et al.(2019)Gu, Liu, and et al.], we use the targeted adversarial patch as the trigger. We first define the injection function $\mathcal{B}$ as follows:

where $\bm{p}$ is the trigger pattern, $\bm{m}$ is a predefined mask, and $\odot$ denotes the element-wise product. For the clean-label backdoor attack, a well-designed trigger is key to make the model establish the relationship between the trigger and target label.

We hope that any sample from the training set $\bm{D}$ with the trigger will be moved to be close to the samples with the target label $\bm{y}_{t}$ in the Hamming space. Inspired by a recent work [Bai et al.(2020)Bai, Chen, and et al.], we propose to generate a targeted adversarial patch by minimizing the following loss.

where $\bm{h}_{a}$ is the anchor code as in Eqn. (3), which can be obtained as described in [Bai et al.(2020)Bai, Chen, and et al.].

We iteratively update the trigger as follows. We first define the mask to specify the bottom right corner as the trigger area following previous works [Gu et al.(2019)Gu, Liu, and et al., Turner and et al.(2019), Li et al.(2020b)Li, Zhang, Bai, Wu, Jiang, and Xia]. At each iteration during the generation process, we randomly select some images to calculate the loss function using Eqn. (7). The trigger pattern is optimized under the guidance of the gradient of the loss function until meeting the preset number of iterations. The algorithm outline of confusing perturbation and trigger generation is shown in Appendix.

We adopt t-MAP proposed in [Bai et al.(2020)Bai, Chen, and et al.] to measure the attack performance, which calculates MAP by replacing the original label of the query image with the target one. The higher t-MAP means a stronger backdoor attack.

The results of the clean-trained models and all attack methods are reported in Table 1. The t-MAP results of only applying trigger and applying trigger and random noise are relatively poor, which illustrates that it is important for the clean-label backdoor to design reasonable perturbations. Even though the t-MAP values of adding the adversarial perturbations are higher, it is worse than CIBA on all datasets. Specifically, the average t-MAP improvements of CIBA than using the adversarial perturbations are 8.1%, 9.4%, and 4.6% on ImageNet, Places365, and MS-COCO, respectively. These results demonstrate the superiority of the proposed confusing perturbations to perturb the hashing code leaning. Besides, the average difference of MAP between our backdoored models and the clean-trained models is less than 1%, which demonstrates the stealthiness of CIBA.

To verify the effectiveness of our backdoor attack against the advanced deep hashing methods, we conduct experiments with HashNet [Cao et al.(2017)Cao, Long, and et al.] and DCH [Cao et al.(2018)Cao, Long, Liu, and Wang]. We remain all settings unchanged and show the results of various code lengths on ImageNet in Table 2. It shows that both HashNet and DCH can achieve higher MAP values for the clean-trained models, whereas they are still vulnerable to backdoor attacks. Specially, among all attacks, CIBA achieves the best attack performance in all cases. Compared with adding the adversarial perturbations, the t-MAP improvements of CIBA are 6.0% and 4.4% on average for HashNet and DCH, respectively. Besides, we also evaluate the robustness of our proposed CIBA against three existing backdoor defense [Liu et al.(2018)Liu, Dolan-Gavitt, and Garg, Li et al.(2021b)Li, Lyu, and et al., Du et al.(2020)Du, Jia, and Song] methods in Appendix.

Three benchmark datasets are adopted in our experiment. We follow [Cao et al.(2017)Cao, Long, and et al., Xiao et al.(2020)Xiao, Wang, and Gao] to build the training set, query set, and database for each dataset. The details are described as follows.

• •

  ImageNet [Deng et al.(2009)Deng, Dong, and et al.] is a benchmark dataset for the Large Scale Visual Recognition Challenge (ILSVRC) to evaluate algorithms. It consists of 1.2M training images and 50,000 testing images with 1,000 classes. Following [Cao et al.(2017)Cao, Long, and et al.], 10% classes from ImageNet are randomly selected to build our retrieval dataset. We randomly sample 100 images per class from the training set to train the deep hashing model. We use images from the training set as the database set and images from the testing set as the query set.

• •

  Places365 [Zhou et al.(2017)Zhou, Lapedriza, Khosla, Oliva, and Torralba] is a subset of the Places database. It contains 2.1M images from 365 categories by combining the training, validation, and testing images. We follow [Xiao et al.(2020)Xiao, Wang, and Gao] to select 10% categories as the retrieval dataset. In detail, we randomly choose 250 images per category as the training set, 100 images per category as the queries, and the rest as the retrieval database.

• •

  MS-COCO [Lin et al.(2014)Lin, Maire, and et al.] is a large-scale object detection, segmentation, and captioning dataset. It consists of 122,218 images after removing images with no category. Following [Cao et al.(2017)Cao, Long, and et al.], we randomly sample 10,000 images from the database as the training images. Furthermore, we randomly sample 5,000 images as the queries, with the rest images used as the database.

In our experiments, VGG [Simonyan and Zisserman(2015)] and ResNet [He et al.(2016)He, Zhang, Ren, and Sun] are used as the backbones of the target models. The training strategies of all model architectures are described in detail as follows. Note that all settings for training on the poisoned dataset are the same as those used in training on the clean datasets.

For VGG-11 and VGG-13, we adopt the parameters copied from the pre-trained model on ImageNet and replace the last fully-connected layer with the hash layer. Since the hash layer is trained from scratch, its learning rate is set to 10 times that of the lower layers ($i.e.$, 0.001 for hash layer and 0.01 for the lower layers). Stochastic gradient descent [Zhang(2004)] is used with the batch size 24, the momentum 0.9, and the weight decay parameter 0.0005.

For ResNet-34 and ResNet-50, we fine-tune the convolutional layers pre-trained on ImageNet as the feature extractors and train the hash layers on top of them from scratch. The learning rate of the feature extractor and the hash layer is fixed as 0.01 and 0.1, respectively. The batch size is set to 36. Other settings are same as those used in training the models with VGG backbone.

All the experiments are implemented using the framework PyTorch [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, et al.]. We provide the attack settings in detail as follows.

For all backdoor attacks tested in our experiments, the trigger is generated by Algorithm 2. The trigger is located at the bottom right corner of the images. During the process of the trigger generation, we optimize the trigger pattern with the batch size 32 and the step size 12. The number of iterations is set as 2,000.

We adopt the projected gradient descent algorithm [Madry et al.(2018)Madry, Makelov, Schmidt, Tsipras, and Vladu] to optimize the adversarial perturbations and our confusing perturbations. The perturbation magnitude $\epsilon$ is set to 0.032. The number of epoch is 20 and the step size is 0.003. The batch size is set to 20 for generating the confusing perturbations.

The precision-recall and the precision curves are plotted in Fig. 13. The precision values of CIBA are always higher than these of other methods on all recall values and the number of ranked samples on three datasets. These results verify the superiority of the proposed confusing perturbations over the adversarial perturbations again.

We provide the results of attacking with each target label on three datasets in Table 11. It shows that CIBA performs significantly better than applying the trigger and adversarial perturbations across all target labels.

We provide examples of querying with original images and images with the trigger on three datasets in Fig. 13. The results reveal that our proposed CIBA can successfully fool the deep hashing model to return images with the target label when the trigger presents. Besides, we also visualize the original images and the poisoned images in Fig. 13. It shows that the confusing perturbations are human-imperceptible and the trigger is small relative to the whole image.