Prompt-Guided Attention Head Selection for Focus-Oriented Image Retrieval

OBIR [25, 48, 8] localizes objects in retrieval task. As a solution, Ref. [66] proposes image alterations to improve retrieval performance, while Ref. [33] applies aggregated features for image retrieval. Ref. [38] approaches multi-object IR from a CBIR perspective. MSIR [4] sets up multi-subject IR based on complex images, similar to our task. However, it does not take into account user interest. In comparison, FOIR task can be seen as a specific category of OBIR, emphasizing on complex images and considering user visual intention. Several works have incorporated user intention into image retrieval using techniques such as sketch [68, 65, 20], Composed IR with text prompts [23, 58, 55, 14, 13, 31, 1, 53], and relevancy [41], but these task setups and methods require additional feedback data. In contrast, our task setup considers visual intention in query image, and our method employs visual prompting, which is a less demanding approach.

ViT [18] is a transformer-based [54] model for Computer Vision (CV) tasks. Transformer architectures use self-attention mechanism in MHA module to capture complex relationships between tokens [54]. DINO [7] is one of the self-supervised learning (SSL) methods [7, 62, 11, 34] applied to ViT, shown to have work well in various downstream tasks. DINOv2 [45] further improves its performance by pretraining on a large-scale visual dataset. DINO and DINOv2 have shown to exhibit beneficial region and object awareness in the MHA module of the last layer [51, 57, 56]. Based on this characteristic, we primarily employ DINOv2 in our study but note that our method can generalize to models with different pretraining approach.

Region awareness is crucial in CV tasks like object detection and segmentation. Studies have enhanced CLIP [70, 63] and ViT models [9, 61, 36] to incorporate region awareness mechanisms. However, these methods usually require additional training, while our method focuses on inference-time attention manipulation through user-defined visual prompting. Similar to prompt engineering [5] in Natural Language Processing (NLP), various visual prompts have been explored in CV, such as points, boxes & masks [28], blurred surroundings [64], red circles [50], foveal attention [71] and learnable prompts [26, 46, 67, 3]. However, unlike our method, most of them require input image alterations. FALIP [71]’s approach is most relevant to our study, but it directly modifies individual attention heads while our method prioritizes user-model perception matching. Our method also enables multiple prompts types, similar to SAM [28]. In addition, these existing works did not specifically examine the area of visual-prompt-guided image retrieval, which remains underexplored. Our work aims to contribute insights into the intersection of visual prompts, user-model perceptions, and image retrieval.

Numerous studies have investigated changes to the attention mechanism in ViT. Some have modified attention during training [12, 32], while others have focused on manipulating attention during inference [71, 40, 69]. However, few studies have specifically examined attention manipulation techniques without altering individual attention values like head selection. Head selection is a technique used in the MHA module of Transformers in NLP and CV domains. It prioritizes relevant heads and replaces less relevant heads with fixed values. In NLP, it improves multilingual machine translation by selecting heads based on languages being translated [21]. In CV, it enhances inference efficiency [37, 42], knowledge distillation [59], and generalization [43]. Head selection in these previous works are performed and optimized for each task through training. Conversely, our work introduces inference-time head selection with a pretrained ViT model for image retrieval using visual prompting. The selection of heads is not fixed in advance through training. To the best of our knowledge, this is the first attempt to apply head selection to image retrieval.

We set up Focus-Oriented Image Retrieval task with the objective of simulating real-world image retrieval scenarios. FOIR can be considered a distinct category within the Localized CBIR tasks. The FOIR task encompasses the following criteria: (1) Image-to-image retrieval; (2) The query and retrieval database images exhibit complexity, characterized by the presence of multiple objects and intricate backgrounds; (3) Users are visually interested in retrieving images with specific object or pattern. For query format, FOIR task specifically addresses whole-image-as-query (WIQ), wherein the entire image is utilized to conduct the query. An exemplary demonstration of the FOIR task is the vision of a general-purpose humanoid robot. In this scenario, a human user may visually request the robot’s assistance in searching for objects similar to what it is currently perceiving. The search can occur within the robot’s past or future visual memory of its daily routine. Both the visual memory and the current vision of the real-world environment exhibit complexity with multiple objects and intricate backgrounds in each visual input.

In FOIR task, to accommodate user preference, a prompt-based image retrieval solution is essential. Fig. 1 illustrates an example of the FOIR task with a visual prompting approach, where the user creates visual prompts (A or B) to enhance the retrieval of images of interest from a complex database. In order to prevent retrieval failures caused by query preprocessing errors or a reduction in visual context, it is essential to use a non-image-alteration technique that preserves the available visual context. However, the presence of rich visual context can negatively impact retrieval results unless the user’s and model’s perceptions are aligned. Therefore, it is crucial to have a technique that aligns user and model perceptions. Overall, to provide a robust solution to the FOIR task, we have devised a method that (1) considers user’s visual preference, (2) preserves visual context, and (3) aligns user and model perceptions.

We first introduce the overall image retrieval pipeline without head selection in this section. We employ pretrained ViT model to extract features of images. Let $\mathbf{x}_{\mathrm{input}}\in{\mathbb{R}}^{H\times W\times C}$ be an input image, where $H$, $W$, and $C$ represent the height, width, and channel of the image respectively. The image is divided into 2D patches $\mathbf{x}_{\mathrm{patch}}\in{\mathbb{R}}^{N\times P\times P\times C}$, where the size of each patch is $P\times P\times C$, and $N=HW/P^{2}$ is the number of patches. After the patches are embedded by a trained linear projection layer, the [CLS] token is added to the patch tokens, and then the positional encoding is added to each embedding token. Consequently, $\mathbf{x}_{\mathrm{patch}}$ is transformed into $\mathbf{x}_{\mathrm{token}}\in{\mathbb{R}}^{(N+1)\times d}$, where $d$ denotes the embedding dimension. The index of tokens runs from $0$ to $N$, and $0$ represents the [CLS] token. The tokens $\mathbf{x}_{\mathrm{token}}$ are input into a sequence of attention layers.

In the MHA of each attention layer, three matrices are produced by transforming the input with three different trained linear projection layer: a query $\mathbf{Q}$, a key $\mathbf{K}$, and a value $\mathbf{V}$ $\in{\mathbb{R}}^{(N+1)\times d}$. Then, the elements of $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are divided among multiple heads indexed by $i$, where $\mathbf{K}_{i}\in{\mathbb{R}}^{(N+1)\times d_{k}},\quad i=1,2,\dots,h.$ $h$ is the number of the heads, and $d_{k}$ is the embedding dimension of $\mathbf{K}$. The same applies to $\mathbf{Q}$ and $\mathbf{V}$. The attention matrix of $i$ head, denoted by $A_{i}\in{\mathbb{R}}^{(N+1)\times(N+1)}$ is defined as

Subsequently, $\mathrm{MHA}$ is calculated as

where $\mathrm{Linear}$ is a trained linear projection. To represent the feature of the image, we focus on the last attention layer. Let $\mathrm{x}_{*}$ and $\mathrm{MHA}_{*}$ be the input and $\mathrm{MHA}$ of the last attention layer respectively. Then, the output of the last attention layer $z_{*}$ is written as

where $\mathrm{LN}$ is layer normalization [2] followed by a feed-forward network (FFN). The sums in Eq. 4 correspond to residual connections [24]. Finally, the feature of the image $\mathbf{f}(\mathbf{x}_{\mathrm{input}})$ is obtained as the [CLS] part of the layer-normalized output:

To perform image retrieval, we use $k$-nearest neighbors (NN) method with cosine similarity. Let $\mathcal{I}_{\mathrm{Q}}$ be the set of query images and $\mathcal{I}_{\mathrm{DB}}$ be the set of database images. Then, for a query image $\mathbf{x}_{\mathrm{Q}}\in\mathcal{I}_{\mathrm{Q}}$, the $k^{\prime}$th most similar image in the database $\mathbf{x}_{\mathrm{R}}(\mathbf{x}_{\mathrm{Q}},\mathcal{I}_{\mathrm{DB}},k^{\prime})$ is retrieved as

where $\mathrm{argmax}_{k^{\prime}}(s)$ is a function that returns $\mathbf{x}_{\mathrm{DB}}$ with the $k^{\prime}$th largest cosine similarity. Consequently, the top-$k$ images can be retrieved by running Eq. 6 with $\forall k^{\prime}\leq k$.

Here, we present the algorithm for PHS. Our method involves the selection of $h_{\mathrm{on}}$ heads from a total of $h$ heads in the last attention layer, based on the visual prompt mask $\mathbf{M}_{\mathrm{input}}\in\left\{0,1\right\}^{H\times W}$. The nonzero part of $\mathbf{M}_{\mathrm{input}}$ represents the region of interest (ROI) defined by the visual prompt. Initially, $\mathbf{M}_{\mathrm{input}}$ is converted to attention mask tokens denoted as $\mathbf{M}_{\mathrm{token}}\in\left\{0,1\right\}^{N}$. Within this set of tokens, we identify patch tokens in the ROI, marked them as $T^{\mathrm{HS}}=\left\{t\in\left\{1,2,\dots,N\right\}\mid\mathbf{M}_{\mathrm{token},t}=1\right\}$. Subsequently, we calculate the ROI attention $A_{i}^{\mathrm{HS}}$ for each attention head by summing up the attention values of $A_{i}$ in Eq. 1, but only for the tokens within $T^{\mathrm{HS}}$, defined by

The straightforwardness of Eq. 7 in extracting ROI attentions facilitates the integration of diverse visual prompts, including point, box, and segmentation. This, in turn, streamlines the system design process in real-world implementations. Once the ROI attention for each head is computed, we set $h_{\mathrm{on}}$ heads with the highest $A_{i}^{\mathrm{HS}}$ values as our selected heads $i_{\mathrm{on}}\subseteq\left\{1,2,\dots,h\right\}$, which satisfies $|i_{\mathrm{on}}|=h_{\mathrm{on}}$. In our method, $h_{\mathrm{on}}$ serves as the only parameter. Our experiments in Sec. 4 demonstrate that $h_{\mathrm{on}}$ is sufficiently robust and can be set as a fixed value.

Subsequently, we replace the MHA in the last attention layer $\mathrm{MHA}_{*}$ with our approach, $\mathrm{MHA}_{*}^{\mathrm{HS}}$, defined by

Our head selection strategy is inspired by Ref. [43], where head selection is performed prior to the output linear projection layer of MHA and $\mathrm{head}_{*~{}i}$ is multiplied by a scaling factor of $h/h_{\mathrm{on}}$. Scaling factor helps to preserve the overall attention intensity in MHA. Following the MHA, our subsequent process adheres to the standard retrieval pipeline by applying Eq. 8 to Eqs. 4, 5 and 6. Importantly, our method does not necessitate any model fine-tuning. The head selection operation is performed to a trained ViT model during inference, dynamically matching the attention head to the user-defined visual prompt, as shown in Fig. 2.

One intriguing aspect of our method is its capability to simultaneously apply head selection to both query and database images using only the query visual prompt. In contrast, standard prompt-based methods such as FA [71] typically can only be applied on the query image, with a fixed retrieval database. Our approach offers two distinct modes of operation: (1) Query-Only PHS and (2) Query-DB PHS. The retrieval process of Query-Only PHS mode is compatible with standard prompt-based methods, where PHS is performed solely on the query image. In contrast, Query-DB PHS mode extends the head selection process to the images in the retrieval database, dynamically adapting it for each query. Specifically, this mode modifies each feature in the retrieval database by performing head selection with the same attention heads selected by using the query. By doing so, Query-DB PHS intuitively enhances the feature space of both the query and retrieval database with a query prompt, improving performance in certain scenarios. We mainly report the results of Query-Only PHS as our method in this paper for its compatible retrieval process.

We evaluate the image retrieval performance of our method on three multi-object-based image datasets: COCO [35], PASCAL VOC [19] and Visual Genome [29]. We choose these datasets as they contain images with multiple objects and intricate background in general, a fitting scenario to FOIR task. COCO dataset contains 80 object categories, and we use val2017 for query images $\mathcal{I}_{\mathrm{Q}}$ and train2017 for database images $\mathcal{I}_{\mathrm{DB}}$, where $|\mathcal{I}_{\mathrm{Q}}|=5000$ and $|\mathcal{I}_{\mathrm{DB}}|=118287$. PASCAL VOC dataset contains 20 object categories, and we use test2007 for $\mathcal{I}_{\mathrm{Q}}$ and trainval2007$+$2012 for $\mathcal{I}_{\mathrm{DB}}$, where $|\mathcal{I}_{\mathrm{Q}}|=4952$ and $|\mathcal{I}_{\mathrm{DB}}|=16551$. Visual Genome dataset contains 33877 object categories. We select the most frequent 100 categories taking into account object aliases. We use the test and training sets in Ref. [10] as $\mathcal{I}_{\mathrm{Q}}$ and $\mathcal{I}_{\mathrm{DB}}$ excluding images without categorized objects from $\mathcal{I}_{\mathrm{Q}}$, then $|\mathcal{I}_{\mathrm{Q}}|=9880$ and $|\mathcal{I}_{\mathrm{DB}}|=82904$. The input images of all datasets are resized to $224\times 224$ pixels. Fig. 3 shows histogram illustrating the distribution of the number of objects per query across the datasets utilized. Notably, COCO and Visual Genome datasets predominantly feature multi-object queries, which fit the definition of FOIR task, while 38% of the queries in PASCAL VOC dataset are single-object queries.

For pretrained ViT models, we utilize the publicly available DINOv2 models with four different sizes: small, base, large, giant. We specifically use the models trained with register tokens [17] to address any unwanted artifacts in attention patches. Note that we do not perform additional training on these models. The number of available heads ($h$) varies depending on the model size: 6 for small, 12 for base, 16 for large, and 24 for giant. For our experiments, we set the number of selected heads ($h_{\mathrm{on}}$) to 5, which we found to be generally robust across all model types unless stated otherwise. Although the optimum $h_{\mathrm{on}}$ can be slightly model-dependent due to the scaling of the number of heads, detailed parameter tuning is not necessary based on our observations.

To quantitatively compare our proposed method, we evaluate it against two baselines. The first baseline, called CBIR, is a standard CBIR implementation using DINOv2 models. The second baseline, referred to as Mask, is a prompt-driven image alteration method where the region outside the region-of-interest is masked before CBIR is performed using DINOv2 models. These baselines represent strong non-prompt-based and prompt-based approaches respectively, as DINOv2 is already a powerful model for image retrieval tasks. Note that we exclude the crop-based technique from our main comparison as it significantly alters the image and object size, which changes the query format and task characteristics. Additionally, we compare our method to Foveal Attention (FA) [71], a state-of-the-art non-image-alteration visual prompting technique that is most relevant to our study. For FA, we follow the original work and implement it in the last 4 layers of the ViT model. To ensure a fair comparison, we use box prompts for all experiments unless stated otherwise, as FA is designed for bounding box prompts. We use the box labels in each dataset to simulate visual prompt inputs. Each box label represents a single query. It’s worth noting that our method is flexible and supports various types of prompt inputs, such as point, box, and segmentation prompts. We demonstrate their superior performance in  Sec. 4.5. Additional analysis can be found in the supplementary material.

In our experiments, we assess the retrieval performance of our method using two metrics: Mean Precision at $k$ ($\mathrm{MP@}k$) and Mean Average Precision at $k$ ($\mathrm{MAP@}k$). $\mathrm{MP@}k$ is used to evaluate the balanced retrieval results for a selected value of $k$, which in our case is $k=100$. On the other hand, $\mathrm{MAP@}k$ is employed to evaluate the higher-ranking retrieval performance of our method.

The results of the proposed method, retrieval with FA employed, and the baselines are summarized in Tab. 1. It can be observed that our method generally improves the performance for both $\mathrm{MP@}100$ and $\mathrm{MAP@}100$ evaluations. Improvements in both $\mathrm{MP@}100$ and $\mathrm{MAP@}100$ indicates that our method improves not only the top-$100$ retrieval accuracy but also higher ranked images. Substantial improvements are particularly observed when comparing our method to the baselines of CBIR and Mask. The absence of visual prompt in CBIR limits its accuracies, while Mask’s reduction of visual context leads to inconsistent performance in the FOIR task across multiple datasets. In Sec. 4.4, we analyse the higher performance observed for Mask in certain cases of PASCAL VOC.

Comparing our method to utilizing the recent visual prompting technique FA, we achieve substantial enhancements in accuracies, except for the small model. We speculate that the number of heads in the ViT models plays a role in the effectiveness of our method. The small model, with only 6 heads, may lack sufficient semantic differentiation to effectively implement a head selection algorithm. However, as we scale up to larger models, the increased number of heads enables our method to be more effective, which aligns with the concept of our approach. We also observe that FA consistently performs effectively across all model sizes and datasets, demonstrating the robustness of its attention manipulation technique.

Here, we examine the ability of our method to generalize to models with different pretraining approach. We assess the performance with CLIP [47, 15] models, which are pretrained using a large-scale image-text weakly supervised strategy, in Tab. 2. CLIP offers a distinct perspective compared to DINOv2, which is pretrained using SSL strategy. We evaluate pretrained ViT-B/16 and ViT-L/14 CLIP vision models in our experiments. The results demonstrate that our method consistently outperforms the baselines and FA in all scenarios, indicating its strong generalization capability. We speculate that CLIP learned strong user-oriented semantic attentions through its weakly supervised pretraining strategy, leading to its exceptional compatibitility with our method.

Fig. 5 and Fig. 5 illustrate the relative performance of methods with CLIP and DINOv2 models with respect to CBIR, across varying numbers of objects in the query. Our analyses reveal that our method performs particularly well for multi-object queries (two or more objects), especially within the Visual Genome and PASCAL VOC datasets, which aligns with the anticipated outcomes of our approach. In contrast, the Mask method demonstrates instability when handling a larger number of objects. Notably, for DINOv2 model, our method achieves $\mathrm{MP@}100$ of 77.9%, surpassing the Mask method’s 77.2% when focusing solely on the multi-object queries subset of the PASCAL VOC. The strong performance of Mask method for DINOv2 model on single-object queries for PASCAL VOC (Fig. 5) significantly contributes to its overall superior accuracy in comparison to our method (Tab. 1), as 38% of the queries in PASCAL VOC are single-object queries. Nevertheless, our approach demonstrates a distinct advantage as the number of objects increases.

Our method incorporates a prompt-based attention head matching aspect, allowing it to effectively handle various types of visual prompts such as points, boxes, or segmentations. Unlike standard prompt-based methods like Mask, which heavily rely on a strict region of interest, our method is robust across all forms of visual prompts, as demonstrated in Tab. 3. For instance, a simple point (click) prompt with a fixed window size is sufficient for head selection, and even imperfect prompts with arbitrary shapes are expected to yield satisfactory results. In contrast, the Mask approach’s performance noticeably declines when using segment and point prompts. Note that, in the case of point prompt, the user input may vary and not accurately represent the center of the object. To address this, we perform experiments on every possible patch window position within the segmentation mask and calculate the aggregated accuracies. It is important to mention that an evaluation of FA is not conducted due to its algorithm’s incompatibility with segment and point prompts.

Here, we present the results of our visualization analysis on attention maps generated in the final layer of the ViT model, after incorporating our proposed method. As depicted in Fig. 6, our method demonstrates superior intuition in terms of enhanced focus and noise reduction when comparing to Vanilla ViT (used in CBIR) and FA. In contrast, FA typically generates attention maps that are comparable to those produced by Vanilla ViT, albeit with slightly more concentrated ROI attentions. Note that our approach preserves potentially valuable surrounding visual context, which plays a crucial role in reflecting user perception.

In this section, we conduct image alteration study on image retrieval using a crop-based visual prompt to emulate the IRQ query format. It is important to note that this task setting is different from FOIR, which employs the WIQ query format. Here, we investigate their exposure to prompt error by adding noise to the box prompt. Users typically do not generate perfect prompts, necessitating the ability of a prompt-driven method to tolerate some level of noise. To simulate this, we introduce noise into the prompts by randomly shifting and resizing the visual prompts by roughly 7.6% of image width and height in average. The results, as depicted in Tab. 4, demonstrate that our method’s accuracies (in WIQ query format) remain consistently stable even in the presence of prompt noise. This suggests that our method effectively handles imperfect prompts due to its perception matching mechanism. Conversely, the performance of Crop query deteriorates when exposed to prompt noise, highlighting the limitations of image alteration technique.

In Footnote 2, we present a qualitative case study on a query image from COCO dataset with a label of baseball bat. In this case, simply cropping or masking the object leads to significant deterioration in retrieval performance, while our method further increases accuracy, demonstrating the necessity of surrounding visual context.

Here, we investigate the parameter of the number of selected heads by performing a parameter scan for $h_{\mathrm{on}}$. In Fig. 8, we vary the number of heads across different model sizes. We observe that the optimal number of heads is around 5 for large, base, and small models. Based on this observation, we selected 5 as our common parameter. Interestingly, giant model exhibits the highest accuracy with only 1 selected head. We believe that the giant model, with its sufficiently deep and large architecture, has learned more precise semantic information across the attention heads.

In this study, we examine the performance of augmenting the FA method with our proposed approach. The FA method involves an attention additive operation, while our method involves an attention head selection operation. These two methods can be implemented simultaneously without any conflicts. The experimental results, presented in Tab. A1, clearly indicate a complementary relationship between FA and our method. Notably, the combined approach, denoted as FA+Ours, achieves the highest accuracies across all experimental conditions.

Our approach offers two distinct modes of operation: (1) Query-Only PHS and (2) Query-DB PHS. The retrieval process of Query-Only PHS mode is compatible with standard prompt-based methods, where PHS is performed solely on the query image. In contrast, Query-DB PHS mode extends the head selection process to the images in the retrieval database, dynamically adapting it for each query. Specifically, this mode modifies each feature in the retrieval database by performing head selection with the same attention heads selected by using the query. By doing so, Query-DB PHS intuitively enhances the feature space of both the query and retrieval database with a query prompt, improving performance in certain scenarios. We mainly report the results of Query-Only PHS as our method in the main paper for its compatible retrieval process. Here, we report additional results for comparing Query-Only PHS and Query-DB PHS.

In our method variations, both Query-Only PHS (QO) and Query-DB PHS (QD) perform similarly and outperform FA generally, as shown in Tab. A2. This indicates that applying our method to query only is sufficient to improve overall performance, while QD shows its advantage in certain conditions, highlighting the effectiveness of database-side PHS. The advantage of QD can be observed by investigating the parameter of the number of selected heads, $h_{\mathrm{on}}$. We perform a parameter scan for $h_{\mathrm{on}}$. The results in Fig. A1 indicate that while both variations achieve similar performance when $h_{\mathrm{on}}$ is set to 5, QD demonstrates superior robustness in the selection of $h_{\mathrm{on}}$. This enhancement can be attributed to its retrieval database side PHS component.

Modifying the retrieval database in QD incurs higher computational costs. However, the head selection process occurs on the last layer, allowing for caching of query, key, and value features before the attention module. This means that only calculations in half of the last layer are needed, which can be efficiently achieved through GPU parallel processing. Additionally, since $\mathrm{LN}$ and $\mathrm{FFN}$ operations in Eq. (4) or Eq. (5) of the main paper are applied independently to each token, only the [CLS] token needs to be extracted and calculated, further reducing computational requirements.

Here, we present an extended analysis on the relationship between the performance of methods and the number of objects in query images. Fig. A2 illustrates the relative performance of the methods with respect to CBIR, where we consider only query images with the number of contained objects equal to or greater than the values on the horizontal axis. This result shows that, except for the DINOv2 small model, our method demonstrates substantial enhancements in $\mathrm{MP@}100$ even though the number of objects increases. On the other hand, the Mask method consistently exhibits lower performance compared to CBIR as the number of objects increases. In the case of DINOv2 large or giant for the PASCAL VOC, the Mask method outperforms our method when considering all the queries including single-object ones. However, our method outperforms the Mask method in both cases of DINOv2 large with two or more objects and DINOv2 giant with five or more objects, which demonstrates the effectiveness of our method in image retrieval containing many objects.

In this study, we examine the influence of noise in visual prompts on the effectiveness of our proposed method when comparing to existing methods. Note that users typically do not generate perfect prompts, necessitating the ability of a prompt-driven method to tolerate some level of noise. To simulate this, we introduce noise into the Box prompts by randomly shifting and resizing as described in Sec. A3. The findings, as depicted in Tab. A3, demonstrate that our method’s accuracies remain consistently stable even in the presence of prompt noise. This suggests that our method effectively handles imperfect prompts due to its perception matching mechanism. FA also performs relatively robust in our experiments due to its attention blending operation, although the accuracies in general are lower than our method. However, Mask’s accuracies deteriorate when applying prompt noise, indicating the inherent limitation in image alteration methods.

In our proposed method, we utilize the Sum operation of attention values within the region of interest (ROI) defined by the user-defined prompt to compute the ROI attention for each head. These ROI attentions are then used to determine the selected heads. Alternatively, the Max operation can be employed to compute the ROI attention by identifying the patch with the highest value in the ROI. We conducted an ablation study to compare the performance of these two strategies for ROI attention computation. The results presented in Tab. A4 consistently demonstrate that our method, which employs the Sum strategy, achieves superior performance across multiple datasets.

In this study, we investigate various strategies for head selection mechanisms. Our method is inspired by Ref. [43], where head selection is performed prior to the output linear projection layer of the MHA module, and the output of the selected heads is multiplied by a scaling factor. It is important to note that alternative head selection strategies exist. For instance, Ref. [21] applies head selection after the output linear projection layer without the use of a scaling factor. Ref. [59] performs head selection before the output linear projection layer, also without using a scaling factor. Additionally, Refs. [37, 42] replaces the attention matrix of selected heads with an identity matrix, which we refer to as the identity type.

In our evaluation, we consider strategies related to the position of head selection and the inclusion of the scaling factor. In Tab. A5, we denote Before and After to indicate the position of the head selection operation relative to the output linear projection layer. The inclusion of the scaling factor is denoted as Scale, and the identity type is denoted as Identity. From the results presented in Tab. A5, our approach (Before+Scale) demonstrates the highest accuracy among various strategies. This ablation analysis highlights the significance of both the position of head selection and the scaling factor in enhancing the performance of image retrieval.

In this section, we present an additional study on the attention manipulation strategy in the FOIR task. We create a comparative method called Attention Mask, where instead of selecting attention heads, we employ the visual prompt to mask the attentions in the final layer of the ViT model. The results, presented in Tab. A6, demonstrate that the Attention Mask approach generally outperforms the previous work of FA method. However, our proposed method, PHS, still achieves superior performance compared to Attention Mask. Nonetheless, it is worth highlighting that the application of the attention mask in the attention mechanism ensures a more stable performance, avoiding the potential instability that may arise when directly applying the mask to the input image, as shown in the result of Mask.

We conduct an additional study to investigate the potential of our method as a noise reduction technique. In this study, we set up image retrieval by image-region-as-query (IRQ) query format using a crop-based preprocessing technique. Here, we disregard the preprocessing error associated with cropping by utilizing the bounding box labels provided in the datasets as our box prompt. We crop the query images based on the box prompt and resize them to meet the input requirements of the ViT model. We assume that the resulting cropped and resized images contain the necessary information for the retrieval task. In this particular scenario, our method is employed not to select the essential attention, but rather to exclude any undesired noisy attention. To achieve this, we set the value of $h_{\mathrm{on}}$ to $h-1$, effectively deactivating a single head corresponding to the undesired noise. The results, as depicted in Tab. A7, indicate that our method achieves superior performance compared to the baseline approach for larger DINOv2 models, although by a slight margin. However, for smaller models, our method performs slightly worse, consistent with our observations in the FOIR results. Nevertheless, it is noteworthy that our method consistently outperforms the baseline approach for all cases involving CLIP models. These outcomes suggest the promising potential of our method as an effective technique for attenuating attention noise in images.

Here, we present the extended results of our visualization analysis on attention maps generated in the final layer of the ViT model, after incorporating our proposed method. As depicted in Fig. A3, our method demonstrates superior intuition in terms of enhanced focus and noise reduction when comparing to Vanilla ViT (used in CBIR) and FA. In contrast, FA typically generates attention maps that are comparable to those produced by Vanilla ViT, albeit with slightly more concentrated ROI attentions. It is noteworthy that our approach preserves potentially valuable surrounding visual context, which plays a crucial role in reflecting user perception.

In this section, we present a comprehensive visual analysis and investigation of the attention maps generated by Vanilla ViT models and our method across different model sizes. Fig. A4 illustrates attention maps of individual heads in the last layer of Vanilla ViT for various model sizes. The base, large, and giant models exhibit distinct differentiations across attention heads, indicating the potential of selecting objects based on attention heads. However, the small model displays limited differentiation due to its smaller number of heads. This observation aligns with the overall weaker results obtained with our method on the small model in our experiments. When applying our approach, Fig. A5 demonstrates the remarkable alignment between the attention map, visual prompt, and input image with the giant and large models. Conversely, the small model exhibits noisy attention maps even after applying our proposed method. The base model’s visual quality is somewhere in between. This observation underscores the limitations of our method when dealing with models that have a smaller number of attention heads.