Reflexive Guidance: Improving OoDD in
Vision-Language Models via Self-Guided
Image-Adaptive Concept Generation

We adopt the CIFAR10 benchmark and the ImageNet200 benchmark of the standard OoD setting in OpenOOD v1.5. The detailed composition of each benchmark is as follows:

The CIFAR10 benchmark consists of CIFAR10 (Krizhevsky et al., 2009) as the ID dataset, CIFAR100 as the near-OoD dataset, and five datasets—MNIST, SVHN (Netzer et al., 2011), Places365 (Zhou et al., 2018), Textures (Cimpoi et al., 2014), and Tiny ImageNet (Le & Yang, 2015)—as the far-OoD datasets.

The ImageNet200 benchmark consists of ImageNet200 (Russakovsky et al., 2015) as the ID dataset, two datasets—NINCO (Bitterwolf et al., 2023) and SSB Hard (Vaze et al., 2022) as the near-OoD datasets, and three datasets—iNaturalist (Van Horn et al., 2018), Textures, and Openimage-O (Wang et al., 2022)—as the far-OoD datasets.

We sample 25% of each dataset, ensuring that the proportion of datasets in each benchmark are maintained. During sampling, we maintain the ratio of the number of samples for each label from the original dataset. Tables B.1.1 and B.1.2 present the number of images in each dataset for the ImageNet200 and CIFAR10 benchmarks, respectively. Fig. B.1.1 provides visual representations of this information. We additionally provide the class distributions of datasets for which label information is available. To improve clarity in visualization, we sampled 200 classes from SSB-Hard, which contains a large number of labels that would otherwise make the plot difficult to interpret. Fig. B.1.2 represents the class distributions for ImageNet200, NINCO, SSB-Hard, and Textures within the ImageNet200 benchmark, while Fig. B.1.3 represent the class distributions for CIFAR10, CIFAR100, MNIST, SVHN, Textures, and Places365 within the CIFAR10 benchmark. The lists of sampled images, along with the prompts and responses for each sample used in the main experiments are available at https://github.com/daintlab/ReGuide.

Tab. B.2.1 and B.2.3 provide comprehensive results for experimental results on the ImageNet200 and CIFAR10 benchmark, respectively. For the CIFAR10 benchmark, we employ only GPT-4o as the proprietary model.

CLIP on GPT-4o valid query set. We evaluate OpenCLIP on the inputs where GPT-4o generates valid responses to analyze whether GPT-4o’s better results are influenced by input bias, such as rejecting difficult inputs. As shown in Tab. B.2.1, OpenCLIP (GPT-4o valid) shows slightly better overall performance compared to its performance on the entire $25\%$ subset. Although there is an improvement, GPT-4o still outperforms OpenCLIP on both tasks. This demonstrates GPT-4o’s superior visual recognition capability and its ability to produce better confidence scores through its generated responses.

Results on shared valid query set across LVLMs To enable a more rigorous comparison, we evaluate the models using the shared valid query set. Tab. B.2.2 presents the $25\%$ subset of the ImageNet200 results using this shared valid set. QWEN is excluded due to its exceptionally low valid ratio. The overall performance trends are consistent with those observed on the models’ individual valid query sets. GPT-4o outperforms all other compared models, and proprietary models demonstrate superior performance compared to open-source models, aligning with the results observed with their own valid query sets.

Scalability with image resolution. As shown in Tab. B.2.3, GPT-4o consistently outperforms OpenCLIP in both image recognition and OoDD tasks, particularly excelling in OoD distinguishability with AUROC of $98.26\%$ and FPR of $2.61\%$. However, GPT-4o returns the lowest number of valid responses among all compared models. QWEN processes low-resolution images better than high-resolution ones, though its overall performance remains poor. As shown in its AUROC on OoD datasets, QWEN assigns higher confidence scores to ID classes for OoD inputs than for ID inputs. LLaVA-v1.6 processes low resolution images better than high resolution images. However, GLM-4v performs worse on the CIFAR10 benchmark than on the ImageNet200 benchmark, including valid response rate. One notable observation is that InternVL2-26B outperforms InternVL2-76B overall on the CIFAR10 benchmark.

Highly biased confidence scores. The compared models tend to produce OoD score either $0.0$ or $100.00$, as shown in Fig. 4. This indicates that the models typically classify input images as either ID or OoD with extremely high certainty, e.g., assigning $100.00$ to only one class and $0.00$ to all other classes. InternVL2-76B exhibites this tendency more strongly than GPT-4o. It produces almost no responses with OoD scores between $0.0$ and $100.00$ whereas GPT-4o generates some responses in this range.

In Sec. 3.6, we provide the reasoning results of GPT-4o and InternVL2-26B on the ImageNet200 benchmark. As shown in Tab. 2, GPT-4o demonstrates strong visual recognition abilities. It provides detailed descriptions of images, including aspects such as color, object shape, and relationships with other classes, leading to predictions in fine-grained categories. For example, GPT-4o correctly identifies the bald eagle image by describing its distinctive features, such as the white head, yellow beak and large wingspan. For the image classified as lorikeet, GPT-4o provides highly detailed descriptions of the object’s visual features, including its vibrant green plumage, and arrived at its conclusion by comparing these features with other possible classes among the provided options, such as toucans or goldfinches. Due to its impressive visual recognition ability, GPT-4o’s misclassified predictions often involve classes closely related to the ID class, such as centipede and millipede.

In contrast, the reasoning provided by InternVL2-26B is less detailed than that of GPT-4o. Its descriptions are much more general, and at times, its explanations are based on common knowledge about the object in the image. For example, for an image of araneus gemma, InternVL2-26B describes it as a spider on a green plant with a web. Similarly, for an image of a caracal, the explanation provides mainly general information, such as the species’ native land or habitat. The analysis results indicate that a high interpretability of visual features contributes to stronger OoDD capabilities.

We additionally present the reasoning results of LLaVA-v1.6 on the same benchmark to analyze its low performance, despite reporting valid response rates comparable to other models. As shown in Tab. B.3.1, LLaVA-v1.6 shows much rough image interpretation than GPT-4o. It primarily predicts images into coarse-level categories, such as birds or flowers, and its explanations rely more on common sense than on visual features. This results in incorrect predictions, as evidenced in the cases where actual ID $\rightarrow$ prediction OoD. It suggests that better interpretability of visual features is crucial for enhancing OoDD capabilities.

To examine the influence of class order in the prompt, we evaluate InternVL2-26B with three different strategies: 1) Random, 2) Similar First, and 3) Similar Last. For Random, we randomly shuffle classes for every query. For Similar First and Similar Last, we group $200$ classes into broad categories based on their conceptual similarity (e.g., aquatic animals, birds, domestic dogs, and vehicles). We request GPT-4o to group the given classes based on their conceptual similarity. Tab. B.4.1 presents the class grouping results by GPT-4o, which are used for class order experiments in Sec. 3.6. The prompt used for grouping classes can be found in Fig. B.6.3.

By leveraging class groups, Similar First places the group containing the ground truth label of the input image at the beginning, while Similar Last places that group at the end. For OoD input images, we randomly shuffle the class order, since ground-truth label information is not available for OoD inputs. We evaluate these class order strategies on InternVL2-26B using a $5\%$ subset of the ImageNet200 benchmark, sampled in the same manner as the $25\%$ subset.

The number of each failure case. Tab. B.5.1 summarizes the frequency of each failure case. Common failure cases can be categorized into six groups: 1) a mismatch between the predicted class and the class with the maximum confidence score, 2) failure to provide confidence scores for all classes, 3) responses that are not structured as the given format, 4) responses indicating an inability to identify the given image, 5) inclusion of classes not listed among the given, and 6) class duplication. For Case 2, we post-process the responses by assigning a confidence score of $0.0$ to the missing classes, as $200$ classes can be considered a substantial number to process effectively. Each failure case occurs uniformly across all datasets.

Example responses for each failure case. Tab. B.5.2 provides two examples for each failure case. Case 2 is omitted as it is self-explanatory. Bold text indicates the elements that contribute to the response being classified as a failure case.

Examples for inappropriate content. Tab. B.5.3 shows examples of images and their responses from each proprietary model. All inappropriate or harmful images we found are from SSB Hard. As discussed in Sec. 3.6, All three proprietary models decline to follow a given prompt if an input image contains such content, instead they return responses with “I’m sorry, I can’t help with that” or “I apologize, but I do not feel comfortable classifying or describing this image in detail”. Claude 3.5 Sonnet rarely generates responses corresponding Case 4, but its frequency increases for the inappropriate classes in SSB Hard, such as undies or uzi. However, all open-source models generate predictions for them.

The effectiveness of guidelines and examples. We examine the effects of guidelines and examples to mitigate invalid responses using InternVL2-76B on the $5\%$ subset of the ImageNet200 benchmark. Baseline refers to a prompt that includes only the basic elements: a task description and an explanation of the rejection class. As shown in Tab. B.5.4, the inclusion of guidelines and examples results in a nearly $90\%$ reduction in failure cases compared to prompts without them. Examples help address all types of failure cases, while guidelines are especially effective in reducing Failure Case (2), where responses do not follow the required format.

We provide the full prompt formats used for experiments in Sec. 3.5 and 3.6. In the prompt format, yellow, green, red, blue boxes indicate a task description, an explanation of the reject class, guidelines, and examples for response format, respectively.

Despite the guidelines provided in Stage 1, responses that deviated from the guidelines are observed. These failure cases can be broadly categorized into four cases: 1) inclusion of ID class names, 2) failure to suggest exactly $N$ classes for each near- and far-OoD group, 3) duplicated classes, and 4) no suggestion due to failure in recognizing the image. We post-process the suggested answers to refine them into a feasible set of auxiliary OoD classes, except for Case 4. For Case 1, we exclude suggestions that are character-level identical to ID classes. For Case 2, if the total number of suggestions exceeds $2N$, we sample only $2N$ classes from the suggestions. If the total number of suggestions is less than $2N$, we use the suggestions as is without any supplementation. For Case 3, we remove one of duplicated classes.

Tab. C.2.1 presents the comprehensive results for ReGuide.

As in Sec. B.2, we evaluate the effectiveness of ReGuide on the shared valid query set. As shown in Tab. C.2.2, the overall performance trends among the LVLMs remained consistent with their performance on their own valid query sets. ReGuide enhances OoDD performance across all models, consistent with the results observed on their own valid query sets. We additionally evaluate their FPR at a TPR threshold of $90\%$, considering the high sensitivity to TPR thresholds, as discussed in Sec. 3.6. At the $90\%$ TPR threshold, FPR is significantly reduced, and ReGuide consistently demonstrates lower FPR compared to the baseline across all OoD datasets.

Tab. C.2.3 and C.2.4 present example responses of each models in Tab. 4. The first row in Tab. C.2.3 indicates the classes suggested by GPT-4o when ID class names are provided (i.e., GPT-text). The prompt used to obtain class suggestions of GPT-text can be found in Sec. C.5. A class name below each image is the ground truth of the corresponding image. The class suggestion set in GPT-text remains fixed regardless of the input images, as ID classes are static information, whereas ReGuide’s suggestions change based on the input image due to its image-adaptive nature.

The results on InternVL2-26B/-76B and GPT-4o demonstrate the effectiveness of ReGuide. Firstly, ReGuide suggests helpful classes for OoDD. For example, as shown in Tab. C.2.3, InternVL2+ReGuide suggests near-OoD classes for dotted image, such as polka dot, cherry red, or salmon-colored, which are highly close to its ground-truth. These suggested classes successfully guide the model to classify OoD inputs into one of the auxiliary OoD classes. For softball image, the prediction of ReGuide is also closest to the ground-truth.

To further investigate the FPR increase in larger models such as InternVL2-76B and GPT-4o in Tab. 4, we analyze the ROC curves for each model with and without ReGuide. Fig. C.3.1 shows the ROC curves for InternVL2-26B and GPT-4o. Both models exhibit improved ROC curves after applying ReGuide; however, the FPR declines more gradually for GPT-4o+ReGuide compared to InternVL2-26B+ReGuide as TPR decreases from $100\%$ as shown in Fig. C.3.1(b) and C.3.1(d). While the reduction in FPR for GPT-4o+ReGuide is less pronounced at high TPR thresholds (e.g., 100%–95%), it achieves significantly lower FPR as the TPR threshold approaches 90%.

To explore the causes of the FPR change observed across models in Fig. C.3.1, we analyze examples at the two extremes –$100.00\%$ and $0.0\%$– of the ROC curves, as well as the shifts in OoD score distributions induced by ReGuide. This analysis reveals that the increase in FPR@95%TPR for larger models such as InternVL2-76B and GPT-4o stems from their superior visual recognition ability, which leads to more confident predictions with highly suitable class suggestions.

Tab. C.3.1 presents examples on the extremes of Fig. C.3.1(b) and C.3.1(d). This highlights that labeling issues arise from insufficiently thorough annotations. First, a small subset of datasets exhibits a conceptual mismatch between the objects in images and their ground-truth labels. For instance, the grand piano image demonstrates a misprediction caused by such labeling issues, as the object is more accurately described as a grand piano mechanism, a class suggested by GPT-4o, rather than the ground-truth label grand piano. Another case involves images with ground-truth labels corresponding to objects that occupy only a small portion of the image, as shown in the bee and ball examples. In the bee image, the dominant object is a plant resembling a cactus, yet the ground-truth label is bee. For such images, LVLMs often suggest classes based on the dominant objects and classify them accordingly.

The suggested classes provided by GPT-4o for label-mismatched inputs are significantly fine-grained and contextually appropriate. Tab. C.3.2 presents the classes suggested by GPT-4o for images with ambiguous or inaccurate labels. For example, GPT-4o suggests classes such as model ape, gorilla figurine, orangutan statue, and monkey figurine as near-OoD classes. Based on the shape and texture of the object in the image, these suggestions are highly relevant and more suitable classifications. Similarly, for the cockshell image, GPT-4o suggests near-OoD classes such as motorboat, sailboat, rowboat, and kayak, which are fine-grained categories closely related to the depicted object. As shown in Tab. C.3.2, GPT-4o’s strong image recognition capabilities allow it to provide fine-grained, contextually appropriate classifications for images with ambiguous or inaccurate labels.

Fig. C.3.2(a) and C.3.2(b) depict the OoD score distributions for InternVL2-26B and GPT-4o, respectively. As shown in Fig. C.3.2(a), with ReGuide, InternVL2-26B produces OoD scores ranging between $0.00$ and $100.00$, which is not observed without ReGuide. However, the effect of ReGuide on GPT-4o differs from that on InternVL2-26B. The OoD score distribution for GPT-4o becomes more biased with ReGuide, as illustrated in Fig. C.3.2(b). The number of responses with OoD scores between $0.00$ and $100.00$ decrease significantly. We infer that this outcome is due to the strong visual recognition ability of GPT-4o. As shown in Tab. 2, the suggestions of GPT-4o include classes that are highly similar to the given image.

For more detailed analysis of the ReGuide effect, we examine the OoD score distributions for ID and OoD inputs separately. Fig. C.3.3 displays OoD score distributions for ID and OoD inputs on InternVL2-26B and GPT-4o.

A common observation regarding ReGuide for InternVL2-26B and GPT-4o is the shift in the OoD score distribution of OoD inputs. As shown in Fig. C.3.3(b) and C.3.3(d), ReGuide increases the number of OoD inputs with an OoD score of $0.0$ while reducing those with a score of $100$. This shift demonstrates that the models effectively classify OoD inputs into one of the suggested OoD classes, thereby improving OoD detectability and confirming that ReGuide functions as intended.

The increase in inputs with an OoD score of $0.0$ is also observed for ID inputs. As previously discussed in Tab. C.3.1, this is caused by ID images with ambiguous or inaccurate labels. For such inputs, GPT-4o+ReGuide effectively classifies them into a single class with higher confidence, as the suggested classes often include one closely aligned with the ground-truth label. This explanation is further supported by the improvement in FPR@90%TPR, where the influence of this small subset of images is reduced, as shown in Tab. 4.

When incorporating available auxiliary OoD classes, one of effective scoring strategies is to include confidence estimates for these auxiliary OoD classes. For example, NegLabel (Jiang et al., 2024) uses the scoring function as the ratio of the sum of ID softmax to the sum of both ID and OoD softmax, and EOE (Cao et al., 2024) utilizes the difference between max ID softmax and max OoD softmax.

We compare three different scoring functions: 1) the maximum ID softmax, 2) the difference between the sum of ID softmax and OoD softmax, and 3) the difference between the maximum ID softmax and maximum OoD softmax. Each of the scoring functions is represented, in order, as follows:

where $\tau$ is a temperature.

Tab. C.4.1 presents the results with these scoring functions. All results are on a $5\%$ subset of ImageNet200 benchmark using InternVL2-26B+ReGuide. No significant differences are observed across the scoring functions. LVLMs tend to assign high confidence to a single predicted class, resulting in similar results regardless of these different scoring designs. Therefore, all results reported in our paper are measured by $\mathcal{S}_{\text{max}}$.

We provide the full prompts used for experiments in Sec. 4.1. In the prompt, yellow, green, red, blue boxes indicate a task description, an explanation of rejection class, guidelines, and examples for response format, respectively.

While ReGuide, as a two-stage approach, incurs higher computational costs compared to single-turn generation (Section 3), the inference cost does not double. This is because caching image representations eliminates the need to process the same image twice, and the shorter prompt in Stage 1 (class suggestion) reduces the token count. For open-source models, we analyze the inference cost from a computational complexity perspective, whereas for proprietary models, the analysis focuses on API usage costs.

We estimate the inference time $FLOPs$ following Li et al. (2024b). This standard practical estimate is based on a simplified scaling law, assuming that the inference cost of the LLM scales linearly with both the number of tokens processed and the model size. The inference time $FLOPs$ is estimated as:

where $N$ is the number of parameters in the LLM and $T$ denotes the total number of tokens involved in the inference process. Specifically, $T$ is the sum of $Q$, $V$, and $G$, which correspond to the number of text input tokens, image tokens, and generated tokens, respectively. For example, for InternVL2-76B, the prompt depicted in Fig. B.6.1 with ImageNet200 classes includes 5,350 input tokens. The number of generated tokens is 1,936, assuming the output follows the provided example structures, and the number of image tokens is 256. Thus, the baseline $FLOPs$ is estimated as $7\times 10^{10}\times(256+5,350+1,936)$.

Similarly, for ReGuide, $FLOPs$ is estimated as:

where $Q_{Stage1},G_{Stage1}$ and $Q_{Stage2},G_{Stage2}$ denote the number of text input and generated tokens for Stage 1 and Stage 2, respectively. Although ReGuide involves two stages, the input image is processed only once as the same image is reused across both stages and can be cached. Note that the inference cost of the vision encoder is excluded, as it remains consistent for both the baseline and ReGuide within the same VLM.

Tab. C.6.1 presents $FLOPs$ for the baseline and ReGuide on InternVL2 models. These values are estimated on one input pair (image-prompt) of the ImageNet200 benchmark, assuming that the generated outputs follow one of the provided example structures.

ReGuide requires only 1.14$\times$ the computational cost compared to the baseline. It is because 1) the prompt used in Stage 1 is short and simple (e.g., $917$ tokens for Stage 1), and 2) the image tokens are processed only once. Consequently, the increase in computational cost arises solely from the processing of the Stage 1 prompt. Additional optimization techniques for multi-turn inference could further reduce the computational cost gap between the baseline and ReGuide.

Tab. C.6.2 shows API costs for proprietary models under the baseline and ReGuide experiments on the 5% ImageNet200 benchmark. From the perspective of API costs, ReGuide exhibited a higher proportional increase compared to computational costs above, with a 1.47$\times$ increase relative to the baseline. Since we processed each stage separately, the API cost reflects a scenario where image token caching is not applied. Additionally, we did not employ multi-turn inference, as it provided no advantages in inference time due to the absence of batch inference support. Implementing multi-turn batch inference could significantly improve efficiency and reduce costs.

One possible approach to reduce the inference cost gap between the baseline and ReGuide is to integrate the two-stage prompts into a single prompt while maintaining ReGuide’s process of providing suggested classes. This adjustment would allow ReGuide to maintain its effectiveness with a single forward pass. Fundamentally, it is essential to develop LVLMs to overcome their tendency to limit responses to the options explicitly provided in the user-defined prompt. Such advancements would enable LVLMs to explore and utilize information beyond the immediate constraints of the prompt, establishing robust models capable of generating safe and reliable responses to a wider range of inputs.

While the proposed ReGuide improves performance in both image classification and OoDD tasks, there remain limitations. These challenges are primarily caused by the difficulty in exerting precise control over the behavior of the VLM. Due to the inherent complexity of understanding the implicit dynamics of LVLMs, controlling their behavior through prompting does not always ensure the desired outcome. Although we provide additional guidelines based on the model’s failure cases to mitigate unintended responses, this approach is insufficient for full control over outputs. To address unintended outputs, we can utilize LLMs to iteratively refine prompts based on the model’s responses. Leveraging external LLMs for this purpose is expected to function similarly to model ensemble principles, resulting in prompts better suited to the model compared to heuristic refinements.

From a visual perspective, ReGuide relies on the image understanding capabilities of LVLMs. Any potential improvement depends on the model’s ability to recognize visuals accurately. Additionally, since our proposed approach is image-adaptive, the quality of the suggested class set is influenced by the image context. For instance, if the target object occupies only a small portion of the image, an LVLM may focus more on other objects in the background. This tendency can result in class suggestions that are either uninformative or overly broad.