GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection

Prompt learning has emerged as a key technique to optimize the performance of VLMs for specific scenarios by incorporating carefully designed prompts into input images or text. Initially, prompt engineering utilized static prompt templates to guide VLMs. However, these static prompts often struggle with generalization due to their rigidity and vulnerability to diverse data distributions (Zhou et al., 2022b; a) To mitigate this limitation, methods such as CPE were proposed, combining multiple pre-defined prompts to improve robustness across varied data domains (Jeong et al., 2023). Recent advancements introduced learnable tokens to enable dynamic adaptation in prompt design. Zhou et al. (2022b) proposed the context optimization (CoOp) method, which integrates learnable tokens into text prompt, enhancing the expressiveness of prompts beyond static templates. Furthermore, Li et al. (2024) extended this concept with a semantic concatenation approach, generating multiple negative samples in prompt learning. Notably Zhou et al. (2023) introduced an object-agnostic prompt learning framework that learns generalized features for both normal and anomalous cases without relying on specific object semantics, as shown in the left part of Fig 1(a). Building upon this foundation, we propose a novel semantic-aware prompt design strategy that transforms these object-agnostic prompts into global and local semantic prompts, as shown in right part of Fig 1(a), enabling more comprehensive feature extraction while maintaining object-agnostic properties.

CLIP consists of text and vision encoders, where the vision encoder is composed of multilayer networks based on ViT (Dosovitskiy, 2020). For ZSAD, CLIP generates a text embedding $t_{c}\in\mathbb{R}^{D}$ by passing a text prompt $\mathbb{T}$, which incorporates the class $c$ from the target class set $C$, through the text encoder. $\mathbb{T}$ follows the format A photo of a [class], where [class] represents the target class name. The vision encoder takes an input image $x_{i}$ and extracts visual features, where the class token $f_{i}\in\mathbb{R}^{D}$, referred to as [cls] token, is treated as global visual embedding. Additionally, patch token $f_{i}^{m}\in\mathbb{R}^{H\times W\times D}$, extracted from detailed regions of the image, are used as local visual embedding. The probability of $x_{i}$ belonging to class $c$ is calculated based on the cosine similarity between $t_{c}$ and $f_{i}$, as shown in the following expression (Zhou et al., 2023):

where $\tau$ represents the temperature hyperparameter and $<\cdot,\cdot>$ represents the cosine similarity. In this study, we assume that object-agnostic prompts are necessary when using CLIP for ZSAD. Under this assumption, we designed two text prompts to distinguish between normal and anomalous conditions and performed anomaly detection by calculating the anomaly probability based on their similarities. Consequently, $t_{c}$ is represented by two types of text embeddings, where one is the normal text embedding $t_{n}$ and the other is the abnormal text embedding $t_{a}$. The anomaly score was denoted as $P(t_{a},f_{i})$. For local visual embeddings, the probabilities of the normal and anomalous conditions for each pixel $(j,k)$, where $j\in[1,H]$ and $k\in[1,W]$, are calculated as $P(t_{n},f_{i}^{m(j,k)})$ and $P(t_{a},f_{i}^{m(j,k)})$. These probabilities are then used to obtain the normal and anomaly localization maps, $S_{n}\in\mathbb{R}^{H\times W}$ and $S_{a}\in\mathbb{R}^{H\times W}$, respectively.

We proposes the GlocalCLIP, which explicitly separates global and local prompts to learn general normal and anomalous features in a complementary manner. The overall structure is shown in Fig. 2 and comprises four steps: (1) Text Encoder: Prompts from the object-agnostic glocal semantic prompt are passed through the text encoder to generate both global and local text embeddings. (2) Vision Encoder: The input image is processed by the vision encoder, which returns the global and local visual embeddings through the [CLS] and patch token, respectively. (3) Anomaly Scoring: Anomaly scores are calculated by measuring the similarity between global visual embedding and global text embedding, as well as between local visual embedding and local text embedding. (4) Glocal Contrastive Learning: Complementary learning is achieved through contrastive learning between global and local text embeddings. During inference, anomaly detection and localization are achieved by computing an anomaly score and generating a localization map.

In GlocalCLIP, we propose an object-agnostic glocal semantic prompt design to capture subtle contextual differences between normal and anomalous situations. These prompts are generated by concatenating a suffix indicating an anomaly with a base prompt that represents normal conditions. For example, appending the suffix with a crack in the corner to the normal prompt A crystal clear window transforms it into an anomaly prompt, A crystal clear window with a crack in the corner. Thus, GlocalCLIP can learn both the fine details of an object and its defects through semantic changes in prompts. The text prompts are expressed as follows:

Normal prompt $p_{n}$ and anomaly prompt $p_{a}$ are designed in binary form, following an object-agnostic prompt structure. Here, $[N_{i}]\ (i\in[1,E])$ denotes a learnable token for normal conditions, representing the general state of each object. While, $[A_{j}]\ (j\in[1,L])$ denotes a learnable token for abnormal conditions, indicating defects or damage specific to each object. Instead of focusing on learning class semantic within an image, these prompts are designed to capture both global and local features that distinguish normal from anomalous conditions. By utilizing the term object in a generalized context, the design enables the prompts to learn object-agnostic representations, facilitating a more generalized approach to anomaly detection without reliance on class-specific semantics. In this context, the term damaged is manually incorporated into the anomaly prompt to explicitly represent anomalous conditions. By explicitly separating these prompts into global and local contexts, they can be defined as follows:

Here, $g_{n}$ and $g_{a}$ represent the global prompts for normal and abnormal conditions, respectively, while $l_{n}$ and $l_{a}$ correspond to the local prompts for the same conditions. The learnable tokens used in each prompt, $[N_{i}^{g}]$, $[N_{i}^{l}]$ and $[A_{j}^{g}]$, $[A_{j}^{l}]\ (i\in[1,E],j\in[1,L])$, correspond to normal and abnormal states. This separation is designed to learn features from different perspectives in anomaly detection. The global prompts capture the overall context of the image to determine normality or abnormality, while the local prompts focus on fine-grained characteristics of localized defects, such as scratches or contamination. This explicit design enables accurate anomaly detection across various domains by effectively capturing both global and local details. The effect of the learnable prompts when varying their positions is detailed in Appendix C.

To learn between global and local prompts in a complementary manner, we propose a glocal contrastive learning mechanism that aligns text embeddings across different semantic levels. The glocal contrastive learning (GCL) named $L_{gcl}$ operates on triplets of prompts, where global prompt serves as an anchor to regulate its relationship with local prompts. This design choice is motivated by the hierarchical nature of visual understanding: global prompts capture the overall image context, while local prompts refine this understanding with fine-grained details. The loss is formulated as:

where ${\bm{a}}$, ${\bm{p}}$, and ${\bm{n}}$ represent the embeddings of anchor, positive, and negative prompts respectively, and margin determines the minimum required distance between anchor and negative prompts. In GCL, the global normal prompt serves as the anchor, encouraging the local normal prompt to move closer as a positive example, while the local anomaly prompt is pushed farther away as a negative example. Similarly, when the global anomaly prompt is used as the anchor, the local anomaly prompt is brough closer, and the local normal prompt is pushed farther away. This dual alignment ensures that prompts are learned relative to the semantic context of global normality and abnormality. Consequently, the loss function minimizes the distance between semantically similar prompts and maximizes it for dissimilar ones, enabling the model to learn discriminative features where the global context aids the refinement of local details. The total glocal contrastive loss is defined as:

where $L_{gcl}^{normal}$ focuses on aligning prompts under normal conditions, and $L_{gcl}^{anomaly}$ operates on anomaly conditions.

We utilized deep-text prompt tuning by inserting learnable tokens into each layer of the text encoder, refining text embeddings and enhancing their interaction with visual embeddings. Specifically, at the $i$-th layer, the learnable token $t_{i}^{learnable}$ is concatenated with the text embedding $t_{i}$ to adjust the text embedding. This process is represented as follows:

The updated text embeddings in each layer are passed to the next layer, allowing more detailed text information can be learned using a new $t_{i}^{learnable}$ token in each layer. Then the text embeddings are aligned with visual features, enabling a more accurate detection of normal and anomaly patterns.

In vision encoder, ViT is used to obtain global and local visual embeddings to effectively learn global and local visual features. The original ViT captures a global feature using a [CLS] token, while local visual embedding is derived from patch token. In GlocalCLIP, a V-V attention is applied instead of the conventional QKV attention layer to detect fine defects by focusing on local regions. V-V attention replaces both the query and key with the same value, intensifying the correlation among local features. As a result, this modification focus on the fine details of the image, facilitating the detection of subtle anomaly patterns. V-V attention is calculated using

where $V$ represents the patch token embedding of the vision encoder, and $D$ denotes the dimension of the visual embedding. The depth at which the V-V attention layer starts to be applied can be adjusted as a hyperparameter to control the focus on local regions.

The training loss is composed of three complementary components: global loss $L_{global}$, local loss $L_{local}$ and glocal contrastive loss $L_{gcl}^{total}$ defined in Section 3.3. The global and local loss components in our training objective are inspired by the design principles of Zhou et al. (2023), which effectively balances anomaly detection and localization tasks. The total training loss is defined as:

where $\mathcal{M}$ denotes a set of intermediate layers used for extracting local features, and $\lambda$ is a hyperparameter controlling interaction between global and local prompt. $L_{global}$ is computed using the binary cross-entropy. Next, $L_{local}$, which is based on the predicted and actual anomaly regions in each measurement, is given by

$S_{n,M_{k}}^{(j,k)}$ and $S_{a,M_{k}}^{(j,k)}$ denote the similarities corresponding to the normal and anomalous cases, respectively. Furthermore, let $S\in\mathbb{R}^{H\times W}$ be the ground-truth localization mask, where $S_{j,k}=1$ denotes that a pixel is anomalous, and $S_{j,k}=0$ otherwise. The local loss, $L_{local}$, is calculated using

where $Focal(\cdot,\cdot)$ and $Dice(\cdot,\cdot)$ represent the loss functions proposed by Ross & Dollár (2017) and Li et al. (2019), respectively. Focal loss assigns higher weights to important samples in imbalanced data, while dice loss is used to reduce the difference between the predicted and actual anomaly regions. $Up(\cdot,\cdot)$ and $[\cdot,\cdot]$ represent upsampling and channel-wise concatenation, respectively, and $I$ denotes a matrix with all elements equal to 1. During inference, anomaly detection and localization are performed based on the anomaly score and localization map, as described in Eq. 1. The anomaly localization map, denoted as $Map\in\mathbb{R}^{H\times W}$, is computed as $Map=G_{\sigma}(\sum_{M_{k}\in\mathcal{M}}(\frac{1}{2}(I-Up(S_{n,M_{k}}))+\frac{1}{2}Up(S_{a,M_{k}})))$, where $G_{\sigma}$ represents a Gaussian filter and the parameter $\sigma$ controls the smoothing effect.