VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation

Our approach follows the generalized ZSAS methods adopted in works [5, 33], which requires segmenting the anomalies in unseen products $C^{u}$ after training on seen products $C^{s}$ with pixel-annotations. During the training stage, the model generates pixel-wise classification results based on two categories of textual descriptions: normal and abnormal. During the testing stage, the model is expected to directly segment anomalies in unseen products. It is worth noting that $C^{u}\cap C^{s}=\emptyset$ and the products used in the training and testing stages come from different datasets. This undoubtedly poses a significant challenge to the model’s domain generalization capability.

Existing CLIP-based improvement methods have three main drawbacks: 1) manually designing text prompts is time-consuming and labor-intensive, 2) product-specific text prompts cannot adapt to data privacy scenarios, and 3) the localization results are easily influenced by the semantics of product categories in the text prompts [33]. To address the aforementioned issues, we propose a baseline that incorporates two main designs: unified text prompting (UTP) and deep text prompting (DPT). As shown in Fig. 3, given an input image $X\in\mathbb{R}^{h\times w\times 3}$ and two-class text prompts, the designed baseline (marked in red dashed) first extracts patch-level image features and text features separately. Then, the patch-level image features are mapped to a joint space, where the similarity between image features and text features is computed to generate anomaly maps. Finally, anomaly maps from multiple intermediate layers of the image encoder are fused after upsampling to obtain the final results.

Unified text prompting (UTP). A unified template for generating normal and abnormal text prompts is designed as follows:

where $v_{i},i\in\{1,2,\cdots r\}$ is a $C$-dimensional learnable vector embedded into the word embedding space, used to learn the unified textual context of the product categories. A pair of opposing [state] words, such as "good/damaged" and "perfect/flawed", is utilized to generate normal and abnormal text prompts, respectively. $H$ represents the word embedding matrix corresponding to specific prompts in the textual space. In this paper, we choose a common state word pair, i.e. "good/damaged".

Deep text prompting (DTP). Before statement, let us first review the inference process of the CLIP text encoder briefly. Before being fed into the text encoder, [SOS] and [EOS] are respectively added to the front and back of the text prompt, indicating the beginning and end of the sentence. Afterwards, these tokens are mapped to a discrete word embedding space, capped to a fixed length of 77 in CLIP. Let us denote the word embeddings as $[s,H,e,J]\in\mathbb{R}^{77\times C}$, where $s$ and $e$ are $C$-dimensional word embeddings corresponding to [SOS] and [EOS] tokens, respectively. $J$ is a placeholder matrix initialized to zero to ensure a fixed length of the word embeddings. The final output text embedding at the position of the [EOS] token is aligned with the image features after passing through a linear projection layer.

To better align fine-grained normal and anomalous visual semantics with text, deep text prompting is designed to further refine the textual space as shown in Fig. 3. In specific, continuous trainable embeddings are inserted at the beginning of text embedding in each transformer layer of the text encoder. Assuming the text encoder’s (i+1)-th layer is represented as $Layer_{i+1}^{text}$, the inserted embeddings are $P_{i}\in\mathbb{R}^{n\times C}$ and the output text embedding is $g$. The process is formulated as follows: \linenomathAMS

where $i=1,2,\cdots N_{t}$, $s_{0}=s$, $H_{0}=H$, $e_{0}=e$. $N_{t}$ is the number of text encoder layers. $TextProj(\cdot)$ and $Norm(\cdot)$ respectively denote final text projection and LayerNorm [1] layers. For normal and abnormal text prompts, we denote the embeddings after DTP as $g_{n}$ and $g_{a}$, respectively. Since the masked self-attention is employed in the text encoder, $[s_{i},P_{i},H_{i},e_{i},J_{i}]$ and $[s_{i},H_{i},P_{i},e_{i},J_{i}]$ are not mathematically equivalent. We adopted the former because the model can only attend to tokens before itself, thus placing the learnable embeddings at the beginning of the sentence leads to a greater degree of refinement in the textual space. More details are shown in the Appendix B.2.

How to acquire the anomaly map? For an input image $X\in\mathbb{R}^{h\times w\times 3}$, patch-level visual feature map $Z_{s}^{l}\in\mathbb{R}^{H\times W\times d_{I}},l=1,2,\cdots,B$ are extracted from the image encoder layers, where $H=h/patchsize,W=w/patchsize$, $d_{I}$ is the size of image embeddings and $B$ is the number of extracted intermediate patch-level feature layers. Then, the feature maps are mapped to a joint space and align with text embeddings using a single linear layer by calculating the cosine similarity. Let us respectively denote the visual and textual features in the joint space as $F_{s}^{l}\in\mathbb{R}^{HW\times C}$ and $F_{t}=[g_{n},g_{a}]\in\mathbb{R}^{2\times C}$, where $C$ is the embedding size in the joint space. The process of acquiring the anomaly map can be formulated as:

where $\tau_{1}$ denotes the temperature coefficient, which is set as a learnable parameter. $Up(\cdot)$ is an upsampling operation with bilinear interpolation. $\widetilde{(\cdot)}$ represents the $L_{2}$-normalized version along the embedding dimension.

The baseline has made some progress, but still faces the following three main problems: 1) The unified text prompt does not consider specific visual contexts. 2) Overfitting phenomena may occur in the unified text prompt. 3) Insufficient interaction between information from different modalities limits further improvement in segmentation performance. In this end, we further designed two novel visual context prompting modules, namely Pre-VCP and Post-VCP as shown in Fig. 3. In contrast to the baseline, the global features of the image are encoded into the text prompt using the Pre-VCP module. The Post-VCP module receives patch-level features from the image encoder and text features from the text encoder as inputs to generate the anomaly map.

Pre-VCP module. We designed a Pre-VCP module to introduce global image features into the text prompts of the baseline. Due to the extensive alignment of image-text pairs during the pretraining process of CLIP, the embedding at the [CLS] token position of the image encoder encompasses rich global image features. We combine the global image features with learnable vectors in the baseline to facilitate the fusion with the unified category contexts. Specifically, the global image features are initially mapped to the word embedding space through a small neural network, namely Mini-Net. This can be expressed as $\{x_{i}\}_{i=1}^{r}=h(x)$, where $x_{i}\in\mathbb{R}^{1\times C},i=1,2,\cdots r$ represents the mapping results, which are combined with embeddings corresponding to the product category:

where $z_{i}=x_{i}+v_{i}$. For the Mini-Net $h(\cdot)$, a parameter-efficient design utilizing only a one-dimensional convolutional layer with ($r$, $1\times 3$) kernels is employed. The final text prompt based on Pre-VCP can be expressed as follows:

For convenience in the subsequent text, we refer to the text prompt template as "a photo of a [state] [$z(x,v)$]".

Post-VCP module. To further enable the text embedding to adapt based on fine-grained image features, we devised a Post-VCP module, as illustrated in Fig. 3. The text embedding $F_{t}\in\mathbb{R}^{2\times C}$ and flattened visual embedding $Z_{s}^{l}\in\mathbb{R}^{HW\times d_{I}}$ from each layer are projected into a latent space with $C$-dimension. Then the learnable queries $Q_{t}$, keys $K_{s}^{l}$, and values $V_{s}^{l}$ can be obtained:

where $W_{t}^{q}\in\mathbb{R}^{C\times C},W_{s}^{k}\in\mathbb{R}^{d_{I}\times C},W_{s}^{v}\in\mathbb{R}^{d_{I}\times C}$ are linear projection matrices in the PreProj layer. To capture richer visual features for fine-tuning text, a multi-head structure is adopted for computing attention maps to update text features within each head using matrix multiplication: \linenomathAMS

where $m=1,2,\cdots,M$. $M$ is the number of heads, $Q_{t}^{(m)}\in\mathbb{R}^{2\times(C/M)},K_{s}^{l(m)}\in\mathbb{R}^{HW\times(C/M)},V_{s}^{l(m)}\in\mathbb{R}^{HW\times(C/M)}$ represent the features within each head after the $Split(\cdot)$ operation for partitioning along the embedding dimension. $A_{t}^{l(m)}\in\mathbb{R}^{2\times HW}$ and $O_{t}^{l(m)}\in\mathbb{R}^{2\times(C/M)}$ respectively refer to the attention maps and the text features updated through the image feature within each head. After concatenating all features along the embedding dimension using the $Concat(\cdot)$ operation, a PostProj layer with weight matrix $W_{t}^{o}\in\mathbb{R}^{C\times d_{I}}$ is employed to obtain the final updated text embedding $O_{t}^{l}\in\mathbb{R}^{2\times d_{I}}$ from $F_{t}$. Then, the updated anomaly map is calculated as:

where $\tau_{2}$ is a temperature coefficient set as a learnable parameter.

To visually validate the effectiveness of the Post-VCP module, we show the attention maps $A_{t}^{l(m)}$ under different heads corresponding to normal and abnormal text embeddings. These maps reveal that abnormal text embeddings concentrate more on defective regions of the image compared to normal text embeddings. This clear differentiation stems from employing fine-grained visual contexts in the Post-VCP module to update text embeddings from $F_{t}$ to $O_{t}^{l}$.

Loss function. In this work, we employed focal loss [15] and dice loss [18] to supervise the learning of VCP-CLIP. The total loss function of VCP-CLIP is calculated as:

where the loss function consists of two components, one for the baseline and the other for additional VCP module. $M_{1}^{l}$ and $M_{2}^{l},l=1,2,\cdots B$ are anomaly maps generated from the two branches mentioned above. $S\in\mathbb{R}^{h\times w}$ is the ground truth corresponding to the input image.

Inference. The ultimate anomaly maps come from different layers of the image encoder by summation. The anomaly maps generated from the two branches are represented as $M_{1}$ and $M_{2}$. To further enhance the ZSAS capability, we introduced a weighted fusion policy to generate the final anomaly map, $M_{a}=(1-\alpha)M_{1}+\alpha M_{2}$ , where $\alpha\in[0,1]$ is a fusion weight designed as a hyperparameter to balance the importance of different anomaly maps.

Datasets and metrics. To assess the performance of the model, ten real industrial anomaly segmentation datasets are used, including MVTec-AD [2], VisA [35], BSD [25], GC [17], KSDD2 [3], MSD [31], Road [26], RSDD [29], BTech [19], DAGM [20]. Since the products in VisA do not overlap with those in other datasets, we use VisA as the training dataset for evaluation on other datasets. For VisA itself, we assess it after training on MVTec-AD. Please refer to the Appendix C for more details. To ensure a fair comparison, pixel-level AUROC (Area Under the Receiver Operating Characteristic), PRO (Per-Region Overlap), and AP (Average Precision) are employed as the evaluation metrics, following the recent works [5, 6].

Implementation details. In the experiments, we adopt the CLIP model with ViT-L-14-336 pretrained by OpenAI [22] by default. Specifically, we set the number of layers $B$ for extracting patch-level features to 4. Since the image encoder comprises 24 transformer layers, we evenly extract image features from layers {6, 12, 18, 24}. All images are resized to a resolution of $518\times 518$, and then fed into the image encoder. The length of the learnable category vectors $r$ and the length of the learnable text embeddings $n$ in each text encoder layer are set to 2 and 1, respectively, by default. The number of attention heads $M$ in the Post-VCP module is set to 8. The fusion weight $\alpha$ for different anomaly maps is set to 0.75 as the default value. The Adam optimizer [12] with an initial learning rate of 4e-5 is used, and the model is trained for continuous 10 epochs with a batch size of 32. All experiments are conducted on a single NVIDIA GeForce RTX 3090. We conducted three runs using different random seeds and then averaged the results. More details can be found in Appendix A.

Two kinds of state-of-the-art approaches are used to compare with ours: training-free approaches and training-required approaches. The training-free approaches include WinCLIP [10] and AnVoL [8], which do not require auxiliary datasets for fine-tuning the model but necessitate more complex manual prompt designs and inference processes. The training-required approaches comprise CoCoOp [32], AnomalyGPT [9] and APRIL-GAN [5], which adhere to the protocol of training on the seen products and testing on the unseen products.

Quantitative comparison. Table 1 shows the quantitative performance comparison with other state-of-the-art methods on ZSAS. The best results are shown in bold, and the second best results are underlined. It can be observed that the proposed VCP-CLIP outperforms all other methods across all metrics, particularly in terms of AP. Due to the tiny anomaly regions on the Visa dataset, its anomaly segmentation is more challenging. However, VCP-CLIP still maintains its advantage compared to other methods. Notably, it achieves state-of-the-art results on VisA dataset, with AUROC score of 95.7%, PRO score of 90.7% and AP score of 30.1%. It is noteworthy that our baseline approach has already achieved nearly superior performance compared to existing methods such as CoCoOp, which similarly introduces global image information in the text prompts. This is because our method simultaneously adjusts text embeddings using fine-grained image features.

Qualitative comparison. For a more intuitive understanding of the results, we visualized the anomaly segmentation results of our VCP-CLIP alongside another five methods: WinCLIP [10], AnVoL [8], CoCoOp [32], AnomalyGPT [9], and APRIL-GAN [5] on the MVTec-AD and VisA datasets in Fig. 5. The visualization results clearly indicate that the compared approaches have a tendency to generate incomplete or false-positive results, which can negatively impact the performance of anomaly localization. In contrast, our VCP-CLIP effectively mitigates these issues, providing a more accurate and reliable approach to ZSAS. More quantitative and qualitative comparisons are provided in the Appendix D.

Same prompts during training and testing. To better validate the effectiveness of VCP-CLIP, we compared it with the proposed baseline on MVTec-AD and VisA. Fig. 7 illustrates the AP improvement of VCP-CLIP over the baseline for each product. In specific, VCP-CLIP demonstrates varying degrees of improvement among 13 out of the 15 products and 10 out of the 12 products on the MVTec-AD and VisA datasets, respectively. This affirms the robust generalization capability of VCP-CLIP, which is attributed to both the global visual context in Pre-VCP and the fine-grained local visual context in Post-VCP.

Different prompts during training and testing. To validate the robustness of VCP-CLIP during the test process with different text prompts, we employed text prompts different from those used during training on the MVTec-AD and VisA datasets. Specifically, during training, the default state words "good/damaged" were used. During testing, we reported the metric AP when the state words were respectively "normal/abnormal", "perfect/flawed", and "pristine/broken". As shown in Fig. 7, our baseline performance sharply declined on two datasets, while the performance of VCP-CLIP remained relatively stable. This indicates that after incorporating VCP, the model can adaptively adjust the output text embeddings based on input images, thereby avoiding dependence on the specific text prompts used during training.

Influence of different components. To assess the impact of different components on VCP, experiments were conducted on MVTec-AD. Results in Table 3 indicate performance when using DTP, Pre-VCP or Post-VCP individually. Notably, the optimal performance for VCP is achieved when all combined. It can been seen that the performance decline is more pronounced after removing Post-VCP compared to Pre-VCP. We also attempted to remove the learnable text embeddings from each layer of the text encoder (without DTP), which resulted in a decrease of 0.3% in AUROC, 0.6% in PRO, and 1.2% in AP. This is because the original text space cannot directly comprehend the global features of images, while DTP ensures deep fine-tuning of each text encoder layer, thereby fostering mutual understanding and fusion of different modalities.

Influence of ensemble of different patch-level image layers. In Table 3, we explore the impact of patch-level features from different image encoder layers on VCP-CLIP’s performance. The experiments were conducted on the MVTec-AD dataset. An intuitive observation is that image features from intermediate layers (i.e. the 12th and 18th layers), contribute more to the final segmentation result. Image features from lower layers (i.e., the 6th layer) are too low-level, while those from higher layers (i.e., the 24th layer) are overly abstract. Their effectiveness is not as pronounced as those from intermediate layers. However, We observed a positive correlation between incorporated layer numbers and improved segmentation results. To maintain high performance, we adopted all patch-level features from {6, 12, 18, 24} layers in VCP-CLIP.

Ablation on text prompt design. As demonstrated in Table 4, we considered two commonly used text prompt templates and explored the impact of different prompting state words in the proposed VCP-CLIP on MVTec-AD. Specifically, we designed the following two text prompt templates: 1) this is a [state] photo of [$z(x,v)$]; 2) a photo of a [state] [$z(x,v)$]. The state words (e.g. "perfect/flawed") are respectively inserted into the template to generate normal and abnormal text prompts. It can be observed that for the same template with different state words, our VCP-CLIP model consistently maintains similar performance, validating the robustness towards the state words. Furthermore, the second type of template, default employed in VCP-CLIP, outperforms the first type overall, which may be attributed to the repeated usage of similar template during the pre-training process of the vanilla CLIP.

Ablation on different pretrained models and resolutions. In Table 6 and Table 6, we conducted a comprehensive analysis of the impact of varying input image resolution and pre-trained backbone on MVTec-AD. The former is tested using ViT-L-14-336, while the latter reports the optimal performance under different backbones pre-trained by OpenAI. The inference time was simultaneously tested for a single image (average of 200 images). We observe that a moderate increase in input image resolution contributes to more precise segmentation (higher AP). However, deviations from the original pre-training resolution ($336^{2}$ to $798^{2}$), leading to model degradation. This outcome can be attributed to the model deviating from the original image space. The result in Table 6 shows that our VCP-CLIP achieves the optimal segmentation performance in ViT-L-14-336. Therefore, we have chosen it as the default backbone.