MVREC: A General Few-shot Defect Classification Model Using Multi-View Region-Context

Various models, including object detection, segmentation, and classification, have been applied to defect detection. In recent years, the MVTec AD (Bergmann et al. 2019) dataset has been widely studied for anomaly detection tasks (Lyu, Mo, and Wong 2024; Pang et al. 2022). These models learn from normal samples to identify anomalies. However, defect classification, which involves identifying specific defect types, is more challenging due to the rarity and diversity of defects and the limited availability of relevant datasets. FSDMC models, such as CAO (Cao et al. 2023), Fabric (Zhan, Zhou, and Xu 2022), and FANet (Zhao et al. 2023), have been proposed. However, these methods are often dataset-specific and require complex training processes, including meta-learning and metric learning. Additionally, using a subset of defect types as base classes to train a base model and then evaluating novel classes is common, but this approach may not be practical for real-world applications.

The most common few-shot methods include meta-learning and metric learning. Meta-learning methods learn a model that can quickly adapt to new tasks with minimal training data. Metric learning methods learn a distance metric that can effectively measure the similarity between samples. Recently, large language models and multi-modal pre-training models, such as GPT (Ouyang et al. 2022; Zhu et al. 2023) and CLIP (Radford et al. 2021), have emerged as powerful tools. Related research (Gu et al. 2024; Jeong et al. 2023) has been applied to defect detection tasks, showing impressive performance. Numerous adapter-based methods have been proposed to adapt the CLIP model to specific tasks with few samples, such as CLIP-Adapter (Gao et al. 2024), Tip-Adapter (Zhang et al. 2022), CoOp (Zhou et al. 2022), and SuS-X (Udandarao, Gupta, and Albanie 2023), but most of these methods are designed to jointly learn image and text features for general vision tasks.

Traditional classification networks typically evolve by cropping the defect region and resizing it, without explicitly utilizing the region context. When defects vary in size, as shown in the example in Figure 1, cropping and resizing the defect region may result in the loss of crucial contextual information. To address this issue, region-context models incorporate region context as a prompt to predict target information, thereby preserving the contextual information of the target. For example, the SAM network uses prompts in the form of points and bounding boxes. To enable CLIP to focus on specific regions within the entire image, various methods (Zhong et al. 2022; Zhou, Loy, and Dai 2022; Shtedritski, Rupprecht, and Vedaldi 2023; Sun et al. 2024) have been explored. AlphaCLIP is an innovative enhancement of the CLIP model, designed to improve its ability to focus attention on specific regions (Sun et al. 2024). This architecture enables AlphaCLIP to provide precise control over the emphasis of image content.

We first introduce the multi-view region-context (MVREC) feature extraction process for defect instances. To effectively capture the visual representation of defects and explicitly mine contextual information, we employ the pre-trained AlphaCLIP model to extract visual features from images using their mask prompts. Given the small data volume characteristic of few-shot learning tasks, we utilize multi-view context augmentation to generate multi-view patches of defect images, thereby expanding the available dataset for subsequent processing. Specifically, we employ two context augmentation methods to achieve this. The first method, multi-scale augmentation, involves cropping $\text{Num}_{scale}$ patches at different scales from the defect patches and their corresponding masks, centered on the defect. The second method involves offsetting the center of the defect to generate $\text{Num}_{offset}$ defect patches with different offsets at each scale. By applying these two augmentation methods, a total of $V=\text{Num}_{scale}\times\text{Num}_{offset}$ patches can be obtained for each defect instance. The AlphaCLIP model extracts MVREC patch embeddings $E\in\mathbb{R}^{V\times C}$ from the multi-view patches, where $C$ is the number of feature channels. These embeddings are then averaged to produce a single MVREC feature $F\in\mathbb{R}^{C}$.

For $N$-way $K$-shot classification tasks, the MVREC patch embeddings $E_{SUPP}\in\mathbb{R}^{NK\times V\times C}$ and MVREC features $F_{SUPP}\in\mathbb{R}^{NK\times C}$ are first extracted for the support set. Then, the one-hot encoded class labels $Y_{SUPP}\in\mathbb{R}^{NK\times N}$ are extracted. The MVREC features $F_{SUPP}$ and $Y_{SUPP}$ are used to build the cached key-value pairs for the FSDMC task. Additionally, the MVREC patch embeddings $E_{SUPP}$ and the one-hot encoded labels $Y_{SUPP}$ are used as training data to fine-tune the Zip-Adapter.

In this section, we introduce the method of utilizing MVREC visual features to construct the zero-initialized projection classifier (Zip-Adapter) for FSDMC tasks. The Zip-Adapter classifier consists of a zero-initialized projection (ZIP) module and a scaled dot-product attention (SDPA) module. It stores the MVREC features $F_{SUPP}$ along with the encoded labels $Y_{SUPP}$ of the support set samples. The ZIP module includes a single linear layer, a residual connection, and a SiLU activation function, with the linear layer initialized to zeros. The output of the ZIP module is the adapted feature ${F^{\prime}}$, generated as follows:

$$F^{\prime}=\text{SiLU}\left(\text{Linear}\left(F\right)\right)+F,$$

(1)

here, $F$ represents the MVREC feature for either the support sample or the query sample. In the Zip-Adapter, the ZIP module is designed to serve as an identity transformation by initializing the linear layer with zeros and using a residual connection. The SDPA module is a scaled dot-product attention mechanism, which calculates the visual similarity between the query defect instance and the support set. It then produces the classification logits by performing a weighted sum of the support encoded labels $Y_{SUPP}$. The SDPA module is defined as follows:

$$logits_{query}=Y_{SUPP}\cdot\psi\left(Sim\left(F^{\prime}_{query},F^{\prime}_{SUPP}\right)\right),$$

(2)

where the $\psi$ is the activation function (Zhang et al. 2022) for modulating the cosine similarity:

$$\psi(x)=\exp\left(-\beta(1-x)\right),$$

(3)

$\beta$ controls the sharpness of the curve. And $Sim$ is the cosine similarity function. The output of the SDPA module, $logits_{query}$, represents the classification logits of the query defect instance. The class with the highest logit value is identified as the predicted class.

Zip-Adapter-F enhances the visual features for better performance by fine-tuning the Zip-Adapter classifier, making both the ZIP module and the cached visual features of the support set learnable. Our Zip-Adapter-F combines a cache-based mechanism with an adapter-based mechanism, using the Zip-Adapter as the base model. The fine-tuning process involves two training objectives: (1) optimizing the cross-entropy (CE) loss between the predicted logits $logits_{query}$ and the labels $Y_{query}$. The CE loss $\mathcal{L}_{CE}$ is defined as:

$$\mathcal{L}_{CE}(logits_{query},Y_{query})=-\sum_{i}y_{i}\log(p_{i}),$$

(4)

where $y_{i}$ and $p_{i}$ represents the label and predicted probability distribution for class $i$.
The second part uses the triplet loss to optimize the intra-class compactness and inter-class separability of the adapted feature $f_{\text{adapted}}$ of the ZIP module within a batch. The triplet loss $\mathcal{L}_{triplet}$ is defined as:

$$\begin{split}\mathcal{L}_{triplet}(F^{\prime}_{query})&=\max(d(F^{\prime}_{\text{anchor}},F^{\prime}_{\text{positive}})\\ &-d(F^{\prime}_{\text{anchor}},F^{\prime}_{\text{negative}})+\alpha,0)\end{split},$$

(5)

where $F^{\prime}_{\text{anchor}}$, $F^{\prime}_{\text{positive}}$, and $F^{\prime}_{\text{negative}}$ are the embeddings (feature vectors) of an anchor sample, a positive sample (same class as an anchor), and a negative sample (different class from anchor), within a batch, respectively. $d(\cdot,\cdot)$ is a distance function used to measure the similarity between embeddings. $\alpha$ is a margin hyperparameter that specifies the minimum difference between the distances of positive and negative pairs required for the loss to be zero. The overall loss function for finetuning Zip-Adapter-F is:

$$\mathcal{L}_{Zip-Adapter-F}=\mathcal{L}_{CE}+\lambda\cdot\mathcal{L}_{triplet},$$

(6)

where $\lambda$ is a hyperparameter that balances the importance of the two parts in the overall loss. After Zip-Adapter-F is trained, it can be used to classify query defect instances similar to the Zip-Adapter classifier.

We conducted experiments on the MVTec-FS dataset and four other datasets to evaluate our MVREC using accuracy metrics. The few-shot setup is defined as N-way K-shot, where K is set to 1, 3, or 5. The support set is sampled from the training set, and the query set consists of all images in the testing set. The evaluation was conducted on the query set for each of the five sampled support sets, and the average classification accuracy was calculated to provide a more robust assessment. Ablation studies were performed on the MVTec-FS dataset to assess the effectiveness of the various components. For AlphaCLIP, we selected the ViT-L/14 (Dosovitskiy et al. 2020) backbone. The MVREC visual feature used three scales, representing the commonly used large, medium, and small settings. The number of offsets was set to 9, based on a grid layout similar to a tic-tac-toe board. We set $\beta$ to 32 for the Zip-Adapter and 1 for the Zip-Adapter-F classifiers. When training the Zip-Adapter-F, we used the AdamW optimizer with a learning rate of 0.0001. The model was updated for 500 iterations, training on all MVREC features of the support set in each iteration. For the triplet loss item, the hyperparameters $\alpha$ and $\lambda$ were set to 0.5 and 4, respectively.

In our experiment, we evaluated a variety of baseline classifiers based on the AlphaCLIP backbone, including: 1. CLIP-ZeroShot (Radford et al. 2021): This approach leverages the zero-shot capability of the CLIP model. We generated text embeddings for each class description and computed the similarity between the test image embeddings and these text embeddings, classifying them based on the highest similarity. 2. CLIP-KNN: This method uses the K-Nearest Neighbors algorithm with the CLIP features. The most similar K (K=1) support samples are retrieved for a query sample, and the class with the majority vote is selected as the prediction. 3. CLIP-ProtoNet: This approach builds a Prototypical Network (Snell, Swersky, and Zemel 2017) on top of the CLIP, where proxy features representing each class are calculated from the support set, and test images are classified based on their similarity to these class proxies. 4. CLIP-Adapter (Gao et al. 2024): This method involves adding adapter layers on top of the CLIP. These layers are trained for the new classification task, adjusting the image features accordingly. 5. Tip-Adapter (Zhang et al. 2022): This method constructs a key-value cache model using CLIP-extracted features from the few-shot data and performs recognition in a retrieval-based manner. Tip-Adapter-F treats the visual cache as learnable parameters and optimizes them to improve performance.

In Table 1, we present the classification accuracy (%) of various few-shot models evaluated on the MVTec-FS dataset under different few-shot learning configurations. To ensure a fair comparison, we report results across several few-shot settings, specifically with 0, 1, 3, and 5 shots. The accuracies for 14 product categories, as well as the average accuracy, are reported in separate columns. As shown in the table, our MVREC demonstrates outstanding performance in all few-shot setups, regardless of whether Zip-Adapter or Zip-Adapter-F is used. The Zip-Adapter-F achieves the highest accuracy of 89.4% with 5 shots, which is 6.9% higher than the second-best, LinearProb. By comparing different product categories, it is evident that the Zip-Adapter-F achieves the highest accuracy in most categories, demonstrating its effectiveness in few-shot learning scenarios.

To better illustrate the function of MVREC, we used t-SNE (Van der Maaten and Hinton 2008) to visualize the support MVREC features in Zip-Adapter-F, as shown in Figure 5. Different colors represent 5 defect classes from the 5-shot leather images of the MVTec-FS. The changes in the distribution indicate that multi-view augmentation and fine-tuning help the model learn more discriminative features.

We evaluated MVREC on several public datasets, including: 1) NEU-DET (He et al. 2020), a metal surface defect dataset for detection model research; 2) PCB Defect Dataset (Huang and Wei 2019), released by The Open Lab on Human-Robot Interaction of Peking University; 3) Magnetic Tile Surface Defects (MTD) (Huang, Qiu, and Yuan 2020), which contains 6 common magnetic tile defects; and 4) AITEX Fabric Defect (Silvestre-Blanes et al. 2019), a fabric defect dataset with 12 types of defects, from which seven defect types with at least 10 samples are selected. For each dataset, 50% of the data is used as the training set for sampling the support set, and the other 50% is used as the testing set (query set). In addition to 1, 3, and 5 shots, we also evaluated performance with 10, 15, and 20 shots on the NEU-DET, PCB Defect, and MTD datasets for a more comprehensive comparison. The results, in Figure 6, demonstrate that Zip-Adapter-F (MVREC) achieves the best performance on all datasets, with performance improving as the number of shots increases.

In creating the MVTec-FS dataset, we began by selecting all 1,228 anomaly images and their corresponding masks from the testing set of the MVTec AD dataset. The original anomaly masks were annotated at the image level, which is a coarse form of labeling. For instance, as shown in Figure 7, when multiple different types of anomalies appear in the same image, they are labeled as a single ”combined” type. Additionally, when multiple defects of the same type appear in an image, the image-level mask is treated as a single instance. Given these problems, we refined the mask labels by converting them into instance-level defect masks using the connected component algorithm, assigning a class label to each defect instance. Subsequently, we manually reviewed and adjusted the defect instance labels to ensure accurate labeling of each defect instance. Figure 7 illustrates examples of these label modifications. Specifically, the label modification involved two main actions: 1) verifying and revising the defect instance masks, and 2) checking and correcting the instance class labels. For the ”combined” type, where multiple defect instances exist within a single image, we modified the instance class labels to ensure that each defect instance was labeled correctly.
The MVTec-FS dataset, as summarized in Table 7, showcases a diverse collection of sub-datasets, each representing different product categories with varying numbers of anomaly types and defect instances. Across these sub-datasets, the number of anomaly categories ranges from 3 to 7, reflecting the distinct defect characteristics of each product type. Notably, each sub-dataset within MVTec-FS is carefully constructed to ensure that there are at least five instances of each anomaly type in the training set. This design allows for effective few-shot learning experiments, supporting scenarios where 1-shot, 3-shot, and 5-shot learning paradigms can be evaluated.
Moreover, MVTec-FS is not only suitable for few-shot classification tasks but also serves as a valuable resource for few-shot object detection and unified multi-modal classification tasks. We hope that this dataset will stimulate further research in these areas, fostering advancements in both few-shot learning and multi-modal learning fields.