Excision And Recovery: Visual Defect Obfuscation Based
Self-Supervised Anomaly Detection Strategy

There have been efforts to enhance the UAD performance based on widely known NNs, such as auto-encoder (AE) or U-Net [32], without changing much of their structure. Among AE variants are, MS-CAM [23] presents a multi-scale channel attention module with an adversarial learning strategy. GANomaly [2] adopts feature distance loss to perform better normal pattern reconstruction. SCADN [42] performs multi-scale striped masking before feeding input to their NN. In cases of U-Net [32] variants, DAAD [16] includes block-wise memory module, and RIAD [45] proposes the reconstruction-by-inpainting strategy with multiple square patched disjoint masks. These approaches maximize the UAD performance while keeping the scale of the NN relatively small.

We also employ an U-Net [32] structure and at the same time, pursue a practically deployable solution that allows NNs to operate properly in industrial environments as a way for edge computing.

UAD based on reconstruction-by-inpainting is an effective self-supervision technique for representation learning to prevent an UAD model from accurately reconstructing unseen anomalous patterns [45, 7, 18, 21, 17, 28, 31, 3]. Specifically, methods such as random masking [45, 7, 18, 21], multiple disjoint masking [45, 17], and progressive inpainting from the initial masks [24, 28, 17, 31, 3] have been developed.

The common limitation of multiple masking and progressive inpainting is inference latency due to the multiple inferences. In addition, the random masking strategy causes the problem of output inconsistency when applied to the reconstruction-by-inpainting approach. Thus, to develop a practically deployable solution for ensuring real-time defect detection and output reliability, we should consider the following: 1) deterministic mask generation strategy, 2) minimizing the number of masks, and 3) immediate inpainting strategy rather than a progressive inpainting strategy.

To meet the above requirements, we exploit a pre-trained attention model for deterministic single masking. The deterministic single masking strategy allows real-time processing and, at the same time, secures the output reliability.

Researches report that attention-based saliency masking [19, 25] or non-saliency masking [5, 34] is more effective and helpful for representation learning. Their intention is to eliminate unnecessary input information for their objective, representation learning or object recognition.

However, since those masking methods will empty all the information in the suspected anomalous regions, accurate reconstruction of normal patterns becomes hard, especially when the masked region is large. To ease this situation, an additional strategy that randomly leaves a few patches within masked saliency areas as hint information for reconstruction can be considered [19, 25]. This strategy serves to provide initial information for inpainting the masked regions and accurate reconstruction. However, the randomness of their patch hint-providing causes the output inconsistency problem.

We present a visual obfuscation-based hint-providing scheme to promote the accurate reconstruction of normal patterns.

Due to the class imbalance problem stemming from the scarcity of abnormal situations, we adopt a self-supervised learning strategy to conduct target representation learning of normal samples. An overall schematic diagram of EAR is shown in Figure 1. The excision stage is composed of two steps. First, we generate a deterministic single saliency mask, $S$, from attention map, $A$, by exploiting the ImageNet [9] pre-trained DINO-ViT [6]. The resulting saliency mask, $S$, indicates suspected anomalous regions. Then, we provide a mosaic hint of the masked regions for reconstruction. To provide a proper hint by obfuscation, mosaic scale, $m$, is estimated from the part of the given product image that correspond to the saliency region. The result of recovery, $\hat{I}$, will be obtained by feeding $I^{\prime}$ into the U-Net [32]. The magnitude of the reconstruction error, especially the maximum value of multi-scale gradient magnitude similarity (MSGMS) [45], between $I$ and $\hat{I}$ is used to determine whether the product is defective or not.

We aim to develop a real-time and reliable solution by avoiding inference latency and output inconsistency. For this, we propose a deterministic saliency masking strategy by exploiting a pre-trained self-attention model. Specifically, the ImageNet [9] pre-trained DINO-ViT [6] is used in this study which is trained with a self-distillation strategy. First, we feed an input image $I$ into the DINO-ViT [6] and get an attention map, $A$, by averaging [CLS] tokens, multi-heads of the last layer. Then, we binarize the attention map, $A$, by thresholding the upper quartile value $\mu+0.674\sigma$ ($Q3$) [37] of pixel-wise attention scores to generate a binary saliency mask, $S$. Referring to the probable error [10], we regard upper $Q3$ values as suspected anomalous regions. This allows the mask size to be sufficiently large enough to cover suspected anomalous regions while keeping it reasonably small.

$S$ is used to cut out the suspected anomalous regions in normal samples in training, and the UAD model will be optimized to inpaint the empty region. After training, the recovered masked region by the UAD model will be accurately matched when the $I$ does not include any unseen anomalous patterns. However, if the masked region covers anomalous patterns, the UAD model will struggle to recover its original defective form. Therefore, defective products can be effectively detected due to relatively large reconstruction errors.

Saliency masking empties the defective information in the suspected defective regions to help transform masked unseen anomalous regions into normal forms. However, not leaving any clues in the masked region could cause inaccurate reconstruction of normal patterns, degrading the UAD performance.

We propose a hint-providing strategy with visual obfuscation on masked saliency regions for accurate reconstruction of normal patterns. For visual obfuscation, we adopt mosaicing of proper scale depending on the defect scale. For mosaicing, we create each single representative value within each square patch of $m\times m$ pixels by average pooling. Thus, the mosaic scale is represented by $m$. Determining a proper mosaic scale is described in Section 3.4. The average pooled image is upscaled into the original scale with the nearest interpolation and combined with a saliency mask to provide the masked regions with the proper mosaic hint as shown in Figure 1. When the mosaic method described above is denoted by $M$, the hint-providing method is expressed as (1). The processed image $I^{\prime}$ will be fed into the UAD model for reconstruction.

This mosaicing with the proper mosaic scale makes anomalous regions visually obfuscated to an extent that helps efficient reconstruction of normal patterns, and contains accurate reconstruction of anomalous patterns.

Depending on the mosaic scale, there is a difference in the information details of provided hints for masked regions. Since the mosaic scale to use is a factor that determines the reconstruction quality, it directly affects the UAD performance. As the optimal scale of the mosaic for each product is not known in advance, estimating the proper mosaic scale is necessary to give the best possible hint.

To construct a mosaic scale estimation model, we first obtain the optimal mosaic scale $m^{*}$ for each product in the MVTec AD dataset [4] through grid search. For setting a representative value for each product, we exploit the degree of edge response, $r$, in (2), which was used in SIFT [27].

For summarizing the pixel-wise edge response, we only take the saliency region in $I$ and average the top 10% of them. Let us denote this by $r_{10}$. We could successfully relate $r_{10}$ to $m^{*}$ and their linear relation is shown in Figure 2. They show a strong correlation. Products with detailed features or rough surfaces give high values of $r_{10}$ while products with relatively smooth surfaces show low values.

We optimize the linear function of the mosaic scale estimation model for each object and texture subset. An estimated mosaic scale, $\hat{m}$, is determined by quantizing $f(r_{10})$ to the nearest power of 2. For experiments, $\hat{m}$ will be used for EAR training, and the results using $m^{*}$ will also be presented to verify the effectiveness of the proposed mosaic scale estimation method.

A prior study [45] has shown a satisfactory result to detect various-sized defects by employing MSGMS as in (3). Their training objective also includes $\mathcal{L}_{2}$ (pixel-wise distance) and structural similarity index measure (SSIM) [38] which are widely used for training of a reconstruction model. We also inherit the above for training and anomaly scoring. In MSGMS, we set multiple scales $N$ to 3.

Three loss terms $\mathcal{L}_{2}$, $\mathcal{L}_{ssim}$, and $\mathcal{L}_{msgms}$ are combined by applying weights $\lambda$ as (4). Then, we apply a loss transformation method LAMP [29] on (4). LAMP [29] is known to enhance the UAD performance by only loss amplification of the training process. In addition, it can be applied to any UAD training process because it does not depend on NN structures or preprocessing methods. The final loss function for training EAR is (5).

To evaluate the performance of EAR, we use the public industrial visual inspection dataset, MVTec AD [4], for the experiments. MVTec AD [4] provides a total of 15 subtasks with 10 objects and 5 textures. Each training set for these tasks only provides normal, anomaly-free samples. The test set includes both normal and defective samples.

Implementation details. We simply inherit a well-known U-Net [32]-like structure as a reconstruction model for experiments. Specifically, we construct an U-Net as in RIAD [45]. The reconstruction model is structured with five convolutional blocks for the encoder and decoder respectively, and the $i$-th layer in the encoder is concatenated with $(5$$-i)$-th layer in the decoder. We repeat ‘convolution $\rightarrow$ batch normalization $\rightarrow$ leaky ReLU activation’ three times for the encoder, and ‘upsampling $\rightarrow$ convolution $\rightarrow$ batch normalization $\rightarrow$ leaky ReLU activation’ three times for the decoder. Note that, the stride is set to 2 in the third layer of each encoder block for spatial downscaling. Also, upsampling with scaling factor 2 and the nearest interpolation is applied in the first layer of each decoder block.

To activate EAR, a pre-trained attention model is required. Many variants of pre-trained ViT are publicly available. We adopt one of the state-of-the-art models, specifically ViT-S/8. It is provided in the official GitHub repository¹¹1https://github.com/facebookresearch/dino published by Caron et al. [6]. This ViT is known to have the ability to emphasize class-specific spatial features that lead to unsupervised object segmentation. We leverage this property to emphasize saliency regions within a given image and cut them out for inpainting.

Mosaic scale estimation. Our work includes an optimal mosaic scale estimation process. To construct a mosaic scale estimation model, we need the ground truth of optimal mosaic scale $m^{*}$ for each product. We have initially found them in the grid search manner. The results of finding the initial ground truth are given in Figure 3, and the linear regression model for mosaic scale estimation is shown in Figure 2. A summary of mosaic scale estimation for each product is also given in Table 1. As can be seen in Table 1, $m^{*}$ and $\hat{m}$ match in most objects. However, there are mismatches for the textures. Accordingly, we could observe that the AD performance for them using $\hat{m}$ is in general clearly lower than using $m^{*}$. The experiments will be conducted with $\hat{m}$ to train an UAD model for each subtask. The UAD results from $m^{*}$ will also be shown for comparison.

Training conditions. We perform hyperparameter tuning in all UAD experiments for fair comparison of each model in the best performance condition. The tuned hyperparameters are: 1) kernel size 2) learning rate 3) scheduling method of learning rate. As learning rate scheduling methods, we used fixed learning rates, learning rate warm-up [13], and SGDR [26]. The values used as hyperparameters are summarized in Table 2.

Evaluation metric. To evaluate the performance of UAD experiments, we use the area under the receiver operating characteristic curve (AUROC) [11]. The AUROC is measured based on the anomaly scores for each normal and defective sample within the test set. For anomaly scoring, we adopt the maximum value of MSGMS between the input $I$ and reconstruction-by-inpainting result $\hat{I}$ of the UAD model which is capable of detecting various sizes of defects [45]. When the MSGMSs of the UAD model for the unseen anomalous patterns are relatively larger compared to the normal pattern cases, AUROC will be close to 1.

We report visual comparisons of reconstruction results to see whether the reconstruction of normal patterns is accurate when the proposed method EAR is applied. First, as can be seen in Figure 4, we check the effect of disabling either of the important design components: saliency masking and visual obfuscation by mosaicing. EAR${}_{\mbox{{w/o obf}}}$ disables visual obfuscation for hint-providing. That is, EAR${}_{\mbox{{w/o obf}}}$ empties saliency region without any hint. EAR${}_{\mbox{{w/o attn}}}$ does not utilize the ImageNet [9] pre-trained DINO-ViT [6] to cut out suspected anomalous regions. Thus, EAR${}_{\mbox{{w/o attn}}}$ reconstructs the whole image that is obfuscated. The results for each model are shown for the best hyperparameter conditions. EAR${}_{\mbox{{w/o obf}}}$ shows an inpainting mistake on the background region marked with a red box, and EAR${}_{\mbox{{w/o attn}}}$ struggles reconstructing the normal region within the defective sample. The full model, EAR, accurately reconstructs normal regions within a defective sample, marked in the yellow box. In addition, red-boxed anomalous regions are successfully transformed into a normal form without inpainting mistakes. Those reconstruction results confirm that both saliency masking and mosaic obfuscation for hint-providing play an essential role in achieving contained generalization ability [29] by complementing each other.

In Figure 5, we present visual comparisons of the EAR variants with RIAD [45]. RIAD [45] features reconstruction by inpainting with multiple disjoint random masks and provides a cumulated error map. The reconstruction results of RIAD [45] are just one case from multiple masking. The overall edge error, MSGMS, is large due to a lot of random patch masks being located with the edge of the object in RIAD [45] case. EAR${}_{\mbox{{w/o obf}}}$ case shows the inaccurate reconstruction of the normal region by a large binary mask. Especially in a defective pill case, binary masking causes confusion as to whether the empty space should be filled with a red dot pattern or a white color. EAR${}_{\mbox{{w/o attn}}}$ shows scattered minute errors all over the region are produced because of the spatial discontinuity of the input image due to the mosaic. In contrast, EAR accurately reconstructs normal patterns by leveraging the hint-providing strategy from mosaic obfuscation; specifically, the logo and digit printings of the capsule. It successfully transforms the scratched white digit printing ‘500’ into a normal form.

We train EAR with anomaly-free samples. Then, the AUROC is measured for each subtask with an MSGMS-based anomaly scoring method. The measured performance is summarized in Table 3. As our study proposes a strategy to maximize the performance without changing the NN structure, we compare the performance with recent studies that use NNs of the same or similar scale.

EAR achieves the best performance in hazelnut, pill, and transistor cases compared to other models. The common characteristic of defective samples in these subtasks is surface damage which can be recovered into normal form by EAR. In cases of capsules, screws, and zippers that show sophisticated features, AUROC is relatively low compared to the highest performance. This is because the detailed pattern alignment of screw thread or the zipper teeth by reconstruction may be slightly missed due to saliency masking and visual obfuscation in the suspected anomalous regions. In the other cases that show AUROC close to 1, EAR also achieves similar cutting-edge performance with a very slight margin.

In conclusion, EAR achieves AUROCs of 0.922 and 0.942 by utilizing $\hat{m}$ and $m^{*}$ respectively. The performance 0.922 with $\hat{m}$ indicates that EAR with mosaic scale estimation enhances the UAD performance compared to prior state-of-the-art models. When the mosaic scale tuning with grid search is conducted for each subtask to use the true optimal mosaic scale, we can further enhance the UAD performance.

The measured processing time is summarized in Table 4. RIAD [45] takes the longest time for both training and inference due to the multiple masking strategy. For reference, the number of masks used in RIAD [45] is set to 12 as suggested in the original paper. In Table 4, EAR${}_{\mbox{{w/o attn}}}$ shows the fastest speed because it does not involve generating saliency maps through a pre-trained attention model, ImageNet [9] pre-trained DINO-ViT [6]. EAR${}_{\mbox{{w/o obf}}}$ and EAR are somewhat slower than EAR${}_{\mbox{{w/o attn}}}$ because they generate saliency maps via a pre-trained attention model. They show 2.35 $\times$ and 1.86 $\times$ faster inference than RIAD [45], respectively. EAR shows an inference speed fast enough for real-time processing with the highest UAD performance.

We have conducted an ablation study to see how deterministic saliency masking and obfustication by mosaicing for hint-providing affect the UAD performance. In addition, we also check the effect of applying the knowledge distillation (KD) method, part of SQUID [40], which uses two of the same NNs as teacher and student, respectively, during the training stage.

The results of the ablation study are summarized in Table 5. The 2nd column shows the cases of EAR${}_{\mbox{{w/o obf}}}$. It appears to be difficult to achieve a high performance because binary masking empties all the information in the suspected defective regions, causing inaccurate reconstruction on both normal and anomalous patterns. On the other hand, EAR${}_{\mbox{{w/o attn}}}$, shown in the 3rd column, confirms that the obfustication by mosaicing achieves better UAD performance compared to RIAD [45] and EAR${}_{\mbox{{w/o obf}}}$ because of the relatively accurate reconstruction of normal patterns, reducing false positives. Also refer to the visual comparisons of the above cases in Figure 5.

We confirm that the UAD performance is further improved when mosaic is used for hint-providing as shown in the last column, which is the case of EAR. Note that, the full model of EAR exploits both hint-providing and pre-trained attention-based saliency masking.

When the KD strategy from SQUID [40] is additionally applied (the 5th column in Table 5), there is almost no change in the performance. Referring to the expensive training cost of KD due to the use of two identical NNs (for each teacher and student), we do not see any advantage of additionally employing KD for EAR.