PyramidFlow: High-Resolution Defect Contrastive Localization
using Pyramid Normalizing Flow

With the rise of deep learning, numerous works apply generalized computer vision methods for defect detection. Some works are based on object detection[14, 31, 30, 28, 13], which relies on annotated rectangular boxes, enabling locating and classifying defects end-to-end. The other is applying semantic segmentation[18, 24, 5], which enables pixel-level localization, suit for complex scenarios with difficult-to-locate boundaries. However, these works still rely on supervised learning, they attempt to collect sufficient defective samples to learn well-defined representations.

Recently, some promising work has considered the scarcity of defects in real-world scenarios, where defect-free samples are only obtained. These methods can be classified as reconstruction-based and anomaly-based. The reconstruction-based method relies on generative models such as VAE or GAN, which encode a defective image and reconstruct it to a defect-free image, then localize the defect with the contrast of these two images. The reconstruction-based method performs well on single textural images, but they cannot generalize to non-textural images for ill-posedness and degeneracy[25]. The anomaly-based method treats defects as anomalous, applying neural networks to discriminate between normality and anomalous. These methods extract pre-trained features, then estimate their probability density using Mahalanobis distances or K-NearestNeighbor, while the lower probability indicates where the image patches are abnormal. Although anomaly-based methods had achieved great success in defect detection, it locates defects with low pixel-level resolution compared with reconstruction-based methods, usually 1/16th or even lower, which greatly limits practical industrial applications.

To overcome existing shortcomings, we propose a latent template-based defect contrastive localization paradigm, which breaks the limitation of low-frequency texture-aware-only models, enabling more accurate results.

Normalizing flow is a kind of invertible neural network with bijective mappings and traceable Jacobi determinants. It was first proposed for nonlinear independent component estimation[8] and applied to anomaly detection[21] recently for its invertibility helps prevent mode collapse. The normalizing flow comprises coupling blocks, these basic modules for realizing nonlinear mappings and calculating Jacobi determinants. Originally, NICE[8] proposed the additive coupling layer with unitary Jacobi determinants, while RealNVP[9] further proposed the affine coupling layer that enables the generation of non-volume-preserving mappings. However, redundant volume degrees of freedom can lead to increased optimization complexity, creating a domain gap between maximum likelihood estimation and anomaly metrics, which may potentially compromise the generalization performance in anomaly detection.

Previous works[21, 10, 27] on anomaly localization usually follow the methods proposed in RealNVP, but some challenges remain. Some studies[22] have found that convolutional normalizing flow focuses on local rather than semantic correlations, which are usually addressed by image embeddings[12]. Hence, earlier studies[21] adopted pre-trained backbones, while recent trends used pre-trained encoders to extract image patches[22, 10, 27]. However, pre-trained-based methods rely on task-irrelevant external priors, which limit generalization in unforeseen scenarios.

To address the above challenges, we propose a pyramid-like normalizing flow called PyramidFlow, which utilizes volume normalization to preserve volume mappings that include task-relevant implicit priors. Additionally, our method offers the option of using pre-trained models, and we have observed that external priors from pre-trained models can improve generalization performance. We will discuss these contributions in Sec. 4.3 and Sec. 4.4.

Defect images contain various frequency components. Usually, the low-frequency components represent the slow gradient background, while the high-frequency components correspond to details or defects. To decouple the frequency components and identify each frequency component independently, we propose invertible pyramids, which enable multi-scale decomposition and composition for a single feature. To facilitate feature learning, previous work applies pre-trained encoders to extract features. Although pre-trained methods with external priors help performance improvement, to fully explore the advantages of our approach in our primary study, let’s consider a baseline without any pre-trained model.

For a three-channel image $I$, apply orthogonally initialized $1\times 1$ convolution $\mathbf{W}\in\mathbb{R}^{C\times 3}$ to the image for obtain features $x=\mathbf{W}I$. Given a feature $x$ and a positive integer $L$, the pyramid decomposition is a mapping from features to feature sets $\mathcal{L}_{dec}:x\to\{x_{d}|d\in\mathbb{Z}_{L-1}\}$, where $x_{d}$ is the $d-$level pyramid can be calculated as

Differing from Gaussian pyramid, Eq. 2 indicates that there is always an inverse operation for pyramid decomposition, which is called pyramid composition. The method based on Eqs. 1 and 2, enabling perform multi-scale feature decomposition and composition, is a critical invertible module for PyramidFlow.

As shown in Fig. 3(a), the conventional affine coupling block splits a single feature along the channel dimension, where one sub-feature keeps its identity while another is performed affine transformation controlled by it. Denote the splitted features are $x_{0},x_{1}$ and its outputs are $y_{0},y_{1}$, then the corresponding transformation can be described as

Denote the element at position $i,j$ of $s(\cdot)$ as $s_{i,j}(\cdot)$. As the Jacobian matrix of transformation(3) is a triangular matrix, its logarithmic determinant can be estimated as

Eqs. 3, 4 and 5 are the basis of all affine coupling blocks. However, the coupling block shown in Fig. 3(a) remains identical for one part. Therefore the reverse cascade architecture is proposed in NICE[8] such that both parts are transformed, as shown in Fig. 3(b). The previous works construct the holistic invertible normalizing flow by iterative applying the structure shown in Fig. 3(b).

In addition, we employ invertible 1x1 convolution[11] for feature fusion within features. Specifically, denoting the full rank matrix corresponding to the invertible $1\times 1$ convolution as $\mathbf{A}$, which can be decomposed by PLU as

In summary, Eqs. 3, 4, 5, 6 and 7 describe proposed pyramidal coupling block mathematically, as shown in Fig. 3(c). First, multi-scale feature fusion(3-5) is performed, and then apply linear fusion(6-7) for shuffle channels. Furthermore, we propose a dual coupling block as shown in Fig. 3(d), which is equivalent to the reverse parallel of the coupling block in Fig. 3(c). The dual coupling block is reparameterized in our implementation, and its affine parameters $s(\cdot),t(\cdot)$ are estimated from concatenated features.

However, this approach relies on the basic distribution assumption and ignores the effect of the implicit prior in the probability density transform on generalization. When such approaches are applied to anomaly detection, the inconsistency between the training objectives and the anomaly evaluation results in domain gaps.

Similar to batch normalization or instance normalization in deep learning, the proposed volume normalization will be employed for volume-preserving mappings, as illustrated in Fig. 4. Particularly, for the affine coupling block, the parameter $s(\cdot)$ is subtracted from its mean value before performing Eq. 3; for the invertible convolution, the parameter $s_{i}$ is subtracted from its mean value before calculating the matrix $\mathbf{A}$ based on Eq. 6. Depending on the statistical dimension, we propose Spatial Volume Normalization (SVN) and Channel Volume Normalization (CVN). SVN performs mean statistics along the spatial dimension, while CVN is along the channel dimension. Various volume normalization methods contain different priors, then we will explore their impact in Sec. 4.2.

Suppose a training batch with 2 normal samples, its latent variables are $z_{d}(i),z_{d}(j)$. Previous studies train neural networks using spatial difference $\Delta z_{d}=\|z_{d}(i)-z_{d}(j)\|^{2}$. However, it ignored the impact of high-frequency defects. To address the above shortcoming, we propose the following Fourier loss function.

The Eq. 10 shows that the total anomaly is a composition of anomalies at various scales, which is consistent with the empirical method proposed by Rudolph, et al. [22].

The normalizing flow based on Eqs. 3 and 4 is computationally invertible, which indicates that only one copy of the variables is necessary for all stages. This feature decreases the memory footprint during backpropagation from linear to constant complexity. We have analyzed the above characteristics based on a fixed number of pyramid layers $L=8$, image resolution with $256\times 256$, and channels $C=24$, then changed the number of stacked layers $D$ to explore the trends of memory usage and model parameters. The forward and memory-saving backpropagation is implemented based on the self-developed PyTorch-based[17] framework autoFlow. All indicators are recorded during steady-state training, then plotted as bar and line graphs, as shown in Fig. 5.

The memory usage based on auto-differentiation increases linearly with depth $D$, while the implementation based on normalizing flow achieves approximate depth-independent memory usage. The memory superiority enables the proposed method to be trained in memory-constrained devices below 4G without powerful hardware. The line graph shows the exponential trends between model parameters and depth, while the horizontal line represents the parameters of the usual pre-trained model. We mainly adopt methods with $D<8$ or even shallower, where the number of parameters is far smaller than popular pre-training-based methods. In summary, in scenarios of memory constraint, the proposed PyramidFlow enables dealing with larger images and requires fewer parameters than others.

In this subsection, based on MVTecAD, we investigate the impact of volume normalization on generalization. The experiment without data augmentation, fixed pyramid layers $L=4$, channels $C=16$, and linear interpolated image to $256\times 256$. During the training, the volume normalization applies sample mean normalization and updates the running mean with 0.1 momenta, while in the testing, the volume normalization is based on the running mean. We sufficiently explored the volume normalization methods proposed in Sec. 3.2. Some representative categories are shown as Tab. 1.

The result in Tab. 1 shows the performance differences between the various volume normalization methods: CVN outperforms SVN for the first three classes, while the latter behaves the opposite. We further visualize these defect distributions in Fig. 6, which shows that SVN-superior classes are commonly textural images with a larger range of defects, while CVN-superior classes are object images.

SVN with larger receptive fields achieves non-local localization by aggregating an extensive range of texture features, while CVN realizes accurate localization by shuffling channels. In a word, different volume normalization techniques implicitly embody distinct task-specific priors. Furthermore, our ablation study in Sec. 4.3 shows that volume normalization does help to improve average performance.

This subsection discusses the impact of some proposed methods on performance. The study is conducted on full MVTecAD, and other settings are the same as Sec. 4.2. We ablate four methods from baseline individually. In particular, experiment I is based on latent Gaussian assumption and Eq. 8 without volume normalization. For experiment II , the category-independent zero template $\bar{z}_{d}=0$ is applied. Then, Experiment III does not adopt the method of Eq. 10 but composes the pyramid first and localizes its difference later. Finally, experiment IV adopts a spatial version of the loss function instead of Eq. 9. The result of the above ablation experiments is shown in Tab. 2.

Tab. 2 demonstrates that experiments I -IV present various performance degradation. Experiment I has the largest average degradation, with the object classes being more affected. Although the non-volume-preserving enables larger outputs and higher Image-AUROC performance, the implicit prior in volume normalization discussed in Sec. 4.2 is more helpful for generalization. For experiment II , it shows that the latent template benefits the performance, and object classes are improved greatly. It is because the category-specific latent template reduces intra-class variance, helping convergence during training. Then, experiment III suggests that multi-scale differences had a more pronounced impact on textural classes, as higher-level operators with larger receptive fields correspond to large defects. Finally, Experiment IV reveals that Fourier loss (9) is the icing on the cake that helps performance improvement. To summarize, methods I -III are critical for the proposed model, while tricks IV help further improvements.

First, we take three methods based on image contrast, AnoGAN[23], Vanilla VAE[15], and AE-SSIM[4]. They are not dependent on external datasets, so it is fair to compare them with our FNF model. Furthermore, we also compared our method to those that utilize external priors, such as S-T, SPADE, etc. All methods were reproduced based on the official implementation or AnomaLib[1]. For fair comparisons, we adapted the $1\times 1$ convolution $\mathbf{W}$ to the pre-trained encoder, where the pre-trained encoder is the first two layers of ResNet18 for extracting the image into features of original 1/4 size and with 64 channels.

As shown in Tab. 3, our FNF method greatly outperforms the comparable methods without external priors, even exceeding S-T, SPADE, and CS-Flow that using external priors. Most of the reconstruction-based methods in Tab. 3 suffer from ill-posedness in complex scenarios (e.g., tile and wood.), while our method achieves the best AUPRO score owing to high-resolution contrast in latent space. However, a larger resolution implies larger intra-class variance, which degrades the overall AUROC performance for hard-to-determine anomaly boundaries. Furthermore, Fig. 7(a) visualizes representative examples of MVTecAD anomaly localization, which shows that our method achieves precise localization with reasonable scale.

In principle, template-based methods require pixel-level registration between images and templates, which is typically satisfied in most real-world scenarios, but may fail in some cases. This subsection explores the performance of the proposed method on unregistered (e.g. rotation, shift) categories (e.g. grid, metal nut, screw, and hazelnut), as shown in Tab. 5. The reconstruction-based AE-SSIM heavily relies on pixel-level contrast, which decreases the average localization accuracy (AUPRO%). In contrast, the anomaly-based SPADE avoids registration issues and achieves better performance. Fortunately, our ResNet18-based method, which utilizes normalizing flow to reduce patch variance, remains competitive in unregistered scenes, although it falls short of state-of-the-art performance. We visualize these categories in Fig. 7(c).