Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers

Currently, a large body of works on self-supervised learning focus on discriminative approaches [1, 2, 3, 4, 5, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], which regard each image as a different class and trains the model by discriminating them up to data augmentations. From a mutual-information perspective [31, 18, 1], models are trained to maximize the mutual information between different augmented views of an image. Plenty of positive and negative samples are needed to discover the similarity and dissimilarity. In practice, this requires large batches or memory banks. More recent works [4, 32, 33] have shown that we can learn high-quality representations without negative samples.

Self-supervised Vision Transformers. MoCov3 [7] first explored the iceberg of transformer-based self-supervised learning and presented a training recipe to let ViT perform reasonably well on ImageNet-1K linear evaluation. MoBY [6] adopts Swin Transformer [34] as the backbone to evaluate the learned representations on downstream tasks such as object detection and semantic segmentation. DINO [5] presents a new self-supervised learning method that shows good synergy with the transformer architecture and achieves the comparable performance of large self-supervised ConvNets using small/medium-size Transformers. Based on DINO, EsViT [8] further pursues efficient solutions to self-supervised vision transformers with two major insights: a multi-stage transformer architecture with sparse self-attentions, and a region-matching pre-training task. Concurrent BERT[35] like approachs (BEiT[36], MAE[10], CAE[37]) propose to reconstruct raw pixels via mask image modeling. And [38] demonstrates that them also implicitly learns occlusion-invariant features, which are analogous to other discriminative methods, while the latter learns other invariance. iBOT [9] performs masked image modeling via self-distillation as DINO with an online tokenizer. Different from these works on objective, our work aims at developing an automatic view learning method rather than using hand-crafted view policies.

Generation-based Pretext Tasks. At present, only a very small amount of work has explored automatic view generation for self-supervised learning on small-scale datasets preliminarily. InfoMin [18] leverages an adversarial semi-supervised view learning strategy. Except the flow-based view generation model which is adversarially trained to minimize mutual information, they need two classifiers on each of the learned views to perform classification using labels for the downstream task during the view learning process. InfoMin needs to first train the generative network with labels, then the frozen generator is used to generate views for self-supervised representation learning. Viewmaker [19] apply adversarial robustness literature to generate a variety of residual perturbations for a single input, with an iterative optimized image-to-image neural network as their Viewmaker network, requiring optimized in alternating steps with encoder. Their ineffectiveness prevents them from applying to self-supervised vision transformers that already require a large amount of computing resources. In this work, we employ a constrained adversarial training method that adaptively reduces the mutual information between different views while preserving useful input features for the encoder to learn from just through one-step optimization.

Inspired by recent advancements in neural architecture search (NAS) [39, 40], researchers try to learn augmentation policies from data automatically [12, 17, 13, 14, 15, 41]. AutoAugment [12] first adopts reinforcement learning for auto-augmentation tasks but it requires searching for thousands of GPU-days. AdvAA [17] proposes online search manners by jointly optimizing augmentation policies and training the target networks. RandAug [13] directly uses naive grid search to find the best policy. DDAS [15] completes differentiable augmentations search through meta-learning with one-step gradient update. DADA [14] introduces trainable policy parameters and adopts the Gumbel technique to update it with network weights by gradient descent algorithm alternately. These methods are based on supervised search space and need target validation sets, either have large search costs for multi-step optimization or multiple experiments, which are not suitable for self-supervised learning. Thus this work proposes a self-regularized adversarial AutoView that can be optimized with self-supervised networks simultaneously in each single forward-backward step.

View-based Self-supervised Vision Transformer. View-based self-supervised vision transformers have shown remarkable results in representation learning recently, whereby one image $\bm{x}$ is augmented using two separate data augmentations to get augmented views $\bm{v}_{1},\bm{v}_{2}$, and then tranformers are trained to maximize the mutual information between different views i.e., $I(\bm{v}_{1};\bm{v}_{2})$, through maximize $\textit{sim}(f_{1}(\bm{v}_{1}),f_{2}(\bm{v}_{2}))$, where $\textit{sim}(\cdot,\cdot)$ is a similarity function and $f_{1},f_{2}$ are transformer encoder network. And $f_{1}$ parameters are updated with an exponential moving average (ema) of the $f_{2}$ parameters. But generating views through careful construction of data augmentation strategies is still a significant limitation, as several barriers are hindering automatic view learning development and application on self-supervised vision transformers.

Adversarial View Learning Through One-Step Optimization. Without label supervision, adversarial learning could be a useful way to learn augmentation policies for self-supervised vision transformer. For an input image $\bm{x}$, the view generator is defined as $g$ . Given $\bm{v_{1}},\bm{v_{2}}=g(\bm{x})$, and two encoders $f_{1},f_{2}$ are trained to maximize $I(\bm{v_{1}};\bm{v_{2}})$. Meanwhile, $g$ is adversarially trained to minimize $I(\bm{v_{1}};\bm{v_{2}})$. Then the min-max objective is:

However, the ineffectiveness of multi-step optimization of the encoder and generator prevents it from applying to self-supervised vision transformers that already require a large amount of computing resources.

Hence, we employ a constrained adversarial training method that enables the model to adaptively reduce the mutual information between different views just through one-step optimization. In practice, we follow [5] to learn to match the encoder networks’ output probability distributions by minimizing the cross-entropy loss w.r.t. the parameters of the $f_{2}$ encoder network with the $f_{1}$ network fixed. And our $g$ generates views by sampling according to probability, then we have:

where $H(a,b)=-a\log b$. We relax the policy selection to be differentiable through Gumbel-Softmax reparameterization [42] to achieve differentiable optimization of our image transforming $g$, details in Sec. 3.3. It is worth noting that we realize the one-step optimization of the generator and encoder through Gradient reversal layer (GRL)[43]. Remarkably, through that, the search cost of AutoView is nearly zero as views and network parameters are simultaneously optimized in each single forward-backward step. Through one-step optimization, AutoView addresses the high learning cost and successfully performs AutoAugment for self-supervised learning in large-scale datasets for the first time.

With the loss function in Eqn. (2), AutoView intends to increase the sampling probability of those transformations that can generate samples with high training loss. By sampling such transformations, AutoView can pay more attention to more aggressive augmentation strategies and increase model robustness against difficult samples. However, blindly minimizing the mutual information, i.e., increasing the difficulty of samples may cause the augment ambiguity phenomenon [44]: augmented images may be far away from the majority of clean images, which could cause the under-fitting of models and deteriorate the learning process. InfoMin designed a semi-supervised method using a supervised constraint guiding the generator to alleviate this problem, which is not suitable for self-supervised learning without labels.

The Information Bottleneck (IB) theory [45] uses the information theoretic objective to constrain the mutual information between the input and the representation, which can help us to preserve information. As previous work [46] shows that the information constraining objective in supervised setting has the same form as that of self-supervised setting except the target outputs. Therefore, we unify these two objectives by using $\bm{t}$ as the output of the downstream tasks. In supervised setting, $\bm{t}$ represents the label of input $\bm{x}$. In self-supervised setting, $\bm{t}$ represents the input $\bm{x}$ itself. This leads to the unified objective function linking the representation $\bm{z}$ of input $\bm{x}$ and target $\bm{t}$ as:

This unified objective describes a constraint with the goal of maximizing the mutual information between the representation $\bm{z}$ and the target $\bm{t}$. Hence, we propose a self-regularized loss to avoid information collapse caused by the lack of label supervision:

where $\bm{x}$ is the original image, $\bm{v}\sim g(\bm{x})$ is the augmented image. So this self-regularized loss is to keep the information $I(\bm{v},\bm{x})$ to encourage distinguishable image features that share relevant semantic information with the original data.

In summary of Eqn. (2) and Eqn. (4), the training process of our AutoView is formulated as follow:

where the second term of the loss function is actually Eqn.(4). We use $f_{1}$ to achieve our information constraint on $g$, which will not affect the training of $f_{2}$, and then accomplish the conversion of maximization and minimization by the negative sign. So that we integrate the self-regularized loss into our adversarial view learning framework without undermining its advantage of one-step optimization.

The current search space designed for supervised AutoAugment is unsuitable for view-based self-supervised learning as analyzed previously. Meanwhile, existing view learning methods, both the generative Viewmaker model and flow-based view generation model of InfoMin are pixel-wise generators, which are not suitable for view-based self-supervised learning of embedding space and can’t be applied to large scale datasets due to the limitations of the pixel-wise image generator. Thus we present a curated augmentation policy search space and design our online policy search framework based it.

In detail, we construct our search space of image pre-processing operations with 12 candidates. We denote the set of operations as $\mathcal{O}$: {AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Hue, Grayscale, Gaussianblur}, removing geometric transformation(e.g., ShearY, TranslateX) [47] and adding augmentations commonly used in self-supervised methods (e.g., Gaussianblur, Grayscale) [3]. Here we intuitively list the augmentation operations in our curated search space:

Each augmentation policy $\varphi$ contains $K$ image pre-processing operations: $\varphi={\left\{o_{k}\right\}}^{K}_{k=1}$, where $o_{k}\in\mathcal{O}$. With $|\cdot|$ represents the number of elements in the set, we define sampling weights ${\bm{\pi}}\in\mathbb{R}^{K\times|\mathcal{O}|}$ to represent the sampling probability for each candidate operation, which is the parameter that the generator needs to learn. Therefore, the $k$-th image operation of policy can be sampled as:

For $i$-th policy, each image operation $o^{i}_{k}$ owns two parameters: the magnitude $m^{i}_{k}$ that determines the strength of the transformation and the probability of applying the transformation, $p^{i}_{k}$. So that the output of image $\bm{x}$ pre-processing $\tilde{o}^{i}_{k}$ can be computed as follows:

And it is worth noting that if the policy is not applied, that is when there is no augmentation, the self-supervised training will easily collapse. So instead of sampling and applying the whole policy by probability as in previous work [14], as shown in the left part of Fig. 2, we sample the policy by layers and the execution probability of the first layer is fixed at 1. Finish formulating our search space and then we relax the policy selection to be differentiable through Gumbel-Softmax reparameterization [42].

As to search adversarial policy online with encoder, we use the Gumbel-Softmax reparameterization trick [42] to achieve the differentiable relaxation. Thus, we regard Eqn. (7) as $bO(\bm{x};m)+(1-b)\bm{x}$, where $b\in\{0,1\}$ is sampled from Bernoulli distribution. Since this distribution is non-differentiable, we instead use Relaxed Bernoulli distribution:

where $u$ is sampled from a uniform distribution on $[0,1]$. Using this reparameterization, each operation can be differentiable w.r.t. its probability parameter $p$. For policy sample, we approximate their gradient in a similar manner to the straight-through estimator. For forward, we first sample operation by Eqn. (6), e.g.$i$-th operation. Then:

where $\bm{c}=\mathrm{softmax}(\bm{\pi}_{k})$ and $\bm{h}=one\_hot(\bm{c},\bm{i})$. So the backward pass uses differentiable variables $\bm{c}$ and the forward pass uses discrete variables $\bm{h}$.

Through the above relaxation, we can achieve differentiable optimization of image transforming $g$. So then the search cost of AutoView is nearly zero as we can learn views and network parameters simultaneously in Eqn. (5).

Architectures. We use the Vision Transformers with different amounts of parameters, ViT-S/16, ViT-B/16 as the encoder network $f$. For ViTs, /16 denotes the patch size being 16. We pretrain and finetune the transformers with 224-size images, so the total number of patch tokens is 196. The projection head $h$ is a 3-layer MLPs with $l_{2}$-normalized bottleneck following DINO[5].

Pretraining Details. We pretrain the models on the ImageNet or OpenImages without labels. For self-supervised training, the hyperparameters follow closely to DINO. We use the adamw optimizer with a cosine decay learning rate schedule. The learning rate is linearly ramped up during the first 10 epochs to its base value (0.0005 for the batchsize of 256). The weight decay also follows a cosine schedule from 0.04 to 0.4.

For AutoView policy training, we made a simplification on our preliminary search space, we discretized the continuous magnitude of augmentation and considered the same augmentation with different magnitudes as multiple operations. The operation number in an augmentation policy is set to $K=2$ and the two operations share sampling weight $\bm{\pi}$. Then we adopt Adam optimizer with $\beta=(0.5,0.999)$ optimizing the policy weights $\pi$ with a step decay learning rate schedule, and the initial learning rate is 6e-5. The execution probability $p$ also optimized by Adam optimizer with $\beta=(0.5,0.999)$ with learning rate 1e-5.

For progressive learning, we divide the training process into four stages with about 25 epochs per stage: the early stage uses a small image size, while the later stages use larger image sizes with stronger regularization. And the minimum (for the first stage) and maximum (for the last stage) values of image size are 128, 224. The small view size of multi-crop training is from 55 to 96. Both of them change linearly according to the training stage.

For grid search of RandAug, RandAug makes the following simplifying assumptions: all operations share a single, discrete magnitude, $M\in\left[0,10\right]$; all policies apply the same number of operations, $N$; all operations are applied with uniform probability. We select the best $k$-NN result from a grid search over $\left(N,M\right)$.

Linear and $k$-NN Evaluation Details. We follow the evaluation protocols in DINO. For linear evaluation, we sweep over different learning rates. For $k$-NN evaluation, we sweep over different numbers of nearest neighbors.

End-to-end fine-tuning Details. We follow the protocol used in DeiT[11] and finetune the features on downstream transfer tasks, e.g., Cifar10 and Flowers.

Fine-Tuning Setting of Semantic Segmentation on ADE20K. For semantic segmentation, we follow the configurations in iBOT [9], fine-tuning $160$k iterations with $512\times 512$ images and a layer decay rate of $0.65$ and set the learning rate on $8e^{-5}$. We do not use multi-scale training and testing. To produce hierarchical feature maps, we use the features output from layer $4$, $6$, $8$, and $12$, with additional deconvolution layers appended after.

We evaluate our method on two large scale image datasets, ImageNet-1K [48] and OpenImages [49]. ImageNet contains 1.2M train set images and 50K val set images in 1,000 classes. OpenImages consists of a total of 9.1 million images. We follow [50] to construct a subset of the OpenImages dataset that has 212K images present across 208 classes.

Linear and $k$-NN results on ImageNet. To evaluate the quality of pre-trained features, we either use a linear classifier on the frozen representation or a $k$-nearest neighbor classifier ($k$-NN) . In Tab. I, AutoView achieves remarkable improvements over RandAug baseline on $k$-NN and linear classification (+10.2% and +7.3%) and steadily boosts both the top-1 and $k$-NN classification accuracy on ImageNet for different frameworks (e.g. DINO, iBOT, MoCoV3) , different architectures (e.g.ViT-S,ViT-B) and different training lengths (e.g.100$\sim$800 epochs). To be specific, AutoView can promote the $k$-NN accuracy of DINO with ViT-S of various train periods by 0.8%, 1.2%, 0.4% respectively, and the results are shown in Fig. 3 intuitively show our continuous boost on different training lengths(e.g. 100$\sim$800 epochs). Particularly, AutoView consistently boosts the DINO on the progressive training setting with the gain of 1.3% $k$-NN accuracy.

Why we focusing on ViTs? First, data augmentation is more critical in ViTs than CNNs. The vanilla ViT needs to be trained with a large dataset with full supervision (e.g., ImageNet-21k). DeiT improves the vanilla ViT-L by up to 2.2% using better augmentation policies proposed by RangAugment. Second, more and more current sota self-supervised methods use ViTs as their backbone, and our method is in line with them. Moreover, our AutoView is a general AutoAugment method for self-supervised learning. As shown in Tab. I, our method consistently improves the performance when applied to CNN-based self-supervised learning approach.

Classification results on OpenImages. We firstly pretrain models on OpenImages dataset. After pre-training, we freeze the backbone weights and train a linear classifier with a multi-class logistic regression loss. We follow the mAP metric as described in [50]. The last row in Tab. II shows the continuous boost obtained by using our AutoView on OpenImages.

As self-supervised training is time-consuming and image size plays an important role in training efficiency. [51, 52, 53, 54] have proposed different kinds of progressive training, which dynamically change the training settings or networks. We take the insight of EfficientNetV2[54] and present an efficient self-supervised learning scheme by progressively increasing the image size during training. We divide the training process into four stages. In the early training epochs, we trained the network with smaller images such that the network can learn simple representations easily and fast and then gradually increase image size. As the result shown in Tab. III, we indeed improve the training speed but it also comes with a drop in accuracy. In this section, all models are trained for 100 epochs with ViT-S/16 and train time is counted on the same machine using 8 RTX 2080Ti GPUs with the total batch size 256.

Progressive Learning with AutoView. As EfficientNetV2 points out using the same regularization for all image sizes causes the drop in accuracy. To improve both training speed and accuracy, they also adaptively adjust regularization linearly according to image size. To continue to explore this property, we adapt grid search over $(N,M)$ as RandAug for DINO progressive training in our search space to select the best result of linear classification on imagenet.

With the search result $\{(2,3),(2,6),(2,6),(2,8)\}$ of four train stage, we improve both the accuracy and training time. The above results prove that adaptive regularization helps progressive learning gain increase in accuracy. But the RandAug search results also show that simple linear schedule regularization just according to image size is not always the most suitable. Then thanks to AutoView, we use it to dynamically adjust augmentations during model training and get better results than costly and tedious manual search, which clearly demonstrates the effectiveness and generalization of AutoView.

We first evaluate our method in the dense downstream tasks, semantic segmentation, on which we observe improvements over the vanilla pre-trained baselines. We then evaluate properties of the learned features of models pretrained for 300 epochs with ViT-S/16 in terms of nearest neighbor search and transferability to different datasets.

Semantic Segmentation. We evaluate our method in the dense downstream task, semantic segmentation, on which we observe improvements over the vanilla pretrained baseline. And in this experiment models are pretrained for 400 epochs with ViT-B/16. Our semantic segmentation experiments on ADE20K[55] use UperNet [56] following the code in iBOT. We use the task layer in UperNet and fine-tune the entire network. From Tab. IV, we can see that AutoView advances DINO with ViT-B/16 with a clear margin of 1.2% on mAcc.

Video Object Segmentation. We evaluate the output patch tokens on the DAVIS-2017 video instance segmentation benchmark[57]. We segment scenes with the nearest neighbor between consecutive frames, while do not train any model on top of the features, nor fine-tune any weights for the task, and follow the experimental protocol as DINO. According to Tab. V, AutoView is superior to vanilla DINO on all metrics.

Image Retrieval. We consider the revisited Oxford [58], which contains 3 different splits of gradual difficulty with query/database pairs. We freeze the features and directly apply $k$-NN for retrieval as DINO, reporting the Mean Average Precision (mAP) for the Medium and Hard splits. Since AutoView has higher $k$-NN results on Imagenet-1K, the performance is also better for AutoView in the image retrieval task (up to 2.8%) as in Tab. V.

Transfer learning. We evaluate the quality of the features pretrained on ImageNet-1K and fine-tune on several smaller datasets. The results are demonstrated in Tab. VI. While the results on CIFAR10 and CIFAR100 have almost plateaued, AutoView consistently boosts the baseline.

AutoView enables models to cover more augmentation policies. This property could improve model robustness to novel examples. To verify if AutoView can improve ViT-based models’ robustness and out-of-distribution performance, we evaluated our pretrained models on two robustness scenarios including natural adversarial example and out-of-distribution detection. And we also verify that our method has no overfitting problem on ImageNetV2 [59]. In this section, we study the robustness and overfitting with ViT-S/16 pretrained for 300 epochs and then linearly evaluated for 100 epochs.

Natural Adversarial Example. The dataset ImageNet-A [60] adversarially collects 7500 unmodified, natural but “hard” real-world images which are drawn from some challenging scenarios (e.g., fog scene and occlusion). The metric for assessing classifiers’ robustness to adversarially filtered examples includes the top-1 accuracy and Area Under the Response Rate Accuracy Curve (AURRA). AURRA is an uncertainty estimation metric introduced in [60]. As shown in Tab. VII, the ViT trained by AutoView improves over the vanilla pretrained DINO by 2.1% AURRA.

Out-of-distribution Detection. The dataset ImageNet-O[60] is an adversarial out-of-distribution detection dataset which adversarially collects 2000 images from outside ImageNet-1K. The anomalies of unforeseen classes should result in low-confidence predictions. The metric is the area under the precision-recall curve (AUPR) . As Tab. VII indicates, AutoView outperforms baseline by 1.0% AUPR.

Performance on ImageNetV2. Tab. VII reports the additional evaluation on ImageNet V2[59], that has a test set distinct from the ImageNet validation, which reduces overfitting on the validation set. The results (+0.9% $k$-NN) verify the generalization and effectiveness of our method. Note that our method does not use the validation set for augmentation searching, so there should be no overfitting problem.

In this section, we perform extensive ablation studies to analyze each components of our proposed AutoView. All the results in this section are obtained by training ViT-S/16 with different schemes for 100 epochs.

Effectiveness of the Self-Regularized Loss. We compare the pretraining objective with and without self-regularized loss. As shown in Fig. 4, the training curve with our self-regularized loss is much stabler, and converges to a lower loss value, which demonstrate that the self-regularized loss can effectively avoid information clapse. As shown in the fourth column of Tab. IX, without self-regularized loss (Reg. Loss), the performance dropped by 0.4% in regular AutoView training and dropped considerably by 3.7% in progressive learning, proving its importance in AutoView.

Effectiveness of the Search Space. For our search space, we compared our present curated augmentation policy search space with the one designed for supervised learning that is generally used by popular AutoAugment methods. As shown in Tab. VIII, AutoView and RangAug both gain a large margin of linear and $k$-NN accuracy with our search space. For the first time, the AutoAugment method for self-supervised learning is proved to be able to perform in a policy search space. For AutoView, we can find it significantly outperforms RandAug on same search space (up to +3.8% $k$-NN accuracy). And it is worth mentioning that the results of RandAug are obtained through tedious and time-consuming grid search.

Importance of Searching. As shown in Tab. IX, we further investigate the effect of searching for sampling weights and execution probability of AutoView. In the first column, we search policy hierarchically (H. Policy) instead of sampling and applying the whole policy by probability. Then we fix the execution probability at 0.5 without searching (Exc. Prob.). Next, we randomly select views in our policy search framework rather than sampling by the weights searched (Weight). The clear accuracy drop reflects the searching of hierarchical policy, execution probability and views are all vital components of AutoView.

The trade-off of two objectives in AutoView. There exists a trade-off between $min(f_{1}(v_{1}),f_{2}(v_{2}))$ and $max(f_{1}(v_{1}),f_{1}(x))$ in our AutoView. And we have considered and verified the trade-off between these two objectives through a loss weight $\alpha$, which multiplies on the self-regularized item. The results are shown in Tab. X, where we can see that $\alpha$ is 1.0 is the optimal setting we adopt in this paper. Moreover, our self-regularized loss is the constraint of the augmented view and the original view on the single network $f_{1}$, which is much weaker than the loss between the two siamese networks and just can be seen as one regularization term, and the regularization term is adaptively adjusted during training as the Fig. 5 shows.

Method Complexity. Our searched augmentation policy is general and universal. One can directly use our searched policy to get the same performance results without searching. As shown in Tab. XI, the model retraining with searched policy (Policy-retrain in Tab. XI) performs comparable with the the model that use AutoView in an online way (AutoView in Tab. XI).

Search Space Design. We adopt the supervised RA search space and make non-trivial modifications. The vanilla RA search space did not contain the transformations commonly used in SSL. We first added them, but then we found that the effect was unsatisfactory (see Tab. XI). With many geometric transformations combined, the object often shifts from the center of the image [47], which hurts SSL model learning, so we removed these transformations.

Choice of $K$. First, the $K$ of sota methods (i.e., AA, RA, DADA, etc.) is set to 2. Second, our method is based on RA. During RA searching (see Sec. 4.3), we found that $K$=2 achieved the best results. We also explore different $K$s in Tab. XI.

Result when $g$ is not shared for $v_{1}$ and $v_{2}$. We have considered and verified various cases of network structure design. When $g$ is not shared, the result is sub-optimal (see Tab. XI).

Search Overhead. The search overhead of different methods in Tab. XII shows the advantage of our one-step optimization. AutoView learns adversarial augmentation policies efficiently by learning views and network parameters simultaneously in a single forward-backward step without additional search costs. Noting that, the search overhead there is just a rough estimate, as for InforMin and Viewmaker, we did not consider the generation time of the pixel-wise image generator and for DADA, the search is only conducted on a small proxy subdataset.

Qualitative Studies of AutoView Searching. To demonstrate the effectiveness of our AutoView searching, we visualize the different operations’ sampling probability searched in Fig. 7. In particular, we put geometric transformation (e.g.,ShearY, TranslateX) back into our search space in this experiment. We can see that the probabilities of geometric transformation operations are getting smaller while GrayScale, the augmentations commonly used in self-supervised methods, is getting larger, which proves the correctness of our AutoView.

Visualization Results of Attention Maps In Fig. 6, we indicate that our AutoView shows visually stronger ability to capture the information of objects, especially in situations with complex backgrounds, compared with DINO Manual. For example, in the first column and the last column, the attention maps learned by AutoView are the complete information of objects (e.g., cats and cabinets) and eliminates the interference of background as much as possible than DINO Manual.

As Fig. 8 shows, without self-regularized loss, the features learned by the model are chaotic. By comparison, the model trained with our self-regularized loss can learn the target object information well.