Improving Contrastive Learning by Visualizing Feature Transformation

Let us start from the basic procedures of contrastive learning, as shown in Figure 2. Each data sample $x$ passes through two separate data augmentation pipeline $t_{q}$ and $t_{k}$, which are randomly sampled from the same data augmentation pool, and two views $v_{q}$ and $v_{k}$ will be acquired to construct positive pairs [6, 12]. The encoder $q$ and $k$ ²²2Encoder $q$ and $k$ might be the same [6] or different network [14, 12]. will respectively map two views into feature embedding space. An $\ell_{2}$ normalization is applied on feature vector $h_{q}$ and $h_{k}$ to project the corresponding vector $h_{q}$ and $h_{k}$ (i.e., $z_{q}=h_{q}/\|h_{q}\|_{2}$) onto the unit sphere and obtain $z_{q}$ and $z_{k}$. Their inner product will produce the cos similarity score, namely one positive pair score $S_{q\cdot k^{+}}$ and $K$ negative pair scores $S_{q\cdot k^{-}}$. These pair scores are input to InfoNCE loss 1 for contrastive learning:

Here we roughly defined Feature Transformation process as certain manipulations on encoder embeddings $h_{q}$ and $h_{k}$, in order to reshape the distribution of the output pos/neg pair score ($S_{q\cdot k^{+}}$ and $S_{q\cdot k^{-}}$.), for better contrastive learning in the follow-up InfoNCE loss. The most common FT applied in current SOTA is the [4, 6, 42, 7, 14] unit-sphere projection of $\ell_{2}$ normalization. We provide empirical studies of this regular FT and illustrate it importance for significant constriction of feature length ($\ell_{2}$) in Supp F.

We choose to visualize the score distribution of pos/neg pairs instead of the loss curves and transfer accuracy, as the inside training dynamics can unearth the learning capability of the model. Specifically, there are two practical reasons: (1) The basic idea of InfoNCE loss is to compare the pos/neg scores in a log-softmax manner, so visualizing the input score pairs can help study the contrastive learning process. (2) The normalized feature vectors $z_{q}$ and $z_{k}$ are high-dimensional, which is challenging for storage and visualization; The exponential amplification of scores is too large to observe the details of characteristics of pos/neg scores. However, $S_{q\cdot k}$ is one-dimensional and limited to $[-1,1]$, which is suitable to observe inside the contrastive process.

Notice that this practical visualization tool is offline and doesn’t affect training speed with negligible computation. Even with larger datasets and batch size, it’s still feasible. The details of the visualization tool are present in Supp A.

We choose the computationally-efficient model, MoCo [14] as an example to demonstrate our visualization design.

In the following sections, we provide thorough experiments and visualization analysis to show how the parameter $m$ affects the contrastive learning process. We attempt various $m$ for MoCo on ImageNet-100 (denoted as IN-100) [41] with linear readout protocol for evaluation (details in Supp B). As the Tab 1 shown, with the decrease of $m$ (increasing the update speed of encoder $f_{k}$), the accuracy presents an inverse U-shape and the max $56.24\%$ locates at $m=0.99$ and the model collapse³³3Model collapse means that the transfer accuracy with linear readout protocol can not achieve the accuracy of training from random initialization, i.e., $15.90\%$, indicating the negative effect brought by pre-train. when $m\leq 0.5$. The trend of these results is similar with BYOL [12].

We choose three non-trivial statistics to visualize the score distribution: the mean of pos/neg scores (indicating the approximate average of the pos/neg pair distance) and the variance of negative scores (indicating the fluctuation degree of the negative samples in the memory queue). As shown in Fig 3(a), when $m$ becomes smaller, the update speed of encoder $k$ is increasing, leading to incremental differences of features among training steps, which is reflected as the growing variance of negative scores of the queue, namely the inconsistency. Specifically, when $m=1$ (no update of $f_{k}$ during training), the variance is closed to zero (blue line) while the variance of $m=0.9$ (red) is larger but relatively unstable. $m=0.5$ (grey) brings more violent fluctuations/inconsistency in the memory queue, leading to a poor transfer accuracy even model collapse.

However, in Fig 5(c), when we increase $m$ from $0.9$ (green) to $0.99$ (orange), the easy pos pair becomes hard pos pair (from very similar $~{}0.9$ to less similar $~{}0.7$), leading to a higher transfer accuracy ($46.5\%$ v.s $56.2\%$, $9.7\%$ increased). Note that this observation (converting easy positive to hard one) could be explained by InfoMin principle [42]: Raising the view variance between $z_{q}$ and $z_{k^{+}}$ corresponds to increasing the mutual information for contrastive learning, which forces the encoder learns a more robust embedding and thus improves the transfer accuracy.

In the guarantee of stable and smooth score distribution and gradient, we can adopt some feature transformation methods which create hard ones by decreasing easy positive scores. Thus, we propose a positive feature extrapolation method to improve transfer accuracy in section 4.1.

Following the discussion in Sec 3.3 which indicates that lowering the easy positive pair scores to create hard positive pairs during training could be beneficial for the final transfer performance. Thus we would like to explore a way to manipulate the positive features $z_{q}$ and $z_{k^{+}}$ to increase the view variance between them during training.

First, we simply adopt weighted addition for the two positive features to generate new feature:

where $\hat{z}_{q}$ and $\hat{z}_{k^{+}}$ are the transformed new features. Meanwhile, considering the design principle of mixup [44, 58], we make sure that the summation of weights equals to $1$. More importantly, we should guarantee than the transformed pos score $\hat{S}_{q\cdot k^{+}}$ is smaller than the original pos score $S_{q\cdot k^{+}}$, namely $\hat{z}_{q}\hat{z}_{k^{+}}\leq z_{q}z_{k^{+}}$. Take Equation 3 into the transformed score:

Because $S_{q\cdot k^{+}}\in[-1,1]$ and thus $(1-S_{q\cdot k^{+}})\geq 0$. To make sure the lower score $\hat{S}_{q\cdot k^{+}}\leq S_{q\cdot k^{+}}$, we need to set $\lambda_{ex}\geq 1$ to let $2\cdot\lambda_{ex}(1-\lambda_{ex})\leq 0$. So we choose $\lambda_{ex}\sim Beta(\alpha_{ex},\alpha_{ex})+1$ ⁴⁴4We choose to set the two parameter $\alpha_{ex}$ of the beta distribution to be the same, because the two mixed features are symmetrical. And the same applies to the negative feature interpolation. is sampled from a beta distribution and then adding $1$ results in a range of $(1,2)$. And the range of transformed pos score will be $\hat{S}_{q\cdot k^{+}}\in[-4+5S_{q\cdot k^{+}},S_{q\cdot k^{+}}]$.

Intuitively, it can be seemed as a simple approach to push away $z_{q}$ and $z_{k^{+}}$ in feature space. After extrapolation, the distance between the extrapolated feature vector is enlarged. Therefore the extrapolation can serve as a feature transformation to create hard positives from easy ones. As shown in Fig 6(b), it brings a minor direction change for two positive vectors and meanwhile conveying a larger view variance of a sample for better contrastive learning. The visualization of lowering pos score by extrapolation is shown in Fig 1(c).

We evaluate the efficacy of positive extrapolation on IN-100 and attempt various $\alpha_{ex}$ in Tab 2. The positive extrapolation with various $\alpha_{ex}$ consistently improves the accuracy from the baseline MoCo ($71.1\%$), which clearly demonstrates the efficacy of positive extrapolation. It is interesting that $\alpha_{ex}\textgreater 1$ will get better results than those of $\alpha_{ex}\textless 1$. Because the beta distribution with $\alpha_{ex}\textless 1$ provides extreme large or small $\lambda_{ex}$ with high probability, e.g., $1.1$ or $1.9$, while the beta distribution with $\alpha_{ex}\textgreater 1$ gives neutral $\lambda_{ex}=1.5$ with high probability ⁵⁵5The beta distribution with $\alpha_{ex}\textgreater 1$ shows an inverted U shape which samples 0.5 with a greater probability and thus making $\lambda_{ex}$ to have a greater chance to be $1.5$.. According to Equation 4, extreme $\lambda_{ex}$ will bring too much/little hardness, so the corresponding performance is not robust as the neutral one.

Previous contrastive models [6, 14] do not make full use of negative samples. e.g., In MoCo, there are many repetitive negative features stores in the memory queue iteration by iteration. Thus we could design a new strategy to fully utilize negative features and increase the diversity of the memory queue. With sufficient randomness, we propose the negative interpolation in memory queue, which intuitively provides diversified negatives for each training step.

Specifically, we denote the negative memory queue of MoCo as $Z_{neg}=\{z_{1},z_{2},\dots,z_{K}\}$ where $K$ is the size of the memory queue, and $Z_{perm}$ as the random permutation of $Z_{neg}$. We propose to use a simple interpolation between two memory queue to create a new queue $\hat{Z}_{neg}=\{\hat{z}_{1},\hat{z}_{2},\dots,\hat{z}_{K}\}$:

where $\lambda_{in}\sim Beta(\alpha_{in},\alpha_{in})$ is in the range of $(0,1)$, as Fig 6(a) shown. The transformed memory queue $\hat{Z}_{neg}$ provides fresh interpolated negatives for contrastive loss iteration by iteration, where the random permutation and $\lambda_{in}$ ensure the diversity of $\hat{Z}_{neg}$ of each training step. The diversity makes the model to compare with much more linear combinations of previous negatives in each training step. Positive extrapolation increases the view variance between two pos features while the negative interpolation similarly boosts the “sample variance” (diversity) of the memory queue. We conjecture that original queue $Z_{neg}$ provides discrete distribution of negative samples but our method can fill in the incomplete sample points of the distribution by random interpolation, leading to a more discriminative model We evaluate the efficacy of negative interpolation on IN-100 and attempt various $\alpha_{in}$ in Tab 4. The neg interpolation is fairly robust with various $\alpha_{in}$, with the improvement of $2.2\%$-$3.5\%$ from the baseline ($71.1\%$). More interesting discussions about negative feature transformation (hard negatives & negative extrapolation) are shown in Supp G.

Previous works have explored the method leveraging image-level [38] and feature-level [20] mixing in contrastive learning. Our method differs from the previous works in three ways, first is the motivation, we are motivated by our observation in Sec 3.2 to propose the feature transformation strategies. Second, the way we extrapolate between two positive features is novel and outperforms the other two methods on several experiments in Tab 8 and 9. Third, the negative interpolation aims at fully utilizing negative samples in each training step. Both FT methods focus on exploring an effective way to perform feature transformation, not simply extending hard negatives to memory queue [20], neither the image-level mixup [38]. In the following sections, we provide inside discussions for the proposed FT, including (1) What if extending memory queue instead of FT. (2) When to add FT? (3) Dimension-level mixing rather than linear mixup. (4) Could the gains brought by FT vanish if training longer?

We compare the performance of using only the interpolated queue $\hat{Z}_{neg}$, original $Z_{neg}$ with $K$/$2K$ negative samples, and their combination $\tilde{Z}_{neg}$, in Tab 5. We found that using the combination queue shows negligible improvement over the performance ($74.73\%$) of using the interpolated queue alone ($74.64\%$). We consider that the interpolated negative features contain sufficient diversified negatives compared with the original queue. So even the double negative samples (more mutual information) of the extended queue ($\tilde{Z}_{neg}$) cannot boost the performance. Notably, the extended queue requires double times computation for each contrastive loss. Thus we recommend feature transformations with less computation but more efficacy rather than feature augmentation.

We adopt the linear readout protocol [14] to compare performance for image classification on IN-100, where we freeze the features and train a supervised linear classifier using softmax. Tab 7 summarizes the results of ablation studies. We observe that the positive extrapolation and negative interpolation components are complementary which can improve the top-1 accuracy by 5.77%/2.72$\%$ when combined on MoCoV1/MoCoV2. The dimension-level mix also shows improvement based on the already high performance of both components. The performance-boosting of ablation studies over MoCo shows the efficacy of our FT. Notice that the transformed features are not necessarily on the unit sphere (i.e., has a norm of 1), we did not need to re-perform $\ell_{2}$ norm for transformed features, because the performance difference is negligible ($76.87\%$ v.s post-norm $76.68\%$). More discussions about $\ell_{2}$ for vector length are put in Supp F. Here we strongly recommend to re-perform $\ell_{2}$ norm for the transformed features on all the datasets, for the sake of contrasting all the scores on the unit-sphere.

We apply our FT to various contrastive models in Tab 7. It presents that our FT brings $5.77\%$, $3.93\%$ , $1.3\%$, $1.1\%$, and $0.7\%$ improvement over MoCo [14], SimCLR [6], InfoMin [42], SWAV [4] and SimSiam [8], respectively on IN-100 dataset (200 epoch). It is worthy to point out that the series of ablation studies of our FT can boosts the SimCLR model. The experiments shows our FT is generic and robust for various contrastive models.

After ablations on IN-100 dataset, we use the best settings of $\alpha_{in}$ and $\alpha_{ex}$ to train a model on ImageNet-1k (IN-1K). Note that the dimension-level mix is not used for the experiments on IN-1K due to computational constraints. We apply our method on the baseline MoCo [14] and MoCoV2 [7], which are both trained on IN-1K with 200 epochs. The results and comparison are summarized in Tab 8. Our method improves MoCoV1 and MoCoV2 by 1.3% and 2.1% on Top-1 accuracy respectively which are significant on a large dataset like IN-1K. UnMix [38] and MoCHi [20] are the methods that also leverage mixup to better aid the contrastive learning process. Notably, we can observe that our method with MoCoV2 can provide larger performance gain than UnMix and MoCHi respectively.