Unbiased and Efficient Self-Supervised Incremental Contrastive Learning

In order to eliminate the deviation caused by the change of the noise distribution in the learning of the old data, we first propose a metric for measuring the change ratio of the noise distribution.

To explore how noise distribution works in contrastive learning, we first rewrite the InfoNCE loss into the form of softmax-based categorical cross-entropy:

The softmax term in Eq. (4) represents the final prediction result of the model, in which the sum of predicted probabilities for all classes changes with the noise distribution, that is, $\sum_{x_{k}\in X}f(x_{i},x_{k}^{+})\to\sum_{x_{k}\in{X\cup\Delta X}}f(x_{i},x_{k}^{+})$. Therefore, we propose to use the ratio of the sum of predicted probabilities for new classes to the one for old classes.

With the proposed change ratio in Eq. (6), we can measure how much the noise distribution changes. If the new data maintain the same noise distribution as the old data (i.e., $p_{n}^{\Delta X}=p_{n}^{X}$), the change ratio $r_{i}^{X\to\Delta X}$ equals $1$. The deviation of $r_{i}^{X\to\Delta X}$ from 1 reflects the degree of change of noise distribution.

Next, we design a novel contrastive loss function, leveraging the change ratio of the noise distribution.

In NCE-II loss, the change ratio for each sample $r_{i}^{\Delta X\to X}$ is weighted by a coefficient $\alpha$, which reflects the increment of the data. Specifically, when there is no new data (i.e., $\alpha=0$) or the noise distribution remains the same (i.e., $r_{i,\Delta X\to X}$=1), the NCE-II equals $0$, which is in line with the intuition.

Given NCE-II, the encoder is trained on the old data $X$ with two contrastive loss functions (Eq.(2) and Eq.(8)), the sum of which is equivalent to the loss function of retraining on all data $X^{\prime}$.

Finally, the objective function of ICL is defined as

which is an unbiased estimation of both old data and new data w.r.t. retraining. In Table 2, we provide the comparison of bias between different strategies to show the superiority of the proposed method.

Although the proposed loss function NCE-II is equal to the difference of InfoNCE with old data and all data, the final empirical risk $\mathcal{R}$ during the entire training process is nevertheless different due to the change of data distribution. Therefore, we provide the bound analysis to guarantee the correctness of the proposed NCE-II.

Given the number of epochs $\Delta T$ in the incremental learning process, the time complexity of the proposed method is $\mathcal{O}(\Delta T((2K+1)N+(K+1)\Delta N)C)$, where $C$ represents for the time consuming of the encoder and function $f(\cdot,\cdot)$. As the parameters of the encoder have been optimized, the number of epochs is much smaller than the one of the retraining process. We further guarantee it in the next section.

For plasticity, we treat the optimization of ICL for new data as transfer learning (the new data may contain new classes or from different domains) and propose a meta-optimization strategy for fast adaption, which is formulated as a form of model agnostic meta-learning (MAML) (Finn et al., 2017).

We treat the old data $X$ as the support set and the new data $\Delta X$ as the query set. In the meta-training stage, we calculate the loss on the support set by Eq.(8) and gain the new parameters $\theta^{\prime}$ in a few gradient descent steps:

where $lr_{s}$ is the learning rate of the meta-training process on the support set $X$. For fairness, the number of steps is set to $\max\{\lceil(1-\alpha)/\alpha\rceil,1\}$ to ensure that the total number of samples trained in one epoch and the size of all data are as equal as possible, under the premise that there is at least one support sample for each query.

In the meta-testing stage, after obtaining the adapted parameters $\theta^{\prime}$, we update the parameters $\theta$ on the query set with

where $lr_{q}$ is the learning rate of the meta-testing process on the query set $\Delta X$.

The learning rate for gradient descent is commonly set as a hyperparameter and needs to be manually searched for the optimal value. However, it is a challenge to determine the values of two learning rates ($lr_{q}$ and $lr_{s}$) in our proposed meta-optimization. It is (a) unreasonable to simply set the two values to the same or use the previous one because the two learning rates are used for different training processes and data, (b) hard to search for the optimal value due to the large searching space, and (c) the change rules of learning rates should be automatically learned according to the status of the training environment instead of applying the human-designed ones. Therefore, we aim to find a solution that can adaptively control learning rates while speeding up the convergence process.

We proposed a Learning rate Learning (LRL) mechanism to learn the two learning rates ($lr_{s}$ and $lr_{q}$) adaptively according to the current state of the encoder in a reinforcement learning approach (Kaelbling et al., 1996; Xu et al., 2017; Li et al., 2021b). In our reinforcement learning setting, we propose to use a function $\mathcal{F}$ to generate the state $s_{t}$ which encodes the observation of the training process (the inputs $X$ and the parameters $\theta_{t}$) at the time step $t$:

The action $a_{t}$ is defined as the learned learning rate based on the state $s_{t}$ and $a_{t}\in\mathbb{R}$ is a continuous value.

To resolve the continuous action space issue, (Xu et al., 2017) uses an actor-critic algorithm (Sutton et al., 1999; Silver et al., 2014) to generate the learning rate, employing two neural networks: the actor network for learning rate generation and the critic network for criticizing the actions. However, because the inputs of the actor network are ordered and related (i.e., not i.i.d.), the reinforcement learning process is therefore unstable and problematic (Mnih et al., 2013). Therefore, we propose to use the Deep Deterministic Policy Gradient (DDPG) to learn the two learning rates respectively, which contains the experience replay mechanism to resolve the i.i.d. issue and the separate target network mechanism for a stable training (Lillicrap et al., 2016).

In general, DDPG networks $\mathcal{D}=\{\mu,Q,\mu^{\prime},Q^{\prime}\}$ consist of the (online) actor network $\mu(s|\theta^{\mu})$ and (online) critic network $Q(s,a|\theta^{Q})$, and the target actor network $\mu^{\prime}$ and the target critic network $Q^{\prime}$ with the same initial weights as the online ones. We first select an action $a_{t}$ in step $t$ as

where $\mathcal{N}$ is the Ornstein-Uhlenbeck (OU) process (Uhlenbeck and Ornstein, 1930) to generate noise for action exploration. Next, the action $a_{t}$ is executed (updating encoder’s parameters with $a_{t}$ as the learning rate) and the new state $s_{t+1}$ and reward $r_{t}$ are observed. As the aim of LRL is to accelerate the convergence, we define the reward $r_{t}$ as the loss decrement:

Finally, we optimize the the critic network using Temporal-Difference (TD) learning with a replay buffer and actor network with the chain rule (Lillicrap et al., 2016). The target networks are updated by a momentum mechanism. Please refer to the online repositories for more details.

We use four benchmark datasets across computer vision (CV) and graph representation learning (GRL). For CV, we use the following two datasets: (a) ImageNet is a benchmark dataset consisting of around 1.28 million images (Deng et al., 2009). In order to reduce the influence of additional factors for an accurate efficiency evaluation, we extract a subset with 2 classes, named ImageNet-2, to manage that the entire training process can be completed on 1 GPU within 6 hours. (b) MNIST is a handwritten digit dataset (LeCun et al., 1998). Similarly, we extract a subset with 2 classes, named MNIST-2. For GRL, we use the following two datasets: (a) PROTEINS is a bioinformatics dataset with 1,113 molecular graphs (Morris et al., 2020). (b) REDDIT is a social network dataset consisting of 2,000 graphs in 2 classes (Morris et al., 2020). We use Local Degree Profile (LDP) algorithm (Cai and Wang, 2018) to generate node features.

We compare four baselines including the simple strategies (retraining and fine-tuning) and commonly used techniques (replay and distillation).

• •

  Retraining uses both old data and new data to train a new encoder. Since the aim of this study is to find an approach for unbiased estimation, we first compare ICL with retraining which serves as the baseline of efficiency and the upper bound of effectiveness.

• •

  Fine-tuning updates the parameters of the pre-trained encoder using the new data.

Furthermore, since ICL is under the self-supervised setting which lacks related research, we survey the replay and distillation technique used in incremental learning (Rebuffi et al., 2017; Castro et al., 2018; Fang et al., 2020; Cha et al., 2021; Lin et al., 2021).

• •

  Replay uses the previously seen data (partially stored old data) and the new data to train the encoder for avoiding catastrophic forgetting. classes,

• •

  Distillation learns a more compact encoder from the old one to prevent over-drift of representations from previous data when learning new ones.

For fairness, we use the exact same network architecture for all methods.

We further introduce the following three variants of ICL to verify the effectiveness of the components (LRL and meta-optimization):

• •

  ICL without LRL mechanism (w/o LRL), i.e., with NCE-II and meta-optimization.

• •

  ICL without meta-optimization (w/o M), i.e., with NCE-II and LRL only generating one learning rate.

• •

  ICL without meta-optimization and LRL mechanism (w/o M+LRL), i.e., only with NCE-II.

We adopt a two-stage scheme (You et al., 2020; Li et al., 2021a, 2022b). In the training process, for fairness, all of the methods use the same single encoder to generate the representations. In the testing process, an extra SVM classifier (You et al., 2020) is trained with the fixed embedding for evaluation. In order to evaluate the efficiency of the methods more accurately and reduce the influence of external factors, we conduct each experiment on one single Nvidia V100 32G GPU for 5 times independently. We record the mean value as the final accuracy for each experiment and omit the standard deviations (all deviations are around 0.01). (a) The encoder we have used for CV is ResNet-18 (He et al., 2016) and a two-layer Graph Convolution Network (GCN) (Kipf and Welling, 2017) with 32 hidden units is used for GRL. (b) We choose {random resizing, 224×224-pixel cropping, random color jittering, random grayscale conversion, random horizontal flip} (Wu et al., 2018; He et al., 2020) as the data augmentation for images, and randomly sample one from {random node dropping, random node attribute masking, random subgraph selection} (You et al., 2020) for GRL. (c) The similarity measurement we have used is the cosine similarity function $sim(z_{i},z_{j})\coloneqq z_{i}\cdot z_{j}/||z_{i}||\cdot||z_{j}||$. (d) The state generated by function $\mathcal{F}$ in LRL is defined as the average loss of the current training process on the mini-batch data. (e) A two-layer LSTM with 20 hidden units is used as the actor network and a three-layer neural network (NN) with 10 hidden units is used as the critic network.

For common hyper-parameters, the number of negative samples $K=31$ (i.e., a batch size $bs=32$), the initial learning rate is searched from $lr\in\{10^{-3},10^{-4},10^{-5}\}$, the temperature parameters $\tau=0.1$, and the patience to wait for convergence is $50$ epochs (i.e., the process is terminated when the loss no longer drops for 50 epochs). For LRL, the momentum term $m=10^{-3}$.