Improving Vision Transformers for
Incremental Learning

Slow convergence when the number of classes is small: As ViT is significantly behind ResNet-18 for the first 50 classes, a reasonable guess is that ViT model is not fully trained even though it consumes longer training. As shown in Fig. 4, ViT-ti takes more than 2000 training epochs to achieve saturated performance. In contrast, ResNet usually is trained for only 90 epochs. When we increase training length from 400 to 800 epochs, it results in significant performance boosts from $74.72\%$ to $82.28\%$ (shown in Fig. 4). However, it is still lower than accuracy of ResNet-18. Although previous works [13, 33] report that ViT models require longer training schedule (300-400 epochs) to achieve saturated performance, this issue becomes even more severe when the number of classes is small, which is typical in the beginning of incremental leaning. As shown in Fig. 4, ViT model requires more training epochs to achieve saturated performance on 10 classes than on 50 classes. In addition, the initial model is crucial for the performance of the following incremental steps. As shown in Fig. 2(a), when the initial model is improved by training for 800 epochs, the average accuracy of all incremental steps is also significantly improved from $60.02\%$ to $67.11\%$.

More bias towards new classes: We also compare ViT and ResNet on confusion matrices in Fig. 3, where clearly more bias towards new classes is observed in ViT (highlighted in a red box). Quantitatively, compared to ResNet-18, ViT-ti has 9.3% more testing samples from old classes are falsely classified as new classes (11.2% vs. 20.5%). We believe this is caused by the conflict between the margin ranking loss and the data augmentation like Mixup or CutMix. Margin ranking loss [17] is an important technique to prevent bias, but requires hard class label per sample. Such hard class label is not available in ViT as it heavily relies on augmentation like Mixup or CutMix that generates soft (or mixed) label. In contrast, ResNet is not affected as strong augmentation is not necessary. Note that to avoid bias caused by long training schedule of ViT, we have already reduced training length for incremental steps $t>1$. More details about training setting are described in Sec. 4.2.

Underfitted classifier when using the same learning rate to ViT feature extractor: Another clear difference between ResNet and ViT is observed in the magnitude of learned softmax temperature $\eta$ in Equ. 1: $\eta=34$ for ResNet vs. $\eta=10$ for ViT in the final model after the last incremental step. We hypothesize that (a) the magnitude of $\eta$ is correlated to the learning rate, and (b) the small learning rate $2.5\times 10^{-4}$ is good for ViT as feature extractor but too low for the classifier. To validate this, we conduct experiments by applying different learning rates ($2.5\times 10^{-4}$, $5\times 10^{-4}$, $2.5\times 10^{-3}$) for the classifier alone while keeping the learning rate of ViT feature extractor unchanged ($2.5\times 10^{-4}$). Experimental results validate our hypothesis. As shown in Fig. 6, $\eta$ is highly correlated to the learning rate. Fig. 6 shows increasing learning rate from $2.5\times 10^{-4}$ to $2.5\times 10^{-3}$ for classifier boosts both the final and average incremental accuracy significantly.

Convolutional stem for faster convergence:

Bias correction via balanced finetuning: We correct the biases in classifier by finetuning it with balanced dataset, which is denoted as $\mathcal{X}^{\prime}_{o}\cup\mathcal{X}^{\prime}_{n}$, where $\mathcal{X}^{\prime}_{o}$ and $\mathcal{X}^{\prime}_{n}$ are the exemplar set from old and new classes respectively. The idea of using balanced dataset to correct biases in the classifier was studied in [35], where a linear layer with two parameters is learned. However, it does not apply to our case, since our classifier has normalized weights per class. Instead, we finetune all parameters in the classifier. As shown in Fig. 3(c), this finetuning strategy effectively mitigates bias towards new classes for using ViT model. This results in a two-stage incremental learning. In the first stage, parameters in both feature extractor and classifier (denoted as $\Theta_{t}$ for incremental step $t$) are updated. The second stage deals with balance finetuning of classifier where only the parameters in classifier (denoted as $\Theta^{g}_{t}$) are finetuned, with feature extractor frozen.

Large learning rate for classifier: It is straightforward to address the underfitting of classifier, by using a larger learning rate for it. In this paper, we use learning rate $2.5\times 10^{-3}$ on ImageNet dataset, which is ten times higher than the learning rate for the backbone feature extractor.

ViT architecture: We follow the design of ViT models in [13], and adopt ViT-ti as the base model for ImageNet dataset. Existing CIL works apply ResNet-18 as base model for CIL on ImageNet dataset. ViT-ti has 5M parameters and 1.1G FLOPs, less than ResNet-18 with 12M parameters and 1.8G FLOPs. CLS token of final transformer block is treated as the feature output of backbone ViT model, and fed into classifier head, which is cosine linear layer in our case. For CIL on CIFAR-100 dataset, existing works apply a modified 32-layer ResNet with 0.47M parameters, 69.8M FLOPs. For a fair comparison, we modify our ViT model with embedding dimension of 120, depth of 4, mlp ratio of 1, and patch size of 4. This lightweight model contains 0.47M parameters, and 41.0M FLOPs, with input image resolution 32$\times$32 same as existing works.

Convolutional stem architecture: Follow [36], we replace the patchify stem and the first transformer block in ViT-ti model with a small convolutional network. It consists of 4 convolution layers with kernel size $3$ and stride size $2$. The number of output channels is [24, 48, 96, 192]. Each convolution layer is followed by a batch norm layer and a ReLU layer. As pointed out in [36], convolutional stem has negligible impact on the model’s FLOPs.

Loss: Our loss function, $L=\sum_{x\in\mathcal{X}^{\prime}_{o}\cup\mathcal{X}_{n}}(L_{ce}(x)+\lambda L_{dis}(x))$, includes a standard cross entropy loss $L_{ce}$, and distillation loss $L_{dist}$, where $\lambda$ is a balancing factor. Same as in [17], $L_{dis}$ penalizes the change of feature. $L_{dis}(x)=1-\left\langle\bar{f}^{\ast}(x),\bar{f}(x)\right\rangle$, where $f^{\ast}(x)$ represents feature vector extracted by old model. In the finetuning stage, only cross entropy loss $L_{ce}$ is calculated on new class exemplar set $\mathcal{X}_{n}^{\prime}$ and old class exemplar set $\mathcal{X}_{o}^{\prime}$. With feature extractor frozen, only cosine linear classifier parameters are updated. $\lambda$ is set to 3. Same as [17], we use adaptive $\lambda$.

Augmentation: We adopt the same augmentation recipe in [36] for ImageNet dataset, including Mixup, CutMix, soft label, and AutoAugmentation. All images are re-scaled to 224$\times$224. For CIFAR-100 dataset, we use same data augmentation with images re-scaled to 32$\times$32.

Optimizer: Note that in all experiments, all ViT-based models are trained from scratch, without using any pre-training or extra dataset. We adopt commonly used AdamW optimizer. Batch size is set to 1024. Weight decay is set to 0.24. Learning rate is set to $2.5\times 10^{-4}$ for feature extractor, and $2.5\times 10^{-3}$ for classifier. Learning rate is scaled based on batch size as $lr\times$BatchSize/512. Cosine learning rate scheduler is applied with 5 warmup epochs. For the initial step $t=1$, the number of training epochs is set to $800$. For the following incremental steps $t>1$, number of training epochs is set to $50$ for B50/B500 settings, and $200$ for B0 setting. This is because when the incremental learning starts with half of total classes, learned feature representation is more robust and can better generalize to the other half classes. For CIFAR-100 dataset, since it contains less training data, number of training epochs is set to $2000$ for initial step $t=1$, and $500$ for following incremental steps $t>1$. Batch size is set to 512. Learning rate is set to $2.5\times 10^{-3}$ for feature extractor, and $5\times 10^{-3}$ for classifier.

Datasets: Our experiments are conducted on three widely used CIL evaluation datasets, ImageNet-1000 [10], ImageNet-100 and CIFAR-100 [21]. ImageNet-1000 is a large-scale dataset with 1000 classes. ImageNet-100 is a subset of ImageNet-1000 with 100 classes. Same as in [17], we use random seed 1993 to shuffle 1000 classes and choose the first 100 classes as ImageNet-100. CIFAR-100 is a dataset with small image resolution $32\times 32$.

Protocols: CIL protocols in existing works are different on three main factors: number of classes in the initial step, number of new classes each incremental step adds, and the number of exemplars. For simplicity, we use abbreviated notation of CIL protocol as in Fig. 1 (b). Existing works mainly adopt three CIL protocols. For example, (1) B500 R20, (2) B0 R20, (3) B0 T20K on ImageNet-1000 dataset. B500 denotes initial step contains half of total 1000 classes. B0 denotes model starts from scratch, and each step adds same number of new classes. R20 denotes each old class keeps same number of 20 exemplars. T20K denotes each incremental step keeps total 20000 exemplars. These protocols are evaluated with different numbers of new classes each incremental step adds, e.g., C100 denotes each incremental step adds 100 new classes. For ImageNet-100 and CIFAR-100, B500 is changed to B50, and T20K is changed to T2K. We conduct experiments on C2, C5 and C10 settings for CIFAR-100, C10 and C5 settings for ImageNet-100, C100 setting for ImageNet-1000.

Baselines: In our experiments, we compare with 5 baselines, including iCaRL [31], LUCIR [17], BiC [35], PODNet [14], and DDE [18]. Note that these baselines may only provide results on part of the three protocols in their original papers. Here, we report their reported results for the protocols they have evaluated. For the protocols they do not evaluate, we use either their official code release, or our own implementation. BiC [35] is evaluated with our PyTorch implementation. Note that in the released code of PODNet [14] for ImageNet, stride size of first convolution layer in ResNet-18 is changed from 2 to 1, which results in larger spatial size of every convolution feature map and much larger FLOPs (6.9G FLOPs vs 1.8G FLOPs). To investigate this issue, we evaluated it with two settings. In the first one, we adopt official ResNet-18 setting for feature extractor. In the second one, we use the modified ResNet same as in [14]. Our method is also evaluated with these two settings.

ImageNet-100: The quantitative results are shown in Tab. 1. The incremental accuracy of each incremental learning step is shown in Fig. 7. Compared with CNN counter part, LUCIR [17], the proposed method outperforms by $8.96\%$, $14.35\%$, $11.07\%$, $8.83\%$, $15.85\%$, $10.37\%$ for 6 CIL settings on ImageNet-100, respectively. Note that LUCIR, PODNet, BiC only perform well on partial CIL protocols. In contrast, our method consistently outperforms existing methods on all CIL protocols. Moreover, our accuracy degradation slope is flatter than CNN counterpart, LUCIR, indicating a better ability to address forgetting. For example, with B50 C10 R20 in Fig. 7(a), accuracy improvement between ours and LUCIR is $4.94\%$ for the initial step, and $12.68\%$ for the final incremental step.

ImageNet-1000: As shown in Tab. 1, the results on ImageNet-1000 is consistent with those on ImageNet-100. With more classes, the catastrophic forgetting of ViT models is more severe compared with ImageNet-100. The proposed method outperforms state-of-the-art methods by at least $1.69\%$ with B500 setting, by at least $7.27\%$ on B0 R20 setting, and by at least $6.28\%$ with B0 T20K setting. Compared with CNN counterpart, baseline LUCIR, our improvements on three CIL protocols are $4.86\%$, $7.99\%$, and $8.10\%$, respectively.

CIFAR-100: The results are shown in Tab. 2. Our method outperforms baselines with clear margin, using same input resolution 32$\times$32. Moreover, our method has 0.47M parameters and 41.0M FLOPs, lower than 0.47M parameters 69.8M FLOPs of commonly adopted ResNet-32 model in existing works. Note that our model is trained from scratch, without using any pre-training or extra dataset.

Model with larger FLOPs: Here we change the first convolution layer of our method to stride size 1, and obtain a model with 6.9G FLOPs. In the released code of PODNet [14], the first convolution layer of ResNet-18 is modified with kernel size of 3 and stride size of 1. With this modification, PODNet has 6.9G FLOPs. For a fair comparison, we compare with our model with 6.9G FLOPs. The results are shown in Tab. 4. Our method still outperforms PODNet consistently on ImageNet-100, ImageNet-1000 with all CIL protocols.

Convolutional stem: As shown in Tab. 4, without convolutional stem, the naive ViT model’s performance is significantly lower than its ResNet counterpart, LUCIR. Moreover, with the same setting of classifier learning rate and bias correction, the models without CNN stem perform significantly worse than the models with CNN stem. This is mainly due to the inferior optimizability of ViT models, which requires much longer training schedule.

Bias correction: As shown in Tab. 4, without bias correction, the average incremental accuracy is much lower, especially for the protocol with more incremental steps with B0 protocols. This is because bias towards new classes is more severe when there are more incremental steps.

Large classifier learning rate: As shown in Tab. 4, when the model has bias correction, larger classifier learning rate further improves average incremental accuracy, which is caused by underfitted classifier parameters. Moreover, it also improves ViT models without CNN stem. In particular, without CNN stem, the initial step accuracy of ViT-ti is slightly below LUCIR on B50 C10 R20 setting. However, with bias correction and large classifier lr, it still outperforms LUCIR on average incremental accuracy by $3.66\%$.

Average incremental forgetting: As shown in Tab. 5, forgetting of our method is significantly lower than CNN counterpart, LUCIR. Please note that lower forgetting does NOT necessarily lead to higher average accuracy, when initial accuracy is the same. For example, in Tab. 5 B50 R20, forgetting of BiC is lower than LUCIR. However, as in Fig. 7(a), average incremental accuracy of BiC is lower than LUCIR, and their initial step accuracy is very close. This is because average forgetting only measures model stability, while CIL pursues a trade-off between stability and plasticity. A trivial solution, with model frozen after initial step, can achieve 0 forgetting, while not handling new classes.