Memory-based Jitter: Improving Visual Recognition on Long-tailed Data
with Diversity In Memory

MBJ has a novel re-balancing strategy, compared with prior works on long-tailed visual recognition. Generally, re-balancing aims to highlight the tail classes during training. In prior works, there are two major re-balancing types, i.e., re-weighting[11, 40, 6] and re-sampling[24, 46, 2, 3]. Re-weighting strategy allocates larger weights to tail classes in loss function. Re-sampling over-samples the raw images of the tail classes for training.

Different from these prior works, MBJ re-samples the features / prototypes to highlight the tail classes. It thus avoids directly re-sampling the raw data. Since directly re-sampling the raw data actually compromises the deep embedding learning [48, 12], avoiding such operation substantially benefits MBJ. An ablation study carried out on the long-tailed CIFAR-10 dataset shows that when we completely remove the jitter augmentation, this novel re-sampling strategy still brings $+2.1\%$ improvement over the baseline. The details are to be accessed in Section 4.5.2.

The memory bank plays a critical role in MBJ. Both the weight jitters and the feature jitters are scattered among sequential training iterations. To accumulate these jitters for tail data augmentation, we employ a memory bank. Since memory-based learning has been explored in several computer vision domains, including unsupervised learning, semi-supervised learning and supervised learning [9, 31, 14, 39, 23, 49], we make a detailed comparison as follows.

In unsupervised learning, [9] employs memory to include more data in the dictionary. It shows that larger optimization scope within a optimization step is beneficial for unsupervised learning [9]. In semi-supervised learning, [14, 31] enforce consistency between historical predictions. Such consistency offers auxiliary supervision for the unlabeled data. In supervised deep metric learning, [39] uses memory to enhance the hard mining effect. Regardless of their objectives of using memory, they all hold a negative attitude towards the jitters. [9] and [14, 31] suppress the jitters with momentum and consistency constraint, respectively. [39] tries to avoid the jitters by delaying the injection of memory.

In contrast to their negative attitude towards jitters, we find that the jitters are informative for long-tailed visual recognition. As a major contribution of this work, we analyze the mechanism in Section 3.1 and experimentally validate its effectiveness in Section 4.5.2.

Moreover, we notice a recent work IEM [50] also employs memory for long-tailed image classification. We compare MBJ against IEM in details for clarity. Our method significantly differs from IEM in three aspects, i.e., the applied task, the mechanism and the achieved performance. First, IEM is specified for image classification, while MBJ improves both image classification and deep metric learning with a uniform motivation. Second, IEM considers tail classes are harder to recognize, and thus employ more prototypes from the memory for higher redundancy, while MBJ employs the jitters in memory to augment the diversity of tail data. Finally, on image classification task, MBJ maintains competitive performance with significantly higher computing efficiency. IEM requires extraordinary large amount of memory (up to $50,000$ per class), and achieves Top-1 accuracy of $67\%$ on iNaturelist18. In contrast, MBJ is more memory-efficient and more accuracy. For example, on iNaturelist18 [34], MBJ only stores $40,000$ memorized features in total and achieves Top-1 accuracy of $66.9\%$ and $70.0\%$ at $90$ and $200$ epochs, respectively.

With these comparison, we find that MBJ is featured for the memory-based feature augmentation. It is orthogonal to many prior works. Specifically, we note a very recent work RIDE [38] using multiple classifiers (experts) ensemble to improve the accuracy of head and tail classes, simultaneously. MBJ can be integrated into RIDE [38] for better performance gains.

To illustrate the phenomena of weight jitters and feature jitters, we conduct a toy experiment on CIFAR-10 [13]. We set a specified class to contain very limited samples (i.e., 50 samples) so that it turns into a tail class. We train a deep classification model to convergence and then continue the training for observation purpose. Within the following iterations, we record two objects, i.e., 1) the prototype (i.e., the weight vector in the classification layer) of the tail class and 2) the feature of a single tail sample. As the training iterates, both the prototypes and the features accumulate, allowing a quantitative statistic on their variances. We visualize the geometrical angular variance of the accumulated features / prototypes in Fig. 2, from which we draw two observations.

First, we observe considerable variance among the accumulated weight vectors (i.e., prototypes), as well as the accumulated features. It indicates that among multiple iterations, both the prototype of a single class and the feature of a single image keep on changing itself, yielding the phenomena of weight jitters and feature jitters, respectively.

Second, we observe that the above-described jitters require certain training iterations to accumulate before they reach stable status. When there is only one single feature / prototype, the corresponding variance is naturally $0$. As the number of accumulated features / prototypes increases, the variance gradually grows until reaching a stable level.

Based on the above observation, we device MBJ. MBJ uses a memory bank to collect the historical features / prototypes, so as to accumulate the feature / weight jitters for tail augmentation.

The pipeline of MBJ for deep image classification is illustrated in Fig.3. In the raw image space, MBJ randomly samples the images without re-balancing the head and tail classes. In each training iteration, the deep model transforms the raw images into a batch of features. Given the features in current mini-batch, MBJ stores them into a memory bank. The memory bank has a larger size than the mini-batch, so it is capable to accumulate historical features across multiple training iterations.

When collecting the features, MBJ lays emphasis on the tail classes, so that the tail and head data will be re-balanced. Specifically, MBJ assigns small sampling probabilities to head classes, as well as relatively large sampling probabilities to tail classes, which is formulated as:

where $P_{i}$ is the corresponding sampling probability of class $i$, $N_{i}$ is the sample number of the $i$-th class, $C$ is the total number of classes, and $\beta$ is a hyper-parameter to control the strength of re-balancing. A larger $\beta$ results in higher priority on accumulating the tail features. We use $\beta=1.5$ in all of our experiments.

To control the memory size, we use a queue strategy for updating the memory bank. After the memory bank reaches its size limitation, we enqueue the newest features (i.e., the features in current mini-batch), and dequeue the oldest ones.

Given the feature memory and the batch features, MBJ combines both of them to learn the classifier in a joint optimization manner. Specifically, MBJ uses the features in current mini-batch and the weight vectors in the classification layer to deduce a cross-entropy loss, i.e., the loss $\mathcal{L}_{batch}$. Meanwhile, MBJ uses the memorized features and the weight vectors in the classification layer to deduce another cross-entropy loss, i.e., the loss $\mathcal{L}_{memory}$. MBJ sums up those two losses by:

in which $\eta$ is a weighting factor. Algorithm 1 provides the pseudo-code of MBJ for deep image classification task.

A popular baseline [21, 17, 26, 25, 18, 27, 28] for deep metric learning is as follows: during training, we learn a classification model on the training set. The weight vectors in the classification layer are typically recognized as prototypes for each class. During testing, the distance between two images are measured under the learned deep embedding. Based on this baseline, MBJ collects the historical prototypes into a prototype memory with emphasis on the tail classes.

The sampling strategy is exactly the same as in Eq.1, so we omit the detailed description. Given the historical prototypes in memory and the up-to-date prototypes, MBJ combines both to learn the features. For clarity, we illustrate the learning process with focus on a single feature $x$ under optimization.

To learn with the up-to-date prototypes $W=\{w_{1},w_{2},\cdots,w_{C}\}$ ($C$ is the total number of training classes), MBJ adopts a popular deep metric learning method, i.e., CosFace [36], which is formulated as:

in which $C$ is the number of classes, $w_{y}$ is the prototype of the target class, $\alpha$ is a scale factor and $\delta$ is a margin for better similarity separation.

To learn with the prototype memory, MBJ needs to deal with multiple positive prototypes associated with feature $x$. It is because the weight vector of the target class in the classifier may be sampled at several training iterations. Let us assume there are $K$ positive prototypes $\{u_{1},u_{2},\cdots,u_{K}\}$(i.e., weight vectors of the target class), and $L$ negative prototypes $\{v_{1},v_{2},\cdots,v_{L}\}$ (i.e., weight vectors of the non-target class). We find that a recent deep metric learning method, i.e., Circle Loss[26], allows multiple similarities associated with a single sample feature. Accordingly to Circle Loss, we define the loss function associated with the prototype memory by:

Similar to Eq.2, MBJ sums the above two losses (i.e., $\mathcal{L}_{batch}$ and $\mathcal{L}_{memory}$) to optimize the feature $x$. We note that two editions of MBJ share a unified framework, except for the jitter type. To improve the classification accuracy, MBJ memorizes the feature jitters; To improve the retrieval accuracy, MBJ memorizes the prototype jitters. They have a dual pattern against each other. Algorithm 2 provides the pseudo-code of MBJ for deep metric learning task.

MBJ is featured for its re-balancing strategy and its augmentation manner. Although re-balancing the features / prototypes (instead of re-balancing the raw data) considerably benefits MBJ (as introduced in Section 2.1), we note that the improvement is mainly because the accumulated jitters increase the tail data diversity. Removing the jitters or using other augmentation method significantly compromise MBJ. The details are to be accessed in Section 4.5.2.

For CIFAR datasets, we synthesize several long-tailed version, following [4]. We use an imbalance ratio to denote the ratio between sample size of the most frequent and least frequent class. Imbalanced ratio (IR) in our experiments are set to $10$, $50$ and $100$, respectively.

ImageNet-LT, Places-LT and iNaturalist18 are publicly available long-tailed dataset. In ImageNet-LT, the maximum of images per class is $1280$ and the minimum of images per class is $5$. In Places-LT, the largest class has $4980$ images while the smallest ones have $5$ images. Their test set is balanced. The iNaturalist18 dataset is a large-scale dataset with extremely imbalanced label distribution. It has $437,513$ images from $8,142$ classes. We adopt the official training and validation splits for our experiments.

We evaluate MBJ under a popular deep metric learning task (i.e., re-ID). We note that the long-tail problem on this task has been noticed recently, and the competing methods are relatively few. Table 4 compares MBJ with re-ID baseline (CosFace  [37]) and a state-of-the-art method (Feature Cloud  [17]), from which we draw two observations:

First, under typical long-tailed distribution, MBJ significantly improves the re-ID baseline. When there are only 20 head classes (“H20”), MBJ achieves $+11.1\%$ and $+10.9\%$ mAP on Market-1501 and DukeMTMC-reID, respectively.

Second, MBJ marginally surpasses the recent state-of-the-art, i.e., Feature Cloud [17]. For example, under three long-tailed condition on Market-1501, MBJ achieves $72.6\%$, $68.8\%$ and $66.7\%$ mAP, which are higher than Feature Cloud by $+3.9\%$, $+1.5\%$ and $+2.6\%$, respectively. MBJ obtain the new state-of-the-art performance.