Free: Faster and Better Data-Free Meta-Learning

DFML Setup. We are given only a collection of pre-trained models $\mathcal{M}_{pool}=\{M_{i}\}$, each designed to solve a specific task without accessing their training data. DFML aims to learn a meta-learner $F(\cdot;\boldsymbol{\theta})$ that can be rapidly transferred to new unseen tasks using pre-trained models $\mathcal{M}_{pool}$. During meta-testing, we sample “$N$-way $K$-shot” tasks. For such a task, there are $N$ classes and each class only has $K$ labeled samples $(\boldsymbol{S},\boldsymbol{Y_{S}})$ named the support set, and $U$ unlabelled samples per class $\boldsymbol{Q}$ called the query set, where $\boldsymbol{Y_{Q}}$ is the corresponding ground truth. We use the support set $(\boldsymbol{S},\boldsymbol{Y_{S}})$ to adapt the meta-learner $F(\cdot;\boldsymbol{\theta})$ to each specific task. The query set $\boldsymbol{Q}$ is what we actually need to predict.

Model inversion. Model inversion [10, 24, 48, 21] aims to recover the training data $\boldsymbol{\hat{X}}$ from the pre-trained model $M$ as an alternative to the inaccessible original data $\boldsymbol{X}$. So a generator $G(\cdot;\boldsymbol{\theta}_{G})$ can be introduced to estimate the distribution of $\boldsymbol{X}$ by designing an inversion loss with the pre-trained model $M$. Specifically, given a latent code $\boldsymbol{z}$ as a low-dimensional representation, the generator maps $\boldsymbol{z}$ to the intended data approximation $\boldsymbol{\hat{x}}$. In this case, the generator is only responsible for a small part of the data distribution. Compared with the pixel updating strategy used in [15] that updates different pixels independently, the generator can provide stronger regularization on pixels because they are produced from the shared weights.

Efficiency dilemma. Model inversion generally requires a generate-forward-backward computation for each optimization step, with the gradient back-propagation process being the most time-intensive [13, 31]. As the structure of the pre-trained model expands, or the scale of the generated data increases, the time and computational costs rise substantially. We quantitatively measure the GFLOPs required for the gradient back-propagation of a single image through different model architectures, as shown in Tab. 1. The total computation for inversion can be expressed as $N_{iter}\times N_{img}\times$ GFLOPs. Prior works optimize pixels individually or train a unique generator from scratch for each pre-trained model to reconstruct specific training distribution. Such pixel optimization or generator training often demands hundreds of generate-forward-backward computations. However, certain foundational features and textures are shared across different tasks, typically as the shallow layers of the generator [26]. If we train a meta-generator that retains common initialization parameters, rapid adaptation to specific tasks can be achieved in a few steps.

Model heterogeneity. Pre-trained models which contain random classes can be perceived as sub-domains with their diverse data distributions, due to statistical properties differences such as pixel intensity or texture variation [3, 35]. Diverse model architectures also contribute to model heterogeneity, influencing how data is represented. Moreover, our knowledge regarding the quality of pre-trained models is limited, as we have no data for evaluation. Given the chance that they stem from different domains, the tasks recovered from them inherently exhibit a distribution gap. We provide a $t$-SNE visualization to illustrate the gap among them in Fig. 5. Since neural networks tend to favor shortcuts and frequently learn simpler biases [12], a straightforward ERM across different tasks will cause the meta-learner to develop an undue bias towards specific tasks, limiting its generalization capabilities for more challenging unseen tasks. As shown in Tab. 5, applying ERM leads to a 3.04% decrease in our approach’s performance on the unseen test set.

Adaptive Task Recovery. The meta-generator $G(\cdot;\boldsymbol{\theta}_{G})$ takes the standard Gaussian noise $\boldsymbol{Z}$ as inputs and outputs the recovered data $\boldsymbol{\hat{X}}=G(\boldsymbol{Z};\boldsymbol{\theta}_{G})$. For a pre-trained model $M$, we aim to recover a pseudo task $\mathcal{T}=\{(\boldsymbol{\hat{X}},\boldsymbol{Y})\}$ via the meta-generator. In order to obtain the training distribution of the pre-trained model, we need to perform $k$ steps of rapid adaptation on the meta-generator. We utilize the DeepInversion loss [47] for updating $\boldsymbol{Z}$ and $\boldsymbol{\theta}_{G}$, which contains a classification loss to condition the recovered data on pre-defined target label $\boldsymbol{Y}$, and an adversarial loss to encourage data diversity by maximizing the KL divergence between the pre-trained model $M$ and the meta-learner $F(\cdot;\boldsymbol{\theta})$:

where $\eta=\mathbb{I}\{\mathop{\arg\max}M(\hat{\boldsymbol{X}})=\mathop{\arg\max}F(\hat{\boldsymbol{X}};\boldsymbol{\theta})\}$ and $\hat{\boldsymbol{X}}=G(\boldsymbol{Z};\boldsymbol{\theta}_{G})$. The function $\mathbb{I}(\cdot)$ is an indicator to enable $\hat{\boldsymbol{X}}$ with the same prediction from the pre-trained model and the meta-learner ($\eta$ = 1), otherwise disable it ($\eta$ = 0).

Meta-Generator Learning. In the inner loop, the adaptation process provides a sequence of losses $\mathcal{L}_{G}^{0},\mathcal{L}_{G}^{1},\ldots,\mathcal{L}_{G}^{k-1}$, i.e., losses on different data generated by the updating generator over $k$ steps. To allow the meta-generator to build an internal representation suitable for a wide range of pre-trained models, the outer loop attempts to make the task-specific parameters reachable within the $k$-step adaptation:

Multi-Task Knowledge Distillation. Upon obtaining the recovered tasks from pre-trained models, we propose executing knowledge distillation to transfer the task-specific knowledge from the pre-trained model (acting as the teacher) to the meta-learner (acting as the student) using the recovered task. We optimize the meta-learner by minimizing the disagreement of predictions between them. This approach is favored because knowledge distillation offers richer supervision. It leverages the semantic class relationships present in the soft-label predictions from the pre-trained model $M$, as opposed to solely relying on the hard-label supervision from the generated data $\boldsymbol{\hat{X}}$.

Implicit Gradient Alignment. To understand how it results in the desired alignment between different tasks, we can analyze the proposed objective in Eq. 3. We employ Taylor’s expansion to elucidate the connection between the gradient $\nabla_{\boldsymbol{\widetilde{\theta}}}\mathcal{L}_{KD}(\mathcal{T}_{i};\boldsymbol{\widetilde{\theta}})$ and gradient at initial point $\nabla_{\boldsymbol{\theta}}\mathcal{L}_{KD}(\mathcal{T}_{i};\boldsymbol{\theta})$, as described in Lemma 1.

Optimization. However, optimizing the meta-learner based on Eq. 3 requires high-order gradients when computing $\nabla_{\boldsymbol{\theta}}\mathcal{L}_{KD}(\mathcal{T}_{i};\boldsymbol{\widetilde{\theta}})$, making the back-propagation highly inefficient. We could apply a first-order approximation [23] to further accelerate the gradient computing. We continue to update $\boldsymbol{\widetilde{\theta}}$ using the gradient $\nabla_{\boldsymbol{\widetilde{\theta}}}\mathcal{L}_{KD}(\mathcal{T}_{i};\boldsymbol{\widetilde{\theta}})$. Then, the meta-learner $F(\cdot;\boldsymbol{\theta})$ is updated through a weighted difference between the resulted parameters $\boldsymbol{\widetilde{\theta}}$ and the original parameters $\boldsymbol{\theta}$ (cf. Algorithm 2).

Cross Task Replay. A small number of pre-trained models (e.g., 100) are insufficient to represent the actual underlying task distribution, leading to an over-reliance on specific tasks. Hence, we employ a memory bank $\mathcal{B}$ with a first-in-first-out structure to store the previously recovered tasks, and replay new tasks interpolated across different tasks, i.e., a random combination of classes. This further alleviates potential conflicts between tasks, assisting the meta-learner in generalizing across tasks. For these replayed tasks $\hat{\mathcal{T}}$, which include the support set $\boldsymbol{S}$, the query set $\boldsymbol{Q}$ and class labels $\boldsymbol{Y}$, we update the meta-learner as follows:

Datasets and pre-trained models. We conduct experiments on two widely-used DFML benchmark datasets, and one fine-grained dataset, including miniImageNet [29], CIFAR-FS [1] and CUB [37]. Following standard splits [43], we split each dataset into the meta-training, meta-validating and meta-testing subsets with disjoint label spaces. In the DFML setting, we have no access to the meta-training data. Following [39, 16], we collect 100 models pre-trained on 100 $N$-way tasks sampled from the meta-training subset, and those models are used as the meta-training resources.

Implementation details. For the model architecture, we adopt Conv4 as the architecture of the meta-learner and the pre-trained models for a fair comparison with existing works. We provide the detailed structure for the meta-generator in Appendix B. For hyperparameters, the batch size $b$ is set to 4, and the learning rate $\gamma$, $\alpha$, and $\epsilon$ are all set to 0.001. We report the average accuracy over 600 meta-testing tasks. We leave the other setup in Appendix A.

Baselines. (i) Random. Learn a classifier using the support set from scratch for each meta-testing task. (ii) Best-Model. We select the pre-trained model with the highest reported accuracy to directly predict the query set during meta-testing. (iii) Average. Average all pre-trained models and then finetune it using the support set. (iv) OTA [33]. Calculate the weighted average of all pre-trained models and then finetune it using the support set. (v) DRO [39]. Meta-learn a hyper-network to fuse all pre-trained models into one single model, which serves as the meta-initialization and can be adapted to each meta-testing task using the support set. (vi) PURER [15]. Adversarially train the meta-learner with a learnable dataset, where a batch of pseudo tasks is sampled for meta-training at each iteration. (vii) BiDf-MKD [16]. A bi-level data-free meta knowledge distillation framework to transfer general knowledge in the white-box setting.

Metrics. In addition to the standard few-shot classification accuracy, we also focus on the speed of model inversion in DFML, quantifying this through the metric of GPU hours. Note that the time cost of meta-learner training is omitted since we only focus on the data recovery process. For consistency in comparison, all GPU hours are obtained using a single GeForce RTX 3090 GPU.

Comparisons with baselines. Tab. 2 shows the results for 5-way classification compared with current baselines. To ascertain the efficacy of Free, we compare it against two DFML algorithm categories: the non-inversion methods, which facilitate model fusion in the parameter space, and the inversion-based methods that generate pseudo data for training. As shown in Tab. 2, our approach considerably outperforms all non-inversion methods, and is significantly faster than other inversion-based methods. For instance, PURER generates data by constructing a smaller pseudo dataset, with each class comprising only 20 images. Nevertheless, PURER demands 15000 iterations to converge in optimizing the dataset. In contrast, we only require 2/5 steps for different tasks, owning to our adaptive meta-generator.

Multi-domain scenario. We conduct experiments in a challenging multi-domain scenario where all pre-trained models are tailored to address distinct tasks from multiple meta-training subsets (CIFAR-FS, miniImageNet, and CUB). For meta-testing, we assess the meta-learner on unseen tasks spanning CIFAR-FS, miniImageNet, and CUB concurrently, demanding the meta-learner to possess generalization capabilities across multiple domains. Tab. 3 showcases our results. Our approach achieves commendable results, outperforming the baseline by 6.66% and 15.69% in 1-shot and 5-shot learning, respectively. This solidly demonstrates that BelL effectively aligns tasks from different domains, optimizing them towards a unified direction, resulting in robust generalization across various domains.

Multi-architecture scenario. We also conduct experiments in a multi-architecture scenario where each pre-trained model differs in architecture. For each pre-trained model, the architecture is randomly chosen from Conv4, ResNet-10 and ResNet-18. The results, presented in Tab. 4, demonstrate the efficacy of our approach. It outperforms all baselines and can apply to multi-architecture scenario without any change. This flexibility is attributable to our method’s lack of constraints on the underlying architecture or scale of the pre-trained models, which enables effective task alignment across heterogeneous models.

Ablation studies. Tab. 5 evaluates the impact of each module on the CIFAR-FS. Initially, we set a baseline only performing meta-learning via cross task replay, using a sequential generator. The generator trained on a previous pre-trained model is directly used as an initialization for subsequent tasks. However, updating the generator with only 5 steps for each pre-trained model proves to be insufficient. This is attributed to the non-reusability of the sequential generator across diverse pre-trained models. Incorporating FIve module offers notable enhancements, suggesting that the meta-generator can rapidly adapt and inverse task-specific training data. Nonetheless, the current method (i.e., ERM) still merely aggregates knowledge distillation losses from multiple tasks to update the meta-learner, overlooking the alignment of optimization directions across tasks. By adding BelL module, we observe a significant improvement (i.e., Sequence), demonstrating the effectiveness of the conflicting gradient alignment. With all modules, we achieve the best performance with a boosting improvement of 2.8% and 4.91%, showing their complementarity.

Heterogeneity among pre-trained models. We present a $t$-SNE visualization of the features extracted from different pre-trained models in Fig. 5. We randomly sample 10 images of the same class and use 10 pre-trained models from CIFAR-FS to extract their features. The visualization clearly shows that the features of the same image, when extracted by different pre-trained models, are scattered in the feature space. This implies that the inversed tasks from distinct pre-trained models naturally inherit varying feature distributions. This observation underscores the importance of addressing the heterogeneity among pre-trained models.

Gradient inner product maximization. In our approach, we incorporate BelL to align gradients across different tasks, guiding the meta-learner towards a unified direction. To further validate whether BelL implicitly maximizes the gradient inner product, we chart its evolution during meta-learner training in Fig. 5. We contrast BelL with the ERM method, monitoring the mean gradient inner product between tasks derived from varied pre-trained models in each inner loop. As evident from Fig. 5, the gradient inner product for BelL gradually rises with training progression. Conversely, ERM maintains lower values throughout. This confirms BelL’s effectiveness in implicitly aligning the gradient direction by enhancing the gradient inner product.

Effects of the number of pre-trained models. We conduct experiments on DFML with varying numbers of pre-trained models. Each model is pre-trained on a 5-way subset of CIFAR-FS. As shown in Fig. 6, the performance improves with an increasing number of models. This can be attributed to the fact that a greater quantity of pre-trained models offers a more comprehensive understanding of various classes, thereby enhancing generalization capabilities for unseen classes. Interestingly, we observe that ERM delivers promising results with a smaller set of pre-trained models, at times even outperforming our approach. However, as the number of pre-trained models continues to climb, ERM’s performance starts to drop and is eventually surpassed by our approach. This decline can be attributed to the growing issues of model heterogeneity and task conflicts with an increase in the number of pre-trained models. The poor performance of ERM highlights the challenge of utilizing numerous available pre-trained models from sources like GitHub to enable efficient learning on unseen downstream tasks. Contrarily, we effectively navigate this challenge, consistently delivering performance enhancements.

Visualizations. Fig. 7 presents a 5-way task recovered from the Conv4. A significant advantage of DFML lies in its ability to learn from weaker pre-trained models without data privacy leakage. Our approach generates images that are visually distinct from the originals, thereby addressing privacy concerns. The images we generate display high-frequency patterns and textures, which are recognized for their particular value in training neural networks [11, 22].