Generalization Bounds for Deep Transfer Learning Using Majority Predictor Accuracy

Deep transfer learning, the problem of transferring representations (or features) learned by deep neural networks from one task to another, has become a crucial part for training deep learning models in practice [14, 10, 19]. Despite this fact, the current literature still lacks a theory for understanding the generalization of models obtained by deep transfer learning. In this paper, we close this gap between the theory and practice of deep transfer learning by proving novel generalization bounds for models learned through such transfer learning methods.

To prove the bounds, we develop the Majority Predictor Accuracy (MPA), a simple and easy-to-compute quantity defined as the accuracy of the classifier that returns the most probable target label conditioned on a given source label. Using the MPA, we can show that when the source and target data share the input set, the true risk of the transferred model is upper bounded by the sum of the empirical risk of the source model, $1-\text{MPA}$, and an $\widetilde{\mathcal{O}}(1/\sqrt{n})$ sample complexity with high probability. We further extend this result to the more general setting where the source and target datasets contain different inputs. This extension is achieved by using dummy source labels, a technique previously developed for transferability estimation [13].

We also demonstrate the usefulness of our theoretical bounds in practice by showing empirically that the MPA can be used as a transferability measure, defined as a numeric score that can tell whether deep transfer learning would be effective when transferring between a given pair of source-target tasks. Specifically, our experiments on the large-scale CUB-200 dataset [18] show that the MPA scores are highly correlated with the actual accuracies of the transferred models with statistical significance, thus indicating that the MPA is a good measure of transferability.

To summarize, our paper makes the following contributions: (1) developing the new MPA score, (2) proving novel deep transfer learning bounds using the MPA, and (3) showing our bounds are practically useful through experiments.

Related Work. Transfer learning [10, 20] is a long-standing research area of machine learning. Several previous work has provided theoretical analysis and generalization bounds for transfer learning, especially under the domain adaptation setting, such as [5, 6, 11, 4, 1]; however, these results were not explicitly developed for deep learning and the settings commonly used in practice, where a learned representation is adapted to the new domain [14, 19]. Our paper, on the other hand, provides generalization bounds explicitly for these commonly used deep transfer learning settings. Furthermore, unlike these previous work, our bounds are useful in practice for understanding the transferability between different tasks, as demonstrated in our experiments.

Our work is also related to a recent attempt to develop transferability measures for deep transfer learning [2, 17, 13, 15, 20, 16, 8]. Transferability measures aim to estimate the effectiveness of deep transfer learning between tasks and have been used for model or task selection [2, 17, 13, 20], checkpoint ranking [8], and few-shot learning [16]. Although some theoretical properties were shown for these transferability measures [17, 13, 16], they only focused on the empirical risk instead of the true risk as in our paper.

Deep transfer learning [10] refers to the problem of transferring a learned deep neural network representation from a source task to a target task. In this section, we formalize the deep transfer learning setting considered in our paper. This setting is commonly used in practice for several large-scale deep learning models [14, 19].

Let $\mathcal{S}=\{(x_{1},s_{1}),(x_{2},s_{2}),\ldots,(x_{n},s_{n})\}$ be a train dataset for a source classification task where each input-label pair $(x_{i},s_{i})\in\mathbb{R}^{d}\times[m_{S}]$ is drawn iid from a joint distribution $\mathbb{P}_{X,S}$, with $[n]=\{1,2,\ldots,n\}$ for any positive integer $n$. Consider a target classification task with a train set ${\mathcal{T}=\{(x_{1},t_{1}),(x_{2},t_{2}),\ldots,(x_{n},t_{n})\}}$ where each target example $(x_{i},t_{i})\in\mathbb{R}^{d}\times[m_{T}]$ is drawn iid from $\mathbb{P}_{X,T}$. We will first consider this simple case where the source and target datasets share the same inputs $\{x_{1},x_{2},\ldots,x_{n}\}$, with each $x_{i}$ being a $d$-dimensional vector having the same marginal distribution $\mathbb{P}_{X}$ in both tasks. Here the source task has $m_{S}$ classes and the target task has $m_{T}$ classes. The more general case with different input sets will be discussed in Section V.

In our deep transfer learning setting, we first train a source model $h{\,{\circ}\,}w$ using $\mathcal{S}$, where $w(x)\in\mathbb{R}^{r}$ is the $r$-dimensional representation (also called embedding or feature vector) of the input $x$ extracted from the network $w$, and $h{\,{\circ}\,}w(x)=h(w(x))\in[m_{S}]$ is the source label returned by the network $h$ with the representation $w(x)$ as input. The functions $w$ and $h$ are usually called the feature extractor and the head classifier respectively. In deep learning, the whole model $h{\,{\circ}\,}w$ is a deep neural network with $w$ being its parameters from the input up to some layer $L$, and $h$ being the network parameters from layer $L$ to the output. We obtain the optimal source model on $\mathcal{S}$ by minimizing the empirical risk:¹¹1Throughout our paper, we assume $\operatorname*{arg\,min}$ and $\operatorname*{arg\,max}$ follow any deterministic tie-breaking strategy.

where $\Omega_{w}$ and $\Omega_{h}$ are the spaces of all possible $w$’s and $h$’s respectively, and with $\mathbf{1}[\cdot]$ being the indicator function,

In practice, the optimal feature extractor $w^{*}$ often learns generic feature representations (e.g., edges or shapes in images) that can be reused for several tasks, while the optimal head classifier $h^{*}$ is often specialized for a particular source task. To transfer this trained model $h^{*}{{\circ}\,}w^{*}$ to a target task, the usual practice is to discard $h^{*}$ and reuse $w^{*}$ for the target task. Specifically, we will re-train a new head classifier $k^{*}$ on the target dataset $\mathcal{T}$ using the features extracted from $w^{*}$.

In this paper, we allow the target head classifier to return real-valued scores for all target labels. That means for each example $x$, a head classifier $k$ on the target task would take $w^{*}(x)$ as input and return $k{\,{\circ}\,}w^{*}(x)=k(w^{*}(x))\in\mathbb{R}^{m_{T}}$, the scores (before softmax) for all target labels. We will consider the optimal target head classifier $k^{*}$ obtained by minimizing the empirical risk with a given margin $\gamma\geq 0$:

where $\Omega_{k}$ is the space of all $k$’s, and for all $w\in\Omega_{w}$, $k\in\Omega_{k}$:

with $k{\,{\circ}\,}w(x_{i})_{t}$ being the $t$-th element of the vector $k{\,{\circ}\,}w(x_{i})$. Here $\widehat{R}_{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\bullet$}}}},\gamma}$ is a general version of the empirical risk in Eq. (2). The margin $\gamma$ measures the gap between the prediction probability of the correct label and those of the other labels, and has often been used in generalization bounds for deep learning [3, 9].

Our paper shall prove generalization bounds for the optimal transferred model $k^{*}{{\circ}\,}w^{*}$. For this purpose, we introduce in the next section the majority predictor accuracy, a transferability measure that we will use for our bounds.

The Majority Predictor Accurcacy (MPA) is defined as the accuracy of the simple predictor (classifier) that maps each source label to the target label with maximal empirical conditional probability. Formally, given the source dataset $\mathcal{S}$ and the target dataset $\mathcal{T}$, the empirical joint distribution between all possible source-target label pairs ${(s,t)\in[m_{S}]\times[m_{T}]}$ is $\hat{P}(s,t)=\frac{1}{n}{|\{i\in[n]:s_{i}=s\ \mathrm{and}\ t_{i}=t\}|}$, the empirical marginal distribution over the source labels is ${\hat{P}(s)=\sum_{t\in[m_{T}]}\hat{P}(s,t)},\forall s\in[m_{S}]$, and the empirical conditional distribution of a target label $t$ given a source label $s$ is $\hat{P}(t|s)=\hat{P}(s,t)/\hat{P}(s)$, for all $(s,t)\in[m_{S}]\times[m_{T}]$.

To define the MPA, consider the following majority predictor $f_{\text{mp}}$ that takes a source label $s\in[m_{S}]$ and simply returns a target label $t$ that maximizes the empirical conditional probability $\hat{P}(t|s)$:

The MPA is then defined as the accuracy of $f_{\text{mp}}$ on the target dataset, as stated in the following definition.

The MPA is very simple to compute, requiring only $O(n)$ computational time by looping through the datasets a few times to compute the empirical distributions, $f_{\text{mp}}$, and its accuracy. In Section VII, we will show that it can also be used as a transferability measure that estimates the effectiveness of transfer learning between two tasks. We now prove generalization guarantees for the transferred models in the next section.

This section proves our generalization bounds for deep transfer learning based on the MPA where the source and target training sets are assumed to share the inputs. In particular, we bound the true risk of the transferred model $k^{*}{{\circ}\,}w^{*}$ on the target distribution $\mathbb{P}_{X,T}$, which is:

for $(x,t)\sim\mathbb{P}_{X,T}$. We will prove the bounds for both fully connected neural networks and convolutional neural networks. For this purpose, we consider the head classifier $f_{\text{mp}}{\,{\circ}\,}h^{*}$ defined on any representation $w(x)\in\mathbb{R}^{r}$, where:

Throughout our paper, we will make an assumption that using a deep neural network as the target head classifier can achieve better empirical risk than using the naive classifier $f_{\text{mp}}{\,{\circ}\,}h^{*}$. This assumption is usually satisfied in practice because of the expressiveness of neural network models [21].

In this assumption, $\widehat{R}_{\mathcal{T}}(w^{*},f_{\text{mp}}{\,{\circ}\,}h^{*})$ is defined similarly as in Eq. (2). We also note that $\widehat{R}_{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\bullet$}}}},\gamma}$ is non-decreasing in $\gamma$, so the assumption implies, for all $\gamma\in[0,\bar{\gamma}]$, $\widehat{R}_{\mathcal{T},\gamma}(w^{*},\bar{k})\leq\widehat{R}_{\mathcal{T},\bar{\gamma}}(w^{*},\bar{k})\leq\widehat{R}_{\mathcal{T}}(w^{*},f_{\text{mp}}{\,{\circ}\,}h^{*})$. Under this assumption, we first prove the following lemma relating the empirical risks of the optimal source and transferred models using the MPA.

As a remark, the bound in Lemma 1 gets tighter when $\text{MPA}(\mathcal{T}|\mathcal{S})\rightarrow 1$, that is, when $f_{\text{mp}}$ is more accurate. Using this lemma, we now prove the generalization bounds for the transferred model $k^{*}{{\circ}\,}w^{*}$. Section IV-A below proves the bound for fully connected neural networks, while Section IV-B proves the bound for convolutional neural networks.

Up until now, we have only considered the case where source and target datasets share the same input set. In this section, we extend our results to the case where the source and target datasets contain different inputs. Formally, let $\mathcal{S}=\{(x_{1},s_{1}),(x_{2},s_{2}),\ldots,(x_{n},s_{n})\}$ be the source dataset where $(x_{i},s_{i})\in\mathbb{R}^{d}\times[m_{S}]$ is drawn iid from a joint distribution $\mathbb{P}_{X,S}$, and $\mathcal{T}=\{(z_{1},t_{1}),(z_{2},t_{2}),\ldots,(z_{p},t_{p})\}$ be the target dataset where $(z_{i},t_{i})\in\mathbb{R}^{d}\times[m_{T}]$ is drawn iid from $\mathbb{P}_{Z,T}$. We also follow the deep transfer learning procedure in Section II and first train the optimal model $h^{*}{{\circ}\,}w^{*}$ on the source data $\mathcal{S}$ using Eq. (1). Then we freeze $w^{*}$ and train the target head classifier $k^{*}$ using Eq. (3) with the target dataset $\mathcal{T}$, where we will apply $w^{*}$ to the target inputs $\{z_{1},z_{2},\ldots,z_{p}\}$ to get the representations $\{w^{*}(z_{1}),w^{*}(z_{2}),\ldots,w^{*}(z_{p})\}$.

To prove the generalization bounds, we will consider a new source dataset $\tilde{\mathcal{S}}:=\{(z_{i},h^{*}{{\circ}\,}w^{*}(z_{i}))\}_{i=1}^{p}$ induced by $h^{*}{{\circ}\,}w^{*}$ and the target inputs $\{z_{1},z_{2},\ldots,z_{p}\}$. In essence, $\tilde{\mathcal{S}}$ contains the target inputs with “dummy” source labels generated by $h^{*}{{\circ}\,}w^{*}$. This technique of using these dummy labels was previously employed to develop the LEEP transferability measure [13], and is useful for proving our bounds as well. With the new source dataset $\tilde{\mathcal{S}}$, we consider the majority predictor $f_{\text{mp}}$ constructed from $(\tilde{\mathcal{S}},\mathcal{T})$, as well as the corresponding majority predictor accuracy $\text{MPA}(\mathcal{T}|\tilde{\mathcal{S}})$. We still keep Assumption 1 in Section IV, but adapt it to the new $h^{*}{{\circ}\,}w^{*}$ and $f_{\text{mp}}$. The following lemma is the analogue of Lemma 1 for the different inputs setting.

Similar to Section IV, we derive the following generalization bounds, which are analogues of Theorems 1 and 2. The proofs of these theorems are similar to those of Theorems 1 and 2, with Lemma 1 being replaced by Lemma 2.

The technique used to prove our theorems is general and can be combined with other generalization bounds for deep neural networks. Although we proved our results using the norm-based bounds of [9], we emphasize that our proof technique can also be used with other generalization bounds for neural networks, such as those of [3].

The bounds in Theorems 1 and 2 depend on both the optimal source empirical risk $\widehat{R}_{\mathcal{S}}(w^{*},h^{*})$ and $\text{MPA}(\mathcal{T}|\mathcal{S})$. These bounds get better when $\widehat{R}_{\mathcal{S}}(w^{*},h^{*})\rightarrow 0$ and $\text{MPA}(\mathcal{T}|\mathcal{S})\rightarrow 1$. The bounds in Theorems 3 and 4 do not contain the source empirical risk, since it has been indirectly measured in $\text{MPA}(\mathcal{T}|\tilde{\mathcal{S}})$ when we use $h^{*}{{\circ}\,}w^{*}$ to construct $\tilde{\mathcal{S}}$.

From our results, we can see that $\text{MPA}(\mathcal{T}|\mathcal{S})$ (or $\text{MPA}(\mathcal{T}|\tilde{\mathcal{S}})$ for the setting with different inputs) can be used as a transferability measure. Specifically, for well-trained and deep enough neural networks, it has been observed empirically [21] that ${\widehat{R}_{\mathcal{S}}(w^{*},h^{*})\approx 0}$. Furthermore, the $\widetilde{\mathcal{O}}(\cdot)$ terms in our theorems are near $0$ for large enough $n$. In this case, our results imply that $\text{MPA}(\mathcal{T}|\mathcal{S})\lessapprox 1-R_{T}(w^{*},k^{*})$ or ${\text{MPA}(\mathcal{T}|\tilde{\mathcal{S}})\lessapprox 1-R_{T}(w^{*},k^{*})}$. This means that $\text{MPA}(\mathcal{T}|\mathcal{S})$ or $\text{MPA}(\mathcal{T}|\tilde{\mathcal{S}})$ lower bounds the expected accuracy of the transferred model $k^{*}{{\circ}\,}w^{*}$, and thus can be used as a transferability measure. We now validate this observation empirically.

We show the usefulness of our theoretical bounds in practice by empirically illustrating the ability of MPA as a transferability measure on the large-scale Caltech-UCSD Birds-200 dataset [18], which contains 11,788 images of 200 bird species labeled with 312 binary attributes. We keep the train-test split as provided in dataset, with 5,994 train images and 5,794 test images. We pick 4 attributes Curved Bill, Iridescent Wings, Brown Upper Parts and Olive Under Parts for training source models, and randomly choose 100 different attributes as target tasks. Regarding the model architecture, we use ResNet18 [7] without the last fully connected layer as the feature extractor $w$. In all tests, we train our source model $h^{*}{{\circ}\,}w^{*}$ and the transferred model $k^{*}{{\circ}\,}w^{*}$ using the cross-entropy loss with batch size 32 and run the stochastic gradient descent optimizer with momentum for 40 epochs. The initial learning rate is set at 0.01 and is divided by 10 every 10 epochs.

Following the settings in [17, 13], we estimate the correlations between the MPA scores and the actual test accuracies of the transferred models to evaluate the relationship between these two quantities. High correlations mean the MPA score is a good measure for comparing test accuracies of the transferred models, and thus is a good transferability measure. For the 4 source tasks above with 100 randomly chosen target tasks, our experimental results give the following Pearson correlation coefficients: 0.9534 (Curved Bill), 0.9452 (Iridescent Wings), 0.9484 (Brown Upper Parts), and 0.9611 (Olive Under Parts). These coefficients show that the MPA scores and the test accuracies are highly positive correlated with statistical significance ($p<10^{-4}$), which clearly indicates that the MPA is a reliable transferability measure for estimating the performance of transferred models.

We proved novel generalization bounds for transfer learning of deep neural networks using a new quantity, the majority predictor accuracy, that can be computed easily and efficiently from data. We showed the usefulness of our bounds in practice by demonstrating that the majority predictor accuracy can be used for estimating the effectiveness of deep transfer learning. Our theory can potentially be extended to analyze more complex transfer scenarios such as continual learning [12].