On Hypothesis Transfer Learning of Functional Linear Models

Advances in technologies enable us to collect and process densely observed data over some temporal or spatial domains, which are coined functional data [Ramsay et al., 2005; Kokoszka and Reimherr, 2017]. While functional data analysis (FDA) has been proven useful in various fields like finance, genetics and etc., and has been researched widely in the statistical community, its effectiveness relies on having sufficient training samples drawn from the same distribution. However, this may not hold for functional data under some applications due to collection expenses or other constraints. Transfer learning (TL)[Torrey and Shavlik, 2010] leverages additional information from some similar (source) tasks to enhance the learning procedure on the original (target) task, and is an appealing mechanism when there is a lack of training samples. The goal of this paper is to develop TL algorithms for functional linear regression (FLR), one of the most prevalent models in the FDA. The FLR concerned in this paper is Scalar-on-Function regression, which takes the form:

where $Y$ is a scalar response, $X:\mathcal{T}\to\mathbb{R}$ and $\beta:\mathcal{T}\to\mathbb{R}$ are the square-integrable functional predictor and coefficient function respectively over a compact domain $\mathcal{T}\subset\mathbb{R}$, and $\epsilon$ is a random noise with zero mean.

A classical approach to estimating $\beta$ is to reduce the problem to classical multivariate linear regression by expanding the $X$ and $\beta$ under the same finite basis, like deterministic basis functions, e.g. Fourier basis, or the eigenbasis of the covariance function of $X$ [Cardot et al., 1999; Yao et al., 2005; Hall and Hosseini-Nasab, 2006; Hall and Horowitz, 2007], which we refer to truncation-based FLR in this paper. Conceptually, the offset transfer learning techniques developed in multivariate/high-dimensional linear regression [Kuzborskij and Orabona, 2013, 2017; Li et al., 2022; Bastani, 2021] can be applied to truncation-based FLR methods, though they lack a theoretical foundation in this context due to the truncation error inherent in a basis expansion of $\beta$. In particular, a key property distinguishing functional data from multivariate data is that they are inherently infinite-dimensional and generated through smooth underlying processes. Omitting this fact and treating the finite coefficients as multivariate parameters loses the benefit that data are generated from smooth processes, see detailed discussion in Section 2. Observing these limitations we develop the first TL algorithms for FLR with statistical risk guarantees under the supervised learning setting.

We summarize our main contributions as follows.

1. 1.

   We propose using the Reproducing Kernel Hilbert Space (RKHS) distance between tasks’ coefficients as a measure of task similarity. The transferred information is thus tied to the RKHS’s properties and makes the transfer more interpretable. One can tailor the employed RKHS to the task’s nature, offering flexibility to embed diverse structural elements, like smoothness or periodicity, into TL processes.

2. 2.

   Built upon the offset transfer learning (OTL) paradigm, we propose TL-FLR, a variant of OTL for multiple positive transfer sources. We establish the minimax optimality for TL-FLR. Intriguingly, the result reveals that the faster statistical rate of TL-FLR, compared to non-transfer learning, not only depends on source sample size and the magnitude of discrepancy across tasks like most existing works, but also the signal ratio between offset and source model.

3. 3.

   To deal with the practical scenario in which there is no available prior task similarity information, we propose Aggregation-based TL-FLR (ATL-FLR), utilizing sparse aggregation to mitigate negative transfer effects. We establish the upper bound for ATL-FLR and show the aggregation cost decreases faster than the transfer learning risk, demonstrating an ability to identify optimal sources without too much extra cost compared to TL-FLR. We further extend this framework to Functional Generalized Linear Models (FGLM) with theoretical guarantees, broadening its applicability.

4. 4.

   In developing statistical guarantees, we uncovered unique requirements for making OTL theoretically feasible in the functional data context. These include the necessity for covariate functions across tasks to exhibit similar structural properties to ensure statistical convergence, and the coefficient functions of negative sources can be separated from positive ones within a finite-dimensional space to ensure optimal source identification.

In this section, we study the theoretical properties of the prediction accuracy of the proposed algorithms. We evaluate the proposed algorithms via excess risk, i.e.

where the expectation is taken over an independent test data point $Z^{(0)}$ from the target distribution. To study the excess risk of TL-FLR and ATL-FLR, we define the parameter space as

To establish the theoretical analysis of the proposed algorithms, we first state some assumptions. For $t\in 0\cup[T]$, denote $\{s_{j}^{(t)}\}_{j\geq 1}$ and $\{\phi_{j}^{(t)}\}_{j\geq 1}$ as the eigenvalues and eigenfunctions of $\Gamma^{(t)}:=K^{\frac{1}{2}}C^{(t)}K^{\frac{1}{2}}$ respectively.

The polynomial EDR assumption is standard in FLR literature like Cai and Yuan [2012]; Reimherr et al. [2018]. RKHSs that satisfy this assumption, like Sobolev spaces, are natural choices when one is concerned about the smoothness being the structural properties in the TL processes.

We note that under the posterior drift setting, both conditions in Assumption 2 are satisfied automatically. Although neither condition implies the other, both conditions primarily focus on how the smoothness of the source kernel $\Gamma^{(t)}$ relates to that of the target kernel $\Gamma^{(0)}$. Specifically, Condition 1 implies $L_{\Gamma^{(0)}}$ and $L_{\Gamma^{(t)}}$ not only share the same eigenspace but also have similar magnitudes of the projection onto the $j$-th dimension of the eigenspace, which commonly appears in FDA literature [Yuan and Cai, 2010; Balasubramanian et al., 2022]. This helps us to control the excess risk of $\hat{\beta}_{S}$ over the target domain. Condition 2 implies the probability measures of $X^{(0)}$ and $X^{(t)}$ are equivalent, see Baker [1973]. Collectively, these conditions indicate the feasibility of OTL for functional data relies on the fact that source covariance function $C^{(t)}$ should behave similarly to the target’s $C^{(0)}$. Either a too “smooth” or a too “rough” $X^{(t)}$ can degrade the optimality. This aligns with the principles of standard target-only FLR, where the performance of the estimator is jointly determined by the covariance function and reproducing kernel.

In this section, we show our approaches in the FLR model can be naturally extended to the functional generalized linear model (FGLM) settings, which includes wider application scenarios like classification. To start, consider the following series of FGLM models similar to the FLR setting (1),

where $i\in[n_{t}]$ and $t\in[T]$, $\theta_{i}^{(t)}=\langle X_{i}^{(t)},\beta^{(t)}\rangle_{L^{2}}$ is the canonical parameter. The functions $\rho,\eta,\psi,d$ are known, and $\tau$ is either known or out-of-interest parameter that is independent of $X^{(t)}$. In this paper, we consider $\eta$ to take the canonical form, i.e. $\eta(x)=x$. The GLMs are characterized by the different $\psi$. For example, in linear regression with Gaussian response, $\psi(x)=x^{2}/2$; in the logistic regression with binary response, $\psi(x)=log(1+e^{x})$; and in Poisson regression with non-negative integer response, $\psi(x)=e^{x}$.

A standard method for addressing GLM involves minimizing the loss function defined as the negative log-likelihood. Therefore, to implement the transfer learning for FGLM, one can naturally substitute the square loss in TL-FLR (Algorithm 1) and ATL-FLR (Algorithm 2) with the negative log-likelihood loss, i.e.

We refer to these transfer learning algorithms for FGLM as TL-FGLM and ATL-FGLM. To establish the optimality of TL-FGLM and ATL-FGLM, the following technical assumptions are required.

Assumption 4 is natural in most GLM literature and is satisfied by many popular exponential families. Assumption 5 restricts the $\psi^{\prime\prime}$ in the bounded region and thus restricts the variance of $y$.

Since the conditional mean for FGLM is $\operatorname{E}[Y_{i}|X_{i}]=\eta^{\prime}(\langle\beta,X^{(0)}\rangle_{L^{2}})$, we evaluate the accuracy by excess risk, i.e. $\mathcal{E}(\hat{\beta}):=\operatorname{E}_{X^{(0)}}[\eta^{\prime}(\langle\hat{\beta},X^{(0)}\rangle_{L^{2}})-\eta^{\prime}(\langle\beta^{(0)},X^{(0)}\rangle_{L^{2}})]^{2}$.

We illustrate our algorithms for FLR by providing results using simulated data and defer the financial market data application and FGLM results to Appendix E. We consider the following, OFLR, TL-FLR, ATL-FLR, Detection Transfer Learning (Detect-TL) [Tian and Feng, 2022] and Exponential Weighted ATL-FLR (ATL-FLR (EW)). To set up the RKHS, we consider the setting in Cai and Yuan [2012]. Let $\psi_{k}(s)=\sqrt{2}\cos(\pi ks)$ for $j\geq 1$ and define the reproducing kernel $K$ of $\mathcal{H}_{K}$ as $K(\cdot,\cdot)=\sum_{k=1}^{\infty}k^{-2}\psi_{k}(\cdot)\psi_{k}(\cdot)$.

For the target model, $\beta^{(0)}(s)$ is set to be (1) $\beta_{1}^{(0)}(s)=\sum_{k=1}^{\infty}4\sqrt{2}(-1)^{k-1}k^{-2}\psi_{k}(s)$; (2) $\beta_{2}^{(0)}(s)=4\cos(3\pi s)$; (3) $\beta_{3}^{(0)}(s)=4\cos(3\pi s)+4\sin(3\pi s)$. For a specific $h$, let $\mathcal{S}=\{l:\|\beta_{0}-\beta_{t}\|_{K}\leq h\}$, then we generate source models as follows. We scale each target model such that their RKHS norm is $20$. If $t\in\mathcal{S}$, then $\beta_{t}(t)$ is set to be $\beta_{t}(s)=\beta_{0}(s)+\sum_{k=1}^{\infty}\left(\mathcal{U}_{k}(\sqrt{12}h/\pi k^{2})\right)\psi_{k}(s)$ with $\mathcal{U}_{k}$’s i.i.d. uniform random variable on $[-1,1]$. If $t\in\mathcal{S}^{c}$, then $\beta_{t}(s)$ is generated from a Gaussian process with mean function $\cos(2\pi s)$ with kernel $\exp(-15|s-t|)$ as covariance kernel. The predictors $X^{(t)}$ are i.i.d. generated from a Gaussian process with the mean function $\sin(\pi s)$ and the covariance kernels are set to be Matérn kernel $C_{\nu,\rho}$ [Cressie and Huang, 1999] where the parameter $\nu$ controls the smoothness of $X^{(t)}$. We set the covariance kernel of $X^{(t)}$ as $C_{1/2,1}$ for the target tasks and $C_{3/2,1}$ for source tasks, to fulfill Assumptions 2. We note that such a setting is more challenging than assuming that the target and source tasks have the same covariance kernel. All functions are generated on $[0,1]$ with 50 evenly spaced points and we set $n_{0}=150$ and $n_{t}=100$. For each algorithm, we set the regularization parameters as $\lambda_{1}$ and $\lambda_{2}$ as the optimal values in Theorem 1 and select the constants using cross-validation. The excess risk for the target tasks is calculated via the Monte-Carlo method by using $1000$ newly generated predictors $X^{(0)}$.

In the left panel of Figure 2 we compare TL-FLR with OFLR by considering relative excess risk, i.e. the ratio of TL-FLR’s excess risk to OFLR’s. We note that since RKHS of $\beta^{(0)}$ is fixed, the magnitude of $h$ is proportional to $\xi(h,\beta_{\mathcal{S}})$ and a large $h$ indicates less similarity between sources and target tasks. Overall, the effectiveness of TL-FLR for different $\beta^{(0)}$ presents a consistent pattern, i.e, with more transferable sources involved and smaller $h$ (bottom right), TL-FLR has a more significant improvement, while with fewer sources and larger $h$ (top left), the transfer may be worse than OFLR.

In the right panel of Figure 2, we evaluate ATL-FLR under unknown $\mathcal{S}$ cases. We set $\mathcal{S}$ as a random subset of $\{1,2,\cdots,20\}$ such that $|\mathcal{S}|$ is equal to $0,2,\cdots,20$. We also implement TL-FLR by using true $\mathcal{S}$ and OFLR as baseline. In all scenarios, ATL-FLR outperforms all its competitors. Comparing ATL-FLR with ATL-FLR(EW), even though ATL-FLR(EW) has similar patterns as ATL-FLR, we can see the gaps between the two curves are larger when the proportion of $\mathcal{S}$ is small, showing that ATL-FLR(EW) is more sensitive to source tasks in $\mathcal{S}^{c}$, while ATL-FLR is less affected. Detect-TL only has a considerable reduction on the excess risk with relatively small $h$, but provides limited improvement when $h$ is large, indicating its limited learning effect when there is only limited knowledge available in sources.

In this paper, we study offset transfer learning in functional regression settings including FLR and FGLM. We derive excess risk and show a faster statistical rate depending on both source sample size and the magnitude of similarity across tasks. Our theoretical analysis will help researchers better understand the transfer dynamic of offset transfer learning. Moreover, we leverage the sparse aggregation to alleviate the negative transfer effect while having faster-decreasing aggregation costs compared to transfer learning risk.