Sparse Overlapping Sets Lasso for Multitask Learning and its Application to fMRI Analysis

Multitask learning exploits the relationships between several learning tasks in order to improve performance, which is especially useful if a common subset of features are useful for all tasks at hand. The group lasso (Glasso) [21, 10] is naturally suited for this situation: if a feature is selected for one task, then it is selected for all tasks. This may be too restrictive in many applications, and this motivates a less rigid approach to multitask feature selection. Suppose that the available features can be organized into overlapping subsets according to a notion of similarity, and that the features useful in one task are similar, but not necessarily identical, to those best suited for other tasks. In other words, a feature that is useful for one task suggests that the subset it belongs to may contain the features useful in other tasks (Figure 1).

In this paper, we introduce the sparse overlapping sets lasso (SOSlasso), a convex program to recover the sparsity patterns corresponding to the situations explained above. SOSlasso generalizes lasso [18] and Glasso, effectively spanning the range between these two well-known procedures. SOSlasso is capable of exploiting the similarities between useful features across tasks, but unlike Glasso it does not force different tasks to use exactly the same features. It produces sparse solutions, but unlike lasso it encourages similar patterns of sparsity across tasks. Sparse group lasso [16] is a special case of SOSlasso that only applies to disjoint sets, a significant limitation when features cannot be easily partitioned, as is the case of our motivating example in fMRI. The main contribution of this paper is a theoretical analysis of SOSlasso, which also covers sparse group lasso as a special case (further differentiating us from [16]). The performance of SOSlasso is analyzed, error bounds are derived for general loss functions, and its consistency is shown for squared error loss. Experiments with real and synthetic data demonstrate the advantages of SOSlasso relative to lasso and Glasso.

We formalize the notations used in the sequel. Lowercase and uppercase bold letters indicate vectors and matrices respectively. We assume a multitask learning framework, with a data matrix $\bm{\Phi}_{t}\in\mathbb{R}^{n\times p}$ for each task $t\in\{1,2,\ldots,\mathcal{T}\}$. We assume there exists a vector $\bm{x}^{\star}_{t}\in\mathbb{R}^{p}$ such that measurements obtained are of the form $\bm{y}_{t}=\bm{\Phi}_{t}\bm{x}^{\star}_{t}+\eta_{t}~\ \eta_{t}\sim\mathcal{N}(0,\sigma^{2}\bm{I})$. Let $\bm{X}^{\star}:=\left[\bm{x}_{1}^{\star}~\ \bm{x}_{2}^{\star}~\ \ldots\bm{x}_{\mathcal{T}}^{\star}\right]\in\mathbb{R}^{p\times\mathcal{T}}$. Suppose we are given $M$ (possibly overlapping) groups $\tilde{\mathcal{G}}=\{\tilde{G}_{1},\tilde{G}_{2},\ldots,\tilde{G}_{M}\}$, so that $\tilde{G}_{i}\subset\{1,2,\ldots,p\}~\ \forall i$, of maximum size $B$. These groups contain sets of “similar” features, the notion of similarity being application dependent. We assume that all but $k\ll M$ groups are identically zero. Among the active groups, we further assume that at most only a fraction $\alpha\in(0,1)$ of the coefficients per group are non zero. We consider the following optimization program in this paper

where $\bm{x}=[\bm{x}_{1}^{T}\bm{x}_{2}^{T}\ldots\bm{x}_{\mathcal{T}}^{T}]^{T}$, $h(\bm{x})$ is a regularizer and $\mathcal{L}_{t}:=\mathcal{L}_{\bm{\Phi}_{t}}(\bm{x}_{t})$ denotes the loss function, whose value depends on the data matrix $\bm{\Phi}_{t}$. We consider least squares and logistic loss functions. In the least squares setting, we have $\mathcal{L}_{t}=\frac{1}{2n}\|\bm{y}_{t}-\bm{\Phi}_{t}\bm{x}_{t}\|^{2}$. We reformulate the optimization problem (1) with the least squares loss as

where $\bm{y}=[\bm{y}_{1}^{T}\bm{y}_{2}^{T}\ldots\bm{y}_{\mathcal{T}}^{T}]^{T}$ and the block diagonal matrix $\bm{\Phi}$ is formed by block concatenating the $\bm{\Phi}_{t}^{\prime}s$. We use this reformulation for ease of exposition (see also [10] and references therein). Note that $\bm{x}\in\mathbb{R}^{\mathcal{T}p},~\ \bm{y}\in\mathbb{R}^{\mathcal{T}n},~\ \mbox{ and }\bm{\Phi}\in\mathbb{R}^{\mathcal{T}n\times\mathcal{T}p}$. We also define $\mathcal{G}=\{G_{1},G_{2},\ldots,G_{M}\}$ to be the set of groups defined on $\mathbb{R}^{\mathcal{T}p}$ formed by aggregating the rows of $\bm{X}$ that were originally in $\tilde{\mathcal{G}}$, so that $\bm{x}$ is composed of groups $G\in\mathcal{G}$.

We next define a regularizer $h$ that promotes sparsity both within and across overlapping sets of similar features:

where the $\alpha_{G}>0$ are constants that balance the tradeoff between the group norms and the $\ell_{1}$ norm. Each $\bm{w}_{G}$ has the same size as $\bm{x}$, with support restricted to the variables indexed by group $G$. $\mathcal{W}$ is a set of vectors, where each vector has a support restricted to one of the groups $G\in\mathcal{G}$:

where $[\bm{w}_{G}]_{i}$ is the $i^{th}$ coefficient of $\bm{w}_{G}$. The SOSlasso is the optimization in (1) with $h(\bm{x})$ as defined in (3).

We say the set of vectors $\bm{w}_{G}$ is an optimal decomposition of $\bm{x}$ if they achieve the $\inf$ in (3). The objective function in (3) is convex and coercive. Hence, $\forall\bm{x}$, an optimal decomposition always exists.

The example in Table 1 gives an insight into the kind of sparsity patterns preferred by the function $h(\bm{x})$. The optimization problems (1) and (2) will prefer solutions that have a small value of $h(\cdot)$. Consider 3 instances of $\bm{x}\in\mathbb{R}^{10}$, and the corresponding group lasso, $\ell_{1}$, and $h(\bm{x})$ function values. The vector is assumed to be made up of two groups, $G_{1}=\{1,2,3,4,5\}\mbox{ and }G_{2}=\{6,7,8,9,10\}$. $h(\bm{x})$ is smallest when the support set is sparse within groups, and also when only one of the two groups is selected. The $\ell_{1}$ norm does not take into account sparsity across groups, while the group lasso norm does not take into account sparsity within groups.

To solve (1) and (2) with the regularizer proposed in (3), we use the covariate duplication method of [6], to reduce the problem to a non overlapping sparse group lasso problem. We then use proximal point methods [8] in conjunction with the MALSAR [22] package to solve the optimization problem.

We derive certain key properties of the regularizer $h(\cdot)$ in (3), independent of the loss function used.

The proof follows from basic properties of norms and because if $\bm{w}_{G},\bm{v}_{G}$ are optimal decompositions of $\bm{x},\bm{y}$, then it does not imply that $\bm{w}_{G}+\bm{v}_{G}$ is an optimal decomposition of $\bm{x}+\bm{y}$. For a detailed proof, please refer to the supplementary material.

The dual norm of $h(\bm{x})$ can be bounded as

(i) follows from the fact that the constraint set in (i) is a superset of the constraint set in the previous statement, since $\|\bm{a}\|_{2}\leq\|\bm{a}\|_{1}$. (4) follows from noting that the maximum is obtained by setting $\bm{w}_{G^{*}}=\frac{\bm{u}_{G^{*}}}{2\|\bm{u}_{G^{*}}\|_{2}}$, where $G^{*}=\arg\max_{G\in\mathcal{G}}\|\bm{u}_{G}\|_{2}$. The inequality (4) is far more tractable than the actual dual norm, and will be useful in our derivations below. Since $h(\cdot)$ is a norm, we can apply methods developed in [11] to derive consistency rates for the optimization problems (1) and (2). We will use the same notations as in [11] wherever possible.

The result follows in a straightforward way from noting that supports of decompositions for vectors in $sA$ and $sB^{\perp}$ do not overlap. We defer the proof to the supplementary material.

Proof For any vector $\bm{x}$ with $supp(\bm{x})\subset\mathcal{G}^{\star}$, there exists a representation $\bm{x}=\sum_{G\in\mathcal{G}^{\star}}\bm{w}_{G}$, such that the supports of the different $\bm{w}_{G}$ do not overlap. Then,

We see that $(1+\sqrt{B\alpha})\sqrt{|\mathcal{G}^{\star}|}$ (Lemma 3.5) gives an upper bound on the subspace compatibility constant with respect to the $\ell_{2}$ norm for the subspace indexed by the support of the vector, which is contained in the span of the union of groups in $\mathcal{G}^{\star}$.

In this paper, we consider vectors $\bm{x}^{\star}$ that lie exactly in $k\ll M$ groups, and display within-group sparsity. This implies that the tolerance $\tau(\bm{x}^{\star})=0$, and we will ignore this term henceforth.

The result above shows a general bound on the error using the lasso with sparse overlapping sets. Note that the regularization parameter $\lambda_{n}$ as well as the RSC constant $\kappa$ depend on the loss function $\mathcal{L}_{\bm{\Phi}}(\bm{x})$. Convergence for logistic regression settings may be derived using methods in [1]. In the next section, we consider the least squares loss (2), and show that the estimate using the SOSlasso is consistent.

We first need to bound the dual norm of the gradient of the loss function, so as to bound $\lambda_{n}$. Consider $\mathcal{L}:=\mathcal{L}_{\bm{\Phi}}(\bm{x})=\frac{1}{2n}\|\bm{y}-\bm{\Phi}\bm{x}\|^{2}$. The gradient of the loss function with respect to $\bm{x}$ is given by $\nabla\mathcal{L}=\frac{1}{n}\bm{\Phi}^{T}(\bm{\Phi}\bm{x}-\bm{y})=\frac{1}{n}\bm{\Phi}^{T}\eta$ where $\eta=[\eta_{1}^{T}\eta_{2}^{T}\ldots\eta_{\mathcal{T}}^{T}]^{T}$ (see Section 2). Our goal now is to find an upper bound on the quantity $h^{*}(\nabla\mathcal{L})$, which from (4) is

where $\bm{\Phi}_{G}$ is the matrix $\bm{\Phi}$ restricted to the columns indexed by the group $G$. We will prove an upper bound for the above quantity in the course of the results that follow.

Proof From the chi-squared tail bound in [3], $\mathbb{P}(z_{i}\geq c^{2}d)\leq\exp\left(-\frac{(c-1)^{2}d}{2}\right)$. The result follows from a union bound and inverting the expression.  

Proof Let $\gamma\sim\chi^{2}_{\mathcal{T}|G|}$. We begin with the upper bound obtained for the dual norm of the regularizer in (4):

where $(i)$ follows from the formulation of the gradient of the loss function and the fact that the square of maximum of non negative numbers is the maximum of the squares of the same numbers. In $(ii)$, we have defined $\sigma_{m}=\max_{G}\sigma_{mG}$. Finally, we have made use of Lemma 4.1 in $(iii)$. We then set

to obtain the result.  

Proof Follows from substituting in Theorem 3.7 the results from Lemma 3.5 and Lemma 4.2.  

From [11], we see that the convergence rate matches that of the group lasso, with an additional multiplicative factor $\alpha$. This stems from the fact that the signal has a sparse structure “embedded” within a group sparse structure. Visualizing the optimization problem as that of solving a lasso within a group lasso framework lends some intuition into this result. Note that since $\alpha<1$, this bound is much smaller than that of the standard group lasso.

We have introduced SOSlasso, a function that recovers sparsity patterns that are a hybrid of overlapping group sparse and sparse patterns when used as a regularizer in convex programs, and proved its theoretical convergence rates when minimizing least squares. The SOSlasso succeeds in a multi-task fMRI analysis, where it both makes better inferences and discovers more theoretically plausible brain regions that lasso and Glasso. Future work involves experimenting with different parameters for the group and l1 penalties, and using other similarity groupings, such as functional connectivity in fMRI.