Feature Selection: A Data Perspective

According to the availability of supervision (such as class labels in classification problems), feature selection can be broadly classified as supervised, unsupervised and semi-supervised methods.

Supervised feature selection is generally designed for classification or regression problems. It aims to select a subset of features that are able to discriminate samples from different classes (classification) or to approximate the regression targets (regression). With supervision information, feature relevance is usually assessed via its correlation with the class labels or the regression target. The training phase highly depends on the selected features: after splitting the data into training and testing sets, classifiers or regression models are trained based on a subset of features selected by supervised feature selection. Note that the feature selection phase can be independent of the learning algorithms (filter methods); or it may iteratively take advantage of the learning performance of a classifier or a regression model to assess the quality of selected features so far (wrapper methods); or make use of the intrinsic structure of a learning algorithm to embed feature selection into the underlying model (embedded methods). Finally, the trained classifier or regression model predicts class labels or regression targets of unseen samples in the test set with the selected features. In the following context, for supervised methods, we mainly focus on classification problems, and use label information, supervision information interchangeably.

Unsupervised feature selection is generally designed for clustering problems. As acquiring labeled data is particularly expensive in both time and efforts, unsupervised feature selection has gained considerable attention recently. Without label information to evaluate the importance of features, unsupervised feature selection methods seek alternative criteria to define feature relevance. Different from supervised feature selection, unsupervised feature selection usually uses all instances that are available in the feature selection phase. The feature selection phase can be independent of the unsupervised learning algorithms (filter methods); or it relies on the learning algorithms to iteratively improve the quality of selected features (wrapper methods); or embed the feature selection phase into unsupervised learning algorithms (embedded methods). After the feature selection phase, it outputs the cluster structure of all data samples on the selected features by using a standard clustering algorithm [Guyon and Elisseeff (2003), Liu and Motoda (2007), Tang et al. (2014a)].

Supervised feature selection works when sufficient label information is available while unsupervised feature selection algorithms do not require any class labels. However, in many real-world applications, we usually have a limited number of labeled data. Therefore, it is desirable to develop semi-supervised methods by exploiting both labeled and unlabeled data samples.

Concerning different selection strategies, feature selection methods can be broadly categorized as wrapper, filter and embedded methods.

Wrapper methods rely on the predictive performance of a predefined learning algorithm to evaluate the quality of selected features. Given a specific learning algorithm, a typical wrapper method performs two steps: (1) search for a subset of features; and (2) evaluate the selected features. It repeats (1) and (2) until some stopping criteria are satisfied. Feature set search component first generates a subset of features; then the learning algorithm acts as a black box to evaluate the quality of these features based on the learning performance. For example, the whole process works iteratively until such as the highest learning performance is achieved or the desired number of selected features is obtained. Then the feature subset that gives the highest learning performance is returned as the selected features. Unfortunately, a known issue of wrapper methods is that the search space for $d$ features is $2^{d}$, which is impractical when $d$ is very large. Therefore, different search strategies such as sequential search [Guyon and Elisseeff (2003)], hill-climbing search, best-first search [Kohavi and John (1997), Arai et al. (2016)], branch-and-bound search [Narendra and Fukunaga (1977)] and genetic algorithms [Golberg (1989)] are proposed to yield a local optimum learning performance. However, the search space is still extremely huge for high-dimensional datasets. As a result, wrapper methods are seldom used in practice.

Filter methods are independent of any learning algorithms. They rely on characteristics of data to assess feature importance. Filter methods are typically more computationally efficient than wrapper methods. However, due to the lack of a specific learning algorithm guiding the feature selection phase, the selected features may not be optimal for the target learning algorithms. A typical filter method consists of two steps. In the first step, feature importance is ranked according to some feature evaluation criteria. The feature importance evaluation process can be either univariate or multivariate. In the univariate scheme, each feature is ranked individually regardless of other features, while the multivariate scheme ranks multiple features in a batch way. In the second step of a typical filter method, lowly ranked features are filtered out. In the past decades, different evaluation criteria for filter methods have been proposed. Some representative criteria include feature discriminative ability to separate samples [Kira and Rendell (1992), Robnik-Šikonja and Kononenko (2003), Yang et al. (2011), Du et al. (2013), Tang et al. (2014b)], feature correlation [Koller and Sahami (1995), Guyon and Elisseeff (2003)], mutual information [Yu and Liu (2003), Peng et al. (2005), Nguyen et al. (2014), Shishkin et al. (2016), Gao et al. (2016)], feature ability to preserve data manifold structure [He et al. (2005), Zhao and Liu (2007), Gu et al. (2011b), Jiang and Ren (2011)], and feature ability to reconstruct the original data [Masaeli et al. (2010), Farahat et al. (2011), Li et al. (2017b)].

Embedded methods is a trade-off between filter and wrapper methods which embed the feature selection into model learning. Thus they inherit the merits of wrapper and filter methods – (1) they include the interactions with the learning algorithm; and (2) they are far more efficient than the wrapper methods since they do not need to evaluate feature sets iteratively. The most widely used embedded methods are the regularization models which target to fit a learning model by minimizing the fitting errors and forcing feature coefficients to be small (or exact zero) simultaneously. Afterwards, both the regularization model and selected feature sets are returned as the final results.

It should be noted that some literature classifies feature selection methods into four categories (from the selection strategy perspective) by including the hybrid feature selection methods [Saeys et al. (2007), Shen et al. (2012), Ang et al. (2016)]. Hybrid methods can be regarded as a combination of multiple feature selection algorithms (e.g., wrapper, filter, and embedded). The main target is to tackle the instability and perturbation issues of many existing feature selection algorithms. For example, for small-sized high-dimensional data, a small perturbation on the training data may result in totally different feature selection results. By aggregating multiple selected feature subsets from different methods together, the results are more robust and hence the credibility of the selected features is enhanced.

Laplacian Score [He et al. (2005)] is an unsupervised feature selection algorithm which selects features that can best preserve the data manifold structure. It consists of three phases. First, it constructs the affinity matrix such that ${\bf S}(i,j)=e^{-\frac{\|{\bf x}_{i}-{\bf x}_{j}\|_{2}^{2}}{t}}$ if $x_{i}$ is among the $p$-nearest neighbor of $x_{j}$; otherwise ${\bf S}(i,j)=0$. Then, the diagonal matrix ${\bf D}$ is defined as ${\bf D}(i,i)=\sum_{j=1}^{n}{\bf S}(i,j)$ and the Laplacian matrix ${\bf L}$ is ${\bf L}={\bf D}-{\bf S}$. Lastly, the Laplacian Score of each feature $f_{i}$ is computed as:

As Laplacian Score evaluates each feature individually, the task of selecting the $k$ features can be solved by greedily picking the top $k$ features with the smallest Laplacian Scores. The Laplacian Score of each feature can be reformulated as:

where $\|{\bf D}^{\frac{1}{2}}\tilde{{\bf f}}_{i}\|_{2}$ is the standard data variance of feature $f_{i}$, and the term $\tilde{{\bf f}}_{i}/\|{\bf D}^{\frac{1}{2}}\tilde{{\bf f}}_{i}\|_{2}$ is interpreted as a normalized feature vector of ${\bf f}_{i}$. Therefore, it is obvious that Laplacian Score is a special case of utility maximization in Eq. (1).

SPEC [Zhao and Liu (2007)] is an extension of Laplacian Score that works for both supervised and unsupervised scenarios. For example, in the unsupervised scenario, the data similarity is measured by RBF kernel; while in the supervised scenario, data similarity can be defined by: $\small\mathbb{S}(i,j)=\left\{\begin{array}[]{ll}\frac{1}{n_{l}}&\text{if }y_{i}=y_{j}=l\\ 0&\text{otherwise}\\ \end{array}\right.$ , where $n_{l}$ is the number of data samples in the $l$-th class. After obtaining the affinity matrix ${\bf S}$ and the diagonal matrix ${\bf D}$, the normalized Laplacian matrix ${\bf L}_{norm}={\bf D}^{-\frac{1}{2}}({\bf D}-{\bf S}){\bf D}^{-\frac{1}{2}}$. The basic idea of SPEC is similar to Laplacian Score: a feature that is consistent with the data manifold structure should assign similar values to instances that are near each other. In SPEC, the feature relevance is measured by three different criteria:

In the above equations, $\hat{{\bf f}_{i}}={\bf D}^{\frac{1}{2}}{\bf f}_{i}/\|{\bf D}^{\frac{1}{2}}{\bf f}_{i}\|_{2}$; $(\lambda_{j},\xi_{j})$ is the $j$-th eigenpair of the normalized Laplacian matrix ${\bf L}_{norm}$; $\alpha_{j}=\cos\theta_{j}$, $\theta_{j}$ is the angle between $\xi_{j}$ and ${\bf f}_{i}$; $\gamma(.)$ is an increasing function to penalize high frequency components of the eigensystem to reduce noise. If the data is noise free, the function $\gamma(.)$ can be removed and $\gamma(x)=x$. When the second evaluation criterion $SPEC\_score2(f_{i})$ is used, SPEC is equivalent to the Laplacian Score. For $SPEC\_score3(f_{i})$, it uses the top $m$ eigenpairs to evaluate the importance of feature $f_{i}$.

All these three criteria can be reduced to the the unified similarity based feature selection framework in Eq. (1) by setting ${\bf\hat{f}}_{i}$ as ${\bf f}_{i}\|{\bf D}^{\frac{1}{2}}{\bf f}_{i}\|_{2}$, $({\bf f}_{i}-\mu{\bf 1})/\|{\bf D}^{\frac{1}{2}}{\bf f}_{i}\|_{2}$, ${\bf f}_{i}\|{\bf D}^{\frac{1}{2}}{\bf f}_{i}\|_{2}$; and ${\bf\hat{S}}$ as ${\bf D}^{\frac{1}{2}}{\bf U}({\bf I}-\gamma({\bf\Sigma})){\bf U}^{\prime}{\bf D}^{\frac{1}{2}}$, ${\bf D}^{\frac{1}{2}}{\bf U}({\bf I}-\gamma({\bf\Sigma})){\bf U}^{\prime}{\bf D}^{\frac{1}{2}}$, ${\bf D}^{\frac{1}{2}}{\bf U}_{m}(\gamma(2{\bf I})-\gamma({\bf\Sigma}_{m})){\bf U}_{m}^{\prime}{\bf D}^{\frac{1}{2}}$ in SPEC_score1, SPEC_score2, SPEC_score3, respectively. ${\bf U}$ and ${\bf\Sigma}$ are the singular vectors and singular values of the normalized Laplacian matrix ${\bf L}_{norm}$.

Fisher Score [Duda et al. (2012)] is a supervised feature selection algorithm. It selects features such that the feature values of samples within the same class are similar while the feature values of samples from different classes are dissimilar. The Fisher Score of each feature $f_{i}$ is evaluated as follows:

where $n_{j}$, $\mu_{i}$, $\mu_{ij}$ and $\sigma_{ij}^{2}$ indicate the number of samples in class $j$, mean value of feature $f_{i}$, mean value of feature $f_{i}$ for samples in class $j$, variance value of feature $f_{i}$ for samples in class $j$, respectively. Similar to Laplacian Score, the top $k$ features can be obtained by greedily selecting the features with the largest Fisher Scores.

According to [He et al. (2005)], Fisher Score can be considered as a special case of Laplacian Score as long as the affinity matrix is $\mathbb{S}(i,j)=\left\{\begin{array}[]{ll}\frac{1}{n_{l}}&\text{if }y_{i}=y_{j}=l\\ 0&\text{otherwise,}\\ \end{array}\right.$. In this way, the relationship between Fisher Score and Laplacian Score is $fisher\_score(f_{i})=1-\frac{1}{laplacian\_score(f_{i})}$. Hence, the computation of Fisher Score can also be reduced to the unified utility maximization framework.

The trace ratio criterion [Nie et al. (2008)] directly selects the global optimal feature subset based on the corresponding score, which is computed by a trace ratio norm. It builds two affinity matrices ${\bf S}_{w}$ and ${\bf S}_{b}$ to characterize within-class and between-class data similarity. Let ${\bf W}=[{\bf w}_{i_{1}},{\bf w}_{i_{2}},...,{\bf w}_{i_{k}}]\in\mathbb{R}^{d\times k}$ be the selection indicator matrix such that only the $i_{j}$-th entry in ${\bf w}_{i_{j}}$ is 1 and all the other entries are 0. With these, the trace ratio score of the selected $k$ features in $\mathcal{S}$ is:

where ${\bf L}_{b}$ and ${\bf L}_{w}$ are Laplacian matrices of ${\bf S}_{a}$ and ${\bf S}_{b}$ respectively. The basic idea is to maximize the data similarity for instances from the same class while minimize the data similarity for instances from different classes. However, the trace ratio problem is difficult to solve as it does not have a closed-form solution. Hence, the trace ratio problem is often converted into a more tractable format called the ratio trace problem by maximizing $tr[({\bf W}^{\prime}{\bf X}^{\prime}{\bf L}_{w}{\bf X}{\bf W})^{-1}({\bf W}^{\prime}{\bf X}^{\prime}{\bf L}_{b}{\bf X}{\bf W})]$. As an alternative, [Wang et al. (2007)] propose an iterative algorithm called ITR to solve the trace ratio problem directly and was later applied in trace ratio feature selection [Nie et al. (2008)].

Different ${\bf S}_{b}$ and ${\bf S}_{w}$ lead to different feature selection algorithms such as batch-mode Lalpacian Score and batch-mode Fisher Score. For example, in batch-mode Fisher Score, the within-class data similarity and the between-class data similarity are ${\bf S}_{w}(i,j)=\left\{\begin{array}[]{ll}1/n_{l}&\text{if }y_{i}=y_{j}=l\\ 0&\text{otherwise}\\ \end{array}\right.$ and ${\bf S}_{b}(i,j)=\left\{\begin{array}[]{ll}1/n-1/n_{l}&\text{if }y_{i}=y_{j}=l\\ 1/n&\text{otherwise}\\ \end{array}\right.$ respectively. Therefore, maximizing the trace ratio criterion is equivalent to maximizing $\frac{\sum_{s=1}^{k}{\bf f}_{i_{s}}^{\prime}{\bf S}_{w}{\bf f}_{i_{s}}}{\sum_{s=1}^{k}{\bf f}_{i_{s}}^{\prime}{\bf f}_{i_{s}}}=\frac{{\bf X}_{\mathcal{S}}^{\prime}{\bf S}_{w}{\bf X}_{\mathcal{S}}}{{\bf X}_{\mathcal{S}}^{\prime}{\bf X}_{\mathcal{S}}}.$ Since ${\bf X}_{\mathcal{S}}^{\prime}{\bf X}_{\mathcal{S}}$ is constant, it can be further reduced to the unified similarity based feature selection framework by setting ${\bf\hat{f}}={\bf f}/\|{\bf f}\|_{2}$ and ${\bf\hat{S}}={\bf S}_{w}$. On the other hand in batch-mode Laplacian Score, the within-class data similarity and the between-class data similarity are ${\bf S}_{w}(i,j)=\left\{\begin{array}[]{ll}e^{-\frac{\|{\bf x}_{i}-{\bf x}_{j}\|_{2}^{2}}{t}}&\text{if }{\bf x}_{i}\in\mathcal{N}_{p}({\bf x}_{j})\text{ or }{\bf x}_{j}\in\mathcal{N}_{p}({\bf x}_{i})\\ 0&\text{otherwise}\\ \end{array}\right.$ and ${\bf S}_{b}=({\bf 1}^{\prime}{\bf D}_{w}{\bf 1})^{-1}{\bf D}_{w}{\bf 1}{\bf 1}^{\prime}{\bf D}_{w}$ respectively. In this case, the trace ratio criterion score is $\frac{tr({\bf W}^{\prime}{\bf X}^{\prime}{\bf L}_{b}{\bf X}{\bf W})}{tr({\bf W}^{\prime}{\bf X}^{\prime}{\bf L}_{w}{\bf X}{\bf W})}=\frac{\sum_{s=1}^{k}{\bf f}_{i_{s}}^{\prime}{\bf D}_{w}{\bf f}_{i_{s}}}{\sum_{s=1}^{k}{\bf f}_{i_{s}}^{\prime}({\bf D}_{w}-{\bf S}_{w}){\bf f}_{i_{s}}}$. Therefore, maximizing the trace ratio criterion is also equivalent to solving the unified maximization problem in Eq. (1) where ${\bf\hat{f}}={\bf f}/\|{\bf D}_{w}^{\frac{1}{2}}{\bf f}\|_{2}$ and ${\bf\hat{S}}={\bf S}_{w}$.

ReliefF [Robnik-Šikonja and Kononenko (2003)] selects features to separate instances from different classes. Assume that $l$ data instances are randomly selected among all $n$ instances, then the feature score of $f_{i}$ in ReliefF is defined as follows:

where NH$(j)$ and NM$(j,y)$ are the nearest instances of $x_{j}$ in the same class and in class $y$, respectively. Their sizes are $m_{j}$ and $h_{jy}$, respectively. $p(y)$ is the ratio of instances in class $y$.

ReliefF is equivalent to selecting features that preserve a special form of data similarity matrix. Assume that the dataset has the same number of instances in each of the $c$ classes and there are $q$ instances in both $NM(j)$ and $NH(j,y)$. Then according to [Zhao and Liu (2007)], the ReliefF feature selection can be reduced to the utility maximization framework in Eq. (1).

Mutual Information Maximization (MIM) (a.k.a. Information Gain) [Lewis (1992)] measures the importance of a feature by its correlation with class labels. It assumes that when a feature has a strong correlation with the class label, it can help achieve good classification performance. The Mutual Information score for feature $X_{k}$ is:

It can be observed that in MIM, the scores of features are assessed individually. Therefore, only the feature correlation is considered while the feature redundancy is completely ignored. After it obtains the MIM feature scores for all features, we choose the features with the highest feature scores and add them to the selected feature set. The process repeats until the desired number of selected features is obtained.

It can also be observed that MIM is a special case of linear combination of Shannon information terms in Eq. (13) where both $\beta$ and $\lambda$ are equal to zero.

A limitation of MIM criterion is that it assumes that features are independent of each other. In reality, good features should not only be strongly correlated with class labels but also should not be highly correlated with each other. In other words, the correlation between features should be minimized. Mutual Information Feature Selection (MIFS) [Battiti (1994)] considers both the feature relevance and feature redundancy in the feature selection phase, the feature score for a new unselected feature $X_{k}$ can be formulated as follows:

In MIFS, the feature relevance is evaluated by $I(X_{k};Y)$, while the second term penalizes features that have a high mutual information with the currently selected features such that feature redundancy is minimized.

MIFS can also be reduced to be a special case of the linear combination of Shannon information terms in Eq. (13) where $\beta$ is between zero and one, and $\lambda$ is zero.

[Peng et al. (2005)] proposes a Minimum Redundancy Maximum Relevance (MRMR) criterion to set the value of $\beta$ to be the reverse of the number of selected features:

Hence, with more selected features, the effect of feature redundancy is gradually reduced. The intuition is that with more non-redundant features selected, it becomes more difficult for new features to be redundant to the features that have already been in $\mathcal{S}$. In [Brown et al. (2012)], it gives another interpretation that the pairwise independence between features becomes stronger as more features are added to $\mathcal{S}$, possibly because of noise information in the data.

MRMR is also strongly linked to the Conditional likelihood maximization framework if we iteratively revise the value of $\beta$ to be $\frac{1}{|\mathcal{S}|}$, and set the other parameter $\lambda$ to be zero.

Some studies [Lin and Tang (2006), El Akadi et al. (2008), Guo and Nixon (2009)] show that in contrast to minimize the feature redundancy, the conditional redundancy between unselected features and already selected features given class labels should also be maximized. In other words, as long as the feature redundancy given class labels is stronger than the intra-feature redundancy, the feature selection will be affected negatively. A typical feature selection under this argument is Conditional Infomax Feature Extraction (CIFE) [Lin and Tang (2006)], in which the feature score for a new unselected feature $X_{k}$ is:

Compared with MIFS, it adds a third term $\sum_{X_{j}\in\mathcal{S}}I(X_{j};X_{k}|Y)$ to maximize the conditional redundancy. Also, CIFE is a special case of the linear combination of Shannon information terms by setting both $\beta$ and $\gamma$ to be 1.

MIFS and MRMR reduce feature redundancy in the feature selection process. An alternative criterion, Joint Mutual Information [Yang and Moody (1999), Meyer et al. (2008)] is proposed to increase the complementary information that is shared between unselected features and selected features given the class labels. The feature selection criterion is listed as follows:

The basic idea of JMI is that we should include new features that are complementary to the existing features given the class labels.

JMI cannot be directly reduced to the condition likelihood maximization framework. In [Brown et al. (2012)], the authors demonstrate that with simple manipulations, the JMI criterion can be re-written as:

Therefore, it is also a special case of the linear combination of Shannon information terms by iteratively setting $\beta$ and $\lambda$ to be $\frac{1}{|\mathcal{S}|}$.

Previously mentioned criteria could be reduced to a linear combination of Shannon information terms. Next, we show some other algorithms that can only be reduced to a non-linear combination of Shannon information terms. Among them, Conditional Mutual Information Maximization (CMIM) [Vidal-Naquet and Ullman (2003), Fleuret (2004)] iteratively selects features which maximize the mutual information with the class labels given the selected features so far. Mathematically, during the selection phase, the feature score for each new unselected feature $X_{k}$ can be formulated as follows:

Note that the value of $I(X_{k};Y|X_{j})$ is small if $X_{k}$ is not strongly correlated with the class label $Y$ or if $X_{k}$ is redundant when $\mathcal{S}$ is known. By selecting the feature that maximizes this minimum value, it can guarantee that the selected feature has a strong predictive ability, and it can reduce the redundancy w.r.t. the selected features.

The CMIM criterion is equivalent to the following form after some derivations:

Therefore, CMIM is also a special case of the conditional likelihood maximization framework in Eq. (12).

In [Vidal-Naquet and Ullman (2003)], the authors propose a feature selection criterion called Informative Fragments (IF). The feature score of each new unselected features is given as:

The intuition behind Informative Fragments is that the addition of the new feature $X_{k}$ should maximize the value of conditional mutual information between $X_{k}$ and existing features in $\mathcal{S}$ over the mutual information between $X_{j}$ and $Y$. An interesting phenomenon of IF is that with the chain rule that $I(X_{k}X_{j};Y)=I(X_{j};Y)+I(X_{k};Y|X_{j})$, IF has the equivalent form as CMIM. Hence, it can also be reduced to the general framework in Eq. (12).

Interaction Capping [Jakulin (2005)] is a similar feature selection criterion as CMIM in Eq. (21), it restricts the term $I(X_{j};X_{k})-I(X_{j};X_{k}|Y)$ to be nonnegative:

Apparently, it is a special case of non-linear combination of Shannon information terms by setting the function $g(.)$ to be $-\max[0,I(X_{j};X_{k})-I(X_{j};X_{k}|Y)]$.

Another class of information theoretical based methods such as Double Input Symmetrical Relevance (DISR) [Meyer and Bontempi (2006)] exploits normalization techniques to normalize mutual information [Guyon et al. (2008)]:

It is easy to validate that DISR is a non-linear combination of Shannon information terms and can be reduced to the conditional likelihood maximization framework.

There are other information theoretical based feature selection methods that cannot be simply reduced to the unified conditional likelihood maximization framework. Fast Correlation Based Filter (FCBF) [Yu and Liu (2003)] is an example that exploits feature-class correlation and feature-feature correlation simultaneously. The algorithm works as follows: (1) given a predefined threshold $\delta$, it selects a subset of features $\mathcal{S}$ that are highly correlated with the class labels with $SU\geq\delta$, where $SU$ is the symmetric uncertainty. The $SU$ between a set of features $X_{\mathcal{S}}$ and the class label $Y$ is given as follows:

A specific feature $X_{k}$ is called predominant iff $SU(X_{k},Y)\geq\delta$ and there does not exist a feature $X_{j}\in\mathcal{S}$ $(j\neq k)$ such that $SU(X_{j},X_{k})\geq SU(X_{k},Y)$. Feature $X_{j}$ is considered to be redundant to feature $X_{k}$ if $SU(X_{j},X_{k})\geq SU(X_{k},Y)$; (2) the set of redundant features is denoted as $\mathcal{S}_{P_{i}}$, which will be further split into $\mathcal{S}_{P_{i}}^{+}$ and $\mathcal{S}_{P_{i}}^{-}$ where they contain redundant features to feature $X_{k}$ with $SU(X_{j},Y)>SU(X_{k},Y)$ and $SU(X_{j},Y)<SU(X_{k},Y)$, respectively; and (3) different heuristics are applied on $\mathcal{S}_{P}$, $\mathcal{S}_{P_{i}}^{+}$ and $\mathcal{S}_{P_{i}}^{-}$ to remove redundant features and keep the features that are most relevant to the class labels.

First, we consider the binary classification or univariate regression problem. To achieve feature selection, the $\ell_{p}$-norm sparsity-induced penalty term is added on the classification or regression model, where $0\leq p\leq 1$. Let ${\bf w}$ denotes the feature coefficient, then the objective function for feature selection is:

where $loss(.)$ is a loss function, and some widely used loss functions $loss(.)$ include least squares loss, hinge loss and logistic loss. $\|{\bf w}\|_{p}=(\sum_{i=1}^{d}\|w_{i}\|^{p})^{\frac{1}{p}}$ is a sparse regularization term, and $\alpha$ is a regularization parameter to balance the contribution of the loss function and the sparse regularization term for feature selection.

Typically when $p=0$, the $\ell_{0}$-norm regularization term directly seeks for the optimal set of nonzero entries (features) for the model learning. However, the optimization problem is naturally an integer programming problem and is difficult to solve. Therefore, it is often relaxed to a $\ell_{1}$-norm regularization problem, which is regarded as the tightest convex relaxation of the $\ell_{0}$-norm. One main advantage of $\ell_{1}$-norm regularization (LASSO) [Tibshirani (1996)] is that it forces many feature coefficients to become smaller and, in some cases, exactly zero. This property makes it suitable for feature selection, as we can select features whose corresponding feature weights are large, which motivates a surge of $\ell_{1}$-norm regularized feature selection methods [Zhu et al. (2004), Xu et al. (2014), Wei et al. (2016a), Wei and Yu (2016), Hara and Maehara (2017)]. Also, the sparse vector ${\bf w}$ enables the ranking of features. Normally, the higher the value, the more important the corresponding feature is.

Here, we discuss how to perform feature selection for the general multi-class classification or multivariate regression problems. The problem is more difficult because of the multiple classes and multivariate regression targets, and we would like the feature selection phase to be consistent over multiple targets. In other words, we want multiple predictive models for different targets to share the same parameter sparsity patterns – each feature either has small scores or large scores for all targets. This problem can be generally solved by the $\ell_{p,q}$-norm sparsity-induced regularization term, where $p>1$ (most existing work focus on $p=2$ or $\infty$) and $0\leq q\leq 1$ (most existing work focus on $q=1$ or $0$). Assume that ${\bf X}$ denotes the data matrix, and ${\bf Y}$ denotes the one-hot label indicator matrix. Then the model is formulated as follows:

where $\|{\bf W}\|_{p,q}=(\sum_{j=1}^{c}(\sum_{i=1}^{d}|{\bf W}(i,j)|^{p})^{\frac{q}{p}})^{\frac{1}{q}}$; and the parameter $\alpha$ is used to control the contribution of the loss function and the sparsity-induced regularization term. Then the features can be ranked according to the value of $\|{\bf W}(i,:)\|_{2}^{2}({i=1,...,d})$, the higher the value, the more important the feature is.

Authors in [Nie et al. (2010)] propose an efficient and robust feature selection (REFS) method by employing a joint $\ell_{2,1}$-norm minimization on both the loss function and the regularization. Their argument is that the $\ell_{2}$-norm based loss function is sensitive to noisy data while the $\ell_{2,1}$-norm based loss function is more robust to noise. The reason is that $\ell_{2,1}$-norm loss function has a rotational invariant property [Ding et al. (2006)]. Consistent with $\ell_{2,1}$-norm regularized feature selection model, a $\ell_{2,1}$-norm regularizer is added to the $\ell_{2,1}$-norm loss function to achieve group feature sparsity. The objective function of REFS is:

To solve the convex but non-smooth optimization problem, an efficient algorithm is proposed with strict convergence analysis.

It should be noted that the aforementioned REFS is designed for multi-class classification problems where each instance only has one class label. However, data could be associated with multiple labels in many domains such as information retrieval and multimedia annotation. Recently, there is a surge of research work study multi-label feature selection problems by considering label correlations. Most of them, however, are also based on the $\ell_{2,1}$-norm sparse regularization framework [Gu et al. (2011a), Chang et al. (2014), Jian et al. (2016)].

Most of existing sparse learning based approaches build a learning model with the supervision of class labels. The feature selection phase is derived afterwards on the sparse feature coefficients. However, since labeled data is costly and time-consuming to obtain, unsupervised sparse learning based feature selection has received increasing attention in recent years. Multi-Cluster Feature Selection (MCFS) [Cai et al. (2010)] is one of the first attempts. Without class labels to guide the feature selection process, MCFS proposes to select features that can cover multi-cluster structure of the data where spectral analysis is used to measure the correlation between different features.

MCFS consists of three steps. In the first step, it constructs a $p$-nearest neighbor graph to capture the local geometric structure of data and gets the graph affinity matrix ${\bf S}$ and the Laplacian matrix ${\bf L}$. Then a flat embedding that unfolds the data manifold can be obtained by spectral clustering techniques. In the second step, since the embedding of data is known, MCFS takes advantage of them to measure the importance of features by a regression model with a $\ell_{1}$-norm regularization. Specifically, given the $i$-th embedding ${\bf e}_{i}$, MCFS regards it as a regression target to minimize:

where ${\bf w}_{i}$ denotes the feature coefficient vector for the $i$-th embedding. By solving all $K$ sparse regression problems, MCFS obtains $K$ sparse feature coefficient vectors ${\bf W}=[{\bf w}_{1},...,{\bf w}_{K}]$ and each vector corresponds to one embedding of ${\bf X}$. In the third step, for each feature $f_{j}$, the MCFS score for that feature can be computed as $\text{\emph{MCFS}}(j)=\max_{i}|{\bf W}(j,i)|$. The higher the MCFS score, the more important the feature is.

In [Yang et al. (2011)], the authors propose a new unsupervised feature selection algorithm (UDFS) to select the most discriminative features by exploiting both the discriminative information and feature correlations. First, assume $\tilde{{\bf X}}$ is the centered data matrix such $\tilde{{\bf X}}={\bf H}_{n}{\bf X}$ and ${\bf G}=[{\bf G}_{1},{\bf G}_{1},...,{\bf G}_{n}]^{\prime}={\bf Y}({\bf Y}^{\prime}{\bf Y})^{-\frac{1}{2}}$ is the weighted label indicator matrix, where ${\bf H}_{n}={\bf I}_{n}-\frac{1}{n}{\bf 1}_{n}{\bf 1}_{n}^{\prime}$. Instead of using global discriminative information, they propose to utilize the local discriminative information to select discriminative features. The advantage of using local discriminative information are two folds. First, it has been demonstrated to be more important than global discriminative information in many classification and clustering tasks. Second, when it considers the local discriminative information, the data manifold structure is also well preserved. For each data instance $x_{i}$, it constructs a $p$-nearest neighbor set for that instance $\mathcal{N}_{p}(x_{i})=\{x_{i_{1}},x_{i_{2}},...,x_{i_{p}}\}$. Let ${\bf X}_{\mathcal{N}_{p}(i)}=[{\bf x}_{i},{\bf x}_{i_{1}},...,{\bf x}_{i_{p}}]$ denotes the local data matrix around $x_{i}$, then the local total scatter matrix ${\bf S}_{t}^{(i)}$ and local between class scatter matrix ${\bf S}_{b}^{(i)}$ are $\tilde{{\bf X}_{i}}^{\prime}\tilde{{\bf X}_{i}}$ and $\tilde{{\bf X}_{i}}^{\prime}{\bf G}_{i}{\bf G}_{i}^{\prime}\tilde{{\bf X}_{i}}$ respectively, where $\tilde{{\bf X}_{i}}$ is the centered data matrix and ${\bf G}_{(i)}=[{\bf G}_{i},{\bf G}_{i_{1}},...,{\bf G}_{i_{k}}]^{\prime}$. Note that ${\bf G}_{(i)}$ is a subset from ${\bf G}$ and ${\bf G}_{(i)}$ can be obtained by a selection matrix ${\bf P}_{i}\in\{0,1\}^{n\times(k+1)}$ such that ${\bf G}_{(i)}={\bf P}_{i}^{\prime}{\bf G}$. Without label information in unsupervised feature selection, UDFS assumes that there is a linear classifier ${\bf W}\in\mathbb{R}^{d\times s}$ to map each data instance ${\bf x}_{i}\in\mathbb{R}^{d}$ to a low dimensional space ${\bf G}_{i}\in\mathbb{R}^{s}$. Following the definition of global discriminative information [Yang et al. (2010), Fukunaga (2013)], the local discriminative score for each instance $x_{i}$ is :