Distributionally Robust Safe Screening

The DR setting has been explored in various ML problems, aiming to enhance model robustness against data distribution variations. A DR learning problem is typically formulated as a worst-case optimization problem since the goal of DR learning is to ensure model performance under the worst-case data distribution within a specified range. Hence, a variety of optimization techniques tailored to DR learning have been investigated within both the ML and optimization communities [8, 9, 1]. The proposed DRSS method is one of such DR learning methods, focusing specifically on the problem of sample/feature deletion. The ability to identify irrelevant samples/features is of practical significance. For example, in the context of continual learning (see, e.g., [10]), it is crucial to effectively manage data by selectively retaining and discarding samples/features, especially in anticipation of changes in future data distributions. Incorrect deletion of essential data can lead to catastrophic forgetting [11], a phenomenon where a ML model, after being trained on new data, quickly loses information previously learned from older datasets. The proposed DRSS method tackles this challenge by identifying samples/features that, regardless of future data distribution shifts, will not have any influence on all possible newly trained model in the future.

SS refers to optimization techniques in sparse learning that identify and exclude irrelevant samples or features from the learning process. SS can reduce computational cost without changing the final trained model. Initially, SfS was introduced by [2] for the Lasso. Subsequently, SsS was proposed by [3] for the SVM. Among various SS methods developed so far, the most commonly used is based on the duality gap [12, 13]. Our proposed DRSS method also adopts this approach. Over the past decade, SS has seen diverse developments, including methodological improvements and expanded application scopes [14, 15, 16, 17, 18, 19, 20, 21]. Unlike other SS studies that primarily focused on reducing computational costs, this study adopts SS for a different purpose. We employ SS across scenarios where data distribution varies within a defined range, aiming to discard unnecessary samples/features. To our knowledge, no existing studies have utilized SS within the DR learning framework.

We mainly assume the weighted regularized empirical risk minimization (weighted RERM) for linear prediction. This may include kernelized versions, which are discussed in Appendix C. Suppose that we learn the model parameters as linear prediction coefficients, that is, learn $\bm{\beta}^{*(\bm{w})}\in\mathbb{R}^{d}$ such that the outcome for a sample $\bm{x}\in\mathbb{R}^{d}$ is predicted as $\bm{x}^{\top}\bm{\beta}^{*(\bm{w})}$.

Since $\ell$ and $\rho$ are convex, we can easily confirm that $P_{\bm{w}}(\bm{\beta})$ is convex with respect to $\bm{\beta}$.

Applying Fenchel’s duality theorem (Appendix A.2), we have the following dual problem of (1):

where $\bm{\gamma}$ is a positive-valued vector. The relationship between the original problem (1) (called the primal problem) and the dual problem (2) are described as follows:

In weighted RERM, we call that a loss function $\ell$ induces sample-sparsity if elements in $\bm{\alpha}^{*(\bm{w})}$ are easy to become zero. Due to (5), this can be achieved by $\ell$ such that $\{t\in\mathbb{R}\mid 0\in\partial\ell_{y}(t)\}$ is not a point but an interval.

Similarly, we call that a regularization function $\rho$ induces feature-sparsity if elements in $\bm{\beta}^{*(\bm{w})}$ are easy to become zero. Due to (4), this can be achieved by $\rho$ such that $\{\bm{v}\in\mathbb{R}^{d}\mid\exists j\in[d-1]:~{}0\in[\partial\rho^{*}(\bm{v})]_{j}\}$ is not a point but a region.

For example, the hinge loss $\ell_{y}(t)=\max\{0,1-t\}$ ($y\in\{-1,+1\}$) is a sample-sparse loss function since $\{t\in\mathbb{R}\mid 0\in\partial\ell_{y}(t)\}=[1,+\infty)$. Similarly, the L1-regularization $\rho(\bm{v})=\lambda\sum_{j=1}^{d-1}|v_{j}|$ ($\lambda>0$: hyperparameter) is a feature-sparse regularization function since $\{\bm{v}\in\mathbb{R}^{d}\mid\exists j\in[d-1]:~{}0\in[\partial\rho^{*}(\bm{v})]_{j}\}=\{\bm{v}\in\mathbb{R}^{d}\mid\exists j\in[d-1]:~{}|v_{j}|\leq\lambda,~{}v_{d}=0\}$. See Section 4 for examples of using them.

We consider identifying training samples that do not affect the training result $\bm{\beta}^{*(\bm{w})}$. Due to the relationship (4), if there exists $i\in[n]$ such that $\alpha^{*(\bm{w})}_{i}=0$, then the $i^{\mathrm{th}}$ row (sample) in $\check{X}$ does not affect $\bm{\beta}^{*(\bm{w})}$. However, since computing $\bm{\alpha}^{*(\bm{w})}$ is as costly as $\bm{\beta}^{*(\bm{w})}$, it is difficult to use the relationship as it is. To solve the problem, the SsS first considers identifying the possible region ${\cal B}^{*(\bm{w})}\subset\mathbb{R}^{d}$ such that $\bm{\beta}^{*(\bm{w})}\in{\cal B}^{*(\bm{w})}$ is assured. Then, with ${\cal B}^{*(\bm{w})}$ and (5), we can conclude that the $i^{\mathrm{th}}$ training sample do not affect the training result $\bm{\beta}^{*(\bm{w})}$ if $\bigcup_{\bm{\beta}\in{\cal B}^{*(\bm{w})}}\partial\ell_{y_{i}}(\check{X}_{i:}\bm{\beta})=\{0\}$.

First we show how to compute ${\cal B}^{*(\bm{w})}$. In this paper we adopt the computation methods that is available when the regularization function $\rho$ in $P_{\bm{w}}$ (and also $P_{\bm{w}}$ itself) of (1) are strongly convex.

The proof is presented in Appendix A.3. The amount $P_{\bm{w}}(\hat{\bm{\beta}})-D_{\bm{w}}(\hat{\bm{\alpha}})$ is known as the duality gap, which must be nonnegative due to (3). So we obtain the following gap safe sample screening rule from Lemma 3.1:

The proof is presented in Appendix A.4.

We consider identifying $j\in[d]$ such that $\beta_{j}^{*(\bm{w})}=0$, that is, identifying that the $j^{\mathrm{th}}$ feature is not used in the prediction, even when the sample weights $\bm{w}$ are changed.

For simplicity, suppose that the regularization function $\rho$ is decomposable, that is, $\rho$ is represented as $\rho(\bm{\beta}):=\sum_{j=1}^{d}\sigma_{j}(\beta_{j})$ ($\sigma_{1},\sigma_{2},\dots,\sigma_{d}$: $\mathbb{R}\to\mathbb{R}$). Then, since $\rho^{*}(\bm{v})=\sum_{j=1}^{d}\sigma^{*}_{j}(v_{j})$ and therefore $[\partial\rho^{*}(\bm{v})]_{j}=\partial\sigma^{*}_{j}(v_{j})$, from (4) we have

If we know $\bm{\alpha}^{*(\bm{w})}$, we can identify whether $\beta_{j}^{*(\bm{w})}=0$ holds. However, like SsS (Section 3.1), we would like to check the condition without computing $\bm{\alpha}^{*(\bm{w})}$ or $\bm{\beta}^{*(\bm{w})}$.

So, like SsS, SfS first considers identifying the possible region ${\cal A}^{*(\bm{w})}\subset\mathbb{R}^{n}$ such that $\bm{\alpha}^{*(\bm{w})}\in{\cal A}^{*(\bm{w})}$ is assured. Then we can conclude that $\beta^{*(\bm{w})}_{j}=0$ is assured if $\bigcup_{\bm{\alpha}\in{\cal A}^{*(\bm{w})}}\partial\sigma^{*}_{j}(\check{\check{X}}_{:j}^{(\bm{\gamma},\bm{w})\top}\bm{\alpha})=\{0\}$.

Then we show how to compute ${\cal A}^{*(\bm{w})}$. With Lemma A.3, we can calculate ${\cal A}^{*(\bm{w})}$ as follows, if the loss function $\ell_{y}$ in $P_{\bm{w}}$ of (1) is smooth:

The proof is presented in Appendix A.5. Similar to Lemma 3.2, we obtain the gap safe feature screening rule from Lemma 3.3:

The proof is almost same as Lemma 3.2.

In Sections 3.1 and 3.2 we showed the conditions when samples or features are screened out. In this section we show how to use the conditions for the change of sample weights $\bm{w}$.

For Definition 3.5, we have only to apply SS rules in Lemma 3.2 or 3.4 by setting $\hat{\bm{\beta}}\leftarrow\bm{\beta}^{*(\tilde{\bm{w}})}$ and $\hat{\bm{\alpha}}\leftarrow\bm{\alpha}^{*(\tilde{\bm{w}})}$. On the other hand, for Definition 3.6, we need to maximize or minimize the interval in Lemma 3.2 or 3.4 in $\bm{w}\in{\cal W}$.

Thus, solving the maximizations and/or minimizations in Theorem 3.7 provides DRSsS and DRSfS rules. However, how to solve it largely depends on the choice of $\ell$, $\rho$ and ${\cal W}$. In Section 4 we show specific calculations of Theorem 3.7 for some typical setups.

L1-loss L2-regularized SVM is a sample-sparse model for binary classification ($\bm{y}\in\{-1,+1\}^{n}$) that satisfies the preconditions to apply SsS (Lemma 3.1). Detailed calculations are presented in Appendix B.1.

For L1-loss L2-regularized SVM, we set $\rho$ and $\ell$ as:

Then $\rho$ is $\lambda$-strongly convex. Setting $\bm{\gamma}=\bm{1}_{n}$, the dual objective function is described as

Here, in the viewpoint of minimization, we may consider this problem as a maximization with the constraint “$\forall i\in[n]:0\leq\alpha_{i}\leq 1$”.

Optimality conditions (4) and (5) are described as:

Noticing that ${\cal Z}[\ell_{y_{i}}]=(1,+\infty)$, by Theorem 3.7, the DRSsS rule for ${\cal W}$ is calculated as:

Here, we can find that $P_{\bm{w}}(\bm{\beta}^{*(\tilde{\bm{w}})})-D_{\bm{w}}(\bm{\alpha}^{*(\tilde{\bm{w}})})$, which we need to maximize in reality, is the sum of linear function and convex quadratic function with respect to $\bm{w}\in{\cal W}$. (Since $(\bm{\alpha}^{*(\tilde{\bm{w}})}{\raisebox{0.80002pt}{$\times$}\Box}\check{X})(\bm{\alpha}^{*(\tilde{\bm{w}})}{\raisebox{0.80002pt}{$\times$}\Box}\check{X})^{\top}$ is positive semidefinite, we know that it is convex quadratic). Although constrained maximization of a convex function is difficult in general, for this case we can algorithmically maximize it (Section 4.3).

L2-loss L1-regularized SVM is a feature-sparse model for binary classification ($\bm{y}\in\{-1,+1\}^{n}$) that satisfies the preconditions to apply SfS (Lemma 3.3). Detailed calculations are presented in Appendix B.2.

For L2-loss L1-regularized SVM, we set $\sigma_{j}$ (and consequently $\rho$) and $\ell$ as:

Notice that $\sigma_{d}(\beta_{d})$ is not defined as $\lambda|\beta_{d}|$ but $0$: we rarely regularize the intercept with L1-regularization.

Setting $\bm{\gamma}=\lambda\bm{1}_{n}$, the dual objective function is described as

Optimality conditions (4) and (5) are described as

Noticing that ${\cal Z}[\sigma_{j}^{*}]=(-\lambda,\lambda)$, by Theorem 3.7, the DRSfS rule for ${\cal W}$ is calculated as:

where

Here, the expressions in $\underline{L}$ and $\overline{L}$ are linear with respect to $\bm{w}$, and the expression in $N$ inside the square root is convex and quadratic with respect to $\bm{w}$. Also, $\overline{R}$ is decomposed to two maximizations $\frac{2\mu}{\min_{i\in[n]}w_{i}\gamma_{i}^{2}}$ and $P_{\bm{w}}(\bm{\beta}^{*(\bm{w})})-D_{\bm{w}}(\bm{\alpha}^{*(\bm{w})})$, where the former is easily computed while the latter is linear with respect to $\bm{w}$. So, similar to L1-loss L2-regularized SVM, we can obtain the maximization result by maximizing or minimizing the linear terms by Lemma A.4 in Appendix A, and maximizing the convex quadratic function by the method of Section 4.3.

To derive DRSS rules of Sections 4.1 and 4.2, we need to compute the following forms of optimization problems:

The proof is presented in Appendix A.6.