Attribute-Efficient Learning of Halfspaces with Malicious Noise: Near-Optimal Label Complexity and Noise Tolerance

We informally present our main results below; readers are referred to Theorem 4 in Section 4 for a precise statement.

First of all, note that the noise tolerance is near-optimal as [KL88] showed that a noise rate greater than $\frac{\epsilon}{1+\epsilon}$ cannot be tolerated by any algorithm regardless of the computational power. The following fact establishes the optimality of our label complexity.

To see this lemma, observe that there exist $\epsilon$-packings of $s$-sparse halfspaces with sizes $(\frac{1}{\epsilon})^{\Omega(s)}$ [Lon95] and $(\frac{d}{s})^{\Omega(s)}$ [RWY11]; applying Theorem 1 of [KMT93] gives the lower bound.

[KL88] presented a general analysis on efficiently learning halfspaces, showing that even without any distributional assumptions, it is possible to tolerate the malicious noise at a rate of ${\Omega}(\epsilon/d)$, but a noise rate greater than $\frac{\epsilon}{1+\epsilon}$ cannot be tolerated. The noise model was further studied by [Sch92, Bsh98, CDF⁺99], and [KKMS05] obtained a noise tolerance $\Omega(\epsilon/d^{1/4})$ when $D$ is the uniform distribution. [KLS09] improved this result to $\Omega(\epsilon^{2}/\log(d/\epsilon))$ for the uniform distribution, and showed a noise tolerance $\Omega(\epsilon^{3}/\log^{2}(d/\epsilon))$ for isotropic log-concave distributions. A near-optimal result of $\Omega(\epsilon)$ was established in [ABL17] for both uniform and isotropic log-concave distributions.

Achieving attribute efficiency has been a long-standing goal in machine learning and statistics [Blu90, BHL95], and has found a variety of applications with strong theoretical backend. A partial list includes online classification [Lit87], learning decision lists [Ser99, KS04, LS06], compressed sensing [Don06, CW08, TW10, SL18], one-bit compressed sensing [BB08, PV16], and variable selection [FL01, FF08, SL17a, SL17b].

Label-efficient learning has also been broadly studied since gathering high quality labels is often expensive. The prominent approaches include disagreement-based active learning [Han11, Han14], margin-based active learning [BBZ07, BL13, YZ17], selective sampling [CCG11, DGS12], and adaptive one-bit compressed sensing [ZYJ14, BFN⁺17]. There are also a number of interesting works that appeal to extra information to mitigate the labeling cost, such as comparison [XZS⁺17, KLMZ17] and search [BH12, BHLZ16].

Recent works such as [DKK⁺16, LRV16] studied mean estimation under a strong noise model where in addition to returning dirty instances, the adversary has also the power of eliminating a few clean instances, similar to the nasty noise model in learning halfspaces [BEK02]. The main technique of robust mean estimation is a novel outlier removal paradigm, which uses the spectral norm of the covariance matrix to detect dirty instances. This is similar in spirit to the idea of [KLS09, ABL17] and the current work. However, there is no direct connection between mean estimation and halfspace learning since the former is an unsupervised problem while the latter is supervised (although any connection would be very interesting). Very recently, such technique was extensively investigated in a variety of problems such as clustering and linear regression; we refer the reader to a comprehensive survey by [DK19] for more information.

Our main algorithm, namely Algorithm 1, is based on the celebrated margin-based active learning framework [BBZ07]. The key observation is that a good classifier can be learned by concentrating on fitting only the most informative labeled instances, measured by the closeness to the current decision boundary (i.e. the closer the more informative). In our algorithm, the sampling region is set as $\mathbb{R}^{d}$ at phase $k=1$, and is set as the band $X_{w_{k-1},b_{k}}=\{x\in\mathbb{R}^{d}\mathrel{\mathop{\mathchar 58\relax}}\left\lvert w_{k-1}\cdot x\right\rvert\leq b_{k}\}$ at phases $k\geq 2$. Once we obtain the instance set $\bar{T}$, we perform a pruning step that removes all instances having large $\ell_{\infty}$-norm. This is motivated by our analysis that with high probability, all clean instances in $\bar{T}$ must have small $\ell_{\infty}$-norm provided that Assumption 1 is satisfied. Since the oracle $\mathrm{EX}_{\eta}^{x}(D,w^{*})$ may output dirty instances, we design an attribute-efficient soft outlier removal procedure, which aims to find proper weights for all instances in $T$, such that the clean instances (i.e. those from $D_{w_{k-1},b_{k}}$) have overwhelming weights compared to dirty instances. Equipped with the learned weights, it is possible to minimize the reweighted hinge loss to obtain a refined halfspace. However, this would lead to a suboptimal label complexity since we have to query the label for all instances in $T$. Our remedy is to randomly sample a few points from $T$ according to their importance, which is crucial for us to obtain near-optimal label complexity.

When minimizing the hinge loss, we carefully construct the constraint set $W_{k}$ with three properties. First, it has an $\ell_{2}$-norm constraint. As a useful fact of isotropic log-concave distributions, the $\ell_{2}$-distance to the underlying halfspace $w^{*}$ is of the same order as the error rate. Thus, if we were able to ensure that the target halfspace $w^{*}$ stays in $W_{k}$, we would show that the error rate of $w_{k}$ is as small as $O(r_{k})$, the radius of the $\ell_{2}$-ball. Second, $W_{k}$ has an $\ell_{1}$-norm constraint, which is well-known for its power to promote sparse solutions and to guarantee attribute-efficient sample complexity [Tib96, CDS98, CT05, PV13b]. Lastly, the $\ell_{2}$ and $\ell_{1}$ radii of $W_{k}$ shrinks by a constant factor in each phase; hence, when Algorithm 1 terminates, the radius of the $\ell_{2}$-ball will be as small as $O(\epsilon)$. Notably, [Zha18] also utilizes such constraint for active learning of sparse halfspaces, but only under the setting of label noise.

The last step in Algorithm 1 is to perform hard-thresholding $\mathcal{H}_{s}$ on the solution $v_{k}$ followed by $\ell_{2}$-normalization. Roughly speaking, these two steps will produce an iterate $w_{k}$ consistent with the structure of $w^{*}$ (i.e. $w_{k}$ is guaranteed to belong to the concept class $\mathcal{C}$), and more importantly, will be useful to show that $w^{*}$ lies in $W_{k}$ in all phases.

We elaborate on our hyper-parameter setting that is used in Algorithm 1 and our analysis. Let $g(t)=c_{2}\mathinner{\bigl{(}2t\exp(-t)+\frac{c_{3}\pi}{4}\exp\mathinner{\bigl{(}-\frac{c_{4}t}{4\pi}\bigr{)}}+16\exp(-t)\bigr{)}}$, where the constants are specified in Appendix A. Observe that there exists an absolute constant $\bar{c}\geq 8\pi/c_{4}$ satisfying $g(\bar{c})\leq 2^{-8}\pi$, since the continuous function $g(t)\to 0$ as $t\to+\infty$ and all the involved quantities in $g(t)$ are absolute constants. Given such constant $\bar{c}$, we set $b_{k}=\bar{c}\cdot 2^{-k-3}$, $\tau_{k}=c_{0}\kappa\cdot\min\{b_{k},1/9\}$, $\delta_{k}=\frac{\delta}{(k+1)(k+2)}$,

We set the constant $\kappa=\exp(-\bar{c})$, and choose $\xi_{k}=\min\big{\{}\frac{1}{2},\frac{\kappa^{2}}{16}\mathinner{\bigl{(}1+4\sqrt{C_{2}}{z_{k}}/{\tau_{k}}\bigr{)}}^{-2}\big{\}}$. Observe that all $\xi_{k}$’s are lower bounded by the constant $c_{6}\mathrel{\mathop{\mathchar 58\relax}}=\min\Big{\{}\frac{1}{2},\frac{\kappa^{2}}{16}\mathinner{\Bigl{(}1+\frac{4}{c_{0}\kappa\bar{c}}\sqrt{C_{2}\bar{c}^{2}+C_{2}}\Bigr{)}}^{-2}\Big{\}}$. Our theoretical guarantee holds for any noise rate $\eta\leq c_{5}\epsilon$, where the constant $c_{5}\mathrel{\mathop{\mathchar 58\relax}}=\frac{c_{8}}{2\pi}\bar{c}c_{1}c_{6}$.

We set the total number of phases $k_{0}=\big{\lceil}\log\mathinner{\bigl{(}\frac{\pi}{16c_{1}\epsilon}\bigr{)}}\big{\rceil}$ in Algorithm 1. Consider any phase $k\geq 1$. We use $n_{k}=\tilde{O}\mathinner{\bigl{(}s^{2}\log^{4}\frac{d}{b_{k}}\cdot\mathinner{\bigl{(}\log d+{\log^{3}\frac{1}{\delta_{k}}}\bigr{)}}\bigr{)}}$ as the size of unlabeled instance set $\bar{T}$. We will show that by making $N_{k}=O\mathinner{\left(n_{k}/b_{k}\right)}$ calls to $\mathrm{EX}_{\eta}^{x}(D,w^{*})$, Algorithm 1 is guaranteed to obtain such $\bar{T}$ in each phase with high probability. We set $m_{k}=\tilde{O}\mathinner{\bigl{(}s\log^{2}\frac{d}{b_{k}\delta_{k}}\cdot\log\frac{d}{\delta_{k}}\bigr{)}}$ as the size of labeled instance set $S_{k}$, which is also the number of calls to $\mathrm{EX}_{\eta}^{y}(D,w^{*})$. Note that $N\mathrel{\mathop{\mathchar 58\relax}}=\sum_{k=1}^{k_{0}}N_{k}$ is the sample complexity of Algorithm 1, and $m\mathrel{\mathop{\mathchar 58\relax}}=\sum_{k=1}^{k_{0}}m_{k}$ is its label complexity.

Our soft outlier removal procedure is inspired by [ABL17]. We first briefly describe their main idea. Then we introduce a natural extension of their approach to the high-dimensional regime and show why it fails. Lastly, we present our novel outlier removal scheme.

To ease our discussion, we decompose $T=T_{\mathrm{C}}\cup T_{\mathrm{D}}$ where $T_{\mathrm{C}}$ is the set of clean instances in $T$ and $T_{\mathrm{D}}$ consists of all dirty instances. Ideally, we would expect to find a function $q\mathrel{\mathop{\mathchar 58\relax}}T\rightarrow[0,1]$ such that $q(x)=1$ for all $x\in T_{\mathrm{C}}$ and $q(x)=0$ otherwise. Suppose that $\xi$ is the fraction of dirty instances in $T$. Then one would expect that the total weights $\sum_{x\in T}q(x)$ is as large as $(1-\xi)\left\lvert T\right\rvert$ in order to include such ideal function. On the other hand, we must restrict the weights of dirty instances; namely, we need to characterize under what conditions $T_{\mathrm{C}}$ can be distinguished from $T_{\mathrm{D}}$. The key observation made in [KLS09] and [ABL17] is that if the dirty instances want to deteriorate the hinge loss (which is the purpose of the adversary), they must lead to a variance³³3We follow [ABL17] and slightly abuse the word “variance” without subtracting the squared mean of $w\cdot x$. of $w\cdot x$ orders of magnitude larger than $\Omega(b^{2}+r^{2})$ on the direction of a particular halfspace. Thus, it suffices to find a proper weight for each instance, such that the reweighted variance $\frac{1}{\left\lvert T\right\rvert}\sum_{x\in T}q(x)(w\cdot x)^{2}$ is as small as $O(b^{2}+r^{2})$ for all feasible halfspaces $w\in W$. Now it remains to resolve two questions: 1) how many instances do we need to draw in order to guarantee the existence of such function $q$; and 2) how to find a feasible function $q$ in polynomial time.

If label complexity were our only objective, we could have used the soft outlier removal procedure of [ABL17] directly, i.e. we set $W=B_{2}(u,r)$, which in conjunction with the $\ell_{1}$-norm constrained hinge loss minimization of [Zha18] would result in an $\tilde{O}\mathinner{\bigl{(}\frac{d^{2}}{\epsilon}\bigr{)}}$ sample complexity and a $\operatorname{poly}\mathinner{\left(s,\log d,\log(1/\epsilon)\right)}$ label complexity. However, as we would also like to optimize for the learner’s sample complexity by utilizing the sparsity assumption, we need an attribute-efficient outlier removal procedure.

Localized sampling, also known as margin-based active learning, is a useful technique proposed in [BBZ07]. Interestingly, under isotropic log-concave distributions, [BL13] showed that if the band width $b$ is large enough, the region outside the band, i.e. $\{x\in\mathbb{R}^{d}\mathrel{\mathop{\mathchar 58\relax}}\left\lvert w\cdot x\right\rvert>b\}$, can be safely “ignored”, in the sense that, if $w$ is close enough to $w^{*}$, it is guaranteed to incur a small error rate therein. Motivated by this elegant finding, theoretical analyses in the literature are often dedicated to bounding the error rate within the band, and it is now well understood that a constant error rate within the band suffices to ensure significant progress in each phase [ABHU15, ABL17, Zha18]. We follow this line of reasoning and our technical contribution is to show how to obtain such constant error rate with near-optimal label complexity and noise tolerance.

Our analysis will rely on the condition that $\bar{T}$ has sufficiently many instances. Specifically, in order to collect $n_{k}$ instances to form the working set $\bar{T}$, we need to call $\mathrm{EX}_{\eta}^{x}(D,w^{*})$ enough number of times since our sampling is localized within the band $X_{k}\mathrel{\mathop{\mathchar 58\relax}}=\mathinner{\left\{x\mathrel{\mathop{\mathchar 58\relax}}\left\lvert w_{k-1}\cdot x\right\rvert\leq b_{k}\right\}}$. The following lemma characterizes the sample complexity at phase $k$.

We summarize the performance guarantee of Algorithm 2 in the following proposition.

Again, we remind that the key difference between our algorithm and that of [ABL17] is in Constraint 3 of Algorithm 2: we require that the “variance proxy” $\sum_{x\in T}q(x)x^{\top}Hx$ of the reweighted instances are small for all positive semidefinite $H$ that lies in an intersection of a trace-norm ball and an $\ell_{1}$-norm ball. On the statistical side, this favorable constraint set of $H$, in conjunction with Adamczak’s bound in empirical processes literature [Ada08], results in sufficient uniform concentration of the variance proxy $x^{\top}Hx$ with a sample complexity of $\operatorname{poly}\mathinner{\left(s,\log d\right)}$. This significantly improves the sample complexity of $\operatorname{poly}\mathinner{\left(d\right)}$ established in [ABL17]. The detailed proof can be found in Appendix D.3.

Next, we describe the statistical property of the distribution $p$ (obtained by normalizing $q$ returned by Algorithm 2). Observe that the noise rate within the band is at most $\eta/b_{k}\leq O(\eta/\epsilon)\leq\xi_{k}$ since the probability mass of the band is $\Theta(b_{k})$ – an important property of isotropic log-concave distributions. Also, it is possible to show that the variance of clean instances on directions $H\in\mathcal{M}_{k}$ is $O(b_{k}^{2}+r_{k}^{2})$ (see Lemma 16). Therefore, Algorithm 2 is essentially searching for a weighting such that clean instances have overwhelming weights over dirty instances, and that the variance of the weighted instances is similar to that of the clean instances. Recall that $T_{\mathrm{C}}\subset T$ is the set of clean instances in $T$. Let $\tilde{T}_{\mathrm{C}}=\{(x,y_{x})\}_{x\in T_{\mathrm{C}}}$ be the unrevealed labeled set where each instance is correctly annotated by $w^{*}$. The following proposition, which is similar to Lemma 4.7 of [ABL17] but with refinement, states that the reweighted hinge loss $\ell_{\tau_{k}}(w;p)\mathrel{\mathop{\mathchar 58\relax}}=\sum_{x\in T}p(x)\ell_{\tau_{k}}(w;x,y_{x})$, is a good proxy for the hinge loss evaluated exclusively on clean labeled instances $\tilde{T}_{\mathrm{C}}$.

Note that though this proposition is phrased in terms of the hinge loss on pairs $(x,y_{x})$, it is only used in the analysis and our algorithm does not require the knowledge of the labels $y_{x}$ – the algorithm even does not need to exactly identify the set of clean instances $T_{\mathrm{C}}$. As a result, the size of $T_{\mathrm{C}}$ does not count towards our label complexity. Proposition 7 together with Proposition 8 implies that with high probability, Algorithm 2 produces a desired probability distribution in polynomial time, which justifies its computational and statistical efficiency.

In addition, let $L_{\tau_{k}}(w)\mathrel{\mathop{\mathchar 58\relax}}=\operatorname{\mathbb{E}}_{x\sim D_{w_{k-1},b_{k}}}\mathinner{\bigl{[}\ell_{\tau_{k}}\mathinner{\left(w;x,\operatorname{sign}\mathinner{\left(w^{*}\cdot x\right)}\right)}\bigr{]}}$ be the expected loss on $D_{w_{k-1},b_{k}}$. The following result links $L_{\tau_{k}}(w)$ to the empirical hinge loss on clean instances.

In light of Proposition 8, one may want to find an iterate by minimizing its reweighted hinge loss $\ell_{\tau_{k}}(w;p)$. This requires collecting labels for all instances in $T$, which leads to a suboptimal label complexity $O\mathinner{\bigl{(}s^{2}\cdot\operatorname{polylog}\mathinner{\left(d,1/\epsilon\right)}\bigr{)}}$. As a remedy, we perform a random sampling process, which draws $m_{k}$ instances from $T$ according to the distribution $p$ and then query their labels, resulting in the labeled instance set $S_{k}$. By standard uniform convergence arguments, it is expected that $\ell_{\tau_{k}}(w;S_{k})\approx\ell_{\tau_{k}}(w;p)$ provided that $m_{k}$ is large enough, as is shown in the following proposition.

We remark that when establishing the performance guarantee, the $\ell_{1}$-norm constraint on the hypothesis space, together with an $\ell_{\infty}$-norm upper bound on the localized instance space, leads to a Rademacher complexity that has a linear dependence on the sparsity (up to a logarithmic factor). Technically speaking, our analysis is more involved than that of [ABL17]: applying their analysis to the setting of learning sparse halfspaces along with the fact that the VC dimension of the class of $s$-sparse halfspaces is $O(s\log(d/s))$ would give a label complexity quadratic in $s$.

Our analysis involves building uniform concentration bounds. The primary issue of applying standard concentration results, e.g. Theorem 1 of [KST08], is that the instances are not contained in a pre-specified $\ell_{\infty}$-ball with probability $1$ under isotropic log-concave distribution. [ABL17, Zha18] construct a conditional distribution, on which the data are all bounded from above, and then measure the difference between this conditional distribution and the original one. We circumvent such technical complication by using the Adamczak’s bound [Ada08] in the empirical process literature, which provides a generic way to analyze concentration inequalities for well-behaved distributions with unbounded support. See Appendix C for a concrete treatment.