A study of the classification of low-dimensional data with supervised manifold learning

Consider a setting with $M$ data classes where the samples of each class $m\in\{1,\dots,M\}$ are drawn from a probability measure $\nu_{m}$ in a Hilbert space $H$ such that $\nu_{m}$ has a bounded support $\mathcal{M}_{m}\subset H$. Let $\mathcal{X}=\{x_{i}\}_{i=1}^{N}\subset H$ be a set of $N$ training samples such that each $x_{i}$ is drawn from one of the probability measures $\nu_{m}$, and the samples drawn from each $\nu_{m}$ are independent and identically distributed. We denote the class label of $x_{i}$ by $C_{i}\in\{1,2,\dots,M\}$.

Let $Y=\{y_{i}\}_{i=1}^{N}\subset\mathbb{R}^{d}$ be a $d$-dimensional embedding of $\mathcal{X}$, where each $y_{i}$ corresponds to $x_{i}$. We consider supervised embeddings such that $Y$ is linearly separable. Linear separability is defined as follows:

The above definition of separability implies the following. For any given class $m$, there exists a set of hyperplanes $\{\omega_{mk}\}_{k\neq m}\subset\mathbb{R}^{d}$, $\|\omega_{mk}\|=1$, and a set of real numbers $\{b_{mk}\}_{k\neq m}\subset\mathbb{R}$ that separate class $m$ from other classes, such that for all $y_{i}$ of class $C_{i}=m$

$$\omega_{mk}^{T}\,y_{i}+b_{mk}>\gamma/2,\quad\forall k\neq m$$

(2)

and for all $y_{i}$ of class $C_{i}\neq m$, there exists a $k$ such that

$$\omega_{mk}^{T}\,y_{i}+b_{mk}<-\gamma/2.$$

(3)

These hyperplanes are obtained by setting $\omega_{km}=-\omega_{mk}$, $b_{km}=-b_{mk}$.

Figure 1 shows an illustration of a linearly separable embedding of data samples from two classes. Manifold learning methods typically compute a low-dimensional embedding $Y$ of training data $\mathcal{X}$ in a pointwise manner, i.e., the coordinates $y_{i}$ are computed only for the initially available training samples $x_{i}$. However, in a classification problem, in order to estimate the class label of a new data sample $x$ of unknown class, $x$ needs to be mapped to the low-dimensional domain of embedding as well. The construction of a function $f:H\rightarrow\mathbb{R}^{d}$ that generalizes the learnt embedding to the whole space is known as the out-of-sample generalization problem. Smooth functions are commonly used for out-of-sample interpolation, e.g. as in (Qiao et al., 2013), (Peherstorfer et al., 2011).

Now let $x$ be a test sample drawn from the probability measure $\nu_{m}$, hence, the true class label of $x$ is $m$. In our study, we consider two basic classification schemes in the domain of embedding:

Linear classifier. The embeddings of the training samples are used to compute the separating hyperplanes, i.e., the classifier parameters $\{\omega_{mk}\}$ and $\{b_{mk}\}$. Then, mapping $x$ to the low-dimensional domain as $f(x)\in\mathbb{R}^{d}$, the class label of $x$ is estimated as $\hat{C}(x)=l$ if there exists $l\in\{1,\dots,M\}$ such that

$$\omega_{lk}^{T}\,f(x)+b_{lk}>0,\quad\forall k\in\{1,\dots,M\}\setminus\{l\}.$$

(4)

Note that the existence of such an $l$ is not guaranteed in general for any $x$, but for a given $x$ there cannot be more than one $l$ satisfying the above condition. Then $x$ is classified correctly if the estimated class label agrees with the true class label, i.e., $\hat{C}(x)=l=m$.

Nearest neighbor classification. The test sample $x$ is assigned the class label of the closest training point in the domain of embedding, i.e., $\hat{C}(x)=C_{i^{\prime}}$, where

$$i^{\prime}=\arg\min_{i=1,\dots,N}\|y_{i}-f(x)\|$$

In the rest of this section, we study the generalization performance of supervised dimensionality reduction methods. We first consider in Section 2.2 interpolation functions that vary regularly on each class support and we search for a lower bound on the probability of correctly classifying a new data sample in terms of the regularity of $f$, the separation of the embedding, and the sampling density. Then in Section 2.3, we study the classification performance for a particular type of interpolation functions, namely RBF interpolators, which is one of the most popular ones (Peherstorfer et al., 2011), (Chin and Suter, 2008). We focus particularly on Gaussian RBF interpolators in Section 2.4 and derive some results regarding the existence of an optimal kernel scale parameter. Lastly, we discuss our results in comparison with previous literature in Section 2.5.

In the results in Sections 2.2-2.4, we keep a generic formulation and simply treat the supports $\{\mathcal{M}_{m}\}$ as arbitrary bounded subsets of $H$, each of which represents a different data class. Nevertheless, from the perspective of manifold learning, our results are of interest especially when the data is assumed to have an underlying low-dimensional structure. In Section 2.5, we study the implications of our results for the setting where $\mathcal{M}_{m}$ are low-dimensional manifolds. We then examine how the proposed bounds vary in relation to the intrinsic dimensions of $\{\mathcal{M}_{m}\}$.

Let $f:H\rightarrow\mathbb{R}^{d}$ be an out-of-sample interpolation function such that $f(x_{i})=y_{i}$ for each training sample $x_{i}$, $i=1,\dots,N$. Assume that $f$ is Lipschitz continuous with constant $L>0$ when restricted to any one of the supports $\mathcal{M}_{m}$; i.e., for any $m\in\{1,\dots,M\}$ and any $u,v\in\mathcal{M}_{m}$

$$\|f(u)-f(v)\|\leq L\,\|u-v\|$$

where $\|\cdot\|$ denotes above the $\ell_{2}$-norm if the argument is in $\mathbb{R}^{d}$, and the norm induced from the inner product in $H$ if the argument is in $H$.

We will find a relation between the classification accuracy and the number of training samples via the covering number of the supports $\mathcal{M}_{m}$. Let $B_{\epsilon}(x)\subset H$ denote an open ball of radius $\epsilon$ around $x$

$$B_{\epsilon}(x)=\{u\in H:\|x-u\|<\epsilon\}.$$

The covering number $\mathcal{N}(\epsilon,A)$ of a set $A\subset H$ is defined as the smallest number of open balls $B_{\epsilon}$ of radius $\epsilon$ whose union contains $A$ (Kulkarni and Posner, 1995)

$$\mathcal{N}(\epsilon,A)=\inf\{k:\exists\,u_{1},\dots,u_{k}\in H\,\text{s.t.}\,A\subset\bigcup_{i=1}^{k}B_{\epsilon}(u_{i})\}.$$

We assume that the supports $\mathcal{M}_{m}$ are totally bounded, i.e., $\mathcal{M}_{m}$ has a finite covering number $\mathcal{N}(\epsilon,\mathcal{M}_{m})$ for any $\epsilon>0$.

We state below a lower bound for the probability of correctly classifying a sample $x$ drawn from $\nu_{m}$, in terms of the number of training samples drawn from $\nu_{m}$, the separation of the embedding and the regularity of $f$.

The proof of the theorem is given in Appendix A.1. Theorem 2 establishes a link between the classification performance and the separation of the embedding of the training samples. In particular, due to the condition $\epsilon\leq\gamma/(2L)$, the increase in the separation $\gamma$ allows a larger value for $\epsilon$, provided that the interpolator regularity is not affected much. This reduces the covering number $\mathcal{N}(\epsilon/2,\mathcal{M}_{m})$ in return and increases the probability of correct classification. Similarly, from the condition $\epsilon\leq\gamma/(2L)$, one can also observe that at a given separation $\gamma$, a smaller Lipschitz constant $L$ for the interpolation function allows the parameter $\epsilon$ to take a larger value. This reduces the covering number $\mathcal{N}(\epsilon/2,\mathcal{M}_{m})$ and therefore increases the correct classification probability. Thus, choosing a more regular interpolator at a given separation helps improve the classification performance. If the $\epsilon$ parameter is fixed, the Lipschitz constant of the interpolator is allowed to increase only proportionally to the separation margin. The condition that the interpolator must be sufficiently regular in comparison with the separation suggests that increasing the separation too much at the cost of impairing the interpolator regularity may degrade the classifier performance. In the case that the supports $\mathcal{M}_{m}$ are low-dimensional manifolds, the covering number $\mathcal{N}(\epsilon/2,\mathcal{M}_{m})$ increases at a geometric rate with the intrinsic dimension $D$ of the manifold, since a $D$-dimensional manifold is locally homeomorphic to $\mathbb{R}^{D}$. Therefore, from the condition on the number of samples, $N_{m}$ should increase at a geometric rate with $D$.

In Theorem 2 the probability of misclassification decreases with the number $N_{m}$ of training samples at a rate of $O(N_{m}^{-1})$. In the rest of this section, we show that it is in fact possible to obtain an exponential convergence rate with linear and NN-classifiers under certain assumptions. We first present the following lemma.

Lemma 3 is proved in Appendix A.2. The inequality in (6) shows that as the number $Q$ of training samples falling in a neighborhood of a test point $x$ increases, the probability of the deviation of $f(x)$ from its average within the neighborhood decreases. The parameter $\epsilon$ captures the relation between the amount and the probability of deviation.

When studying the classification accuracy in the main result below, we will use the following generalized definition of the linear separation.

The above definition of separability is more flexible than the one in Definition 1. Clearly, an embedding that is linearly separable with margin $\gamma$ has a $Q$-mean separability margin of $\gamma_{Q}\geq\gamma$ for any $Q$. As in the previous section, we consider that the test sample $x$ is classified with the linear classifier (4) in the low-dimensional domain, defined with respect to the set of hyperplanes given by $\{\omega_{mk}\}$ and $\{b_{mk}\}$ as in (2) and (3).

In the following result, we show that an exponential convergence rate can be obtained with linear classifiers in supervised manifold learning. We define beforehand a parameter depending on $\delta$, which gives the smallest possible measure of the $\delta$-neighborhood $B_{\delta}(x)$ of a point $x$ in support $\mathcal{M}_{m}$.

$$\eta_{m,\delta}:=\inf_{x\in\mathcal{M}_{m}}\nu_{m}(B_{\delta}(x)).$$

Theorem 5 is proved in Appendix A.3. The theorem shows how the classification accuracy is influenced by the separation of the classes in the embedding, the smoothness of the out-of-sample interpolant, and the number of training samples drawn from the density of each class. The condition in (8) points to the tradeoff between the separation and the regularity of the interpolation function. As the Lipschitz constant $L$ of the interpolation function $f$ increases, $f$ becomes less “regular”, and a higher separation $\gamma_{Q}$ is needed to meet the condition. This is coherent with the expectation that, when $f$ becomes irregular, the classifier becomes more sensitive to the perturbations of the data, e.g., due to noise. The requirement of a higher separation is then for ensuring a larger margin in the linear classifier, which compensates for the irregularity of $f$. From (8), it is also observed that the separation should increase with the dimension $d$ as well, and also with $\epsilon$, whose increase improves the confidence of the bound (9). Note that the condition in (8) implies also the following: When computing an embedding, it is not advisable to increase the separation of training data unconditionally. In particular, increasing the separation too much may violate the preservation of the geometry and yield an irregular interpolator. Hence, when designing a supervised dimensionality reduction algorithm, one must pay attention to the regularity of the resulting interpolator as much as the enhancement of the separation margin.

Next, we discuss the roles of the parameters $Q$ and $\delta$. The term $\exp(-Q\,\epsilon^{2}/(2L^{2}\delta^{2}))$ in the correct classification probability bound (9) shows that, for fixed $\delta$, the confidence increases with the value of $Q$. Meanwhile, due to the numerator of the term $\exp(-2\,(N_{m}\,\eta_{m,\delta}-Q)^{2}/N_{m})$, for a high confidence, the number of samples $N_{m}$ should also be relatively big with respect to $Q$ to have a high overall confidence. Similarly, at fixed $Q$, $\delta$ should be made smaller to increase the confidence due to the term $\exp(-(Q\,\epsilon^{2})/(2L^{2}\delta^{2}))$, which then reduces the parameter $\eta_{m,\delta}$ and eventually requires the number of samples $N_{m}$ to take a sufficiently large value in order to make the term $\exp(-2\,(N_{m}\,\eta_{m,\delta}-Q)^{2}/N_{m})$ small and have a high confidence. Therefore, these two parameters $Q$ and $\delta$ behave in a similar way, and determine the relation between the number of samples and the correct classification probability, i.e., they indicate how large $N_{m}$ should be in order to have a certain confidence of correct classification.

Theorem 5 studies the setting where the class labels are estimated with a linear classifier in the domain of embedding. We also provide another result below that analyses the performance when a nearest-neighbor classifier is used in the domain of embedding.

Theorem 6 is proved in Appendix A.4. Theorem 6 is quite similar to Theorem 5 and can be interpreted similarly. Unlike in the previous result, the separability condition of the embedding is based on the pairwise distances of samples from different classes here. The condition (10) suggests that the result is useful when the parameter $D_{2\delta}$ is sufficiently small, which requires the embedding to map nearby samples from the same class in the ambient space to nearby points.

In this section, we have characterized the regularity of the interpolation functions via their rates of variation when restricted to the supports $\mathcal{M}_{m}$. While the results of this section are generic in the sense that they are valid for any interpolation function with the described regularity properties, we have not examined the construction of such functions. In a practical classification problem where one uses a particular type of interpolation functions, one would also be interested in the adaptation of these results to obtain performance guarantees for the particular type of function used. Hence, in the following section we focus on a popular family of smooth functions; radial basis function (RBF) interpolators, and study the classification performance of this particular type of interpolators.

Here we consider an RBF interpolation function $f:H\rightarrow\mathbb{R}^{d}$ of the form

$$f(x)=[f^{1}(x)\,f^{2}(x)\,\dots f^{d}(x)]$$

such that each component $f^{k}$ of $f$ is given by

$$f^{k}(x)=\sum_{i=1}^{N}c_{i}^{k}\,\phi(\|x-x_{i}\|)$$

where $\phi:\mathbb{R}\rightarrow\mathbb{R}^{+}$ is a kernel function, $c_{i}^{k}\in\mathbb{R}$ are coefficients, and $x_{i}$ are kernel centers. In interpolation with RBF functions, it is common to choose the set of kernel centers as the set of available data samples. Hence, we assume that the set of kernel centers $\{x_{i}\}_{i=1}^{N}$ is selected to be the same as the set of training samples $\mathcal{X}$. We consider a setting where the coefficients $c_{i}^{k}$ are set such that $f(x_{i})=y_{i}$, i.e., $f$ maps each training point in $\mathcal{X}$ to its embedding previously computed with supervised manifold learning.

We consider the RBF kernel $\phi$ to be a Lipschitz continuous function with constant $L_{\phi}>0$, hence, for any $u,v\in\mathbb{R}$

$$|\phi(u)-\phi(v)|\leq L_{\phi}\,|u-v|.$$

Also, let $\mathcal{C}$ be an upper bound on the coefficient magnitudes such that for all $k=1,\dots,d$

$$\sum_{i=1}^{N}|c_{i}^{k}|\leq\mathcal{C}.$$

In the following, we analyze the classification accuracy and extend the results in Section 2.2 to the case of RBF interpolators. We first give the following result, which probabilistically bounds how much the value of the interpolator $f$ at a point $x$ randomly drawn from $\nu_{m}$ may deviate from the average interpolator value of the training points of the same class within a neighborhood of $x$.

The proof of Lemma 7 is given in Appendix A.5. The lemma states a result similar to the one in Lemma 3; however, is specialized to the case where $f$ is an RBF interpolator.

We are now ready to present the following main result.

The theorem is proved in Appendix A.6. The theorem bounds the classification accuracy in terms of the smoothness of the RBF interpolation function and the number of samples. The condition in (14) characterizes the compromise between the separation and the regularity of the interpolator, which depends on the Lipschitz constant of the RBF kernels and the coefficient magnitude. As the Lipschitz constant $L_{\phi}$ and the coefficient magnitude parameter $\mathcal{C}$ increase (i.e., $f$ becomes less “regular”), a higher separation $\gamma_{Q}$ is required to provide a performance guarantee. When the separation margin of the embedding and the interpolator satisfy the condition in (14), the misclassification probability decays exponentially as the number of training samples increases, similarly to the results in Section 2.2.

Theorem 8 studies the misclassification probability when the class labels in the low-dimensional domain are estimated with a linear classifier. We also present below a bound on the misclassification probability when the nearest-neighbor classifier is used in the low-dimensional domain.

Theorem 9 is proved in Appendix A.7. While it provides the exact convergence rate as in Theorem 8, the necessary condition in (16) includes also the parameter $D_{2\delta}$. Hence, if the embedding maps nearby samples from the same class to nearby points, and a compromise is achieved between the separation and the interpolator regularity, the misclassification probability can be upper bounded.

In data interpolation with RBFs, it is known that the accuracy of interpolation is quite sensitive to the choice of the shape parameter for several kernels including the Gaussian kernel (Baxter, 1992). The relation between the shape parameter and the performance of interpolation has been an important problem of interest (Piret, 2007). In this section, we focus on the Gaussian RBF kernel, which is a popular choice for RBF interpolation due to its smoothness and good spatial localization properties. We study the choice of the scale parameter of the kernel within the context of classification.

We consider the RBF kernel given by

$$\phi(r)=e^{-\frac{r^{2}}{\sigma^{2}}}$$

where $\sigma$ is the scale parameter of the Gaussian function. We focus on the condition (14) in Theorem 8

$$\sqrt{d}\,\mathcal{C}\,(L_{\phi}\delta+\epsilon)\leq\gamma_{Q}/2,$$

(or equivalently the condition (16) if the nearest neighbor classifier is used), which relates the interpolation function properties with the separation. In particular, for a given separation margin, this condition is satisfied more easily when the term on the left hand side of the inequality is smaller. Thus, in the following, we derive an expression for the left hand side of the above inequality by deriving the Lipschitz constant $L_{\phi}$ and the coefficient bound $\mathcal{C}$ in terms of the scale parameter $\sigma$ of the Gaussian kernel. We then study the scale parameter that minimizes $\sqrt{d}\,\mathcal{C}\,(L_{\phi}\delta+\epsilon)$.

Writing the condition $f(x_{i})=y_{i}$ in a matrix form for each dimension $k=1,\dots,d$, we have

$$\Phi c^{k}=y^{k}$$

(18)

where $\Phi\in\mathbb{R}^{N\times N}$ is a matrix whose $(i,j)$-th entry is given by $\Phi_{ij}=\phi(\|x_{i}-x_{j}\|)$, $c^{k}\in\mathbb{R}^{N\times 1}$ is the coefficient vector whose $i$-th entry is $c_{i}^{k}$, and $y^{k}\in\mathbb{R}^{N\times 1}$ is the data coordinate vector giving the $k$-th dimensions of the embeddings of all samples, i.e., $y_{i}^{k}=Y_{ik}$. Assuming that the embedding is computed with the usual scale constraint $Y^{T}Y=I$, we have $\|y^{k}\|=1$. The norm of the coefficient vector can then be bounded as

$$\|c^{k}\|\leq\|\Phi^{-1}\|\|y^{k}\|=\|\Phi^{-1}\|.$$

(19)

In the rest of this section, we assume that the data $\mathcal{X}$ are sampled from the Euclidean space, i.e., $H=\mathbb{R}^{n}$. We first use a result by Narcowich et al. (1994) in order to bound the norm $\|\Phi^{-1}\|$ of the inverse matrix. From (Narcowich et al., 1994, Theorem 4.1) we get¹¹1The result stated in (Narcowich et al., 1994, Theorem 4.1) is adapted to our study by taking the measure as $\beta(\rho)=\delta(\rho-\rho_{0})$ so that the RBF kernel defined in (Narcowich et al., 1994, (1.1)) corresponds to a Gaussian function as $F(r)=\exp(-\rho_{0}\,r^{2})$. The scale of the Gaussian kernel is then given by $\sigma={\rho_{0}}^{-1/2}$.

$$\|\Phi^{-1}\|\leq\beta\,\sigma^{-n}e^{\alpha\sigma^{2}}$$

(20)

where $\alpha>0$ and $\beta>0$ are constants depending on the dimension $n$ and the minimum distance between the training points $\mathcal{X}$ (separation radius) (Narcowich et al., 1994). As the $\ell_{1}$-norm of the coefficient vector can be bounded as $\|c^{k}\|_{1}\leq\sqrt{N}\|c^{k}\|$, from (19) one can set the parameter $\mathcal{C}$ that upper bounds the coefficients magnitudes as

$$\mathcal{C}=a\sigma^{-n}e^{\alpha\sigma^{2}}$$

where $a=\beta\sqrt{N}$.

Next, we derive a Lipschitz constant for the Gaussian kernel $\phi(r)$ in terms of $\sigma$. Setting the second derivative of $\phi$ to zero

$$\frac{d^{2}\phi}{dr^{2}}=e^{-\frac{r^{2}}{\sigma^{2}}}\left(\frac{4r^{2}}{\sigma^{4}}-\frac{2}{\sigma^{2}}\right)=0$$

we get that the maximum value of $|d\phi/dr|$ is attained at $r=\sigma/\sqrt{2}$. Evaluating $|d\phi/dr|$ at this value, we obtain

$$L_{\phi}=\sqrt{2}e^{-\frac{1}{2}}\sigma^{-1}.$$

Now rewriting the condition (14) of the theorem, we have

$$\sqrt{d}\,\mathcal{C}\,(L_{\phi}\delta+\epsilon)=a_{1}\sigma^{-n-1}e^{\alpha\sigma^{2}}\,+a_{2}\sigma^{-n}e^{\alpha\sigma^{2}}\leq\gamma_{Q}/2$$

where $a_{1}=\sqrt{2d}\,a\,e^{-1/2}\delta$ and $a_{2}=\sqrt{d}\,a\,\epsilon$. We thus determine the Gaussian scale parameter $\sigma$ that minimizes

$$F(\sigma)=a_{1}\sigma^{-n-1}e^{\alpha\sigma^{2}}\,+a_{2}\sigma^{-n}e^{\alpha\sigma^{2}}.$$

First, notice that as $\sigma\rightarrow 0$ and $\sigma\rightarrow\infty$, the function $F(\sigma)\rightarrow\infty$. Therefore, it has at least one minimum. Setting

$$\frac{dF}{d\sigma}=e^{\alpha\sigma^{2}}\sigma^{-n-2}\big{(}2\alpha a_{2}\sigma^{3}+2\alpha a_{1}\sigma^{2}-a_{2}n\sigma-a_{1}(n+1)\big{)}=0$$

we need to solve

$$2\alpha a_{2}\sigma^{3}+2\alpha a_{1}\sigma^{2}-a_{2}n\sigma-a_{1}(n+1)=0.$$

(21)

The leading and the second-degree coefficients are positive, while the first-degree and the constant coefficients are negative in the above cubic polynomial. Then, the sum of the roots is negative and the product of the roots is positive. Therefore, there is one and only one positive root $\sigma_{opt}$, which is the unique minimizer of $F(\sigma)$.

The existence of an optimal scale parameter $0<\sigma_{opt}<\infty$ for the RBF kernel can be intuitively explained as follows. When $\sigma$ takes too small values, the support of the RBF function concentrated around the training points does not sufficiently cover the whole class supports $\mathcal{M}_{m}$. This manifests itself in (14) with the increase in the term $L_{\phi}$, which indicates that the interpolation function is not sufficiently regular. This weakens the guarantee that a test sample will be interpolated sufficiently close to its neighboring training samples from the same class and mapped to the correct side of the hyperplane in the linear classifier. On the other hand, when $\sigma$ increases too much, the stability of the linear system (18) is impaired and the coefficients $c$ increase too much. This results in an overfitting of the interpolator and, therefore, decreases the classification performance. Hence, the analysis in this section provides a theoretical justification of the common knowledge that $\sigma$ should be set to a sufficiently large value while avoiding overfitting.

Remark: It is also interesting to observe how the optimal scale parameter changes with the number of samples $N$. In the study (Narcowich et al., 1994), the constants $\alpha$ and $\beta$ in (20) are shown to vary with the separation radius $q$ at rates $\alpha=O(q^{-2})$ and $\beta=O(q^{n})$, where the separation radius $q$ is proportional to the smallest distance between two distinct training samples. Then a reasonable assumption is that the separation radius $q$ should typically decrease at rate $O(N^{-1/n})$ as $N$ increases. Using this relation, we get that $\alpha$ and $\beta$ should vary at rates $\alpha=O(N^{2/n})$ and $\beta=O(N^{-1})$ with $N$. It follows that $a=\beta\sqrt{N}=O(N^{-1/2})$, and the parameters $a_{1}$, $a_{2}$ of the cubic polynomial in (21) also vary with $N$ at rates $a_{1}=O(N^{-1/2})$, $a_{2}=O(N^{-1/2})$. The equation (21) in $\sigma$ can then be rearranged as

$$b_{3}\sigma^{3}+b_{2}\sigma^{2}-b_{1}\sigma-b_{0}=0$$

such that the constants vary with $N$ at rates $b_{3}=O(N^{2/n})$, $b_{2}=O(N^{2/n})$, $b_{1}=O(1)$, $b_{0}=O(1)$. We can then inspect how the roots of this equation change with $N$ as $N$ increases. Since $b_{3}$ and $b_{2}$ dominate the other coefficients for large $N$, three real roots will exist if $N$ is sufficiently large, two of which are negative and one is positive. The sum of the pairwise products of the roots is negative and it decays with $N$ at rate $O(N^{-2/n})$, and the product of the roots also decays with $N$. Then at least two of the roots must decay with $N$. Meanwhile, the sum of the three roots is $O(1)$ and negative. This shows that one of the negative roots is $O(1)$, i.e., does not decay with $N$. From the product of three roots, we then observe that the product of the two decaying roots is $O(N^{-2/n})$. However, their sum also decays at the same rate (from the sum of the pairwise products), which is possible if their dominant terms have the same rate and cancel each other. We conclude that both of the decaying roots vary at rate $O(N^{-1/n})$, one of which is the positive root and the optimal value $\sigma_{opt}$ of the scale parameter. This analysis shows that the scale parameter of the Gaussian kernel should be adapted to the number of training samples, and a smaller kernel scale must be preferred for a larger number of training samples. In fact, the relation $\sigma_{opt}=O(N^{-1/n})$ is quite intuitive, as the average or typical distance between two samples will also decrease at rate $O(N^{-1/n})$ as the number of samples $N$ increases in an $n$-dimensional space. Then the above result simply suggests that the kernel scale should be chosen as proportional to the average distance between the training samples.

In Theorems 8 and 9, we have presented a result that characterizes the performance of classification with RBF interpolation functions. In particular, we have considered a setting where an RBF interpolator is fitted to each dimension of a low-dimensional embedding where different classes are separable. Our study has several links with RBF networks or least-squares regression algorithms. In this section, we interpret our findings in relation with previously established results.

Several previous works study the performance of learning by considering a probability measure $\rho$ defined on $X\times Y$, where $X$ and $Y$ are two sets. The “label” set $Y$ is often taken as an interval $[-L,L]$. Given a set of data pairs $\{(x_{j},y_{j})\}_{j=1}^{N}$ sampled from the distribution $\rho$, the RBF network estimates a function $\hat{f}$ of the form

$$\hat{f}(x)=\sum_{i=1}^{R}c_{i}\,\phi\left(\frac{\|x-t_{i}\|}{\sigma_{i}}\right).$$

(22)

The number of RBF terms $R$ may be different from the number of samples $N$ in general. The function $\hat{f}$ minimizes the empirical error

$$\hat{f}=\arg\min_{f}\sum_{j=1}^{N}\left(f(x_{j})-y_{j}\right)^{2}.$$

The function $\hat{f}$ estimated from a finite collection of data samples is often compared to the regression function (Cucker and Smale, 2002)

$$f_{o}(x)=\int_{Y}y\,d\rho(y|x)$$

where $d\rho(y|x)$ is the conditional probability measure on $Y$. The regression function $f_{o}$ minimizes the expected risk as

$$f_{o}=\arg\min_{f}\int_{X\times Y}\big{(}f(x)-y\big{)}^{2}d\rho.$$

As the probability measure $\rho$ is not known in practice, the estimate $\hat{f}$ of $f_{o}$ is obtained from data samples. Several previous works have characterized the performance of learning by studying the approximation error (Niyogi and Girosi, 1996), (Lin et al., 2014)

$$\mathbb{E}[(f_{o}-\hat{f})^{2}]=\int_{X}(f_{o}(x)-\hat{f}(x))^{2}d\rho_{X}(x)$$

(23)

where $\rho_{X}$ is the marginal probability measure on $X$. This definition of the approximation error can be adapted to our setting as follows. In our problem the distribution of each class is assumed to have a bounded support, which is a special case of modeling the data with an overall probability distribution $\rho$. If the supports $\mathcal{M}_{m}$ are assumed to be nonintersecting, the regression function $f_{o}$ is given by

$$f_{o}(x)=\sum_{m=1}^{M}m\,I_{m}(x)$$

which corresponds to the class labels $m=1,\dots,M$, where $I_{m}$ is the indicator function of the support $\mathcal{M}_{m}$. It is then easy to show that the approximation error $\mathbb{E}[(f_{o}-\hat{f})^{2}]$ can be bounded as a constant times the probability of misclassification $P(\hat{C}(x)\neq m)$. Hence, we can compare our misclassification probability bounds in Section 2.3 with the approximation error in other works.

The study in (Niyogi and Girosi, 1996) assumes that the regression function is an element of the Bessel potential space of a sufficiently high order and that the sum of the coefficients $|c_{i}|$ is bounded. It is then shown that for data sampled from $\mathbb{R}^{n}$, with probability greater than $1-\delta$ the approximation error in (23) can be bounded as

$$\mathbb{E}[(f_{o}-\hat{f})^{2}]\leq O\left(\frac{1}{R}\right)+O\left(\sqrt{\frac{Rn\log(RN)-\log(\delta)}{N}}\right)$$

(24)

where $R$ is the number of RBF terms.

The analysis by Lin et al. (2014) considers families of RBF kernels that include the Gaussian function. Supposing that the regression function $f_{o}$ is of Sobolev class $W_{2}^{r}$, and that the number of RBF terms is given by $R=N^{\frac{n}{n+2r}}$ in terms of the number of samples $N$, the approximation error is bounded as

$$\mathbb{E}[(f_{o}-\hat{f})^{2}]\leq O(N^{-\frac{2r}{n+2r}}\log^{2}(N)).$$

(25)

Next, we overview the study by Hernández-Aguirre et al. (2002), which studies the performance of RBFs in a Probably Approximately Correct (PAC)-learning framework. For $X\subset\mathbb{R}^{n}$, a family $\mathcal{F}$ of measurable functions from $X$ to $[0,1]$ is considered and the problem of approximating a target function $f_{0}$ known only through examples with a function in $\hat{f}\in\mathcal{F}$ is studied. The authors use a previous result from (Vidyasagar, 1997) that relates the accuracy of empirical risk minimization to the covering number of $\mathcal{F}$ and the number of samples. Combining this result with the bounds on covering number estimates of Lipschitz continuous functions (Kolmogorov and Tihomirov, 1961), the following result is obtained for PAC function learning with RBF neural networks with Gaussian kernel. Let the coefficients be bounded as $|c_{i}|\leq A$, a common scale parameter be chosen as $\sigma_{i}=\sigma$, and $\mathbb{E}[|f_{0}-\hat{f}|]$ be computed under a uniform probability measure $\rho$. Then if the number of samples satisfies

$$N\geq\frac{8}{\varepsilon^{2}}\log\bigg{(}\frac{\sqrt{2}RnA}{e^{-1/2}\sigma\zeta}\bigg{)}$$

(26)

an approximation of the target function is obtained with accuracy parameter $\varepsilon$ and confidence parameter $\zeta$:

$$P(\mathbb{E}[|f_{0}-\hat{f}|]>\varepsilon)\leq\zeta.$$

(27)

In the above expression, the expectation is over the test samples, whereas the probability is over the training samples; i.e., over all possible distributions of training samples, the probability of having the average approximation error larger than $\varepsilon$ is bounded. Note that, our results in Theorems 8 and 9, when translated into the above PAC-learning framework, correspond to a confidence parameter of $\zeta=0$. This is because the misclassification probability bound of a test sample is valid for any choice of the training samples, provided that the condition (14) (or the condition (16)) holds. Thus, in our result the probability running over the training samples in (27) has no counterpart. When we take $\zeta=0$, the above result does not provide a useful bound since $N\rightarrow\infty$ as $\zeta\rightarrow 0$. By contrast, our result is valid only if the conditions (14), (16) on the interpolation function holds. It is easy to show that, assuming nonintersecting class supports $\mathcal{M}_{m}$, the expression $\mathbb{E}[|f_{0}-\hat{f}|]$ is given by a constant times the probability of misclassification. The accuracy parameter $\varepsilon$ can then be seen as the counterpart of the misclassification probability upper bound given on the right hand sides of (15) and (17) (the expression subtracted from 1). At fixed $N$, the dependence of the accuracy on the kernel scale parameter is monotonic in the bound (26); $\varepsilon$ decreases as $\sigma$ increases. Therefore, this bound does not guide the selection of the scale parameter of the RBF kernel, while the discussion in Section 2.4 (confirmed by the experimental results in Section 4.2) suggests the existence of an optimal scale.

Finally, we mention some results on the learning performance of regularized least squares regression algorithms. In (Caponnetto and De Vito, 2007) optimal rates are derived for the regularized least squares method in a Reproducing Kernel Hilbert Space (RKHS) in the minimax sense. It is shown that, under some hypotheses concerning the data probability measure and the complexity of the family of learnt functions, the maximum error (yielded by the worst distribution) obtained with the regularized least squares method converges at a rate of $O(1/N)$. Next, the work in (Steinwart et al., 2009) shows that, in regularized least squares regression over a RKHS, if the eigenvalues of the kernel integral operator decay sufficiently fast, and if the $\ell_{\infty}$-norms of regression functions can be bounded, the error of the classifier converges at a rate of up to $O(1/N)$ with high probability. Steinwart et al. also examine the learning performance in relation with the exponent of the function norm in the regularization term and show that the learning rate is not affected by the choice of the exponent of the function norm.

We now overview the three bounds given in (24), (25), and (26) in terms of the dependence of the error on the number of samples. The results in (24) and (25) provide a useful bound only in the case where the number of samples $N$ is larger than the number of RBF terms $R$, contrary to our study where we treat the case $R=N$. If it is assumed that $N$ is sufficiently larger than $R$, the result in (24) predicts a rate of decay of only $O(\sqrt{\log(N)/N})$ in the misclassification probability. The bound in (25) improves with the Sobolev regularity of the regression function; however, the dependence of the error on the number of samples is of a similar nature to the one in (24). Considering $\varepsilon$ as a misclassification error parameter in the bound in (26), the error decreases at a rate of $O(N^{-1/2})$ as the number of samples increases. The analysis in (Caponnetto and De Vito, 2007) and (Steinwart et al., 2009) also provide the similar rates of convergence of $O(N^{-1})$. Meanwhile, our results in Theorems 8 and 9 predict an exponential decay in the misclassification probability as the number of samples $N$ increases (under the reasonable assumption that $N_{m}=O(N)$ for each class $m$). The reason why we arrive at a more optimistic bound is the specialization of the analysis to the considered particular setting, where the support of each class is assumed to be restricted to a totally bounded region in the ambient space, as well as the assumed relations between the separation margin of the embedding and the regularity of the interpolation function.

Another difference between these previous results and ours is the dependence on the dimension. The results in (24), (25), and (26) predict an increase in the error at the respective rates of $O(\sqrt{n})$, $O(e^{-1/n})$, and $O(\sqrt{\log n})$ with the ambient space dimension $n$. While these results assume that the data $\mathcal{X}\subset\mathbb{R}^{n}$ is in an Euclidean space of dimension $n$, our study assumes the data $\mathcal{X}$ to be in a generic Hilbert space $H$. The results in Theorems 5-8 involve the dimension $d$ of the low-dimensional space of embedding and does not explicitly depend on the dimension of the ambient Hilbert space $H$ (which could be infinite-dimensional). However, especially in the context of manifold learning, it is interesting to analyze the dependence of our bound on the intrinsic dimension of the class supports $\mathcal{M}_{m}$.

In order to put the expressions (15), (17) in a more convenient form, let us reduce one parameter by setting $Q=N_{m}\eta_{m,\delta}/2$. Then the misclassification probability is of

$$O\left(\exp(-N_{m}\eta_{m,\delta}^{2})+N\exp\left(-\frac{N_{m}\,\eta_{m,\delta}\,\epsilon^{2}}{L_{\phi}^{2}\,\delta^{2}}\right)\right).$$

We can relate the dependence of this expression on the intrinsic dimension as follows. Since the supports $\mathcal{M}_{m}$ are assumed to be totally bounded, one can define a parameter $\Theta$ that represents the “diameter” of $\mathcal{M}_{m}$, i.e., the largest distance between any two points on $\mathcal{M}_{m}$. Then the measure $\eta_{m,\delta}$ of the minimum ball of radius $\delta$ in $\mathcal{M}_{m}$ is of $O((\delta/\Theta)^{D})$, where $D$ is the intrinsic dimension of $\mathcal{M}_{m}$. Replacing this in the above expression gives the probability of misclassification as

$$O\left(\exp\left(-\frac{N_{m}\,\delta^{2D}}{\Theta^{2D}}\right)+N\exp\left(-\frac{N_{m}\,\delta^{D-2}\,\epsilon^{2}}{L_{\phi}^{2}\,\Theta^{D}}\right)\right).$$

This shows that in order to retain the correct classification guarantee, as the intrinsic dimension $D$ grows, the number of samples $N_{m}$ should increase at a geometric rate with $D$. In supervised manifold learning problems, data sets usually have a low intrinsic dimension, therefore, this geometric rate of increase can often be tolerated. Meanwhile the dimension of the ambient space is typically high, so that performance bounds independent of the ambient space dimension are of particular interest. Note that generalization bounds in terms of the intrinsic dimension have been proposed in some previous works as well (Bickel and Li, 2007), (Kpotufe, 2011), for the local linear regression and the K-NN regression problems.