Universal consistency of the $k$-NN rule in metric spaces and Nagata dimension

The $k$-nearest neighbour classifier, in spite of being arguably the oldest supervised learning algorithm in existence, still retains his importance, both practical and theoretical. In particular, it was the first classification learning rule whose (weak) universal consistency (in the finite-dimensional Euclidean space ${\mathbb{R}}^{d}=\ell^{2}(d)$) was established, by Charles J. Stone in [20].

Stone’s result is easily extended to all finite-dimensional normed spaces, see, e.g., [6]. However, the $k$-NN classifier is no longer universally consistent already in the infinite-dimensional Hilbert space $\ell^{2}$. A series of examples of this kind, obtained in the setting of real analysis, belongs to Preiss, and the first of them [17] is so simple that it can be described in a few lines. We will reproduce it in the article, since the example remains virtually unknown in the statistical machine learning community.

There is sufficient empirical evidence to support the view that the performance of the $k$-NN classifier greatly depends on the chosen metric on the domain (see e.g., [9]). There is a supervised learning algorithm, Large Margin Nearest Neighbour Classifier (LMNN), based on the idea of optimizing the $k$-NN performance over all Euclidean metrics on a finite dimensional vector space [21]. At the same time, it appears that a theoretical foundation for such an optimization over a set of distances is still lacking. The first question to address in this connection, is of course to characterize those metrics (generating the original Borel structure of the domain) for which the $k$-NN classifier is (weakly) universally consistent.

While the problem in this generality remains still open, a great advance in this direction was made by Cérou and Guyader in [2]. They have shown that the $k$-NN classifier is consistent under the assumption that the regression function $\eta(x)$ satisfies the weak Lebesgue–Besicovitch differentiation property:

where the convergence is in measure, that is, for each ${\epsilon}>0$,

The probability measure $\mu$ above is the sample distribution law. The proof extended the ideas of the paper [4], in which it was previously observed that Stone’s universal consistency can be deduced from the classical Lebesgue–Besicovitch differentiation theorem: every $L^{1}(\mu)$-function $f$ on ${\mathbb{R}}^{d}$ satisfies Eq. (1), even in the strong sense (convergence almost everywhere). See also [8].

Those separable metric spaces in which the weak Lebesgue–Besicovitch differentiation property holds for every Borel probability measure (equivalently, for every sigma-finite locally finite Borel measure) have not yet been characterized. But the complete separable metric spaces in which the strong Lebesgue–Besicovitch differentiation property holds for every such measure as above have been described by Preiss [18]: they are exactly those spaces that are sigma-finite dimensional in the sense of Nagata [13, 16]. (For finite dimensional spaces in the sense of Nagata, the sketch of a proof by Preiss, in the sufficiency direction, was elaborated by Assouad and Quentin de Gromard in [1]. The completeness assumption on the metric space is only essential for the necessity part of the result.) In particular, it follows that every sigma-finite dimensional separable metric space satisfies the weak Lebesgue–Besicovitch differentiation property for every probability measure.

Combining the result of Preiss with that of Cérou–Guyader, one concludes that the $k$-NN classifier is universally consistent in every sigma-finite dimensional separable metric space, as was noted in [2].

The authors of [2] mention in their paper that “[Stone’s theorem] is based on a geometrical result, known as Stone’s Lemma. This powerful and elegant argument can unfortunately not be generalized to infinite dimension.” The aim of this article is to show that at least Stone’s original proof, including Stone’s geometric lemma as its main tool, can be extended from the Euclidean case to the sigma-finite dimensional metric spaces. In fact, as we will show, the geometry behind Stone’s lemma, even if it appears to be essentially based on the Euclidean structure of the space, is captured by the notion of Nagata dimension, which is a purely metric concept. In this way, Stone’s geometric lemma and indeed the original Stone’s proof of the universal consistency of the $k$-NN classifier, become applicable to a wide range of metric spaces.

In the absence of distance ties (that is, in case where every sphere is a $\mu$-negligible set with regard to the underlying measure $\mu$), the extension is quite straightforward, indeed almost literal. However, this is not so in the presence of distance ties: an example shows that the conclusion of Stone’s geometric lemma may not hold. Another example shows that even in the compact metric spaces of Nagata dimension zero, the distance ties may be everpresent. We also show that an attempt to reduce the case to the situation without distance ties by learning in the product of $\Omega$ with the unit interval (an additional random variable used for tie-breaking) cannot work, because already the product of a zero-dimensional space in the sense of Nagata with the interval (which has dimension one) can have an infinite Nagata dimension. Stone’s geometric lemma has to be modified, to parallel the Hardy–Littlewood inequality in the geometric measure theory.

We do not touch upon the subject of strong universal consistency in general metric spaces. The main open question left is whether every metric space in which the $k$-NN classifier is universally consistent is necessarily sigma-finite dimensional. A positive answer, modulo the work of [2] and [18], would also answer in the affirmative an open question in real analysis going back to Preiss: suppose a metric space $X$ satisfies the weak Lebesgue–Besicovitch differentiation property for every sigma-finite locally finite Borel measure, will it satisfy the strong Lebesgue–Besicovitch differentiation property for every such measure?

Here we will recall the standard probabilistic model for statistical learning theory. The domain, $\Omega$, means a standard Borel space, that is, a set equipped with a sigma-algebra which coincides with the sigma-algebra of Borel sets generated by a suitable separable complete metric. (Recall that the Borel structure generated by a metric $\rho$ on a set $\Omega$ is the smallest sigma-algebra containing all open subsets of the metric space $(\Omega,\rho)$.) The distribution laws for datapoints, both unlabelled and labelled, are Borel probability measures defined on the corresponding Borel sigma-algebra.

Since we will be dealing with the $k$-NN classifier, the domain, $\Omega$, will actually be a metric space, which we also assume to be separable.

Labelled data pairs $(x,y)$, where $x\in\Omega$ and $y\in\{0,1\}$, will follow an unknown probability distribution $\tilde{\mu}$, that is, a Borel probability measure on $\Omega\times\{0,1\}$. We denote the corresponding random element $(X,Y)\sim\tilde{\mu}$. Define two Borel measures on $\Omega$, $\mu_{i}$, $i=0,1$, by $\mu_{i}(A)=\tilde{\mu}(A\times\{i\})$. In this way, $\mu_{0}$ is governing the distribution of the elements labelled $0$, and similarly for $\mu_{1}$. The sum $\mu=\mu_{0}+\mu_{1}$ (the direct image of $\tilde{\mu}$ under the projection from $\Omega\times\{0,1\}$ onto $\Omega$) is a Borel probability measure on $\Omega$, the distribution law of unlabelled data points. Clearly, $\mu_{i}$ is absolutely continuous with regard to $\mu$, that is, if $\mu(A)=0$, then $\mu_{i}(A)=0$ for $i=0,1$. The corresponding Radon-Nikodým derivative in the case $i=1$ is just the conditional probability for a point $x$ to be labeled $1$:

In statistical terminology, $\eta$ is the regression function.

Together with the Borel probability measure $\mu$ on $\Omega$, the regression function allows for an alternative, and often more convenient, description of the joint law $\tilde{\mu}$. Namely, given $A\subseteq\Omega$,

and

which allows to reconstruct the measure $\tilde{\mu}$ on $\Omega\times\{0,1\}$.

Let ${\mathcal{B}}(\Omega,\{0,1\})$ denote the collection of all Borel measurable binary functions on the domain, that is, essentially, the family of all Borel subsets of $\Omega$. Given such a function $f\colon\Omega\to\{0,1\}$ (a classifier), the misclassification error is defined by

The Bayes error is the infimal misclassification error taken over all possible classifiers:

It is a simple exercise to verify that the Bayes error is achieved on some classifier (and thus is the minimum), which is called a Bayes classifier. For instance, every classifier satisfying

is a Bayes classifier.

The Bayes error is zero if and only if the learning problem is deterministic, that is, the regression function $\eta$ is equal almost everywhere to the indicator function, $\chi_{C}$, of a concept $C\subseteq\Omega$, a Borel subset of the domain.

A learning rule is a family of mappings ${\mathcal{L}}=\left({\mathcal{L}}_{n}\right)_{n=1}^{\infty}$, where

and the functions ${\mathcal{L}}_{n}$ satisfy the following measurability assumption: the associated maps

are Borel (or just universally measurable). Here $\sigma=(x_{1},\ldots,x_{n},y_{1},\ldots,y_{n})$ is a labelled learning sample.

The data is modelled by a sequence of independent identically distributed random elements $(X_{n},Y_{n})$ of $\Omega\times\{0,1\}$, following the law $\tilde{\mu}$. Denote $\varsigma$ an infinite sample path. In this context, $\mathcal{L}_{n}$ only gets to see the first $n$ labelled coordinates of $\varsigma$. A learning rule $\mathcal{L}$ is weakly consistent, or simply consistent, if ${\mathrm{err}}_{\mu}{\mathcal{L}}_{n}(\varsigma)\to\ell^{\ast}$ in probability as $n\to\infty$. If the convergence occurs almost surely (that is, along almost all sample paths $\varsigma\sim\tilde{\mu}^{\infty}$), then $\mathcal{L}$ is said to be strongly consistent. Finally, $\mathcal{L}$ is universally (weakly / strongly) consistent if it is weakly / strongly consistent under every Borel probability measure $\tilde{\mu}$ on the standard Borel space $\Omega\times\{0,1\}$.

The learning rule we study is the $k$-NN classifier, defined by selecting the label ${\mathcal{L}}_{n}(\sigma)(x)\in\{0,1\}$ by the majority vote among the values of $y$ corresponding to the $k=k_{n}$ nearest neighbours of $x$ in the learning sample $\sigma$.

If $k$ is even, then a voting tie may occur. This is of lesser importance, and can be broken in any way. For instance, by always assigning the value $1$ in case of a voting tie, or by choosing the value randomly. The consistency results usually do not depend on it. Intuitively, if voting ties keep occurring asymptotically at a point $x$ along a sample path, it means that $\eta(x)=1/2$ and so any value of the classifier assigned to $x$ would do.

It may also happen that the smallest closed ball containing $k$ nearest neighbours of a point $x$ contains more than $k$ elements of a sample (distance ties). This situation is more difficult to manage and requires a consistent tie-breaking strategy, whose choice may affect the consistency results.

Given $k$ and $n\geq k$, we define $r^{\varsigma_{n}}_{k\mbox{\tiny-NN}}(x)$ as the smallest radius of a closed ball around $x$ containing at least $k$ nearest neighbours of $x$ in the sample $\varsigma_{n}$:

As the corresponding open ball around $x$ contains at most $k-1$ elements of the sample, the ties may only occur on the sphere.

We adopt the combinatorial notation $[n]=\{1,2,\ldots n\}$. If $\sigma\in\Omega^{n}$ and $\sigma^{\prime}\in\Omega^{k}$, $k\leq n$, the symbol

means that there is an injection $f\colon[k]\to[n]$ such that

A $k$ nearest neighbour map is a function

with the properties

1. (1)

   $k\mbox{-NN}^{\sigma}(x)\sqsubset\sigma$, and

2. (2)

   all points $x_{i}$ in $\sigma$ that are at a distance strictly less than $r^{\varsigma_{n}}_{k\mbox{\tiny-NN}}(x)$ to $x$ are in $k\mbox{-NN}^{\sigma}(x)$.

The mapping $k\mbox{-NN}^{\sigma}$ can be deterministic or stochastic, in which case it will depend on an additional random variable, independent of the sample.

An example of the former kind is based on the natural order on the sample, $x_{1}<x_{2}<\ldots<x_{n}$. In this case, from among the points belonging to the sphere of radius $r_{k\mbox{\tiny-NN}^{\sigma}}(x)$ around $x$ we choose points with the smaller index: $k\mbox{-NN}^{\sigma}(x)$ contains all the points of $\sigma$ in the open ball, $B_{r_{k\mbox{\tiny-NN}^{\sigma}}(x)}(x)$, plus a necessary number (at least one) of points of $\sigma\cap S_{r_{k\mbox{\tiny-NN}^{\sigma}}(x)}(x)$ having smallest indices.

An example of the second kind is to use a similar procedure, after applying a random permutation of the indices first. A random learning input will consist of a pair $(W_{n},P_{n})$, where $W_{n}$ is a random $n$-sample and $P_{n}$ is a random element of the group of permutations of rank $n$. An equivalent (and more common) way would be to use a sequence of i.i.d. random elements $Z_{n}$ of the unit interval or the real line, distributed according to the uniform (resp. gaussian) law, and in case of a tie, give a preference to a realization $x_{i}$ over $x_{j}$ provided the value $z_{i}$ is smaller than $z_{j}$.

Now, a formal definition of the $k$-NN learning rule can be given as follows:

Here, $\chi_{[0,\infty)}$ is the Heaviside function, the sign of the argument:

The empirical measure $\mu_{k\mbox{\tiny-NN}^{\sigma}(x)}$ is a uniform measure supported on the set of $k$ nearest neighbours of $x$ within the sample $\sigma$, and the label ${\epsilon}$ is seen as a function ${\epsilon}\colon\{x_{1},x_{2},\ldots,x_{n}\}\to\{0,1\}$.

The expression appearing under the argument,

is the empirical regression function. In the presence of a law of labelled points, it is a random variable, and so we have the following immediate, yet important, observation.

Let $(\mu,\eta)$ be a learning problem in a separable metric space $(\Omega,d)$. If the values of the empirical regression function, $\eta_{n,k}$, converge to $\eta$ in probability (resp. almost surely) in the region

then the $k$-NN classifier is consistent (resp. strongly consistent) under $(\mu,\eta)$.

We conclude this section by recalling an important technical tool.

[Cover-Hart lemma [3]] Let $\Omega$ be a separable metric space, and let $\mu$ be a Borel probability measure on $\Omega$. Almost surely, the function $r^{\varsigma_{n}}_{k\mbox{\tiny-NN}}$ (Eq. (2)) converges to zero uniformly over any precompact subset $K\subseteq\mbox{supp}\,\mu$.

Here we will discuss a 1979 example of Preiss [17]. Preiss’s aim was to prove that the Lebesgue–Besicovitch differentiation property (Eq. (1)) fails in the infinite-dimensional Hilbert space $\ell^{2}$. However, as already suggested in [2], his example can be easily adapted to prove that the $k$-NN learning rule is not universally consistent in the infinite-dimensional separable Hilbert space $\ell^{2}$ either.

Recall the notation $[n]=\{1,2,\ldots n\}$. Let $(N_{k})$ be a sequence of positive natural numbers $\geq 2$, to be selected later. Denote by

the Cartesian product of finite discrete spaces equipped with the product topology. It is a Cantor space (the unique, up to a homeomorphism, totally disconnected compact metrizable space without isolated points).

Let $\pi_{k}$ denote the canonical cordinate projections of $Q$ on the $k$-dimensional cubes $Q_{k}=\prod_{i=1}^{k}[N_{i}]$. Denote $Q^{\ast}=\cup_{k=1}^{\infty}Q_{k}$ a disjoint union of the cubes $Q_{k}$, and let ${\mathcal{H}}=\ell^{2}(Q^{\ast})$ be a Hilbert space spanned by an orthonormal basis $(e_{\bar{n}})$ indexed by elements $\bar{n}$ of this union.

For every $\bar{n}=(n_{1},\ldots,n_{i},\ldots)\in Q$ define

The map $f$ is continuous and injective, thus a homeomorphism onto its image. Denote $\nu$ the Haar measure on $Q$ (the product of the uniform measures on all $[N_{k}]$). Let $\mu_{1}=f_{\ast}(\nu)$ be the direct image of $\nu$, a compactly-supported Borel probability measure on $\mathcal{H}$. If $r>0$ satisfies $2^{-k}\leq r^{2}<2^{-k+1}$, then for each $\bar{n}=(n_{1},n_{2},\ldots)\in Q$,

Now, for every $k$ and each $\bar{n}=(n_{1},\ldots,n_{k})\in Q_{k}\subseteq Q^{\ast}$ define in a similar way

Note that the closure of $f(Q^{\ast})$ contains $f(Q)$ (as a proper subset). Now define a purely atomic measure $\mu_{0}$ supported on the image of $Q^{\ast}$ under $f$, having the following special form:

The weights $a_{k}>0$ are chosen so that the measure is finite:

Since for $r$ satisfying $2^{-k}\leq r^{2}<2^{-k+1}$ and $\bar{n}\in Q$ the ball $B_{r}(f(\bar{n}))$ contains in particular $f(n_{1}n_{2},\ldots,n_{k})$, we have

Assuming in addition that

we conclude:

Clearly, the conditions (4) and (5) can be simultaneously satisfied by a recursive choice of $(N_{k})$ and $(a_{k})$.

Now renormalize the measures $\mu_{0}$ and $\mu_{1}$ so that $\mu=\mu_{0}+\mu_{1}$ is a probability measure, and interpret $\mu_{i}$ as the distribution of points labelled $i=0,1$. Thus, the regression function is deterministic, $\eta=\chi_{C}$, where we are learning the concept $C=f(Q)={\mathrm{supp}}\,\mu_{1}$, $\mu_{1}(C)>0$.

For a random element $X\in{\mathcal{H}}$, $X\sim\mu$, the distance $r_{k}(X)$ to the $k$-th nearest neighbour within an i.i.d. $n$-sample goes to zero almost surely when $k/n\to 0$, according to a lemma of Cover and Hart, and the convergence is uniform on the precompact support of $\mu$. It follows that the probability of one of the $k$ nearest neighbours to a random point $X\in{\mathcal{H}}$ to be labelled one, conditionally on $r^{\varsigma_{n}}_{k\mbox{\tiny-NN}}=r$, converges to zero, uniformly in $r$. The $k$-NN learning rule will almost surely predict a sequence of classifiers converging to the identically zero classifier, and so is not consistent.

Recall that a family $\gamma$ of subsets of a set $\Omega$ has multiplicity $\leq\delta$ if the intersection of more than $\delta$ different elements of $\gamma$ is always empty. In other words,

Let $\delta\in{\mathbb{N}}$, $s\in(0,+\infty]$. We say that a metric space $(\Omega,d)$ has Nagata dimension $\leq\delta$ on the scale $s>0$, if every finite family $\gamma$ of closed balls of radii $<s$ admits a subfamily $\gamma^{\prime}$ of multiplicity $\leq\delta+1$ which covers the centres of all the balls in $\gamma$. The smallest $\delta$ with this property, if it exists, and $+\infty$ otherwise, is called the Nagata dimension of $\Omega$ on the scale $s>0$. A space $\Omega$ has Nagata dimension $\delta$ if it has Nagata dimension $\delta$ on a suitably small scale $s\in(0,+\infty]$. Notation: $\dim^{s}_{Nag}(\Omega)=\delta$, or simply $\dim_{Nag}(\Omega)=\delta$.

Sometimes the following reformulation is more convenient.

A metric space $(\Omega,d)$ has Nagata dimension $\leq\delta$ on the scale $s\in(0,+\infty]$ if and only if it satisfies the following property. For every $x\in\Omega$, $r<s$, and a sequence $x_{1},\ldots,x_{\delta+2}\in\bar{B}_{r}(x)$, there are $i,j$, $i\neq j$, such that $d(x_{i},x_{j})\leq\max\{d(x,x_{i}),d(x,x_{j})\}$.

The property of a metric space $\Omega$ having Nagata dimension zero on the scale $+\infty$ is equivalent to $\Omega$ being a non-archimedian metric space, that is, a metric space satisfying the strong triangle inequality, $d(x,z)\leq\max\{d(x,y),d(y,z)\}$.

Indeed, $\dim_{Nag}^{+\infty}(\Omega)=0$ means exactly that for any sequence of $\delta+2=2$ points, $x_{1},x_{2}$, contained in a closed ball $B_{r}(x)$, we have $d(x_{1},x_{2})\leq\max\{d(x,x_{1}),d(x,x_{2})\}$.

It follows from Proposition 4 that $\dim_{Nag}({\mathbb{R}})=1$. Let $x_{1},x_{2},x_{3}$ be three points contained in a closed ball, that is, an interval $[x-r,x+r]$. Without loss in generality, assume $x_{1}<x_{2}<x_{3}$. If $x_{2}\leq x$, then $\lvert x_{1}-x_{2}\rvert\leq\lvert x_{1}-x\rvert$, and if $x_{2}\geq x$, then $\lvert x_{3}-x_{2}\rvert\leq\lvert x_{3}-x\rvert$.

The following example suggests that the Nagata dimension is relevant for the study of the $k$-NN classifier, as it captures in an abstract context the geometry behind Stone’s lemma.

The Nagata dimension of the Euclidean space $\ell^{2}(d)$ is finite, and it is bounded by $C(d)-1$, where $C(d)$ is the value of the constant in Stone’s geometric lemma (Lemma 8).

Indeed, let $x_{1},\ldots,x_{C(d)+1}$ be points belonging to a ball with centre $x$. Using the argument in the proof of Stone’s geometric lemma with $k=1$, mark $\leq C(d)$ points $x_{i}$ belonging to the $\leq C(d)$ cones with apex at $x$. At least one point, say $x_{j}$, has not been marked; it belongs to some cone, which therefore already contains a marked point, say $x_{i}$, different from $x_{j}$, and ${\left\|\,x_{i}-x_{j}\,\right\|}\leq{\left\|\,x_{j}-x\,\right\|}$.

A similar argument shows that every finite-dimensional normed space has finite Nagata dimension.

In ${\mathbb{R}}^{2}={\mathbb{C}}$ the family of closed balls of radius one centred at the vectors $\exp(2\pi ki/5)$, $k=1,2,\ldots,5$, has multiplicity $5$ and admits no proper subfamily containing all the centres. Therefore, the Nagata dimension of $\ell^{2}(2)$ is at least $5$. Since the plane can be covered with $6$ cones having the central angle $\pi/3$, Example 4 implies that $\dim_{Nag}(\ell^{2}(2))=5$.

The problem of calculating the Nagata dimension of the Euclidean space $\ell^{2}(d)$ is mentioned as “possibly open” by Nagata [15], p. 9 (where the value $\dim_{Nag}+2$ is called the “crowding number”). Nagata also remarks that $\dim_{Nag}{\mathbb{R}}^{1}=1$ and $\dim_{Nag}\ell^{2}(3)=5$ (without a proof).

Notice that the property of the Euclidean space established in the proof of Stone’s geometric lemma is strictly stronger than the finiteness of the Nagata dimension. There exists a finite $\delta$ (in general, higher than the Nagata dimension) such that, given a sequence $x_{1},\ldots,x_{\delta+2}\in\bar{B}_{r}(x)$, $r<s$, there are $i,j$, $i\neq j$, such that $d(x_{i},x_{j})<\max\{d(x,x_{i}),d(x,x_{j})\}$. The inequality here is strict, cf. Remark 2. This is exactly the property that removes the problem of distance ties in the Euclidean space. However, adopting this as a definition in the general case would be too restrictive, removing from consideration a large class of metric spaces in which the $k$-NN classifier is still universally consistent, such as all non-archimedean metric spaces.

Let $e_{n}$ denote the $n$-th standard basic vector in the separable Hilbert space $\ell^{2}$, that is, a sequence whose $n$-th coordinate is $1$ and the rest are zeros. The convergent sequence $(1/n)e_{n}$, $n\geq 0$, together with the limit $0$, viewed as a metric subspace of $\ell^{2}$, has infinite Nagata dimension on every scale $s>0$. This is witnessed by the family of closed balls $B_{1/n}((a/n)e_{n})$, having zero as the common point, and having the property that every centre belongs to exactly one ball of the family. Realizing ${\mathbb{R}}$ as a continuous curve in $\ell^{2}$ without self-intersections passing through all elements of the sequence as well as the limit leads to an equivalent metric on ${\mathbb{R}}$ having infinite Nagata dimension on each scale.

The Nagata–Ostrand theorem [13, 16] states that the Lebesgue covering dimension of a metrizable topological space is the smallest Nagata dimension of a compatible metric on the space (and in fact this is true on every scale $s>0$, [12]). This is the historical origin of the concept of the metric dimension.

There appears to be no single comprehensive reference to the concept of Nagata dimension. Various results are scattered in the journal papers [1, 12, 13, 15, 16, 18], see also the book [14], pages 151–154.

Metric spaces of finite Nagata dimension admit an almost literal version of Stone’s geometric lemma in case where the sample has no distance ties, that is, the values of the distances $d(x_{i},x_{j})$, $i\neq j$, are all pairwise distinct.

[Stone’s geometric lemma, finite Nagata dimension, no ties] Let $\Omega$ a metric space of Nagata dimension $\delta<\infty$ on a scale $s>0$. Let

be a finite sample in $\Omega$, and let $x\in\Omega$ be any. Suppose there are no distance ties inside the sample

and $k$ is such that, inside the above sample, $r_{k\mbox{\tiny-NN}}(x_{i})<s$ for all $i$. The number of $i$ having the property that $x\neq x_{i}$ and $x$ is among the $k$ nearest neighbours of $x_{i}$ inside the sample above is limited by $(k+1)(\delta+1)$.

Now the same argument as in the original proof of Stone (Subs. 3.3) shows that the $k$-NN classifier is consistent under each distritution $\mu$ on $\Omega\times\{0,1\}$ with the property that the distance ties occur with zero probability. Since we are going to give a proof of a more general result, we will not repeat the argument here, only mention that due to the Cover–Hart lemma, if $n$ is sufficiently large, then with arbitrarily high probability, the $k$ nearest neighbours of a random point inside a random sample will all lie at a distance $<s$.

In this section we will construct a series of examples to illustrate the difficulties arising in the presence of distance ties in general metric spaces that are absent in the Euclidean case. The fundamental difference between the two situations is the inequality in the equivalent definition of the Nagata dimension (proposition 4) that is, unlike in the Euclidean space, no longer strict.

As we have already noted (remark 2), the conclusion of Stone’s geometric lemma 8 remains valid even if we allow the adversary to break the distance ties and pick the $k$ nearest neighbours. Our first example shows that it is no longer the case in a metric space of finite Nagata dimension.