On the Probabilistic Approximation in
Reproducing Kernel Hilbert Spaces

Let $X$ be a set, $\mathbb{F}=\mathbb{R}$ or $\mathbb{C}$, and $\mathscr{F}(X,\mathbb{F})$ the set of functions from $X$ to $\mathbb{F}$. $\mathscr{F}(X,\mathbb{F})$ is naturally equipped with the vector space structure over $\mathbb{F}$ by pointwise addition and scalar multiplication:

A vector subspace $\mathscr{H}\subset\mathscr{F}(X,\mathbb{F})$ is said to be a reproducing kernel Hilbert space (RKHS) on $X$ if

• •

  $\mathscr{H}$ is endowed with a Hilbert space structure $\langle\cdot,\cdot\rangle$. Our convention is that this is $\mathbb{F}$-linear in the first argument.

• •

  for every $x\in X$, the linear evaluation functional $E_{x}:\mathscr{H}\rightarrow\mathbb{F}$, defined by $E_{x}(f)=f(x)$, is bounded.

If $\mathscr{H}$ is an RKHS on $X$, then Riesz representation theorem shows that for each $x\in X$, there exists a unique vector $k_{x}\in\mathscr{H}$ such that for any $f\in\mathscr{H}$,

The function $k_{x}$ is called the reproducing kernel for the point $x$, and the function $K:X\times X\rightarrow\mathbb{F}$ defined by $K(y,x)=k_{x}(y)$ is called the reproducing kernel for $\mathscr{H}$. One can check that $K$ is indeed a kernel function, meaning that for any $n\in{\mathbb{N}}$ and any $n$ distinct points $\{x_{1},\cdots,x_{n}\}\subset X$, the matrix $(K(x_{i},x_{j}))$ is symmetric (Hermitian) and positive semidefinite. It is well-known that there is a one-to-one correspondence between RKHSs and kernel functions on $X$: by Moore’s theorem [5], if $K:X\times X\rightarrow\mathbb{F}$ is a kernel function, then there exists a unique RKHS $\mathscr{H}$ on $X$ such that $K$ is the reproducing kernel of $\mathscr{H}$. We let $\mathscr{H}(K)$ denote the unique RKHS with the reproducing kernel $K$, and define the feature map $\phi:X\rightarrow\mathscr{H}(K)$ by $\phi(x)=k_{x}$. We refer to [1, 2, 4, 6, 7, 8] for more details on the RKHS and its applications.

One of the interesting topics on the RKHS is interpolation. Let $\mathscr{H}(K)$ be an RKHS on $X$, $F=\{x_{1},\cdots,x_{N}\}$ a finite set of distinct points in $X$, and $\{c_{1},\dots,c_{N}\}\subset\mathbb{F}$. If the matrix $(K(x_{i},x_{j}))$ is invertible, then there exists $f\in\mathscr{H}(K)$ such that $f(x_{i})=c_{i}$ for all $1\leq i\leq N$. However, if $(K(x_{i},x_{j}))$ is not invertible, such $f$ may not exist. In this case, one is often interested in finding the best approximation in $\mathscr{H}(K)$ to minimize the least square error:

The theorem below shows the existence of the optimizer and describes its structure:

Now, let $\mathscr{P}(X)$ be the set of probability measures on $X$ and $\mu_{N}=\frac{1}{N}\sum\limits_{i=1}^{N}\delta_{x_{i}}\in\mathscr{P}(X)$. Then the above least square error problem is equivalent to

where $g$ is any given function with $g(x_{i})=c_{i}$ for all $i=1,\dots,N$. This inspires us to replace $\mu_{N}$ with any probability measure $\mu\in\mathscr{P}(X)$ and consider the probabilistic approximation problem in the RKHS.

The general formulation is as follows. Throughout this paper, we assume that $X$ is a Polish space, and all functions and measures considered are Borel measurable. Let $g:X\rightarrow\mathbb{F}$ be a given function, $\mu\in\mathscr{P}(X)$, and $c:\mathbb{F}\times\mathbb{F}\rightarrow\mathbb{R}^{+}$ a nonnegative cost function. We then consider the following minimization problem

Our first result is about the case when the cost function $c$ is from the $L^{p}$ norm, and the feature map $\phi$ is a continuous $p$-frame ¹¹1A set of vectors $\{f_{x}\}_{x\in X}$ in $\mathscr{H}(K)$ is a continuous $p$-frame with respect to $\mu\in\mathscr{P}(X)$ if there exist $0<A\leq B$ such that for any $f\in{\mathscr{H}(K)}$, $A\|f\|^{p}_{\mathscr{H}(K)}\leq\int_{X}|\langle f,f_{x}\rangle|^{p}d\mu(x)\leq B\|f\|^{p}_{\mathscr{H}(K)}$. for $\mathscr{H}(K)$ with respect to $\mu$.

Note that the continuous $p$-frame condition is the same as that the $L^{p}$ norm is equivalent to the Hilbert space norm, as in the following inequality:

for some $0<C_{1}\leq C_{2}<\infty$. Thus we can rewrite Theorem 1.2 as the following:

In the special case where $p=2$ and $\mathscr{H}$ is a Hilbert subspace of $L^{2}(X,\mu)$, such a unique closest vector is classically given by the orthogonal projection of $g$ onto $\mathscr{H}$. Although $L^{p}(X,\mu)$ is not a Hilbert space for general $p\neq 2$, our corollary (under some assumptions) provides such a unique optimizer in the probabilistic approximation sense, which can be viewed as "projections" to the given subspace.

Our next result involves adding an extra regularization term to the minimization problem. In statistical regression and machine learning, a regularization term is preferred to perform variable selection, enhance the prediction accuracy, and prevent overfitting. Examples of such practice are ridge regression [3] and Lasso regression [10]. In this paper, we consider the following minimization problem with regularization:

We show the existence and uniqueness of the optimizer in the following theorem:

The theorems in this section will be proved in the last part of this paper. In the following sections, we will show some representer-type theorems describing the optimizer, mainly that from Theorem 1.4.

As witnessed by the work of Wahba [11] and followed by Schölkopf, Herbrich, and Smola [9], the celebrated representer theorem (Theorem 2.1 below) states that the optimizer in an RKHS that minimizes the regularized loss function is in the linear span of the kernel function at the given points.

The representer theorem is useful in practice since it turns the minimization problem into a finite-dimensional optimization problem. It is natural to ask whether there is a corresponding version of the representer theorem under the probabilistic approximation setting as we introduced in the previous section. We confirm this speculation with the following representer theorem in measure representation forms.

The finiteness condition 2.1 in Theorem 2.2 holds when $\mu$ is finitely supported, or when $X$ is compact and $K$ is continuous. Also, the condition 2.1 holds on the Hardy space $H^{2}(D)$ when $\operatorname{supp}\mu$ is a compact set in the open unit disk $D$, as we will see in Example 3.2 later.

We can furthermore assume that the $\mathbb{F}$-measures $\xi$ on $X$ are finitely supported, and the set $\mathscr{A}$ then becomes the linear span of $\{k_{x}\}_{x\in\operatorname{supp}(\mu)}$. This leads to the following corollary:

Note that when $\mu$ is finitely supported, Corollary 2.3 recovers the celebrated Representer Theorem 2.1. On the other hand, when $\mathscr{H}(K)$ is finite-dimensional, the unique optimizer is also in the linear span of $\{k_{x}\}_{x\in\operatorname{supp}(\mu)}$, which does not depend on the cardinality of the support of the measure $\mu$. Both these two cases indicate that the probabilistic approximation problem turns out to be a measure quantization problem about the measure $\mu$ with respect to the loss function 1.2.

The preferable result in the probabilistic representer theorem (Theorem 2.2) is that the minimizer $\hat{f}$ can be represented directly by an $\mathbb{F}$-measure $\nu$, instead of an approximating sequence $\{\nu_{n}\}$. We conjecture such nice form holds merely under the Assumption 2.1:

To support this conjecture as well as to illustrate Theorem 2.2, we here provide the following example: