Tractable Computation of Expected Kernels (Supplementary material)

Let ${p}_{n}$ and ${q}_{m}$ be two compatible probabilistic circuits over variables $\mathbf{X}$ whose output units $n$ and $m$ are sum units, denoted by ${p}_{n}(\mathbf{X})=\sum_{i\in\mathsf{in}(n)}\theta_{i}{p}_{i}(\mathbf{X})$ and ${q}_{m}(\mathbf{X})=\sum_{j\in\mathsf{in}(m)}\delta_{j}{q}_{j}(\mathbf{X})$ respectively. Let $k_{l}$ be a kernel circuit with its output unit being a sum unit $l$, denoted by $k_{l}(\mathbf{X})=\sum_{c\in\mathsf{in}(l)}\gamma_{c}k_{c}(\mathbf{X})$. Then it holds that

Let ${p}_{n}$ and ${q}_{m}$ be two compatible probabilistic circuits over variables $\mathbf{X}$ whose output units $n$ and $m$ are product units, denoted by ${p}_{n}(\mathbf{X})={p}_{n_{\mathsf{L}}}(\mathbf{X}_{\mathsf{L}}){p}_{n_{\mathsf{R}}}(\mathbf{X}_{\mathsf{R}})$ and ${q}_{m}(\mathbf{X})={q}_{m_{\mathsf{L}}}(\mathbf{X}_{\mathsf{L}}){q}_{m_{\mathsf{R}}}(\mathbf{X}_{\mathsf{R}})$. Let $k$ be a kernel circuit that is kernel-compatible with the circuit pair ${p}_{n}$ and ${q}_{m}$ with its output unit being a product unit denoted by $k(\mathbf{X},\mathbf{X}^{\prime})=k_{\mathsf{L}}(\mathbf{X}_{\mathsf{L}},\mathbf{X}_{\mathsf{L}}^{\prime})k_{\mathsf{R}}(\mathbf{X}_{\mathsf{R}},\mathbf{X}_{\mathsf{R}}^{\prime})$. Then it holds that

Following the assumptions in Theorem LABEL:thm:_double_sum_complexity, the squared maximum mean discrepancy $\mathit{MMD}[\mathcal{H},p,q]$ in RKHS $\mathcal{H}$ associated with kernel $k$ as defined in gretton2012kernel can be tractably computed.

Following the assumptions in Theorem LABEL:thm:_double_sum_complexity, if the probabilistic circuit $p$ further satisfies determinism, the kernelized discrete Stein discrepancy (KDSD) $\mathbb{D}^{2}(q\leavevmode\nobreak\ \parallel\leavevmode\nobreak\ p)=\mathbb{E}_{\mathbf{x},\mathbf{x}^{\prime}\sim q}[k_{p}(\mathbf{x},\mathbf{x}^{\prime})]$ in the RKHS associated with kernel $k$ as defined in yang2018goodness can be tractably computed.

Before showing the proof for Corollary LABEL:cor:_tractable_kdsd, we first give definitions that are necessary for defining KDSD as follows to be self-contained.

Given the above definitions, the discrete Stein discrepancy between two distributions $p$ and $q$ is defined as

where $\bm{f}:\mathcal{X}\rightarrow\mathbb{R}^{D}$ is a test function, belonging to some function space $\mathcal{H}$ and $\mathcal{T}_{p}$ is the so-called Stein difference operator, which is defined as

If the function space $\mathcal{H}$ is an reproducing kernel Hilbert space (RKHS) on $\mathcal{X}$ equipped with a kernel function $k(\cdot,\cdot)$, then a kernelized discrete Stein discrepancy (KDSD) is defined and admits a closed-form representation as

Here, the kernel function $k_{p}$ is defined as

where the difference operator $\Delta^{\mathbf{x}}$ is as in Definition 1.3. The superscript $\mathbf{x}$ specifies the variables that it operates on.

Let $f(\mathbf{x})$ be a test function. Assume that $f-\mathbb{E}_{p}[f]\in\mathcal{H}_{p}$, with $\mathcal{H}_{p}$ being the RKHS associated with the kernel function $k_{p}$, and $\sum_{i}w_{i}=1$, then it holds that

where $C_{f}:=\parallel f-\mathbb{E}_{p}f\parallel_{\mathcal{H}_{p}}$. Moreover, the convergence rate is $\mathcal{O}(N^{-1/2})$.

Let $p(\mathbf{X_{c}}\mid\mathbf{x_{s}})$ be a PC that encodes a conditional distribution over variables $\mathbf{X_{c}}$ conditioned on $\mathbf{X_{s}}=\mathbf{x_{s}}$, and $k$ be a KC. If the PC $p(\mathbf{X_{c}}\mid\mathbf{x_{s}})$ and $p(\mathbf{X_{c}}\mid\mathbf{x_{s}}^{\prime})$ are compatible and $k$ is kernel-compatible with the PC pair for any $\mathbf{x_{s}}$, $\mathbf{x_{s}}^{\prime}$, then the conditional kernel function $k_{p,\mathbf{s}}$ as defined in Proposition LABEL:pro:_kdsd can be tractably computed.