Properties of f-divergences and f-GAN training

Given a strictly convex twice continuously differentiable function $f\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}_{>0}\to\mathbb{R}$, the $f$-divergence between probability distributions with densities¹¹1Most results also hold for “discrete” probability distributions. $p$ and $q$ over $\mathbb{R}^{K}$ is defined as

For simplicity, we assume the probability distributions $p$ and $q$ are suitably nice, e.g. absolutely continuous with respect to the Lebesgue measure on $\mathbb{R}^{K}$, $p(x),q(x)>0$ for $x\in\mathbb{R}^{K}$, and $p$ and $q$ continuously differentiable. These assumptions are discussed in §3.4. We refer to $f$ as the defining function of the divergence $D_{f}$.

There is some redundancy in the above definition. Adding an affine-linear term to the defining function just adds a constant to the divergence: if $g(u)=f(u)+a+bu$ for $a,b\in\mathbb{R}$ then $D_{g}(p,q)=D_{f}(p,q)+a+b$ for all $p$ and $q$. Typically we do not care about an overall additive shift and would regard $D_{f}$ and $D_{g}$ as essentially the same divergence. Thus we have identified two unnecessary degrees of freedom in the specification of the defining function, and these may be removed by fixing $f(1)=f^{\prime}(1)=0$. This results in no loss of generality since any defining function can be put in this form by adding a suitable affine-linear term.²²2This affine-linear term also does not affect the various bounds and finite sample approximations derived below, as long as the reparameterization trick is used for the generator gradient as is standard practice. In the discrete case or if other finite sample approximation such as naive REINFORCE is used then adding an affine-linear term to $f$ may affect the variance of the finite sample approximation of the generator gradient. From here on we assume $f(1)=f^{\prime}(1)=0$ as part of our definition of an f-divergence. The choice $f(1)=0$ ensures $D_{f}(p,q)=0$ when $p=q$. The choice $f^{\prime}(1)=0$ has the ancillary benefit of (1) being non-negative even if $p$ and $q$ are positive functions that do not integrate to one.

There is also a multiplicative degree of freedom in the defining function that is often irrelevant: multiplying the defining function by a constant $k>0$ just multiplies the divergence by the same constant. Removing this superficial source of variation makes different f-divergences easier to compare. We refer to an f-divergence as being canonical if $f^{\prime\prime}(1)=1$. Any f-divergence can be made canonical by scaling appropriately. Intuitively, fixing $f^{\prime\prime}(1)$ corresponds to fixing the behavior of the f-divergence in the region $u\approx 1$, corresponding to $p\approx q$. This will be made precise below.

f-divergences satisfy several mathematical properties:

• •

  $D_{f}$ is linear in $f$.

• •

  $D_{f}(p,q)\geq 0$ for all distributions $p$ and $q$ with equality iff $p=q$. This justifies referring to $D_{f}$ as a divergence.

• •

  $D_{f}$ uniquely determines $f$.

• •

  All f-divergences agree up to an overall scale factor on the divergence between nearby distributions: If $p\approx q$ then $D_{f}(p,q)\approx f^{\prime\prime}(1)\operatorname{KL}(p\,\|\,q)$ to second order. In particular all canonical f-divergences agree when $p\approx q$.

• •

  For many common f-divergences, $f^{\prime\prime}$ has a simpler algebraic form than $f$ and is easier to work with.

Linearity is straightforward to verify. If $D_{f}$ and $D_{g}$ are two f-divergences and $k>0$ then $D_{f+g}=D_{f}+D_{g}$ and $D_{(kf)}=kD_{f}$. If $f$ and $g$ are strictly convex then so is $f+g$ and $kf$, and $(f+g)(1)=f(1)+g(1)=0$ and $(kf)(1)=kf(1)=0$, and similarly for $f^{\prime}$, so $D_{f+g}$ and $D_{kf}$ are valid f-divergences.

The non-negativity of $D_{f}$ follows from the convexity of $f$. Since $f(1)=f^{\prime}(1)=0$ and $f$ is strictly convex, $f(u)\geq 0$ with equality iff $u=1$. Thus

Thus $D_{f}(p,q)\geq 0$ for all $p$ and $q$. In general, if $\int g(x)\operatorname{d\!}x=0$ for a continuous integrable non-negative function $g\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}$ then $g(x)=0$ for all $x$. Applying this to $g(x)=q(x)f(p(x)/q(x))$ we see that we have equality in (2) iff $p(x)/q(x)=1$ for all $x$. Thus $D_{f}(p,q)=0$ implies $p=q$. The non-negativity of $D_{f}$ can also be seen by plugging the constant function $u(x)=1$ into (14).

We now show that $D_{f}$ completely determines $f$. Suppose $D_{f}=D_{g}$. We wish to show that $f=g$. Consider first the discrete case where $p$ and $q$ are distributions over a two-point set $\{0,1\}$. Given $u>1$, choose $p_{u}$ and $q_{u}$ such that $p_{u}(0)/q_{u}(0)=u$ and $p_{u}(1)/q_{u}(1)=\tfrac{1}{2}$. It is straightforward to show that such $p_{u}$ and $q_{u}$ exist and are given by

Thus

Subtracting the equivalent equation for $g$ and using $D_{f}=D_{g}$, we see that $f$ and $g$ differ only in an affine-linear term: $g(u)=f(u)+a(u-1)$ for all $u>1$ for some $a\in\mathbb{R}$. Therefore $g^{\prime}(u)=f^{\prime}(u)+a$ for $u>1$, and taking the limit as $u\to 1$ we have $g^{\prime}(1)=f^{\prime}(1)+a$. But $f^{\prime}(1)=g^{\prime}(1)=0$, so $a=0$, so $f(u)=g(u)$ for $u>1$. A similar argument applies for $0<u<1$. In this case we choose $p_{u}(0)/q_{u}(0)=u$ and $p_{u}(1)/q_{u}(1)=2$, and we have

and

From here we apply the same argument as above. Thus $f(u)=g(u)$ for $0<u<1$. Combining this with the above result for $u>1$ and the fact that $f(1)=g(1)$, we have $f=g$ as desired. For the continuous case where $p$ and $q$ are densities over $\mathbb{R}^{K}$, we reduce this to the discrete case by considering mixtures of Gaussians with shrinking covariances. Consider the $u>1$ case. Fix any two distinct points $\mu_{0},\mu_{1}\in\mathbb{R}^{K}$. Let $\tilde{p}_{u\sigma}(x)=p_{u}(0)\,\mathcal{N}(x;\mu_{0},\sigma^{2}I)+p_{u}(1)\,\mathcal{N}(x;\mu_{1},\sigma^{2}I)$ and $\tilde{q}_{u\sigma}(x)=q_{u}(0)\,\mathcal{N}(x;\mu_{0},\sigma^{2}I)+q_{u}(1)\,\mathcal{N}(x;\mu_{1},\sigma^{2}I)$ where $p_{u}$ and $q_{u}$ are as specified for the discrete $u>1$ case above. As $\sigma\to 0$, $D_{f}(\tilde{p}_{u\sigma},\tilde{q}_{u\sigma})\to D_{f}(p_{u},q_{u})$, so $(2u-1)D_{f}(\tilde{p}_{u\sigma},\tilde{q}_{u\sigma})\to f(u)+2(u-1)f(\tfrac{1}{2})$, and similarly for $g$. But $D_{f}=D_{g}$, so we have the same quantity tending to two limits, so the limits must be equal. Thus $g(u)=f(u)+a(u-1)$ for some $a\in\mathbb{R}$ and the remainder of the argument is as above. A similar argument applies for $0<u<1$. Thus $f=g$ also holds in the continuous case.

Different f-divergences may behave very differently when $p$ and $q$ are far apart but are essentially identical when $q\approx p$. One way to make this precise is to consider a parametric family $\{q_{\lambda}\mathrel{\mathop{\mathchar 58\relax}}\lambda\in\Lambda\}$ of densities. A Taylor expansion of $D_{f}\left(q_{\lambda},q_{\lambda+\varepsilon v}\right)$ in terms of $\epsilon$ shows that

where $\varepsilon\in\mathbb{R}$, $v\in\mathbb{R}^{K}$, and

is the Fisher information matrix of the parametric family, also known as the Fisher metric. Thus all f-divergences agree up to a constant factor on the divergence between two nearby distributions, and they are all just scaled versions of the Fisher distance in this regime. Alternatively this may be stated in the non-parametric form

where $v\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}$ satisfies $\int v(x)\operatorname{d\!}x=0$. Informally we may state this as

Thus all f-divergences agree up to a constant factor on the divergence between nearby distributions.

The fact that $f^{\prime\prime}$ has a simpler algebraic form than $f$ for many common f-divergences will be seen when we consider specific f-divergences below. Considering $f^{\prime\prime}$ is natural for a number of reasons. Adding an affine-linear term to $f$ does not change $f^{\prime\prime}$. Thus even without the constraints $f(1)=f^{\prime}(1)=0$, $f^{\prime\prime}$ would not have the two unnecessary degrees of freedom mentioned above and $D_{f}$ would uniquely determine $f^{\prime\prime}$. Strict convexity of $f$ corresponds to the simple condition $f^{\prime\prime}(u)>0$ for all $u$. We will also see below that various gradients of $D_{f}$ and of its lower bound $E_{f}$ depend on $f$ only through $f^{\prime\prime}$. Considering $f^{\prime\prime}$ also provides a simple view of how various f-divergences are related. For example, three common symmetric divergences are the Jensen-Shannon, squared Hellinger and Jeffreys divergences. In the $f^{\prime\prime}$ domain, these may all be viewed as forms of average of KL and reverse KL, specifically as the harmonic mean, geometric mean and arithmetic mean respectively.

We first derive a variational lower bound on the f-divergence $D_{f}$. Since $f$ is strictly convex, its graph lies at or above any of its tangent lines and only touches in one place. That is, for $k,u>0$,

with equality iff $k=u$. This inequality is illustrated in Figure 1. Substituting $p(x)/q(x)$ for $k$ and $u(x)$ for $u$, for any continuously differentiable function $u\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}_{>0}$ we obtain

with equality iff $u=u^{*}$, where $u^{*}(x)=p(x)/q(x)$. The function $u$ is referred to as the critic. It will be helpful to have a concise notation for this bound. Writing $u(x)=\exp(d(x))$ without loss of generality, for any continuously differentiable function $d\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}$, we have

with equality iff $d=d^{*}$, where

Note that both $a_{f}$ and $b_{f}$ are linear in $f$. Their derivatives $a_{f}^{\prime}(\log u)=uf^{\prime\prime}(u)$ and $b_{f}^{\prime}(\log u)=u^{2}f^{\prime\prime}(u)$ depend on $f$ only through $f^{\prime\prime}$. Note that $b_{f}^{\prime}(d)=a_{f}^{\prime}(d)\exp(d)$.

The bound (15) leads naturally to variational divergence estimation. The $f$-divergence between $p$ and $q$ can be estimated by maximizing $E_{f}$ with respect to $d$ (Nguyen et al., 2010). Conveniently $E_{f}$ is expressed in terms of expectations and may be approximately computed and maximized with respect to $d$ using only samples from $p$ and $q$. Ultimately this property derives from the fact that the tangent lines of $f(u)$ are affine-linear in $u$, and the constant term leads to expectations with respect to $q$ and the linear term leads to expectations with respect to $p$. If we parameterize $d$ as a neural net $d_{\nu}$ with parameters $\nu$ then we can approximate the divergence by maximizing $E_{f}(p,q,d_{\nu})$ with respect to $\nu$. This does not compute the exact divergence for several reasons: there is no guarantee that $d^{*}$ lies in the family $\{d_{\nu}\mathrel{\mathop{\mathchar 58\relax}}\nu\}$ of functions representable by the neural net; gradient-based optimization may find a local but not global minimum; and we have to be careful not to overfit given a finite set of samples from $p$ (typically $q$ is a model and so we have access to arbitrarily many samples). However we hope for sufficiently flexible neural nets and careful optimization that the approximation will be close.

In this section we express several common divergences in terms of (1) and the variational lower bound (16). For each f-divergence, we give explicit expressions for $f$, $f^{\prime\prime}$, $D_{f}$, $E_{f}$, $a_{f}$, $b_{f}$, $a_{f}^{\prime}$ and $b_{f}^{\prime}$. We also list the tail weights, which are related to the asymptotic behavior of $f^{\prime\prime}(u)$ as $u\to 0$ and $u\to\infty$ and which determine the key qualitative properties of an f-divergence (Shannon et al., 2020). We mention below how some of the divergences are related to each other by softening (Shannon et al., 2020). The p-softening of a divergence $D$ is the divergence $\tilde{D}$ given by $\tilde{D}(p,q)=4D(\tfrac{1}{2}p+\tfrac{1}{2}q,q)$. Similarly the q-softening is given by $\tilde{D}(p,q)=4D(p,\tfrac{1}{2}p+\tfrac{1}{2}q)$. The factor of $4$ here ensures that $\tilde{D}$ remains canonical (Shannon et al., 2020).

The Kullback-Leibler (KL) divergence (or I divergence or relative entropy) satisfies:

The KL divergence has $(1,2)$ tail weights. The KL divergence defining function is sometimes given as $f(u)=u\log u$. The additional affine-linear term in our expression for $f$ is due to the constraints $f(1)=f^{\prime}(1)=0$. We note in passing that this precisely corresponds to what is sometimes referred to as the generalized KL divergence $D(p,q)=\int p(x)\log\frac{p(x)}{q(x)}\operatorname{d\!}x-\int p(x)\operatorname{d\!}x+\int q(x)\operatorname{d\!}x$. This has the property that $D(p,q)\geq 0$ with equality iff $p=q$ even if we remove the constraint that $p$ and $q$ be valid densities that integrate to one.

The reverse KL divergence satisfies:

The reverse KL divergence has $(2,1)$ tail weights. Note the explicit symmetry between the representations of KL and reverse KL in terms of $a_{f}$ and $b_{f}$. Their symmetric relationship is less apparent from $f$ and $f^{\prime\prime}$. In general if $D_{g}(p,q)=D_{f}(q,p)$ then $a_{g}(d)=b_{f}(-d)$ and $b_{g}(d)=a_{f}(-d)$.

The canonicalized Jensen-Shannon divergence (Lin, 1991) (or Jensen difference (Burbea and Rao, 1982) or capacitory discrimination (Topsoe, 2000)) satisfies:

Here $\operatorname{JS}$ is the conventional, non-canonical Jensen-Shannon divergence with $f^{\prime\prime}(1)=1/4$. The Jensen-Shannon divergence has $(1,1)$ tail weights. Its $f^{\prime\prime}(u)$ is the harmonic mean of $f^{\prime\prime}(u)$ for KL and $f^{\prime\prime}(u)$ for reverse KL. The square root of the Jensen-Shannon divergence defines a metric on the space of probability distributions (Endres and Schindelin, 2003; Österreicher and Vajda, 2003).

The canonicalized squared Hellinger distance (closely related to the Freeman-Tukey statistic for hypothesis testing) satisfies:

Here $\int\sqrt{p(x)q(x)}\operatorname{d\!}x$ is known as the Bhattacharyya coefficient. The squared Hellinger distance has $(\frac{3}{2},\frac{3}{2})$ tail weights. Its $f^{\prime\prime}(u)$ is the geometric mean of $f^{\prime\prime}(u)$ for KL and $f^{\prime\prime}(u)$ for reverse KL. The Hellinger distance defines a metric on the space of probability distributions (Vajda, 2009).

The Jeffreys divergence (or J divergence) $(p,q)\mapsto\tfrac{1}{2}\operatorname{KL}(p,q)+\tfrac{1}{2}\operatorname{KL}(q,p)$ is the arithmetic mean of the KL divergence and reverse KL divergence. Since $f$, $f^{\prime\prime}$, $D_{f}$, $E_{f}$, $a_{f}$, $b_{f}$, $a_{f}^{\prime}$ and $b_{f}^{\prime}$ are all linear in $f$, they are also all just the arithmetic mean of the corresponding quantities for KL and reverse KL and so we do not list them separately. The Jeffreys divergence has $(2,2)$ tail weights.

The canonicalized squared Le Cam distance (or squared Puri-Vincze distance or triangular discrimination) satisfies:

Here $\Delta$ is the conventional, non-canonical squared Le Cam distance with $f^{\prime\prime}(1)=\tfrac{1}{2}$, and $\chi^{2}$ is the Pearson $\chi^{2}$ divergence defined below. The squared Le Cam distance has $(0,0)$ tail weights. It is symmetric with respect to $p$ and $q$. The canonicalized squared Le Cam distance may be obtained by q-softening the canonicalized Pearson $\chi^{2}$ divergence or p-softening the canonicalized Neymann divergence. The Le Cam distance defines a metric on the space of probability distributions (Vajda, 2009).

The canonicalized Pearson $\chi^{2}$ divergence (or Kagan divergence) satisfies:

Here $\chi^{2}$ is the conventional, non-canonical Pearson $\chi^{2}$ divergence with $f^{\prime\prime}(1)=2$. The Pearson $\chi^{2}$ divergence has $(0,3)$ tail weights. The p-softened canonicalized Pearson $\chi^{2}$ divergence is itself. The q-softened canonicalized Pearson $\chi^{2}$ divergence is the canonicalized squared Le Cam distance.

The canonicalized Neymann divergence satisfies: