Concavity of entropy under thinning

This paper considers information-theoretic properties of the thinning map, an operation on the space of discrete random variables, based on random summation.

Equivalently, if the probability mass function (pmf) of $X$ is $f$, then the pmf of $T_{\alpha}X$ is

where $bi(i;j,\alpha)=\binom{j}{i}\alpha^{i}(1-\alpha)^{j-i}$ is the binomial pmf. (Note that we write $T_{\alpha}$ for the map acting on the pmf as well as acting on the random variable.)

We briefly mention other notation used in this paper. We use ${\rm Po}(\lambda)$ to denote the Poisson distribution with mean $\lambda$, i.e., the pmf is $po(\lambda)=\{po(i;\lambda),\ i=0,1,\ldots\},\ po(i;\lambda)=\lambda^{i}e^{-\lambda}/i!$. The entropy of a discrete random variable $X$ with pmf $f$ is defined as

and the relative entropy between $X$ (with pmf $f$) and $Y$ (with pmf $g$) is defined as

For convenience we write $D(X)=D(X||po(\lambda))$ where $\lambda=EX$.

The thinning operation is intimately associated with the Poisson distribution and Poisson convergence theorems. It plays a significant role in the derivation of a maximum entropy property for the Poisson distribution (Johnson [7]). Recently there has been evidence that, in a number of problems related to information theory, the operation $T_{\alpha}$ is the discrete counterpart of the operation of scaling a random variable by $\sqrt{\alpha}$; see [5, 6, 7, 14]. Since scaling arguments can give simple proofs of results such as the Entropy Power Inequality, we believe that improved understanding of the thinning operation could lead to discrete analogues of such results.

For example, thinning lies at the heart of the following result (see [5, 6, 14]), which is a Poisson limit theorem with an information-theoretic interpretation.

For Part (4), we recall that a random variable $X$ on $\mathbf{Z}_{+}$ is called ultra-log-concave, or ULC, if its pmf $f$ is such that the sequence $i!f_{i},\ i=0,1,\ldots,$ is log-concave. Examples of ULC random variables include the binomial and the Poisson. In general, a sum of independent (but not necessarily identically distributed) Bernoulli random variables is ULC. Informally, a ULC random variable is less “spread out” than a Poisson with the same mean. Note that in Part (4) the ULC assumption is natural since, among ULC distributions with a fixed mean, the Poisson achieves maximum entropy ([7, 14]).

Parts (2) and (3) of Theorem 1 (see [5, 6]) resemble the entropic central limit theorem of Barron [2], in that convergence in relative entropy, rather than the usual weak convergence, is established. The monotonicity statements in Parts (3) and (4), proved in [14], can be seen as the discrete analogue of the monotonicity of entropy in the central limit theorem, conjectured by Shannon and proved much later by Artstein et al. [1].

In this work we further explore the behavior of entropy under thinning. Our main result is the following concavity property.

Theorem 2 is interesting on two accounts. Firstly, it can be seen as an analogue of the inequality

where $X$ and $Y$ are continuous random variables with finite variances and $h$ denotes the differential entropy. The difference between thinning by $\alpha$ in (1) and scaling by $\sqrt{\alpha}$ in (2) is required to control different moments. In the discrete case, the law of small numbers [5] and the corresponding maximum entropy property [7] both require control of the mean, which is achieved by this thinning factor. In the continuous case, the central limit theorem [2] requires control of the variance, which is achieved by this choice of scaling. It is well-known that (2) is a reformulation of Shannon’s entropy power inequality ([12, 3]). Thus Theorem 2 may be regarded as a first step towards a discrete entropy power inequality (see Section IV for further discussion).

Secondly, Theorem 2 is closely related to an open problem of Shepp and Olkin [11] concerning Bernoulli sums. With a slight abuse of notation let $H(a_{1},\ldots,a_{n})$ denote the entropy of the sum $\sum_{i=1}^{n}X_{i}$, where $X_{i}$ is an independent Bernoulli random variable with parameter $a_{i},\ i=1,\ldots,n$.

As noted by Shepp and Olkin [11], $H(a_{1},\ldots,a_{n})$ is concave in each $a_{i}$ and is concave in the special case where $a_{1}=\ldots=a_{n}$ and $b_{1}=\ldots=b_{n}$. We provide further evidence supporting Conjecture 1, by proving another special case, which is a consequence of Theorem 2 when applied to Bernoulli sums.

Conjecture 1 remains open. We are hopeful, however, that the techniques introduced here could help resolve this long-standing problem.

In Section II we collect some basic properties of thinning and ULC distributions, which are used in the proof of Theorem 2 in Section III. Possible extensions are discussed in Section IV.

Basic properties of thinning include the semigroup relation ([7])

and the commuting relation ($*$ denotes convolution)

It is (5) that allows us to deduce Corollary 1 from Theorem 2 easily.

Concerning the ULC property, three important observations ([7]) are

1. 1.

   a pmf $f$ is ULC if and only if the ratio $(i+1)f_{i+1}/f_{i}$ is a decreasing function of $i$;

2. 2.

   if $f$ is ULC, then so is $T_{\alpha}f$;

3. 3.

   if $f$ and $g$ are ULC, then so is their convolution $f*g$.

A key tool for deriving Theorem 2 and related results ([7, 13]) is Chebyshev’s rearrangement theorem, which states that the covariance of two increasing functions of the same random variable is non-negative. In other words, if $X$ is a scalar random variable, and $g$ and $\tilde{g}$ are increasing functions, then (assuming the expectations are finite)

The basic idea is to use the decomposition

where as before $D(X)=D(X||po(\lambda))$ with $\lambda=EX$, and $L(X)=E\log(po(X;\lambda))$.

The behavior of the relative entropy $D(X)$ under thinning is fairly well-understood. In particular, by differentiating $D(T_{\alpha}X)$ with respect to $\alpha$ and then using a data-processing argument, Yu [14] shows that

Further, for any independent $U$ and $V$, the data-processing inequality shows that $D(U+V)\leq D(U)+D(V).$ By taking $U=T_{\alpha}X$ and $V=T_{1-\alpha}Y$, one concludes that

Therefore we only need to prove the corresponding result for $L$, that is

Unfortunately, matters are more complicated because there is no equivalent of the data-processing inequality, i.e., the inequality $L(U+V)\leq L(U)+L(V)$ does not always hold. (Consider for example $U$ and $V$ i.i.d. Bernoulli with parameter $p\in(0,1)$. This inequality then reduces to $2p\leq p^{2}$, which clearly fails for all $p$.)

Nevertheless, it is possible to establish (7) directly. We illustrate the strategy with a related but simpler result, which involves the equivalent of Equation (6) for $L$.

Before proving Theorem 2, we note that although (1) is stated for $\alpha+\beta\leq 1$, only the case $\alpha+\beta=1$ need to be considered. Indeed, if (1) holds for $\alpha+\beta=1$, then for general $\alpha,\ \beta\geq 0$ such that $\alpha+\beta=\gamma\leq 1$, we have

where (4) and (5) are used in the equality, and Proposition 1 is used in (8).

In the continuous case, (2) is quickly shown (see [4]) to be equivalent to Shannon’s entropy power inequality

valid for independent $X$ and $Y$ with finite variances, with equality if and only if $X$ and $Y$ are normal. We aim to formulate a discrete analogue of (11), with the Poisson distribution playing the same role as the normal since it has the corresponding infinite divisibility and maximum entropy properties.

Observe that the function $\exp(2t)$ appearing in (11) is (proportional to) the inverse of the entropy of the normal with variance $t$. That is, if we write $e(t)=h(N(0,t))=\log(\sqrt{2\pi t})$ then the entropy power $v(X)=e^{-1}(h(X))=\exp(2h(X))/(2\pi)$, so Equation (11) can be written as

Although there does not exist a corresponding closed form expression for the entropy of a Poisson random variable, we can denote ${\cal E}(t)=H(po(t))$. Then ${\cal E}(t)$ is increasing and concave. (The proof of Proposition 1, when specialized to the Poisson case, implies this concavity.) Define

That is, $H(po(V(X)))=H(X)$. It is tempting to conjecture that the natural discrete analogue of Equation (11) is

for independent discrete random variables $X$ and $Y$, with equality if and only if $X$ and $Y$ are Poisson. However, this is not true. A counterexample, provided by an anonymous referee, is the case where $X$ and $Y$ both have the pmf $p(0)=1/6$, $p(1)=2/3$, $p(2)=1/6.$ Since this pmf even lies within the ULC class, the conjecture still fails when restricted to this class.

We believe that the discrete counterpart of the entropy power inequality should involve the thinning operation described above. If so, the natural conjecture is the following, which we refer to as the thinned entropy power inequality.

In a similar way to the continuous case, (12) easily yields the concavity of entropy, Equation (1), as a corollary. Indeed, by (12) and the concavity of ${\cal E}(t)$, we have

and (1) follows.

Unlike the continuous case, (1) does not easily yield (12). The key issue is the question of scaling. That is, in the continuous case, the entropy power $v(X)$ satisfies $v(\sqrt{\alpha}X)=\alpha v(X)$ for all $\alpha$ and $X$. It is this result that allows Dembo et al. [4] to deduce (11) from (2).

Such an identity does not hold for thinned random variables. However, we conjecture that

for all $\alpha$ and ULC $X$. Note that this Equation (13), which we refer to as the restricted thinned entropy power inequality (RTEPI), is simply the case $Y=0$ of the full thinned entropy power inequality (12). If (13) holds, we can use the argument provided by [4] to deduce the following result, which is in some sense close to the full thinned entropy power inequality, although $\beta+\gamma<1$ in general.

Again assuming (13), Proposition 2 yields the following special case of (12). The reason this argument works is that, as in [4], if $X$ is Poisson then (13) holds with equality for all $\alpha$.

We hope to report progress on (12) in future work. Given the fundamental importance of (11), it would also be interesting to see potential applications of (12) (if true) and (1). For example, Oohama [9] used the entropy power inequality (11) to solve the multi-terminal source coding problem. This showed the rate at which information could be transmitted from $L$ sources, producing correlated Gaussian signals but unable to collaborate or communicate with each other, under the addition of Gaussian noise. It would be of interest to know whether (12) could lead to a corresponding result for discrete channels.

Note: Since the submission of this paper to ISIT09, we have found a proof of the restricted thinned entropy power inequality, i.e., Equation (13). The proof, based on [7], is somewhat technical and will be presented in a future work.