Continuity and Additivity Properties
of Information Decompositions

Together with the information decomposition framework, Williams and Beer (2010) also defined an information decomposition $I_{\min}$. Let

be the specific information of the outcome $S=s$ about $Y$ and $Z$, respectively. Then

$I_{\min}$ has been criticized, because it assigns relatively large values of shared information, conflating “the same amount of information” with “the same information” (Harder et al., 2013; Griffith and Koch, 2014).

A related information decomposition is the minimum mutual information (MMI) decomposition given by

Even more severely than $I_{\min}$, this information decomposition conflates “the same amount of information” with “the same information.” Still, formally, this definition produces a valid bivariate information decomposition and thus serves as a useful benchmark. The axioms imply that $SI(S;Y,Z)\leq SI_{\operatorname{MMI}}(S;Y,Z)$ for any other bivariate information decomposition. For multivariate Gaussian variables, many information decompositions actually agree with $I_{\operatorname{MMI}}$ (Barrett, 2015).

To address the criticism of $I_{\min}$, Harder et al. (2013) introduced a bivariate information decomposition $I_{\operatorname{red}}$ based on a notion of redundant information as follows. Let $\mathcal{Z}^{\prime}:=\{z\in\mathcal{Z}:P(Z=z)>0\}$ be the support of $Z$, and let

Then

Banerjee et al. (2018a) showed that the quantity $I(S;Y)-I_{S}(Y\searrow Z)$ has a decision-theoretic interpretation by way of channel deficiencies (Raginsky, 2011).

Motivated from decision-theoretic considerations, Bertschinger et al. (2014) introduced the bivariate information decomposition $I_{\operatorname{BROJA}}$ (eponymously named after the authors in (Bertschinger et al., 2014)). Given $P\in\mathbb{P}_{\mathcal{S}\times\mathcal{Y}\times\mathcal{Z}}$, let $\Delta_{P}$ denote the set of joint distributions of $(S,Y,Z)$ that have the same marginals on $(S,Y)$ and $(S,Z)$ as $P$. Then define the unique information that $Y$ conveys about $S$ with respect to $Z$ as

where the subscript $Q$ in $I_{Q}$ denotes the joint distribution on which the function is computed. Computation of this decomposition was investigated by Banerjee et al. (2018b). $I_{\operatorname{BROJA}}$ leads to a concept of synergy that agrees with the synergy measure defined by Griffith and Koch (2014).

James et al. (2018) define the following bivariate decomposition: Given the joint distribution $P\in\mathbb{P}_{\mathcal{S}\times\mathcal{Y}\times\mathcal{Z}}$ of $(S,Y,Z)$, let $P_{Y-S-Z}=P(S,Y)P(S,Z)/P(S)$ be the probability distribution in $\mathbb{P}_{\mathcal{S}\times\mathcal{Y}\times\mathcal{Z}}$ that maximizes the entropy among all distributions $Q$ with $Q(S,Y)=P(S,Y)$ and $Q(S,Z)=P(S,Z)$. Similarly, let $P_{\Delta}$ be the probability distribution in $\mathbb{P}_{\mathcal{S}\times\mathcal{Y}\times\mathcal{Z}}$ that maximizes the entropy among all distributions $Q$ with $Q(S,Y)=P(S,Y)$ and $Q(S,Z)=P(S,Z)$ and $Q(Y,Z)=P(Y,Z)$ (unlike for $P_{Y-S-Z}$, we do not have an explicit formula for $P_{\Delta}$). Then

This definition is motivated in terms of a lattice of all sensible marginal constraints when maximizing the entropy, as in the definition of $P_{Y-S-Z}$ and $P_{\Delta}$ (see (James et al., 2018) for the details).

The information decompositions $I_{\cap}^{\wedge}$ (Griffith et al., 2014), $I_{\cap}^{\operatorname{GH}}$ (Griffith and Ho, 2015) and $I_{\cap}^{*}$ (Kolchinsky, 2022) are motivated from the notion of common information due to Gács and Körner (1973) and present three different approaches that try to represent the shared information in terms of a random variable $Q$:

where the optimization runs over all pairs of (deterministic) functions $f,f^{\prime}$ (for $SI_{\cap}^{\wedge}$), all joint distributions of four random variables $S,X,Y,Q$ that extend the joint distribution of $S,X,Y$ (for $SI_{\cap}^{\operatorname{GH}}$), and all pairs of stochastic matrices $\lambda_{q|y},\lambda^{\prime}_{q|z}$ (for $SI_{\cap}^{*}$), respectively. One can show that $SI_{\cap}^{\wedge}(S;Y,Z)\leq SI_{\cap}^{\operatorname{GH}}(S;Y,Z)\leq SI_{\cap}^{*}(S;Y,Z)$ (Kolchinsky, 2022).

The $I_{\cap}^{*}$-decomposition draws motivation from considerations of channel preorders, in a similar spirit as Banerjee et al. (2018a), and it is related to ideas from Bertschinger and Rauh (2014). Kolchinsky (2022) shows that there is an analogy between $I_{\cap}^{*}$ and $I_{\operatorname{BROJA}}$.

Niu and Quinn (2019) presented a bivariate information decomposition $I_{\operatorname{IG}}$ based on information geometric ideas. While their construction is very elegant, it only works for joint distributions $P$ of full support (i.e. $P(s,y,z)>0$ for all $s,y,z$). It is unknown whether it can be extended meaningfully to all joint distributions. Numerical evidence exists that a unique continuous extension is possible at least to some joint distributions with restricted support (see examples in (Niu and Quinn, 2019)).

For any $t\in\mathbb{R}$, consider the joint distribution

where $c_{t}$ is a normalizing constant, and let

Then

An interesting aspect about these definitions is that, by the Generalized Pythagorean Theorem (see (Amari, 2018)), $D(P\|P_{S-Z-Y})=D(P\|P^{*})+D(P^{*}\|P_{S-Z-Y})$.

Given an information measure that captures some aspect of unique information but that fails to satisfy the consistency condition (2), one may construct a corresponding bivariate information decomposition as follows:

We call the construction of Lemma 1 the UI construction. The unique information $UI_{\delta}$ returned by the UI construction is the smallest $UI$-function of any bivariate information decomposition with $UI\geq\delta$.

As Banerjee et al. (2018a) show, the decomposition $I_{\operatorname{red}}$ is an example of this construction. As another example, as Banerjee et al. (2018a) and Rauh et al. (2019) suggested, the UI construction can be used to obtain bivariate information decompositions from the one- or two-way secret key rates and related information functions that have been defined as bounds on the secret key rates, such as the intrinsic information (Maurer and Wolf, 1997), the reduced intrinsic information (Renner and Wolf, 2003), or the minimum intrinsic information (Gohari and Anantharam, 2010).

Several other measures have been proposed that are motivated by the framework of Williams and Beer (2010) but that leave the framework. Ince (2017) defines a decomposition $I_{\operatorname{ccs}}$, which satisfies (1), but in which $SI_{\operatorname{ccs}}$, $UI_{\operatorname{ccs}}$ and $CI_{\operatorname{ccs}}$ may take negative values. The SPAM decomposition of Finn and Lizier (2018) consists of non-negative information measures that decompose the mutual information, but this decomposition has a different structure, with alternating signs and twice as many terms. Both approaches construct “pointwise” decompositions, in the sense that $SI$, $UI$ and $CI$ can be naturally expressed as expectations, in a similar way that entropy and mutual information can be written as expectations (see (Finn and Lizier, 2018) for details). Recent works have proposed other decompositions based on a different lattice (Ay et al., 2020) or singling out features of the target variable (Magri, 2021).

Since these measures do not lie in our direct focus, we omit their definitions. Nevertheless, one can ask the same questions: Are the corresponding information measures continuous, and are they additive? For the constructions in (Finn and Lizier, 2018), both continuity and additivity (as a consequence of a chain rule) are actually postulated. The decomposition in (Ay et al., 2020) is additive. On the other hand, $I_{\operatorname{ccs}}$ is neither continuous (as can be seen from its definition) nor additive (since it does not satisfy the identity property).

We discuss two further related properties, namely, asymptotic continuity and locking (Banerjee et al., 2018a; Rauh et al., 2019), which we explain shortly below. $I_{\operatorname{BROJA}}$ is asymptotically continuous and does not exhibit locking. It is not known if other information decompositions satisfy these properties.

Operational quantities in information theory such as channel capacities and compression rates are usually defined in the spirit of Shannon (1948) - in the asymptotic regime of many independent uses of the channel or many independent realizations of the underlying source distribution. In the asymptotic regime, real-valued functionals of distributions that are asymptotically continuous are especially useful as they often provide lower or upper bounds for operational quantities of interest (Cerf et al., 2002; Banerjee et al., 2018a; Chitambar and Gour, 2019).

Asymptotic continuity is a stronger notion of continuity that considers convergence relative to the dimension of the underlying state space (Synak-Radtke and Horodecki, 2006; Chitambar and Gour, 2019; Fannes, 1973; Winter, 2016). Concretely, a function $f$ is said to be asymptotically continuous if

for all joint distributions $P,P^{\prime}\in{\mathbb{P}_{\mathcal{S}}}$, where $C$ is some constant, $\epsilon=\tfrac{1}{2}\|P-P^{\prime}\|_{1}$, and $\zeta:[0,1]\to\mathbb{R}_{+}$ is any continuous function converging to zero as $\epsilon\to 0$ (Chitambar and Gour, 2019).

As an example, entropy is asymptotically continuous (see, e.g., (Csiszár and Körner, 2011, Lemma 2.7)): For any $P,P^{\prime}\in\mathbb{P}_{\mathcal{S}}$, if $\tfrac{1}{2}\|P-P^{\prime}\|_{1}\leq\epsilon$, then

where $h(\cdot)$ is the binary entropy function, $h(p)=-p\log p-(1-p)\log(1-p)$ for $p\in(0,1)$ and $h(0)=h(1)=0$. Likewise, the conditional mutual information satisfies asymptotic continuity in the following sense (Renner and Wolf, 2003; Christandl and Winter, 2004): For any $P,P^{\prime}\in\mathbb{P}_{\mathcal{S}\times\mathcal{Y}\times\mathcal{Z}}$, if $\tfrac{1}{2}\|P-P^{\prime}\|_{1}\leq\epsilon$ then

Note that the right-hand side of the above inequality does not depend explicitly on the cardinality of $Z$.

As Banerjee et al. (2018a) show, $UI_{\operatorname{BROJA}}$ is asymptotically continuous:

Locking is motivated from the following property of the conditional mutual information (Renner and Wolf, 2003; Christandl et al., 2007): For arbitrary discrete random variables $S$, $Y$, $Z$, and $U$,

The conditional mutual information does not exhibit “locking” in the sense that any additional side information $U$ accessible to $Z$ cannot reduce the conditional mutual information by more than the entropy of the side information.

As Rauh et al. (2019) show, $UI_{\operatorname{BROJA}}$ does not exhibit locking:

This property is useful, for example, in a cryptographic context (Rauh et al., 2019) where it ensures that the unique information that $Y$ has about $S$ w.r.t. an adversary $Z$ cannot “unlock”, i.e., drop by an arbitrarily large amount on giving away a bit of information to $Z$.