Information-Theoretic Lower Bounds for Distributed Function Computation

The problem of distributed function computation arises in such applications as inference and learning in networks and consensus or coordination of multiple agents. Each node of the network has an initial random observation and aims to compute a common function of the observations of all the nodes by exchanging messages with its neighbors over discrete memoryless point-to-point channels and by performing local computations. A problem of theoretical and practical interest is to determine the fundamental limits on the computation time, i.e., the minimum number of steps needed by any distributed computation algorithm to guarantee that, when the algorithm terminates, each node has an accurate estimate of the function value with high probability.

Formally, a network consisting of nodes connected by point-to-point channels is represented by a directed graph $G=({\mathcal{V}},{\mathcal{E}})$, where ${\mathcal{V}}$ is a finite set of nodes and ${\mathcal{E}}\subseteq{\mathcal{V}}\times{\mathcal{V}}$ is a set of edges. Node $u$ can send messages to node $v$ only if $(u,v)\in{\mathcal{E}}$. Accordingly, to each edge $e\in{\mathcal{E}}$ we associate a discrete memoryless channel with finite input alphabet ${\mathsf{X}}_{e}$, finite output alphabet ${\mathsf{Y}}_{e}$, and stochastic transition law $K_{e}$ that specifies the transition probabilities $K_{e}(y_{e}|x_{e})$ for all $(x_{e},y_{e})\in{\mathsf{X}}_{e}\times{\mathsf{Y}}_{e}$. The channels corresponding to different edges are assumed to be independent. Initially, each node $v$ has access to an observation given by a random variable (r.v.) $W_{v}$ taking values in some space ${\mathsf{W}}_{v}$. We assume that the joint probability law ${\mathbb{P}}_{W}$ of $W\triangleq(W_{v})_{v\in{\mathcal{V}}}$ is known to all the nodes. Given a function $f:\prod_{v\in{\mathcal{V}}}{\mathsf{W}}_{v}\to{\mathsf{Z}}$, each node aims to estimate the value $Z=f(W)$ via local communication and computation. For example, when $f$ is given by the identity mapping $Z=W$, the goal of each node is to estimate the observations of all other nodes in the network.

The operation of the network is synchronized, and takes place in discrete time. A $T$-step algorithm ${\mathcal{A}}$ is a collection of deterministic encoders $(\varphi_{v,t})$ and estimators $(\psi_{v})$, for all $v\in{\mathcal{V}}$ and $t\in\{1,\ldots,T\}$, given by mappings

where ${\mathsf{X}}_{v\rightarrow}=\prod_{u\in{\mathcal{N}}_{v\rightarrow}}{\mathsf{X}}_{(v,u)}$ and ${\mathsf{Y}}_{v\leftarrow}=\prod_{u\in{\mathcal{N}}_{v\leftarrow}}{\mathsf{Y}}_{(u,v)}$. Here, ${\mathcal{N}}_{v\leftarrow}\triangleq\{u\in{\mathcal{V}}:(u,v)\in{\mathcal{E}}\}$ and ${\mathcal{N}}_{v\rightarrow}\triangleq\{u\in{\mathcal{V}}:(v,u)\in{\mathcal{E}}\}$ are, respectively, the in-neighborhood and the out-neighborhood of node $v$. The algorithm operates as follows: at each step $t$, each node $v$ computes $X_{v,t}\triangleq(X_{(v,u),t})_{u\in{\mathcal{N}}_{v\rightarrow}}=\varphi_{v,t}\big{(}W_{v},Y^{t-1}_{v}\big{)}\in{\mathsf{X}}_{v\rightarrow}$, and then transmits each message $X_{(v,u),t}$ along the edge $(v,u)\in{\mathcal{E}}$. For each $(u,v)\in{\mathcal{E}}$, the received message $Y_{(u,v),t}$ at each $t$ is related to the transmitted message $X_{(u,v),t}$ via the stochastic transition law $K_{(u,v)}$. At step $T$, each node $v$ computes ${\widehat{Z}}_{v}=\psi_{v}(W_{v},Y^{T}_{v})$ as an estimate of $Z$, where $Y_{v,t}\triangleq(Y_{(u,v),t})_{u\in{\mathcal{N}}_{v\leftarrow}}\in{\mathsf{Y}}_{v\leftarrow}$ for $t\in\{1,\ldots,T\}$.

Given a nonnegative distortion function $d:{\mathsf{Z}}\times{\mathsf{Z}}\to\mathbb{R}^{+}$, we use the excess distortion probability ${\mathbb{P}}\big{[}d(Z,{\widehat{Z}}_{v})>\varepsilon\big{]}$ to quantify the computation fidelity of the algorithm at node $v$. A key fundamental limit of distributed function computation is the $(\varepsilon,\delta)$-computation time:

If an algorithm ${\mathcal{A}}$ has the property that

then we say that it achieves accuracy $\varepsilon$ with confidence $1-\delta$. Thus, $T(\varepsilon,\delta)$ is the minimum number of time steps needed by any algorithm to achieve accuracy $\varepsilon$ with confidence $1-\delta$. The objective of this paper is to derive general lower bounds on $T(\varepsilon,\delta)$ for arbitrary network topologies, discrete memoryless channel models, continuous or discrete observations, and functions $f$.

Previously, this problem (for real-valued functions and quadratic distortion) has been studied by Ayaso et al. [1] and by Como and Dahleh [2] using information-theoretic techniques. This problem is also related to the study of communication complexity of distributed computing over noisy channels. In that context, Goyal et al. [3] studied the problem of computing Boolean functions in complete graphs, where each pair of nodes communicates over a pair of independent binary symmetric channels (BSCs), and obtained tight lower bounds on the number of serial broadcasts using an approach tailored to that special problem. The technique used in [3] has been extended to random planar networks by Dutta et al. [4]. Other related, but differently formulated, problems include communication complexity and information complexity in distributed computing over noiseless channels, surveyed in [5]; minimum communication rates for distributed computing [6, 7, 8], compression, or estimation based on infinite sequences of observations, surveyed in [9, Chap. 21]; and distributed computing in wireless networks, surveyed in [10]. Some achievability results for specific distributed function computation problems can be found in [11, 12, 13, 1, 14, 15, 16, 17, 18].

Our analysis builds upon the information-theoretic framework proposed by Ayaso et al. [1] and Como and Dahleh [2]. The underlying idea is rather natural and exploits a fundamental trade-off between the minimal amount of information any good algorithm must necessarily extract about the function value $Z$ when it terminates and the maximal amount of information any algorithm is able to obtain due to time and communication constraints. To be more precise, given any set of nodes ${\mathcal{S}}\subseteq{\mathcal{V}}$, let $W_{\mathcal{S}}\triangleq(W_{v})_{v\in{\mathcal{S}}}$ denote the vector of observations at all the nodes in ${\mathcal{S}}$. The quantity that plays a key role in the analysis is the conditional mutual information $I(Z;{\widehat{Z}}_{v}|W_{\mathcal{S}})$ between the actual function value $Z$ and the estimate ${\widehat{Z}}_{v}$ at an arbitrary node $v$, given the observations in an arbitrary subset of nodes ${\mathcal{S}}$ containing $v$.

Consider an arbitrary $T$-step algorithm ${\mathcal{A}}$ that achieves accuracy $\varepsilon$ with confidence $1-\delta$. Then, as we show in Lemma 1 of Sec. II-A, this mutual information can be lower-bounded by

where $h_{2}(\delta)\triangleq-\delta\log\delta-(1-\delta)\log(1-\delta)$ is the binary entropy function, and

is the conditional small ball probability of $Z=f(W)$ given $W_{\mathcal{S}}=w_{\mathcal{S}}$. The conditional small ball probability quantifies the difficulty of localizing the value of $Z=f(W)$ in a “distortion ball” of size $\varepsilon$ given partial knowledge about the value of $W$, namely $W_{\mathcal{S}}=w_{\mathcal{S}}$. For example, as discussed in Sec. IV, when $f$ is a linear function of the observations $W$, the conditional small ball probability can be expressed in terms of so-called Lévy concentration functions [19], for which tight estimates are available under various regularity conditions.

On the other hand, if ${\mathcal{A}}$ is a $T$-step algorithm, then the amount of information any node $v$ has about $Z$ once ${\mathcal{A}}$ terminates can be upper-bounded by a quantity that increases with $T$ and also depends on the network topology and on the information transmission capabilities of the channels connecting the nodes. To quantify this amount of information, we consider a cut of the network, i.e., a partition of the set of nodes ${\mathcal{V}}$ into two disjoint subsets ${\mathcal{S}}$ and ${\mathcal{S}}^{c}\triangleq{\mathcal{V}}\backslash{\mathcal{S}}$, such that $v\in{\mathcal{S}}$. The underlying intuition is that any information that nodes in ${\mathcal{S}}$ receive about $W_{{\mathcal{S}}^{c}}$ must flow across the edges from nodes in ${\mathcal{S}}^{c}$ to nodes in ${\mathcal{S}}$. The set of these edges, denoted by ${\mathcal{E}}_{\mathcal{S}}$, is referred to as the cutset induced by ${\mathcal{S}}$. Figure 1 illustrates these concepts on a simple four-node network. We then have the following upper bound [1, 2] (see also Lemma 2 in Sec. II-B):

The quantity $C_{\mathcal{S}}$, referred to as the cutset capacity, is the sum of the Shannon capacities of all the channels located on the edges in the cutset ${\mathcal{E}}_{\mathcal{S}}$. Thus, if there exists a cut $({\mathcal{S}},{\mathcal{S}}^{c})$ with a small value of $C_{\mathcal{S}}$, then the amount of information gained by the nodes in ${\mathcal{S}}$ about $Z$ will also be small. Note that the cutset upper bound grows linearly with $T$. However, when the initial observations $W$ are discrete, we also know that

where $H(W_{{\mathcal{S}}^{c}}|W_{\mathcal{S}})$ is the conditional entropy of $W_{{\mathcal{S}}^{c}}$ given $W_{{\mathcal{S}}}$, which does not depend on $T$. In fact, we sharpen this bound by showing in Lemma 5 in Sec. II-D that

Here, $\eta_{v}$ is defined as

where the supremum is over all triples $(U,X_{v},Y_{v})$ of r.v.’s, such that $U$ takes values in an arbitrary alphabet, $U\to X_{v}\to Y_{v}$ is a Markov chain, $X_{v}$ takes values in ${\mathsf{X}}_{v\leftarrow}$, $Y_{v}$ takes values in ${\mathsf{Y}}_{v\leftarrow}$, and the conditional probability law ${\mathbb{P}}_{Y_{v}|X_{v}}$ is equal to the product of all the channels entering $v$. As we discuss in detail in Sec. II-C, this constant is related to so-called strong data processing inequalities (SDPIs) [20], and quantifies the information transmission capabilities of the channels entering $v$. When $\eta_{v}<1$, the upper bound (4) is strictly smaller than $H(W_{{\mathcal{S}}^{c}}|W_{\mathcal{S}})$. With the upper bound (4), we can strengthen the cutset bound to the following:

Combining the bounds in (2) and (5), we conclude that, if there exists a $T$-step algorithm ${\mathcal{A}}$ that achieves accuracy $\varepsilon$ with confidence $1-\delta$, then

moreover, this inequality holds for all choices of ${\mathcal{S}}\subset{\mathcal{V}}$ and $v\in{\mathcal{S}}$. The precise statements of the resulting lower bounds on $T(\varepsilon,\delta)$ are given in Theorem 1 and Theorem 2.

The lower bound in (I-B) accounts for the difficulty of estimating the value of $Z=f(W)$ given only a subset of observations $W_{\mathcal{S}}$ through the small ball probability $L(W_{\mathcal{S}},\varepsilon)$, and for the communication bottlenecks in the network through the cutset capacity $C_{\mathcal{S}}$ and the constants $\eta_{v}$. The presence of $L(W_{\mathcal{S}},\varepsilon)$ in the bound ensures the correct scaling of $T(\varepsilon,\delta)$ in the high-accuracy limit $\varepsilon\to 0$. In particular, when the function $f$ is real-valued and the probability distribution of $Z=f(W)$ has a density, it is not hard to see that $L(W_{\mathcal{S}},\varepsilon)=O(\varepsilon)$, and therefore $T(\varepsilon,\delta)$ grows without bound at the rate of $\log(1/\varepsilon)$ as $\varepsilon\to 0$. By contrast, the bounds of Ayaso et al. [1] saturate at a finite constant even when no computation error is allowed, i.e., when $\varepsilon=0$. Detailed comparison with existing bounds is given in Sec. IV, where we particularize our lower bounds to the computation of linear functions. Moreover, in certain cases our lower bound on $T(\varepsilon,\delta)$ tends to infinity in the high-confidence regime $\delta\to 0$. By contrast, existing lower bounds that rely on cutset capacity estimates remain bounded regardless of how small we make $\delta$.

Throughout the paper, we provide several concrete examples that illustrate the tightness of the general lower bound in (I-B). In particular, Example 1 in Sec. II-E concerns the problem of computing the mod-$2$ sum of two independent ${\rm Bern}(\tfrac{1}{2})$ random variables in a network of two nodes communicating over binary symmetric channels (BSCs). For that problem, we obtain a lower bound on $T(0,\delta)$ that matches an achievable upper bound within a factor of $2$. In Example 2 in Sec. II-E, we consider the case where the nodes aim to distribute their discrete observations to all other nodes, and obtain a lower bound on $T(0,\delta)$ that captures the conductance of the network, which plays a prominent role in the previously published bounds of Ayaso et al. [1]. In Sec. V, we study two more examples: computing a sum of independent Rademacher random variables in a dumbbell network of BSCs, and distributed averaging of real-valued observations in an arbitrary network of binary erasure channels (BECs). Our lower bound for the former example can precisely capture the dependence of the computation time on the number of nodes in the network, while for the latter example it captures the correct dependence of the computation time on the accuracy parameter $\varepsilon$.

A significant limitation of the analysis based on a single cut $({\mathcal{S}},{\mathcal{S}}^{c})$ of the network is that it only captures the flow of information across the cutset ${\mathcal{E}}_{\mathcal{S}}$, but does not account for the time it takes the algorithm to disseminate this information to all the nodes in ${\mathcal{S}}$. We address this limitation in Sec. III through a multi-cutset analysis. The main idea is to partition the set of nodes ${\mathcal{V}}$ into several subsets ${\mathcal{S}}_{1},\ldots,{\mathcal{S}}_{n}$, such that, for all ${\mathcal{P}}_{i}\triangleq{\mathcal{S}}_{1}\cup\ldots\cup{\mathcal{S}}_{i}$, the cutsets ${\mathcal{E}}_{{\mathcal{P}}_{1}},\ldots,{\mathcal{E}}_{{\mathcal{P}}_{n-1}}$, ${\mathcal{E}}_{{\mathcal{P}}^{c}_{1}},\ldots,{\mathcal{E}}_{{\mathcal{P}}^{c}_{n-1}}$ are disjoint, and to analyze the flow of information across this sequence of cutsets. Once such a partition is selected, the analysis is based on a network reduction argument (Lemma 7), which lumps all the nodes in each ${\mathcal{S}}_{i}$ into a single virtual “supernode.” The construction of the partition ensures that each supernode $i$ only communicates with supernodes $i-1$ and $i+1$, and can also send noisy messages to itself (this is needed to simulate noisy communication among the nodes within ${\mathcal{S}}_{i}$ in the original network). Thus, the reduced network takes the form of a chain with $n$ nodes communicating with their nearest neighbors over bidirectional noisy links and, in addition, sending noisy messages to themselves. We refer to this network as a bidirected chain of length $n-1$. Figure 2a shows the partition of a six-node network, and the bidirected chain reduced from this network is shown in Fig. 2b.

Once this reduction is carried out, we can convert any $T$-step algorithm ${\mathcal{A}}$ running on the original network into a randomized $T$-step algorithm ${\mathcal{A}}^{\prime}$ running on the reduced network with the same accuracy and confidence guarantees as ${\mathcal{A}}$. Consequently, it suffices to analyze distributed function computation in bidirected chains. The key quantitative statement that emerges from this analysis can be informally stated as follows: For any bidirected chain with $n>3$ nodes, there exists a constant $\eta\in[0,1]$ that plays the same role as $\eta_{v}$ in (4) and quantifies the information transmission capabilities of the channels in the chain, such that, for any algorithm ${\mathcal{A}}$ that runs on this chain and takes time $T=O(n/\eta)$, the conditional mutual information between the function value $Z$ and its estimate ${\widehat{Z}}_{n}$ at the rightmost node $n$ given the observations of nodes $2$ through $n$ is upper-bounded by

where $C_{(1,2)}$ is the Shannon capacity of the channel from node $1$ to node $2$. The precise statement is given in Lemma 8 in Sec. III-A. Intuitively, this shows that, unless the algorithm uses $\Omega(n/\eta)$ steps, the information about $W_{1}$ will dissipate at an exponential rate by the time it propagates through the chain from node $1$ to node $n$. Combining (7) with the lower bound on $I(Z;{\widehat{Z}}_{n}|W_{2:n})$ based on small ball probabilities, we can obtain lower bounds on the computation time $T(\varepsilon,\delta)$. The precise statement is given in Theorem 3. Moreover, as we show, it is always possible to reduce an arbitrary network with bidirectional point-to-point channels between the nodes to a bidirected chain whose length is equal to the diameter of the original network, which implies that, for networks with sufficiently large diameter, and for sufficiently small values of $\varepsilon,\delta$,

where ${\rm diam}(G)$ denotes the diameter. This dependence on ${\rm diam}(G)$, which cannot be captured by the single-cutset analysis, is missing in almost all of the existing lower bounds on computation time. An exception is the paper by Rajagopalan and Schulman [13] that gives an asymptotic lower bound on the time required to broadcast a single bit over a chain of unidirectional BSCs. Our multi-cutset analysis applies to both discrete and continuous observations, and to general network topologies. It can be straightforwardly particularized to specific networks, such as bidirected chains, rings, trees, and grids, as discussed in Sec. III-B. We note that techniques involving multiple (though not necessarily disjoint) cutsets have also been proposed in the study of multi-party communication complexity by Tiwari [21] and more recently by Chattopadhyay et al. [22], while our concern is the influence of network topology and channel noise on the computation time.

The remainder of the paper is structured as follows. We start with the single-cutset analysis in Sec. II. The lower bound on the conditional mutual information via the conditional small ball probability is presented in Sec. II-A. The cutset upper bound and the SDPI upper bound on the conditional mutual information are presented in Sec. II-B and Sec. II-D. An introduction on SDPIs given in Sec. II-C. The lower bound on computation time is given in Sec. II-E, along with two concrete examples. Sec. III is devoted to the multi-cutset analysis, where we first present the network reduction argument in Sec. III-A, then derive general lower bounds on computation time and particularize the results to special networks in Sec. III-B. In Sec. IV, we discuss lower bounds for computing linear functions, where we relate the conditional small ball probability to Lévy concentration functions, and evaluate them in a number of special cases. We also make detailed comparisons of our results with existing lower bounds in Sec. IV-D. In Sec. V, we compare the lower bounds on computation time with the achievable upper bounds for two more examples: computing a sum of independent Rademacher random variables in a dumbbell network of BSCs, and distributed averaging of real-valued observations in an arbitrary network of binary erasure channels (BECs). We conclude this paper and point out future research directions in Sec. VI. A couple of lengthy technical proofs are relegated to a series of appendices.

For any $\varepsilon\geq 0$, ${\mathcal{S}}\subset{\mathcal{V}}$, and $w_{\mathcal{S}}\in\prod_{v\in{\mathcal{S}}}{\mathsf{W}}_{v}$, define the conditional small ball probability of $Z$ given $W_{\mathcal{S}}=w_{\mathcal{S}}$ as

This quantity measures how well the conditional distribution of $Z$ given $W_{\mathcal{S}}=w_{\mathcal{S}}$ concentrates in a small region of size $\varepsilon$ as measured by $d(\cdot,\cdot)$. The following lower bound on $I(Z;{\widehat{Z}}_{v}|W_{\mathcal{S}})$ in terms of the conditional small ball probability is essential for proving lower bounds on $T(\varepsilon,\delta)$.

Our first upper bound involves the cutset capacity $C_{\mathcal{S}}$, defined as

Here, $C_{e}$ denotes the Shannon capacity of the channel $K_{e}$.