Tight Bounds for Distributed Functional Monitoring

David P. Woodruff
IBM Almaden
dpwoodru@us.ibm.com Qin Zhang
IBM Almaden
qinzhang@cse.ust.hk Most of this work was done while Qin Zhang was a postdoc in MADALGO (Center for Massive Data Algorithmics - a Center of the Danish National Research Foundation), Aarhus University.

Abstract

We resolve several fundamental questions in the area of distributed functional monitoring, initiated by Cormode, Muthukrishnan, and Yi (SODA, 2008), and receiving recent attention. In this model there are $k$ sites each tracking their input streams and communicating with a central coordinator. The coordinator’s task is to continuously maintain an approximate output to a function computed over the union of the $k$ streams. The goal is to minimize the number of bits communicated.

Let the $p$ -th frequency moment be defined as $F_{p}=\sum_{i}f_{i}^{p}$ , where $f_{i}$ is the frequency of element $i$ . We show the randomized communication complexity of estimating the number of distinct elements (that is, $F_{0}$ ) up to a $1+\varepsilon$ factor is $\tilde{\Omega}(k/\varepsilon^{2})$ , improving upon the previous $\Omega(k+1/\varepsilon^{2})$ bound and matching known upper bounds up to a logarithmic factor. For $F_{p}$ , $p>1$ , we improve the previous $\Omega(k+1/\varepsilon^{2})$ bits communication bound to ${\Omega}(k^{p-1}/\varepsilon^{2})$ . We obtain similar improvements for heavy hitters, empirical entropy, and other problems. Our lower bounds are the first of any kind in distributed functional monitoring to depend on the product of $k$ and $1/\varepsilon^{2}$ . Moreover, the lower bounds are for the static version of the distributed functional monitoring model where the coordinator only needs to compute the function at the time when all $k$ input streams end; surprisingly they almost match what is achievable in the (dynamic version of) distributed functional monitoring model where the coordinator needs to keep track of the function continuously at any time step. We also show that we can estimate $F_{p}$ , for any $p>1$ , using $\tilde{O}(k^{p-1}{\mathrm{poly}}(\varepsilon^{-1}))$ bits of communication. This drastically improves upon the previous $\tilde{O}(k^{2p+1}N^{1-2/p}{\mathrm{poly}}(\varepsilon^{-1}))$ bits bound of Cormode, Muthukrishnan, and Yi for general $p$ , and their $\tilde{O}(k^{2}/\varepsilon+k^{1.5}/\varepsilon^{3})$ bits bound for $p=2$ . For $p=2$ , our bound resolves their main open question.

Our lower bounds are based on new direct sum theorems for approximate majority, and yield improvements to classical problems in the standard data stream model. First, we improve the known lower bound for estimating $F_{p},p>2,$ in $t$ passes from $\tilde{\Omega}(n^{1-2/p}/(\varepsilon^{2/p}t))$ to ${\Omega}(n^{1-2/p}/(\varepsilon^{4/p}t))$ , giving the first bound that matches what we expect when $p=2$ for any constant number of passes. Second, we give the first lower bound for estimating $F_{0}$ in $t$ passes with $\Omega(1/(\varepsilon^{2}t))$ bits of space that does not use the hardness of the gap-hamming problem.

1 Introduction

Recent applications in sensor networks and distributed systems have motivated the distributed functional monitoring model, initiated by Cormode, Muthukrishnan, and Yi [20]. In this model there are $k$ sites and a single central coordinator. Each site $S_{i}\ (i\in[k])$ receives a stream of data $A_{i}(t)$ for timesteps $t=1,2,\ldots$ , and the coordinator wants to keep track of a function $f$ that is defined over the multiset union of the $k$ data streams at each time $t$ . For example, the function $f$ could be the number of distinct elements in the union of the $k$ streams. We assume that there is a two-way communication channel between each site and the coordinator so that the sites can communicate with the coordinator. The goal is to minimize the total amount of communication between the sites and the coordinator so that the coordinator can approximately maintain $f(A_{1}(t),\ldots,A_{k}(t))$ at any time $t$ . Minimizing the total communication is motivated by power constraints in sensor networks, since communication typically uses a power-hungry radio [25]; and also by network bandwidth constraints in distributed systems. There is a large body of work on monitoring problems in this model, including maintaining a random sample [21, 50], estimating frequency moments [18, 20], finding the heavy hitters [5, 42, 45, 54], approximating the quantiles [19, 54, 35], and estimating the entropy [4].

We can think of the distributed functional monitoring model as follows. Each of the $k$ sites holds an $N$ -dimentional vector where $N$ is the size of the universe. An update to a coordinate $j$ on site $S_{i}$ causes $v^{i}_{j}$ to increase by $1$ . The goal is to estimate a statistic of $v=\sum_{i=1}^{k}v^{i}$ , such as the $p$ -th frequency moment $F_{p}=\|v\|_{p}^{p}$ , the number of distinct elements $F_{0}=|\textrm{support}(v)|$ , and the empirical entropy $H=\sum_{i}\frac{v_{i}}{\|v\|_{1}}\log\frac{\|v\|_{1}}{v_{i}}$ . This is the standard insertion-only model. For many of these problems, with the exception of the empirical entropy, there are strong lower bounds (e.g., $\Omega(N)$ ) if allowing updates to coordinates that cause $v^{i}_{j}$ to decrease [4]. The latter is called the update model. Thus, except for entropy, we follow previous work and consider the insertion-only model.

To prove lower bounds, we consider the static version of the distributed functional monitoring model, where the coordinator only needs to compute the function at the time when all $k$ input streams end. It is clear that a lower bound for the static case is also a lower bound for the dynamic case in which the coordinator has to keep track of the function at any point in time. The static version of the distributed functional monitoring model is closely related to the multiparty number-in-hand communication model, where we again have $k$ sites each holding an $N$ -dimensional vector $v^{i}$ , and they want to jointly compute a function defined on the $k$ input vectors. It is easy to see that these two models are essentially the same since in the former, if site $S_{i}$ would like to send a message to $S_{j}$ , it can always send the message first to the coordinator and then the coordinator can forward the message to $S_{j}$ . Doing this will only increase the total amount of communication by a factor of two. Therefore, we do not distinguish between these two models in this paper.

There are two variants of the multiparty number-in-hand communication model we will consider: the blackboard model, in which each message a site sends is received by all other sites, i.e., it is broadcast, and the message-passing model, in which each message is between the coordinator and a specific site.

Despite the large body of work in the distributed functional monitoring model, the complexity of basic problems is not well understood. For example, for estimating $F_{0}$ up to a $(1+\varepsilon)$ -factor, the best upper bound is $\tilde{O}(k/\varepsilon^{2})$ ¹¹1 We use $\tilde{O}(f)$ to denote a function of the form $f\cdot\log^{O(1)}(Nk/\varepsilon)$ . [20] (all communication and information bounds in this paper, if not otherwise stated, are in terms of bits), while the only known lower bound is $\Omega(k+1/\varepsilon^{2})$ . The dependence on $\varepsilon$ in the lower bound is not very insightful, as the $\Omega(1/\varepsilon^{2})$ bound follows just by considering two sites [4, 16]. The real question is whether the $k$ and $1/\varepsilon^{2}$ factors should multiply. Even more embarrassingly, for the frequency moments $F_{p}$ , $p>2$ , the known algorithms use communication $\tilde{O}(k^{2p+1}N^{1-2/p}{\mathrm{poly}}(1/\varepsilon))$ , while the only known lower bound is $\Omega(k+1/\varepsilon^{2})$ [4, 16]. Even for $p=2$ , the best known upper bound is $\tilde{O}(k^{2}/\varepsilon+k^{1.5}/\varepsilon^{3})$ [20], and the authors’ main open question in their paper is “It remains to close the gap in the $F_{2}$ case: can a better lower bound than $\Omega(k)$ be shown, or do there exist $\tilde{O}(k\cdot{\mathrm{poly}}(1/\varepsilon))$ solutions?”

Our Results: We significantly improve the previous communication bounds for approximating the frequency moments, entropy, heavy hitters, and quantiles in the distributed functional monitoring model. In many cases our bounds are optimal. Our results are summarized in Table 1, where they are compared with previous bounds.

	Previous work	This paper	Previous work	This paper
Problem	LB	LB (all static)	UB	UB
$F_{0}$	$\tilde{\Omega}(k)$ [20]	$\mathbf{\tilde{\Omega}(k/\varepsilon^{2})}$	$\tilde{O}(k/\varepsilon^{2})$ [20]	–
$F_{2}$	$\Omega(k)$ [20]	$\mathbf{{\Omega}(k/\varepsilon^{2})}$ (BB)	$\tilde{O}(k^{2}/\varepsilon+k^{1.5}/\varepsilon^{3})$ [20]	$\mathbf{\tilde{O}(\frac{k}{\mathrm{poly}(\varepsilon)})}$
$F_{p}\ (p>1)$	$\Omega(k+1/\varepsilon^{2})$ [4, 16]	$\mathbf{{\Omega}(k^{p-1}/\varepsilon^{2})}$ (BB)	$\tilde{O}(\frac{p}{\varepsilon^{1+2/p}}k^{2p+1}N^{1-2/p})$ [20]	$\mathbf{\tilde{O}(\frac{k^{p-1}}{\mathrm{poly}(\varepsilon)})}$
All-quantile	$\tilde{\Omega}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})$ [35]	$\mathbf{{\Omega}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})}$ (BB)	$\tilde{O}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})$ [35]	–
Heavy Hitters	$\tilde{\Omega}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})$ [35]	$\mathbf{{\Omega}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})}$ (BB)	$\tilde{O}(\min\{\frac{\sqrt{k}}{\varepsilon},\frac{1}{\varepsilon^{2}}\})$ [35]	–
Entropy	$\tilde{\Omega}(1/\sqrt{\varepsilon})$ [4]	$\mathbf{{\Omega}(k/\varepsilon^{2})}$ (BB)	$\tilde{O}(\frac{k}{\varepsilon^{3}})$ [4], $\tilde{O}(\frac{k}{\varepsilon^{2}})$ (static) [33]	–
$\ell_{p}\ (p\in(0,2])$	–	$\mathbf{{\Omega}(k/\varepsilon^{2})}$ (BB)	$\tilde{O}(k/\varepsilon^{2})$ (static) [40]	–

Table 1: UB denotes upper bound; LB denotes lower bound; BB denotes blackboard model.

N

denotes the universe size. All bounds are for randomized algorithms. We assume all bounds hold in the dynamic setting by default, and will state explicitly if they hold in the static setting. For lower bounds we assume the message-passing model by default, and state explicitly if they also hold in the blackboard model.

We have three main results, each introducing a new technique:

1.

We show that for estimating $F_{0}$ in the message-passing model, $\tilde{\Omega}(k/\varepsilon^{2})$ communication is required, matching an upper bound of [20] up to a polylogarithmic factor. Our lower bound holds in the static model in which the $k$ sites just need to approximate $F_{0}$ once on their inputs.
2.

We show that we can estimate $F_{p}$ , for any $p>1$ , using $\tilde{O}(k^{p-1}{\mathrm{poly}}(\varepsilon^{-1}))$ communication in the message-passing model²²2We assume the total number of updates is ${\mathrm{poly}}(N)$ .. This drastically improves upon the previous bound $\tilde{O}(k^{2p+1}N^{1-2/p}{\mathrm{poly}}(\varepsilon^{-1}))$ of [20]. In particular, setting $p=2$ , we resolve the main open question of [20].
3.

We show ${\Omega}(k^{p-1}/\varepsilon^{2})$ communication is necessary for approximating $F_{p}\ (p>1)$ in the blackboard model, significantly improving the prior $\Omega(k+1/\varepsilon^{2})$ bound. As with our lower bound for $F_{0}$ , these are the first lower bounds which depend on the product of $k$ and $1/\varepsilon$ . As with $F_{0}$ , our lower bound holds in the static model in which the sites just approximate $F_{p}$ once.

Our other results in Table 1 are explained in the body of the paper, and use similar techniques.

We would like to mention that after the conference version of our paper, our results found applications in proving a space lower bound at each site for tracking heavy hitters in the functional monitoring model [36], and a communication complexity lower bound of computing $\varepsilon$ -approximations of range spaces in $\mathbb{R}^{2}$ in the message-passing model [34].

Our Techniques: Lower Bound for $F_{0}$ : For illustration, suppose $k=1/\varepsilon^{2}$ . There are $1/\varepsilon^{2}$ sites each holding a random independent bit. Their task is to approximate the sum of the $k$ bits up to an additive error $1/\varepsilon$ . Call this problem $k$ -APPROX-SUM.³³3In the conference version of this paper we introduced a problem called $k$ -GAP-MAJ, in which sites need to decide if at least $1/(2\varepsilon^{2})+1/\varepsilon$ of the bits are $1$ , or at most $1/(2\varepsilon^{2})-1/\varepsilon$ of the bits are $1$ . We instead use $k$ -APPROX-SUM here since we feel it is easier to work with: This problem is stronger than $k$ -GAP-MAJ thus is easier to lower bound, and it suffices for our purpose. $k$ -GAP-MAJ will be introduced and used in Section 6.1 for heavy-hitters and quantiles. We show any correct protocol must reveal $\Omega(1/\varepsilon^{2})$ bits of information about the sites’ inputs. We “compose” this with $2$ -party disjointness ( $2$ -DISJ) [48], in which each party has a bitstring of length $1/\varepsilon^{2}$ and either the strings have disjoint support (the solution is $0$ ) or there is a single coordinate which is $1$ in both strings (the solution is $1$ ). Let $\tau$ be the hard distribution for $2$ -DISJ, shown to require $\Omega(1/\varepsilon^{2})$ bits of communication to solve [48]. Suppose the coordinator and each site share an instance of $2$ -DISJ in which the solution to $2$ -DISJ is a random bit, which is the site’s effective input to $k$ -APPROX-SUM. The coordinator has the same input for each of the $1/\varepsilon^{2}$ instances, while the sites have an independent input drawn from $\tau$ conditioned on the coordinator’s input and output bit determined by $k$ -APPROX-SUM. The inputs are chosen so that if the output of $2$ -DISJ is $1$ , then $F_{0}$ increases by $1$ , otherwise it remains the same. This is not entirely accurate, but it illustrates the main idea. Now, the key is that by the rectangle property of $k$ -party communication protocols, the $1/\varepsilon^{2}$ different output bits are independent conditioned on the transcript. Thus if a protocol does not reveal $\Omega(1/\varepsilon^{2})$ bits of information about these output bits, by an anti-concentration theorem we can show that the protocol cannot succeed with large probability. Finally, since a $(1+\varepsilon)$ -approximation to $F_{0}$ can decide $k$ -APPROX-SUM, and since any correct protocol for $k$ -APPROX-SUM must reveal $\Omega(1/\varepsilon^{2})$ bits of information, the protocol must solve $\Omega(1/\varepsilon^{2})$ instances of $2$ -DISJ, each requiring $\Omega(1/\varepsilon^{2})$ bits of communication (otherwise the coordinator could simulate $k-1$ of the sites and obtain an $o(1/\varepsilon^{2})$ - communication protocol for $2$ -DISJ with the remaining site, contradicting the communication lower bound for $2$ -DISJ on this distribution). We obtain an $\tilde{\Omega}(k/\varepsilon^{2})$ bound for $k\geq 1/\varepsilon^{2}$ by using similar arguments. One cannot show this in the blackboard model since there is an $\tilde{O}(k+1/\varepsilon^{2})$ bound for $F_{0}$ ⁴⁴4The idea is to first obtain a $2$ -approximation. Then, sub-sample so that there are $\Theta(1/\varepsilon^{2})$ distinct elements. Then the first party broadcasts his distinct elements, the second party broadcasts the distinct elements he has that the first party does not, etc..

Lower Bound for $F_{p}$ : Our ${\Omega}(k^{p-1}/\varepsilon^{2})$ bound for $F_{p}$ cannot use the above reduction since we do not know how to turn a protocol for approximating $F_{p}$ into a protocol for solving the composition of $k$ -APPROX-SUM and $2$ -DISJ. Instead, our starting point is a recent $\Omega(1/\varepsilon^{2})$ lower bound for the $2$ -party gap-hamming distance problem GHD [16]. The parties have a length- $1/\varepsilon^{2}$ bitstring, $x$ and $y$ , respectively, and they must decide if the Hamming distance $\Delta(x,y)>1/(2\varepsilon^{2})+1/\varepsilon$ or $\Delta(x,y)<1/(2\varepsilon^{2})-1/\varepsilon$ . A simplification by Sherstov [49] shows a related problem called $2$ -GAP-ORT also has communication complexity of $\Omega(1/\varepsilon^{2})$ bits. Here there are two parties, each with $1/\varepsilon^{2}$ -length bitstrings $x$ and $y$ , and they must decide if $|\Delta(x,y)-1/(2\varepsilon^{2})|>2/\varepsilon$ or $|\Delta(x,y)-1/(2\varepsilon^{2})|<1/\varepsilon$ . Chakrabarti et al. [15] showed that any correct protocol for $2$ -GAP-ORT must reveal ${\Omega}(1/\varepsilon^{2})$ bits of information about $(x,y)$ . By independence and the chain rule, this means for ${\Omega}(1/\varepsilon^{2})$ indices $i$ , ${\Omega}(1)$ bits of information is revealed about $(x_{i},y_{i})$ conditioned on values $(x_{j},y_{j})$ for $j<i$ . We now “embed” an independent copy of a variant of $k$ -party-disjointness, the $k$ -XOR problem, on each of the $1/\varepsilon^{2}$ coordinates of $2$ -GAP-ORT. In this variant, there are $k$ parties each holding a bitstring of length $k^{p}$ . On all but one “special” randomly chosen coordinate, there is a single site assigned to the coordinate and that site uses private randomness to choose whether the value on the coordinate is $0$ or $1$ (with equal probability), and the remaining $k-1$ sites have $0$ on this coordinate. On the special coordinate, with probability $1/4$ all sites have a $0$ on this coordinate (a “00” instance), with probability $1/4$ the first $k/2$ parties have a $1$ on this coordinate and the remaining $k/2$ parties have a $0$ (a “10” instance), with probability $1/4$ the second $k/2$ parties have a $1$ on this coordinate and the remaining $k/2$ parties have a $0$ (a “01” instance), and with the remaining probability $1/4$ all $k$ parties have a $1$ on this coordinate (a “11” instance). We show, via a direct sum for distributional communication complexity, that any deterministic protocol that decides which case the special coordinate is in with probability $1/4+{\Omega}(1)$ has conditional information cost ${\Omega}(k^{p-1})$ . This implies that any protocol that can decide whether the output is in the set $\{10,01\}$ (the “XOR” of the output bits) with probability $1/2+{\Omega}(1)$ has conditional information cost ${\Omega}(k^{p-1})$ . We do the direct sum argument by conditioning the mutual information on low-entropy random variables which allow us to fill in inputs on remaining coordinates without any communication between the parties and without asymptotically affecting our ${\Omega}(k^{p-1})$ lower bound. We design a reduction so that on the $i$ -th coordinate of $2$ -GAP-ORT, the input of the first $k/2$ -players of $k$ -XOR is determined by the public coin (which we condition on) and the first party’s input bit to $2$ -GAP-ORT, and the input of the second $k/2$ -players of $k$ -XOR is determined by the public coin and the second party’s input bit to $2$ -GAP-ORT . We show that any protocol that solves the composition of $2$ -GAP-ORT with $1/\varepsilon^{2}$ copies of $k$ -XOR , a problem that we call $k$ -BTX , must reveal ${\Omega}(1)$ bits of information about the two output bits of an ${\Omega}(1)$ fraction of the $1/\varepsilon^{2}$ copies, and from our ${\Omega}(k^{p-1})$ information cost lower bound for a single copy, we can obtain an overall ${\Omega}(k^{p-1}/\varepsilon^{2})$ bound. Finally, one can show that a $(1+\varepsilon)$ -approximation algorithm for $F_{p}$ can be used to solve $k$ -BTX .

Upper Bound for $F_{p}$ : We illustrate the algorithm for $p=2$ and constant $\varepsilon$ . Unlike [20], we do not use AMS sketches [3]. A nice property of our protocol is that it is the first $1$ -way protocol (the protocol of [20] is not), in the sense that only the sites send messages to the coordinator (the coordinator does not send any messages). Moreover, all messages are simple: if a site receives an update to the $j$ -th coordinate, provided the frequency of coordinate $j$ in its stream exceeds a threshold, it decides with a certain probability to send $j$ to the coordinator. Unfortunately, one can show that this probability cannot be the same for all coordinates $j$ , as otherwise the communication would be too large.

To determine the threshold and probability to send an update to a coordinate $j$ , the sites use the public coin to randomly group all coordinates $j$ into buckets $S_{\ell}$ , where $S_{\ell}$ contains a $1/2^{\ell}$ fraction of the input coordinates. For $j\in S_{\ell}$ , the threshold and probability are only a function of $\ell$ . Inspired by work on sub-sampling [37], we try to estimate the number of coordinates $j$ of magnitude in the range $[2^{h},2^{h+1})$ , for each $h$ . Call this class of coordinates $C_{h}$ . If the contribution to $F_{2}$ from $C_{h}$ is significant, then $|C_{h}|\approx 2^{-2h}\cdot F_{2}$ , and to estimate $|C_{h}|$ we only consider those $j\in C_{h}$ that are in $S_{\ell}$ for a value $\ell$ which satisfies $|C_{h}|\cdot 2^{-\ell}\approx 2^{-2h}\cdot F_{2}\cdot 2^{-\ell}\approx 1$ . We do not know $F_{2}$ and so we also do not know $\ell$ , but we can make a logarithmic number of guesses. We note that the work [37] was available to the authors of [20] for several years, but adapting it to the distributed framework here is tricky in the sense that the “heavy hitters” algorithm used in [37] for finding elements in different $C_{h}$ needs to be implemented in a $k$ -party communication-efficient way.

When choosing the threshold and probability we have two competing constraints; on the one hand these values must be chosen so that we can accurately estimate the values $|C_{h}|$ from the samples. On the other hand, these values need to be chosen so that the communication is not excessive. Balancing these two constraints forces us to use a threshold instead of just the same probability for all coordinates in $S_{\ell}$ . By choosing the thresholds and probabilities to be appropriate functions of $\ell$ , we can satisfy both constraints. Other minor issues in the analysis arise from the fact that different classes contribute at different times, and that the coordinator must be correct at all times. These issues can be resolved by conditioning on a quantity related to the protocol’s correctness being accurate at a small number of selected times in the stream, and then arguing that the quantity is non-decreasing and that this implies that it is correct at all times.

Implications for the Data Stream Model: In 2003, Indyk and Woodruff introduced the GHD problem [38], where a $1$ -round lower bound shortly followed [52]. Ever since, it seemed the space complexity of estimating $F_{0}$ in a data stream with $t>1$ passes hinged on whether GHD required $\Omega(1/\varepsilon^{2})$ communication for $t$ rounds, see, e.g., Question 10 in [2]. A flurry [9, 10, 16, 51, 49] of recent work finally resolved the complexity of GHD. What our lower bound shows for $F_{0}$ is that this is not the only way to prove the $\Omega(1/\varepsilon^{2})$ space bound for multiple passes for $F_{0}$ . Indeed, we just needed to look at $\Theta(1/\varepsilon^{2})$ parties instead of $2$ parties. Since we have an $\Omega(1/\varepsilon^{4})$ communication lower bound for $F_{0}$ with $\Theta(1/\varepsilon^{2})$ parties, this implies an $\Omega((1/\varepsilon^{4})/(t/\varepsilon^{2}))=\Omega(1/(t\varepsilon^{2}))$ bound for $t$ -pass algorithms for approximating $F_{0}$ . Arguably our proof is simpler than the recent GHD lower bounds.

Our ${\Omega}(k^{p-1}/\varepsilon^{2})$ bound for $F_{p}$ also improves a long line of work on the space complexity of estimating $F_{p}$ for $p>2$ in a data stream. The current best upper bound is $\tilde{O}(N^{1-2/p}\varepsilon^{-2})$ bits of space [28]. See Figure 1 of [28] for a list of papers which make progress on the $\varepsilon$ and logarithmic factors. The previous best lower bound is $\tilde{\Omega}(N^{1-2/p}\varepsilon^{-2/p}/t)$ for $t$ passes [7]. By setting $k^{p}=\varepsilon^{2}N$ , we obtain that the total communication is at least ${\Omega}(\varepsilon^{2-2/p}N^{1-1/p}/\varepsilon^{2})$ , and so the implied space lower bound for $t$ -pass algorithms for $F_{p}$ in a data stream is ${\Omega}(\varepsilon^{-2/p}N^{1-1/p}/(tk))={\Omega}(N^{1-2/p}/(\varepsilon^{4/p}t))$ . This gives the first bound that agrees with the tight $\tilde{\Theta}(1/\varepsilon^{2})$ bound when $p=2$ for any constant $t$ . After our work, Ganguly [29] improved this for the special case $t=1$ . That is, for $1$ -pass algorithms for estimating $F_{p}$ , $p>2$ , he shows a space lower bound of $\Omega(N^{1-2/p}/(\varepsilon^{2}\log n))$ .

Other Related Work: There are quite a few papers on multiparty number-in-hand communication complexity, though they are not directly relevant for the problems studied in this paper. Alon et al. [3] and Bar-Yossef et al. [7] studied lower bounds for multiparty set-disjointness, which has applications to $p$ -th frequency moment estimation for $p>2$ in the streaming model. Their results were further improved in [14, 31, 39]. Chakrabarti et al. [12] studied random-partition communication lower bounds for multiparty set-disjointness and pointer jumping, which have a number of applications in the random-order data stream model. Other work includes Chakrabarti et al. [13] for median selection, Magniez et al. [44] and Chakrabarti et al. [11] for streaming language recognition. Very few studies have been conducted in the message-passing model. Duris and Rolim [23] proved several lower bounds in the message-passing model, but only for some simple boolean functions. Three related but more restrictive private-message models were studied by Gal and Gopalan [27], Ergün and Jowhari [24], and Guha and Huang [32]. The first two only investigated deterministic protocols and the third was tailored for the random-order data stream model.

Recently Phillips et al. [47] introduced a technique called symmetrization for the number-in-hand communication model. The idea is to try to find a symmetric hard distribution for the $k$ players. Then one reduces the $k$ -player problem to a $2$ -player problem by assigning Alice the input of a random player and Bob the inputs of the remaining $k-1$ players. The answer to the $k$ -player problem gives the answer to the $2$ -player problem. By symmetrization one can argue that if the communication lower bound for the resulting $2$ -player problem is $L$ , then the lower bound for the $k$ -player problem is $\Omega(kL)$ . While symmetrization developed in [47] can be used to solve some problems for which other techniques are not known, such as bitwise AND/OR and graph connectivity, it has several limitations. First, symmetrization requires a symmetric hard distribution, and for many problems (e.g., $F_{p}\ (p>1)$ in this paper) this is not known or unlikely to exist. Second, for many problems (e.g., $F_{0}$ in this paper), we need a direct-sum type of argument with certain combining functions (e.g., the majority (MAJ)), while in [47], only outputting all copies or with the combining function OR is considered. Third, the symmetrization technique in [47] does not give information cost bounds, and so it is difficult to use when composing problems as is done in this paper. In this paper, we have further developed symmetrization to make it work with the combining function MAJ and the information cost.

Paper Outline: In Section 3 and Section 4 we prove our lower bounds for $F_{0}$ and $F_{p}$ , $p>1$ . The lower bounds apply to functional monitoring, but hold even in the static model. In Section 5 we show improved upper bounds for $F_{p},p>1,$ for functional monitoring. Finally, in Section 6 we prove lower bounds for all-quantile, heavy hitters, entropy and $\ell_{p}$ for any $p\geq 1$ in the blackboard model.

2 Preliminaries

In this section we review some basics on communication complexity and information theory.

Information Theory

We refer the reader to [22] for a comprehensive introduction to information theory. Here we review a few concepts and notations.

Let $H(X)$ denote the Shannon entropy of the random variable $X$ , and let $H_{b}(p)$ denote the binary entropy function when $p\in[0,1]$ . Let $H(X\ |\ Y)$ denote conditional entropy of $X$ given $Y$ . Let $I(X;Y)$ denote the mutual information between two random variables $X,Y$ . Let $I(X;Y\ |\ Z)$ denote the mutual information between two random variables $X,Y$ conditioned on $Z$ . The following is a summarization of the basic properties of entropy and mutual information that we need.

Proposition 1

Let $X,Y,Z,W$ be random variables.

1.

If $X$ takes value in $\{1,2,\ldots,m\}$ , then $H(X)\in[0,\log m]$ .
2.

$H(X)\geq H(X\ |\ Y)$ and $I(X;Y)=H(X)-H(X\ |\ Y)\geq 0$ .
3.

If $X$ and $Z$ are independent, then we have $I(X;Y\ |\ Z)\geq I(X;Y)$ . Similarly, if $X,Z$ are independent given $W$ , then $I(X;Y\ |\ Z,W)\geq I(X;Y\ |\ W)$ .
4.

(Chain rule of mutual information)

$I(X,Y;Z)=I(X;Z)+I(Y;Z\ |\ X).$

And in general, for any random variables $X_{1},X_{2},\ldots,X_{n},Y$ ,

$\textstyle I(X_{1},\ldots,X_{n};Y)=\sum_{i=1}^{n}I(X_{i};Y\ |\ X_{1},\ldots,X_{i-1})$ .

Thus, $I(X,Y;Z\ |\ W)\geq I(X;Z\ |\ W)$ .
5.

(Data processing inequality) If $X$ and $Z$ are conditionally independent given $Y$ , then $I(X;Y\ |\ Z,W)\leq I(X;Y\ |\ W)$ .
6.

(Fano’s inequality) Let $X$ be a random variable chosen from domain $\mathcal{X}$ according to distribution $\mu_{X}$ , and $Y$ be a random variable chosen from domain $\mathcal{Y}$ according to distribution $\mu_{Y}$ . For any reconstruction function $g:\mathcal{Y}\to\mathcal{X}$ with error $\delta_{g}$ ,

$H_{b}(\delta_{g})+\delta_{g}\log(\left|\mathcal{X}\right|-1)\geq H(X\ |\ Y).$
7.

(The Maximum Likelihood Estimation principle) With the notations as in Fano’s inequality, if the (deterministic) reconstruction function is $g(y)=x$ for the $x$ that maximizes the conditional probability $\mu_{X}(x\ |\ Y=y)$ , then

$\delta_{g}\leq 1-\frac{1}{2^{H(X\ |\ Y)}}.$

Call this $g$ the maximum likelihood function.

Communication complexity

In the two-party randomized communication complexity model (see e.g., [43]), we have two players Alice and Bob. Alice is given $x\in\mathcal{X}$ and Bob is given $y\in\mathcal{Y}$ , and they want to jointly compute a function $f(x,y)$ by exchanging messages according to a protocol $\Pi$ . Let $\Pi(x,y)$ denote the message transcript when Alice and Bob run protocol $\Pi$ on input pair $(x,y)$ . We sometimes abuse notation by identifying the protocol and the corresponding random transcript, as long as there is no confusion.

The communication complexity of a protocol is defined as the maximum number of bits exchanged among all pairs of inputs. We say a protocol $\Pi$ computes $f$ with error probability $\delta\ (0\leq\delta\leq 1)$ if there exists a function $g$ such that for all input pairs $(x,y)$ , $\mathsf{Pr}[g(\Pi(x,y))\neq f(x,y)]\leq\delta$ . The $\delta$ -error randomized communication complexity, denoted by $R^{\delta}(f)$ , is the cost of the minimum-communication randomized protocol that computes $f$ with error probability $\delta$ . The $(\mu,\delta)$ -distributional communication complexity of $f$ , denoted by $D_{\mu}^{\delta}(f)$ , is the cost of the minimum-communication deterministic protocol that gives the correct answer for $f$ on at least a $1-\delta$ fraction of all input pairs, weighted by distribution $\mu$ . Yao [53] showed that

Lemma 1 (Yao’s Lemma)

$R^{\delta}(f)\geq\max_{\mu}D_{\mu}^{\delta}(f)$ .

Thus, one way to prove a lower bound for randomized protocols is to find a hard distribution $\mu$ and lower bound $D_{\mu}^{\delta}(f)$ . This is called Yao’s Minimax Principle.

We will use the notion expected distributional communication complexity $\mathsf{ED}_{\mu}^{\delta}(f)$ , which was introduced in [47] (where it was written as $\mathsf{E}[D_{\mu}^{\delta}(f)]$ , with a bit abuse of notation) and is defined to be the expected cost (rather than the worst case cost) of the deterministic protocol that gives the correct answer for $f$ on at least $1-\delta$ fraction of all inputs, where the expectation is taken over distribution $\mu$ .

The definitions for two-party protocols can be easily extended to the multiparty setting, where we have $k$ players and the $i$ -th player is given an input $x_{i}\in\mathcal{X}_{i}$ . Again the $k$ players want to jointly compute a function $f(x_{1},x_{2},\ldots,x_{k})$ by exchanging messages according to a protocol $\Pi$ .

Information complexity

Information complexity was introduced in a series of papers including [17, 7]. We refer the reader to Bar-Yossef’s Thesis [6]; see Chapter 6 for a detailed introduction. Here we briefly review the concepts of information cost and conditional information cost for $k$ -player communication problems. All of them are defined in the blackboard number-in-hand model.

Let $\mu$ be an input distribution on $\mathcal{X}_{1}\times\mathcal{X}_{2}\times\ldots\times\mathcal{X}_{k}$ and let $X$ be a random input chosen from $\mu$ . Let $\Pi$ be a randomized protocol running on inputs in $\mathcal{X}_{1}\times\mathcal{X}_{2}\times\ldots\times\mathcal{X}_{k}$ . The information cost of $\Pi$ with respect to $\mu$ is $I(X;\Pi)$ . The information complexity of a problem $f$ with respect to a distribution $\mu$ and error parameter $\delta\ (0\leq\delta\leq 1)$ , denoted $\mathrm{IC}_{\mu}^{\delta}(f)$ , is the minimum information cost of a $\delta$ -error protocol for $f$ with respect to $\mu$ . We will work in the public coin model, in which all parties also share a common source of randomness.

We say a distribution $\lambda$ partitions $\mu$ if conditioned on $\lambda$ , $\mu$ is a product distribution. Let $X$ be a random input chosen from $\mu$ and $D$ be a random variable chosen from $\lambda$ . For a randomized protocol $\Pi$ on $\mathcal{X}_{1}\times\mathcal{X}_{2}\times\ldots\times\mathcal{X}_{k}$ , the conditional information cost of $\Pi$ with respect to the distribution $\mu$ on $\mathcal{X}_{1}\times\mathcal{X}_{2}\times\ldots\times\mathcal{X}_{k}$ and a distribution $\lambda$ partitioning $\mu$ is defined as $I(X;\Pi\ |\ D)$ . The conditional information complexity of a problem $f$ with respect to a distribution $\mu$ , a distribution $\lambda$ partitioning $\mu$ , and error parameter $\delta\ (0\leq\delta\leq 1)$ , denoted $\mathrm{IC}_{\mu}^{\delta}(f|\ \lambda)$ , is the minimum information cost of a $\delta$ -error protocol for $f$ with respect to $\mu$ and $\lambda$ . The following proposition can be found in [7].

Proposition 2

For any distribution $\mu$ , distribution $\lambda$ partitioning $\mu$ , and error parameter $\delta\ (0\leq\delta\leq 1)$ ,

R^{\delta}(f)\geq\mathrm{IC}_{\mu}^{\delta}(f)\geq\mathrm{IC}_{\mu}^{\delta}(f\ |\ \lambda).

Statistical distance measures

Given two probability distributions $\mu$ and $\nu$ over the same space $\mathcal{X}$ , the following statistical distance measures will be used in this paper:

1.

Total variation distance: $\mathsf{TV}(\mu,\nu)\stackrel{{\scriptstyle def}}{{=}}\max_{A\subseteq\mathcal{X}}\left|\mu(A)-\nu(A)\right|$ .
2.

Hellinger distance: $h(\mu,\nu)\stackrel{{\scriptstyle def}}{{=}}\sqrt{\frac{1}{2}\sum_{x\in\mathcal{X}}\left(\sqrt{\mu(x)}-\sqrt{\nu(x)}\right)^{2}}$

We have the following relation between total variation distance and Hellinger distance (cf. [6], Chapter $2$ ).

Proposition 3

$h^{2}(\mu,\nu)\leq\mathsf{TV}(\mu,\nu)\leq h(\mu,\nu)\sqrt{2-h^{2}(\mu,\nu)}.$

The total variation distance of transcripts on a pair of inputs is closely related to the error of a randomized protocol. The following proposition can be found in [6], Proposition 6.22 (the original proposition is for the 2-party case, and generalizing it to the multiparty case is straightforward).

Proposition 4

Let $0<\delta<1/2$ , and $\Pi$ be a $\delta$ -error randomized protocol for a function $f:\mathcal{X}_{1}\times\ldots\times\mathcal{X}_{k}\to\mathcal{Z}$ . Then, for every two inputs $(x_{1},\ldots,x_{k}),(x^{\prime}_{1},\ldots,x^{\prime}_{k})\in\mathcal{X}_{1}\times\ldots\times\mathcal{X}_{k}$ for which $f(x_{1},\ldots,x_{k})\neq f(x^{\prime}_{1},\ldots,x^{\prime}_{k})$ , it holds that

\mathsf{TV}(\Pi_{x_{1},\ldots,x_{k}},\Pi_{x^{\prime}_{1},\ldots,x^{\prime}_{k}})>1-2\delta.

Conventions.

In the rest of the paper we call a player a site, as to be consistent with the distributed functional monitoring model. We denote $[n]=\{1,\ldots,n\}$ . Let $\oplus$ be the XOR function. All logarithms are base- $2$ unless noted otherwise. We say $\tilde{W}$ is a $(1+\varepsilon)$ -approximation of $W$ , $0<\varepsilon<1$ , if $W\leq\tilde{W}\leq(1+\varepsilon)W$ .

3 A Lower Bound for $F_{0}$

We introduce a problem called $k$ -APPROX-SUM, and then compose it with $2$ -DISJ (studied, e.g., in [48]) to prove a lower bound for $F_{0}$ . In this section we work in the message-passing model.

3.1 The $k$ -APPROX-SUM Problem

In the $k$ -APPROX-SUM_f,τ problem, we have $k$ sites $S_{1},S_{2},\ldots,S_{k}$ and the coordinator. Let $f:\mathcal{X}\times\mathcal{Y}\to\{0,1\}$ be an arbitrary function, and let $\tau$ be an arbitrary distribution on $\mathcal{X}\times\mathcal{Y}$ such that for $(X,Y)\sim\tau$ , $f(X,Y)=1$ with probability $\beta$ , and $0$ with probability $1-\beta$ , where $\beta\ (c_{\beta}/k\leq\beta\leq 1/c_{\beta}$ for a sufficiently large constant $c_{\beta}$ ) is a parameter. We define the input distribution $\mu$ for $k$ -APPROX-SUM_f,τ on $\{X_{1},\ldots,X_{k},Y\}\in\mathcal{X}^{k}\times\mathcal{Y}$ as follows: We first sample $(X_{1},Y)\sim\tau$ , and then independently sample $X_{2},\ldots,X_{k}\sim\tau\ |\ Y$ . Note that each pair $(X_{i},Y)$ is distributed according to $\tau$ . Let $Z_{i}=f(X_{i},Y)$ . Thus $Z_{i}$ ’s are i.i.d. Bernoulli $(\beta)$ . Let $Z=\{Z_{1},Z_{2},\ldots,Z_{k}\}$ . We assign $X_{i}$ to site $S_{i}$ for each $i\in[k]$ , and assign $Y$ to the coordinator.

In the $k$ -APPROX-SUM_f,τ problem, the $k$ sites want to approximate $\sum_{i\in[k]}Z_{i}$ up to an additive factor of $\sqrt{\beta k}$ . In the rest of this section, for convenience, we omit subscripts $f,\tau$ in $k$ -APPROX-SUM_f,τ, since our results will hold for all $f,\tau$ having the properties mentioned above.

For a fixed transcript $\Pi=\pi$ , let $q_{i}^{\pi}=\mathsf{Pr}[Z_{i}=1\ |\ \Pi=\pi]$ . Thus $\sum_{i\in[k]}q_{i}^{\pi}=\mathsf{E}[\sum_{i\in[k]}Z_{i}\ |\ \Pi=\pi]$ . Let $c_{0}$ be a sufficiently large constant.

Definition 1

Given an input $(x_{1},\ldots,x_{k},y)$ and a transcript $\Pi=\pi$ , let $z_{i}=f(x_{i},y)$ and $z=\{z_{1},\ldots,z_{k}\}$ . For convenience, we define $\Pi(z)\triangleq\Pi(x_{1},\ldots,x_{k},y)$ . We say

1.

$\pi$ is bad₁ for $z$ (denoted by $z\perp_{1}\pi$ ) if $\Pi(z)=\pi$ , and for at least $0.1$ fraction of $\{i\in[k]\ |\ z_{i}=1\}$ , it holds that $q_{i}^{\pi}\leq\beta/c_{0}$ , and
2.

$\pi$ is bad₀ for $z$ (denoted by $z\perp_{0}\pi$ ) if $\Pi(z)=\pi$ , and for at least $0.1$ fraction of $\{i\in[k]\ |\ z_{i}=0\}$ , it holds that $q_{i}^{\pi}\geq\beta/c_{0}$ .

And $\pi$ is good for $z$ otherwise.

In this section, we will prove the following theorem. Except stated explicitly, all probabilities, expectations and variances are taken with respect to the input distribution $\mu$ .

Theorem 1

Let $\Pi$ be the transcript of any deterministic protocol for $k$ -APPROX-SUM on input distribution $\mu$ with error probability $\delta$ for some sufficiently small constant $\delta$ , then $\mathsf{Pr}[\Pi\textrm{ is good}]\geq 0.96$ .

The following observation, which easily follows from the rectangle property of communication protocols, is crucial to our proof. We have included a proof in Appendix A.

Observation 1

Conditioned on $\Pi$ , $Z_{1},Z_{2},\ldots,Z_{k}$ are independent.

Definition 2

We say a transcript $\pi$ is rare⁺ if $\sum_{i\in[k]}q_{i}^{\pi}\geq 4\beta k$ and rare^- if $\sum_{i\in[k]}q_{i}^{\pi}\leq\beta k/4$ . In both cases we say $\pi$ is rare. Otherwise we say it is normal.

Definition 3

We say $Z=\{Z_{1},Z_{2},\ldots,Z_{k}\}$ is a joker⁺ if $\sum_{i\in[k]}Z_{i}\geq 2\beta k$ , and a joker^- if $\sum_{i\in[k]}Z_{i}\leq\beta k/2$ . In both cases we say $Z$ is a joker.

Lemma 2

Under the assumption of Theorem 1, $\mathsf{Pr}[\Pi\textrm{ is normal}]\geq 0.99$ .

Proof:

First, we can apply a Chernoff bound on random variables $Z_{1},\ldots,Z_{k}$ , and get

\mathsf{Pr}[Z\textrm{ is a joker}^{+}]=\mathsf{Pr}\left[\sum_{i\in[k]}Z_{i}\geq 2\beta k\right]\leq e^{-\beta k/3}.

Second, by Observation 1, we can apply a Chernoff bound on random variables $Z_{1},\ldots,Z_{k}$ conditioned on $\Pi$ being rare⁺,

			$\displaystyle\mathsf{Pr}[Z\textrm{ is a joker}^{+}\ \|\ \Pi\textrm{ is rare}^{+}]$
		$\displaystyle\geq$	$\displaystyle\sum_{\pi}\mathsf{Pr}\left[\Pi=\pi\ \|\ \Pi\textrm{ is rare}^{+}\right]\mathsf{Pr}\left[Z\textrm{ is a joker}^{+}\ \|\ \Pi=\pi,\Pi\textrm{ is rare}^{+}\right]$
		$\displaystyle=$	$\displaystyle\sum_{\pi}\mathsf{Pr}\left[\Pi=\pi\ \|\ \Pi\textrm{ is rare}^{+}\right]\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\geq 2\beta k\ \right\|\sum_{i\in[k]}q_{i}^{\pi}\geq 4\beta k,\Pi=\pi\right]$
		$\displaystyle\geq$	$\displaystyle\sum_{\pi}\mathsf{Pr}\left[\Pi=\pi\ \|\ \Pi\textrm{ is rare}^{+}\right]\left(1-e^{-\beta k/2}\right)$
		$\displaystyle=$	$\displaystyle\left(1-e^{-\beta k/2}\right).$

Finally by Bayes’ theorem, we have that

	$\displaystyle\mathsf{Pr}[\Pi\textrm{ is rare}^{+}]$	$\displaystyle=$	$\displaystyle\frac{\mathsf{Pr}[Z\textrm{ is a joker}^{+}]\cdot\mathsf{Pr}[\Pi\textrm{ is rare}^{+}\ \|\ Z\textrm{ is a joker}^{+}]}{\mathsf{Pr}[Z\textrm{ is a joker}^{+}\ \|\ \Pi\textrm{ is rare}^{+}]}$
		$\displaystyle\leq$	$\displaystyle\frac{e^{-\beta k/3}}{1-e^{-\beta k/2}}\leq 2e^{-\beta k/3}.$

Similarly, we can also show that $\mathsf{Pr}[\Pi\textrm{ is rare}^{-}]\leq 2e^{-\beta k/8}$ . Therefore $\mathsf{Pr}[\Pi\textrm{ is rare}]\leq 4e^{-\beta k/8}\leq 0.01$ (recall that by our assumption $\beta k\geq c_{\beta}$ for a sufficiently large constant $c_{\beta}$ ).

Definition 4

Let $c_{\ell}=40c_{0}$ . We say a transcript $\pi$ is weak if $\sum_{i\in[k]}q_{i}^{\pi}(1-q_{i}^{\pi})\geq\beta k/c_{\ell}$ , and strong otherwise.

Lemma 3

Under the assumption of Theorem 1, $\mathsf{Pr}[\Pi\textrm{ is normal and strong}]\geq 0.98$ .

Proof:

We first show that for a normal and weak transcript $\pi$ , there exists a constant $\delta_{\ell}=\delta_{\ell}(c_{\ell})$ such that

	$\displaystyle\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\leq\sum_{i\in[k]}q_{i}^{\pi}+2\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$	$\displaystyle\geq$	$\displaystyle\delta_{\ell},$		(1)
	$\displaystyle\text{and \quad}\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\geq\sum_{i\in[k]}q_{i}^{\pi}+4\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$	$\displaystyle\geq$	$\displaystyle\delta_{\ell}.$		(2)

The first inequality is a simple application of Chernoff-Hoeffding bound. Recall that for a normal $\pi$ , $\sum_{i\in[k]}q_{i}^{\pi}\leq 4\beta k$ . We have

			$\displaystyle\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\leq\sum_{i\in[k]}q_{i}^{\pi}+2\sqrt{\beta k}\ \right\|\ \Pi=\pi,\Pi\text{ is normal}\right]$
		$\displaystyle\geq$	$\displaystyle 1-\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\geq\sum_{i\in[k]}q_{i}^{\pi}+2\sqrt{\beta k}\ \right\|\ \Pi=\pi,\Pi\text{ is normal}\right]$
		$\displaystyle\geq$	$\displaystyle 1-e^{-\frac{8\sqrt{\beta k}^{2}}{\sum_{i\in[k]}q_{i}^{\pi}}}\geq 1-e^{-2}\geq\delta_{\ell}.\quad(\text{for a sufficiently small constant $\delta_{\ell}$})$

Now we prove for the second inequality. We will need the following anti-concentration result which is an easy consequence of Feller [26] (cf. [46]).

Fact 1

([46]) Let $Y$ be a sum of independent random variables, each attaining values in $[0,1]$ , and let $\sigma=\sqrt{\mathsf{Var}[Y]}\geq 200$ . Then for all $t\in[0,\sigma^{2}/100]$ , we have

\mathsf{Pr}[Y\geq\mathsf{E}[Y]+t]\geq c\cdot e^{-t^{2}/(3\sigma^{2})}

for a universal constant $c>0$ .

For a normal and weak $\Pi=\pi$ , it holds that

$\displaystyle\mathsf{Var}\left[\sum_{i\in[k]}Z_{i}\ \|\ \Pi=\pi\right]$	$\displaystyle=$	$\displaystyle\sum_{i\in[k]}\mathsf{Var}\left[Z_{i}\ \|\ \Pi=\pi\right]\quad(\text{by observation \ref{obs:first}})$
	$\displaystyle=$	$\displaystyle\sum_{i\in[k]}q_{i}^{\pi}(1-q_{i}^{\pi})$
	$\displaystyle\geq$	$\displaystyle\beta k/c_{\ell}.\quad(\text{by definition of a weak $\pi$})$

Recall that by our assumption, $\beta k\geq c_{\beta}$ for a sufficiently large constant $c_{\beta}$ , thus $\sqrt{\beta k}\leq\beta k/(100c_{\ell})$ and $\beta k/c_{\ell}\geq 200^{2}$ . Using Lemma 1, we have for a universal constant $c$ ,

			$\displaystyle\mathsf{Pr}\left[\left.\sum_{i\in[k]}Z_{i}\geq\sum_{i\in[k]}q_{i}^{\pi}+4\sqrt{\beta k}\ \right\|\ \Pi=\pi,\Pi\text{ is weak}\right]$
		$\displaystyle\geq$	$\displaystyle c\cdot e^{-\frac{(4\sqrt{\beta k})^{2}}{3\beta k/c_{\ell}}}\geq c\cdot e^{-16c_{\ell}/3}\geq\delta_{\ell}.\quad(\text{for a sufficiently small constant $\delta_{\ell}$})$

By (1) and (2), it is easy to see that given that $\Pi$ is normal, it cannot be weak with probability more than $0.01$ , since otherwise by Lemma 2 and the analysis above, the error probability of the protocol will be at least $0.99\cdot 0.01\cdot\delta_{\ell}>\delta$ , for an arbitrarily small constant error $\delta$ , violating the success guarantee of the lemma. Therefore,

\mathsf{Pr}[\Pi\text{ is normal and strong}]\geq\mathsf{Pr}[\Pi\text{ is normal}]\ \mathsf{Pr}[\Pi\text{ is strong}\ |\ \Pi\text{ is normal}]\geq 0.99\cdot 0.99\geq 0.98.

Now we analyze the probability of $\Pi$ being good. For a $Z=z$ , let $H_{0}(z)=\{i\ |\ z_{i}=0\}$ and $H_{1}(z)=\{i\ |\ z_{i}=1\}$ . We have the following two lemmas.

Lemma 4

Under the assumption of Theorem 1, $\mathsf{Pr}[\Pi\text{ is bad${}_{0}$}\ |\ \Pi\text{ is normal and strong}]\leq 0.01$ .

Proof:

Consider any $Z=z$ . First, by the definition of a normal $\pi$ , we have $\sum_{i:z_{i}=0}q_{i}^{\pi}\leq\sum_{i\in[k]}q_{i}^{\pi}\leq 4\beta k$ . Therefore the number of $i$ ’s such that $z_{i}=0$ and $q_{i}^{\pi}>(1-\beta/c_{0})$ is at most $4\beta k/(1-\beta/c_{0})\leq 8\beta k$ . Second, by the definition of a strong $\pi$ , we have $\sum_{i:z_{i}=0}q_{i}^{\pi}(1-q_{i}^{\pi})\leq\sum_{i\in[k]}q_{i}^{\pi}(1-q_{i}^{\pi})\leq\beta k/c_{\ell}$ . Therefore the number of $i$ ’s such that $z_{i}=0$ and $\beta/c_{0}\leq q_{i}^{\pi}\leq(1-\beta/c_{0})$ is at most $\frac{\beta k/c_{\ell}}{\beta/c_{0}\cdot(1-\beta/c_{0})}\leq 0.05k\ (c_{\ell}=40c_{0})$ . Also note that if $z$ is not joker, then $\left|H_{0}(z)\right|\geq(k-2\beta k)$ . Thus conditioned on a normal and strong $\pi$ , as well as $z$ is not a joker, the number of $i$ ’s such that $z_{i}=0$ and $q_{i}^{\pi}<\beta/c_{0}$ is at least

\displaystyle(k-2\beta k)-8\beta k-0.05k>0.9k\geq 0.9\left|H_{0}(z)\right|,

where we have used our assumption that $\beta\leq 1/c_{\beta}$ for a sufficiently large constant $c_{\beta}$ . We conclude that

\mathsf{Pr}[\Pi\text{ is bad${}_{0}$}\ |\ \Pi\text{ is normal and strong}]\leq\mathsf{Pr}[Z\text{ is a joker}]\leq 2e^{-\beta k/8}\leq 0.01.

Lemma 5

Under the assumption of Theorem 1, $\mathsf{Pr}[\Pi\text{ is bad${}_{1}$}\ |\ \Pi\text{ is normal}]\leq 0.01$ .

Proof:

Call a $\pi$ is bad₁ for a set $T\subseteq[k]$ (denoted by $T\perp_{1}\pi$ ), if for more than $0.1$ fraction of $i\in T$ , we have $q_{i}^{\pi}\leq\beta/c_{0}$ . Let $\chi(\mathcal{E})=1$ if $\mathcal{E}$ holds and $\chi(\mathcal{E})=0$ otherwise. We have

	$\displaystyle\mathsf{Pr}[\Pi\text{ is bad${}_{1}$}\ \|\ \Pi\text{ is normal}]$	(4)
$\displaystyle=$	$\displaystyle\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]\sum_{z}\mathsf{Pr}[Z=z\ \|\ \Pi=\pi,\Pi\text{ is normal}]\ \chi(z\perp_{1}\pi)$
$\displaystyle\leq$	$\displaystyle\mathsf{Pr}[Z\text{ is a joker}]+\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]$
	$\displaystyle\quad\quad\sum_{\ell\in[{\beta k}/{2},2\beta k]}\sum_{T\subseteq[k]:\left\|T\right\|=\ell}\sum_{z}\mathsf{Pr}[Z=z\ \|\ \Pi=\pi,\Pi\text{ is normal}]\ \chi(H_{1}(z)=T)\ \chi(T\perp_{1}\pi)$
$\displaystyle\leq$	$\displaystyle\mathsf{Pr}[Z\text{ is a joker}]+\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]$
	$\displaystyle\quad\quad\sum_{\ell\in[\beta k/2,2\beta k]}\left(\left.\sum_{\begin{subarray}{c}T\subseteq[k]:\left\|T\right\|=\ell\\ T\perp_{1}\pi\end{subarray}}\prod_{i\in T}q_{i}^{\pi}\ \right\|\ \Pi=\pi,\Pi\text{ is normal}\right)$

The last inequality holds since in (4), in the last term, we count the probability of each possible set $T$ of size $\ell$ and is $\perp_{1}$ to $\pi$ that its elements are all $1$ , which upper bounds the corresponding summation in (4). Now for a fixed $\ell$ , conditioned on a normal $\pi$ , we consider the term

\displaystyle\sum_{\begin{subarray}{c}T\subseteq[k]:\left|T\right|=\ell\\ T\perp_{1}\pi\end{subarray}}\prod_{i\in T}q_{i}^{\pi}.

(5)

W.l.o.g., we can assume that $q_{1}^{\pi}\geq\ldots\geq q_{s}^{\pi}>\beta/c_{0}\geq q_{s+1}^{\pi}\geq\ldots\geq q_{k}^{\pi}$ for an $s=\kappa_{s}k\ (0<\kappa_{s}\leq 1)$ . We consider a pair $(q_{u}^{\pi},q_{v}^{\pi})\ (u,v\in[k])$ . Terms in the summation (5) that includes either $q_{u}^{\pi}$ or $q_{v}^{\pi}$ can be written as

\displaystyle q_{u}^{\pi}\sum_{\begin{subarray}{c}T\subseteq[k]:\left|T\right|=\ell\\ T\perp_{1}\pi\\ u\in T,v\not\in T\end{subarray}}\prod_{i\in T\backslash u}q_{i}^{\pi}+q_{v}^{\pi}\sum_{\begin{subarray}{c}T\subseteq[k]:\left|T\right|=\ell\\ T\perp_{1}\pi\\ v\in T,u\not\in T\end{subarray}}\prod_{i\in T\backslash v}q_{i}^{\pi}+q_{u}^{\pi}q_{v}^{\pi}\sum_{\begin{subarray}{c}T\subseteq[k]:\left|T\right|=\ell\\ T\perp_{1}\pi\\ v\in T,u\in T\end{subarray}}\prod_{i\in T\backslash v,u}q_{i}^{\pi}.

By the symmetry of $q_{u}^{\pi},q_{v}^{\pi}$ , the sets $\{T\backslash u\ |\ T\subseteq[k],\left|T\right|=\ell,T\perp_{1}\pi,u\in T,v\not\in T\}$ and $\{T\backslash v\ |\ T\subseteq[k],\left|T\right|=\ell,T\perp_{1}\pi,v\in T,u\not\in T\}$ are the same. Using this fact and the AM-GM inequality, it is easy to see that the sum will not decrease if we set $(q_{u}^{\pi})^{\prime}=(q_{v}^{\pi})^{\prime}=(q_{u}^{\pi}+q_{v}^{\pi})/{2}$ . Call such an operation an equalization. We repeat applying such equalizations to any pair $(q_{u}^{\pi},q_{v}^{\pi})$ , with the constraint that if $u\in[1,s]$ and $v\in[s+1,k]$ , then we only “average” them to the extent that $(q_{u}^{\pi})^{\prime}=\beta/c_{0},(q_{v}^{\pi})^{\prime}=q_{u}^{\pi}+q_{v}^{\pi}-\beta/c_{0}$ if $q_{u}^{\pi}+q_{v}^{\pi}\leq 2\beta/c_{0}$ , and $(q_{v}^{\pi})^{\prime}=\beta/c_{0},(q_{u}^{\pi})^{\prime}=q_{u}^{\pi}+q_{v}^{\pi}-\beta/c_{0}$ otherwise. We introduce this constraint because we do not want to change $\left|\{i\ |\ (q_{i}^{\pi})^{\prime}\leq\beta/c_{0}\}\right|$ , since otherwise a set $T$ which was originally $\perp_{1}\Pi$ can be $\not\perp_{1}\Pi$ after these equalizations. We cannot further apply equalizations when one of the followings happen.

\displaystyle(q_{1}^{\pi})^{\prime}=\ldots=(q_{s}^{\pi})^{\prime}>\beta/c_{0}=(q_{s+1}^{\pi})^{\prime}=\ldots=(q_{k}^{\pi})^{\prime}.

(6)

\displaystyle(q_{1}^{\pi})^{\prime}=\ldots=(q_{s}^{\pi})^{\prime}=\beta/c_{0}\geq(q_{s+1}^{\pi})^{\prime}=\ldots=(q_{k}^{\pi})^{\prime}.

(7)

We note that actually (7) cannot happen since $\sum_{i\in[k]}(q_{i}^{\pi})^{\prime}=\sum_{i\in[k]}q_{i}^{\pi}$ is preserved during equalizations, and conditioned on a normal $\pi$ , we have $\sum_{i\in[k]}q_{i}^{\pi}\geq\beta k/4>\beta k/c_{0}$ .

Let $q=(q_{1}^{\pi})^{\prime}=\ldots=(q_{s}^{\pi})^{\prime}$ . For a normal $\pi$ , it holds that $\sum_{i\in[k]}(q_{i}^{\pi})^{\prime}=s\cdot q+(k-s)\cdot\beta/c_{0}=r\in[\beta k/4,4\beta k]$ . Let $\alpha\in(0.1,1]$ . Recall that $\ell\in[\beta k/2,2\beta k]$ , and we have set $s=\kappa_{s}k$ . We try to upper bound (5) using (6).

$\displaystyle\sum_{\begin{subarray}{c}T\subseteq[k]:\left\|T\right\|=\ell\\ T\perp_{1}\pi\end{subarray}}\prod_{i\in T}q_{i}^{\pi}.$	$\displaystyle\leq$	$\displaystyle\left({k-s\choose\alpha\ell}\cdot{s\choose(1-\alpha)\ell}\right)\cdot\left(\left(\frac{\beta}{c_{0}}\right)^{\alpha\ell}\cdot\left(\frac{r}{s}-\frac{(k-s)\beta}{c_{0}s}\right)^{(1-\alpha)\ell}\right)$
	$\displaystyle\leq$	$\displaystyle\left(\left(\frac{e(1-\kappa_{s})k}{\alpha\ell}\right)^{\alpha\ell}\cdot\left(\frac{e\kappa_{s}k}{(1-\alpha)\ell}\right)^{(1-\alpha)\ell}\right)\cdot\left(\left(\frac{\beta}{c_{0}}\right)^{\alpha\ell}\cdot\left(\frac{r}{\kappa_{s}k}\right)^{(1-\alpha)\ell}\right)$
	$\displaystyle\leq$	$\displaystyle\left(\frac{e}{\alpha c_{0}}\cdot\frac{\beta k}{\ell}\right)^{\alpha\ell}\cdot\left(\frac{er}{(1-\alpha)\ell}\right)^{(1-\alpha)\ell}$
	$\displaystyle\leq$	$\displaystyle\left(\frac{8e}{(c_{0})^{\alpha}\cdot\alpha^{\alpha}(1-\alpha)^{1-\alpha}}\right)^{\ell}$
	$\displaystyle\leq$	$\displaystyle\left(\frac{8e}{(c_{0})^{0.1}\cdot(1/e)^{2/e}}\right)^{\beta k/2}$	(9)

In (Proof: ), the first term is the number of possible choices of the set $T\ (T=\ell)$ with $\alpha$ fraction of items in $[s+1,\infty]$ , and the rest in $[1,s]$ . And the second term upper bounds $\prod_{i\in T}q_{i}^{\pi}$ according to the discussion above. Here we have assumed $\alpha<1$ , otherwise if $\alpha=1$ , then $(\ref{eq:alpha})\leq{k\choose\ell}\cdot(\beta/c_{0})^{\ell}\leq(2e/c_{0})^{\beta k/2}$ , which is smaller than (9). Now, (4) can be upper bounded by

			$\displaystyle 2e^{-\beta k/8}+\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]\cdot 2\beta k\cdot\left(\frac{8e}{(c_{0})^{0.1}\cdot(1/e)^{2/e}}\right)^{\beta k/2}$
		$\displaystyle=$	$\displaystyle 2e^{-\beta k/8}+2\beta k\cdot\left(\frac{8e}{(c_{0})^{0.1}\cdot(1/e)^{2/e}}\right)^{\beta k/2}$
		$\displaystyle\leq$	$\displaystyle 0.01.\quad\text{(for a sufficiently large constant $c_{0}$)}$

Finally, combining Lemma 3, Lemma 4 and Lemma 5, we get

			$\displaystyle\mathsf{Pr}[\Pi\text{ is good}]\geq\mathsf{Pr}[\Pi\text{ is good, normal and strong}]$
		$\displaystyle=$	$\displaystyle\mathsf{Pr}[\Pi\text{ is normal and strong}](1-\mathsf{Pr}[\Pi\text{ is bad${}_{0}$}\ \|\ \Pi\text{ is normal and strong}]$
			$\displaystyle\quad\quad\quad\quad-\mathsf{Pr}[\Pi\text{ is bad${}_{1}$}\ \|\ \Pi\text{ is normal and strong}])$
		$\displaystyle\geq$	$\displaystyle\mathsf{Pr}[\Pi\text{ is normal and strong}](1-\mathsf{Pr}[\Pi\text{ is bad${}_{0}$}\ \|\ \Pi\text{ is normal and strong}])$
			$\displaystyle\quad\quad\quad\quad-\mathsf{Pr}[\Pi\text{ is normal}]\ \mathsf{Pr}[\Pi\text{ is bad${}_{1}$}\ \|\ \Pi\text{ is normal}]$
		$\displaystyle\geq$	$\displaystyle 0.98\cdot(1-0.01)-0.01\geq 0.96.$

3.2 The $2$ -DISJ Problem

In $2$ -DISJ problem, Alice has a set $x\subseteq[n]$ and Bob has a set $y\subseteq[n]$ . Their goal is to output $1$ if $x\cap y\neq\emptyset$ , and $0$ otherwise.

We define the input distribution $\tau_{\beta}$ as follows. Let $\ell=(n+1)/4$ . With probability $\beta$ , $x$ and $y$ are random subsets of $[n]$ such that $\left|x\right|=\left|y\right|=\ell$ and $\left|x\cap y\right|=1$ . And with probability $1-\beta$ , $x$ and $y$ are random subsets of $[n]$ such that $\left|x\right|=\left|y\right|=\ell$ and $x\cap y=\emptyset$ . Razborov [48] proved that for $\beta=1/4$ , $D^{1/(400)}_{\tau_{1/4}}(\mbox{{$2$-DISJ}})=\Omega(n)$ . It is easy to extend this result to general $\beta$ and the average-case complexity.

Theorem 2 ([47], Lemma 2.2)

For any $\beta\leq 1/4$ , it holds that $\mathsf{ED}^{\beta/100}_{\tau_{\beta}}(\mbox{{$2$-DISJ}})=\Omega(n)$ , where the expectation is taken over the input distribution $\tau_{\beta}$ .

In the rest of the section, we simply write $\tau_{\beta}$ as $\tau$ .

3.3 The Complexity of $F_{0}$

3.3.1 Connecting $F_{0}$ and $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$

Set $\beta=1/(k\varepsilon^{2})$ , $B=20000/\delta$ , where $\delta$ is the small constant error parameter for $k$ -APPROX-SUM in Theorem 1.

We choose $f$ to be $2$ -DISJ with universe size $n=B/\varepsilon^{2}$ , set its input distribution to be $\tau$ , and work on $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$ . Let $\mu$ be the input distribution of $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$ , which is a function of $\tau$ (see Section 3.1 for the detailed construction of $\mu$ from $\tau$ ). Let $\{X_{1},\ldots,X_{k},Y\}\sim\mu$ . Let $Z_{i}=\text{{$2$-DISJ}}(X_{i},Y)$ . Let $\zeta$ be the induced distribution of $\mu$ on $\{X_{1},\ldots,X_{k}\}$ which we choose to be the input distribution for $F_{0}$ . In the rest of this section, for convenience, we will omit the subscripts $2$ -DISJ and $\tau$ in $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$ when there is no confusion.

Let $N=\sum_{i\in[k]}Z_{i}=\sum_{i\in[k]}\text{{$2$-DISJ}}(X_{i},Y)$ . Let $R=F_{0}(\cup_{i\in[k]}X_{i}\bigcap Y)$ . The following lemma shows that $R$ will concentrate around its expectation $\mathsf{E}[R]$ , which can be calculated exactly.

Lemma 6

With probability at least $(1-6500/B)$ , we have $\left|R-\mathsf{E}[R]\right|\leq 1/(10\varepsilon)$ , where $\mathsf{E}[R]=(1-\lambda)N$ for some fixed constant $0\leq\lambda\leq 4/B$ .

Proof:

We can think of our problem as a bin-ball game: Think each pair $(X_{i},Y)$ such that $2$ -DISJ $(X_{i},Y)=1$ are balls (thus we have $N$ balls), and elements in the set $Y$ are bins. Let $\ell=\left|Y\right|$ . We throw each of the $N$ balls into one of the $\ell$ bins uniformly at random. Our goal is to estimate the number of non-empty bins at the end of the process.

By a Chernoff bound, with probability $\left(1-e^{-\beta k/3}\right)\geq(1-100/B)$ , $N\leq 2\beta k=2/\varepsilon^{2}$ . By Fact $1$ and Lemma $1$ in [41], we have $\mathsf{E}[R]=\ell\left(1-(1-1/\ell)^{N}\right)$ and $\mathsf{Var}[R]<4N^{2}/\ell$ . Thus by Chebyshev’s inequality we have

\mathsf{Pr}[\left|R-\mathsf{E}[R]\right|>1/(10\varepsilon)]\leq\frac{\mathsf{Var}[R]}{1/(100\varepsilon^{2})}\leq\frac{6400}{B}.

Let $\theta=N/\ell\leq 8/B$ . We can write

\mathsf{E}[R]=\ell\left(1-e^{-\theta}\right)+O(1)=\theta\ell\left(1-\frac{\theta}{2!}+\frac{\theta^{2}}{3!}-\frac{\theta^{3}}{4!}+\right)+O(1).

This series converges and thus we can write $\mathsf{E}[R]=(1-\lambda)\theta\ell=(1-\lambda)N$ for some fixed constant $0\leq\lambda\leq\theta/2\leq 4/B$ .

The next lemma shows that we can use a protocol for $F_{0}$ to solve $k$ -APPROX-SUM with good properties.

Lemma 7

Any protocol $\cal P$ that computes a $(1+\gamma\varepsilon)$ -approximation to $F_{0}$ (for a sufficiently small constant $\gamma$ ) on input distribution $\zeta$ with error probability $\delta/2$ can be used to compute $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$ on input distribution $\mu$ with error probability $\delta$ .

Proof:

Given an input $\{X_{1},\ldots,X_{k},Y\}\sim\mu$ for $k$ -APPROX-SUM. The $k$ sites and the coordinator use $\cal P$ to compute $\tilde{W}$ which is a $(1+\gamma\varepsilon)$ -approximation to $F_{0}(X_{1},\ldots,X_{k})$ , and then determine the answer to $k$ -APPROX-SUM to be

$\displaystyle\frac{\tilde{W}-(n-\ell)}{1-\lambda}.$

Recall that $0\leq\lambda\leq 4/B$ is some fixed constant, $n=B/\varepsilon^{2}$ and $\ell=(n+1)/4$ .

Correctness.

Given a random input $(X_{1},\ldots,X_{k},Y)\sim\zeta$ , the exact value of $W=F_{0}(X_{1},\ldots,X_{k})$ can be written as the sum of two components.

W=Q+R,

(10)

where $Q$ counts $F_{0}(\cup_{i\in[k]}X_{i}\backslash Y)$ , and $R$ counts $F_{0}(\cup_{i\in[k]}X_{i}\bigcap Y)$ . First, from our construction it is easy to see by a Chernoff bound and the union bound that with probability $\left(1-1/\varepsilon^{2}\cdot e^{-\Omega(k)}\right)\geq 1-100/B$ , we have $Q=\left|\{[n]-Y\}\right|=n-\ell$ , since each element in $\mathcal{S}\backslash Y$ will be chosen by every $X_{i}\ (i=1,2,\ldots,k)$ with a probability at least $1/4$ . Second, by Lemma 6 we know that with probability $(1-6500/B)$ , $R$ is within $1/(10\varepsilon)$ from its mean $(1-\lambda)N$ for some fixed constant $0\leq\lambda\leq 4/B$ . Thus with probability $(1-6600/B)$ , we can write Equation (10) as

W=(n-\ell)+(1-\lambda)N+\kappa_{1},

(11)

for a value $\left|\kappa_{1}\right|\leq 1/(10\varepsilon)$ and $N\leq 2/\varepsilon^{2}$ .

Set $\gamma=1/(20B)$ . Since $F_{0}(X_{1},X_{2},\ldots,X_{k})$ computes a value $\tilde{W}$ which is a $(1+\gamma\varepsilon)$ -approximation of $W$ , we can substitute $W$ with $\tilde{W}$ in Equation (11), resulting in the following.

\tilde{W}=(n-\ell)+(1-\lambda)N+\kappa_{1}+\kappa_{2},

(12)

where $\left|\kappa_{1}\right|\leq 1/(10\varepsilon)$ , $N\leq 2/\varepsilon^{2}$ , and

$\displaystyle\kappa_{2}$	$\displaystyle\leq$	$\displaystyle\gamma\varepsilon\cdot W$
	$\displaystyle=$	$\displaystyle\gamma\varepsilon\cdot((n-\ell)+(1-\lambda)N+\kappa_{1})$
	$\displaystyle\leq$	$\displaystyle\gamma\varepsilon\cdot(B/\varepsilon^{2}+2/\varepsilon^{2}+1/(10\varepsilon))$
	$\displaystyle\leq$	$\displaystyle 1/(10\varepsilon).$

Now we have

	$\displaystyle N$	$\displaystyle=$	$\displaystyle(\tilde{W}-(n-\ell)-\kappa_{1}-\kappa_{2})/(1-\lambda)$
		$\displaystyle=$	$\displaystyle(\tilde{W}-(n-\ell))/(1-\lambda)+\kappa_{3},$

where $\left|\kappa_{3}\right|\leq(1/(10\varepsilon)+1/(10\varepsilon))/(1-4/B)\leq 1/(4\varepsilon)$ . Therefore $(\tilde{W}-(n-\ell))/(1-\lambda)$ approximates $N=\sum_{i\in[k]}Z_{i}$ correctly up to an additive error $1/(4\varepsilon)<\sqrt{\beta k}=1/\varepsilon$ , thus computes $k$ -APPROX-SUM correctly. The total error probability of this simulation is at most $(\delta/2+6600/B)$ , where the first term counts the error probability of $\cal P^{\prime}$ and the second term counts the error probability introduced by the reduction. This is less than $\delta$ if we choose $B=20000/\delta$ .

3.3.2 An Embedding Argument

Lemma 8

Suppose that there exists a deterministic protocol $\cal P^{\prime}$ which computes $(1+\gamma\varepsilon)$ -approximate $F_{0}$ (for a sufficiently small constant $\gamma$ ) on input distribution $\zeta$ with error probability $\delta/2$ (for a sufficiently small constant $\delta$ ) and communication $o(C)$ , then there exists a deterministic protocol $\cal P$ that computes $2$ -DISJ on input distribution $\tau$ with error probability $\beta/100$ and expected communication complexity $o(\log(1/\beta)\cdot C/k)$ , where the expectation is taken over the input distribution $\tau$ .

Proof:

In $2$ -DISJ, Alice holds $X$ and Bob holds $Y$ such that $(X,Y)\sim\tau$ . We show that Alice and Bob can use the deterministic protocol $\cal P^{\prime}$ to construct a deterministic protocol $\cal P$ for $2$ -DISJ $(X,Y)$ with desired error probability and communication complexity.

Alice and Bob first use $\cal P^{\prime}$ to construct a protocol $\cal P^{\prime\prime}$ . During the construction they will use public and private randomness which will be fixed at the end. $\cal P^{\prime\prime}$ consists of two phases.

Input reduction phase. Alice and Bob construct an input for $F_{0}$ using $X$ and $Y$ as follows: They pick a random site $S_{I}\ (I\in[k])$ using public randomness. Alice assigns $S_{I}$ with input $X_{I}=X$ , and Bob constructs inputs for the rest $(k-1)$ sites using $Y$ . For each $i\in[k]\backslash I$ , Bob samples an $X_{i}$ according to $\tau\ |\ Y$ using independent private randomness and assigns it to $S_{i}$ . Let $Z_{i}=\text{{$2$-DISJ}}(X_{i},Y)$ . Note that $\{X_{1},\ldots,X_{k},Y\}\sim\mu$ and $\{X_{1},\ldots,X_{k}\}\sim\zeta$ .

Simulation phase. Alice simulates $S_{I}$ and Bob simulates the rest $(k-1)$ sites, and they run protocol $\cal P^{\prime}$ on $\{X_{1},\ldots,X_{k}\}\sim\zeta$ to compute $F_{0}(X_{1},\ldots,X_{k})$ up to a $(1+\gamma\varepsilon)$ -approximation for a sufficiently small constant $\gamma$ and error probability $\delta/2$ . Let $\pi$ be the protocol transcript, and let $\tilde{W}$ be the output. By Lemma 7, we can use $\tilde{W}$ to compute $k$ -APPROX-SUM with error probability $\delta$ . And then by Theorem 1, for $0.96$ fraction of $Z=z$ over the input distribution $\mu$ and $\pi=\Pi(z)$ , it holds that for $0.9$ fraction of $\{i\in[k]\ |\ z_{i}=0\}$ , $q_{i}^{\pi}<\beta/c_{0}$ , and $0.9$ fraction of $\{i\in[k]\ |\ z_{i}=1\}$ , $q_{i}^{\pi}>\beta/c_{0}$ . Now $\cal P^{\prime\prime}$ outputs $1$ if $q_{I}^{\pi}>\beta/c_{0}$ , and $0$ otherwise. Since $S_{I}$ is chosen randomly among the $k$ sites, and the inputs for the $k$ sites are identically distributed, $\cal P^{\prime\prime}$ computes $Z_{I}=\text{{$2$-DISJ}}(X,Y)$ on input distribution $\tau$ correctly with probability $0.96\cdot 0.9\geq 0.8$ .

We now describe the final protocol $\cal P$ : Alice and Bob repeat $\cal P^{\prime\prime}$ independently for $c_{R}\log(1/\beta)$ times for a large enough constant $c_{R}$ . At the $j$ -th repetition, in the input reduction phase, they choose a random permutation $\sigma_{j}$ of $[n]$ using public randomness, and apply it to each element in $X_{1},\ldots,X_{k}$ before assigning them to the $k$ sites. After running $\cal P^{\prime\prime}$ for $c_{R}\log(1/\beta)$ times, $\cal P$ outputs the majority of the outcomes.

Since $Z_{I}=\text{{$2$-DISJ}(X, Y)}$ is fixed at each repetition, the inputs $\{X_{1},\ldots,X_{k}\}$ at each repetition have a small dependence, but conditioned on $Z_{I}$ , they are all independent. Let $\mu^{\prime}$ to be input distribution of $\{X_{1},\ldots,X_{k},Y\}$ conditioned on $Z_{I}=b$ . Let $\zeta^{\prime}$ be the induced distribution of $\mu^{\prime}$ on $\{X_{1},\ldots,X_{k}\}$ . The successful probability of a run of $\cal P^{\prime\prime}$ on $\zeta^{\prime}$ is at least $0.8-\mathsf{TV}(\zeta,\zeta^{\prime})$ , where $\mathsf{TV}(\zeta,\zeta^{\prime})$ is the total variation distance between distributions $\zeta,\zeta^{\prime}$ , which is at most

\max\{\mathsf{TV}(\text{Binomial}(k,\beta),\text{Binomial}(k-1,\beta)),\mathsf{TV}(\text{Binomial}(k,\beta),\text{Binomial}(k-1,\beta)+1)\},

and can be bounded by $O(1/\sqrt{\beta k})=O(\varepsilon)$ (see, e.g., Fact 2.4 of [30]). Since conditioned on $Z_{I}$ , the inputs at each repetition are independent, and the success probability of each run of $\cal P^{\prime\prime}$ is at least $0.7$ , by a Chernoff bound over the $c_{R}\log(1/\beta)$ repetitions for a sufficiently large $c_{R}$ , we conclude that $\cal P$ succeeds with error probability $\beta/1600$ .

We next consider the communication complexity. At each run of $\cal P^{\prime\prime}$ , let $\mathsf{CC}(S_{I},S_{-I})$ be the expected communication cost between the site $S_{I}$ and the rest players (more precisely, between $S_{I}$ and the coordinator, since in the coordinator model all sites only talk to the coordinator, whose initial input is $\emptyset$ ), where the expectation is taken over the input distribution $\zeta$ and the choice of the random $I\in[k]$ . Since conditioned on $Y$ , all $X_{i}\ (i\in[k])$ are independent and identically distributed, if we take a random site $S_{I}$ , the expected communication between $S_{I}$ and the coordinator should be equal to the total communication divided by a factor of $k$ . Thus we have $\mathsf{CC}(S_{I},S_{-I})=o(C/k)$ . Finally, by the linearity of expectation, the expected total communication cost of the $O(\log(1/\beta))$ runs of $\cal P^{\prime\prime}$ is $o(\log(1/\beta)\cdot C/k)$ .

At the end we fix all the randomness used in construction of protocol $\cal P$ . We first use two Markov inequalities to fix all public randomness such that $\cal P$ succeeds with error probability $\beta/400$ , and the expected total communication cost of the $o(\log(1/\beta)C/k)$ , where both the error probability and the cost expectation are taken over the input distribution $\mu$ and Bob’s private randomness. We next use another two Markov inequalities to fix Bob’s private randomness such that $\cal P$ succeeds with error probability $\beta/100$ , and the expected total communication cost of the $o(\log(1/\beta)C/k)$ , where both the error probability and the cost expectation are taken over the input distribution $\mu$ .

The following theorem is a direct consequence of Lemma 8, Theorem 2 for $2$ -DISJ and Lemma 1 (Yao’s Lemma). Recall that we set $n=O(1/\varepsilon^{2})$ and $1/\beta=\varepsilon^{2}k$ . In the definition for $k$ -APPROX-SUM we need $c_{\beta}/k\leq\beta\leq 1/c_{\beta}$ for a sufficiently large constant $c_{\beta}$ , thus we require $c_{\beta}\leq 1/\varepsilon^{2}\leq k/c_{\beta}$ for a sufficiently large constant $c_{\beta}$ .

Theorem 3

Assuming that $c_{\beta}\leq 1/\varepsilon^{2}\leq k/c_{\beta}$ for a sufficiently large constant $c_{\beta}$ . Any randomised protocol that computes a $(1+\varepsilon)$ -approximation to $F_{0}$ with error probability $\delta$ (for a sufficiently small constant $\delta$ ) has communication complexity $\Omega\left(\frac{k}{\varepsilon^{2}\log(\varepsilon^{2}k)}\right)$ .

4 A Lower Bound for $F_{p}\ (p>1)$

We first introduce a problem called $k$ -XOR which can be considered to some extent as a combination of two $k$ -DISJ (introduced in [3, 7]) instances, and then compose it with $2$ -GAP-ORT (introduced in [49]) to create another problem that we call the $k$ -BLOCK-THRESH-XOR ( $k$ -BTX) problem. We prove that the communication complexity of $k$ -BTX is large. Finally, we prove a communication complexity lower bound for $F_{p}$ by performing a reduction from $k$ -BTX. In this section we work in the blackboard model.

4.1 The $2$ -GAP-ORT Problem

In the $2$ -GAP-ORT problem we have two players Alice and Bob. Alice has a vector $x=\{x_{1},\ldots,x_{1/\varepsilon^{2}}\}\in\{0,1\}^{1/\varepsilon^{2}}$ and Bob has a vector $y=\{y_{1},\ldots,y_{1/\varepsilon^{2}}\}\in\{0,1\}^{1/\varepsilon^{2}}$ . They want to compute

\displaystyle\begin{array}[]{l}2\textrm{-GAP-ORT}(x,y)=\left\{\begin{array}[]{rl}1,&\left|\sum\limits_{i\in[1/\varepsilon^{2}]}\textrm{XOR}(x_{i},y_{i})-\frac{1}{2\varepsilon^{2}}\right|\geq\frac{2}{\varepsilon},\\ 0,&\left|\sum\limits_{i\in[1/\varepsilon^{2}]}\textrm{XOR}(x_{i},y_{i})-\frac{1}{2\varepsilon^{2}}\right|\leq\frac{1}{\varepsilon},\\ ,&\text{otherwise.}\end{array}\right.\end{array}

Let $\phi$ be the uniform distribution on $\{0,1\}^{1/\varepsilon^{2}}\times\{0,1\}^{1/\varepsilon^{2}}$ and let $(X,Y)$ be a random input chosen from distribution $\phi$ . The following theorem is recently obtained by Chakrabarti et al. [15]. ⁵⁵5In our original conference paper, which was published before [15], we proved the theorem for protocols with $\mathrm{poly}(N)$ communication, using a simple argument based on [49] and Theorem 1.3 of [8].

Theorem 4 ([15])

Let $\Pi$ be the transcript of any randomized protocol for $2$ -GAP-ORT on input distribution $\phi$ with error probability $\iota$ , for a sufficiently small constant $\iota>0$ . Then, $I(X,Y;\Pi)\geq{\Omega}(1/\varepsilon^{2})$ .

4.2 The $k$ -XOR Problem

In the $k$ -XOR problem we have $k$ sites $S_{1},\ldots,S_{k}$ . Each site $S_{i}\ (i=1,2,\ldots,k)$ holds a block ${b}_{i}=\{b_{i,1},\ldots,b_{i,n}\}$ of $n$ bits. Let $b=(b_{1},\ldots,b_{k})$ be the inputs of $k$ sites. Let $b_{[k],\ell}$ be the $k$ sites’ inputs on the $\ell$ -th coordinate. W.l.o.g., we assume $k$ is a power of $2$ . The $k$ sites want to compute the following function.

\displaystyle\begin{array}[]{l}k\textrm{-XOR}({b}_{1},\ldots,{b}_{k})=\left\{\begin{array}[]{rl}1,&\text{if $\exists$ $j\in[n]$ such that $b_{i,j}=1$}\\ &\quad\text{for exactly $k/2$ $i$'s},\\ 0,&\text{otherwise.}\end{array}\right.\end{array}

We define the input distribution $\varphi_{n}$ for the $k$ -XOR problem as follows. For each coordinate $\ell\ (\ell\in[n])$ there is a variable $D_{\ell}$ chosen uniformly at random from $\{1,2,\ldots,k\}$ . Conditioned on $D_{\ell}$ , all but the $D_{\ell}$ -th sites set their inputs to $0$ , whereas the $D_{\ell}$ -th site sets its input to $0$ or $1$ with equal probability. We call the $D_{\ell}$ -th site the special site in the $\ell$ -th coordinate. Let $\varphi_{1}$ denote this input distribution on one coordinate.

Next, we choose a random special coordinate $M\in[n]$ and replace the $k$ sites’ inputs on the $M$ -th coordinate as follows: For the first $k/2$ sites, with probability $1/2$ we replace all $k/2$ sites’ inputs with $0$ , and with probability $1/2$ we replace all $k/2$ sites’ inputs with $1$ ; and we independently perform the same operation to the second $k/2$ sites. Let $\psi_{1}$ denote the distribution on this special coordinate. And let $\psi_{n}$ denote the input distribution that on the special coordinate $M$ is distributed as $\psi_{1}$ and on each of the remaining $n-1$ coordinates is distributed as $\varphi_{1}$ .

Let ${B},B_{i},B_{[k],\ell},B_{i,\ell}$ be the corresponding random variables of ${b},b_{i},b_{[k],\ell},b_{i,\ell}$ when the input of $k$ -XOR is chosen according to the distribution $\psi_{n}$ . Let $D=\{D_{1},\ldots,D_{n}\}$ .

4.3 The $k$ -GUESS Problem

The $k$ -GUESS problem can be seen as an augmentation of the $k$ -XOR problem. The $k$ sites are still given an input $B$ , as that in the $k$ -XOR problem. In addition, we introduce another player called the predictor. The predictor will be given an input $Z$ , but it cannot talk to any of the $k$ sites (that is, it cannot write anything to the blackboard). After the $k$ sites finish the whole communication, the predictor computes the final output $g(\Pi(B),Z)$ , where $\Pi(B)$ is the transcript of the $k$ sites’ communication on their input $B$ , and $g$ is the (deterministic) maximum likelihood function (see Proposition 1). In this section when we talk about protocol transcripts, we always mean the concatenation of the messages exchanged by the $k$ sites, but excluding the output of the predictor.

In the $k$ -GUESS problem, the goal is for the predictor to output $(X,Y)$ , where $X=1$ if the inputs of the first $k/2$ sites in the special coordinate $M$ are all $1$ and $X=0$ otherwise, and $Y=1$ if the inputs of the second $k/2$ sites in the special coordinate $M$ are all $1$ and $Y=0$ otherwise. We say the instance $B$ is a $00$ -instance if $X=Y=0$ , a $10$ -instance if $X=1$ and $Y=0$ , a $01$ -instance if $X=0$ and $Y=1$ , and a $11$ -instance if $X=Y=1$ . Let $S\in\{00,01,10,11\}$ be the type of an instance.

We define the following input distribution for $k$ -GUESS: We assign an input $B\sim\psi_{n}$ to the $k$ sites, and $Z=\{D,M\}$ to the predictor, where $D,M$ are those used to construct the $B$ in the distribution $\psi_{n}$ . Slightly abusing notation, we also use $\psi_{n}$ to denote the joint distribution of $B$ and $\{D,M\}$ . We do the same for the one coordinate distributions $\psi_{1}$ and $\varphi_{1}$ . That is, we also use $\psi_{1}$ (or $\varphi_{1}$ ) to denote the joint distribution of $B_{[k],\ell}$ and $D_{\ell}$ for a single coordinate $\ell$ .

Theorem 5

Let $\Pi$ be the transcript of any randomized protocol for $k$ -GUESS on input distribution $\psi_{n}$ with success probability $1/4+\Omega(1)$ . Then we have $I(B;\Pi\ |\ D,M)={\Omega}(n/k)$ , where information is measured ⁶⁶6When we say that the information is measured with respect to a distribution $\mu$ we mean that the inputs to the protocol are distributed according to $\mu$ when computing the mutual information (note that there is also randomness used by $\Pi$ when measuring the mutual information). with respect to the input distribution $\psi_{n}$ .

Proof:

By a Markov inequality, we know that for $\kappa_{1}=\Omega(n)$ of $\ell\in[n]$ , the protocol succeeds with probability $1/4+\Omega(1)$ conditioned on $M=\ell$ . Call an $\ell$ for which this holds eligible. Let $\kappa=n-\kappa_{1}/2$ . We say an $\ell$ is good if $\ell$ is both eligible and $\ell\leq\kappa$ . Thus there are $\kappa_{1}+\kappa-n=\Omega(n)$ good $\ell$ . Let $D_{-\ell}$ denote the random variable $D$ with $\ell$ -th component missing. We say a $d$ is nice for a good $\ell$ if the protocol succeeds with probability $1/4+\Omega(1)$ conditioned on $M=\ell$ and $D_{-\ell}=d$ . By another Markov inequality, it holds that at least an $\Omega(1)$ fraction of $d$ is nice for a good $\ell$ .

Now we consider $I(B;\Pi\ |\ D,M,S=00,M>\kappa)$ . Note that if we can show that $I(B;\Pi\ |\ D,M,S=00,M>\kappa)=\Omega(n/k)$ , then it follows that $I(B;\Pi\ |\ D,M)=\Omega(n/k)$ , since $H(S)=2$ , and $\mathsf{Pr}[(S=00)\wedge(M>\kappa)]=1/4\cdot{\Omega}(1)={\Omega}(1)$ . By the chain rule, expanding the conditioning, and letting $B_{[k],<\ell}$ be the inputs to the $k$ sites on the first $\ell-1$ coordinates, we have

	$\displaystyle I(B;\Pi\ \|\ D,M,S=00,M>\kappa)$	(17)
$\displaystyle=$	$\displaystyle\sum_{\ell\in[n]}I(B_{[k],\ell};\Pi\ \|\ D,M,S=00,M>\kappa,B_{[k],<\ell})$	(17)
$\displaystyle=$	$\displaystyle\sum_{\ell\in[n]}I(B_{[k],\ell};\Pi\ \|\ D,M,S=00,M>\kappa)$
$\displaystyle\geq$	$\displaystyle\sum_{\textrm{good }\ell}I(B_{[k],\ell};\Pi\ \|\ D,M,S=00,M>\kappa)$
$\displaystyle=$	$\displaystyle\sum_{\textrm{good }\ell}\sum_{d}\mathsf{Pr}[D_{-\ell}=d]\cdot I(B_{[k],\ell};\Pi\ \|\ D_{\ell},M,S=00,M>\kappa,D_{-\ell}=d)$
$\displaystyle\geq$	$\displaystyle\sum_{\textrm{good }\ell}\sum_{\text{nice $d$ for $\ell$}}\mathsf{Pr}[D_{-\ell}=d]\cdot I(B_{[k],\ell};\Pi\ \|\ D_{\ell},M,S=00,M>\kappa,D_{-\ell}=d),$	(19)

where (17) to (Proof: ) is because $B_{[k],<\ell}$ is independent of $B_{[k],\ell}$ given other conditions, and we apply item $3$ in Proposition 1.

Now let’s focus on a good $\ell$ and a nice $d$ for $\ell$ . We define a protocol $\Pi_{\ell,d}$ which on input $(A_{1},\ldots,A_{k},R)\sim\psi_{1}$ , attempts to output $(U,V)$ , where $U=1$ if $A_{1}=\ldots=A_{k/2}=1$ and $U=0$ otherwise, and $V=1$ if $A_{k/2+1}=\ldots=A_{k}=1$ and $V=0$ otherwise. Here $A_{1},\ldots,A_{k}$ are inputs of the $k$ sites and $R$ is the input of the predictor. The protocol $\Pi_{\ell,d}$ has $(\ell,d)$ hardwired into it, and works as follows. First, the $k$ sites construct an input $B$ for $k$ -GUESS distributed according to $\psi_{n}$ , using $\{A_{1},\ldots,A_{k}\},d$ and their private randomness, without any communication: They set the input on the $\ell$ -th coordinate to be $B_{[k],\ell}=\{A_{1},\ldots,A_{k}\}$ , and use their private randomness to sample the inputs for coordinates $\ell^{\prime}\neq\ell$ using the value $d$ and the fact that the inputs to the $k$ sites are independent conditioned on $D_{-\ell}=d$ . The predictor sets its input to be $\{D_{\ell}=R,D_{-\ell}=d,M=\ell\}$ . Next, the $k$ sites run $\Pi$ on their input $B$ . Finally, the predictor outputs $(U,V)=g(\Pi(B),R,d,\ell)$ .

Let $\psi_{n,00,d}=(\psi_{n}|S=00,D_{-\ell}=d)$ . Let $\varphi_{n,d}=(\varphi_{n}|D_{-\ell}=d)$ . Let $\psi_{n,00,d}^{\ell}$ and $\varphi_{n,d}^{\ell}$ be the distributions $\psi_{n,00,d}$ and $\varphi_{n,d}$ of $B$ after embedding $(A_{1},\ldots,A_{k})$ to the $\ell$ -th coordinate conditioned on $M>\kappa$ (recall that $\kappa\geq\ell$ for a good $\ell$ , thus $M>\ell$ ), respectively. Since $\ell$ is good and $d$ is nice for $\ell$ , it follows that

	$\displaystyle\mathsf{Pr}[\Pi_{\ell,d}(A_{1},\ldots,A_{k},R)=(U,V)\ \|\ S=00,M>\kappa]$	$\displaystyle=$	$\displaystyle 1/4+{\Omega}(1)-\mathsf{TV}(\psi_{n,00,d}^{\ell},\varphi_{n,d}^{\ell})$
		$\displaystyle\geq$	$\displaystyle 1/4+{\Omega}(1),$

where the probability is taken over $(A_{1},\ldots,A_{k},R)\sim\psi_{1}$ , and $\mathsf{TV}(\psi_{n,00,d}^{\ell},\varphi_{n,d}^{\ell})$ is the total variation distance between distributions $\psi_{n,00,d}^{\ell}$ and $\varphi_{n,d}^{\ell}$ , which can be bounded by $O(1/\sqrt{n-\kappa})=O(1/\sqrt{n})$ . The proof will be given shortly, and here is where we use that $\kappa=n-\Omega(n)$ .

Hence, for a good $\ell$ , and for a nice $d$ for $\ell$ , we have

\displaystyle I(B_{[k],\ell};\Pi\ |\ D_{\ell},M,S=00,M>\kappa,D_{-\ell}=d)\geq I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R,M,S=00,M>\kappa),

(20)

where $\Pi^{\prime}$ is the protocol that minimizes the information cost when the information (on the right side of (20)) is measured with respect to the marginal distribution of $\psi_{n}$ on a good coordinate $\ell$ , and succeeds in outputting $(U,V)$ with probability $1/4+{\Omega}(1)$ when the $k$ sites and the predictor get input $\{A_{1},\ldots,A_{k},R\}\sim\psi_{1}$ , The information on the left side of (20) is measured with respect to $\psi_{n}$ .

Combining (19) and (20), and given that we have $\Omega(n)$ good $\ell$ , as well as at least an $\Omega(1)$ fraction of $d$ that are nice for any good $\ell$ , we have

\displaystyle I(B;\Pi\ |\ D,M,S=00,M>\kappa)\geq{\Omega}(n)\cdot I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R,M,S=00,M>\kappa).

(21)

Now we analyze $\mathsf{TV}(\psi_{n,00,d}^{\ell},\varphi_{n,d}^{\ell})$ . First note that we can just focus on coordinates $\ell^{\prime}>\ell$ , since the distributions $\psi_{n,00,d}^{\ell}$ and $\varphi_{n,d}^{\ell}$ are the same on coordinates $\ell^{\prime}\leq\ell$ . Let $\varsigma_{1}$ and $\varsigma_{2}$ be the distribution of $\psi_{n,00,d}^{\ell}$ and $\varphi_{n,d}^{\ell}$ on coordinates $\{\ell^{\prime}\ |\ \ell^{\prime}>\ell,\ell^{\prime}\in[n]\}$ , respectively. Observe that $\varsigma_{2}$ can be thought as a binomial distribution: for each coordinate $\ell^{\prime}>\ell$ , we set $B_{D_{\ell^{\prime}},\ell^{\prime}}$ randomly to be $0$ or $1$ with equal probability. The remaining $B_{i,\ell^{\prime}}\ (i\neq D_{\ell^{\prime}})$ are all set to be $0$ . Moreover, $\varsigma_{1}$ can be thought in the following way: we first sample according to $\varsigma_{2}$ , and then randomly choose a coordinate $M>\kappa\geq\ell$ and reset $B_{D_{M},M}=0$ . Since $M$ is random, the total variation distance between $\varsigma_{1}$ and $\varsigma_{2}$ is the total variation distance between Binomial $(n-\ell,1/2)$ and Binomial $(n-\ell-1,1/2)$ (that is, by symmetry, only the number of $1$ ’s in $\{B_{D_{\ell^{\prime}},\ell^{\prime}}\ |\ \ell^{\prime}>\ell\}$ matters), which is at most $O(1/\sqrt{n-\ell})\leq O(1/\sqrt{n-\kappa})$ (see, e.g., Fact 2.4 of [30]).

Let $\mathcal{E}$ be the event that all sites have the value $0$ in the $M$ -th coordinate when the inputs are drawn from $\varphi_{n}$ . Observe that $(\varphi_{n}|\mathcal{E})=(\psi_{n}|S=00)$ , thus

I(B;\Pi\ |\ D,M,S=00,M>\kappa)\geq{\Omega}(n)\cdot I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R,M,M>\kappa,\mathcal{E}),

where the information on the left hand side is measured with respect to inputs $(B,\{D,M\})$ drawn from $\psi_{n}$ , and the information on the right hand side is measured with respect to inputs $(A_{1},\ldots,A_{k},R)$ drawn from the marginal distribution of $\varphi_{n}$ on a good coordinate $\ell$ , which is equivalent to $\varphi_{1}$ since $M>\kappa\geq\ell$ . By the third item of Proposition 1, and using that $A_{1},\ldots,A_{k}$ are independent of $\mathcal{E}$ given $M>\kappa\geq\ell$ and $R$ , we obtain

\displaystyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R,M,M>\kappa,\mathcal{E})\geq I(A_{1},\ldots,A_{k}\ ;\Pi^{\prime}\ |\ R,M,M>\kappa).

Finally, since $M,M>\kappa$ are independent of $A_{1},\ldots,A_{k}$ and $R$ , it holds that $I(A_{1},\ldots,A_{k}\ ;\Pi^{\prime}\ |\ R,M,M>\kappa)\geq I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R)$ , where the information is measured with respect to the input distribution $\varphi_{1}$ , and $\Pi^{\prime}$ is a protocol which succeeds with probability $1/4+{\Omega}(1)$ on $\psi_{1}$ .

It remains to show that $I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R)={\Omega}(1/k)$ , where the information is measured with respect to $\varphi_{1}$ , and the correctness is measured with respect to $\psi_{1}$ . Let $\mathbf{0}$ be the all- $0$ vector, $\mathbf{1}$ be the all- $1$ vector and $\mathbf{e_{i}}$ be the standard basis vector with the $i$ -th coordinate being $1$ . By the relationship between mutual information and Hellinger distance (see Proposition 2.51 and Proposition 2.53 of [6]), we have

	$\displaystyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ \|\ R)$	$\displaystyle=$	$\displaystyle(1/k)\cdot\sum_{i\in[k]}I(A_{1},\ldots,A_{k};\Pi^{\prime}\ \|\ R=i)$
		$\displaystyle=$	$\displaystyle\textstyle\Omega(1/k)\cdot\sum_{i\in[k]}h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{e_{i}})),$

where $h(\cdot,\cdot)$ is the Hellinger distance (see Section 2 for a definition). Now we assume $k$ and $k/2$ are powers of $2$ , and we use Theorem $7$ of [39], which says that the following three statements hold:

1.

$\sum_{i\in[k]}h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{e_{i}}))=\Omega(1)\cdot h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))$ .
2.

$\sum_{i\in[k]}h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{e_{i}}))=\Omega(1)\cdot h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2}))$ .
3.

$\sum_{i\in[k]}h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{e_{i}}))=\Omega(1)\cdot h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))$ .

It follows that

	$\displaystyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ \|\ R)$	$\displaystyle=$	$\displaystyle\Omega(1/k)\cdot\left(h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))\right.$
			$\displaystyle\left.+h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2}))+h^{2}(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))\right).$

By the Cauchy-Schwartz inequality we have,

	$\displaystyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ \|\ R)$	$\displaystyle=$	$\displaystyle\Omega(1/k)\cdot\left(h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))\right.$
			$\displaystyle\left.+h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2}))+h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))\right)^{2}.$

We can rewrite this as (by changing the constant in the $\Omega(1/k)$ ):

	$\displaystyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ \|\ R)$	$\displaystyle=$	$\displaystyle\Omega(1/k)\cdot\left(3h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))\right.$
			$\displaystyle\left.+3h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2}))+3h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))\right)^{2}.$

By the triangle inequality of the Hellinger distance, we get

1.

$h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))+h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))\geq h(\Pi^{\prime}(\mathbf{1}),\Pi^{\prime}(1^{k/2}0^{k/2}))$ ,
2.

$h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(\mathbf{1}))+h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2}))\geq h(\Pi^{\prime}(\mathbf{1}),\Pi^{\prime}(0^{k/2}1^{k/2}))$ ,
3.

$h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(0^{k/2}1^{k/2})+h(\Pi^{\prime}(\mathbf{0}),\Pi^{\prime}(1^{k/2}0^{k/2}))\geq h(\Pi^{\prime}(0^{k/2}1^{k/2}),\Pi^{\prime}(1^{k/2}0^{k/2}))$ .

Thus we have

\displaystyle\textstyle I(A_{1},\ldots,A_{k};\Pi^{\prime}\ |\ R)

\displaystyle=

\displaystyle\textstyle\Omega(1/k)\cdot\left(\sum_{a,b\in\{\mathbf{0},\ \mathbf{1},\ 1^{k/2}0^{k/2},\ 0^{k/2}1^{k/2}\}}h(\Pi^{\prime}(a),\Pi^{\prime}(b))\right)^{2}.

The claim is that at least one of $h(\Pi^{\prime}(a),\Pi^{\prime}(b))$ in the RHS in Equation (Proof: ) is ${\Omega}(1)$ , and this will complete the proof. By Proposition 3, this is true if the total variation distance $\mathsf{TV}(\Pi^{\prime}(a),\Pi^{\prime}(b))={\Omega}(1)$ for a pair $(a,b)\in\{\mathbf{0},\ \mathbf{1},\ 1^{k/2}0^{k/2},\ 0^{k/2}1^{k/2}\}^{2}$ . We show that there must be such a pair $(a,b)$ , for the following reasons.

For a $z\in\{\mathbf{0},\ \mathbf{1},\ 1^{k/2}0^{k/2},\ 0^{k/2}1^{k/2}\}$ , let $\chi(z)=xy$ if $z=x^{k/2}y^{k/2}$ . First, there must exist a pair $(a,b)$ such that

\displaystyle\mathsf{TV}(g(\Pi^{\prime}(a),R),g(\Pi^{\prime}(b),R))=\Omega(1),

(22)

since otherwise, if $\mathsf{TV}(g(\Pi^{\prime}(a),R),g(\Pi^{\prime}(b),R))=o(1)$ for all $(a,b)\in\{\mathbf{0},\ \mathbf{1},\ 1^{k/2}0^{k/2},\ 0^{k/2}1^{k/2}\}^{2}$ , then by Proposition 4, for any pair $(a,b)$ with $a\neq b$ , and a $c$ chosen from $\{a,b\}$ uniformly at random, it holds that $\mathsf{Pr}[g(\Pi^{\prime}(c),R)=\chi(c)]\leq 1/2+o(1)$ , where the probability is taken over the distribution of $c$ . Consequently, for a $c$ chosen from $\{\mathbf{0},\ \mathbf{1},\ 1^{k/2}0^{k/2},\ 0^{k/2}1^{k/2}\}$ uniformly at random, it holds that $\mathsf{Pr}[g(\Pi^{\prime}(c),R)=\chi(c)]\leq 1/4+o(1)$ , violating the protocol’s success probability guarantee. Second, since $g$ is a deterministic function, and $R$ is independent of $A_{1},\ldots,A_{k}$ when $(A_{1},\ldots,A_{k},R)\sim\psi_{1}$ , we have

\displaystyle\mathsf{TV}(\Pi^{\prime}(a),\Pi^{\prime}(b))\geq\mathsf{TV}(g(\Pi^{\prime}(a),R),g(\Pi^{\prime}(b),R)).

(23)

The claim follows from (22) and (23).

4.4 The $k$ -BTX Problem

The input of the $k$ -BTX problem is a concatenation of $1/\varepsilon^{2}$ copies of inputs of the $k$ -XOR problem. That is, each site $S_{i}\ (i=1,2,\ldots,k)$ holds an input consisting of $1/\varepsilon^{2}$ blocks each of which is an input for a site in the $k$ -XOR problem. More precisely, each $S_{i}\ (i\in[k])$ holds an input ${b_{i}}=({b}_{i}^{1},\ldots,{b}_{i}^{1/\varepsilon^{2}})$ where ${b}_{i}^{j}=(b_{i,1}^{j},\ldots,b_{i,n}^{j})\ (j\in[1/\varepsilon^{2}])$ is a vector of $n$ bits. Let $b^{j}=(b^{j}_{1},\ldots,b^{j}_{k})$ be the list of inputs to the $k$ sites in the $j$ -th block. Let ${b}=({b_{1}},\ldots,{b_{k}})$ be the list of inputs to the $k$ sites. In the $k$ -BTX problem the $k$ sites want to compute the following.

\displaystyle\begin{array}[]{l}k\textrm{-BTX}({b}_{1},\ldots,{b}_{k})=\left\{\begin{array}[]{rl}1,&\text{if }\left|\sum\limits_{j\in[1/\varepsilon^{2}]}k\textrm{-XOR}({b}_{1}^{j},\ldots,{b}_{k}^{j})-\frac{1}{2\varepsilon^{2}}\right|\geq 2/\varepsilon,\\ 0,&\text{if }\left|\sum\limits_{j\in[1/\varepsilon^{2}]}k\textrm{-XOR}({b}_{1}^{j},\ldots,{b}_{k}^{j})-\frac{1}{2\varepsilon^{2}}\right|\leq 1/\varepsilon,\\ ,&\text{otherwise}.\end{array}\right.\end{array}

We define the input distribution $\nu$ for the $k$ -BTX problem as follows: The input of the $k$ sites in each block is chosen independently according to the input distribution $\psi_{n}$ , which is defined for the $k$ -XOR problem. Let ${B},{B}_{i},B^{j},{B}_{i}^{j},B_{i,\ell}^{j}$ be the corresponding random variables of ${b},{b}_{i},b^{j},{b}_{i}^{j},b_{i,\ell}^{j}$ when the input of $k$ -BTX is chosen according to the distribution $\nu$ . Let ${D}^{j}=({D}_{1}^{j},\ldots,{D}_{n}^{j})$ where ${D}_{\ell}^{j}\ (\ell\in[n],j\in[1/\varepsilon^{2}])$ is the special site in the $\ell$ -th coordinate of block $j$ , and let ${D}=({D}^{1},\ldots,{D}^{1/\varepsilon^{2}})$ . Let ${M}=(M^{1},\ldots,M^{1/\varepsilon^{2}})$ where $M^{j}$ is the special coordinate in block $j$ . Let $S=(S^{1},\ldots,S^{1/\varepsilon^{2}})$ where $S^{j}\in\{00,01,10,11\}$ is the type of the $k$ -XOR instance in block $j$ .

For each block $j\ (j\in[1/\varepsilon^{2}])$ , let $X^{j}=1$ if the inputs of the first $k/2$ sites in the special coordinate $M^{j}$ are all $1$ and $X^{j}=0$ otherwise; and similarly let $Y^{j}=1$ if the inputs of the second $k/2$ sites in the coordinate $M^{j}$ are all $1$ and $Y^{j}=0$ otherwise. Let $X=(X^{1},\ldots,X^{1/\varepsilon^{2}})$ and $Y=(Y^{1},\ldots,Y^{1/\varepsilon^{2}})$ .

Linking $k$ -BTX to $2$ -GAP-ORT.

We show that Alice and Bob, who are given $(X,Y)\sim\phi$ , can construct a $2$ -player protocol $\Pi^{\prime}$ for $2$ -GAP-ORT $(X,Y)$ using a protocol $\Pi$ for $k$ -BTX.

They first construct an input for $k$ -BTX using $(X,Y)$ . Alice simulates the first $k/2$ players, and Bob simulates the second $k/2$ players. Alice and Bob use the public randomness to generate $M^{j}$ and $D^{j}$ for each $j\in[1/\varepsilon^{2}]$ . For each $j\in[1/\varepsilon^{2}]$ , Alice sets the $M^{j}$ -th coordinate of each of the first $k/2$ players to $X^{j}$ . Similarly, Bob sets the $M^{j}$ -th coordinate of each of the last $k/2$ players to $Y^{j}$ . Alice and Bob then use private randomness and the $D^{j}$ vectors to fill in the remaining coordinates. Observe that the resulting inputs $B$ (for $k$ -BTX) is distributed according to $\nu$ .

Alice and Bob then run the protocol $\Pi$ on $B$ . Every time a message is sent between any two of the $k$ players in $\Pi$ , it is appended to the transcript. That is, if the two players are among the first $k/2$ , Alice still forwards this message to Bob. If the two players are among the last $k/2$ , Bob still forwards this message to Alice. If the message is between a player in the first group and the second group, Alice and Bob exchange a message. The output of $\Pi^{\prime}$ is equal to that of $\Pi$ .

Theorem 6

Let $\Pi$ be the transcript of any randomized protocol for $k$ -BTX on input distribution $\nu$ with error probability $\delta$ for a sufficiently small constant $\delta$ . Then $I(X,Y;\Pi\ |\ M,D)={\Omega}(1/\varepsilon^{2})$ , where the information is measured with respect to the uniform distribution on $X,Y$ .

Proof:

By a Markov inequality, we have that for at least $1/2$ fraction of choices of $(M,D)=(m,d)$ , the $k$ -party protocol $\Pi$ computes $k$ -BTX with error probability at most $2\delta$ . Say such a pair $(m,d)$ good. According to our reduction, we have that the transcript of $\Pi$ is equal to the transcript of $\Pi^{\prime}$ and the output of $\Pi^{\prime}$ is the same as the output of that of $\Pi$ . Hence, for a good pair $(m,d)$ , the $2$ -party protocol $\Pi^{\prime}$ computes $2$ -GAP-ORT with error probability at most $2\delta$ on distribution $\phi$ . We have

$\displaystyle I(X,Y;\Pi\ \|\ M,D)$	$\displaystyle=$	$\displaystyle\sum_{(m,d)}\mathsf{Pr}[(M,D)=(m,d)]\left[I(X,Y;\Pi\ \|\ (M,D)=(m,d))\right]$
	$\displaystyle\geq$	$\displaystyle\sum_{\text{good }(m,d)}\mathsf{Pr}[(M,D)=(m,d)]\left[I(X,Y;\Pi\ \|\ (M,D)=(m,d))\right]$
	$\displaystyle=$	$\displaystyle\sum_{\text{good }(m,d)}\mathsf{Pr}[(M,D)=(m,d)]\left[I(X,Y;\Pi^{\prime}\ \|\ (M,D)=(m,d))\right]$
	$\displaystyle\geq$	$\displaystyle 1/2\cdot{\Omega}(1/\varepsilon^{2})\geq{\Omega}(1/\varepsilon^{2}).\quad(\text{By Theorem~\ref{thm:gap}})$

Now we are ready to prove our main theorem for $k$ -BTX.

Theorem 7

Let $\Pi$ be the transcript of any randomized protocol for $k$ -BTX on input distribution $\nu$ with error probability $\delta$ for a sufficiently small constant $\delta$ . We have $I(B;\Pi\ |\ M,D)\geq{\Omega}(n/(k\varepsilon^{2}))$ , where the information is measured with respect to the input distribution $\nu$ .

Proof:

By Theorem 6 we have $I(X,Y;\Pi\ |\ M,D)={\Omega}(1/\varepsilon^{2})$ . Using the chain rule and a Markov inequality, it holds that

I(X^{j},Y^{j};\Pi\ |\ M,D,X^{<j},Y^{<j})={\Omega}(1)

for at least ${\Omega}(1/\varepsilon^{2})$ of $j\in[1/\varepsilon^{2}]$ , where $X^{<j}=\{X^{1},\ldots,X^{j-1}\}$ and similarly for $Y^{<j}$ . We say such a $j$ for which this holds is good.

Now we consider a good $j\in[1/\varepsilon^{2}]$ , and show that

I(B^{j};\Pi\ |\ M,D,B^{<j})={\Omega}(n/k).

Since $B^{<j}$ determines $(X^{<j},Y^{<j})$ given $M$ , and $B^{<j}$ is independent of $B^{j}$ given $M$ and $D$ , by item $3$ of Proposition 1, it suffices to prove that $I(B^{j};\Pi\ |\ M,D,X^{<j},Y^{<j})={\Omega}(n/k)$ . By expanding the conditioning, we can write $I(B^{j};\Pi\ |\ M,D,X^{<j},Y^{<j})$ as

	$\displaystyle\sum_{(x,y,m,d)}\mathsf{Pr}\left[(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y)\right]$
	$\displaystyle\quad\quad\quad\quad\times I\left(B^{j};\Pi\ \|\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y)\right).$

By the definition of a good $j\in[1/\varepsilon^{2}]$ , we know by a Markov bound that with probability ${\Omega}(1)$ over the choice of $(x,y,m,d)$ , we have

I(X^{j},Y^{j};\Pi\ |\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))={\Omega}(1).

Call these $(x,y,m,d)$ for which this holds good for $j$ .

Note that $H(X^{j},Y^{j}\ |\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))=2$ , since $M,D,X^{<j},Y^{<j}$ are independent of $X,Y$ , Therefore, for a good $j$ and a tuple $(x,y,m,d)$ that is good for $j$ , we have

			$\displaystyle H(X^{j},Y^{j}\ \|\ \Pi,M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))$
		$\displaystyle=$	$\displaystyle H(X^{j},Y^{j}\ \|\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))$
			$\displaystyle-I(X^{j},Y^{j};\Pi\ \|\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))$
		$\displaystyle=$	$\displaystyle 2-\Omega(1).$

By the Maximum Likelihood Principle in Proposition 1, the maximum likelihood function $g$ computes $(X^{j},Y^{j})$ from the transcript of $\Pi$ and $M^{j},D^{j},(j,m,d,x,y)$ , with error probability $\delta_{g}$ , over $X^{j},Y^{j},M^{j},D^{j}$ and the randomness of $\Pi$ , satisfying

\displaystyle 1-\delta_{g}\geq\frac{1}{2^{H(X^{j},Y^{j}\ |\ \Pi,M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y))}}\geq\frac{1}{2^{2-{\Omega}(1)}}=\frac{1}{4}+{\Omega}(1),

(26)

Now for a good $j$ , and a tuple $(x,y,m,d)$ that is good for $j$ , we define a protocol $\Pi_{j,x,y,m,d}$ which computes the $k$ -GUESS problem on input $(A_{1},\ldots,A_{k},\{Q,R\})\sim\psi_{n}$ correctly with probability $1/4+\Omega(1)$ . Here $A_{1},\ldots,A_{k}$ are inputs of the $k$ sites and $\{Q,R\}$ is the input of the predictor. The protocol $\Pi_{j,x,y,m,d}$ has $(j,x,y,m,d)$ hardwired into it, and works as follows. First, the $k$ sites construct an input $B$ for the $k$ -BTX problem distributed according to $\nu$ , using $\{A_{1},\ldots,A_{k}\}$ , $(j,x,y,m,d)$ and their private randomness, without any communication: They set $B^{j}=\{A_{1},\ldots,A_{k}\}$ , and use their private randomness to sample inputs for blocks $j^{\prime}\neq j$ using the values $(x,y,m,d)$ and the fact that the inputs to the $k$ sites are independent conditioned on $(x,y,m,d)$ . The predictor sets its input to be $\{M^{j}=Q,D^{j}=R,(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y)\}$ . Next, the $k$ sites run $\Pi$ on their input $B$ . Finally, the predictor outputs $(X^{j},Y^{j})=g(\Pi(B),Q,R,j,m,d,x,y)$ .

Combining these with Theorem 5, we obtain

$\displaystyle I(B;\Pi\ \|\ M,D)$	$\displaystyle\geq$	$\displaystyle\sum_{\text{good }j}I(B^{j};\Pi\ \|\ M,D,B^{<j})$
	$\displaystyle\geq$	$\displaystyle\sum_{\text{good }j}\sum_{\text{good }(x,y,m,d)\text{ for }j}\mathsf{Pr}\left[(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y)\right]$
		$\displaystyle\quad\quad\quad\quad\times I\left(B^{j};\Pi\ \|\ M^{j},D^{j},(M^{-j},D^{-j},X^{<j},Y^{<j})=(m,d,x,y)\right)$
	$\displaystyle=$	$\displaystyle\Omega(1/\varepsilon^{2})\cdot{\Omega}(n/k)\quad(\mbox{By (\ref{eq:error}) and Theorem~\ref{thm:XOR}})$
	$\displaystyle\geq$	$\displaystyle{\Omega}(n/(k\varepsilon^{2})).$

This completes the proof.

By Proposition 2 that says that the randomized communication complexity is always at least the conditional information cost, we have the following immediate corollary.

Corollary 1

Any randomized protocol that computes $k$ -BTX on input distribution $\nu$ with error probability $\delta$ for some sufficient small constant $\delta$ has communication complexity ${\Omega}(n/(k\varepsilon^{2}))$ .

4.5 The Complexity of $F_{p}\ (p>1)$

The input of $\varepsilon$ -approximate $F_{p}\ (p>1)$ is chosen to be the same as $k$ -BTX by setting $n=k^{p}$ . That is, we choose $\{B_{1},\ldots,B_{k}\}$ randomly according to distribution $\nu$ . $B_{i}$ is the input vector for site $S_{i}$ consisting of $1/\varepsilon^{2}$ blocks each having $n=k^{p}$ coordinates. We prove the lower bound for $F_{p}$ by performing a reduction from $k$ -BTX.

Lemma 9

If there exists a protocol $\cal P^{\prime}$ that computes a $(1+\alpha\varepsilon)$ -approximate $F_{p}\ (p>1)$ for a sufficiently small constant $\alpha$ on input distribution $\nu$ with communication complexity $C$ and error probability at most $\delta$ , then there exists a protocol $\cal P$ for $k$ -BTX on input distribution $\nu$ with communication complexity $C$ and error probability at most $3\delta+\sigma$ , where $\sigma$ is an arbitrarily small constant.

Proof:

We pick a random input $B=\{B_{1},\ldots,B_{k}\}$ from distribution $\nu$ . Each coordinate (column) of $B$ represents an item. Thus we have a total of $1/\varepsilon^{2}\cdot k^{p}=k^{p}/\varepsilon^{2}$ possible items, which we identify with the set $[k^{p}/\varepsilon^{2}]$ . If we view each input vector $B_{i}\ (i\in[k])$ as a set, then each site has a subset of $[k^{p}/\varepsilon^{2}]$ corresponding to these $1$ bits. Let $W_{0}$ be the exact value of $F_{p}(B)$ . $W_{0}$ can be written as the sum of three components:

\displaystyle W_{0}

\displaystyle=

\displaystyle\left(\frac{k^{p}-1}{2\varepsilon^{2}}+Q\right)\cdot 1^{p}+\left(\frac{1}{2\varepsilon^{2}}+U\right)\cdot(k/2)^{p}+\left(\frac{1}{4\varepsilon^{2}}+V\right)\cdot k^{p},

(27)

where $Q,U,V$ are random variables (it will be clear why we write it this way in what follows). The first term of the RHS of Equation (27) is the contribution of non-special coordinates across all blocks in each of which one site has $1$ . The second term is the contribution of the special coordinates across all blocks in each of which $k/2$ sites have $1$ . The third term is the contribution of the special coordinates across all blocks in each of which all $k$ sites have $1$ .

Note that $k$ -BTX $(B_{1},\ldots,B_{k})$ is $1$ if $\left|U\right|\geq 2/\varepsilon$ and $0$ if $\left|U\right|\leq 1/\varepsilon$ . Our goal is to use a protocol $\cal P^{\prime}$ for $F_{p}$ to construct a protocol $\cal P$ for $k$ -BTX such that we can differentiate the two cases (i.e., $\left|U\right|\geq 2/\varepsilon$ or $\left|U\right|\leq 1/\varepsilon$ ) with a very good probability.

Given a random input $B$ , let $W_{1}$ be the exact $F_{p}$ -value on the first $k/2$ sites, and $W_{2}$ be the exact $F_{p}$ -value on the second $k/2$ sites. That is, $W_{1}=F_{p}(B_{1},\ldots,B_{k/2})$ and $W_{2}=F_{p}(B_{k/2+1},\ldots,B_{k})$ . We have

\displaystyle W_{1}+W_{2}

\displaystyle=

\displaystyle\left(\frac{k^{p}-1}{2\varepsilon^{2}}+Q\right)\cdot 1^{p}+\left(\frac{1}{2\varepsilon^{2}}+U\right)\cdot(k/2)^{p}+\left(\frac{1}{4\varepsilon^{2}}+V\right)\cdot 2\cdot(k/2)^{p}.

(28)

By Equation (27) and (28) we can cancel out $V$ :

\displaystyle 2^{p-1}(W_{1}+W_{2})-W_{0}

\displaystyle=

\displaystyle(2^{p-1}-1)\left(\left(\frac{k^{p}-1}{2\varepsilon^{2}}+Q\right)+\left(\frac{1}{2\varepsilon^{2}}+U\right)\cdot(k/2)^{p}\right).

(29)

Let $\tilde{W}_{0}$ , $\tilde{W}_{1}$ and $\tilde{W}_{2}$ be the estimated $W_{0}$ , $W_{1}$ and $W_{2}$ obtained by running $\cal P^{\prime}$ on the $k$ sites’ inputs, the first $k/2$ sites’ inputs and the second $k/2$ sites’ inputs, respectively. Observe that $W_{0}\leq(2^{p}+1)k^{p}/\varepsilon^{2}$ and $W_{1},W_{2}\leq 2k^{p}/\varepsilon^{2}$ . By the randomized approximation guarantee of $\cal P^{\prime}$ and the discussion above we have that with probability at least $1-3\delta$ ,

2^{p-1}(W_{1}+W_{2})-W_{0}=2^{p-1}(\tilde{W}_{1}+\tilde{W}_{2})-\tilde{W}_{0}\pm\beta^{\prime}k^{p}/\varepsilon,

(30)

where $\left|\beta^{\prime}\right|\leq 3(2^{p}+1)\alpha$ .

By a Chernoff bound we have that $\left|Q\right|\leq c_{1}k^{p/2}/\varepsilon$ with probability at least $1-\sigma$ , where $\sigma$ is an arbitrarily small constant and $c_{1}\leq\kappa\log^{1/2}(1/\sigma)$ for some universal constant $\kappa$ . Combining this fact with Equation (29) and (30) and letting $\tilde{W}=(2^{p-1}(\tilde{W}_{1}+\tilde{W}_{2})-\tilde{W}_{0})/(2^{p-1}-1)$ , we have that with probability at least $1-3\delta-\sigma$ ,

\displaystyle U

\displaystyle=

\displaystyle\frac{2^{p}\tilde{W}}{k^{p}}-\frac{2^{p}+1}{2\varepsilon^{2}}-\frac{2^{p}\beta}{(2^{p-1}-1)\varepsilon},

(31)

where $\left|\beta\right|\leq 3(2^{p}+1)\alpha+o(1)$ .

Protocol $\cal P$ .

Given an input $B$ for $k$ -BTX, protocol $\cal P$ first uses $\cal P^{\prime}$ to obtain the value $\tilde{W}$ described above, and then determines the answer to $k$ -BTX as follows:

\displaystyle\begin{array}[]{l}k\textrm{-BTX}(B)=\left\{\begin{array}[]{rl}1,&\text{if}\quad\left|2^{p}\tilde{W}/k^{p}-(2^{p}+1)/(2\varepsilon^{2})\right|\geq 1.5/\varepsilon,\\ 0,&\text{otherwise}.\end{array}\right.\end{array}

Correctness.

Note that with probability at least $1-3\delta-\sigma$ , we have $\left|\beta\right|\leq 3(2^{p}+1)\alpha+o(1)$ , where $\alpha>0$ is a sufficiently small constant, and thus $\left|\frac{2^{p}\beta}{(2^{p-1}-1)\varepsilon}\right|<0.5/\varepsilon$ . Therefore, in this case protocol $\cal P$ will always succeed.

Theorem 7 (set $n=k^{p}$ ) and Lemma 9 directly imply the following main theorem for $F_{p}$ .

Theorem 8

Any protocol that computes a $(1+\varepsilon)$ -approximate $F_{p}\ (p>1)$ on input distribution $\nu$ with error probability $\delta$ for some sufficiently small constant $\delta$ has communication complexity ${\Omega}(k^{p-1}/\varepsilon^{2})$ .

5 An Upper Bound for $F_{p}\ (p>1)$

We describe the following protocol to give a factor $(1+\Theta(\varepsilon))$ -approximation to $F_{p}$ at all points in time in the union of $k$ streams each held by a different site. Each site has a non-negative vector $v^{i}\in\mathbb{R}^{m}$ , ⁷⁷7We use $m$ instead of $N$ for universe size only in this section. which evolves with time, and at all times the coordinator holds a $(1+\Theta(\varepsilon))$ -approximation to $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}$ . Let $n$ be the length of the union of the $k$ streams. We assume $n={\mathrm{poly}}(m)$ , and that $k$ is a power of $2$ .

As observed in [20], up to a factor of $O(\varepsilon^{-1}\log n\log(\varepsilon^{-1}\log n))$ in communication, the problem is equivalent to the threshold problem: given a threshold $\tau$ , with probability $2/3$ : when $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}>\tau$ , the coordinator outputs $1$ , when $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}<\tau/(1+\varepsilon)$ , the coordinator outputs $0$ , and for $\tau/(1+\varepsilon)\leq\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}\leq\tau$ , the coordinator can output either $0$ or $1$ ⁸⁸8To see the equivalence, by independent repetition, we can assume the success probability of the protocol for the threshold problem is $1-\Theta(\varepsilon/\log n)$ . Then we can run a protocol for each $\tau=1,(1+\varepsilon),(1+\varepsilon)^{2},(1+\varepsilon)^{3},\ldots,\Theta(n^{2})$ , and we are correct on all instantiations with probability at least $2/3$ ..

We can thus assume we are given a threshold $\tau$ in the following algorithm description. For notational convenience, define $\tau_{\ell}=\tau/2^{\ell}$ for an integer $\ell$ . A nice property of the algorithm is that it is one-way, namely, all communication is from the sites to the coordinator. We leave optimization of the ${\mathrm{poly}}(\varepsilon^{-1}\log n)$ factors in the communication complexity to future work.

5.1 Our Protocol

The protocol consists of four algorithms illustrated in Algorithm 1 to Algorithm 4. Let $v=\sum_{i=1}^{k}v^{i}$ at any point in time during the union of the $k$ streams. At times we will make the following assumptions on the algorithm parameters $\gamma,B,$ and $r$ : we assume $\gamma=\Theta(\varepsilon)$ is sufficiently small, and $B={\mathrm{poly}}(\varepsilon^{-1}\log n)$ and $r=\Theta(\log n)$ are sufficiently large.

r=\Theta(\log n)

/* A parameter used by the sites and coordinator */

for $z=1,2,\ldots,r$ do

for $\ell=0,1,2,\ldots,\log m$ do

Create a set

S^{z}_{\ell}

by including each coordinate in

[m]

independently with probability

2^{-\ell}

Algorithm 1 Intepretation of the random public coin by sites and the coordinator

\gamma=\Theta(\varepsilon),B={\mathrm{poly}}(\varepsilon^{-1}\log n)

. Choose

\eta\in[0,1]

uniformly at random /* Parameters */

for $z=1,2,\ldots,r$ do

for $\ell=0,1,2,\ldots,\log m$ do

for $j=1,2,\ldots,m$ do

f_{z,\ell,j}\leftarrow 0

/* Initialize all frequencies seen to

0

out\leftarrow 0

/* The coordinator’s current output */

Algorithm 2 Initialization at Coordinator

for $z=1,2,\ldots,r$ do

for $\ell=0,1,2,\ldots,\log m$ do

if $j\in S_{\ell}^{z}$ and $v_{j}^{i}>\tau_{\ell}^{1/p}/(kB)$ then

With probability

\min(B/\tau_{\ell}^{1/p},1)

, send

(j,z,\ell)

to the coordinator

Algorithm 3 When Site

i

receives an update

v^{i}\leftarrow v^{i}+e_{j}

for standard unit vector

e_{j}

f_{z,\ell,j}\leftarrow f_{z,\ell,j}+\tau_{\ell}^{1/p}/B

for $h=0,1,2,\ldots,O(\gamma^{-1}\log(n/\eta^{p}))$ do

for $z=1,2,\ldots,r$ do

Choose

\ell

for which

2^{\ell}\leq\frac{\tau}{\eta^{p}(1+\gamma)^{ph}B}<2^{\ell+1}

, or

\ell=0

if no such

\ell

exists

Let

F_{z,h}=\{j\in[m]\mid f_{z,\ell,j}\in[\eta(1+\gamma)^{h},\eta(1+\gamma)^{h+1})\}

\tilde{c}_{h}=\textrm{median}_{z}\ 2^{\ell}\cdot|F_{z,h}|

if $\sum_{h\geq 0}\tilde{c}_{h}\cdot\eta^{p}\cdot(1+\gamma)^{ph}>(1-\varepsilon)\tau$ then

out\leftarrow 1

Terminate the protocol

Algorithm 4 Algorithm at Coordinator if a tuple

(j,z,\ell)

arrives

5.2 Communication Cost

Lemma 10

Consider any setting of $v^{1},\ldots,v^{k}$ for which we have $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}\leq 2^{p}\cdot\tau.$ Then the expected total communication is $k^{p-1}\cdot{\mathrm{poly}}(\varepsilon^{-1}\log n)$ bits.

Proof:

Fix any particular $z\in[r]$ and $\ell\in[0,1,\ldots,\log m]$ . Let $v_{j}^{i,\ell}$ equal $v_{j}^{i}$ if $j\in S_{\ell}$ and equal $0$ otherwise. Let $v^{i,\ell}$ be the vector with coordinates $v_{j}^{i,\ell}$ for $j\in[m]$ . Also let $v^{\ell}=\sum_{i=1}^{k}v^{i,\ell}$ . Observe that ${\bf E}[\|v^{\ell}\|_{p}^{p}]\leq 2^{p}\cdot\tau/2^{\ell}=2^{p}\cdot\tau_{\ell}$ .

Because of non-negativity of the $v^{i}$ ,

\sum_{i=1}^{k}\sum_{j\in S_{\ell}}(v_{j}^{i,\ell})^{p}\leq\sum_{i=1}^{k}\|v^{i,\ell}\|_{p}^{p}\leq\|v^{\ell}\|_{p}^{p}.

Notice that a $j\in S_{\ell}$ is sent by a site with probability at most $B/\tau_{\ell}^{1/p}$ and only if $(v_{j}^{i})^{p}\geq\frac{\tau_{\ell}}{k^{p}B^{p}}$ . Hence the expected number of messages sent for this $z$ and $\ell$ , over all randomness, is

\frac{B}{\tau_{\ell}^{1/p}}{\bf E}\left[\sum_{i,j\ \mid\ (v_{j}^{i})^{p}\geq\frac{\tau_{\ell}}{k^{p}B^{p}}}v_{j}^{i}\right]\leq\frac{B}{\tau_{\ell}^{1/p}}\cdot\frac{{\bf E}[\|v^{\ell}\|_{p}^{p}]}{\tau_{\ell}/(k^{p}B^{p})}\cdot\frac{\tau_{\ell}^{1/p}}{kB}\leq\frac{2^{p}\cdot\tau_{\ell}\cdot k^{p-1}\cdot B^{p}}{\tau_{\ell}}=2^{p}\cdot k^{p-1}\cdot B^{p},

(34)

where we used that $\sum v_{j}^{i}$ is maximized subject to $(v_{j}^{i})^{p}\geq\frac{\tau_{\ell}}{k^{p}B^{p}}$ and $\sum(v_{j}^{i})^{p}\leq\|v^{\ell}\|_{p}^{p}$ when all the $v_{j}^{i}$ are equal to $\tau_{\ell}^{1/p}/(kB)$ . Summing over all $z$ and $\ell$ , it follows that the expected number of messages sent in total is $O(k^{p-1}B^{p}\log^{2}n)$ . Since each message is $O(\log n)$ bits, the expected number of bits is $k^{p-1}\cdot{\mathrm{poly}}(\varepsilon^{-1}\log n)$ .

5.3 Correctness

We let $C>0$ be a sufficiently large constant.

5.3.1 Concentration of Individual Frequencies

We shall make use of the following standard multiplicative Chernoff bound.

Fact 2

Let $X_{1},\ldots X_{s}$ be i.i.d. Bernoulli $(q)$ random variables. Then for all $0<\beta<1$ ,

\mathsf{Pr}\left[|\sum_{i=1}^{s}X_{i}-qs|\geq\beta qs\right]\leq 2\cdot e^{-\frac{\beta^{2}qs}{3}}.

Lemma 11

For a sufficiently large constant $C>0$ , with probability $1-n^{-\Omega(C)}$ , for all $z$ , $\ell$ , $j\in S_{\ell}$ , and all times in the union of the $k$ streams,

1.

$f_{z,\ell,j}\leq 2e\cdot v_{j}+\frac{C\tau_{\ell}^{1/p}\log n}{B}$ , and
2.

if $v_{j}\geq\frac{C(\log^{5}n)\tau_{\ell}^{1/p}}{B\gamma^{10}}$ , then $|f_{z,\ell,j}-v_{j}|\leq\frac{\gamma^{5}}{\log^{2}n}\cdot v_{j}$ .

Proof:

Fix a particular time snapshot in the stream. Let $g_{z,\ell,j}=f_{z,\ell,j}\cdot B/\tau_{\ell}^{1/p}$ . Then $g_{z,\ell,j}$ is a sum of indicator variables, where the number of indicator variables depends on the values of the $v_{j}^{i}$ . The indicator variables are independent, each with expectation $\min(B/\tau_{\ell}^{1/p},1)$ .

First part of lemma. The number $s$ of indicator variables is at most $v_{j}$ , and the expectation of each is at most $B/\tau_{\ell}^{1/p}$ . Hence, the probability that $w=2e\cdot v_{j}\cdot B/\tau_{\ell}^{1/p}+C\log n$ or more of them equal $1$ is at most

{v_{j}\choose w}\cdot\left(\frac{B}{\tau_{\ell}^{1/p}}\right)^{w}\leq\left(\frac{ev_{j}B}{w\tau_{\ell}^{1/p}}\right)^{w}\leq\left(\frac{1}{2}\right)^{C\log n}=n^{-C}.

This part of the lemma now follows by scaling the $g_{z,\ell,j}$ by $\tau_{\ell}^{1/p}/B$ to obtain a bound on the $f_{z,\ell,j}$ .

Second part of lemma. Suppose at this time $v_{j}\geq\frac{C(\log^{5}n)\tau_{\ell}^{1/p}}{B\gamma^{10}}$ . The number $s$ of indicator variables is minimized when there are $k-1$ distinct $i$ for which $v_{j}^{i}=\frac{\tau_{\ell}^{1/p}}{kB}$ , and one value of $i$ for which

v_{j}^{i}=v_{j}-(k-1)\cdot\frac{\tau_{\ell}^{1/p}}{kB}.

Hence,

s\geq v_{j}-(k-1)\cdot\frac{\tau_{\ell}^{1/p}}{kB}-\frac{\tau_{\ell}^{1/p}}{kB}=v_{j}-\frac{\tau_{\ell}^{1/p}}{B}.

If the expectation is $1$ , then $f_{z,\ell,j}=v_{j}-\frac{\tau_{\ell}^{1/p}}{B}$ , and using that $v_{j}\geq\frac{C(\log^{5}n)\tau_{\ell}^{1/p}}{B\gamma^{10}}$ establishes this part of the lemma. Otherwise, applying Fact 2 with $s\geq v_{j}-\frac{\tau_{\ell}^{1/p}}{B}\geq\frac{C(\log^{5}n)\tau_{\ell}^{1/p}}{2B\gamma^{10}}$ and $q=\frac{B}{\tau_{\ell}^{1/p}}$ , and using that $qs\geq\frac{C\log^{5}n}{2\gamma^{10}}$ , we have

\mathsf{Pr}\left[|g_{z,\ell,j}-qs|>\frac{\gamma^{5}qs}{2\log^{2}n}\right]=n^{-\Omega(C)}.

Scaling by $\frac{\tau_{\ell}^{1/p}}{B}=\frac{1}{q}$ , we have

\mathsf{Pr}\left[|f_{s,\ell,j}-s|>\frac{\gamma^{5}s}{2\log^{2}n}\right]=n^{-\Omega(C)},

and since $v_{j}-\frac{\tau_{\ell}^{1/p}}{B}\leq s\leq v_{j}$ ,

\mathsf{Pr}\left[|f_{s,\ell,j}-v_{j}|\geq\frac{\gamma^{5}v_{j}}{2\log^{2}n}+\frac{\tau_{\ell}^{1/p}}{B}\right]=n^{-\Omega(C)},

and finally using that $\frac{\tau_{\ell}^{1/p}}{B}<\frac{\gamma^{5}v_{j}}{2\log^{2}n}$ , and union-bounding over a stream of length $n$ as well as all choices of $z,\ell,$ and $j$ , the lemma follows.

5.3.2 Estimating Class Sizes

Define the classes $C_{h}$ as follows:

C_{h}=\{j\in[m]\mid\eta(1+\gamma)^{h}\leq v_{j}<\eta(1+\gamma)^{h+1}\}.

Say that $C_{h}$ contributes at a point in time in the union of the $k$ streams if

|C_{h}|\cdot\eta^{p}(1+\gamma)^{ph}\geq\frac{\gamma\|v\|_{p}^{p}}{B^{1/2}\log(n/\eta^{p})}.

Since the number of non-zero $|C_{h}|$ is $O(\gamma^{-1}\log(n/\eta^{p}))$ , we have

\displaystyle\sum_{\textrm{ non-contributing }h}|C_{h}|\cdot\eta^{p}(1+\gamma)^{ph+p}=O\left(\frac{\|v\|_{p}^{p}}{B^{1/2}}\right).

(35)

Lemma 12

With probability $1-n^{-\Omega(C)}$ , at all points in time in the union of the $k$ streams and for all $h$ and $\ell$ , for at least a $3/5$ fraction of the $z\in[r]$ ,

|C_{h}\cap S_{\ell}^{z}|\leq 3\cdot 2^{-\ell}\cdot|C_{h}|

Proof:

The random variable $|C_{h}\cap S_{\ell}^{z}|$ is a sum of $|C_{h}|$ independent Bernoulli $(2^{-\ell})$ random variables. By a Markov bound, $\mathsf{Pr}[|C_{h}\cap S_{\ell}^{z}|\leq 3\cdot 2^{-\ell}|C_{h}|]\geq 2/3$ . Letting $X_{z}$ be an indicator variable which is $1$ iff $|C_{h}\cap S_{\ell}^{z}|\leq 3\cdot 2^{-\ell}|C_{h}|$ , the lemma follows by applying Fact 2 to the $X_{z}$ , using that $r$ is large enough, and union-bounding over a stream of length $n$ and all $h$ and $\ell$ .

For a given $C_{h}$ , let $\ell(h)$ be the value of $\ell$ for which we have $2^{\ell}\leq\frac{\tau}{\eta^{p}(1+\gamma)^{ph}B}<2^{\ell+1}$ , or $\ell=0$ if no such $\ell$ exists.

Lemma 13

With probability $1-n^{-\Omega(C)}$ , at all points in time in the union of the $k$ streams and for all $h$ , for at least a $3/5$ fraction of the $z\in[r]$ ,

1.

$2^{\ell(h)}\cdot|C_{h}\cap S_{\ell(h)}^{z}|\leq 3|C_{h}|,$ and
2.

if at this time $C_{h}$ contributes and $\|v\|_{p}^{p}\geq\frac{\tau}{5}$ , then $2^{\ell(h)}\cdot|C_{h}\cap S_{\ell(h)}^{z}|=\left(1\pm\gamma\right)|C_{h}|.$

Proof:

We show this statement for a fixed $h$ and at a particular point in time in the union of the $k$ streams. The lemma will follow by a union bound.

The first part of the lemma follows from Lemma 12.

We now prove the second part. In this case $\|v\|_{p}^{p}\geq\frac{\tau}{5}$ . We can assume that there exists an $\ell$ for which $2^{\ell}\leq\frac{\tau}{\eta^{p}(1+\gamma)^{ph}B}<2^{\ell+1}$ . Indeed, otherwise $\ell(h)=0$ and $|C_{h}\cap S_{\ell(h)}^{z}|=|C_{h}|$ and the second part of the lemma follows.

Let $q(z)=|C_{h}\cap S_{\ell(h)}^{z}|$ , which is a sum of independent indicator random variables and so ${\bf Var}[q(z)]\leq{\bf E}[q(z)]$ . Also,

\displaystyle{\bf E}[q(z)]

\displaystyle=

\displaystyle 2^{-\ell}|C_{h}|\geq\frac{\eta^{p}(1+\gamma)^{ph}B}{\tau}\cdot|C_{h}|.

(36)

Since $C_{h}$ contributes, $|C_{h}|\cdot\eta^{p}\cdot(1+\gamma)^{ph}\geq\frac{\gamma\|v\|_{p}^{p}}{B^{1/2}\log(n/\eta^{p})}$ , and combining this with (36),

{\bf E}[q(z)]\geq\frac{B\gamma\|v\|_{p}^{p}}{B^{1/2}\tau\log(n/\eta^{p})}\geq\frac{B^{1/2}\gamma}{5\log(n/\eta^{p})}.

It follows that for $B$ sufficiently large, and assuming $\eta\geq 1/n^{C}$ which happens with probability $1-1/n^{C}$ , we have ${\bf E}[q(z)]\geq\frac{3}{\gamma^{2}}$ , and so by Chebyshev’s inequality,

\mathsf{Pr}\left[|q(z)-{\bf E}[q(z)]|\geq\gamma{\bf E}[q(z)]\right]\leq\frac{{\bf Var}[q(z)]}{\gamma^{2}\cdot{\bf E}^{2}[q(z)]}\leq\frac{1}{3}.

Since ${\bf E}[q(z)]=2^{-\ell}|C_{h}|$ , and $r=\Theta(\log n)$ is large enough, the lemma follows by a Chernoff bound.

5.3.3 Combining Individual Frequency Estimation and Class Size Estimation

We define the set $T$ to be the set of times in the input stream for which the $F_{p}$ -value of the union of the $k$ streams first exceeds $(1+\gamma)^{i}$ for an $i$ satisfying

0\leq i\leq\log_{(1+\gamma)}2^{p}\cdot\tau.

Lemma 14

With probability $1-O(\gamma)$ , for all times in $T$ and all $h$ ,

1.

$\tilde{c}_{h}\leq 3|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|)$ , and
2.

if at this time $C_{h}$ contributes and $\|v\|_{p}^{p}\geq\frac{\tau}{5}$ , then

$(1-4\gamma)|C_{h}|\leq\tilde{c}_{h}\leq(1+\gamma)|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|).$

Proof:

We assume the events of Lemma 11 and Lemma 13 occur, and we add $n^{-\Omega(C)}$ to the error probability. Let us fix a class $C_{h}$ , a point in time in $T$ , and a $z\in[r]$ which is among the at least $3r/5$ different $z$ that satisfy Lemma 13 at this point in time.

By Lemma 11, for any $j\in C_{h}\cap S_{\ell(h)}^{z}$ for which $v_{j}\geq\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ , if

\displaystyle|\min(v_{j}-\eta(1+\gamma)^{h},\eta(1+\gamma)^{h+1}-v_{j})|\geq\frac{\gamma^{5}}{\log^{2}n}\cdot v_{j},

(37)

then $j\in F_{z,h}$ . Let us first verify that for $j\in C_{h}$ , we have $v_{j}\geq\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ . We have

\displaystyle v_{j}^{p}\geq\eta^{p}(1+\gamma)^{ph}\geq\frac{\tau}{2^{\ell(h)+1}B}\geq\frac{\tau_{\ell(h)}}{2B},

(38)

and so

v_{j}\geq\left(\frac{\tau_{\ell(h)}}{2B}\right)^{1/p}\geq\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}},

where the final inequality follows for large enough $B={\mathrm{poly}}(\varepsilon^{-1}\log n)$ and $p>1$ .

It remains to consider the case when (37) does not hold.

Conditioned on all other randomness, $\eta\in[0,1]$ is uniformly random subject to $v_{j}\in C_{h}$ , or equivalently,

\frac{v_{j}}{(1+\gamma)^{h+1}}<\eta\leq\frac{v_{j}}{(1+\gamma)^{h}}.

If (37) does not hold, then either

\frac{(1-\gamma^{5}/\log^{2}n)v_{j}}{(1+\gamma)^{h}}\leq\eta,\textrm{ or }\eta\leq\frac{(1+\gamma^{5}/\log^{2}n)v_{j}}{(1+\gamma)^{h+1}}.

Hence, the probability over $\eta$ that inequality (37) holds is at least

1-\frac{\frac{\gamma^{5}v_{j}}{(1+\gamma)^{h}\log^{2}n}+\frac{\gamma^{5}v_{j}}{(1+\gamma)^{h+1}\log^{2}n}}{\frac{v_{j}}{(1+\gamma)^{h}}-\frac{v_{j}}{(1+\gamma)^{h+1}}}=1-\frac{\gamma^{4}(2+\gamma)}{\log^{2}n}.

It follows by a Markov bound that

\displaystyle\mathsf{Pr}\left[|C_{h}\cap S_{\ell(h)}^{z}|\geq|C_{h}|\cdot\left(1-\gamma(2+\gamma)\right)\right]\leq\frac{\gamma^{3}}{\log^{2}n}.

(39)

Now we must consider the case that there is a $j^{\prime}\in C_{h^{\prime}}\cap S^{z}_{\ell(h)}$ for which $j^{\prime}\in F_{z,h}$ for an $h^{\prime}\neq h$ . There are two cases, namely, if $v_{j^{\prime}}<\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ or if $v_{j^{\prime}}\geq\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ . We handle each case in turn.

Case: $v_{j^{\prime}}<\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ . Then by Lemma 11,

f_{z,\ell(h),j^{\prime}}\leq 2e\cdot v_{j^{\prime}}+\frac{C\tau_{\ell(h)}^{1/p}\log n}{B}.

Therefore, it suffices to show that

2e\cdot\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}+\frac{C\tau_{\ell(h)}^{1/p}\log n}{B}<\eta(1+\gamma)^{h},

from which we can conclude that $j^{\prime}\notin F_{z,h}$ . But by (38),

\eta(1+\gamma)^{h}\geq\left(\frac{\tau_{\ell(h)}}{2B}\right)^{1/p}>2e\cdot\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}+\frac{C\tau_{\ell(h)}^{1/p}\log n}{B},

where the last inequality follows for large enough $B={\mathrm{poly}}(\varepsilon^{-1}\log n)$ . Hence, $j^{\prime}\notin F_{z,h}$ .

Case: $v_{j^{\prime}}\geq\frac{C(\log^{5}n)\tau_{\ell(h)}^{1/p}}{B\gamma^{10}}$ . We claim that $h^{\prime}\in\{h-1,h+1\}$ . Indeed, by Lemma 11 we must have

\eta(1+\gamma)^{h}-\frac{\gamma^{5}}{\log^{2}n}\cdot v_{j^{\prime}}\leq v_{j^{\prime}}\leq\eta(1+\gamma)^{h+1}+\frac{\gamma^{5}}{\log^{2}n}\cdot v_{j^{\prime}}.

This is equivalent to

\frac{\eta(1+\gamma)^{h}}{1+\gamma^{5}/\log^{2}n}\leq v_{j^{\prime}}\leq\frac{\eta(1+\gamma)^{h+1}}{1-\gamma^{5}/\log^{2}n},

If $j^{\prime}\in C_{h^{\prime}}$ for $h^{\prime}<h-1$ , then

v_{j^{\prime}}\leq\eta(1+\gamma)^{h-1}=\frac{\eta(1+\gamma)^{h}}{1+\gamma}<\frac{\eta(1+\gamma)^{h}}{1+\gamma^{5}/\log^{2}n},

which is impossible. Also, if $j^{\prime}\in C_{h^{\prime}}$ for $h^{\prime}>h+1$ , then

v_{j^{\prime}}\geq\eta(1+\gamma)^{h+2}=\eta(1+\gamma)^{h+1}\cdot(1+\gamma)>\frac{\eta(1+\gamma)^{h+1}}{1-\gamma^{5}/\log^{2}n},

which is impossible. Hence, $h^{\prime}\in\{h-1,h+1\}$ .

Let $N_{z,h}=F_{z,h}\setminus C_{h}$ . Then

\displaystyle{\bf E}[|N_{z,h}|\leq\frac{\gamma^{4}(2+\gamma)}{\log^{2}n}\cdot(|C_{h-1}\cap S_{\ell(h)}^{z}|+|C_{h+1}\cap S_{\ell(h)}^{z}|).

(40)

By (39) and applying a Markov bound to (40), together with a union bound, with probability $\geq 1-\frac{2\gamma^{3}}{\log^{2}n}$ ,

\displaystyle(1-\gamma(2+\gamma))\cdot|C_{h}\cap S_{\ell(h)}^{z}|\leq|F_{z,h}|

(41)

|F_{z,h}|\leq|C_{h}\cap S_{\ell(h)}^{z}|+\gamma(2+\gamma)\cdot(|C_{h-1}\cap S_{\ell(h)}^{z}|+|C_{h+1}\cap S_{\ell(h)}^{z}|).

(42)

By Lemma 12,

2^{\ell(h)}|C_{h-1}\cap S_{\ell(h)}^{z}|\leq 3|C_{h-1}|\textrm{ and }\ 2^{\ell(h)}|C_{h+1}\cap S_{\ell(h)}^{z}|\leq 3|C_{h+1}|.

(43)

First part of lemma. At this point we can prove the first part of this lemma. By the first part of Lemma 13,

\displaystyle 2^{\ell(h)}\cdot|C_{h}\cap S_{\ell(h)}^{z}|\leq 3|C_{h}|.

(44)

Combining (42), (43), and (44), we have with probability at least $1-\frac{2\gamma^{3}}{\log^{2}n}-n^{-\Omega(C)}$ ,

2^{\ell(h)}|F_{z,h}|\leq 3|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|).

Since this holds for at least $3r/5$ different $z$ , it follows that

\tilde{c}_{h}\leq 3|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|),

and the first part of the lemma follows by a union bound. Indeed, the number of $h$ is $O(\gamma^{-1}\log(n/\eta^{p}))$ , which with probability $1-1/n$ , say, is $O(\gamma^{-1}\log n)$ since with this probability $\eta^{p}\geq 1/n^{p}$ . Also, $|T|=O(\gamma^{-1}\log n)$ . Hence, the probability this holds for all $h$ and all times in $T$ is $1-O(\gamma)$ .

Second part of the lemma. Now we can prove the second part of the lemma. By the second part of Lemma 13, if at this time $C_{h}$ contributes and $\|v\|_{p}^{p}\geq\frac{\tau}{5}$ , then

\displaystyle 2^{\ell(h)}\cdot|C_{h}\cap S_{\ell(h)}^{z}|=(1\pm\gamma)|C_{h}|.

(45)

Combining (41), (42), (43), and (45), we have with probability at least $1-\frac{2\gamma^{3}}{\log^{2}n}-n^{-\Omega(C)}$ ,

(1-\gamma(2+\gamma))(1-\gamma)|C_{h}|\leq 2^{\ell(h)}|F_{z,h}|\leq(1+\gamma)|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|).

Since this holds for at least $3r/5$ different $z$ , it follows that

(1-\gamma(2+\gamma))(1-\gamma)|C_{h}|\leq\tilde{c}_{h}\leq(1+\gamma)|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|).

and the second part of the lemma now follows by a union bound over all $h$ and all times in $T$ , exactly in the same way as the first part of the lemma. Note that $1-4\gamma\leq(1-\gamma(2+\gamma))(1-\gamma)$ for small enough $\gamma=\Theta(\varepsilon)$ .

5.3.4 Putting It All Together

Lemma 15

With probability at least $5/6$ , at all times the coordinator’s output is correct.

Proof:

The coordinator outputs $0$ up until the first point in time in the union of the $k$ streams for which $\sum_{h\geq 0}\tilde{c}_{h}\cdot\eta^{p}\cdot(1+\gamma)^{ph}>(1-\varepsilon/2)\tau$ . It suffices to show that

\displaystyle\sum_{h\geq 0}\tilde{c}_{h}\eta^{p}(1+\gamma)^{ph}=(1\pm\varepsilon/2)\|v\|_{p}^{p}

(46)

at all times in the stream. We first show that with probability at least $5/6$ , for all times in $T$ ,

\displaystyle\sum_{h\geq 0}\tilde{c}_{h}\eta^{p}(1+\gamma)^{ph}=(1\pm\varepsilon/4)\|v\|_{p}^{p},

(47)

and then use the structure of $T$ and the protocol to argue that (46) holds at all times in the stream.

Fix a particular time in $T$ . We condition on the event of Lemma 14, which by setting $\gamma=\Theta(\varepsilon)$ small enough, can assume occurs with probability at least $5/6$ .

First, suppose at this point in time we have $\|v\|_{p}^{p}<\frac{\tau}{5}$ . Then by Lemma 14, for sufficiently small $\gamma=\Theta(\varepsilon)$ , we have

$\displaystyle\sum_{h\geq 0}\tilde{c}_{h}\cdot\eta^{p}(1+\gamma)^{ph}$	$\displaystyle\leq$	$\displaystyle\sum_{h\geq 0}(3\|C_{h}\|+3\gamma(2+\gamma)(\|C_{h-1}\|+\|C_{h+1}\|))\cdot\eta^{p}(1+\gamma)^{ph}$
	$\displaystyle\leq$	$\displaystyle\sum_{h\geq 0}\left(3\sum_{j\in C_{h}}v_{j}^{p}+3\gamma(2+\gamma)(1+\gamma)^{2}\sum_{j\in C_{h-1}\cup C_{h+1}}v_{j}^{p}\right)$
	$\displaystyle\leq$	$\displaystyle 4\\|v\\|_{p}^{p}$
	$\displaystyle\leq$	$\displaystyle\frac{4\tau}{5},$

and so the coordinator will correctly output $0$ , provided $\varepsilon<\frac{1}{5}$ .

We now handle the case $\|v\|_{p}^{p}\geq\frac{\tau}{5}$ . Then for all contributing $C_{h}$ , we have

(1-4\gamma)|C_{h}|\leq\tilde{c}_{h}\leq(1+\gamma)|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|),

while for all $C_{h}$ , we have

\tilde{c}_{h}\leq 3|C_{h}|+3\gamma(2+\gamma)(|C_{h-1}|+|C_{h+1}|).

Hence, using (35),

$\displaystyle\sum_{h\geq 0}\tilde{c}_{h}\cdot\eta^{p}(1+\gamma)^{ph}$	$\displaystyle\geq$	$\displaystyle\sum_{\textrm{contributing }C_{h}}(1-4\gamma)\|C_{h}\|\eta^{p}(1+\gamma)^{ph}$
	$\displaystyle\geq$	$\displaystyle\frac{(1-4\gamma)}{(1+\gamma)^{2}}\sum_{\textrm{contributing }C_{h}}\sum_{j\in C_{h}}v_{j}^{p}$
	$\displaystyle\geq$	$\displaystyle(1-6\gamma)\cdot(1-O(1/B^{1/2}))\cdot\\|v\\|_{p}^{p}.$

For the other direction,

			$\displaystyle\sum_{h\geq 0}\tilde{c}_{h}\cdot\eta^{p}(1+\gamma)^{ph}$
		$\displaystyle\leq$	$\displaystyle\sum_{\textrm{contributing }C_{h}}(1+\gamma)\|C_{h}\|\eta^{p}(1+\gamma)^{ph}+\sum_{\textrm{non-contributing }C_{h}}3\|C_{h}\|\eta^{p}(1+\gamma)^{ph}$
			$\displaystyle+\sum_{h\geq 0}3\gamma(2+\gamma)(\|C_{h-1}\|+\|C_{h+1}\|)\eta^{p}(1+\gamma)^{ph}$
		$\displaystyle\leq$	$\displaystyle(1+\gamma)\sum_{\textrm{contributing }C_{h}}\sum_{j\in C_{h}}v_{j}^{p}+O(1/B^{1/2})\cdot\\|v\\|_{p}^{p}+O(\gamma)\cdot\\|v\\|_{p}^{p}$
		$\displaystyle\leq$	$\displaystyle(1+O(\gamma)+O(1/B^{1/2}))\\|v\\|_{p}^{p}.$

Hence, (47) follows for all times in $T$ provided that $\gamma=\Theta(\varepsilon)$ is small enough and $B={\mathrm{poly}}(\varepsilon^{-1}\log n)$ is large enough.

It remains to argue that (46) holds for all points in time in the union of the $k$ streams. Recall that each time in the union of the $k$ streams for which $\|v\|_{p}^{p}\geq(1+\gamma)^{i}$ for an integer $i$ is included in $T$ , provided $\|v\|_{p}^{p}\leq 2^{p}\tau$ .

The key observation is that the quantity $\sum_{h\geq 0}\tilde{c}_{h}\eta^{p}(1+\gamma)^{ph}$ is non-decreasing, since the values $|F_{z,h}|$ are non-decreasing. Now, the value of $\|v\|_{p}^{p}$ at a time $t$ not in $T$ is, by definition of $T$ , within a factor of $(1\pm\gamma)$ of the value of $\|v\|_{p}^{p}$ for some time in $T$ . Since (47) holds for all times in $T$ , it follows that the value of $\sum_{h\geq 0}\tilde{c}_{h}\eta^{p}(1+\gamma)^{ph}$ at time $t$ satisfies

(1-\gamma)(1-\varepsilon/4)\|v\|_{p}^{p}\leq\sum_{h\geq 0}\tilde{c}_{h}\eta^{p}(1+\gamma)^{ph}\leq(1+\gamma)(1+\varepsilon/4)\|v\|_{p}^{p},

which implies for $\gamma=\Theta(\varepsilon)$ small enough that (46) holds for all points in time in the union of the $k$ streams. This completes the proof.

Theorem 9

(MAIN) With probability at least $2/3$ , at all times the coordinator’s output is correct and the total communication is $k^{p-1}\cdot{\mathrm{poly}}(\varepsilon^{-1}\log n)$ bits.

Proof:

Consider the setting of $v^{1},\ldots,v^{k}$ at the first time in the stream for which $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}>\tau$ . For any non-negative integer vector $w$ and any update $e_{j}$ , we have $\|w+e_{j}\|_{p}^{p}\leq(\|w\|_{p}+1)^{p}\leq 2^{p}\|w\|_{p}^{p}$ . Since $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}$ is an integer and $\tau\geq 1$ , we therefore have $\|\sum_{i=1}^{k}v^{i}\|_{p}^{p}\leq 2^{p}\cdot\tau$ . By Lemma 10, the expected communication for these $v^{1},\ldots,v^{k}$ is $k^{p-1}\cdot{\mathrm{poly}}(\varepsilon^{-1}\log n)$ bits, so with probability at least $5/6$ the communication is $k^{p-1}\cdot{\mathrm{poly}}(\varepsilon^{-1}\log n)$ bits. By Lemma 15, with probability at least $5/6$ , the protocol terminates at or before the time for which the inputs held by the players equal $v^{1},\ldots,v^{k}$ . The theorem follows by a union bound.

6 Related Problems

In this section we show that the techniques we have developed for distributed $F_{0}$ and $F_{p}\ (p>1)$ can also be used to solve other fundamental problems. In particular, we consider the problems: all-quantile, heavy hitters, empirical entropy and $\ell_{p}$ for any $p>0$ . For the first three problems, we are able to show that our lower bounds holds even if we allow some additive error $\varepsilon$ . From definitions below one can observe that lower bounds for additive $\varepsilon$ -approximations also hold for their multiplicative $(1+\varepsilon)$ -approximation counterparts.

6.1 The All-Quantile and Heavy Hitters

We first give the definitions of the problems. Given a multiset $A=\{a_{1},a_{2},\ldots,a_{m}\}$ where each $a_{i}$ is drawn from the universe $[N]$ , let $f_{i}$ be the frequency of item $i$ in the set $A$ . Thus $\sum_{i\in[N]}f_{i}=m$ .

Definition 5

( $\phi$ -heavy hitters) For any $0\leq\phi\leq 1$ , the set of $\phi$ -heavy hitters of $A$ is $H_{\phi}(A)=\{x\ |\ f_{x}\geq\phi m\}$ . If an $\varepsilon$ -approximation is allowed, then the returned set of heavy hitters must contain $H_{\phi}(A)$ and cannot include any $x$ such that $f_{x}<(\phi-\varepsilon)m$ . If $(\phi-\varepsilon)m\leq f_{x}<\phi m$ , then $x$ may or may not be included in $H_{\phi}(A)$ .

Definition 6

( $\phi$ -quantile) For any $0\leq\phi\leq 1$ , the $\phi$ -quantile of $A$ is some $x$ such that there are at most $\phi m$ items of $A$ that are smaller than $x$ and at most $(1-\phi)m$ items of $A$ that are greater than $x$ . If an $\varepsilon$ -approximation is allowed, then when asking for the $\phi$ -quantile of $A$ we are allowed to return any $\phi^{\prime}$ -quantile of $A$ such that $\phi-\varepsilon\leq\phi^{\prime}\leq\phi+\varepsilon$ .

Definition 7

(All-quantile) The $\varepsilon$ -approximate all-quantile (QUAN) problem is defined in the coordinator model, where we have $k$ sites and a coordinator. Site $S_{i}\ (i\in[k])$ has a set $A_{i}$ of items. The $k$ sites want to communicate with the coordinator so that at the end of the process the coordinator can construct a data structure from which all $\varepsilon$ -approximate $\phi$ -quantiles for any $0\leq\phi\leq 1$ can be extracted. The cost is defined as the total number of bits exchanged between the coordinator and the $k$ sites.

Theorem 10

Any randomized protocol that computes $\varepsilon$ -approximate QUAN or $\varepsilon$ -approximate $\min\{\frac{1}{2},\frac{\varepsilon\sqrt{k}}{2}\}$ -heavy hitters with error probability $\delta$ for some sufficiently small constant $\delta$ has communication complexity $\Omega(\min\{\sqrt{k}/\varepsilon,1/\varepsilon^{2}\})$ bits.

6.1.1 The $k$ -GAP-MAJ Problem

Before proving Theorem 10, we introduce a problem we call $k$ -GAP-MAJ.

In this section we fix $\beta=1/2$ . In the $k$ -GAP-MAJ problem we have $k$ sites $S_{1},S_{2},\ldots,S_{k}$ , and each site has a bit $Z_{i}\ (1\leq i\leq k)$ such that $\mathsf{Pr}[Z_{i}=0]=\mathsf{Pr}[Z_{i}=1]=1/2$ . Let $\eth$ be the distribution of $\{Z_{1},\ldots,Z_{k}\}$ . The sites want to compute the following function.

\displaystyle\begin{array}[]{l}k\mbox{-GAP-MAJ}(Z_{1},\ldots,Z_{k})=\left\{\begin{array}[]{rl}0,&\text{if }\sum_{i\in[k]}Z_{i}\leq\beta k-\sqrt{\beta k},\\ 1,&\text{if }\sum_{i\in[k]}Z_{i}\geq\beta k+\sqrt{\beta k},\\ ,&\text{otherwise,}\end{array}\right.\end{array}

where $*$ means that the answer can be either $0$ or $1$ .

Notice that $k$ -GAP-MAJ is very similar to $k$ -APPROX-SUM: We set $\beta=1/2$ and directly assign $Z_{i}$ ’s to the $k$ sites. Also, instead of approximating the sum, we just want to decide whether the sum is large or small, up to a gap which is roughly equal to the standard deviation.

We will prove the following theorem for $k$ -GAP-MAJ.

Theorem 11

Let $\Pi$ be the transcript of any private randomness protocol for $k$ -GAP-MAJ on input distribution $\eth$ with error probability $\delta$ for some sufficiently small constant $\delta$ , then $I(Z;\Pi)=\Omega(k)$ .

Remark 1

The theorem holds for private randomness protocols, though for our applications, we only need it to hold for deterministic protocols. Allowing private randomness could be useful when the theorem is used for direct-sum types of arguments in other settings.

The following definition is essentially the same as Definition 8, but with a different setting of parameters. For convenience, we will still use the terms rare and normal. Let $\kappa_{1}$ be a constant chosen later. For a transcript $\pi$ , we define $q_{i}^{\pi}=\mathsf{Pr}_{\eth^{\prime}}[Z_{i}=1\ |\ \Pi=\pi]$ . Thus $\sum_{i\in[k]}q_{i}^{\pi}=\mathsf{E}_{\eth^{\prime}}[\sum_{i\in[k]}Z_{i}\ |\ \Pi=\pi]$ .

Definition 8

We say a transcript $\pi$ is rare⁺ if $\sum_{i\in[k]}q_{i}^{\pi}\geq\beta k+\kappa_{1}\sqrt{\beta k}$ and rare^- if $\sum_{i\in[k]}q_{i}^{\pi}\leq\beta k-\kappa_{1}\sqrt{\beta k}$ . In both cases we say $\pi$ is rare. Otherwise we say it is normal.

Let $\eth^{\prime}$ be the joint distribution of $\eth$ and the distribution of $\Pi$ ’s private randomness. The following lemma is essentially the same as Lemma 2. For completeness we still include a proof.

Lemma 16

Under the assumption of Theorem 11, $\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is normal}]\geq 1-8e^{-\kappa_{1}^{2}/4}$ .

Proof:

Set $\kappa_{2}=\kappa_{1}-1$ . We will redefine the term joker which was defined in Definition 3, with a different setting of parameters. We say $Z=\{Z_{1},Z_{2},\ldots,Z_{k}\}$ is a joker⁺ if $\sum_{i\in[k]}Z_{i}\geq\beta k+\kappa_{2}\sqrt{\beta k}$ , and a joker^- if $\sum_{i\in[k]}Z_{i}\leq\beta k-\kappa_{2}\sqrt{\beta k}$ . In both cases we say $Z$ is a joker.

First, we can apply a Chernoff bound on random variables $Z_{1},\ldots,Z_{k}$ , and obtain

\textstyle\mathsf{Pr}_{\eth}[Z\textrm{ is a joker}^{+}]=\mathsf{Pr}_{\eth}\left[\sum_{i\in[k]}Z_{i}\geq\beta k+\kappa_{2}\sqrt{\beta k}\right]\leq e^{-\kappa_{2}^{2}/3}.

Second, by Observation 1, we can apply a Chernoff bound on random variables $Z_{1},\ldots,Z_{k}$ conditioned on $\Pi$ being bad⁺,

			$\displaystyle\mathsf{Pr}_{\eth^{\prime}}[Z\textrm{ is a joker}^{+}\ \|\ \Pi\textrm{ is bad}^{+}]$
		$\displaystyle\geq$	$\displaystyle\sum_{\pi}\mathsf{Pr}_{\eth^{\prime}}\left[\Pi=\pi\ \|\ \Pi\textrm{ is bad}^{+}\right]\mathsf{Pr}_{\eth^{\prime}}\left[Z\textrm{ is a joker}^{+}\ \|\ \Pi=\pi,\Pi\textrm{ is bad}^{+}\right]$
		$\displaystyle=$	$\displaystyle\sum_{\pi}\mathsf{Pr}_{\eth^{\prime}}\left[\Pi=\pi\ \|\ \Pi\textrm{ is bad}^{+}\right]\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\geq\beta k+\kappa_{2}\sqrt{\beta k}\ \right\|\sum_{i\in[k]}q_{i}^{\pi}\geq\beta k+\kappa_{1}\sqrt{\beta k},\Pi=\pi\right]$
		$\displaystyle\geq$	$\displaystyle\sum_{\pi}\mathsf{Pr}_{\eth^{\prime}}\left[\Pi=\pi\ \|\ \Pi\textrm{ is bad}^{+}\right]\left(1-e^{-(\kappa_{1}-\kappa_{2})^{2}/3}\right)$
		$\displaystyle=$	$\displaystyle\left(1-e^{-(\kappa_{1}-\kappa_{2})^{2}/3}\right).$

Finally by Bayes’ theorem, we have that

	$\displaystyle\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is bad}^{+}]$	$\displaystyle=$	$\displaystyle\frac{\mathsf{Pr}_{\eth}[Z\textrm{ is a joker}^{+}]\cdot\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is bad}^{+}\ \|\ Z\textrm{ is a joker}^{+}]}{\mathsf{Pr}_{\eth^{\prime}}[Z\textrm{ is a joker}^{+}\ \|\ \Pi\textrm{ is bad}^{+}]}$
		$\displaystyle\leq$	$\displaystyle\frac{e^{-\kappa_{2}^{2}/3}}{1-e^{-(\kappa_{1}-\kappa_{2})^{2}/3}}.$

By symmetry (since we have set $\beta=1/2$ ), we can also show that

\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is bad}^{-}]\leq{e^{-\kappa_{2}^{2}/3}}/{(1-e^{-(\kappa_{1}-\kappa_{2})^{2}/3})}.

Therefore $\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is bad}]\leq 2{e^{-(\kappa_{1}-1)^{2}/3}}/{(1-e^{-1/3})}\leq 8e^{-\kappa_{1}^{2}/4}$ (recall that we have set $\kappa_{2}=\kappa_{1}-1$ ).

The following definition is essentially the same as Definition 4, but is for private randomness protocols.

Definition 9

We say a transcript $\pi$ is weak if $\sum_{i\in[k]}q_{i}^{\pi}(1-q_{i}^{\pi})\geq\beta k/(40c_{0})$ (for a sufficiently large constant $c_{0}$ ), and strong otherwise.

The following lemma is similar to Lemma 3, but with the new definition of a normal $\pi$ .

Lemma 17

Under the assumption of Theorem 11, $\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is normal and strong}]\geq 0.98$ .

Proof:

We first show that for a normal and weak transcript $\pi$ , there exists a universal constant $c$ such that

	$\displaystyle\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\leq\beta k-\kappa_{1}\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$	$\displaystyle\geq$	$\displaystyle c\cdot e^{-60c_{0}\kappa_{1}^{2}},$
	$\displaystyle\text{and \quad}\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\geq\beta k+\kappa_{1}\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$	$\displaystyle\geq$	$\displaystyle c\cdot e^{-60c_{0}\kappa_{1}^{2}}.$

We only need to prove the first inequality. The second will follow by symmetry, since we set $\beta=1/2$ . For a good and weak $\Pi=\pi$ , we have

\displaystyle\mathsf{Var}_{\eth^{\prime}}\left(\sum_{i\in[k]}Z_{i}\ |\ \Pi=\pi\right)=\sum_{i\in[k]}\mathsf{Var}_{\eth^{\prime}}(Z_{i}\ |\ \Pi=\pi)\geq\beta k/(40c_{0}).

Set $\kappa_{3}=2\kappa_{1}$ . By Fact 1 we have for a universal constant $c$ ,

			$\displaystyle\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\geq\sum_{i\in[k]}q_{i}^{\pi}+\kappa_{3}\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$
		$\displaystyle\geq$	$\displaystyle c\cdot e^{-\frac{(\kappa_{3}\sqrt{\beta k})^{2}}{3\cdot\beta k/(40c_{0})}}\geq c\cdot e^{-60c_{0}\kappa_{1}^{2}}.$

Together with the fact that $\pi$ is good, we obtain

			$\displaystyle\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\geq\beta k+\kappa_{1}\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$
		$\displaystyle=$	$\displaystyle\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}\geq\beta k+(\kappa_{3}-\kappa_{1})\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$
		$\displaystyle\geq$	$\displaystyle\textstyle\mathsf{Pr}_{\eth^{\prime}}\left[\left.\sum_{i\in[k]}Z_{i}-\sum_{i\in[k]}q_{i}^{\pi}\geq\kappa_{3}\sqrt{\beta k}\ \right\|\ \Pi=\pi\right]$
		$\displaystyle\geq$	$\displaystyle c\cdot e^{-60c_{0}\kappa_{1}^{2}}.$

Now set $\kappa_{1}=\sqrt{4\ln 800}$ . Suppose conditioned on $\Pi$ being normal, it is weak with probability more than $0.01$ . Then the error probability of the protocol (taken over the distribution $\eth^{\prime}$ ) is at least

(1-8e^{-\kappa_{1}^{2}/4})\cdot 0.01\cdot c\cdot e^{-60c_{0}\kappa_{1}^{2}}\geq\delta,

for a sufficiently small constant $\delta$ , violating the success guarantee of Theorem 11. Therefore with probability at least

(1-8e^{-\kappa_{1}^{2}/4})\cdot(1-0.99)\geq 0.98,

$\Pi$ is both normal and strong.

Proof:

(for Theorem 11) Recall that for a transcript $\pi$ , we have defined $q_{i}^{\pi}=\mathsf{Pr}_{\eth^{\prime}}[Z_{i}=1\ |\ \Pi=\pi]$ . Let $p_{i}^{\pi}=\min\{q_{i}^{\pi},1-q_{i}^{\pi}\}$ , thus $p_{i}^{\pi}\in[0,1/2]$ . We will omit the superscript $\pi$ in $p_{i}^{\pi}$ when it is clear from the context.

For a strong $\pi$ , we have $\sum_{i\in[k]}p_{i}\cdot 1/2\leq\sum_{i\in[k]}p_{i}(1-p_{i})<\frac{\beta k}{40c_{0}}$ . Thus $\sum_{i\in[k]}p_{i}<\frac{\beta k}{20c_{0}}$ . For each $p_{i}$ , if $p_{i}<\frac{\beta}{16c_{0}}<1/2$ (for a sufficiently large constant $c_{0}$ ), then $H_{b}(p_{i})<H_{b}\left(\frac{\beta}{16c_{0}}\right)$ (recall that $H_{b}(\cdot)$ is the binary entropy function). Otherwise, it holds that $H_{b}(p_{i})=p_{i}\log\frac{1}{p_{i}}+(1-p_{i})\log\frac{1}{1-p_{i}}\leq p_{i}\log\frac{16c_{0}}{\beta}+2(1-p_{i})p_{i}$ , since $\log\frac{1}{1-p_{i}}=p_{i}+p_{i}^{2}/2+p_{i}^{3}/3+\ldots\leq 2p_{i}$ if $p_{i}\leq 1/2$ . Thus we have

			$\displaystyle\sum_{i\in[k]}H(Z_{i}\ \|\ \Pi=\pi)=\sum_{i\in[k]}H_{b}(p_{i})$
		$\displaystyle\leq$	$\displaystyle\sum_{i\in[k]}\max\left\{H_{b}\left(\frac{\beta}{16c_{0}}\right),\ p_{i}\log\frac{16c_{0}}{\beta}+2(1-p_{i})p_{i}\right\}$
		$\displaystyle\leq$	$\displaystyle\max\left\{k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right),\ \sum_{i\in[k]}p_{i}\cdot\log\frac{16c_{0}}{\beta}+2\sum_{i\in[k]}p_{i}(1-p_{i})\right\}$
		$\displaystyle<$	$\displaystyle\max\left\{k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right),\ \frac{\beta k}{20c_{0}}\cdot\log\frac{16c_{0}}{\beta}+\frac{\beta k}{20c_{0}}\right\}$
		$\displaystyle=$	$\displaystyle k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right).$

Therefore, if $\mathsf{Pr}_{\eth^{\prime}}[\Pi\textrm{ is strong}]\geq 0.98$ , then

			$\displaystyle\sum_{i\in[k]}H(Z_{i}\ \|\ \Pi)$
		$\displaystyle=$	$\displaystyle\sum_{\pi:\pi\textrm{ is strong}}\left(\mathsf{Pr}_{\eth^{\prime}}[\Pi=\pi]\sum_{i\in[k]}H(Z_{i}\ \|\ \Pi=\pi)\right)+\sum_{\pi:\pi\textrm{ is weak}}\left(\mathsf{Pr}_{\eth^{\prime}}[\Pi=\pi]\sum_{i\in[k]}H(Z_{i}\ \|\ \Pi=\pi)\right)$
		$\displaystyle\leq$	$\displaystyle 0.98\cdot k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right)+0.02\cdot k\cdot 1$
		$\displaystyle\leq$	$\displaystyle k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right)+0.02k.$

Now, for a large enough constant $c_{0}$ and $\beta=1/2$ ,

I(Z;\Pi)\geq\sum_{i\in[k]}I(Z_{i};\Pi)\geq\sum_{i\in[k]}(H(Z_{i})-H(Z_{i}\ |\ \Pi))\geq k\cdot H_{b}(\beta)-k\cdot H_{b}\left(\frac{\beta}{16c_{0}}\right)-0.02k\geq\Omega(k).

6.1.2 Proof of Theorem 10

Proof:

We first prove the theorem for QUAN. In the case that $k\geq 1/\varepsilon^{2}$ , we prove an $\Omega(1/\varepsilon^{2})$ information complexity lower bound. We prove this by a simple reduction from $k$ -GAP-MAJ. We can assume $k=1/\varepsilon^{2}$ since if $k>1/\varepsilon^{2}$ then we can just give inputs to the first $1/\varepsilon^{2}$ sites. Set $\beta=1/2$ . Given a random input $Z_{1},Z_{2},\ldots,Z_{k}$ of $k$ -GAP-MAJ chosen from distribution $\eth$ , we simply give $Z_{i}$ to site $S_{i}$ . It is easy to observe that a protocol that computes $\varepsilon/2$ -approximate QUAN on $A=\{Z_{1},Z_{2},\ldots,Z_{k}\}$ with error probability $\delta$ also computes $k$ -GAP-MAJ on input distribution $\eth$ with error probability $\delta$ , since the answer to $k$ -GAP-MAJ is simply the answer to $\frac{1}{2}$ -quantile. The $\Omega(1/\varepsilon^{2})$ lower bound follows from Theorem 11.

In the case that $k<1/\varepsilon^{2}$ , we prove an $\Omega(\sqrt{k}/\varepsilon)$ information complexity lower bound. We again perform a reduction from $k$ -GAP-MAJ. Set $\beta=1/2$ . The reduction works as follows. We are given $\ell=1/(\varepsilon\sqrt{k})$ independent copies of $k$ -GAP-MAJ with $Z^{1},Z^{2},\ldots,Z^{\ell}$ being the inputs, where $Z^{i}=\{Z^{i}_{1},Z^{i}_{2},\ldots,Z^{i}_{k}\}\in\{0,1\}^{k}$ is chosen from distribution $\eth$ . We construct an input for QUAN by giving the $j$ -th site the item set $A_{j}=\{Z^{1}_{j},2+Z^{2}_{j},4+Z^{3}_{j},\ldots,2(\ell-1)+Z^{\ell}_{j}\}$ . It is not difficult to observe that a protocol that computes $\varepsilon/2$ -approximate QUAN on the set $A=\{A_{1},A_{2},\ldots,A_{j}\}$ with error probability $\delta$ also computes the answer to each copy of $k$ -GAP-MAJ on distribution $\eth$ with error probability $\delta$ , simply by returning $(X_{i}-2(i-1))$ for the $i$ -th copy of $k$ -GAP-MAJ, where $X_{i}$ is the $\varepsilon/2$ -approximate $\frac{i-1/2}{\ell}$ -quantile.

On the other hand, any protocol that computes each of the $\ell$ independent copies of $k$ -GAP-MAJ correctly with error probability $\delta$ for a sufficiently small constant $\delta$ has information complexity $\Omega(\sqrt{k}/\varepsilon)$ . This is simply because for any transcript $\Pi$ , by Theorem 11, independence and the chain rule we have that

I(Z^{1},Z^{2},\ldots,Z^{\ell};\Pi)\geq\sum_{i\in[\ell]}I(Z^{i};\Pi)\geq\Omega(\ell k)\geq\Omega(\sqrt{k}/\varepsilon).

(50)

The proof for heavy hitters is done by essentially the same reduction as that for QUAN. In the case that $k=1/\varepsilon^{2}$ (or $k\geq 1/\varepsilon^{2}$ in general), a protocol that computes $\varepsilon/2$ -approximate $\frac{1}{2}$ -heavy hitters on $A=\{Z_{1},Z_{2},\ldots,Z_{k}\}$ with error probability $\delta$ also computes $k$ -GAP-MAJ on input distribution $\eth$ with error probability $\delta$ . In the case that $k<1/\varepsilon^{2}$ , it also holds that a protocol that computes $\varepsilon/2$ -approximate $\frac{\varepsilon\sqrt{k}}{2}$ -heavy hitters on the set $A=\{A_{1},A_{2},\ldots,A_{j}\}$ where $A_{j}=\{Z^{1}_{j},2+Z^{2}_{j},4+Z^{3}_{j},\ldots,2(\ell-1)+Z^{\ell}_{j}\}$ with error probability $\delta$ also computes the answer to each copy of $k$ -GAP-MAJ on distribution $\eth$ with error probability $\delta$ .

6.2 Entropy Estimation

We are given a set $A=\{(e_{1},a_{1}),(e_{2},a_{2}),\ldots,(e_{m},a_{m})\}$ where each $e_{k}\ (k\in[m])$ is drawn from the universe $[N]$ , and $a_{k}\in\{+1,-1\}$ denotes an insertion or a deletion of item $e_{k}$ . The entropy estimation problem (ENTROPY) asks for the value $H(A)=\sum_{j\in[N]}(\left|f_{j}\right|/L)\log(L/\left|f_{j}\right|)$ where $f_{j}=\sum_{k:e_{k}=j}a_{k}$ and $L=\sum_{j\in[N]}\left|f_{j}\right|$ . In the $\varepsilon$ -approximate ENTROPY problem, the items in the set $A$ are distributed among $k$ sites who want to compute a value $\tilde{H}(A)$ for which $\left|\tilde{H}(A)-H(A)\right|\leq\varepsilon$ . In this section we prove the following theorem.

Theorem 12

Any randomized protocol that computes $\varepsilon$ -approximate ENTROPY with error probability at most $\delta$ for some sufficiently small constant $\delta$ has communication complexity ${\Omega}(k/\varepsilon^{2})$ .

Proof:

As with $F_{2}$ , we prove the lower bound for the ENTROPY problem by a reduction from $k$ -BTX. Given a random input $B$ for $k$ -BTX according to distribution $\nu$ with $n=\gamma^{2}k^{2}$ for some parameter $\gamma=\log^{-d}(k/\varepsilon)$ for large enough constant $d$ , we construct an input for ENTROPY as follows. Each block $j\in[1/\varepsilon^{2}]$ in $k$ -BTX corresponds to one coordinate item $e_{j}$ in the vector for ENTROPY; so we have in total $1/\varepsilon^{2}$ items in the entropy vector. The $k$ sites first use shared randomness to sample $\gamma^{2}k^{2}/\varepsilon^{2}$ random $\pm 1$ values for each coordinate across all blocks in $B$ ⁹⁹9By Newman’s theorem (cf. [43], Chapter 3) we can get rid of the public randomness by increasing the total communication complexity by no more than an additive $O(\log(\gamma k/\varepsilon))$ factor which is negligible in our proof.. Let $\{R_{1}^{1},R_{1}^{2},\ldots,R_{\gamma^{2}k^{2}}^{1/\varepsilon^{2}}\}$ be these random $\pm 1$ values. Each site looks at each of its bits $B_{i,\ell}^{j}\ (i\in[k],\ell\in\gamma^{2}k^{2},j\in[1/\varepsilon^{2}])$ , and generates an item $(e_{j},R_{\ell}^{j})$ (recall that $R_{\ell}^{j}$ denotes insertion or deletion of the item $e_{j}$ ) if $B_{i,\ell}^{j}=1$ . Call the resulting input distribution $\nu^{\prime}$ .

We call an item in group $G_{P}$ if the $k$ -XOR instance in the corresponding block is a $00$ -instance; and in group $G_{Q}$ if it is a $11$ -instance; in group $G_{U}$ if it is a $01$ -instance or a $10$ -instance. Group $G_{U}$ is further divided to two subgroups $G_{U_{1}}$ and $G_{U_{2}}$ , containing all $10$ -instance and all $01$ -instance, respectively. Let $P,Q,U,U_{1},U_{2})$ be the cardinalities of these groups. Now we consider the frequency of each item type.

1.

For an item $e_{j}\in G_{P}$ , its frequency $f_{j}$ is distributed as follows: we choose a value $i$ from the binomial distribution on $n$ values each with probability $1/2$ , then we take the sum $\kappa_{j}$ of $i$ i.i.d. $\pm 1$ random variables. We can thus write $|f_{j}|=|\kappa_{j}|$ .
2.

For an item $e_{j}\in G_{Q}$ , its frequency $f_{j}$ is distributed as follows: we choose a value $i$ from the binomial distribution on $n$ values each with probability $1/2$ , then we take the sum $\kappa_{j}$ of $i$ i.i.d. $\pm 1$ random variables. Then we add the value $R_{\ell^{*}}^{j}\cdot k$ , where $\ell^{*}$ is the index of the special column in block $j$ . We can thus write $|f_{j}|$ as $|k+R_{\ell^{*}}^{j}\cdot\kappa_{j}|$ . By a Chernoff-Hoeffding bound, with probability $1-2e^{-\lambda^{2}/2}$ , we have $\left|\kappa_{j}\right|\leq\lambda\gamma k$ . We choose $\lambda=\log(k/\varepsilon)$ , and thus $\lambda\gamma=o(1)$ . Therefore $\kappa_{j}$ will not affect the sign of $f_{j}$ for any $j$ (by a union bound) and we can write $\left|f_{j}\right|=k+R_{\ell^{*}}^{j}\cdot\kappa_{j}$ . Since $\kappa_{j}$ is symmetric about $0$ and $R_{\ell^{*}}^{j}$ is a random $\pm 1$ variable, we can simply drop $R_{\ell^{*}}^{j}$ and write $\left|f_{j}\right|=k+\kappa_{j}$ .
3.

For an item $e_{j}\in G_{U}$ , its frequency $f_{j}$ is distributed as follows: we choose a value $i$ from the binomial distribution on $n$ values each with probability $1/2$ , then we take the sum $\kappa_{j}$ of $i$ i.i.d. $\pm 1$ random variables. Then we add the value $R_{\ell^{*}}^{j}\cdot k/2$ , where $\ell^{*}$ is the index of the special column in block $j$ . We can thus write $|f_{j}|$ as $|k/2+R_{\ell^{*}}^{j}\cdot\kappa_{j}$ . As in the previous case, with probability $1-2e^{-\lambda^{2}/2}$ , $\kappa_{j}$ will not affect the sign of $f_{j}$ and we can write $\left|f_{j}\right|=k/2+\kappa_{j}$ .

By a union bound, with error probability at most $\delta_{1}=1/\varepsilon^{2}\cdot 2e^{-\lambda^{2}/2}=1-o(1)$ , each $\kappa_{j}\ (e_{j}\in G_{Q}\cup G_{U})$ will not affect the sign of the corresponding $f_{j}$ . Moreover, by another Chernoff bound we have that with error probability $\delta_{2}=10e^{-c_{0}^{2}/3}$ , $P,Q,U_{1},U_{2}$ are equal to $1/4\varepsilon^{2}\pm c_{0}/\varepsilon$ , and $U=1/(2\varepsilon^{2})\pm c_{0}/\varepsilon$ . Here $\delta_{2}$ can be sufficiently small if we set constant $c_{0}$ sufficiently large. Thus we have that with arbitrary small constant error $\delta_{0}=\delta_{1}+\delta_{2}$ , all the concentration results claimed above hold. For simplicity we neglect this part of error since it can be arbitrarily small and will not affect any of the analysis. In the rest of this section we will ignore arbitrarily small errors and drop some lower order terms as long as such operations will not affect any the analysis.

The analysis of the next part is similar to that for our $F_{2}$ lower bound, where we end up computing $F_{2}$ on three different vectors. Let us calculate $H_{0},H_{1}$ and $H_{2}$ , which stand for the entropies of all $k$ -sites, the first $k/2$ sites and the second $k/2$ sites, respectively. Then we show that using $H_{0},H_{1}$ and $H_{2}$ we can estimate $U$ well, and thus compute $k$ -BTX correctly with an arbitrarily small constant error. Thus if there is a protocol for ENTROPY on distribution $\nu^{\prime}$ then we obtain a protocol for $k$ -BTX on distribution $\nu$ with the same communication complexity, completing the reduction and consequently proving Theorem 12.

Before computing $H_{0},H_{1}$ and $H_{2}$ , we first compute the total number $L$ of items. We can write

	$\displaystyle L$	$\displaystyle=$	$\displaystyle\sum_{e_{j}\in G_{P}}\left\|f_{j}\right\|+\sum_{e_{j}\in G_{Q}}\left\|f_{j}\right\|+\sum_{e_{j}\in G_{U}}\left\|f_{j}\right\|$		(51)
		$\displaystyle=$	$\displaystyle Q\cdot k+U\cdot k/2+\sum_{e_{j}\in G_{p}}\left\|\kappa_{j}\right\|+\sum_{e_{j}\in G_{Q}\cup G_{U}}\kappa_{j}.$		(51)

The absolute value of the fourth term in (51) can be bounded by $O(\gamma k/\varepsilon)$ with arbitrarily large constant probability, using a Chernoff-Hoeffding bound, which will be $o(\varepsilon L)$ and thus can be dropped. For the third term, by Chebyshev’s inequality we can assume (by increasing the constant in the big-Oh) that with arbitrarily large constant probability, $\sum_{e_{j}\in G_{p}}|\kappa_{j}|=(1\pm\varepsilon)\cdot 1/(4\varepsilon^{2})\cdot{\bf E}[|\kappa_{j}|]$ , where ${\bf E}[|\kappa_{j}|]=\Theta(\gamma k)$ follows by approximating the binomial distribution by a normal distribution (or, e.g., Khintchine’s inequality). Let $z_{1}={\bf E}[|\kappa_{j}|]$ be a value which can be computed exactly. Then, $\sum_{e_{j}\in G_{p}}|\kappa_{j}|=1/(4\varepsilon^{2})\cdot z_{1}\pm O(\gamma k/\varepsilon)=z_{1}/(4\varepsilon^{2})\pm o(\varepsilon L)$ , and so we can drop the additive $o(\varepsilon L)$ term.

Finally, we get,

\displaystyle L

\displaystyle=

\displaystyle Q\cdot k+U\cdot k/2+r_{1}

(52)

where $r_{1}=z_{1}/(4\varepsilon^{2})=\Theta(\gamma k/\varepsilon^{2})$ is a value that can be computed by any site without any comunication.

Let $p_{j}=\left|f_{j}\right|/L\ (j\in[1/\varepsilon^{2}])$ . We can write $H$ as follows.

	$\displaystyle H_{0}$	$\displaystyle=$	$\displaystyle\sum_{e_{j}\in P}p_{j}\log(1/p_{j})+\sum_{e_{j}\in Q}p_{j}\log(1/p_{j})+\sum_{e_{j}\in U}p_{j}\log(1/p_{j}).$		(53)
		$\displaystyle=$	$\displaystyle\log L-S/L$		(53)

where

\displaystyle S

\displaystyle=

\displaystyle\sum_{e_{j}\in P}\left|f_{j}\right|\log\left|f_{j}\right|+\sum_{e_{j}\in Q}\left|f_{j}\right|\log\left|f_{j}\right|+\sum_{e_{j}\in U}\left|f_{j}\right|\log\left|f_{j}\right|

(54)

We consider the three summands in (54) one by one. For the second term in (54), we have

$\displaystyle\sum_{e_{j}\in Q}\left\|f_{j}\right\|\log\left\|f_{j}\right\|$	$\displaystyle=$	$\displaystyle\sum_{e_{j}\in Q}(k+\kappa_{j})\log(k+\kappa_{j})$	(55)
	$\displaystyle=$	$\displaystyle Q\cdot k\log k+k\sum_{e_{j}\in Q}\log(1+\kappa_{j}/k)$
	$\displaystyle=$	$\displaystyle Q\cdot k\log k\pm O(\sum_{e_{j}\in Q}\kappa_{j}).$

The second term in (55) is at most $o(\varepsilon Q\cdot\gamma k)=o(k/eps)$ , and can be dropped. By a similar analysis we can obtain that the third term in (54) is (up to an $o(k/\varepsilon)$ term)

\displaystyle\sum_{e_{j}\in U}\left|f_{j}\right|\log\left|f_{j}\right|=U\cdot(k/2)\log(k/2).

(56)

Now consider the first term. We have

$\displaystyle\sum_{e_{j}\in P}\left\|f_{j}\right\|\log\left\|f_{j}\right\|$	$\displaystyle=$	$\displaystyle\sum_{e_{j}\in P}\left\|\kappa_{j}\right\|\log\left\|\kappa_{j}\right\|$	(57)
	$\displaystyle=$	$\displaystyle(1/4\varepsilon^{2}\pm O(1/\varepsilon))\cdot\mathsf{E}[\left\|\kappa_{j}\right\|\log\left\|\kappa_{j}\right\|]$
	$\displaystyle=$	$\displaystyle(1/4\varepsilon^{2})\cdot\mathsf{E}[\left\|\kappa_{j}\right\|\log\left\|\kappa_{j}\right\|]\pm O(1/\varepsilon)\cdot\mathsf{E}[\left\|\kappa_{j}\right\|\log\left\|\kappa_{j}\right\|]$

where $z_{2}=\mathsf{E}[\left|\kappa_{j}\right|\log\left|\kappa_{j}\right|]=O(\gamma k\log k)$ can be computed exactly. Then the second term in (57) is at most $O(\gamma k\log k/\varepsilon)=o(k/\varepsilon)$ , and thus can be dropped. Let $r_{2}=(1/4\varepsilon^{2})\cdot z_{2}=O(\gamma k\log k/\varepsilon^{2})$ . By Equations (53), (52), (54), (55), (56), (57) we can write

\displaystyle H_{0}

\displaystyle=

\displaystyle\log(Q\cdot k+U\cdot k/2+r_{1})+\frac{Q\cdot k\log k+U\cdot(k/2)\log(k/2)+r_{2}}{Q\cdot k+U\cdot k/2+r_{1}}.

(58)

Let $U=1/(2\varepsilon^{2})+U^{\prime}$ and $Q=1/4\varepsilon^{2}+Q^{\prime}$ , and thus $U^{\prime}=O(1/\varepsilon)$ and $Q^{\prime}=O(1/\varepsilon)$ . Next we convert the RHS of (58) to a linear function of $U^{\prime}$ and $Q^{\prime}$ .

$\displaystyle H_{0}$	$\displaystyle=$	$\displaystyle\log(k/(2\varepsilon^{2})+Q^{\prime}k+U^{\prime}k/2+r_{1})+\frac{\left(\frac{1}{4\varepsilon^{2}}+Q^{\prime}\right)\cdot k\log k+\left(\frac{1}{(2\varepsilon^{2})}+U^{\prime}\right)\cdot\frac{k}{2}\log\frac{k}{2}+r_{2}}{k/(2\varepsilon^{2})+Q^{\prime}k+U^{\prime}k/2+r_{1}}$	(61)
	$\displaystyle=$	$\displaystyle\log(k/(2\varepsilon^{2})+r_{1})+\log\left(1+(U^{\prime}+2Q^{\prime})\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}\right)$
		$\displaystyle+\ \left(\left(\frac{1}{4\varepsilon^{2}}+Q^{\prime}\right)\cdot k\log k+\left(\frac{1}{(2\varepsilon^{2})}+U^{\prime}\right)\cdot\frac{k}{2}\log\frac{k}{2}+r_{2}\right)\cdot\frac{2\varepsilon^{2}}{k+2\varepsilon^{2}r_{1}}\cdot$
		$\displaystyle\left(1-(U^{\prime}+2Q^{\prime})\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}\right)$
		$\displaystyle\pm\ O(\varepsilon^{2})$
	$\displaystyle=$	$\displaystyle\log(k/(2\varepsilon^{2})+r_{1})+(U^{\prime}+2Q^{\prime})\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}$
		$\displaystyle+\ \left(\left(\frac{k(2\log{k}-1)}{4\varepsilon^{2}}+r_{2}\right)+Q^{\prime}\cdot k\log k+U^{\prime}\cdot\frac{k}{2}\log\frac{k}{2}\right)\cdot\frac{2\varepsilon^{2}}{k+2\varepsilon^{2}r_{1}}\cdot$
		$\displaystyle\quad\left(1-U^{\prime}\cdot\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}-Q^{\prime}\cdot\frac{2\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}\right)$
		$\displaystyle\pm\ O(\varepsilon^{2})$
	$\displaystyle=$	$\displaystyle\alpha_{1}+\alpha_{2}U^{\prime}+\alpha_{3}Q^{\prime},$	(62)

up to $o(\varepsilon)$ factors (see below for discussion), where

$\displaystyle\alpha_{1}$	$\displaystyle=$	$\displaystyle\log(k/(2\varepsilon^{2})+r_{1})+\left(\frac{k(2\log{k}-1)}{4\varepsilon^{2}}+r_{2}\right)\cdot\frac{2\varepsilon^{2}}{k+2\varepsilon^{2}r_{1}},$
$\displaystyle\alpha_{2}$	$\displaystyle=$	$\displaystyle\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}+\left(\frac{k}{2}\log\frac{k}{2}-\left(\frac{k(2\log{k}-1)}{4\varepsilon^{2}}+r_{2}\right)\cdot\frac{\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}\right)\cdot\frac{2\varepsilon^{2}}{k+2\varepsilon^{2}r_{1}},$
$\displaystyle\alpha_{3}$	$\displaystyle=$	$\displaystyle\frac{2\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}+\left(k\log k-\left(\frac{k(2\log{k}-1)}{4\varepsilon^{2}}+r_{2}\right)\cdot\frac{2\varepsilon^{2}}{1+2\varepsilon^{2}r_{1}/k}\right)\cdot\frac{2\varepsilon^{2}}{k+2\varepsilon^{2}r_{1}},$	(63)

From (61) to (61) we use the fact that $1/(1+\varepsilon)=1-\varepsilon+O(\varepsilon^{2})$ . From (61) to (61) we use the fact that $\log(1+\varepsilon)=\varepsilon+O(\varepsilon^{2})$ . From (61) to (62) we use the fact that all terms in the form of $U^{\prime}Q^{\prime},{U^{\prime}}^{2},{Q^{\prime}}^{2}$ are at most $\pm o(\varepsilon)$ (we are assuming $O(\varepsilon^{2}\log k)=o(\varepsilon)$ which is fine since we are neglecting polylog $(N)$ factors), therefore we can drop all of them together with the other $\pm O(\varepsilon^{2})$ terms, and consequently obtain a linear function on $U^{\prime}$ and $Q^{\prime}$ .

Next we calculate $H_{1}$ , and the calculation of $H_{2}$ will be exactly the same. The values $t_{1},t_{2}$ used in the following expressions are essentially the same as $r_{1},r_{2}$ used for calculating $H_{0}$ , with $t_{1}=\Theta(\gamma k)$ and $t_{2}=O(\gamma k\log k/\varepsilon^{2})$ . Set $U^{\prime}_{1}=U_{1}-1/4\varepsilon^{2}$ and $U^{\prime}_{2}=U_{2}-1/4\varepsilon^{2}$ .

$\displaystyle H_{1}$	$\displaystyle=$	$\displaystyle\log((Q+U_{1})k/2+t_{1})+\frac{Q\cdot\frac{k}{2}\log\frac{k}{2}+U_{1}\cdot\frac{k}{2}\log\frac{k}{2}+t_{2}}{(Q+U_{1})k/2+t_{1}}$	(64)
	$\displaystyle=$	$\displaystyle\log(k/(4\varepsilon^{2})+(Q^{\prime}+U^{\prime}_{1})k/2+t_{1})+\frac{Q^{\prime}\cdot\frac{k}{2}\log\frac{k}{2}+U^{\prime}_{1}\cdot\frac{k}{2}\log\frac{k}{2}+\frac{k\log(k/2)}{4\varepsilon^{2}}+t_{2}}{k/(4\varepsilon^{2})+(Q^{\prime}+U^{\prime}_{1})k/2+t_{1}}$
	$\displaystyle=$	$\displaystyle\log(k/(4\varepsilon^{2})+t_{1})+(Q^{\prime}+U^{\prime}_{1})\frac{2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}$
		$\displaystyle+\left(Q^{\prime}\cdot\frac{k}{2}\log\frac{k}{2}+U^{\prime}_{1}\cdot\frac{k}{2}\log\frac{k}{2}+\frac{k\log(k/2)}{4\varepsilon^{2}}+t_{2}\right)\cdot\frac{4\varepsilon^{2}}{k+4\varepsilon^{2}t_{1}}\cdot\left(1-\frac{Q^{\prime}\cdot 2\varepsilon^{2}+U^{\prime}_{1}\cdot 2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}\right)$
	$\displaystyle=$	$\displaystyle\beta_{1}+\beta_{2}U^{\prime}_{1}+\beta_{3}Q^{\prime}.$

where

$\displaystyle\beta_{1}$	$\displaystyle=$	$\displaystyle\log(k/(4\varepsilon^{2})+t_{1})+\left(\frac{k\log(k/2)}{4\varepsilon^{2}}+t_{2}\right)\cdot\frac{4\varepsilon^{2}}{k+4\varepsilon^{2}t_{1}},$
$\displaystyle\beta_{2}$	$\displaystyle=$	$\displaystyle\frac{2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}+\left(\frac{k}{2}\log\frac{k}{2}-\left(\frac{k\log(k/2)}{4\varepsilon^{2}}+t_{2}\right)\frac{2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}\right)\frac{4\varepsilon^{2}}{k+4\varepsilon^{2}t_{1}},$
$\displaystyle\beta_{3}$	$\displaystyle=$	$\displaystyle\frac{2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}+\left(\frac{k}{2}\log\frac{k}{2}-\left(\frac{k\log(k/2)}{4\varepsilon^{2}}+t_{2}\right)\frac{2\varepsilon^{2}}{1+4\varepsilon^{2}t_{1}/k}\right)\frac{4\varepsilon^{2}}{k+4\varepsilon^{2}t_{1}}.$	(65)

By the same calculation we can obtain the following equation for $H_{2}$ .

\displaystyle H_{2}

\displaystyle=

\displaystyle\beta_{1}+\beta_{2}U^{\prime}_{2}+\beta_{3}Q^{\prime}.

(66)

Note that $U^{\prime}=U^{\prime}_{1}+U^{\prime}_{2}$ . Combining (64) and (66) we have

\displaystyle H_{1}+H_{2}

\displaystyle=

\displaystyle 2\beta_{1}+\beta_{2}U^{\prime}+2\beta_{3}Q^{\prime}.

(67)

It is easy to verify that Equations (62) and (67) are linearly independent: by direct calculation (notice that $r_{1},r_{2}$ are lower order terms) we obtain $\alpha_{2}=\frac{\varepsilon^{2}}{2}(1\pm o(1))$ and $\alpha_{3}=3\varepsilon^{2}(1\pm o(1)$ . Therefore $\alpha_{2}/\alpha_{3}=(1\pm o(1))/6$ . Similarly we can obtain $\beta_{2}/2\beta_{3}=(1\pm o(1))/2$ . Therefore the two equations are linearly independent. Furthermore, we can compute all the coefficients $\alpha_{1},\alpha_{2},\alpha_{3},\beta_{1},\beta_{2},\beta_{3}$ up to a $(1\pm o(\varepsilon))$ factor. Thus if we have $\sigma\varepsilon$ additive approximations of $H_{0},H_{1},H_{2}$ for a sufficient small constant $\sigma$ , then we can estimate $U^{\prime}$ (and thus $U$ ) up to an additive error of $\sigma^{\prime}/\varepsilon$ for a sufficiently small constant $\sigma^{\prime}$ by Equation (62) and (67), and therefore $k$ -BTX. This completes the proof.

6.3 $\ell_{p}$ for any constant $p\geq 1$

Consider an $n$ -dimensional vector $x$ with integer entries. It is well-known that for a vector $v$ of $n$ i.i.d. $N(0,1)$ random variables that $\langle v,x\rangle\sim N(0,\|x\|_{2}^{2})$ . Hence, for any real $p>0$ , ${\bf E}[|\langle v,x\rangle|^{p}]=\|x\|_{2}^{p}G_{p}$ , where $G_{p}>0$ is the $p$ -th moment of the standard half-normal distribution (see [1] for a formula for these moments in terms of confluent hypergeometric functions). Let $r=O(\varepsilon^{-2})$ , and $v^{1},\ldots,v^{r}$ be independent $n$ -dimensional vectors of i.i.d. $N(0,1)$ random variables. Let $y_{j}=\langle v^{j},x\rangle/G_{p}^{1/p}$ , so that $y=(y_{1},\ldots,y_{r})$ . By Chebyshev’s inequality for $r=O(\varepsilon^{-2})$ sufficiently large, $\|y\|_{p}^{p}=(1\pm\varepsilon/3)\|x\|_{2}^{p}$ with probability at least $1-c$ for an arbitrarily small constant $c>0$ .

We thus have the following reduction which shows that estimating $\ell_{p}$ up to a $(1+\varepsilon)$ -factor requires communication complexity ${\Omega}(k/\varepsilon^{2})$ for any $p>0$ . Let the $k$ parties have respective inputs $x^{1},\ldots,x^{k}$ , and let $x=\sum_{i=1}^{k}x^{i}$ . The parties use the shared randomness to choose shared vectors $v^{1},\ldots,v^{r}$ as described above. For $i=1,\ldots,k$ and $j=1,\ldots,r$ , let $y^{i}_{j}=\langle v^{j},x^{i}\rangle/G_{p}^{1/p}$ , so that $y^{i}=(y^{i}_{1},\ldots,y^{i}_{r})$ . Let $y=\sum_{i=1}^{k}y^{i}$ . By the above, $\|y\|_{p}^{p}=(1\pm\varepsilon/3)\|x\|_{2}^{p}$ with probability at least $1-c$ for an arbitrarily small constant $c>0$ . We note that the entries of the $v^{i}$ can be discretized to $O(\log n)$ bits, changing the $p$ -norm of $y$ by only a $(1\pm O(1/n))$ factor, which we ignore.

Hence, given a randomized protocol for estimating $\|y\|_{p}^{p}$ up to a $(1+\varepsilon/3)$ factor with probability $1-\delta$ , and given that the parties have respective inputs $y^{1},\ldots,y^{k}$ , this implies a randomized protocol for estimating $\|x\|_{2}^{p}$ up to a $(1\pm\varepsilon/3)\cdot(1\pm\varepsilon/3)=(1\pm\varepsilon)$ factor with probability at least $1-\delta-c$ , and hence a protocol for estimating $\ell_{2}$ up to a $(1\pm\varepsilon)$ factor with this probability. The communication complexity of the protocol for $\ell_{2}$ is the same as that for $\ell_{p}$ . By our communication lower bound for estimating $\ell_{2}$ (in fact, for estimating $F_{2}$ in which all coordinates of $x$ are non-negative), this implies the following theorem.

Theorem 13

The randomized communication complexity of approximating the $\ell_{p}$ -norm, $p\geq 1$ , up to a factor of $1+\varepsilon$ with constant probability, is ${\Omega}(k/\varepsilon^{2})$ .

Acknowledgements

We would like to thank Elad Verbin for many helpful discussions, in particular, for helping us with the $F_{0}$ lower bound, which was discovered in joint conversations with him. We also thank Amit Chakrabarti and Oded Regev for helpful discussions, as well as the anonymous referees for useful comments. Finally, we thank the organizers of the Synergies in Lower Bounds workshop that took place in Aarhus for bringing the authors together.

References

[1] http://en.wikipedia.org/wiki/Normal_distribution.
[2] Open problems in data streams and related topics. http://www.cse.iitk.ac.in/users/sganguly/data-stream-probs.pdf, 2006.
[3] N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. In Proc. ACM Symposium on Theory of Computing, 1996.
[4] C. Arackaparambil, J. Brody, and A. Chakrabarti. Functional monitoring without monotonicity. In Proc. International Colloquium on Automata, Languages, and Programming, 2009.
[5] B. Babcock and C. Olston. Distributed top-k monitoring. In Proc. ACM SIGMOD International Conference on Management of Data, 2003.
[6] Z. Bar-Yossef. The complexity of massive data set computations. PhD thesis, University of California at Berkeley, 2002.
[7] Z. Bar-Yossef, T. S. Jayram, R. Kumar, and D. Sivakumar. An information statistics approach to data stream and communication complexity. J. Comput. Syst. Sci., 68:702–732, June 2004.
[8] B. Barak, M. Braverman, X. Chen, and A. Rao. How to compress interactive communication. In Proc. ACM Symposium on Theory of Computing, pages 67–76, 2010.
[9] J. Brody and A. Chakrabarti. A multi-round communication lower bound for gap hamming and some consequences. In IEEE Conference on Computational Complexity, pages 358–368, 2009.
[10] J. Brody, A. Chakrabarti, O. Regev, T. Vidick, and R. de Wolf. Better gap-hamming lower bounds via better round elimination. In APPROX-RANDOM, pages 476–489, 2010.
[11] A. Chakrabarti, G. Cormode, R. Kondapally, and A. McGregor. Information cost tradeoffs for augmented index and streaming language recognition. In Proc. IEEE Symposium on Foundations of Computer Science, pages 387–396, 2010.
[12] A. Chakrabarti, G. Cormode, and A. McGregor. Robust lower bounds for communication and stream computation. In Proc. ACM Symposium on Theory of Computing, pages 641–650, 2008.
[13] A. Chakrabarti, T. S. Jayram, and M. Patrascu. Tight lower bounds for selection in randomly ordered streams. In Proc. ACM-SIAM Symposium on Discrete Algorithms, pages 720–729, 2008.
[14] A. Chakrabarti, S. Khot, and X. Sun. Near-optimal lower bounds on the multi-party communication complexity of set disjointness. In Proc. IEEE Conference on Computational Complexity, pages 107–117, 2003.
[15] A. Chakrabarti, R. Kondapally, and Z. Wang. Information complexity versus corruption and applications to orthogonality and gap-hamming. In APPROX-RANDOM, pages 483–494, 2012.
[16] A. Chakrabarti and O. Regev. An optimal lower bound on the communication complexity of gap-hamming-distance. In Proc. ACM Symposium on Theory of Computing, 2011.
[17] A. Chakrabarti, Y. Shi, A. Wirth, and A. Yao. Informational complexity and the direct sum problem for simultaneous message complexity. In Proc. IEEE Symposium on Foundations of Computer Science, pages 270–278, 2001.
[18] G. Cormode and M. Garofalakis. Sketching streams through the net: Distributed approximate query tracking. In Proc. International Conference on Very Large Data Bases, 2005.
[19] G. Cormode, M. Garofalakis, S. Muthukrishnan, and R. Rastogi. Holistic aggregates in a networked world: Distributed tracking of approximate quantiles. In Proc. ACM SIGMOD International Conference on Management of Data, 2005.
[20] G. Cormode, S. Muthukrishnan, and K. Yi. Algorithms for distributed functional monitoring. ACM Transactions on Algorithms, 7(2):21, 2011.
[21] G. Cormode, S. Muthukrishnan, K. Yi, and Q. Zhang. Optimal sampling from distributed streams. In Proc. ACM Symposium on Principles of Database Systems, 2010. Invited to Journal of the ACM.
[22] T. Cover and J. Thomas. Elements of Information Theory. John Wiley and Sons, Inc., 1991.
[23] P. Duris and J. D. P. Rolim. Lower bounds on the multiparty communication complexity. J. Comput. Syst. Sci., 56(1):90–95, 1998.
[24] F. Ergün and H. Jowhari. On distance to monotonicity and longest increasing subsequence of a data stream. In Proc. ACM-SIAM Symposium on Discrete Algorithms, pages 730–736, 2008.
[25] D. Estrin, R. Govindan, J. S. Heidemann, and S. Kumar. Next century challenges: Scalable coordination in sensor networks. In MOBICOM, pages 263–270, 1999.
[26] W. Feller. Generalization of a probability limit theorem of cramer. Trans. Amer. Math. Soc, 54(3):361–372, 1943.
[27] A. Gál and P. Gopalan. Lower bounds on streaming algorithms for approximating the length of the longest increasing subsequence. In Proc. IEEE Symposium on Foundations of Computer Science, 2007.
[28] S. Ganguly. Polynomial estimators for high frequency moments. CoRR, abs/1104.4552, 2011.
[29] S. Ganguly. A lower bound for estimating high moments of a data stream. CoRR, abs/1201.0253, 2012.
[30] P. Gopalan, R. Meka, O. Reingold, and D. Zuckerman. Pseudorandom generators for combinatorial shapes. In Proceedings of the 43rd annual ACM symposium on Theory of computing, STOC ’11, pages 253–262, New York, NY, USA, 2011. ACM.
[31] A. Gronemeier. Asymptotically optimal lower bounds on the nih-multi-party information complexity of the and-function and disjointness. In Symposium on Theoretical Aspects of Computer Science, pages 505–516, 2009.
[32] S. Guha and Z. Huang. Revisiting the direct sum theorem and space lower bounds in random order streams. In Proc. International Colloquium on Automata, Languages, and Programming, 2009.
[33] N. J. A. Harvey, J. Nelson, and K. Onak. Sketching and streaming entropy via approximation theory. In Proc. IEEE Symposium on Foundations of Computer Science, pages 489–498, 2008.
[34] Z. Huang and K. Yi. Personal communication, 2012.
[35] Z. Huang, K. Yi, and Q. Zhang. Randomized algorithms for tracking distributed count, frequencies, and ranks. CoRR, abs/1108.3413, 2011.
[36] Z. Huang, K. Yi, and Q. Zhang. Randomized algorithms for tracking distributed count, frequencies, and ranks. In Proceedings of the 31st symposium on Principles of Database Systems, PODS ’12, pages 295–306, New York, NY, USA, 2012. ACM.
[37] P. Indyk and D. Woodruff. Optimal approximations of the frequency moments of data streams. In Proc. ACM Symposium on Theory of Computing, 2005.
[38] P. Indyk and D. P. Woodruff. Tight lower bounds for the distinct elements problem. In FOCS, pages 283–288, 2003.
[39] T. S. Jayram. Hellinger strikes back: A note on the multi-party information complexity of and. In APPROX-RANDOM, pages 562–573, 2009.
[40] D. M. Kane, J. Nelson, E. Porat, and D. P. Woodruff. Fast moment estimation in data streams in optimal space. In STOC, pages 745–754, 2011.
[41] D. M. Kane, J. Nelson, and D. P. Woodruff. An optimal algorithm for the distinct elements problem. In Proc. ACM Symposium on Principles of Database Systems, pages 41–52, 2010.
[42] R. Keralapura, G. Cormode, and J. Ramamirtham. Communication-efficient distributed monitoring of thresholded counts. In Proc. ACM SIGMOD International Conference on Management of Data, 2006.
[43] E. Kushilevitz and N. Nisan. Communication Complexity. Cambridge University Press, 1997.
[44] F. Magniez, C. Mathieu, and A. Nayak. Recognizing well-parenthesized expressions in the streaming model. In Proc. ACM Symposium on Theory of Computing, pages 261–270, 2010.
[45] A. Manjhi, V. Shkapenyuk, K. Dhamdhere, and C. Olston. Finding (recently) frequent items in distributed data streams. In Proc. IEEE International Conference on Data Engineering, 2005.
[46] J. Matousek and J. Vondrák. The probabilistic method. Lecture Notes, 2008.
[47] J. M. Phillips, E. Verbin, and Q. Zhang. Lower bounds for number-in-hand multiparty communication complexity, made easy. In Proc. ACM-SIAM Symposium on Discrete Algorithms, 2012.
[48] A. A. Razborov. On the distributional complexity of disjointness. In Proc. International Colloquium on Automata, Languages, and Programming, 1990.
[49] A. A. Sherstov. The communication complexity of gap hamming distance. Electronic Colloquium on Computational Complexity (ECCC), 18:63, 2011.
[50] S. Tirthapura and D. P. Woodruff. Optimal random sampling from distributed streams revisited. In The International Symposium on Distributed Computing, pages 283–297, 2011.
[51] T. Vidick. A concentration inequality for the overlap of a vector on a large set, with application to the communication complexity of the gap-hamming-distance problem. Electronic Colloquium on Computational Complexity (ECCC), 18:51, 2011.
[52] D. Woodruff. Optimal space lower bounds for all frequency moments. In Proc. ACM-SIAM Symposium on Discrete Algorithms, 2004.
[53] A. C. Yao. Probabilistic computations: Towards a unified measure of complexity. In Proc. IEEE Symposium on Foundations of Computer Science, 1977.
[54] K. Yi and Q. Zhang. Optimal tracking of distributed heavy hitters and quantiles. In Proc. ACM Symposium on Principles of Database Systems, 2009.

Appendix A Proof for Observation 1

We first show the rectangle property of private randomness protocols in the message passing model. The proof is just a syntactic change to that in [6], Section 6.4.1, which was designed for the blackboard model. The only difference between the blackboard model and the message-passing model is that in the blackboard model, if one speaks, everyone else can hear.

Property 1

Given a $k$ -party private randomness protocol $\Pi$ on inputs in $\mathcal{Z}_{1}\times\cdots\times\mathcal{Z}_{k}$ in the message passing model, for all $\{z_{1},\ldots,z_{k}\}\in\mathcal{Z}_{1}\times\cdots\times\mathcal{Z}_{k}$ , and for all possible transcripts $\pi\in\{0,1\}^{*}$ , we have

\mathsf{Pr}_{R}[\Pi=\pi\ |\ Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]=\prod_{i=1}^{k}\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ |\ Z_{i}=z_{i}],

(68)

where $\pi_{i}$ is the part of transcript $\pi$ that player $i$ sees (that is, the concatenation of all messages sent from or received by $S_{i}$ ), and $R=\{R_{1},\ldots,R_{k}\}\in\{\mathcal{R}_{1}\times\ldots\times\mathcal{R}_{k}\}$ is the players’ private randomness. Furthermore, we have

\mathsf{Pr}_{\mu,R}[\Pi=\pi]=\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}].

(69)

Proof:

We can view the input of $P_{i}\ (i\in[k])$ as a pair $(z_{i},r_{i})$ , where $z_{i}\in\mathcal{Z}_{i}$ and $r_{i}\in\mathcal{R}_{i}$ . Let $\mathcal{C}_{1}(\pi_{1})\times\cdots\times\mathcal{C}_{k}(\pi_{k})$ be the combinatorial rectangle containing all tuples $\{(z_{1},r_{1}),\ldots,(z_{k},r_{k})\}$ such that $\Pi(z_{1},\ldots,z_{k},r_{1},\ldots,r_{k})=\pi$ .

For each $i\in[k]$ , for each $z_{i}\in\mathcal{Z}_{i}$ , let $\mathcal{C}_{i}(z_{i},\pi_{i})$ be the projection of $\mathcal{C}_{i}(\pi_{i})$ on pairs of the form $(z_{i},*)$ , and $\mathcal{C}_{i}(z_{i})$ be the collection of all pairs of the form $(z_{i},*)$ . Note that $\left|\mathcal{C}_{i}(z_{i},\pi_{i})\right|/\left|\mathcal{C}_{i}(z_{i})\right|=\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ |\ Z_{i}=z_{i}]$ . If each player $P_{i}$ chooses $r_{i}$ uniformly at random from $\mathcal{R}_{i}$ , then the transcript will be $\pi$ if and only if $(z_{i},r_{i})\in\mathcal{C}_{i}(z_{i},\pi_{i})$ . Since the choices of $R_{1},\ldots,R_{k}$ are all independent, it follows that $\mathsf{Pr}_{R}[\Pi=\pi\ |\ Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]=\prod_{i\in[k]}\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ |\ Z_{i}=z_{i}]$ .

To show (69), we sum over all possible values of $Z=\{Z_{1},\ldots,Z_{k}\}$ . Let $Z_{-i}=\{Z_{1},\ldots,Z_{i-1},Z_{i+1},Z_{k}\}$ .

			$\displaystyle\mathsf{Pr}_{\mu,R}[\Pi=\pi]$
		$\displaystyle=$	$\displaystyle\sum_{z}(\mathsf{Pr}_{R}[\Pi=\pi\ \|\ Z=z]\mathsf{Pr}_{\mu}[Z=z])$
		$\displaystyle=$	$\displaystyle\sum_{z}\left(\prod_{i\in[k]}\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ \|\ Z_{i}=z_{i}]\prod_{i\in[k]}\mathsf{Pr}_{\mu}[Z_{i}=z_{i}]\right)\quad(\text{by (\ref{eq:x-1}) and $Z_{i}$ are independent})$
		$\displaystyle=$	$\displaystyle\sum_{z}\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}\wedge Z_{i}=z_{i}]$
		$\displaystyle=$	$\displaystyle\sum_{z_{-i}}\left(\left(\sum_{z_{i}}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}\wedge Z_{i}=z_{i}]\right)\prod_{j\neq i}\mathsf{Pr}_{\mu,R_{j}}[\Pi_{j}=\pi_{j}\wedge Z_{j}=z_{j}]\right)$
		$\displaystyle=$	$\displaystyle\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}]\cdot\sum_{z_{-i}}\prod_{j\neq i}\mathsf{Pr}_{\mu,R_{j}}[\Pi_{j}=\pi_{j}\wedge Z_{j}=z_{j}]$
		$\displaystyle=$	$\displaystyle\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}].$

Now we prove Observation 1. That is, to show

\mathsf{Pr}_{\mu,R}[Z_{1}=z_{1},\ldots,Z_{k}=z_{k}\ |\ \Pi=\pi]=\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[Z_{i}=z_{i}\ |\ \Pi_{i}=\pi_{i}],

We show this using the rectangle property.

			$\displaystyle\mathsf{Pr}_{\mu,R}[Z_{1}=z_{1},\ldots,Z_{k}=z_{k}\ \|\ \Pi=\pi]$
		$\displaystyle=$	$\displaystyle\frac{\mathsf{Pr}_{R}[\Pi=\pi\ \|\ Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]\cdot\mathsf{Pr}_{\mu}[Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]}{\mathsf{Pr}_{\mu,R}[\Pi=\pi]}\quad(\text{Bayes' theorem})$
		$\displaystyle=$	$\displaystyle\frac{\prod_{i\in[k]}\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ \|\ Z_{i}=z_{i}]\cdot\prod_{i\in[k]}\mathsf{Pr}_{\mu}[Z_{i}=z_{i}]}{\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}]}\quad(\text{by }(\ref{eq:x-1}),(\ref{eq:x-2})\text{ and }Z_{i}\text{ are independent})$
		$\displaystyle=$	$\displaystyle\prod_{i\in[k]}\frac{\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ \|\ Z_{i}=z_{i}]\cdot\mathsf{Pr}_{\mu}[Z_{i}=z_{i}]}{\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}]}$
		$\displaystyle=$	$\displaystyle\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[Z_{i}=z_{i}\ \|\ \Pi_{i}=\pi_{i}]\quad(\text{Bayes' theorem})$

	$\displaystyle\mathsf{Pr}[\Pi\text{ is bad${}_{1}$}\ \|\ \Pi\text{ is normal}]$	(4)
$\displaystyle=$	$\displaystyle\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]\sum_{z}\mathsf{Pr}[Z=z\ \|\ \Pi=\pi,\Pi\text{ is normal}]\ \chi(z\perp_{1}\pi)$
$\displaystyle\leq$	$\displaystyle\mathsf{Pr}[Z\text{ is a joker}]+\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]$
	$\displaystyle\quad\quad\sum_{\ell\in[{\beta k}/{2},2\beta k]}\sum_{T\subseteq[k]:\left\|T\right\|=\ell}\sum_{z}\mathsf{Pr}[Z=z\ \|\ \Pi=\pi,\Pi\text{ is normal}]\ \chi(H_{1}(z)=T)\ \chi(T\perp_{1}\pi)$
$\displaystyle\leq$	$\displaystyle\mathsf{Pr}[Z\text{ is a joker}]+\sum_{\pi}\mathsf{Pr}[\Pi=\pi\ \|\ \Pi\text{ is normal}]$
	$\displaystyle\quad\quad\sum_{\ell\in[\beta k/2,2\beta k]}\left(\left.\sum_{\begin{subarray}{c}T\subseteq[k]:\left\|T\right\|=\ell\\ T\perp_{1}\pi\end{subarray}}\prod_{i\in T}q_{i}^{\pi}\ \right\|\ \Pi=\pi,\Pi\text{ is normal}\right)$

			$\displaystyle\mathsf{Pr}_{\mu,R}[Z_{1}=z_{1},\ldots,Z_{k}=z_{k}\ \|\ \Pi=\pi]$
		$\displaystyle=$	$\displaystyle\frac{\mathsf{Pr}_{R}[\Pi=\pi\ \|\ Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]\cdot\mathsf{Pr}_{\mu}[Z_{1}=z_{1},\ldots,Z_{k}=z_{k}]}{\mathsf{Pr}_{\mu,R}[\Pi=\pi]}\quad(\text{Bayes' theorem})$
		$\displaystyle=$	$\displaystyle\frac{\prod_{i\in[k]}\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ \|\ Z_{i}=z_{i}]\cdot\prod_{i\in[k]}\mathsf{Pr}_{\mu}[Z_{i}=z_{i}]}{\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}]}\quad(\text{by }(\ref{eq:x-1}),(\ref{eq:x-2})\text{ and }Z_{i}\text{ are independent})$
		$\displaystyle=$	$\displaystyle\prod_{i\in[k]}\frac{\mathsf{Pr}_{R_{i}}[\Pi_{i}=\pi_{i}\ \|\ Z_{i}=z_{i}]\cdot\mathsf{Pr}_{\mu}[Z_{i}=z_{i}]}{\mathsf{Pr}_{\mu,R_{i}}[\Pi_{i}=\pi_{i}]}$
		$\displaystyle=$	$\displaystyle\prod_{i\in[k]}\mathsf{Pr}_{\mu,R_{i}}[Z_{i}=z_{i}\ \|\ \Pi_{i}=\pi_{i}]\quad(\text{Bayes' theorem})$

Tight Bounds for Distributed Functional Monitoring

Abstract

1 Introduction

2 Preliminaries

Information Theory

Proposition 1

Communication complexity

Lemma 1 (Yao’s Lemma)

Information complexity

Proposition 2

Statistical distance measures

Proposition 3

Proposition 4

Conventions.

3 A Lower Bound for F0F_{0}

3.1 The kk-APPROX-SUM Problem

Definition 1

Theorem 1

Observation 1

Definition 2

Definition 3

Lemma 2

Definition 4

Lemma 3

Fact 1

Lemma 4

Lemma 5

3.2 The 22-DISJ Problem

Theorem 2 ([47], Lemma 2.2)

3.3 The Complexity of F0F_{0}

3.3.1 Connecting F0F_{0} and kk-APPROX-SUM2-DISJ,τ{}_{\text{$2$-DISJ},\tau}

Lemma 6

Lemma 7

Correctness.

3.3.2 An Embedding Argument

Lemma 8

Theorem 3

4 A Lower Bound for Fp​(p>1)F_{p}\ (p>1)

4.1 The 22-GAP-ORT Problem

Theorem 4 ([15])

4.2 The kk-XOR Problem

4.3 The kk-GUESS Problem

Theorem 5

4.4 The kk-BTX Problem

Linking kk-BTX to 22-GAP-ORT.

Theorem 6

Theorem 7

Corollary 1

4.5 The Complexity of Fp​(p>1)F_{p}\ (p>1)

Lemma 9

Protocol 𝒫\cal P.

Correctness.

Theorem 8

5 An Upper Bound for Fp​(p>1)F_{p}\ (p>1)

5.1 Our Protocol

5.2 Communication Cost

Lemma 10

5.3 Correctness

5.3.1 Concentration of Individual Frequencies

Fact 2

Lemma 11

5.3.2 Estimating Class Sizes

Lemma 12

Lemma 13

5.3.3 Combining Individual Frequency Estimation and Class Size Estimation

Lemma 14

5.3.4 Putting It All Together

Lemma 15

Theorem 9

6 Related Problems

6.1 The All-Quantile and Heavy Hitters

Definition 5

Definition 6

Definition 7

Theorem 10

6.1.1 The kk-GAP-MAJ Problem

Theorem 11

Remark 1

Definition 8

Lemma 16

3 A Lower Bound for $F_{0}$

3.1 The $k$ -APPROX-SUM Problem

3.2 The $2$ -DISJ Problem

3.3 The Complexity of $F_{0}$

3.3.1 Connecting $F_{0}$ and $k$ -APPROX-SUM ${}_{\text{$2$-DISJ},\tau}$

4 A Lower Bound for $F_{p}\ (p>1)$

4.1 The $2$ -GAP-ORT Problem

4.2 The $k$ -XOR Problem

4.3 The $k$ -GUESS Problem

4.4 The $k$ -BTX Problem

Linking $k$ -BTX to $2$ -GAP-ORT.

4.5 The Complexity of $F_{p}\ (p>1)$

Protocol $\cal P$ .

5 An Upper Bound for $F_{p}\ (p>1)$

6.1.1 The $k$ -GAP-MAJ Problem

6.3 $\ell_{p}$ for any constant $p\geq 1$