Joint Source-Channel Coding on a Multiple Access Channel with Side Information

In this paper we consider the transmission of information from several correlated sources over a multiple access channel with side information. This system does not satisfy source-channel separation ([12]). Thus for optimal transmission one needs to consider joint source-channel coding. We will provide several good joint source-channel coding schemes.

Although this topic has been studied for last several decades, one recent motivation is the problem of estimating a random field via sensor networks. Sensor nodes have limited computational and storage capabilities and very limited energy [3]. These sensor nodes need to transmit their observations to a fusion center which uses this data to estimate the sensed random field. Since transmission is very energy intensive, it is important to minimize it.

The proximity of the sensing nodes to each other induces high correlations between the observations of adjacent sensors. One can exploit these correlations to compress the transmitted data significantly ([3], [4]). Furthermore, some of the nodes can be more powerful and act as cluster heads ([4]). Nodes transmit their data to a nearby cluster head which can further compress information before transmission to the fusion center. Transmission of data from sensor nodes to their cluster-head requires sharing the wireless multiple access channel (MAC). At the fusion center the underlying physical process is estimated. The main trade-off possible is between the rates at which the sensors send their observations and the distortion incurred in the estimation at the fusion center. The availability of side information at the encoders and/or the decoder can reduce the rate of transmission ([19], [42]).

The above considerations open up new interesting problems in multi-user information theory and the quest for finding the optimal performance for various models of sources, channels and side information have made this an active area of research. The optimal solution is not known except in a few simple cases. In this paper a joint source channel coding approach is discussed under various assumptions on side information and distortion criteria. Sufficient conditions for transmission of discrete/continuous alphabet sources with a given distortion over a discrete/continuous alphabet MAC are provided. These results generalize the previous results available on this problem.

In the following we survey the related literature. Ahlswede [1] and Liao [28] obtained the capacity region of a discrete memoryless MAC with independent inputs. Cover, El Gamal and Salehi [12] made further significant progress by providing sufficient conditions for transmitting losslessly correlated observations over a MAC. They proposed a ‘correlation preserving’ scheme for transmitting the sources. This mapping is extended to a more general system with several principle sources and several side information sources subject to cross observations at the encoders in [2]. However single letter characterization of the capacity region is still unknown. Indeed Duek [15] proved that the conditions given in [12] are only sufficient and may not be necessary. In [26] a finite letter upper bound for the problem is obtained. It is also shown in [12] that the source-channel separation does not hold in this case. The authors of [35] obtain a condition for separation to hold in a multiple access channel.

The capacity region for the distributed lossless source coding problem for correlated sources is given in the classic paper by Slepian and Wolf ([38]). Cover ([11]) extended Slepian-Wolf results to an arbitrary number of discrete, ergodic sources using a technique called ‘random binning’. Other related papers on this problem are [2], [6].

Inspired by Slepian-Wolf results, Wyner and Ziv [42] obtained the rate distortion function for source coding with side information at the decoder. It is shown that the knowledge of side information at the encoders in addition to the decoder, permits the transmission at a lower rate. This is in contrast to the lossless case considered by Slepian and Wolf. The rate distortion function when encoder and decoder both have side information was first obtained by Gray (See [8]). Related work on side information coding is [5], [14], [33]. The lossy version of Slepian-Wolf problem is called multi-terminal source coding problem and despite numerous attempts (e.g.,  [9], [30]) the exact rate region is not known except for a few special cases. First major advancement was in Berger and Tung ([8]) where an inner and an outer bound on the rate distortion region was obtained. Lossy coding of continuous sources at the high resolution limit is studied in [43] where an explicit single-letter bound is obtained. Gastpar ([19]) derived an inner and an outer bound with decoder side information and proved the tightness of his bounds when the sources are conditionally independent given the side information. The authors in [39] obtain inner and outer bounds on the rate region with side information at the encoders and the decoder. In [29] an achievable rate region for a MAC with correlated sources and feedback is given.

The distributed Gaussian source coding problem is discussed in [30], [41]. For two users exact rate region is provided in [41]. The capacity of a Gaussian MAC (GMAC) for independent sources with feedback is given in [32]. In [27] one necessary and two sufficient conditions for transmitting a bivariate jointly Gaussian source over a GMAC are provided. It is shown that the amplify and forward scheme is optimal below a certain SNR. The performance comparison of the schemes given in [27] with a separation-based scheme is given in [34]. GMAC under received power constraints is studied in [18] and it is shown that the source-channel separation holds in this case.

In [20] the authors discuss a joint source channel coding scheme over a MAC and show the scaling behavior for the Gaussian channel. A Gaussian sensor network in distributed and collaborative setting is studied in [24]. The authors show that it is better to compress the local estimates than to compress the raw data. The scaling laws for a many-to-one data-gathering channel are discussed in [17]. It is shown that the transport capacity of the network scales as $\mathcal{O}(logN)$ when the number of sensors $N$ grows to infinity and the total average power remains fixed. The scaling laws for the problem without side information are also discussed in [21] and it is shown that separating source coding from channel coding may require exponential growth, as a function of number of sensors, in communication bandwidth. A lower bound on best achievable distortion as a function of the number of sensors, total transmit power, the degrees of freedom of the underlying process and the spatio-temporal communication bandwidth is given.

The joint source-channel coding problem also bears relationship to the CEO problem [10]. In this problem, multiple encoders observe different, noisy versions of a single information source and communicate it to a single decoder called the CEO which is required to reconstruct the source within a certain distortion. The Gaussian version of the CEO problem is studied in [31].

This paper makes the following contributions. It obtains sufficient conditions for transmission of correlated sources with given distortions over a MAC with side information. The source/channel alphabets can be discrete or continuous. The sufficient conditions are strong enough that previous known results are special cases. Next we obtain a bit to Gaussian mapping which provides correlated Gaussian channel codewords for discrete distributed sources.

The paper is organized as follows. Sufficient conditions for transmission of distributed sources over a MAC with side information and given distortion are obtained in Section II. The sources and the channel alphabets can be continuous or discrete. Several previous results are recovered as special cases in Section III. Section IV considers the important case of transmission of discrete correlated sources over a GMAC and presents a new joint source-channel coding scheme. Section V briefly considers Gaussian sources over a GMAC. Section VI concludes the paper. The proof of the main theorem is given in Appendix A. The proofs of several other results are provided in later appendices.

We consider the transmission of memoryless dependent sources, through a memoryless multiple access channel (Fig. 1). The sources and/or the channel input/output alphabets can be discrete or continuous. Furthermore, side information about the transmitted information may be available at the encoders and the decoder. Thus our system is very general and covers many systems studied earlier.

Initially we consider two sources $(U_{1},U_{2})$ and side information random variables $Z_{1},Z_{2},Z$ with a known joint distribution $F(u_{1},u_{2},z_{1},z_{2},z)$. Side information $Z_{i}$ is available to encoder $i,~i={1,2}$ and the decoder has side information $Z$. The random vector sequence $\{(U_{1n},U_{2n},Z_{1n},Z_{2n},Z_{n}),n\geq 1\}$ formed from the source outputs and the side information with distribution $F$ is independent identically distributed (iid) in time. We will denote $\{U_{1k},~k=1,...,n\}$ by $U_{1}^{n}$. Similarly for other sequences. The sources transmit their codewords $X_{in}$’s to a single decoder through a memoryless multiple access channel. The channel output $Y$ has distribution $p(y|x_{1},x_{2})$ if $x_{1}$ and $x_{2}$ are transmitted at that time. Thus, $\{Y_{n}\}$ and $\{X_{1n},X_{2n}\}$ satisfy $p(y_{k}|y^{k-1},x_{1}^{k},x_{2}^{k})=p(y_{k}|x_{1k},x_{2k})$. The decoder receives $Y_{n}$ and also has access to the side information $Z_{n}$. The encoders at the two users do not communicate with each other except via the side information. The decoder uses the channel outputs and its side information to estimate the sensor observations $U_{in}$ as $\hat{U}_{in},~i={1,2}$. It is of interest to find encoders and a decoder such that $\{U_{1n},U_{2n},n\geq 1\}$ can be transmitted over the given MAC with $E[d_{1}(U_{1},\hat{U_{1}})]\leq D_{1}$ and $E[d_{2}(U_{2},\hat{U_{2}})]\leq D_{2}$ where $d_{i}$ are non-negative distortion measures and $D_{i}$ are the given distortion constraints. If the distortion measures are unbounded we assume that there exist $u_{i}^{*}$ such that $E[d_{i}(U_{i},u_{i}^{*})]<\infty,~i={1,2}$. This covers the important special case of mean square error (MSE) if $E[U_{i}^{2}]<\infty,~i=1,2$.

Source channel separation does not hold in this case.

For discrete sources a common distortion measure is Hamming distance,

For continuous alphabet sources the most common distortion measure is $d(x,x^{\prime})=(x-x^{\prime})^{2}$. To obtain the results for lossless case from our Theorem 1 below, we assume that $d_{i}(x,x^{\prime})=0\Leftrightarrow x=x^{\prime}$, e.g., Hamming distance.

$Definition$: The source $(U_{1}^{n},U_{2}^{n})$ can be transmitted over the multiple access channel with distortions ${\bf{D}}{\buildrel\Delta\over{=}}(D_{1},D_{2})$ if for any $\epsilon>0$ there is an $n_{0}$ such that for all $n>n_{0}$ there exist encoders $f_{E,i}^{n}:\mathcal{U}_{i}^{n}\times\mathcal{Z}_{i}^{n}\rightarrow\mathcal{X}_{i}^{n},i={1,2}$ and a decoder ${{f}_{D}^{n}}:\mathcal{Y}^{n}\times\mathcal{Z}^{n}\rightarrow(\mathcal{\hat{U}}_{1}^{n},\mathcal{\hat{U}}_{2}^{n})$ such that $\frac{1}{n}E\left[\sum_{j=1}^{n}d(U_{ij},\hat{U}_{ij})\right]\leq D_{i}+\epsilon,~i={1,2}$ where $(\hat{U}_{1}^{n},\hat{U}_{2}^{n})=f_{D}(Y^{n},Z^{n})$ and $\mathcal{U}_{i},~\mathcal{Z}_{i},~\mathcal{Z},~\mathcal{X}_{i},~\mathcal{Y},~\hat{\mathcal{U}_{i}}$ are the sets in which $U_{i},~Z_{i},~Z,~X_{i},~Y,~\hat{U}_{i}$ take values.

We denote the joint distribution of $(U_{1},U_{2})$ by $p(u_{1},u_{2})$. Also, $X\leftrightarrow Y\leftrightarrow Z$ will denote that $\{X,Y,Z\}$ form a Markov chain.

Now we state the main Theorem.

$Proof$: See Appendix A.                                                                                                $\blacksquare$

In the proof of Theorem 1 the encoding scheme involves distributed vector quantization $(W_{1}^{n},W_{2}^{n})$ of the sources $(U_{1}^{n},U_{2}^{n})$ and the side information $Z_{1}^{n},Z_{2}^{n}$ followed by a correlation preserving mapping to the channel codewords $(X_{1}^{n},X_{2}^{n})$. The decoding approach involves first decoding $(W_{1}^{n},W_{2}^{n})$ and then obtaining the estimates $(\hat{U}_{1}^{n},\hat{U}_{2}^{n})$ as a function of $(W_{1}^{n},W_{2}^{n})$ and the decoder side information $Z^{n}$.

If the channel alphabets are continuous (e.g., GMAC) then in addition to the conditions in Theorem 1 certain power constraints $E[X_{i}^{2}]\leq P_{i},~i=1,2$ are also needed. In general, we could impose a more general constraint $E[g_{i}(X_{i})]\leq\alpha_{i}$ where $g_{i}$ is some non-negative cost function. Furthermore, for continuous alphabet r.v.s (sources/channel input/output) we will assume that probability density exists so that one can use differential entropy (more general cases can be handled but for simplicity we will ignore them).

The dependence in $(U_{1},U_{2})$ is used in two ways in (1): to $reduce$ the quantities on the left and to increase the quantities on the right. The side information $Z_{1}$ and $Z_{2}$ effectively $increases$ the dependence in the inputs.

If the source-channel separation holds then one can consider the capacity region of the channel. For example, when there is no side information $Z_{1},Z_{2},Z$ and the sources are independent then we obtain the rate region

This is the well known rate region of a MAC ([13]). To obtain (2) from (1), take $(Z_{1},Z_{2},Z)$ independent of $(U_{1},U_{2})$. Also, take $U_{1},U_{2}$ discrete, $W_{i}=U_{i}$ and $X_{i}$ independent of $U_{i},~i=1,~2$.

In Theorem 1 it is possible to include other distortion constraints. For example, in addition to the bounds on $E[d(U_{i},\hat{U}_{i})]$ one may want a bound on the joint distortion $E[d((U_{1},U_{2}),(\hat{U}_{1},\hat{U}_{2}))]$. Then the only modification needed in the statement of the above theorem is to include this also as a condition in defining $f_{D}$.

If we only want to estimate a function $g(U_{1},U_{2})$ at the decoder and not $(U_{1},U_{2})$ themselves, then again one can use the techniques in proof of Theorem 1 to obtain sufficient conditions. Depending upon $g$, the conditions needed may be weaker than those needed in (1). We will explore this in more detail in a later work.

In our problem setup the side information $Z_{i}$ can be included with source $U_{i}$ and then we can consider this problem as one with no side information at the encoders. However, the above formulation has the advantage that our conditions (1) are explicit in $Z_{i}$.

The main problem in using Theorem 1 is in obtaining good source-channel coding schemes providing $(W_{1},W_{2},X_{1},X_{2})$ which satisfy the conditions in the theorem for a given source $(U_{1},U_{2})$ and a channel. A substantial part of this paper will be devoted to this problem.

In the following we show that our result contains several previous studies as special cases. The practically important special case of GMAC will be studied in detail in later sections. There we will discuss several specific joint source-channel coding schemes for GMAC and compare their performance.

This system is practically very useful. For example, in a sensor network, the observations sensed by the sensor nodes are discretized and then transmitted over a GMAC. The physical proximity of the sensor nodes makes their observations correlated. This correlation can be exploited to compress the transmitted data and increase the channel capacity. We present a novel distributed ‘correlation preserving’ joint source-channel coding scheme yielding jointly Gaussian channel codewords which transmit the data efficiently over a GMAC.

Sufficient conditions for lossless transmission of two discrete correlated sources $(U_{1},U_{2})$ (generating $iid$ sequences in time) over a general MAC with no side information are obtained in (4).

In this section, we further specialize these results to a GMAC: $Y=X_{1}+X_{2}+N$ where $N$ is a Gaussian random variable independent of $X_{1}$ and $X_{2}$. The noise $N$ satisfies $E[N]=0$ and $Var(N)=\sigma_{N}^{2}$ . We will also have the transmit power constraints: $E[X_{i}^{2}]\leq P_{i},i=1,2$. Since source-channel separation does not hold for this system, a joint source-channel coding scheme is needed for optimal performance.

The dependence of right hand side (RHS) in (4) on input alphabets prevents us from getting a closed form expression for the admissibility criterion. Therefore we relax the conditions by taking away the dependence on the input alphabets to obtain good joint source-channel codes.

$Proof$: See Appendix C.                                                                                                                $\blacksquare$

Thus from (4),

The relaxation of the upper bounds is only in (12) and (13) and not in (14).

We show that the relaxed upper bounds are maximized if $(X_{1},X_{2})$ is jointly Gaussian and the correlation $\rho$ between $X_{1}$ and $X_{2}$ is high (the highest possible $\rho$ may not give the largest upper bound in (12)-(14)).

$Proof$: See Appendix C.                                                                                                                $\blacksquare$

The difference between the bounds in (12) is

This difference is small if correlation between $(U_{1},U_{2})$ is small. In that case $H(U_{1}|U_{2})$ and $H(U_{2}|U_{1})$ will be large and (12) and (13) can be active constraints. If correlation between $(U_{1},U_{2})$ is large, $H(U_{1}|U_{2})$ and $H(U_{2}|U_{1})$ will be small and (14) will be the only active constraint. In this case the difference between the two bounds in (12) and (13) is large but not important. Thus, the outer bounds in (12) and (13) are close to the inner bounds whenever the constraints (12) and (13) are active. Often (14) will be the only active constraint.

Based on Lemma 2, we use jointly Gaussian channel inputs $(X_{1},X_{2})$ with the transmit power constraints. Thus we take $(X_{1},X_{2})$ with mean vector $[0~~0]$ and covariance matrix $K_{X_{1},X_{2}}=\\ \begin{pmatrix}P_{1}&\rho\sqrt{P_{1}P_{2}}\\ \rho\sqrt{P_{1}P_{2}}&P_{2}\end{pmatrix}$. The outer bounds in (12)-(14) become $0.5\log\left[1+\frac{P_{1}(1-{{\rho}}^{2})}{{\sigma_{N}}^{2}}\right]$, $0.5\log\left[1+\frac{P_{2}(1-{{\rho}}^{2})}{{\sigma_{N}}^{2}}\right]$ and $0.5\log$$\left[1+\frac{P_{1}+P_{2}+{2}{{\rho}}{\sqrt{P_{1}P_{2}}}}{{\sigma_{N}}^{2}}\right]$ respectively. The first two upper bounds decrease as $\rho$ increases. But the third upper bound increases with $\rho$ and often the third constraint is the limiting constraint. Thus, once $(X_{1},X_{2})$ are obtained we can check for sufficient conditions (4). If these are not satisfied for the $(X_{1},X_{2})$ obtained, we will increase the correlation $\rho$ between $(X_{1},X_{2})$ if possible (see details below). Increasing the correlation in $(X_{1},X_{2})$ will decrease the difference in (15) and increase the possibility of satisfying (4) when the outer bounds in (12) and (13) are satisfied. If not, we can increase ${\rho}$ further till we satisfy (4).

The next lemma provides an upper bound on the correlation $\rho$ between $(X_{1},X_{2})$ possible in terms of the distribution of $(U_{1},U_{2})$.

$Proof$: See Appendix C.                                                                                                                $\blacksquare$

It is stated in [35], without proof, that the correlation between $(X_{1},X_{2})$ cannot be greater than the correlation of the source $(U_{1},U_{2})$. Lemma 3 gives a tighter bound in many cases. Consider $(U_{1},U_{2})$ with the joint distribution: $P(U_{1}=0;U_{2}=0)=P(U_{1}=1;U_{2}=1)=0.4444;P(U_{1}=1;U_{2}=0)=P(U_{1}=0;U_{2}=1)=0.0556$. The correlation between the sources is 0.7778 but from Lemma 3, the correlation between $(X_{1},X_{2})$ cannot exceed 0.7055.