Theoretical Analysis of Deep Neural Networks in Physical Layer Communication

As shown in the upper part of Fig. 1, consider the process of message transmission from the perspective of a typical communication system. We assume that an information source generates a sequence of ${\log_{2}}M$-bit message symbols ${s}\in\left\{{1,2,\cdots,M}\right\}$ to be communicated to the destination. Then the modulation modules inside the transmitter map each symbol $s$ to a signal ${\bf{x}}\in{\mathbb{R}^{d}}$, where $d$ denoted the dimension of the signal space. The signal alphabet is denoted by ${{\bf{x}}_{1}},{{\bf{x}}_{2}},\cdots,{{\bf{x}}_{M}}$. During channel transmission, $d$-dimensional signal $\bf{x}$ is corrupted to ${\bf{y}}\in{\mathbb{R}^{d}}$ with conditional probability density function (PDF) $p\left({{\bf{y}}|{\bf{x}}}\right)=\prod_{i=1}^{d}p\left({{y_{i}}|{x_{i}}}\right)$. In this paper, we use $d/2$ bandpass channels, each with separately modulated inphase and quadrature components to transmit the $d$-dimensional signal [33]. Finally, the received signal is mapped by the demodulation module inside the receiver to ${\hat{s}}$ which is an estimate of the transmitted symbol $s$. The procedures mentioned above have been exhaustively presented by Shannon.

From the point of view of filtering or signal inference, the idea of AE-based communication system matches Norbert Wiener’s perspective [34]. As shown in the lower part of the Fig. 1, the AE consists of an encoder and a decoder and each of them is a feedforward neural network with parameters (weights and biases) ${{{\bf{\Theta}}_{f}}}$ and ${{{\bf{\Theta}}_{g}}}$, respectively. Note that each symbol $s$ from information source usually needs to be encoded to a one-hot vector ${\bf{s}}\in{\mathbb{R}^{M}}$ and then is fed into the encoder. Under a given constraint (e.g., average signal power constraint), the PDF of a wireless channel and a loss function to minimize symbol error probability, the encoder and decoder are respectively able to learn to appropriately represent $\bf{s}$ as ${\bf{z}}=f\left({\bf{s}},{{{\bf{\Theta}}_{f}}}\right)$ and to map the corrupted signal $\bf{v}$ to an estimate of transmitted symbol ${\bf{\hat{s}}}=g\left({\bf{v}},{{{\bf{\Theta}}_{g}}}\right)$ where ${\bf{z}},{\bf{v}}\in{\mathbb{R}^{d}}$. Here, we use ${{\bf{z}}_{1}},{{\bf{z}}_{2}},\cdots,{{\bf{z}}_{M}}$ denoted the transmitted signals from the encoder in order to distinguish it from the transmitted signals from the transmitter.

From the perspective of the whole AE (communication system), it aims to transmit information to a destination with low error probability. The symbol error probability, i.e., the probability that the wireless channel has shifted a signal point into another signal’s decision region, is

The AE can use the cross-entropy loss function

to represent the cost brought by inaccuracy of prediction where ${{{\bf{s}}_{i}}\left[j\right]}$ denotes the $j$-th element of the $i$-th symbol in a training set with $n$ symbols. In order to train the AE to minimize the symbol error probability, the optimal parameters could be found by optimizing the loss function

where $P_{{\rm{av}}}$ denotes the average power. In this paper, we set $P_{{\rm{av}}}={1/M}$. Now, it is important to explain how does the mapping ${\bf{z}}=f\left({\bf{s}},{{{\bf{\Theta}}_{f}}}\right)$ vary after the training was done.

The eminent work of [37] proposed a gradient-search algorithm to obtain the optimum constellations. Consider a zero-mean stationary additive white Gaussian noise (AWGN) channel with one-sided spectral density $2N_{0}$. For large signal-to-noise ratio (SNR), the asymptotic approximation of (1) can be written as

To minimize $P_{e}$, the problem can be formulated as

where $\left\{{{\bf{z}}_{m}^{*}}\right\}_{m=1}^{M}$ denotes the optimal signal set. This optimization problem can be solved by using a constrained gradient-search algorithm. We denote $\left\{{{\bf{z}}_{m}}\right\}_{m=1}^{M}$ as an $M\times d$ matrix

Then, the $k$-th step of the constrained gradient-search algorithm can be described by

where $\eta_{k}$ denotes step size and $\nabla{P_{e}}\left({{{\bf{Z}}_{k}}}\right)\in{\mathbb{R}^{M\times d}}$ denotes the gradient of $P_{e}$ respect to the current constellation points. It can be written as

where

The vector ${{{\bf{1}}_{{{\bf{z}}_{m}}-{{\bf{z}}_{i}}}}}$ denotes $d$-dimensional unit vector in the direction of ${{{\bf{z}}_{m}}-{{\bf{z}}_{i}}}$.

Comparing (3) to (5), we can understand the mechanism of the encoder in an AE-based communication system. Its optimized variables of AE method are the parameters ${\Theta_{f}}$ and ${\Theta_{g}}$. In other words, AE learns the constellation design through simultaneously optimizing the parameters of the encoder and decoder. It does not directly optimize the constellation points ${\bf{s}}$. Differently, the gradient-search algorithm directly optimizes the ${\bf{s}}$. Although the optimized variables of these two methods are different, their optimization goals of minimizing the SER are essentially identical. Therefore, the mapping function of encoder can be represented as

when the PDF used for generating training samples is multivariate zero-mean normal distribution ${\bf{v}}-{\bf{z}}\sim{{\cal N}_{d}}({\bf{\vec{0}}},{\mkern 1.0mu}{\bm{\Sigma}})$ where ${{\bf{\vec{0}}}}$ denotes $d$-dimensional zero vector and ${\bm{\Sigma}}=\left({2{N_{0}}/d}\right){\bf{I}}$ is an $d\times d$ diagonal matrix. In the next subsection, detailed explanation is given.

Unlike gradient-search algorithm, models based on neural networks are created directly from data by an algorithm. However, under most cases, these models are "black boxes", which means that humans, even those who design them, cannot understand how variables are being combined to make predictions [38]. At this stage, the task of wireless communication does not require the interpretability of neural networks. However, the theoretical and comparative analyses of a neural network-based communication system are indispensable since an accurate interpretable model has been built.

Let the one-hot vector ${\bf{s}}\in{\mathbb{R}^{M}}$ be the input ${\bf{x}}$, ${{\bf{W}}^{\left(1\right)}}\in{\mathbb{R}^{m\times M}}$ is the first weight matrix, ${{\bf{W}}^{\left(h\right)}}\in{\mathbb{R}^{m\times m}}$ is the weight at the $h$-th layer for $2\leq h\leq{H}$ and $\sigma\left(\cdot\right)$ is the activation function. We assume intermediate layers are square matrices for the sake of simplicity. There are $H_{1}$ and $H_{2}$ hidden layers at the side of transmitter and receiver, respectively. The prediction function can be defined recursively as

where ${c_{\sigma}}={\left({{\mathbb{E}_{x\sim\mathcal{N}\left({0,1}\right)}}\left[{\sigma{{\left(x\right)}^{2}}}\right]}\right)^{-1}}$ is a scaling factor to normalize the input in the initialization phase, and $\left(H_{1}+1\right)$-th layer is defined as channel layer. Note that $\left[{H}\right]\setminus\left\{{{H_{1}}+1}\right\}$ is the set of elements that belong to $\left[{H}\right]$ but not to $\left\{{{H_{1}}+1}\right\}$. To constrain the average power of transmitted signal $P_{{\rm{av}}}$ to ${1/M}$, ${\bf{x}}^{H_{1}}$ is normalized as

Then, the effect on the transmitted signal resulting from a wireless channel can be expressed as

where ${h_{{\Theta_{h}}}}$ is the functional form of the wireless channel with parameter set $\Theta_{h}$²²2In accordance with practice, and without introducing ambiguity, $h$ is used to denote hidden layer and channel in the context of neural network and physical communication, respectively.. Let ${{\bf{x}}^{\left(H_{1}+1\right)}}=\bf{v}$, and finally, the received signal is demodulated as

where ${\sigma^{\left(H\right)}}\left(\cdot\right)={\rm{softmax}}\left(\cdot\right)$.

To theoretically analyze the neural network, some technical conditions on the activation function are imposed. The first condition is Lipschitz and Smooth. The second is that $\sigma\left(\cdot\right)$ is analytic and is not a polynomial function. In this section, softplus ${\sigma^{\left(h\right)}}\left(z\right)=\log\left({1+\exp\left(z\right)}\right)$ is chosen for the hidden layers except for the ${H_{1}}+1$ and $H$-th layers.

While training the deep neural network, randomly initialized gradient descent algorithm is used to optimize the empirical loss (2). Specifically, for every layer $h\in\left[{H}\right]\backslash\left\{{{H_{1}}+1}\right\}$, each entry is sample from a standard Gaussian distribution, ${\bf{W}}_{ij}^{\left(h\right)}\sim\mathcal{N}\left({0,1}\right)$. Then, the values for parameters can be updated by gradient descent, for $k=1,2,\ldots,$ and $h\in\left[{H}\right]\backslash\left\{{{H_{1}}+1}\right\}$ as

where $\eta>0$ is the step size.

The update of parameters ${\Theta_{g}}$ at the side of the receiver can be realized as

since we know the function ${g_{{\Theta_{g}}}}\left(\cdot\right)$ entirely. At the side of the transmitter, it becomes

where the terms $\frac{{\partial{g_{{\Theta_{g}}}}}}{{\partial{h_{{\Theta_{h}}}}}}$ and $\frac{{\partial{h_{{\Theta_{h}}}}}}{{\partial{f_{{\Theta_{f}}}}}}$ are difficult to acquire unless the knowledge about the channel ${{h_{{\Theta_{h}}}}}$ is fully known. In this paper, we consider both the Gaussian channel and the Rayleigh flat fading channel.

The convergence properties of the traditional algorithm, i.e., gradient-search algorithm, have been exhaustively analyzed in [37]. Whereas, the properties of the AE-abased algorithm have not been studied yet, and therefore, we mainly try to analyze the convergence properties of the AE-based algorithm in this subsection.

In [39], Simon S. Du et al. analyze two-layer fully connected ReLU activated neural networks. It has been shown that, with over-parameterization, gradient descent provably converges to the global minimum of the empirical loss at a linear convergence rate. Then, they develop a unified proof strategy for the fully-connected neural network, ResNet and convolutional ResNet [40].

We replace (2) with square loss function as

Then, the individual prediction at the $k$-th iteration can be denoted as

Let ${\bf{\hat{s}}}\left(k\right)={\left({{{\hat{s}}_{1}}\left(k\right),\ldots,{{\hat{s}}_{n}}\left(k\right)}\right)^{T}}\in{\mathbb{R}^{n}}$ where $n$ denotes the size of the training set. Simon S. Du et al. [39, 40] show that for DNN, the sequence $\left\{{{\bf{1}}-{\bf{\hat{s}}}\left(k\right)}\right\}_{k=0}^{\infty}$ admits the dynamics

where

The Gram matrix induced by the weights from $h$-th layer ${{\bf{G}}^{\left(h\right)}}\in{\mathbb{R}^{n\times n}}$ is defined as ${\bf{G}}_{ij}^{\left(h\right)}\left(k\right)=\left\langle{\frac{{\partial{{\hat{s}}_{i}}\left(k\right)}}{{\partial{{\bf{W}}^{\left(h\right)}}\left(k\right)}},\frac{{\partial{{\hat{s}}_{j}}\left(k\right)}}{{\partial{{\bf{W}}^{\left(h\right)}}\left(k\right)}}}\right\rangle$ for $h=1,\ldots,H$. Note for all $h\in\left[H\right]$, each entry of ${{\bf{G}}^{\left(h\right)}}$ is an inner product.

In [40], it has been shown that, if the width is large enough for all layers, for all $k=0,1,\ldots$, ${{\bf{G}}^{H}}\left(k\right)$ is close to a fixed matrix ${{\bf{K}}^{\left(H\right)}}\in{\mathbb{R}^{n\times n}}$ which depends on the input data, neural network architecture and the activation but does not depend on neural network parameters $\Theta$. The Gram matrix ${{\bf{K}}^{\left(H\right)}}$ is recursively defined as follows. For $\left({i,j}\right)\in\left[n\right]\times\left[n\right]$ and $h\in\left[{H-1}\right]$,

However, the existence of the channel layer obstructs this recursive process. Specifically, under the case of Gaussian channel, ${{\bf{W^{\prime}}}^{\left({{H_{1}}+1}\right)}}=\left[{{{\bf{I}}_{m}},{\bf{n}}}\right]$ results that the strictly positive definiteness of matrix ${{\bf{K}}^{\left(H\right)}}$ may not be guaranteed. This leads that the dynamics of gradient descent does not enjoy a linear convergence rate as (25) shown.

Under the case of Rayleigh flat fading channel, ${{\bf{W^{\prime}}}^{\left({{H_{1}}+1}\right)}}=\left[{{\bf{H}},{\bf{n}}}\right]$ makes the situation worse. Since the fading channel is not static, the diagonal elements of the weights matrix ${\bf{H}}\left(k\right)$ in the ${{\bf{W^{\prime}}}^{\left({{H_{1}}+1}\right)}}$ at $k$-th iterative are a random sample from ${{\cal N}_{m}}\left({0,{{\bf{I}}_{m}}}\right)$. At the stage of random initialization, suppose we have some perturbation ${\left\|{{{\bf{G}}^{\left(1\right)}}\left(0\right)-{{\bf{K}}^{\left(1\right)}}}\right\|_{2}}\leq{\varepsilon_{1}}$ in the first layer. This perturbation propagates to the $H$-th layer admits the form

Therefore, we need to have ${\varepsilon_{1}}\leq 1/{2^{O\left(H\right)}}$ and this makes $m$ have exponential dependency on $H$ [40]. Moreover, at the training stage, the perturbation in the $\left({{H_{1}}{\rm{+1}}}\right)$-th layer induced by fading channel can disperse to the whole network, i,e., the averaged Frobenius norm

is not small for all $k=0,1,\ldots$.

In addition, large biases $\bf{n}$ would be introduced by the channel noise when SNR is low. Although the biases do not impact the weights directly, its influence can be spread to the whole network through forward and backward propagation.

Let ${\cal H}_{k,{l}}^{\sigma}$ be the hypothesis class associated with a $l$-layer neural network of size $k$ with activation function $\sigma$. We assume that $\left|{\sigma\left(z\right)}\right|\leq 1$ for all $z\in\mathbb{R}$. More specifically, ${\cal H}_{k,{l}}^{\sigma}$ is the set of all functions $h:{{\mathbb{R}}^{d}}\to{{\mathbb{R}}^{d}}$. Given a random sample $D_{n}=\left\{{\left({{{\bf{v}}_{1}},{{\bf{z}}_{1}}}\right),\ldots,\left({{{\bf{v}}_{n}},{{\bf{z}}_{n}}}\right)}\right\}$, we define

i.e., ${\rm{MMS}}{{\rm{E}}_{k,l,n}}\left({{\bf{z}}|{\bf{v}}}\right)$ is the minimum empirical square loss attained by a $l$-layer neural network of size $k$. Mario Diaz et al. establishes a probabilistic bound for the MMSE in estimating ${\bf{z}}\in{\mathbb{R}^{1}}$ given ${\bf{v}}\in{\mathbb{R}^{d}}$ based on the 2-layer estimator ${\rm{MMS}}{{\rm{E}}_{k,2,n}}\left({{\bf{z}}|{\bf{v}}}\right)$, and the Barron constant of the conditional expectation of $\bf{z}$ given $\bf{v}$ [45, Theorem 1]. We extend this probabilistic bound to explore the MSE performance of DNN-based estimators.

If the joint probability distribution $p\left({{\bf{v}},{\bf{z}}}\right)$ is known, the expected (population) risk ${{\cal C}_{p\left({{\bf{v}},{\bf{z}}}\right)}}\left({{g_{{{\bf{\Theta}}_{g}}}},{{\cal L}_{\log}}}\right)$ can be written as

where $Q\left({\cdot|{\bf{v}}}\right){\rm{=}}{g_{{{\bf{\Theta}}_{g}}}}\left({\bf{v}}\right)\in p\left({\mathcal{Z}}\right)$ and ${D_{{\rm{KL}}}}\left({p\left({{\bf{z}}|{\bf{v}}}\right)||Q\left({{\bf{z}}|{\bf{v}}}\right)}\right)$ denotes Kullback-Leibler divergence between ${p\left({{\bf{z}}|{\bf{v}}}\right)}$ and ${Q\left({{\bf{z}}|{\bf{v}}}\right)}$ [29]⁴⁴4If $\bf{z}$ and $\bf{v}$ are continuous random variables, the sum becomes an integral when their PDFs exist.. If and only if the decoder is given by the conditional posterior ${g_{{{\bf{\Theta}}_{g}}}}\left({\bf{v}}\right){\rm{=}}p\left({{\bf{z}}|{\bf{v}}}\right)$, the expected (population) risk reaches the minimum $\mathop{\min}\limits_{{g_{{{\bf{\Theta}}_{g}}}}}{{\cal C}_{p\left({{\bf{v}},{\bf{z}}}\right)}}\left({{g_{{{\bf{\Theta}}_{g}}}},{{\cal L}_{\log}}}\right)=H\left({{\bf{z}}|{\bf{v}}}\right)$.

In physical layer communication, instead of perfectly knowing the channel-related joint probability distribution $p\left({{\bf{v}},{\bf{z}}}\right)$, we only have a set of $n$ i.i.d. samples ${D_{n}}:=\left\{{\left({{{\bf{v}}_{i}},{{\bf{z}}_{i}}}\right)}\right\}_{i=1}^{n}$ from $p\left({{\bf{v}},{\bf{z}}}\right)$. In this case, the empirical risk is defined as

Practically, the ${\mathcal{D}}_{n}$ from $p\left({{\bf{v}},{\bf{z}}}\right)$ usually is a finite set. This leads the difference between the empirical and expected (population) risks which can be defined as

We now can preliminarily conclude that the DNN-based receiver is an estimator with minimum empirical risk for a given set ${\mathcal{D}}_{n}$, whereas its performance is inferior to the optimal with minimum expected (population) risk under a given joint probability distribution $p\left({{\bf{v}},{\bf{z}}}\right)$.

Furthermore, it is necessary to quantitatively understand how information flows inside the decoder. Fig. 2(b) shows the graph representation of the decoder where ${{{\bf{t}}_{i}}}$ and ${{{\bf{t^{\prime}}}}_{i}}\left({1\leq i\leq S}\right)$ denote $i$-th hidden layer representations starting from the input layer and the output layer, respectively. Usually, the Shannon’s entropy cannot be calculated directly since the exact joint probability distribution of two variables is difficult to acquire. Therefore, we use the method proposed in [46] to illustrate layer-wise mutual information by three kinds of information planes (IPs) where the Shannon’s entropy is estimated by matrix-based functional of Rényi’s $\alpha$-entropy [47]. Its details are given in Appendix.