Capacity of a Nonlinear Optical Channel
with Finite Memory

A coherent optical communication link converts a discrete, complex-valued electric data signal $x_{k}$ to a modulated, continuous optical signal, which is transmitted through an optical fiber, received coherently, and then converted back to a discrete output sequence $Y_{k}$. The coherent link is particularly simple theoretically, in that the transmitter and receiver directly map the electric data to the optical field, which is a linear operation (in contrast with, e.g., direct-detection receivers), and can ideally be performed without distortions. The channel is then well described by the propagation of the (continuous) optical field in the fiber link. It should be emphasized that this assumes the coherent receiver to be ideal, with perfect synchronization and negligible phase noise. Experiments have shown [2] that commercial coherent receivers can indeed perform well enough for the fiber propagation effects to be the main limitations. Two main linear propagation effects in the fiber need to be addressed: dispersion and attenuation. The attenuation effects can be overcome by periodic optical amplification, at the expense of additive Gaussian noise from the inline amplifiers. The dispersion effects are usually equalized electronically by a filter in the coherent receiver. Such a linear optical link can be well-described by an AWGN channel, the capacity of which is unbounded with the signal power.

However, the fiber Kerr-nonlinearity introduces signal distortions, and greatly complicates the transmission modeling. The nonlinear signal propagation in the fiber is described by a nonlinear partial differential equation, the nonlinear Schrödinger equation (NLSE), which includes dispersion, attenuation, and nonlinearity. At high power levels, the three effects can no longer be conveniently separated. However, in contemporary coherent links (distance at least $500~{\textnormal{km}}$ and symbol rate at least $28~{\textnormal{Gbaud}}$), the nonlinearity is significantly weaker than the other two effects, and a perturbation approach can be successfully applied to the NLSE [5, 10, 11, 12]. This leads to the GN model, which will be described in Sec. II-C.

Even today’s highly dispersive optical links have a finite memory. For example, a signal with dispersive length $L_{\text{D}}=1/(\Delta\omega^{2}|\beta_{2}|)$, where $\beta_{2}$ is the group velocity dispersion and $\Delta\omega$ the optical bandwidth, broadens (temporally) a factor $L/L_{\text{D}}$ over a fiber of length $L$. With typical dispersion lengths of $5$–$50~{\textnormal{km}}$, this broadening factor can correspond to hundreds to thousands of adjacent symbols, a large but finite number. The same will hold for interaction among WDM channels; if one interprets $\Delta\omega$ as the channel separation, $L/L_{\text{D}}$ will give an approximation of the number of symbols that two WDM channels separate due to walk-off (and hence interact with nonlinearly during transmission). The channel memory will thus be even larger in the WDM case, and increase with channel separation, but the nonlinear interaction will decrease due to the shorter $L_{\text{D}}$. Thus, the principle of a finite channel memory holds also for WDM signals. To keep notation as simple as possible, we will consider a single, scalar, wavelength channel in this paper. Extensions to dual polarizations and WDM are possible, but will involve obscuring complications such as four-dimensional constellation space [33] in the former case and behavioral models [34] in the latter. We can thus say that in an optical link a certain signal may sense the interference from $N\approx L/L_{\text{D}}$ neighboring symbols, which is the physical reason for introducing a finite-memory model.

If we let the range $N$ of the interfering symbols go to infinity, an even simpler type of model is obtained. The interference is now averaged over infinitely many transmitted symbols. Assuming that an i.i.d. sequence is transmitted, this time average converges to a statistical average, which greatly simplifies the analysis. All models suggested for dispersive optical channels so far belong to this category [5, 23, 35, 10, 4, 11, 12, 14], of which the GN model described in Sec. II-C is the most common.

For a given transmitted complex symbol $x_{k}$, the (complex) single-channel output at each discrete-time $k\in\mathbb{Z}$ is modeled as

where $\{Z_{k}\}$ is a circularly symmetric, complex, white, Gaussian random sequence, independent of $x_{k}$. In (1), $Z_{k}$ is assumed to be independent of the actual transmitted sequence $x_{k}$. However, the variance of $Z_{k}$ depends on the transmit power, as detailed in Sec. II-C and II-D.

For coherent long-haul fiber-optical links without dispersion compensation, Splett et al. [5], Poggiolini et al. [7], and Beygi et al. [11] have all derived models where the nonlinear interference (NLI) appears as Gaussian noise, whose statistics depend on the transmitted signal power via a cubic relationship. The models assume that the transmitted symbols $x_{k}$ in time slot $k\in\mathbb{Z}$ are i.i.d. realizations of the same complex random variable $X$. In this model, the additive noise in (1) is given by

where $\{\tilde{Z}_{k}\}$ are i.i.d. zero-mean unit-variance circularly symmetric complex Gaussian random variables, $P_{\text{ASE}}$ and $\eta$ are real, nonnegative constants, and $P=\mathbb{E}[|X|^{2}]$ is the average transmit power. Therefore, the noise $Z_{k}$ is distributed as $Z_{k}\sim\mathcal{CN}(0,P_{\text{ASE}}+\eta P^{3})$, where $\mathcal{CN}(0,\sigma^{2})$ denotes a circularly symmetric complex Gaussian random variable with mean $0$ and variance $\sigma^{2}$. The parameter $P$, which is a property of the transmitter, governs the behavior of the channel model. This can be intuitively understood as a long-term average of the input power. Mathematically,

for any given $k$, still assuming i.i.d. symbols $x_{k}$. For this reason, we will refer to models that depend on infinitely many past and/or future symbols, via $P$ in (3) or in some other way, as infinite-memory models.

The cubic relation in (2) between the transmit power and the additive noise variance $P_{\text{ASE}}+\eta P^{3}$ is a consequence of the Kerr nonlinearity, and holds for both lumped and distributed amplification schemes. The constant $P_{\text{ASE}}$ represents the total amplified spontaneous emission (ASE) noise of the optical amplifiers for the channel under study, while $\eta$ quantifies the NLI. Several related expressions for this coefficient have been proposed. For example, for distributed amplification and WDM signaling over the length $L$,

were proposed in [5] and [36], resp., where $\gamma$ is the fiber nonlinear coefficient, $B$ is the total WDM bandwidth, and $R_{\text{s}}$ is the symbol rate. Obviously, the expressions in (4) and (5) are qualitatively similar. For dual polarization and single channel transmission over $M$ lumped amplifier spans, the expression

was proposed in [14], and a qualitatively similar formula can be obtained from the results in [10]. Here, $\alpha$ is the attenuation coefficient of the fiber and the coefficient $\epsilon$ is between 0 and 1 (see [10, 14]) depending on how well the nonlinear interference decorrelates between each amplifier span. For single polarization transmission, the coefficient $3$ in (6) should be replaced by $2$ [11].

The benefits of the GN model is that it is very accurate for uncoded transmission with traditional modulation formats,¹¹1The model is not valid for exotic modulation formats such as satellite constellations [37]. as demonstrated in experiments and simulations [38, 15, 9], and that it is very simple to analyze. It is, however, not intended for nonstationary input sequences, i.e., sequences whose statistics vary with time, because the transmit power $P$ in (2) is defined as the (constant) power of a random variable that generates the i.i.d. symbols $x_{k}$. In order to capture the behavior of a wider class of transmission schemes, the GN model can be modified to depend on a time-varying transmit power, which is the topic of the next section.

As mentioned in Sec. I and II-C, a finite-memory model is essential in order to model the channel output corresponding to time-varying input distributions. Therefore, we refine the GN model in Sec. II-C to make it explicitly dependent on the channel memory $N$, in such a way that the model “converges” to the regular GN model as $N\rightarrow\infty$. Many such models can be formulated. In this paper, we aim for simplicity rather than accuracy.

The proposed model assumes that the input–output relation is still given by (1), but the average transmit power $P$ in (2) is replaced by an empirical power, i.e., by the arithmetic average of the squared magnitude of the symbol $x_{k}$ and of the $2N$ symbols around it. Mathematically, (2) is replaced by

for any $k\in\mathbb{Z}$, where $N$ is the (one-sided) channel memory. We refer to (1) and (7) as the finite-memory GN model. Since (second-order) group velocity dispersion causes symmetric broadening with respect to the transit time of the signal, inter-symbol interference from dispersion will act both backwards and forwards in terms of the symbol index. This is why both past and future inputs contribute to the noise power in (7). A somewhat related model for the additive noise in the context of data transmission in electronic circuits has been recently proposed in [39], where the memory is single-sided and the noise scales linearly with the signal power, not cubically as in (7).

Having introduced the finite-memory GN model, we now discuss some particular cases. First, the memoryless AWGN channel model can be obtained from both the GN and finite-memory GN models by setting $\eta=0$. In this case, the noise variance is $\mathds{E}[|Z_{k}|^{2}]=P_{\text{ASE}}$ for all $k$. Second, let us consider the scenario where the transmitted symbols is the random process $\{X_{i}\}$. Then the empirical power $(1/(2N+1))\sum_{i=k-N}^{k+N}|X_{i}|^{2}$ at any discrete time $k$ is a random variable that depends on the magnitude of the $k$th symbol and the $2N$ symbols around it. In the limit $N\rightarrow\infty$, this empirical power converges to the “statistical” power $P$ in (3), for any i.i.d. process with power $P$, as mentioned in Sec. II-C. This observation shows that the proposed finite-memory model in (7) “converges” to the GN model in (2), provided that the channel memory $N$ is sufficiently large and that the process consists of i.i.d. symbols with zero mean and variance $P$.

The purpose of the finite-memory model is to be able to predict the output of the channel when the transmitted symbols are not i.i.d. This is the case for example when the transmitted symbols are a nonstationary process (as will be exemplified in Sec. II-E) and also for coded sequences (which we discuss in Sec. IV). An advantage of the finite-memory model, from a theoretic viewpoint, is that the input–output relation of the channel is modeled as a fixed conditional probability of the output given the input and its history, which is the common notion of a channel model in communication and information theory ever since the work of Shannon [16], [40, p. 74]. This is in contrast to the regular GN model and other channel models, whose conditional distribution change depending on which transmitter the channel is connected to. Specifically, the GN model is represented by a family of such conditional distributions, one for each value of the transmitter parameter $P$.

A drawback with the proposed finite-memory model is that it is more complex than the GN model. Also, our model is not accurate for small values of $N$, since the GN assumption relies on the central limit theorem [7, 11, 12]. Furthermore, we assumed that all the $2N$ symbols around the symbol $x_{k}$ affect the noise variance equally. In practice, this is not the case. We nevertheless use the proposed model in this paper because it is relatively easy to analyze (see Sec. III and IV) and because even this simple finite-memory model captures the quantitative effects caused by non-i.i.d. symbols, which is essential for the capacity analysis in Sec. IV.

Before analyzing the finite-memory GN model, we first quantify the chromatic dispersion of the optical fiber. To this end, we simulated the transmission of a single symbol pulse over a over a single-channel, single-polarization fiber link without dispersion compensation. Ten amplifiers spans over a total distance of $700~{\textnormal{km}}$ are simulated using the lossy NLSE model. We used a raised-cosine pulse with peak power $0.1~{\textnormal{mW}}$ and a duration of $15.6~{\textnormal{ps}}$ at half the maximum amplitude, which corresponds to half the symbol slot in a $32~{\textnormal{Gbaud}}$ transmission system. The result is illustrated in Fig. 1. At this low power, the nonlinear effects are almost negligible. For clarity of illustration, the ASE noise was neglected by setting $P_{\text{ASE}}=0$. The remaining system parameters are given in Table I and will be used throughout the paper, except when other values are explicitly stated. As we can see, the pulse broadens as it propagates along the fiber, having a width corresponding to about $100$ data symbols after $700~{\textnormal{km}}$ of transmission, or a half-width of $N=50$ symbols. This is in good agreement with the relation for symbol memory used in [41, p. 2037], which gives $2N\approx 2\pi|\beta_{2}|LR_{\text{s}}^{2}=97$.

Next, to validate the behavior of the finite-memory model with nonstationary input symbol sequences, we simulated the transmission of independent quadrature phase-shift keying (QPSK) data symbols with a time-varying magnitude, over the same $700$ km fiber link, at $R_{\textnormal{s}}=32~{\textnormal{Gbaud}}$. The transmitted sequence consists of $128$ symbols with $4~{\textnormal{mW}}$ of average signal power, 128 symbols at $0~{\textnormal{mW}}$ power, 128 symbols at $4~{\textnormal{mW}}$, and so on. The statistical power is then $2~{\textnormal{mW}}$. The chosen pulse shape is a raised-cosine return-to-zero pulse. In Fig. 2, we show the amplitude of the transmitted symbols $|x_{k}|$ (red) and received symbols $|Y_{k}|$ (blue) with three different models: the NLSE, the finite-memory GN model with $N=50$, and the regular GN model. In the middle and lower plots of Fig. 2, we used the NLI coefficient $\eta=7244~{\textnormal{W}}^{-2}$, which was calculated from (6), using $\epsilon=0$ for simplicity. Also in Fig. 2, we used $P_{\text{ASE}}=0$ to better illustrate the properties of the nonlinear models.

As can be seen, the agreement between the NLSE simulations and the finite memory model is quite reasonable, but the GN model cannot capture the nonstationary dynamics. The results in Fig. 2 also show that the noise variance in the NLSE simulation is low around the symbols with low input power and high around the symbols with high input power. This behavior is captured by the finite-memory GN model but not by the regular GN model, for which the variance of the noise is the same for any time instant. This illustrates that the GN model (2) should be avoided with nonstationary symbol sequences as the ones used in Fig. 2. This is not surprising, as the model was derived under an i.i.d. assumption. In Sec. V, we will return to this observation when analyzing coded transmission. We believe that the finite-memory GN model proposed here, albeit idealized, is the first model that is able to deal with nonstationary symbol sequences.

We consider the equally spaced 16-QAM constellation shown in Fig. 3. In this case, $\mathcal{S}=\{a+b\sqrt{-1}:a,b\in\{\pm\Delta,\pm 3\Delta\}\}$, the minimum Euclidean distance (MED) of the constellation is $2\Delta$, and the statistical power is $P=10\Delta^{2}$. The binary labeling is the binary reflected Gray code (BRGC) [42], where the first two bits determine the in-phase (real) component of the symbols and the last two bits determine the quadrature (imaginary) components of the symbols. This is shown with colors in Fig. 3.

The maximum-likelihood (ML) symbol-by-symbol detection rule for a given sequence $\boldsymbol{x}^{\text{mem}}_{k}$ chooses the symbol $s_{i}\in\mathcal{S}$ that maximizes $f_{Y_{k}|X_{k},\boldsymbol{X}^{\text{mem}}_{k}}(y|s_{i},\boldsymbol{x}^{\text{mem}}_{k})$ in (III). The decision made by this detector can be expressed as

which shows that, due to the dependency of $\log{\rho(\left\lvert s_{i}\right\rvert^{2}\!+\|\boldsymbol{x}^{\text{mem}}_{k}\|^{2})}$ on $s_{i}$, this detector is not an MED detector. For simplicity, however, we disregard this term and study the MED detector, which chooses the symbol $s_{i}$ being closest, in Euclidean distance, to the channel output $y$. Thus

where $\mathcal{V}_{i}$ denotes the decision region, or Voronoi region, of $s_{i}$.

The following two theorems give closed-form expressions for the BER and SER for the constellation in Fig. 3 when used over the finite-memory GN model.

The BER and SER in the limit $N\to\infty$ can be inferred from Theorems 1 and 2 as shown in the next corollary.

The other extreme case to consider is the memoryless AWGN channel. The BER and SER expressions in this case are given in the following corollary.

The results in Corollaries 1 and 2 correspond to the well-known expressions for the BER and SER for the AWGN channel. In particular, (26) can be found in [43, eq. (10)], [44, eq. (10.36a)] and (27) in [44, eq. (10.32)]. Also, the results in Corollary 2 together with (2) show that the BER and SER for the finite-memory GN model when $N\rightarrow\infty$ converge to the BER and SER for the regular GN model.

We consider the same scenario as in Sec. II-E, with parameters according to Table I. The BER and SER for the 16-QAM constellation in Fig. 3 given by Theorems 1 and 2 are shown in Fig. 4 for different values of $N$. Fig. 4 also shows the asymptotic cases $N=0$ and $N\rightarrow\infty$ given by Corollaries 1 and 2, respectively. Furthermore, results obtained via computer simulations of (1)–(2) are included using the ML detector in (13), marked with squares, and the MED detector in (III-A), marked with circles. As expected, the MED detector yields a perfect match with the analytical expressions, whereas the ML detector deviates slightly for small channel memories.

The results in Fig. 4 show that in the low-input-power regime, the memory in the channel plays no role for the BER and SER, and all the curves follow closely the BER and the SER of a memoryless AWGN channel. However, as $P$ increases, the memory kicks in, causing the BER and SER for finite $N$ to have a minimum, and then to increase as $P$ increases. Physically, this can be explained as follows: in the low-power regime, the BER is limited by the ASE noise, which is independent of the memory depth. In the high-power regime, the Kerr-induced noise dominates, resulting in increasing BER with power. Similar behavior has been reported in most experiments and simulations on nonlinearly-limited links, e.g., [45, 46, 9, 11], [47, Ch. 9]. The reason why the performance improves slightly with the memory depth $N$ is the nonlinear scaling of the Kerr-induced noise. For $N=1$, sequences of two or more high-amplitude symbols will receive high noise power and dominate the average BER. For higher $N$, longer (and less probable) sequences of high-amplitude symbols are required to receive the same, high, noise power. Thus on average the performance improves with $N$, up to a limit given by the GN model.

The results in Fig. 4 also show how the finite-memory model in the high-input power regime approaches the GN model. For $N=50$, the two models yield very similar BER and SER curves.

Fig. 5 shows a generic coded communication system where a message $j$ is mapped to a codeword $\boldsymbol{x}=[x_{1},\dots,x_{n}]$. This codeword is then used to modulate a continuous-time waveform, which is then transmitted through the physical channel. At the receiver’s side, the continuous-time waveform is processed (filtered, equalized, synchronized, matched filtered, sampled, etc.) resulting in a discrete-time observation $\boldsymbol{Y}=[Y_{1},\dots,Y_{n}]$, which is a noisy version of the transmitted codeword $\boldsymbol{x}$. The decoder uses $\boldsymbol{Y}$ to estimate the transmitted message $j$.

When designing a coded communication system, the first step is to choose the set of codewords (i.e., the codebook) that will be transmitted through the channel. Once the codebook has been chosen, the mapping rule between messages and codewords should be chosen, which fully determines the encoding procedure. At the receiver side, the decoder block will use the mapping rule used at the transmitter (as well as the channel characteristics) to give an estimate $\hat{\jmath}$ of the message $j$. The triplet codebook, encoder, and decoder forms a so-called coding scheme. Practical coding schemes are designed so as to minimize the probability that $\hat{\jmath}$ differs from $j$, while at the same time keeping the complexity of both encoder and decoder low.

Channel capacity is the largest transmission rate at which reliable communications can occur. More formally, let $(n,M,\epsilon)$ be a coding scheme consisting of:

• •

  An encoder that maps a message $j\in\{1,\dots,M\}$ into a block of $n$ transmitted symbols $\boldsymbol{x}=[x_{1},\dots,x_{n}]$ satisfying a per-codeword power constraint

  $\displaystyle\frac{1}{n}\sum_{l=1}^{n}|x_{l}|^{2}=P.$

  (28)

• •

  A decoder that maps the corresponding block of received symbols $\boldsymbol{Y}=[Y_{1},\dots,Y_{n}]$ into a message $\hat{\jmath}\in\{1,\dots,M\}$ so that the average error probability, i.e., the probability that $\hat{\jmath}$ differs from $j$, does not exceed $\epsilon$.

Observe that $P$ here is defined differently from in previous sections. It still represents the average transmit power, but while this quantity is Sec. II–III was interpreted in a statistical sense as the mean of an i.i.d. random variable, it is in this section the exact power of every codeword.

The maximum coding rate $R^{*}(n,\epsilon)$ (measured in bit/symbol) for a given block length $n$ and error probability $\epsilon$ is defined as the largest ratio $(\log_{2}M)/n$ for which an $(n,M,\epsilon)$ coding scheme exists. The channel capacity $C$ is the largest coding rate for which a coding scheme with vanishing error probability exists, in the limit of large block length,

By Shannon’s channel coding theorem, the channel capacity of a discrete-time memoryless channel, in bit/symbol, can be calculated as [16], [17, Ch. 7]

where $I(X;Y)$ is the mutual information (MI)

and the maximization in (30) is over all probability distributions $f_{X}$ that satisfy $\mathbb{E}[|X|^{2}]=P$, for a given channel $f_{Y|X}$.

Roughly speaking, a transmission scheme that operates at an arbitrary rate $R<C$ can be designed by creating a codebook of $M=2^{nR}$ codewords of length $n$, whose elements are i.i.d. random samples from the distribution $f_{X}$ that maximizes the mutual information in (30). This codebook is stored in both the encoder and decoder. During transmission, the encoder maps each message $j$ into a unique codeword $\boldsymbol{x}$, and the decoder identifies the codeword that is most similar, in some sense, to the received vector $\boldsymbol{Y}$. An arbitrarily small error probability $\epsilon$ can be achieved by choosing $n$ large enough. This random coding paradigm was proposed already by Shannon [16]. In practice, however, randomly constructed codebooks are usually avoided for complexity reasons.

Since the additive noise in (2) is statistically independent of $X_{k}$, the channel capacity of the GN model (2) can be calculated exactly as [5, 8]

using Shannon’s well-known capacity expression [16, Sec. 24], [17, Ch. 9]. The capacity in (32) can be achieved by choosing the codewords $\boldsymbol{x}$ to be drawn independently from a Gaussian distribution $\mathcal{CN}(0,P)$.

Considered as a function of the transmitted signal power $P$, the capacity in (32) has the peculiar behavior of reaching a peak and eventually decreasing to zero at high enough power, since the denominator of (32) increases faster than the numerator. This phenomenon, sometimes called the “nonlinear Shannon limit” in the optical communications community, conveys the message that reliable communication over nonlinear optical channels becomes impossible at high powers. In the following sections, we shall question this pessimistic conclusion.

The capacity of channels with memory is, under certain assumptions on information stability [48, Sec. I],

where $\boldsymbol{X}_{i}^{j}=(X_{i},X_{i+1},\ldots,X_{j})$, $I(\boldsymbol{X}_{i}^{j};\boldsymbol{Y}_{i}^{j})$ is defined as a multidimensional integral analogous to (31), and the maximization is over all joint distributions of $X_{1},\ldots,X_{n}$ satisfying $\mathds{E}\mathopen{}\left[\|\boldsymbol{X}_{1}^{n}\|^{2}\right]=nP$. In this context, it is worth emphasizing that the maximization in (33) includes sequences $X_{1},\ldots,X_{n}$ that are not i.i.d. Hence, in order to calculate the channel capacity of a transmission link, it is essential that the employed channel model allows non-i.i.d. inputs.

An exact expression for the channel capacity of the finite-memory GN model (7) is not available. Shannon’s formula, which leads to (32), does not apply here, because the sequences $\{X_{k}\}$ and $\{Z_{k}\}$, where $Z_{k}$ was defined in (7), are dependent. A capacity estimation via (33) is numerically infeasible, since it involves integration and maximization over high-dimensional spaces. We therefore turn our attention to bounds on the capacity for the finite-memory model. Every joint distribution of $X_{1},\ldots,X_{k}$ satisfying $\mathds{E}\mathopen{}\left[\|\boldsymbol{X}_{1}^{n}\|^{2}\right]=nP$ gives us a lower bound on capacity. Thus,

for any random process $\{X_{k}\}$ such that the limit exists.

In this section, a lower bound on (33) is derived by applying (34) to the following random input process. In every block of $2N+1$ consecutive symbols, we let the first $N$ symbols and the last $N$ symbols have a constant amplitude, whereas the amplitude of the symbol in the middle of the block follows an arbitrary distribution. The phase of each symbol in the block is assumed uniform. With this random input process, illustrated in Fig. 6, the memory in (7) depends only on a single variable-amplitude symbol. This enables us to derive an analytical expression for the resulting capacity lower bound in (34).

The bound will be numerically computed in the next section.

Theorem 3 yields a lower bound on capacity for every constant $r_{1}$ and every probability distribution $f_{R}$ satisfying (35). Instead of optimizing the bound over all distributions $f_{R}$, which is of limited interest, since the theorem itself provides only a lower bound on capacity, we study a heuristically chosen family of distributions and optimize its parameters along with the constant amplitude $r_{1}$.

An attractive distribution in this context is to let the variable-amplitude symbols follow a circularly symmetric bivariate t-distribution [49, p. 86], [50, p. 1],

where $X$ (with magnitude $R=\left\lvert X\right\rvert$) denotes one such variable-amplitude symbol, $\nu$ is a shape parameter, and $s$ scales the variance, which equals [50, p. 11] $\mathbb{E}[|X|^{2}]=\mathbb{E}[R^{2}]=2\nu s/(\nu-2)$ if $\nu>2$ and is otherwise undefined. The shape of this distribution is similar to a Gaussian, but the heaviness of the tail can be controlled via the shape parameter $\nu$: the closer $\nu$ is to $2$, the heavier tail. This is, as we shall see later, what makes it an interesting choice for nonlinear optical channels.

Again, we consider the same scenario as in Sec. II-E, with the system parameters given in Table I. The distribution of $R=|X|$ is given by $f_{R}(r)=2\pi rf_{X}(r)$, with $f_{X}$ given by (38). The power constraint (35), which reduces to

leaves two degrees of freedom to optimize for each $P$, which we can take to be the shape parameter $\nu$ and the ratio $r_{1}^{2}/s$.

The lower bound on the capacity of the finite-memory model given by Theorem 3 is shown in Fig. 7 as a function of $P$, $\nu$, and $r_{1}^{2}/s$, for the special case $N=1$. The expectation in (36) was estimated by Monte Carlo integration. It can be seen that as the transmit power $P$ increases, the optimum shape parameter $\nu$ gets closer and closer to $2$. In other words, the tail gets heavier, so that at high power, it consumes almost all power, while the probability of transmitting a high amplitude $R$ is still small. In this sense, a t-distribution with a shape parameter near $2$ is similar to a satellite constellation [37].

Selecting the optimum parameters $\nu$ and $r_{1}^{2}/s$ for every power $P$, the capacity bound is plotted in Fig. 8 as a function of transmit power $P$, for selected values of the channel memory $N$. The figure also shows the AWGN channel capacity and the exact capacity of the GN model given by (32). In the linear regime, the capacity bound is close to the AWGN capacity if $N=0$, because the t-distribution is, at high values of $\nu$, approximately equal to the capacity-achieving Gaussian distribution. As $N$ increases, the capacity bound tends, still in the linear regime, to the mutual information of constant-amplitude transmission [51, 52].

Interestingly, we can see that as $N$ increases, the curves approach an asymptotic bound (the curves for $N=10$, $20$, and $50$ almost overlap). It follows that reliable communication in the high input power regime is indeed possible for every finite $N$. This result should be compared with the regular GN model, whose capacity (32) decreases to zero at high average transmit power [8]. It may seem contradictory that the GN model, which can be characterized as a limiting case of the finite-memory model (cf. (7) and (2)–(3)), nevertheless exhibits a fundamentally different channel capacity. This can be intuitively understood as follows. For every block of $2N+1$ symbols, we transmit $2N$ constant-amplitude symbols with low power and only one symbol with variable (potentially very large) power. Although the amplitude of this variable-power symbol is chosen so that the average power constraint is satisfied according to (35) (which requires averaging across many blocks of length $2N+1$), the convergence to average power illustrated in (3) does not occur within a block, even when $N$ is taken very large.

It can be observed that the lower bounds in Fig. 8 all exhibit a low peak, before they converge to their asymptotic values at high $P$. Such bounds can always be improved using the law of monotonic channel capacity [53]. Cast in the framework of this paper, this law states that the channel capacity never decreases with power for any finite-memory channel. This law does not give a capacity lower bound per se, but it provides an instrument by which a lower bound at a certain power $P$ can be propagated to any power greater than $P$. Hence, the part of the curves in Fig. 8 to the right of the peaks can be lifted up to the level of the peaks, which would yield a marginally tighter lower bounds (dashed lines in Fig. 8).