Achievable Rates and Low-Complexity Encoding of Posterior Matching for the BSC

Shannon [3] showed that feedback cannot increase the capacity of discrete memoryless channels (DMC). However, when combined with variable-length coding, Burnashev [4] showed that feedback can help increase the frame error rate’s (FER) decay rate as a function of blocklength. One such variable length coding method was pioneered by Horstein [5]. Horstein’s sequential transmission scheme was presumed to achieve the capacity of the BSC, which was later proved by Shayevitz and Feder [6] showing that it satisfies the criteria of a posterior matching scheme. A posterior matching (PM) scheme was defined by Shayevitz and Feder as one that satisfies the two requirements of the posterior matching principle:

1. 1.

   The input symbol at time $t+1$, $X_{t+1}$, is a fixed function of a random variable $U$, that is independent of the received symbol history $Y^{t}\triangleq\{Y_{1},Y_{2},\dots,Y_{t}\}$; and

2. 2.

   The transmitted message, $\theta$, can be uniquely recovered from $(U,Y^{t})$ a.s.

Gorantla and Coleman [7] used Lyapunov functions for an alternative proof that PM schemes achieve the channel capacity. Later, Li and El-Gamal [8] proposed a capacity achieving “posterior matching” scheme with fixed block-length for DMC channels. Their scheme used a random cyclic shift that was later used by Shayevitz and Feder for a simpler proof that Horstein’s scheme achieves capacity [9]. Naghshvar et. al. [10] proposed a variable length, single phase “posterior matching” scheme for discrete DMC channels with feedback that exhibits Burnashev’s optimal error exponent, and used a sub-martingale analysis to prove that it achieves the channel capacity. Bae and Anastasopoulos [11] proposed a PM scheme that achieves the capacity of finite state channels with feedback. Since then, other “posterior matching” algorithms have been developed, see [12, 13, 14, 15, 16]. Other variable length schemes that attain Burnashev’s optimal error exponent have also been developed, and some can be found in [17, 18, 19, 20, 21].

Feedback communication over the BSC in particular has been the subject of extensive investigation. Capacity-approaching, fixed-length schemes have been developed such as [8], but these schemes only achieve low frame error rates (FERs) at block sizes larger than 1000 bits. For shorter block lengths, capacity-approaching, variable-length schemes have also been developed, e.g., [5], [4], [10]. Recently, Yang et al. [2] provided the best currently available achievability bound for these variable-length schemes. Yang et al. derive an achievable rate using encoders that satisfy the small-enough-difference (SED) constraint. However, the complexity of variable-length schemes satisfying that constraint can grow quickly with message size, becoming too complex for practical implementation even at block lengths significantly below those addressed by the fixed-length schemes such as in [8].

In our precursor conference paper [1], we simplified the implementation of an encoder that enforces the SED constraint both by initially sending systematic bits and by grouping the messages according to Hamming distance from the received systematic bits. The contributions of the current paper include the following:

• •

  This paper provides a new analysis framework for posterior matching on the BSC that avoids martingale analysis in the communication phase in order to show that the achievable rate of [2] can be achieved with a broader set of encoders that satisfy less restrictive criteria than the SED constraint. Thm. 3 provides an example of a constraint, the small-enough-absolute-difference (SEAD) constraint, that meets the new, relaxed criteria.

• •

  The relaxed criteria allow a significant reduction of encoder complexity. Specifically, this paper shows that applying a new partitioning algorithm, thresholding of ordered posteriors (TOP), induces a partitioning that meets the SEAD constraints. The TOP algorithm facilitates further complexity reduction by avoiding explicit computation of posterior updates for the majority of messages, since those posterior updates are not required to compute the threshold position. This low-complexity encoding algorithm achieves that same rate performance that has been previously established for SED encoders in, e.g., [2].

• •

  Our new analysis further tightens the achievable rate bound provided in [2]. This new achievable rate lower bound applies to both the SED encoder analyzed in [2] and to our new, simpler, encoder.

• •

  We also show that using systematic transmissions as in [1] to initially send the message meets both the relaxed criteria including SEAD as well as the SED constraint. Complexity is reduced during the systematic transmission, with the required operations limited to simply storing the received sequence.

• •

  We generalize the concept of the “surrogate process” $U^{\prime}_{i}(t)$, used in Sec V-E of [2], to a broader class of processes that are not necessarily sub-martingales. The ability to construct such “surrogate” processes allows tighter bounds that also apply to the original process.

• •

  Taken together, these results demonstrate that variable-length coding with full noiseless feedback can closely approach capacity with modest complexity.

• •

  Regarding complexity, the simplified encoder associated with SEAD has a complexity below order $O(K^{2})$ and allows simulations for message sizes of at least $1000$ bits. The simplified encoder organizes messages according to their type, i.e. their Hamming distance from the received word, orders messages according to their posterior, and partitions the messages with a simple threshold without requiring any swaps.

• •

  Regarding proximity to capacity, our achievable rate bounds show that with codeword error rate of $10^{-3}$ SEAD posterior matching can achieve $96$% of the channel’s $0.50$-bit capacity for an average blocklength of $199.08$ bits corresponding to a message with $k=47$ bits. Simulations with our simplified encoder achieve $99$% of of the channel’s $0.50$-bit capacity for a target codeword error rate of $10^{-3}$ with an average block size of $201.08$ bits corresponding to a message with $k=49$ bits.

The rest of the paper proceeds as follows. Sec. II describes the communication process, introduces the problem statement, and reviews the highest existing achievability bound, by Yang et al. [2], as well as the scheme that achieves it, by Naghshvar et. al. [10]. Sec. III introduces Thms. 1, 2 and 3 that together relax the sufficient constraints to guarantee a rate above Yang’s lower bound and further tightens Yang’s bound. Sec. IV introduces Lemmas 1-5 and provides the proof of Thm. 1 via Lemmas 1-5. Sec. V provides the proofs of Lemmas 1-5, and the proof of Thm. 2 and Thm. 3. Sec. VI generalizes the new rate lower bound to arbitrary input distributions and derives an improved lower bound for the special case where a uniform input distribution is transformed into a binomial distribution through a systematic transmission phase. Sec. VII describes the TOP partitioning method and implements a simplified encoder that organizes messages according to their type, applies TOP, and employs initial systematic transmissions. Sec. VIII compares performance from simulations using the simplified encoder to the new achievability bounds. Sec. IX provides our conclusions. The AppendixX provides detailed proof of the second part of Thm. 3 and the proof of claim 1.

Our proposed communication scheme and simplified encoding algorithm are based on the single phase transmission scheme proposed by Naghshvar et. al. [10]. Before each transmission, both the transmitter and the receiver partition the message set $\Omega=\{0,1\}^{K}$ into two sets, $S_{0}$ and $S_{1}$. The partition is based on the received symbols $Y^{t}$ according to a specified deterministic algorithm known to both the transmitter and receiver. Then, the transmitter encodes $X_{t}=0$ if $\theta\in S_{0}$ and $X_{t}=1$ if $\theta\in S_{1}$, i.e.

After receiving symbol $Y_{t}$, the receiver computes the posterior probabilities:

The transmitter also computes these posteriors, as it has access to the received symbol $Y_{t}$ via the noiseless feedback channel, which allows both transmitter and receiver to use the same deterministic partitioning algorithm. The process repeats until the first time $\tau$ that a single message $i$ attains a posterior $\rho_{i}(y^{\tau})\geq 1-\epsilon$. The receiver chooses this message $i$ as the estimate $\hat{\theta}$. Since $\theta$ is uniformly sampled, every possible message $j\in\{0,1\}^{K}$ has the same prior: $\Pr(\theta=j)=2^{-K}$.

To prove that the SED scheme of Naghshvar et. al. [10] is a posterior matching BSC scheme as described in [6], it suffices to show that the the scheme uses the same encoding function as [6] applied to a permutation of the messages. Since the posteriors $\rho_{i}(y^{t})$ are fully determined by the history of received symbols $Y^{t}$, a permutation of the messages can be defined concatenating the messages in $S_{0}$ and $S_{1}$, each sorted by decreasing posterior. This permutation induces a c.d.f. on the corresponding posteriors. Then, to satisfy the posterior matching principle, the random variable $U$ could just be the c.d.f. evaluated at the last message before $\theta$. The resulting encoding function is given by $X_{t+1}=0$ if $U<1/2$, otherwise $X_{t+1}=1$.

Naghshvar et. al. proposed two methods to construct the partitions $S_{0}$ and $S_{1}$. The simplest one, described as the small enough difference encoder (SED) [2], consists of an algorithm that terminates when the SED constraint bellow is met:

The algorithm starts with all messages in $S_{0}$ and a vector of posteriors $\bm{\rho}_{t}\triangleq[\rho_{1}(y^{t}),\dots,\rho_{2^{K}}(y^{t})]$ of the messages $\{1,\dots,2^{K}\}$. The items are moved to $S_{1}$ one by one, from smallest to largest posterior. The process ends at any point where rule (3) is met. If the accumulated probability in $S_{0}$ falls below $\frac{1}{2}$, then the labelings of $S_{0}$ and $S_{1}$ are swapped, after which the process resumes.

The worst case scenario complexity of this algorithm is of order $O(M^{2})$, where $M=2^{K}$ is the number of posteriors. The $M$ is squared because part of the process repeats after every swap, and in the worst case scenario the number of swaps is proportional to $M$. However, a likely scenario is that the process ends after very few swaps, in which case the complexity is of order $O(M)=O(2^{K})$.

The second method by which Naghshvar et al. proposed to construct $S_{0}$ and $S_{1}$ consists of an exhaustive search over all possible partitions, i.e., the power set $2^{\Omega}$, and a metric to determine the optimal partition. This search would clearly include the partitioning of the first method, and therefore, also provide the guarantees of equations (9) and (14).

Yang et. al. [2] developed the upper bound (7) on the expected block length $\tau$ of the SED encoder that, to the best of our knowledge, is the best upper bound that has been developed for the model.

The analysis by Yang et al. consists of two steps. The first step, in [2] Thm. $7$, consists of splitting the single phase process from Naghshvar et. al. [10] into a two phase process: the communication phase, with stopping time $T$, where $\rho_{i}(y^{t})<\frac{1}{2}$ and a confirmation phase where $\rho_{i}(y^{t})\geq\frac{1}{2}$, when the transmitted message $\theta$ is the message $i$. This is a method first used by Burnahsev in [4]. With the first step alone, the following upper bound on the expected blocklength can be constructed:

where $C$ is the channel capacity, defined by $C\triangleq 1-H(p)$, and $H(p)\triangleq-p\log_{2}(p)-(1-p)\log_{2}(1-p)$, and the constants $C_{2}$ and $C_{1}$ from [2] are given by:

The second step, in [2] Lemma $4$, consists of synthesizing a surrogate martingale $U^{\prime}_{i}(t)$ with stopping time $T^{\prime}$ that upper bounds $T$, which is a degraded version of the sub-martingale $U_{i}(t)$. The martingale $U^{\prime}_{i}(t)$ guarantees that whenever $U^{\prime}_{i}(t)<0$, then $U^{\prime}_{i}(t+1)\leq\frac{1}{q}\log_{2}(2q)$ and while still satisfying the constraints needed to guarantee the bound (4). An achievability bound on the expected blocklength for the surrogate process, $U^{\prime}_{i}(t)$, is constructed from (4) by replacing some of the $C_{2}$ values by $\frac{1}{q}\log_{2}(2q)$. The new bound from [2] Lemma $4$ is given by:

This bound also applies to the original process $U_{i}(t)$, since the blocklength of the process $U^{\prime}_{i}(t)$ upper bounds that of $U_{i}(t)$. The bound (7) is lower because $\frac{1}{q}\log_{2}(2q)$ is smaller than $C_{2}$. The improvement is more significant as $p\rightarrow 0$ because $\frac{1}{q}\log_{2}(2q)$ grows from $0$ to $1$ as $p\rightarrow 0$, while $C_{2}$, instead, grows from $0$ to infinity. The rate lower bound is given by $\frac{K}{\mathsf{E}[\tau]}$, where $\mathsf{E}[\tau]$ is upper bounded by (7) from Thm. $7$ [2].

Let $\mathcal{F}_{t}\triangleq\sigma(Y^{t})$, the $\sigma$-algebra generated by the sequence of received symbols up to time $t$, where $Y^{t}=[Y_{1},Y_{2},\dots,Y_{t}]$. For each $i=1,\dots,M$, let the processes $U_{i}(Y^{t})$ by:

Yang et. al. show that the SED encoder from Naghshvar et. al. [10] guarantees that the following constraints (9)-(12) are met:

Meanwhile, Naghshvar et. al. show that the SED encoder also satisfied the more strict constraint that the average log likelihood ratio $\mathbf{U}(t)$, as defined in equation (13), is also a submartingale that satisfies equation (14), which is equivalent to (15):

The process $\mathbf{U}(t)$ is a weighted average of values $U_{i}(t)$, some of which increase and some of which decrease after the next transmission $t+1$.

To derive the bound (4), Yang et al. split the decoding time $\tau$ into $T$ and $\tau-T$, where $T$ is an intermediate stopping time defined by the first crossing into the confirmation phase. The expectation $\mathsf{E}[T]$ is analyzed in [2] as a martingale stopping time, and requires that if $\theta=i$, then $U_{i}(t)$ be a strict submartingale that satisfies the inequalities (9) and (10). The expectation $\mathsf{E}[\tau-T]$ is analyzed using a Markov Chain that exploits the larger and fixed magnitude step size (12) and inequality (11). Since $T$ is the time of the first crossing into the confirmation phase, the Markov Chain model, needs to include in the time $\tau-T$ the time that message $i$ takes to return to the confirmation phase if it has fallen back to the communication phase, that is: $\rho_{i}(y^{t})<\frac{1}{2}$ for some $t>T$.

We begin with a theorem that introduces relaxed conditions and shows that they guarantee the performance (4), corresponding to the first step of Yang’s analysis.

In equation (20) the values of $\rho_{j}(y^{t})$ and $U_{j}(t)$ are fixed since they are functions of the $y^{t}$ specified by the conditioning. Then, the expectation $\mathsf{E}[\rho_{j}(y^{t})|Y^{t}=y^{t},\theta=j])$ is simply the constant $\rho_{j}(y^{t})$, and we can also write (20) as the weighted sum of expectations:

Meanwhile the value of $U_{j}(t\!+\!1)$ for each $j$ is a random variable that takes on two possible values depending on the value of $Y_{t+1}$.

The sequential transmission process begins by randomly selecting a message $\theta$ from $\Omega$. Using that selected message, at each time $t$ until the decoding process terminates, the process computes an $X_{t}=x_{t}$, which induces a $Y_{t}=y_{t}$ at the receiver. The original constraint (9) dictates that $\{U_{i}(t),\theta=i\}$ is a sub-martingale and allows for a bound on $U_{i}(t)$ at any future time $t+s$ for any possible selected message $i$, i.e. $\mathsf{E}[U_{i}(t+s)\mid\mathcal{F}_{t},\theta=i]\geq U_{i}(t)+sC$. This is no longer the case with the new constraints in Thm. 1. While equation (16) of the new constraints makes the process $U_{i}(t)$ a sub-martingale, it only guarantees that $\mathsf{E}[U_{i}(t+s)\mid\mathcal{F}_{t},\theta=i]\geq U_{i}(t)+sa$ and $a$ could be any small positive constant. The left side of equation (20) is a sum that includes all $M$ realizations of the message, it is a constraint for each fixed time $t$ and each fixed event $Y^{t}=y^{t}$ that governs the behavior across the entire message set and does not define a sub-martingale. For this reason, the martingale analysis used by Naghshvar et. al. in [10] and Yang et al. in [2] no longer be applies.

A new analysis is needed to derive (21), the bound on the expected stopping time $\tau$, using only the constraints of Thm. 1. This new analysis needs to exploit the property that the expected stopping time is over all messages, that is: $\mathsf{E}[\tau]=\sum_{i=1}^{M}\Pr(\theta=i)\mathsf{E}[\tau\mid\theta=i]$ which the original does not use because it guarantees that the bound (7) holds for each message, i.e., $\mathsf{E}[\tau\mid\theta=i],i=1,\dots,M$. Note, however, that the original constraint (9) does imply that the new constraints are satisfied, so that the results we derive below also apply to the setting of Naghshvar et. al. in [10] and Yang et al. in [2]. The new constraints allow for a much simpler encoder and decoder design. This simpler design motivates our new analysis that forgoes the simplicity afforded by modeling the process $\{U_{i}(t),\theta=i\}$ as a martingale.

The new analysis seeks to accumulate the entire time that a message $i$ is not in its confirmation phase, i.e. the time during which the encoder is either in the communication phase or in some other message’s confirmation phase. For each $n=1,2,3,\dots$, let $T_{n}$ be the time at which the confirmation phase for message $i$ starts for the $n$th time (or the process terminates) and let $t^{(n)}_{0}$ be the time the encoder exits the confirmation phase for message $i$ for the $(n-1)^{th}$ time (or the process terminates). That is, for each $n=1,2,3,\dots$, let $t^{(n)}_{0}$ and $T_{n}$ be defined recursively by $t_{0}^{(1)}=0$ and :

Thus, the total time the process $U_{i}(t)$ is not in its confirmation phase is given by:

First we want to note that the bound (21) is loose compared to (7). It is loose because when the expectation $\mathsf{E}[\tau]$ is split in two parts, $\mathsf{E}[T]$ and $\mathsf{E}[\tau-T]$, to analyze them separately, a sub-optimal factor is introduced in the expression for $\mathsf{E}[T]$, which is $\frac{1}{C}(\log_{2}(M-1)+C_{2})$. The sub-optimality is derived from the term $C_{2}$, which is the largest value that $U_{i}(t)$ can take at the start of the confirmation phase and makes the term $\mathsf{E}[T]$ large. However, this large $C_{2}$ is not needed to satisfy any of the constraints in Thm. 1. To overcome this sub-optimality, we use a surrogate process that is a degraded version of the process $U_{i}(t)$, where the value at the start of the confirmation phase is bounded by a constant smaller than $C_{2}$. The surrogate process is a degradation in the sense that it is always below the value of the original process $U_{i}(t)$.

Perhaps the utility of the surrogate process can be better understood through the following frog-race analogy illustrated in Fig. 2. A frog $f_{1}$ traverses a race track of length $L$ jumping from one point to the next. The distance traveled by frog $f_{1}$ in a single jump is upper bounded by $u_{1}$. The jumps are not necessarily IID, but we know that the expected length of each jump is lower bounded by $l$. It is also possible that $f_{1}$ takes some jumps backwards. With only this information, we want to determine an upper bound on the average number of steps frog $f_{1}$ takes to reach the end of the track. This could be done using Doob’s optional stopping theorem [22] to compute the upper bound as $\frac{L+u_{1}}{l}$, the maximum distance $L+u_{1}$ traveled from the origin to the last jump divided by the lower bound on average distance $l$ of a single jump.

Perhaps this bound can be improved. The final point is located between $L$ and $L+u_{1}$ and is reached in a single jump from a point between $L-u_{1}$ and $L$. If for instance, the frog was restricted to only forward jumps, we could replace $u_{1}/l$ by just $1$, but the process $U_{i}(t)$ actually can take steps backwards. Instead we exploit another property of $U_{i}(t)$, which is that maximum step size $C_{2}$ is not needed to guarantee the lower bound $C$ on the average step size.

Suppose now that a surrogate frog $f_{2}$ participates in the race along $f_{1}$ but with the following restrictions:

1. 1.

   $f_{1}$ and $f_{2}$ start in the same place and always jump at the same time.

2. 2.

   $f_{2}$ is never ahead of $f_{1}$, i.e. when $f_{1}$ jumps forward, $f_{2}$ jumps at most as far, and when $f_{1}$ jumps backwards, $f_{2}$ jumps at least as far.

3. 3.

   Moreover, the forward distance traveled by frog $f_{2}$ in a single jump is upper bounded by $u_{2}<u_{1}$.

4. 4.

   Despite its slower progress, the surrogate frog $f_{2}$ still satisfies the property that the expected length of each jump is lower bounded by $l$.

The average number of steps taken by $f_{2}$ will be upper bounded by $\frac{L+u_{2}}{l}$, also by Doob’s optional stopping theorem. Since $f_{2}$ is never ahead of $f_{1}$ then $f_{2}$ crossing the finish line implies that $f_{1}$ has as well. Thus, $\frac{L+u_{2}}{l}$ is also an upper bound on the average number of jumps required for frog $f_{1}$ reach across $L$.

The equivalent to the surrogate frog $f_{2}$ is what we proceed to define in Thm. 2, where the length $L=\log(M-1)$, $u_{1}=C_{2}$, $u_{2}=B$ and $l=C$.

The following theorem introduces partitioning constraints that guarantee that the constraints in Thm. 2 are satisfied with a value of $B=\log_{2}(2q)/q$ for the surrogate process. The new constraints are looser than the original SED constraint, and therefore are satisfied by an encoder that enforces the SED constraint. Using a new analysis we show that this encoder guarantees an achievability bound tighter than the bound (7) obtained in the second step of Yang’s analysis described in Section II. The value $B=\log_{2}(2q)/q$ is the lowest possible $B$ value that satisfies the constraints (26)-(28) of Thm. 2 for a system that enforces the original SED constraint (3). The new achievability applies to an encoder that satisfies the new relaxed constraint as well as one that satisfies the SED constraint.

The requirement $\rho_{i}(y^{t})\geq\frac{1}{2}\implies S_{0}=\{i\}\;\text{or }S_{1}=\{i\}$, is needed to satisfy constraint (19) and guarantees constraint (18). This requirement is also enforced by the SED partitioning constraints in [10] and [2].

The SEAD partitioning constraint is satisfied whenever the SED constraint (3) is. However, the SEAD partitioning constraint allows for constructions of $S_{0}$, $S_{1}$ that do not meet either of the SED constraints in [10] and [2], and is therefore looser. Particularly, SEAD partitioning allows for the case where $P_{1}-P_{0}>\underset{j\in S_{1}}{\max}\rho_{j}(y^{t})$ that often arises in the implementation shown in Sec VII-A. This case is not allowed under either of the SED constraints because they both demand that:

The expression to bound the expectation expectation $\mathsf{E}[T]$ is constructed via five inequalities (or equalities) each of which derives from one of the following five Lemmas. The proofs of the Lemmas will be provided in Sec. V-A.