Temporal Difference Learning with Compressed Updates:
Error-Feedback meets Reinforcement Learning

Stochastic gradient descent (SGD) is at the heart of large-scale distributed machine learning paradigms such as federated learning (FL) [1]. In these applications, the task of training high-dimensional weight vectors is distributed among several workers that exchange information over networks of limited bandwidth. While parallelization at such an immense scale helps to reduce the computational burden, it creates several other challenges: delays, asynchrony, and most importantly, a significant communication bottleneck. The popularity and success of SGD can be attributed in no small part to the fact that it is extremely robust to such deviations from ideal operating conditions. In fact, by now, there is a rich literature that analyzes the robustness of SGD to a host of structured perturbations that include lossy gradient-quantization [2, 3] and sparsification [4, 5]. For instance, SignSGD - a variant of SGD where each coordinate of the gradient is replaced by its sign - is extensively used to train deep neural networks [6, 7]. Inspired by these findings, in this paper, we ask a different question: Are common reinforcement learning (RL) algorithms also robust to similar structured perturbations?

Motivation. Perhaps surprisingly, despite the recent surge of interest in multi-agent/federated RL, almost nothing is known about the above question. To fill this void, we initiate the study of a robustness theory for iterative RL algorithms with compressed update directions. We primarily focus on the problem of evaluating the value function associated with a fixed policy $\mu$ in a Markov decision process (MDP). Just as SGD is the workhorse of stochastic optimization, the classical temporal difference (TD) learning algorithm [8] for policy evaluation forms the core subroutine in a variety of decision-making algorithms in RL (e.g., Watkin’s Q-learning algorithm). In fact, in their book [9], Sutton and Barto mention: “If one had to identify one idea as central and novel to reinforcement learning, it would undoubtedly be temporal-difference (TD) learning.” Thus, it stands to reason that we center our investigation around a variant of the TD(0) learning algorithm with linear function approximation, where the TD(0) update direction is replaced by a compressed version of it. Moreover, to account for general biased compression operators (e.g., sign and Top-$k$), we couple this scheme with an error-feedback mechanism that retains some memory of TD(0) update directions from the past.

Other than the robustness angle, a key motivation of our work is to design communication-efficient algorithms for the emerging paradigms of multi-agent RL (MARL) and federated RL (FRL). In existing works on these topics [10, 11, 12, 13], agents typically exchange high-dimensional models (parameters) or model-differentials (i.e., gradient-like update directions) over low-bandwidth channels, keeping their personal data (i.e., rewards, states, and actions) private. In the recent survey paper on FRL by [11], the authors explain how the above issues contribute to a major communication bottleneck, just as in the standard FL setting. However, no work on MARL or FRL provides any theory whatsoever when it comes to compression/quantization in RL. Our work takes a significant step towards filling this gap via a suite of comprehensive theoretical results that we discuss below.

We consider a Markov Decision Process (MDP) denoted by $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, where $\mathcal{S}$ is a finite state space of size $n$, $\mathcal{A}$ is a finite action space, $\mathcal{P}$ is a set of action-dependent Markov transition kernels, $\mathcal{R}$ is a reward function, and $\gamma\in(0,1)$ is the discount factor. For the majority of the paper, we will be interested in the policy evaluation problem where the goal is to evaluate the value function $V_{\mu}$ of a given policy $\mu$; here, $\mu:\mathcal{S}\rightarrow\mathcal{A}$. The policy $\mu$ induces a Markov reward process (MRP) characterized by a transition matrix $P$, and a reward function $R$.²²2For simplicity of notation, we have dropped the dependence of $P$ and $R$ on the policy $\mu$. In particular, $P(s,s^{\prime})$ denotes the probability of transitioning from state $s$ to state $s^{\prime}$ under the action $\mu(s)$; $R(s)$ denotes the expected instantaneous reward from an initial state $s$ under the action of the policy $\mu$. The discounted expected cumulative reward obtained by playing policy $\mu$ starting from initial state $s$ is given by:

where $s_{t}$ represents the state of the Markov chain (induced by $\mu$) at time $t$, when initiated from $s_{0}=s$. It is well-known [29] that $V_{\mu}$ is the fixed point of the policy-specific Bellman operator $\mathcal{T}_{\mu}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$, i.e., $\mathcal{T}_{\mu}V_{\mu}=V_{\mu}$, where for any $V\in\mathbb{R}^{n}$,

Linear Function Approximation. In practice, the size of the state space $\mathcal{S}$ can be extremely large. This renders the task of estimating $V_{\mu}$ exactly (based on observations of rewards and state transitions) intractable. One common approach to tackle this difficulty is to consider a parametric approximation $\hat{V}_{\theta}$ of $V_{\mu}$ in the linear subspace spanned by a set $\{\phi_{k}\}_{k\in[K]}$ of $K\ll n$ basis vectors, where $\phi_{k}=[\phi_{k}(1),\ldots,\phi_{k}(n)]^{\top}\in\mathbb{R}^{n}$. Specifically, we have $\hat{V}_{\theta}(s)=\sum_{k=1}^{K}\theta(k)\phi_{k}(s),$ where $\theta=[\theta(1),\ldots,\theta(K)]^{\top}\in\mathbb{R}^{K}$ is a weight vector. Let $\Phi\in\mathbb{R}^{n\times K}$ be a matrix with $\phi_{k}$ as its $k$-th column; we will denote the $s$-th row of $\Phi$ by $\phi(s)\in\mathbb{R}^{K}$, and refer to it as the feature vector corresponding to state $s$. Compactly, $\hat{V}_{\theta}=\Phi\theta$, and for each $s\in\mathcal{S}$, we have that $\hat{V}_{\theta}(s)=\langle\phi(s),\theta\rangle$. As is standard, we assume that the columns of $\Phi$ are linearly independent, and that the feature vectors are normalized, i.e., for each $s\in\mathcal{S}$, $\|\phi(s)\|^{2}_{2}\leq 1.$

The TD(0) Algorithm. The goal now is to find the best parameter vector $\theta^{*}$ that minimizes the distance (in a suitable norm) between $\hat{V}_{\theta}$ and $V_{\mu}$. To that end, we will focus on the classical TD(0) algorithm within the family of TD learning algorithms. Starting from an initial estimate $\theta_{0}$, at each time-step $t=0,1,\ldots$, this algorithm receives as observation a data tuple $X_{t}=(s_{t},s_{t+1},r_{t}=R(s_{t}))$ comprising of the current state, the next state reached by playing action $\mu(s_{t})$, and the instantaneous reward $r_{t}$. Given this tuple $X_{t}$, let us define $g_{t}(\theta)=g(X_{t},\theta)$ as:

The TD(0) update to the current parameter $\theta_{t}$ with step-size $\alpha_{t}\in(0,1)$ can now be described succinctly as:

Under some mild technical conditions, it was shown in [29] that the iterates generated by TD(0) converge almost surely to the best linear approximator in the span of $\{\phi_{k}\}_{k\in[K]}$. In particular, $\theta_{t}\rightarrow\theta^{*}$, where $\theta^{*}$ is the unique solution of the projected Bellman equation $\Pi_{D}\mathcal{T}_{\mu}(\Phi\theta^{*})=\Phi\theta^{*}$. Here, $D$ is a diagonal matrix with entries given by the elements of the stationary distribution $\pi$ of the kernel $P$. Moreover, $\Pi_{D}(\cdot)$ is the projection operator onto the subspace spanned by $\{\phi_{k}\}_{k\in[K]}$ with respect to the inner product $\langle\cdot,\cdot\rangle_{D}$. The key question we explore in this paper is: What can be said of the convergence of TD(0) when the update direction $g_{t}(\theta_{t})$ in equation 2 is replaced by a distorted/compressed version of it? In the next section, we will make the notion of distortion precise, and outline the various technical challenges that make it non-trivial to answer the question that we posed above.

In this section, we propose a general framework for analyzing TD learning algorithms with distorted update directions. Extreme forms of such distortion could involve replacing each coordinate of the TD(0) direction with just its sign, or retaining just $k\in[K]$ of the largest magnitude coordinates and zeroing out the rest. In the sequel, we will refer to these variants of TD(0) as SignTD(0) and Topk-TD(0), respectively. Coupled with a memory mechanism known as error-feedback, it is known that analogous variants of SGD exhibit, almost remarkably, the same behavior as SGD itself [5]. Inspired by these findings, we ask: Does compressed TD(0) with error-feedback exhibit behavior similar to that of TD(0)?

Our motivation behind studying the above question is to build an understanding of: (i) communication-efficient versions of MARL algorithms where agents exchange compressed model-differentials, i.e., the gradient-like updates (see Section 6); and (ii) the robustness of TD learning algorithms to structured perturbations. It is important to reiterate here that our motivation mirrors that of studying perturbed versions of SGD in the context of optimization.

Description of EF-TD. In Algorithm 1, we propose the compressed TD(0) algorithm with error-feedback (EF-TD). Compared to equation 2, we note that the parameter $\theta_{t}$ is updated based on $h_{t}$ - a compressed version of the actual TD(0) direction $g_{t}(\theta_{t})$, where the compression is due to the operator $\mathcal{Q}_{\delta}:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$. The information lost up to time $t-1$ due to compression is accumulated in the memory variable $e_{t-1}$, and injected back into the current step as in equation 3. In line with compressors used for optimization, we consider a fairly general operator $\mathcal{Q}_{\delta}$ that is required to only satisfy the following contraction property for some $\delta\geq 1$:

A distortion perspective. For $\theta\neq 0$, it is easy to verify based on equation 4 that $\langle\mathcal{Q}_{\delta}(\theta),\theta\rangle\geq 1/(2\delta)\|\theta\|^{2}_{2}>0$, i.e., the angle between $\mathcal{Q}_{\delta}(\theta)$ and $\theta$ is acute. This provides an alternative view to the compression perspective: one can think of $\mathcal{Q}_{\delta}(\theta)$ as a tilted version of $\theta$, with a larger $\delta$ implying more tilt, and $\delta=1$ implying no distortion at all. See Figure 1 for a visual interpretation of this point.

The contraction property in equation 4 is satisfied by several popular quantization/sparsification schemes. These include the sign operator, and the Top-$k$ operator that selects $k$ coordinates with the largest magnitude, zeroing out the rest. Importantly, note that the operator $\mathcal{Q}_{\delta}$ does not necessarily yield an unbiased version of its argument. In optimization, error-feedback serves to counter the bias introduced by $\mathcal{Q}_{\delta}$. In fact, without error-feedback, algorithms like SignSGD can converge very slowly, or not converge at all [37]. In a similar spirit, we empirically observe in Fig. 2 (later in Section 8) that without error-feedback, SignTD(0) can end up making little to no progress towards $\theta^{*}$. However, understanding whether error-feedback guarantees convergence to $\theta^{*}$ is quite non-trivial. We now provide some intuition as to why this is the case.

Need for Technical Novelty. Error-feedback ensures that past (pseudo)-gradient information is injected with some delay. Thus, for optimization, error-feedback essentially leads to a delayed SGD algorithm. As long as the objective function is smooth, the gradient does not change much, and the delay-effect can be controlled. This intuition does not carry over to our setting since the TD(0) update direction is not a stochastic gradient of any fixed objective function.³³3As observed by [18] and [34], this can be seen from the fact that the derivative of the TD update direction produces a matrix that is not necessarily symmetric, unlike the symmetric Hessian matrix of a fixed objective function. Thus, controlling the effect of the compressor-induced delay requires new techniques for our setting. Moreover, unlike the SGD noise, the data tuples $\{X_{t}\}$ are part of the same Markov trajectory, introducing further complications. Finally, since the parameter updates of Algorithm 1 are intricately linked with the error variable $e_{t}$, to analyze their joint behavior, we need to perform a more refined Lyapunov drift analysis relative to the standard TD(0) analysis. Despite these challenges, in the sequel, we will establish that EF-TD retains almost the same convergence guarantees as vanilla TD(0).

The goal of this section is to provide a rigorous finite-time analysis of EF-TD under Markovian sampling. To that end, we need to introduce a few concepts and make certain standard assumptions. We start by assuming that all rewards are uniformly bounded by some $\bar{r}>0$, i.e., $|R(s)|\leq\bar{r},\forall s\in\mathcal{S}$. This ensures that the value functions exist. Next, we state a standard assumption that shows up in the finite-time analysis of iterative RL algorithms [18, 14, 16, 49, 10, 13].

The above assumption implies that the Markov chain induced by $\mu$ admits a unique stationary distribution $\pi$ [50]. Let $\Sigma=\Phi^{\top}D\Phi$. Since $\Phi$ is full column rank, $\Sigma$ is full rank with a strictly positive smallest eigenvalue $\omega<1$. To appreciate the implication of the above assumption, let us define the “steady-state” version of the TD update direction as follows: for a fixed $\theta\in\mathbb{R}^{K}$, let

We now introduce the notion of the mixing time $\tau_{\epsilon}$.

Assumption 1 implies that the total variation distance between the conditional distribution $\mathbb{P}\left(s_{t}=\cdot|s_{0}=s\right)$ and the stationary distribution $\pi$ decays geometrically fast for all $t\geq 0$, regardless of the initial state $s\in\mathcal{S}$ [50]. As a consequence of this geometric mixing of the Markov chain, it is not hard to show that $\tau_{\epsilon}$ in Definition 1 is $O\left(\log(1/\epsilon)\right)$; see, for instance, [16]. The precision that suffices for all results in this paper is $\epsilon=\alpha^{2}$, where recall that $\alpha$ is the step-size. Henceforth, we will simply use $\tau$ as a shorthand for $\tau_{\alpha}$. Finally, let us define $\sigma\triangleq\max\{1,\bar{r},\|\theta^{*}\|_{2}\}$ as the variance of our noise model, and $d_{t}\triangleq\|\theta_{t}-\theta^{*}\|_{2}$. We can now state the first main result of this paper.

The proof of Theorem 1 is provided in Appendix D. We now discuss the key implications of this result.

Discussion. Theorem 1 tells us that EF-TD guarantees linear convergence of the iterates to a ball around $\theta^{*}$, where the size of the ball scales with the variance $\sigma^{2}$; this exactly matches the behavior of vanilla TD [18, 14]. Since the step-size $\alpha$ scales inversely with the distortion factor $\delta$, we observe from equation 5 that the exponent of linear convergence gets slackened by $\delta$; once again, this is consistent with analogous results for SGD with (biased) compression [15]. The variance term, namely the second term in equation 5, has the exact same dependence on $\tau,\omega$, and $\gamma$ as one observes for vanilla TD [18, Theorem 3]. Observe that in this term, the effects of $\tau$ and $\delta$ in inflating the variance are additive. Moreover, even in the absence of compression, the dependence of the variance term on the mixing time $\tau$ is known to be unavoidable [51]. This immediately leads to the following interesting conclusion: when the underlying Markov chain induced by the policy mixes “slowly”, i.e., has a large mixing time $\tau$, one can afford to be more aggressive in terms of compression, i.e., use a larger $\delta$, since this would lead to a variance bound that is no worse than in the uncompressed setting. Said differently, Theorem 1 reveals that slowly-mixing Markov chains have a higher tolerance to distortions. This observation is novel since no prior work has studied the effect of distortions to RL algorithms under Markovian sampling. It is also interesting to note here that the phenomenon described above shows up in other contexts too: for instance, the authors in [36] showed that certain quantization mechanisms in optimization automatically come with privacy guarantees.

Theorem 1 is significant in that it is the first result to reveal that coupled with error-feedback, EF-TD is robust to extreme distortions. We will corroborate this phenomenon empirically as well in Section 8. The other key takeaway from Theorem 1 is that the scope of the error-feedback mechanism - now popularly used in distributed learning - extends to stochastic approximation problems well beyond just static optimization settings.

In Appendix B, we analyze a simpler steady-state version of EF-TD to help build intuition. There, we also provide an analysis of compressed TD learning algorithms that do not employ any error-feedback mechanism. Our analysis reveals that such schemes can also converge, provided the compression parameter $\delta$ satisfies a restrictive condition. Notably, as evident from the statement of Theorem 1, we do not need to impose any restrictions on $\delta$ for the convergence of EF-TD.

For the TD(0) algorithm, the update direction $g_{t}(\theta)$ is affine in the parameter $\theta$, i.e., $g_{t}(\theta)$ is of the form $A(X_{t})\theta-b(X_{t})$. As such, the recursion in equation 2 can be seen as an instance of linear stochastic approximation (SA), where the end goal is to use the data samples $\{X_{t}\}$ to find a $\theta$ that solves the linear equation $A\theta=b$; here, $A$ and $b$ are the steady-state versions of $A(X_{t})$ and $b(X_{t})$, respectively. The more involved TD($\lambda$) algorithms within the TD learning family also turn out to be instances of linear SA. Instead of deriving versions of our results for TD($\lambda$) algorithms in particular, we will instead consider a much more general nonlinear SA setting. Accordingly, we now study a variant of Algorithm 1 where $g(X_{t},\theta)$ is a general nonlinear map, and as before, ${X_{t}}$ comes from a finite-state Markov chain that is assumed to be aperiodic and irreducible. We assume that the nonlinear map satisfies the following regularity conditions.

In words, Assumption 2 says that $g(X,\theta)$ is globally uniformly (w.r.t. $X$) Lipschitz in the parameter $\theta$. Assumption 3 is a strong monotone property of the map $-\bar{g}(\theta)$ that guarantees that the iterates generated by the steady-state version of equation 2 converge exponentially fast to $\theta^{*}$. To provide some context, consider the TD(0) setting. Under feature normalization and the bounded rewards assumption, the global Lipschitz property is immediate. Moreover, Assumption 3 corresponds to negative-definiteness of the steady-state matrix $A=\mathbb{E}_{X_{t}\sim\pi}[A(X_{t})]$; this negative-definite property is also easy to verify [14]. For optimization, Assumptions 2 and 3 simply correspond to $L$-smoothness and $\beta$-strong-convexity of the loss function, respectively. For simplicity, we state the main result of this section for $L=2$; the analysis for a general $L$ follows identical arguments.

Discussion. We note that the guarantee in Theorem 2 mirrors that in Theorem 1, and represents our setting in its full generality, accounting for nonlinear SA, Markovian noise, compression, and error-feedback. Providing an explicit finite-time analysis for this setting is one of the main contributions of our paper. Now let us comment on the applications of this result. As noted earlier, our result applies to TD($\lambda$) algorithms and SGD under Markovian noise [52]; the effect of compression and error-feedback was previously unknown for both these settings. More importantly, certain instances of Q-learning with linear function approximation can also be captured via Assumptions 2 and 3 [16]. The key implication is that our analysis framework is not just limited to policy evaluation, but rather extends gracefully to decision-making (control) problems. This speaks to the significance of Theorem 2.

A key motivation of our work is to develop communication-efficient algorithms for MARL. This is particularly relevant for networked/federated versions of RL problems where communication imposes a major bottleneck [11]. To that end, we consider a collaborative MARL setting that has appeared in various recent works [11, 10, 17, 12, 13, 20]. The setup comprises of $M$ agents, all of whom interact with the same MDP, and can communicate via a central server. Every agent seeks to evaluate the same policy $\mu$. The purpose of collaboration is as in the standard FL setting: to achieve a $M$-fold reduction in sample-complexity by leveraging information from all agents. In particular, we ask: By exchanging compressed information via the server, is it possible to expedite the process of learning by achieving a linear speedup in the number of agents? While such questions have been extensively studied for supervised learning, we are unaware of any work that addresses them in the context of MARL. We further note that even without compression or error-feedback, establishing linear speedups under Markovian sampling is highly non-trivial, and the only other paper that does so is the very recent work of [13]. As we explain later in Section 7, our proof technique departs significantly from [13].

We propose and analyze a natural multi-agent version of EF-TD, outlined as Algorithm 2. In a nutshell, multi-agent EF-TD operates as follows. At each time-step, the server sends down a model $\theta_{t}$; each agent $i$ observes a local data sample, computes and uploads the compressed direction $h_{i,t}$, and updates its local error. The server then updates the model. Transmitting compressed TD(0) pseudo-gradients is consistent with both works in FL [53], and in MARL [11, 10, 13], where the agents essentially exchange model-differentials (i.e., the update directions), keeping their raw data private. It is worth noting here that while all agents play the same policy, the realizations of the data tuples $\{X_{i,t}\}$ may vary across agents. Let $\tilde{\theta}_{t}=\theta_{t}+\alpha\bar{e}_{t-1}$, where $\bar{e}_{t}=(1/M)\sum_{i\in[M]}e_{i,t}$, and define $\tilde{d}_{t}\triangleq\|\tilde{\theta}_{t}-\theta^{*}\|_{2}$. We can now state our main convergence result for Algorithm 2.

The proof of Theorem 3 is deferred to Appendix E. There are several key messages conveyed by Theorem 3. We discuss them below.

Message 1: Linear Speedup. Observe that the noise variance terms in equation 8 are $T_{2}$ and $T_{3}$, where $T_{3}$ is $O(\alpha^{2})$, i.e., a higher-order term in $\alpha$. Thus, for small $\alpha$, $T_{2}$ is the dominant noise term. Compared to the noise term for vanilla TD in [18, Theorem 3], $T_{2}$ in our bound has exactly the same dependence on $\tau$, $\omega$, and $\gamma$, and importantly, exhibits an inverse scaling w.r.t. $M$, implying a $M$-fold reduction in the variance $\sigma^{2}$ relative to the single agent setting. This is precisely what we wanted. For exact convergence, we can set $\alpha=O(\log(MT^{2})/T)$ to make $T_{1}$ and $T_{3}$ of order $\tilde{O}(1/T^{2})$, and the dominant term $T_{2}=\tilde{O}{(\sigma^{2}/(MT))}$. Thus, relative to the $O(\sigma^{2}/T)$ rate of TD(0), we achieve a linear speedup w.r.t. the number of agents $M$ in our dominant term $T_{2}$.

Message 2: Communication-efficiency comes (nearly) for free. The second key thing to note is that the dominant $T_{2}$ term is completely unaffected by the compression factor $\delta$. Indeed, the compression factor $\delta$ only affects higher-order terms that decay much faster than $T_{2}$. This means that we can significantly ramp up $\delta$, thereby communicating very little, while preserving the asymptotic rate of convergence of TD(0) and achieving optimal speedup in the number of agents, i.e., communication-efficiency comes almost for free. For instance, suppose $\mathcal{Q}_{\delta}(\cdot)$ is a top-$1$ operator, i.e., $h_{i,t}$ has only one non-zero entry which can be encoded using $\tilde{O}(1)$ bits. In this case, although $\delta=K$, where $K$ is the potentially large number of features, the dominant $\tilde{O}\left(\sigma^{2}/(MT)\right)$ term remains unaffected by $K$. Thus, in the context of MARL, our work is the first to show that asymptotically optimal rates can be achieved by transmitting just $\tilde{O}(1)$ bits per-agent per time-step.

Message 3: Tight Dependence on the Mixing Time. Compared to [13] - the only other paper in MARL that provides a linear speedup under Markovian sampling - our dominant term $T_{2}$ has a tighter $O(\tau)$ dependence on the mixing time $\tau$ as compared to the $O(\tau^{2})$ dependence in [13, Theorem 4.1]. It should be noted here that the $O(\tau)$ dependence is known to be information-theoretically optimal [51].

A few remarks are in order before we end this section.

We now discuss the novel steps in our analysis. Complete proof details are provided in the Appendix.

Proof Sketch for Theorem 1. Inspired by the perturbed iterate framework of [54], our first step is to define the perturbed iterate $\tilde{\theta}_{t}=\theta_{t}+\alpha e_{t-1}$. Simple calculations reveal: $\tilde{\theta}_{t+1}=\tilde{\theta}_{t}+\alpha g_{t}(\theta_{t})$. Notice that this recursion is almost the same as equation 2, other than the fact that the TD(0) update direction is evaluated at $\theta_{t}$ instead of $\tilde{\theta}_{t}$. Thus, to analyze the above recursion, we need to account for the gap between $\theta_{t}$ and $\tilde{\theta}_{t}$, the cause of which is the memory variable $e_{t-1}$. Accordingly, our next key step is to construct a novel Lyapunov function $\psi_{t}$ that captures the joint dynamics of $\tilde{\theta}_{t}$ and $e_{t-1}$:

As far as we are aware, a potential function of the above form has not been analyzed in prior RL work. Our goal now is to exploit the geometric mixing property in Assumption 1 to establish that $\psi_{t}$ decays over time (up to noise terms). This is precisely where the intricate coupling between the parameter $\tilde{\theta}_{t}$, the memory variable $e_{t}$, and the Markovian data tuples $\{X_{t}\}$ makes the analysis quite challenging. Let us elaborate. To exploit mixing-time arguments, we need to condition sufficiently into the past and bound drift terms of the form $\|\tilde{\theta}_{t}-\tilde{\theta}_{t-\tau}\|_{2}.$ This is where the coupling between $\tilde{\theta}_{t}$ and $e_{t}$ introduces non-standard delay terms, precluding the direct use of prior approaches in RL [18, 14, 16]. This difficulty does not show up in compressed optimization since one deals with i.i.d. data, precluding the need for mixing-time arguments. A workaround here is to employ a projection step (as in [18, 10]) to simplify the analysis. This is not satisfying for two reasons: (i) as we show in the Appendix, the simplification in the analysis comes at the cost of a sub-optimal dependence on the distortion $\delta$; and (ii) to project, one needs prior knowledge of the set containing $\theta^{*}$; this turns out to be unnecessary. Our key innovation here to bound the drift $\|\tilde{\theta}_{t}-\tilde{\theta}_{t-\tau}\|$ is the following technical lemma.

The above lemma relates the drift to damped “shocks” (delay terms) from the past, where (i) the amplitude of the shocks is captured by our Lyapunov function; (ii) the duration of these shocks is the mixing time $\tau$; and (iii) the damping arises from the fact that shocks are scaled by $\alpha^{2}$. In attempting to overcome one difficulty, we have created another for ourselves via the “max” term in equation 10. This is where we make a connection to the analysis of the IAG algorithm in [19] where a similar “max” term shows up. Via this connection, we are able to analyze the following final recursion:

Proof Sketch for Theorem 3. To establish the linear speedup property, we need a finer Lyapunov function (and much finer analysis) relative to Theorem 1. Accordingly, we construct:

and $C$ is a suitable design parameter. Using the techniques in [18], [14], and [16] to bound $\|g_{i,t}(\theta)\|^{2}$ unfortunately do not yield the desired linear speedup. Moreover, it is unclear whether the Generalized Moreau Envelope framework in [13] can be extended to analyze equation 12. As such, we depart from these works by establishing the following key result by carefully exploiting the geometric mixing property.

The above norm bound on the average TD direction turns out to be the main ingredient in bounding the drift terms for Algorithm 2, and eventually establishing a recursion of the form in equation 11 for $\Xi_{t}$.

To corroborate our theory, we construct a MDP with $100$ states, and use a feature matrix $\Phi$ with $K=10$ independent basis vectors. Using this MDP, we generate the state transition matrix $P_{\mu}$ and reward vector $R_{\mu}$ associated with a fixed policy $\mu$. We compare the performance of the vanilla uncompressed TD(0) algorithm with linear function approximation to SignTD(0) with and without error-feedback. We perform $30$ independent trials, and average the errors from each of these trials to report the mean-squared error $E_{t}=\|\theta_{T}-\theta^{*}\|^{2}_{2}$ for each of the algorithms mentioned above. The results of this experiment are reported in Fig. 2. We observe that in the absence of error-feedback, SignTD(0) can make little to no progress towards $\theta^{*}$. In contrast, the behavior of SignTD(0) with error-feedback almost exactly matches that of vanilla uncompressed TD(0). Our results align with similar observations made in the context of optimization, where SignSGD with error-feedback retains almost the same behavior as SGD with no compression [5, 37]. We provide some additional experiments below.

$\bullet$ Simulation of Topk-TD(0) with Error-Feedback. We simulate the behavior of EF-TD with the operator $\mathcal{Q}_{\delta}(\cdot)$ chosen to be a top-$k$ operator. We consider the same MDP as above, with the rewards for this experiment chosen from the interval $\begin{bmatrix}0&1\end{bmatrix}$. The size of the parameter vector $K$ is $50$, and the discount factor $\gamma$ is set to be $0.5$. We vary the level of distortion introduced by $\mathcal{Q}_{\delta}(\cdot)$ by changing the number of components $k$ transmitted. Note that $\delta=K/k$. As one might expect, increasing the distortion $\delta$ by transmitting fewer components impacts the exponential decay rate; this is clearly reflected in Fig. 3.

Simulation of Multi-Agent EF-TD. To simulate the behavior of multi-agent EF-TD (Algorithm 2), we consider the same MDP on 100 states as before with rewards in $[0\hskip 2.84526pt1]$. The dimension $K$ of the parameter vector is set to $10$ and the discount factor $\gamma$ to $0.3$. We consider two cases: one where $\mathcal{Q}_{\delta}(\cdot)$ is the sign operator, and another where it is a top-$2$ operator, i.e., only two components are transmitted by each agent at every time-step. We report our findings in Fig. 4, where we vary the number of agents $M$, and observe that for both the sign- and top-$k$ operator experiments, the residual mean-squared error goes down as we scale up $M$. Since the residual error is essentially an indicator of the noise variance, our plots display a clear effect of variance reduction by increasing the number of agents. Our observations thus align with Theorem 3.

We contributed to the development of a robustness theory for iterative RL algorithms subject to general compression schemes. In particular, we proposed and analyzed EF-TD - a compressed version of the classical TD(0) algorithm coupled with error-compensation. We then significantly generalized our analysis to nonlinear stochastic approximation and multi-agent settings. Concretely, our work conveys the following key messages: (i) compressed TD learning algorithms with error-feedback can be just as robust as their optimization counterparts; (ii) the popular error-feedback mechanism extends gracefully beyond the static optimization problems it has been explored for thus far; and (iii) linear convergence speedups in multi-agent TD learning can be achieved with very little communication. Our work opens up several research directions. Studying alternate quantization schemes for RL and exploring other RL algorithms (beyond TD and Q-learning) are immediate next steps. A more open-ended question is the following. SignSGD with momentum [7] is known to exhibit convergence behavior very similar to that of adaptive optimization algorithms such as ADAM [55], explaining their use in fast training of deep neural networks. In a similar spirit, can we make connections between SignTD and adaptive RL algorithms? This is an interesting question to explore since its resolution can potentially lead to faster RL algorithms.