Communication Efficient Parallel Reinforcement Learning

Reinforcement Learning (RL) is being increasingly deployed and trained with parallel agents. In many cases, each agent interacts with identical and independent environments. For example, in autonomous cars, the agents do not share the environment as they are located possibly far away [Kiran et al., 2020]. Similarly, in ride/freight sharing services, different RL agents (vehicles) make decisions in parallel to minimize their costs and maximize the profits [Al-Abbasi et al., 2019, Manchella et al., 2021]. Further, consider an example of an e-commerce company using RL agents in servers for recommending products to customer. Based on the location of the customer, each customer’s query may potentially be routed to a particular server (or agent) [Sankararaman et al., 2019]. Finally in the field of robotics, parallel robots are often deployed in practice [Hu et al., 2020, Sartoretti et al., 2019]. From these examples, we note that every agent would gain by sharing their data collected, otherwise they would be wasting the knowledge accumulated by the remaining agents. Such parallel policy learning with limited amount of communication is the focus of this paper.

Sharing of data across agents poses a new set of problems which comes from communicating the samples. If the agents communicate the observation tuple (state, action, reward, next state) at every time step, their communication complexity will increase. For various power constrained devices such as small robots, this luxury might not always be available [Sankararaman et al., 2019]. In this paper, we aim to work on the problem to reduce the number of communication steps while obtaining the same regret bounds as that of an strategy in which the agents always communicate.

For a system with $M$ parallel agents, the parallel agents generate $M$ times more data. From the lens of regret analysis, this setup loosely translates to a setup where a single agent works for time horizon of $MT$. For the setup where a single agent runs for $MT$ steps, the regret is lower bounded by $O(\sqrt{DSMAT}$ and upper bounded by $\tilde{O}(DS\sqrt{MAT})$ [Jaksch et al., 2010, Agrawal and Jia, 2017]. We show that, if the agents take their actions sequentially and communicate after every interaction with their respective environments, then the agents can collectively obtain a regret bound of $\tilde{O}(DS\sqrt{MAT})$. This results in a faster convergence rate of $O(1/\sqrt{MT})$ to the optimal policy as compared to the convergence rate of $O(1/\sqrt{T})$ when using one RL agent. Note that in practice, a sequential decision making for parallel independent agents cannot be guaranteed. Thus, this result acts as a lower bound to parallel reinforcement learning. In this paper, we further provide an algorithm, in which agents work in parallel, that achieves these bounds with limited communication.

We consider a setup where $M$ reinforcement learning agents interact with $M$ identical environments or Markov Decision Processes (MDPs) [Sutton and Barto, 2018, Puterman, 1994]. Similar to the examples considered above, the $M$ environments are independent. We assume that there also exists a central coordinator (server), although we suggest a method to relax this assumption. All the agents report their experiences to the central coordinator which then computes and shares a policy with the agents. We consider that the central coordinator uses a model based algorithm to compute the policy and each agent stores the number of visitations made to any particular (state, action, next state) tuple. The central coordinator also shares the total visitations to each state-action pair back to all the agents.

Based on this setup, we provide a novel communication efficient dist-UCRL algorithm. In dist-UCRL algorithm, the agents communicate with the central coordinator whenever any agent visits any state action pair $s,a$ in the current epoch (since last synchronization) a fraction ($1/M$) of the total visitation counts in $s,a$ till last synchronization, instead of every $N\geq 1$ time step. Using this synchronization strategy, for an MDP with diameter $D$, number of states $S$, and number of actions $A$, using dist-UCRL algorithm, we bound the cumulative regret of $M$ agents for $T$ time horizon scales as $O(DS\sqrt{MAT})$ using only $O(MAS\log_{2}(MT))$ synchronization steps¹¹1We use the term communication round and synchronization step interchangeably.. Note that dist-UCRL not only achieves the lower bound regret scaling ($\tilde{O}(\sqrt{MT})$) but also with limited communication.

To obtain our results, we also derive a concentration bound on $M$ independent Martingale sequences which can be of independent interest. To the best of our knowledge, this is the first work in the direction of obtaining the performance guarantees using regret analysis for a setup where $M$ agents interact and collaborate.

We also evaluate our algorithms empirically. We run the proposed dist-UCRL algorithm in multiple environments. We compare the dist-UCRL algorithm against the modified UCRL algorithm, following which the agents communicates after every time step. We show that dist-UCRL algorithm obtain similar regret bound with reduced communication.

The rest of the paper is organized as follows. Section II summarizes the key related works. Section III describes the complete system model. Section IV describes the dist-UCRL algorithm, and the regret guarantees of the algorithm are provided in Section V. Section VI describes a modification of UCRL2, mod-UCRL2, and its regret guarantees. Section VII tests the proposed algorithms in multiple environments, and Section VIII concludes the paper with some possible directions for future works.

Optimal planning using Markov Decision Processes has seen numerous significant contributions in history. Many algorithms have been proposed to find the optimal policies. The major algorithms include Q-learning, and policy iteration to find optimal policies [Howard, 1960, Puterman, 1994, Bertsekas, 1995, Sutton and Barto, 2018]. Fundamentally, the algorithms work by calculating the utility of taking certain actions in states that maximizes the utility of the state. These algorithms provide optimal policy when the transition probabilities of the Markov Decision Process is known or an $\epsilon$-optimal policy if using iterative algorithms [Puterman, 1994].

A large body of work is also done in the setup where the transition probabilities are not known. The model-based algorithms work by reducing the number samples required to obtain a close estimate of transition probabilities [Bartlett and Tewari, 2009, Jaksch et al., 2010]. The model-free algorithms work by directly estimating the utilities obtained by taking an action in a state [Jin et al., 2018]. These algorithms apply optimism in the face of uncertainty principle to find a model near the empirical estimates that provides the highest reward. There are also algorithms that sample the transition probabilities using posterior sampling and obtain regret bound [Ian et al., 2013, Agrawal and Jia, 2017]. The algorithms suggest that using parallel agents interacting independent and identical environments will provide tighter concentration inequalities and hence will help in reducing the regret.

In the domain of multiple agents using RL, most of the work is done where the agents interact with a dependent environment and the decision of one agent impacts all the other agents. This area is known as Multi-Agent Reinforcement Learning (MARL) [Gupta et al., 2017, Zhang et al., 2018]. The agents may cooperate with each other, for example, in the case of autonomous vehicles yielding on a busy road. Or the agents may also compete with each other, for example, in the case of car racing where only one car may win. Although, we do have multiple agents in our setup, the environments are independent. Hence, we will refrain from using MARL terminology in this paper.

With the introduction of deep learning to the field of reinforcement learning, many “Deep RL” algorithms have been provided to minimize the sample complexity to find the optimal policies [Mnih et al., 2013, Schulman et al., 2015, Mnih et al., 2016, Schulman et al., 2017, Haarnoja et al., 2018]. Recently, various algorithms are using parallel actors to learn better policies using deep reinforcement learning [Nair et al., 2015, Clemente et al., 2017, Horgan et al., 2018, Espeholt et al., 2018, Assran et al., 2019]. These algorithms consist of parallel agents that share the entire sequence of (state, action, reward, next state) tuples after every $n$ epochs, and update a common neural network with the gradients computed from possibly parallel learners. Compared to these works, we consider a model based setup to obtain regret guarantees with only $O(MAS\log_{2}(MT))$ number of synchronization rounds with a common controller learning the policy.

Recently, in the area of Bandits [Lattimore and Szepesvári, 2020], there has been a thrust on distributed bandits with reduced number of communication rounds. Various algorithms have been proposed to minimize the regret for setups where the agents synchronize a central coordinator [Kanade et al., 2012, Hillel et al., 2013, Wang et al., 2019, Dubey and Pentland, 2020]. Recently, Wang et al. [2019] showed that it is possible to obtain optimal regret guarantees with number of rounds that are independent of $T$ in stochastic Multi-Armed Bandits and Linear Bandits. Further, Dubey and Pentland [2020] considered a linear bandit setup and aimed to protect the privacy of the agents collaborating along with minimizing the number of communications rounds. Moreover, [Chawla et al., 2020, Sankararaman et al., 2019] considered a gossiping setup to communicate only the index of the best arm, thus reducing not only the number of communication rounds, but also the number of bits in each communication round. Similar to the bandit setup, we also attempt to find rigorous regret guarantees of the dist-UCRL algorithm and a bound on the number of communication rounds.

Let $[K]$ be the set of $K$ elements, or $[K]=\{1,2,\cdots,K\}$. We consider an MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},P,\bar{r})$, where $\mathcal{S}=[S]$ is the set of finite states and $\mathcal{A}=[A]$ is the set of finite actions. $P$ denotes the transition probabilities, i.e., on taking action $a\in\mathcal{A}$ in state $s\in\mathcal{S}$, the next state $s^{\prime}\in\mathcal{S}$ follows distribution $P(\cdot|s,a)$. Also, on taking action $a\in\mathcal{A}$ in state $s\in\mathcal{S}$, an agent receives a stochastic reward $r$ drawn from a distribution over $[0,1]$ with mean $\bar{r}(s,a)$.

We consider $M$ agents in the system, each interacting with $M$ identical environments. Let $i\in[M]$ be the indexing for agents and their corresponding environment. For an agent $i$, at time step $t$, let $s_{i,t}$ denote the state of the agent, $a_{i,t}$ be the action taken by the agent, and $r_{i,t}$ denote the reward obtained by the agent on taking action $a_{i,t}$ in state $s_{i,t}$. We assume that the $M$ environments are independent. Mathematically, $\forall\leavevmode\nobreak\ i\in[M]\text{ and }\forall\leavevmode\nobreak\ t\geq 1$, we have,

and we take a similar assumption over the rewards $r_{i,t}$ as well. This means, the distribution of the next state $s_{i,t+1}$ and the rewards $r_{i,t}$ of agent $i\in[M]$ are conditioned only on the knowledge of the current state and action $(s_{i,t},a_{i,t})$ of the agent $i$ and is independent of the other states and actions of the remaining agents.

Let policy $\pi:\mathcal{S}\to\mathcal{A}$ be a function to determine which action to select in state $s\in\mathcal{S}$. Note that each policy induces a Markov Chain on the states $\mathcal{S}$ with transition probabilities $P_{s,s}=P(\cdot|s,\pi(s))$. After defining a policy, we can now define the diameter of the MDP $\mathcal{M}$ as:

We assume that the MDP $\mathcal{M}$ has a finite diameter which means that there is a policy, such that following that policy all $s\in\mathcal{S}$ communicate with each other.

Any agent $i\in[M]$ starting from an initial random state $s_{i,1}=s$ follows an algorithm $\mathscr{A}$ till $T$ time steps to collect a cumulative reward of $R_{i}$. Also, let $\rho_{i}$ denote the average reward of the agent following algorithm $\mathscr{A}$. Or,

Let there be an algorithm $\mathscr{A}$ which always selects action according to a stationary policy $\pi:\mathcal{S}\to\mathcal{A}$. Then, we denote $\rho(\mathcal{M},\mathscr{A},s)=\rho(\mathcal{M},\pi,s)$. The optimal average reward does not depend on the state [Puterman, 1994, Section 8.3.3], and hence for the optimal policy $\pi^{*}$ which maximizes the average reward $\rho(\mathcal{M},\pi,s)$ we have,

Further, the optimal average reward satisfies [Puterman, 1994, Theorem 8.4.7],

where $a^{*}=\pi^{*}(s)$, and $v(s)$ is called the bias of the state $s$ and it denotes the extra reward obtained from starting in the state $s$. Note that $v(s)$ is not unique since if $v(s)\leavevmode\nobreak\ \forall\leavevmode\nobreak\ s$ satisfy Equation (5), then so does $v(s)+c$ for all $c\in\mathbb{R}$ and hence, the bias is translation invariant.

We aim to maximize the cumulative reward collected by all the agents. Hence, we want to develop an algorithm that, starting from no knowledge about the system, learns a policy that minimizes the regret. The regret of an algorithm $\mathscr{A}$, for starting state $s$, and running for time $T$, is defined as:

We will now present our algorithm dist-UCRL that uses upper confidence bounds to bound the regret $\Delta(\mathcal{M},\textsc{dist-UCRL},s,T)$ with high probability with only $O(MAS\log_{2}(MT))$ communication rounds.

We consider that each agent runs an instance of the dist-UCRL algorithm. The dist-UCRL algorithm running at agent $i$ is described in Algorithm 1. The algorithm proceeds in epochs indexed as $k=1,2,\cdots$. The start of every epoch also coincides with the synchronization step where every agent communicates with the central node to share data and update policies. This also implies that the number of synchronization rounds requires by the algorithm are the same as the number of epochs for the algorithm runs.

Algorithm 1 running at agent $i$, maintains two counters, $\nu_{i,k}(s,a)$ and $P_{i}(s,a,s^{\prime})$. $\nu_{i,k}(s,a)$ counts the number of visitations to state action pair $(s,a)$ in epoch $k$, and $P_{i}(s,a,s^{\prime})$ counts the instances when the agent moves to state $s^{\prime}$ on taking action $a$ in state $s$. The agent also stores $\hat{r}_{i}(s,a)$ as the cumulative reward obtained in $(s,a)$.

Let $N_{k}(s,a)=\sum_{i=1}^{M}\sum_{k^{\prime}=1}^{k-1}\nu_{i,k}(s,a)$ be the total number of visitations till the start of epoch $k$ for all agents. And hence, $N_{k}(s,a)=0$ at $k=1$ for all $s,a\in\mathcal{S}\times\mathcal{A}$. At the start of every epoch $k\geq 1$, the agents obtains the policy for epoch $k$ and total visitations to any state action pair $N_{k}(s,a)$. We will denote the time step at which epoch $k$ start and agents synchronize of the $k^{th}$ time with $t_{k}$. At $t=1$, the algorithm synchronizes all the agents for the very first time, or $t_{1}=1$. Later, a new epoch is triggered whenever any of the agent requests for synchronization²²2The synchronization request can be sent to server, which will in turn pause and synchronize the entire system.. An agent $i$ requests synchronization whenever $\nu_{i,k}(s,a)$ becomes at least $1/M$ of $N_{k}(s,a)$ for any state action pair. We assume that every agent is able to receive the synchronization signal instantly and stop further processing of the current epoch. The algorithm calls synchronize algorithm every time after a new epoch starts and updates the policy $\pi_{k}$ and $N_{k}(s,a)$ values. Every agent now selects actions according to the policy $\pi_{k}$ in the epoch $k$.

The synchronize algorithm is described in Algorithm 2. This algorithm calculates the estimates of the transition probability $\hat{p}(\cdot|s,a)$ and the mean rewards $\hat{\bar{r}}(s,a)$ using the samples from all the $M$ agents. We now consider a set of all plausible MDPs $\mathscr{M}(t)$ that exist in the neighborhood of the estimated MDP $\widehat{\mathcal{M}}=(\mathcal{S},\mathcal{A},\hat{p},\hat{\bar{r}})$. The mean rewards $r^{\prime}(s,a)$ and the transition probabilities $p^{\prime}(s,a)$ for all the MDPs in the set $\mathscr{M}(t)$ satisfies:

After obtaining $\mathscr{M}(t)$, Algorithm 2 calls the Extended Value Iteration algorithm which then computes the optimal policy for the optimistic MDP $\tilde{\mathcal{M}}_{t}$ in the set $\mathscr{M}(t)$. The optimistic MDP satisfies $\rho^{*}(\tilde{\mathcal{M}})=\sup_{\mathrm{M}\in\mathscr{M}(t)}\rho^{*}(\mathrm{M})$. As described in [Jaksch et al., 2010], it is not trivial to directly find the optimistic MDP in $\mathscr{M}(t)$. Hence, we consider an extended MDP $\mathcal{M}_{t}^{+}$ which is constructed with the same state space and a continuous action space $(a,q(\cdot|\cdot,a))\in\mathcal{A}\times\mathscr{P}_{t}$, where $\mathscr{P}_{t}$ is the set of transition probabilities for action $a\in\mathcal{A}$ that satisfies Equation (7). When $\mathscr{M}(t)$ contains the true MDP $\mathcal{M}$, the diameter of the extended MDP $\mathcal{M}_{t}^{+}$ is bounded by $D$ as the policy for which all states communicate with each other for MDP $\mathcal{M}$ also ensures that all states communicate in the extended MDP $\mathcal{M}_{t}^{+}$ as well.

The Extended Value Iteration algorithm (Algorithm 3) follows the design of the Extended Value Iteration of UCRL2 algorithm by Jaksch et al. [2010]. As described by Jaksch et al. [2010], Extended Value Iteration (EVI) obtains a policy $\pi$ that is $\epsilon$-optimal for the extended MDP $\mathcal{M}_{t}^{+}$ and in turn the optimistic MDP $\tilde{\mathcal{M}}$. Algorithm 3 calculates the values of the states of the extended MDP $\mathcal{M}_{t}^{+}$. The extended value iteration calculates the utilities of the states and the actions that achieve this utility. Note that unlike the extended value iteration in UCRL2 algorithm, we consider Algorithm 3 to be converged when we have $\max_{s}(u_{i+1}-u_{i}(s))-\min_{s}(u_{i+1}-u_{i}(s))\leq\epsilon=1/\sqrt{Mt}$. This is because we now have $M$ times more samples till any time step $t$ as compared to the UCRL algorithm. The EVI, at start of epoch $k$, returns a policy $\pi_{k}$ that satisfies $\rho(\tilde{\mathcal{M}},\pi_{k})\geq\rho^{*}(\tilde{\mathcal{M}}_{t_{k}})-1/\sqrt{Mt_{k}}$.

We also note that the central controller is not necessariliy required if the agents are in a completely connected network, they can share there data with each other and run the algorithms for the central controller by themselves. Further, the completely connected assumption can also be relaxed by considering a setup where all agents forward the messages they get. This allows the broadcast of the information. Hence, the proposed algorithm can be generalized to any network structure as long as all the agents are connected via some path.

After describing the algorithm, we now bound the regret of the dist-UCRL algorithm. We show that the regret bound holds with high probability. We bound the regret incurred by the dist-UCRL algorithm in the form of following theorem.

We let $m$ be the total number synchronizations done by agents running dist-UCRL algorithm till time $T$. Then, we bound $m$ deterministically in the following theorem.

We now state the lemmas required for the proof of the Theorem 1. The first three lemmas are used to handle the stochastic nature of the algorithm and environment. The first lemma provides concentration bounds on the $\ell_{1}$-deviation of the transition probabilities $\hat{p}(\cdot|s,a)$ for any $(s,a)$.

The next lemma provides concentration bounds on the sum of $M$ independent Martingale sequences for length $T$.

We now bound the growth rate of the total number of visitations to state action pair $(s,a)$, $\sum_{i=1}^{M}\nu_{i,k}(s,a)$, in any epoch $k$. If the total visitations are large, then the agents will incur large regret from a possibly sub-optimal policy. Hence, we have the following lemma:

The last lemma states that the span of the bias of the optimal policy defined as $\max_{s}v(s)-\min_{s}v(s)$ is bounded by the diameter $D$.

After stating all the necessary lemmas, we are now ready to prove the regret bound of the dist-UCRL algorithm or $\Delta(\mathcal{M},\textsc{dist-UCRL},s,T)$. We provide a detailed sketch here and provide the complete proof in Appendix C.

The proof of the Theorem 1 and Theorem 2 suggests that the analysis could potentially be extended to various other algorithms that follow the epoch completion condition from the UCRL-2 Algorithm of [Jaksch et al., 2010].

The proposed dist-UCRL algorithm does not require the agents to work sequentially, and hence the $M$ agents can truly work in parallel. For comparison of the proposed algorithm and completeness, we also consider an extension of UCRL2 algorithm by Jaksch et al. [2010] for $M$ parallel agents which we call mod-UCRL2. In the mod-UCRL2 algorithm, we assume that all the agents communicate to a centralized server at each time step $t$, and the server decides the actions for the agents at each time $t$. Thus, the communication rounds for mod-UCRL2 algorithm is $O(T)$, while the regret analysis is not apriori clear, and is the focus of this section. We also note that even though this algorithm is an easy extension of UCRL2, the analysis of regret is not straightforward, and uses the approach that was used to prove the regret guarantees of dist-UCRL because of the presence of $M$ agents.

At every time step $t$, every agent $i\in[M]$ observes its state $s_{i,t}$ and sends it to the server. The server after receiving all the states, process the requests sequentially in the order $s_{1,t},s_{2,t},\cdots,s_{M,t}$. The central server runs an instance of the UCRL2 algorithm with state sequence $s_{1,1},s_{2,1},\cdots,s_{M,1},s_{1,2},\cdots,$ and the corresponding action sequence $a_{1,1},a_{2,1},\cdots,a_{M,1},a_{1,2},\cdots$. The UCRL2 algorithm, running at the server, proceeds in epoch with epoch $1$ starting with $(i,t)=(1,1)$. We consider an epoch $k$ contains observations for $\{(\underline{i}_{k},\underline{t}_{k}),(\underline{i}_{k},\underline{t}_{k})+1,\cdots,(\bar{i}_{k},\bar{t}_{k})\}$ for some $\underline{i}_{k},\underline{t}_{k},\bar{i}_{k},\bar{t}_{k}$. The server maintains counters $\nu_{k}(s,a)$ denoting the number of times a state action pair $s,a$ is visited in the epoch $k$, and counter $N_{k}(s,a)$ denoting the number of times a state action pair $s,a$ is visited before the start of the epoch $k$ as:

The server updates an epoch $k$ whenever $\nu_{k}(s,a)=\max\{1,N_{k}(s,a)\}$ for some $s,a$. Following the UCRL2 algorithm, the server updates the policy at the beginning of every epoch using the observations collected till the beginning of the epoch. The complete algorithm is provided in Appendix E.

Note that in the mod-UCRL2 algorithm, the agents only behaves as interfaces to the $M$ independent environments, and is essentially not practical because of the sequential interface. In the following result, we formally state the result that the regret is upper bounded by $\tilde{O}(DS\sqrt{MAT})$.

In this section, we analyze the performance of the proposed dist-UCRL algorithm empirically. We test the dist-UCRL algorithm in multiple environments and also vary the number of agents in all the cases to study the regret growth with respect to $M$. Further, we also evaluate the average number of communication steps used by the dist-UCRL  algorithm till time step $T$.

We first run the dist-UCRL algorithm for the riverswim environment, which is a standard benchmark for model-based RL algorithm [Ian et al., 2013, Tossou et al., 2019] with $6$ states and $2$ actions. Next, we construct an extended riverswim environment with $12$ states and $2$ actions. Finally, we use a Grid-World environment [Sutton and Barto, 2018] for a $7\times 7$ grid which amounts to $20$ states and $4$ actions.

We compare the dist-UCRL algorithm against the mod-UCRL2 algorithm. We also compare with the standard UCRL2 algorithm for $M=1$. Note that for the case of $M=1$, both, dist-UCRL algorithm and the mod-UCRL2 algorithm reduces to the UCRL2 algorithm.

We run $50$ independent iterations of the algorithm. We plot the average per-agent regret, $\Delta(\mathcal{M},\textsc{dist-UCRL},s,T)/M$, over the $50$ iterations and the error bars for both dist-UCRL and mod-UCRL2 algorithm in Figure 1. We vary the number of agents $M$ as $1,4$, and $16$ to reduce clutter in figures. From Figure 1(a) and Figure 1(c), we note that the per-agent regret decreases approximately by a factor of $2$ for every $4$-fold increase in the number of agents. This is expected and was indeed the goal.

Note that in Figure 1(b), the per-agent regret falls from being linear in UCRL2 ($M=1$) to significantly sublinear as the number of agents were increased to $M=4$ and $M=16$. For the extended RiverSwim environment with $12$ states, the diameter is approximately $5\times 10^{4}$. Hence, the available time horizon of $T=10^{5}$ was not sufficient for UCRL2. However, distributed RL algorithms still observed a sub-linear regret by collating their knowledge. This further demonstrates the advantage of deploying multiple parallel agents in complex environments to enhance exploration.

We also empirically evaluate the number of communication rounds required by the dist-UCRL algorithm. Note that the number of communications for mod-UCRL2 algorithm is same as the total length for which the algorithm runs. Hence, we only plot the number of communications rounds required for the dist-UCRL algorithm. We vary the number of agents $M$ in multiples for $2$ and starting from $M=2$ agents. We again run the experiment for $50$ independent iterations and plot the average number of synchronization rounds till time step $t$.

We plot the number of communication rounds of the dist-UCRL algorithm in Figure 2. We observe that the number of the synchronization rounds increase very slowly with $t$. Further, as suggested from Theorem 2, the number of synchronization rounds are increasing in $M$. However, we note that for large values of $t$, the increase in the number of communication rounds is sub-linear. This is because for the bound on the number of communication rounds, we take pessimistic estimates on the increase on the visitation counts $N_{k}(s,a)$ where only one agent $i$ updates total $N_{k}(s,a)$ by a factor of $(1+1/M)$ only when the agent $i$ triggers the synchronization round for $s,a$. However, in practice, $N_{k}(s,a)$ grows faster than $(1+1/M)$, as the synchronization rounds triggered for other state action pairs and other agents also contribute towards $N_{k}(s,a)$.

In this work, we considered the problem of simultaneously reducing the cumulative regret and the number of communication rounds between $M$ agents. The $M$ agents interact with $M$ independent and identical Markov Decision Processes and share their data to learn the optimal policy faster. To this end, we proposed dist-UCRL, an epoch-based algorithm, following which the agents communicate in the beginning of every epoch. The data collected from the $M$ agents allows obtaining tighter deviation bounds and hence a smaller confidence set. Further, the agents trigger an epoch only after collecting sufficient samples in every epoch. This allows us to bound the regret of the dist-UCRL algorithm as $\tilde{O}(DS\sqrt{MAT})$ and the number of communication rounds as $O(MAS\log_{2}(MT))$. To analyze our algorithm, we also provide a concentration inequality for $M$ independent Martingale sequence of equal length which may be of an independent interest. We also evaluate the algorithm empirically and found that the average per-agent regret decreases as $O(1/\sqrt{M})$.

For comparison, we consider an extension of the UCRL2 algorithm for $M$ parallel agents working in round-robin sequence, and denote this algorithm as mod-UCRL2. We show that this algorithm also achieves the same regret bound as dist-UCRL algorithm, while with $O(T)$ communication rounds. The regret guarantee required the techniques developed for dist-UCRL. The evaluation results demonstrate the effectiveness of the proposed dist-UCRL algorithm in achieving the same regret bound as mod-UCRL2, while with lower communication. Thus, the proposed dist-UCRL algorithm can be utilized in training multiple parallel power starved devices using reinforcement learning.

Possible future works for the proposed works include analysing parallel RL and dist-UCRL algorithm with differential privacy, where agents may send data with additional noise to protect privacy.