DM²: Decentralized Multi-Agent Reinforcement Learning via Distribution Matching

Many algorithms for multi-agent cooperation tasks require some degree of information sharing between agents. Centralized training decentralized execution (CTDE) methods use a single centralized critic that aggregates information during training, but is no longer required at execution time (Lowe et al. 2017; Sunehag et al. 2018; Rashid et al. 2018; Foerster et al. 2018; Yu et al. 2022). In practical implementations, agent networks often share parameters during training as well.

Rather than sharing model components, methods may also explicitly communicate information between agents. Agents may be allowed to directly communicate information to each other (Jaques et al. 2019; Li and He 2020; Konan, Seraj, and Gombolay 2022). There might also be a central network that provides coordinating signals to all agents (He et al. 2020; Liu et al. 2021). Knowledge of other agents’ policies during training may also be assumed to limit the deviation of the joint policy (Wen et al. 2021).

This work studies the fully decentralized setting without communication or shared model components. To our knowledge, relatively few works consider this setting. Early work analyzed simple cases where two agents with similar but distinct goals could cooperate for mutual benefit under a rationality assumption (Rosenschein and Breese 1989; Genesereth, Ginsberg, and Rosenschein 1986). More recently, in the ALAN system for multi-agent navigation (Godoy et al. 2018), agents learn via a multi-armed bandits method that does not require any communication. Jiang and Lu (2021) study the decentralized multi-agent cooperation in the offline setting—in which each agent can only learn from its own data set of pre-collected behavior without communication—and propose a learning technique that relies on value and transition function error correction.

Ho and Ermon (2016) originally proposed adversarial distribution matching as a way to perform imitation learning in the single agent setting (the gail algorithm). Song et al. (2018) extend gail to the multi-agent setting in certain respects. Their analysis sets up independent imitation learning as searching for a Nash equilibrium, and assumes that a unique equilibrium exists. Their experiments focus on training the agent policies in the CTDE paradigm, rather than the fully distributed setting. This work instead leverages recent single-agent gail convergence theory (Guan, Xu, and Liang 2021) to demonstrate convergence to the joint expert policy, and performs experiments with distributed learning. Wang et al. (2021) study MARL using copula functions to explicitly model the dependence between marginal agent policies for multi-agent imitation learning. Durugkar, Liebman, and Stone (2020) and Radke, Larson, and Brecht (2022) show that balancing individual preferences (such as matching the state-action visitation distribution of some strategies) with the shared task reward can accelerate progress on the shared task. In contrast to these works, the goal of this paper is not to study imitation learning, but rather to study how distribution matching by independent agents can enhance performance in cooperative tasks.

Perhaps most closely related to this work, Hao et al. (2019) use self-imitation learning (sil) (Oh et al. 2018) to encourage agents to repeat actions that led to high returns in the past. The above approach can be considered as a special case of the setting this paper studies, where the target distribution can be non-stationary, and is generated by the agents themselves. This paper further presents a theoretical analysis, showing that in the case where the target distribution is generated by demonstrations (and is therefore stationary), each agent attempting to minimize mismatch to their individual target distributions leads to convergence to the joint target policy. Due to the non-stationary nature of the target distribution in sil, similar guarantees cannot be obtained.

A Markov game (Littman 1994) or a stochastic game (Gardner and Owen 1983) with $K$ agents is defined as a tuple $\langle K,\mathcal{S},\bm{\mathcal{A}},\rho_{0},\mathcal{T},\bm{R},\gamma\rangle$, where $\mathcal{S}$ is the set of states, and $\bm{\mathcal{A}}\equiv\mathcal{A}^{K}$ is the product of the set of actions $\mathcal{A}$ available to each agent. The initial state distribution is described by $\rho_{0}:\mathcal{S}\mapsto\Delta(\mathcal{S})$, where $\Delta(\cdot)$ indicates a distribution over the corresponding set. The transitions between states are controlled by the transition distribution $\mathcal{T}:\mathcal{S}\times\mathcal{A}_{0}\times\mathcal{A}_{1}\times\ldots\times\mathcal{A}_{K-1}\longmapsto\Delta(\mathcal{S})$. Each agent $i$ acts according to a parameterized policy $\pi_{\theta_{i}}:\mathcal{S}\mapsto\Delta(\mathcal{A}_{i})$, and the joint policy $\bm{\pi}_{\theta}=[\pi_{\theta_{1}},\cdots,\pi_{\theta_{K}}]$ is the vector of the individual agent policies. Occasionally, the policy parameters $\theta$ are omitted for convenience. Note that each agent observes the full state. We use subscript ${i-}$ to refer to all agents except $i$, i.e., $\pi_{i-}$ refers to the agent policies, $\{\pi_{0},\ldots,\pi_{i-1},\pi_{i+1},\ldots,\pi_{K-1}\}$.

Each agent $i$ is also associated with a reward function $R_{i}:\mathcal{S}\times\mathcal{A}_{0}\times\ldots\times\mathcal{A}_{K-1}\longmapsto\mathbb{R}$. The agent aims to maximize its expected return $\mathbb{E}_{\bm{\pi}}[\sum_{t=0}^{\infty}\gamma^{t}r_{i,t}]$, where $r_{i,t}$ is the reward received by agent $i$ at time step $t$, and the discount factor $\gamma\in[0,1)$ specifies how much to discount future rewards. In the cooperative tasks considered by this paper, the task rewards are identical across agents.

In Markov games, the optimal policy of an agent depends on the policies of the other agents. The best response policy is the best policy an agent can adopt, given the other agent’s policies $\pi_{i}^{*}=\operatorname*{\arg\!\max}_{\pi_{i}}\mathbb{E}_{\pi_{i},\pi_{i-}}[\sum_{t=0}^{\infty}\gamma^{t}r_{i,t}]$. If no agent can unilaterally change its policy without reducing its return, then the policies are considered to be in a Nash equilibrium. That is, $\forall i\in[0,K-1],\forall\hat{\pi}_{i}\neq\pi_{i},\mathbb{E}_{\pi_{i},\pi_{i-}}[\sum_{t=0}^{\infty}\gamma^{t}r_{i,t}]\geq\mathbb{E}_{\hat{\pi}_{i},\pi_{i-}}[\sum_{t=0}^{\infty}\gamma^{t}r_{i,t}]$.

The theoretical analysis in Section 4 deals with the above fully observable setting, and assumes a discrete and finite state and action space. However, the experiments are conducted in partially observable MDPs (POMDPs) with continuous states, which can be formalized as Dec-POMDPs in the multi-agent setting (Oliehoek 2012). Dec-POMDPs include two additional elements: the set of observations $\Omega$ and each agent’s observation function $O_{i}:\mathcal{S}\longmapsto\Delta(\Omega)$.

Imitation learning (Bakker and Kuniyoshi 1996; Ross, Gordon, and Bagnell 2011; Schaal 1997) is a problem setting where an agent tries to mimic trajectories $\{\xi_{0},\xi_{1},\ldots\}$ where each trajectory $\xi=\{(s_{0},a_{0}),(s_{1},a_{1}),\ldots\}$ is demonstrated by an expert policy $\pi_{E}$. Various methods have been proposed to address the imitation learning problem. Behavioral cloning (Bain and Sammut 1995) applies supervised learning to expert demonstrations to recover the maximum likelihood policy. Inverse reinforcement learning (IRL) (Ng, Russell et al. 2000) recovers a reward function which can then be used to learn the expert policy using reinforcement learning. To do so, $\mathtt{IRL}(\pi_{E})$ aims to recover a reward function under which the trajectories demonstrated by $\pi_{E}$ are optimal.

Ho and Ermon (2016) formulate imitation learning as a distribution matching problem and propose the gail algorithm. Let the state-action visitation distribution of a joint policy $\bm{\pi}=\langle\pi_{1},\ldots\pi_{K}\rangle$ be:

In a multi-agent setting, for agent $i$,

refers to the marginal state-action visitation distribution of agent $i$’s policy $\pi_{i}$, given the other agents’ policies $\pi_{i-}$. In the single agent setting, a policy that minimizes the mismatch of its state-action visitation distribution to the one induced by the expert’s trajectories and maximizes its causal entropy $H(\pi)$ is a solution to the $\mathtt{RL}\circ\mathtt{IRL}(\pi_{E})$ problem (Ho and Ermon 2016). That is, distribution matching is a solution to the imitation learning problem.

Guan, Xu, and Liang (2021) showed that in the single-agent case, the gail algorithm converges to the expert policy under a variety of policy gradient techniques, including TRPO (Schulman et al. 2015). Let $r_{\phi}$ be a reward function (based on a discriminator) parameterized by $\phi$, and let $\psi(\phi)$ be a convex regularizer. Guan, Xu, and Liang (2021) formulate the gail problem as the following min-max problem:

where $V(\pi,r)=\operatorname*{\mathbb{E}}_{s_{0}\sim\rho_{0}}\operatorname*{\mathbb{E}}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{i,t}]$ is the expected return from some start state when following policy $\pi$ and using reward function $r$.

In the multi-agent setting, imitation learning has the added complexity that the expert trajectories are generated by the interaction of multiple expert policies $\langle\pi_{E_{0}},\ldots,\pi_{E_{K}}\rangle$. Successful imitation in this setting thus involves the coordination of all $K$ agents’ policies.

Experiments were conducted on the StarCraft Multi-Agent Challenge domain (Samvelyan et al. 2019). It features cooperative tasks where a team of controllable allied agents must defeat a team of enemy agents. The enemy agents are controlled by a fixed AI. The battle is won and the episode terminates if the allies can defeat all enemy agents. The allies each receive a team reward every time an enemy agent is killed, and when the battle is won. StarCraft is a partially observable domain, where an allied agent can observe features about itself, as well as allies and enemies within a fixed radius. The specific StarCraft tasks used here (with two additional tasks in Appendix B) are:

• •

  5v6: 5 Marines (allies) and 6 Marines (enemies)

• •

  3sv4z: 3 Stalkers (allies) and 4 Zealots (enemies)

• •

  3sv3z: 3 Stalkers (allies) and 3 Zealots (enemies)

dm² is compared against a naive decentralized MARL algorithm, independent ppo (Schulman et al. 2017) (ippo), where individual ppo agents directly receive the team environment reward. Although agents trained under the ippo scheme cannot share information and see only local observations, prior work has shown that ippo can be surprisingly competitive with CTDE methods (Yu et al. 2022). We also compare against two widely used CTDE methods, qmix (Rashid et al. 2018) and rmappo (Yu et al. 2022). These CTDE methods have the advantage of a shared critic network that receives the global state during training. Thus, their performance is expected to be better than that of decentralized methods with no communication.

dm² uses the same ippo implementation as the baseline, with the addition of a gail discriminator for each independent agent $i$ to generate an imitation reward signal, $r_{i,\textsc{gail}}$. The scaled gail reward is added to the environment reward $r_{env}$, with scaling coefficient $c\in\mathbb{R}$: $r_{i,mix}=r_{env}+r_{i,\textsc{gail}}*c$. Learning curves of all algorithms are the mean of 5 runs executed with independent random seeds, where each run is evaluated for 32 test episodes at regular intervals during training. The shaded regions on the plots show the standard error. The evaluation metric is the mean rate of battles won against enemy teams during test episodes.

The data for the gail discriminator consists of 1000 joint state-only trajectories (no actions). The data is sampled from checkpoints during training runs of baseline ippo with the environment reward, and qmix with the environment reward. In runs of dm², each agent imitates the marginal observations of the corresponding agent from the dataset (i.e., agent $i$ will imitate agent $i$’s observations from the dataset) ⁴⁴4The allied agent teams in our experiments have the same state/action spaces. Thus, the mapping of agents to demonstration trajectories does not matter, as long as it is fixed. For each task, demonstrations are sampled from ippo and qmix-trained joint expert policies, executed stochastically for ippo and with an $\epsilon$-greedy sampling for qmix. The win rates achieved by the demonstration policies are plotted as horizontal lines on the graphs. Additionally, sil(Hao et al. 2019) can be seen as a variation of dm², with the target demonstrations being the agent’s most successful prior trajectories. Experimental details such as hyperparameters are specified in Appendix C.