Asynchronous Cooperative Multi-Agent Reinforcement Learning with Limited Communication

Through communication, agents can obtain a better understanding of their own environment and other agents’ behaviors, thus improving their ability to coordinate. For a comprehensive survey on research on the impact of communication in multi-agent collaboration, we refer the reader to [6]. Seminal works such as CommNet [7] used a shared neural network to process local observations for each agent. Each agent makes decisions based on its observations and a mean vector of messages from other agents. Although agents can operate independently with copies of the shared network, instantaneous communication with all agents is necessary. Subsequent works [8, 9] aimed to investigate methodologies that improved coordination by using a small subset of agents. In ATOC [9], the authors used a probabilistic gate mechanism to communicate with nearby agents in an observable field and a bi-LSTM to concatenate messages together. Further improvement to multi-agent collaboration was made by learning encodings of local information via attention mechanisms. TarMAC [10] uses an attention mechanism to generate encodings of the content of messages so that nearby agents can learn the message’s significance via a neural network implicitly. While attention-based mechanisms can help agents improve their abilities to utilize individual pieces of communication, attention-based approaches neglect larger inter-agent relationships. Graph-based methods like DGN [11], DICG [12], InforMARL [13], MAGIC [14], and EMP [15] formulate interactions between different agents as a graph. These methods use graph structures to learn relational representations from nearby nodes. However, a similar communication problem to CommNet arises with these methodologies, as they often require fully connected graphs during the learning process. For example, in EMP, all agents must know the position of all other entities in the graph at the beginning of an episode. This assumption is a key limitation, as it requires that all agents will be able to coordinate synchronously.

Event-triggered control is a control strategy in which the system updates its control actions only when certain events or conditions occur, rather than continuously or at regular time intervals [16]. By reducing the number of control updates, event-triggered control can significantly decrease the workload on controllers, a beneficial attribute for systems with limited resources. ETC [17], VBC [18], and MBC [19] have proposed event-triggered control methodologies to reduce the communication frequency and address communication constraints. These works focus on optimizing when transmission can occur rather than optimizing how to achieve better performance with less communication. Menda et al. [20] frame the asynchronous decision-making problem where agents choose actions when prompted by an event or some set of events occurring in the environment. By relying on a continuous state-space representation and an event-driven simulator, the agents step from event to event, and lower-level timestep simulations are eliminated. This improves the scalability of the algorithm but does not capture low-level agent-agent interactions during an event.

An alternative approach to asynchronous environments formulates the problem as a Macro-Action Decentralized Partially Observable Markov Decision Process (MacDec-POMDP) [21], where agents can start and end higher level ’macro’ actions at different time steps. Xiao et al. [22] propose a methodology for multi-agent policy gradients that allow agents to asynchronously learn high-level policies over a set of pre-defined macro actions. The method relies on an independent actor independent centralized critic training framework, named IAICC, to train each agent. Hong et al. [23] alter the IAICC framework to encode agent time history independently to avoid duplicate macro-observations in the centralized critic. These works focus on learning algorithms for asynchronous macro-actions that occur across multiple time steps. By contrast, our work focuses on the micro level, looking for planning opportunities while agents achieve the same macro-level task.

A decentralized partially observable Markov decision process (DEC-POMDP) is a multi-agent extension of a Markov decision process (MDP) where multiple players (agents) interact in a shared environment, but each agent has only limited or partial information about the current state of the environment and the state of other agents. A DEC-POMDP for $\mathcal{N}$ agents can be defined by a set of global states $\mathcal{S}$, a set of private observations for each agent $o^{(1)},o^{(2)},..,o^{(n)}$, a set of actions for each agent,$a^{(1)},a^{(2)},..,a^{(n)}$, and the transition function $T:\mathcal{S}\times a^{(1)}\times..\times a^{(N)}\rightarrow\mathcal{S}$. Agent $i$ chooses actions $a^{(i)}\in\mathcal{A}^{(i)}$, and then obtains a reward as a function of its state-action pairing: $r^{(i)}:\mathcal{S}\times a^{(i)}\rightarrow\mathbb{R}$. It also receives a local observation $o^{(i)}$. Agent $i$ aims to maximize its reward $R^{(i)}=\sum_{t=0}^{T}r^{(i)}$.

The policy gradient method is used in RL tasks to perform gradient ascent on the agent policy parameters, $\theta$, to optimize the total discounted reward, $J(\theta)=\mathbb{E}_{s\sim\rho^{\pi},a\sim\pi_{\theta}}[R]$. Here, $\rho^{\pi}$ is the state distribution, $\pi_{\theta}$ is the policy distribution and $R$ represents the reward at that time $t$, $R_{t}=\sum_{t^{\prime}=t}^{T}r(s_{t^{\prime}},a_{t^{\prime}})$.

In lieu of $R_{t}$, we use the advantage function $A^{\pi}(s_{t},a_{t})=Q^{\pi}(s_{t},a_{t})-V^{\pi}(s_{t})$ to decrease the variance of the estimated policy gradient, where $V^{\pi}(s_{t})$ is the value function.

We adopt a centralized training decentralized execution learning paradigm [24, 25]. In execution, agents learned policies are conditioned only on their own action-observation history. In training, the critic has access to the global state $\mathcal{S}$ of the environment, which is used to train the model end to end.

In traditional multi-agent reinforcement learning, agents always take action steps synchronously without considering the communication constraints associated with their action selection. For instance, when exploring unknown environments, a single rover may navigate to an area where they are unable to communicate with others, or the duration of transmitting a single message may take several time steps. To model these communication-action costs, we define a new time scale $\tau$ to account for the specific actions each agent has taken at different time steps. As shown in Figure 1, $\tau_{1}^{(i)}$ represents the time of the first action that agent $i$ has taken. In Figure 1 panel (b), all three agents take their first action at the same $t$, resulting in the same time reference point for $\tau_{1}$. However, the next time $t$ that each agent takes their next action is different (agent 1 $\tau_{2}^{(1)}=t_{7}$, agent 2 $\tau_{2}^{(2)}=t_{5}$ agent 3 $\tau_{2}^{(3)}=t_{9}$). Rather than incorporating all time steps into the replay buffer, we only include those steps where the agents are taking an action. As a result, the replay buffer of each agent’s trajectory is based on their $\tau$ sequence of actions instead of $t$. To improve the generalizability of our algorithm to different periods of time between actions $\tau_{1}$ and $\tau_{2}$, during training, we randomly generate the period between different actions. We refer to this randomized parameter as $\mu$, and it determines when an agent will take a subsequent action. $\mu$ is chosen from a uniform distribution, with different intervals used for training and testing to ensure diversity. The $\mu$ range is kept relatively small to model asynchronous behavior that maintains staggered interactions between agents without diverging to entirely independent timescales. For each initiation, the delay is selected randomly from this distribution and remains constant throughout the episode rather than changing after specific actions. This decision avoids creating an artificial link between certain movement actions and specific delays, as we consider such realism beyond the scope of this work and more applicable to domain-specific problems.

Graph transformers are a specialized type of transformer model designed to handle graph-structured data. In addition to the traditional transformer’s ability to capture long-range dependencies, graph transformers enable the model to understand both local node interactions via graph edges and global context via self-attention.

As an input into our graph transformer, we form an agent-entity graph [15], where every object in the environment is assumed to be either an agent, an obstacle, or a goal. All objects within the environment are transformed to be nodes on the graph, with node features $x_{j}=[p^{j}_{i},v^{j}_{i},p^{\mathrm{goal},j}_{i},\texttt{entity\_type(j)}]$ where $p^{j}_{i},v^{j}_{i},p^{\mathrm{goal},j}_{i}$ are the relative position, velocity, and position of the goal of the object at node $j$ with respect to agent $i$, respectively. If node $j$ corresponds to a (static/dynamic) obstacle or a goal, we set $p^{\mathrm{goal},j}_{i}\equiv p^{j}_{i}$. To process the entity_type categorical variable, we use an embedding layer.

Our graph formulation is dynamic, meaning that edges are formed depending on an entity’s proximity to other objects in the graph and their corresponding communication interval, as shown by $(b)$ in Figure 1. We define an edge to exist $e\in\mathcal{E}$ between an agent and another agent if they are within a ‘communication radius’ $\lambda$ of each other and if they are taking an action at the same time step $t$. Edges are formed between agents and obstacles or landmarks when they are merely within the agent’s communication radius. As landmarks and obstacles do not have decision-making abilities, their presence can be sensed at every time step the agent is in proximity. When an edge is formed, $e_{ij}$, it has an associated edge feature given by the Euclidean distance between the entities involved, $i$ and $j$.

For a complete representation of our graph structure, we create an adjacency matrix to represent the connectivity between different nodes on a graph. An adjacency matrix $\mathbb{A}$ is a square matrix used to describe the connections between nodes on a graph, where each element $\mathbb{A}_{i,j}$ represents the presence or absence of an edge between node $i$ and node $j$. Our graph is directed and weighted, so the value in the adjacency matrix for $\mathbb{A}_{i,j}$ is proportionate to the weight of the edge between nodes $i$ and $j$. To account for the fact that our agents can take actions at different time steps, we first introduce the variable $d_{i}$ to represent the current status of the agent, where $d_{i}=1$ if the agent is active and $d_{i}=0$ if the agent is inactive. We create an additional matrix to reflect the activity status of all nodes in the graph $\mathcal{D}\in\mathbb{R}^{N\times N}$ where $\mathcal{D}[i,j]=d_{i}\cdot d_{j}$, where $d_{i}$ and $d_{j}$ reflect whether node $i$ and node $j$ are active. This is used to ensure that interactions can only occur if both nodes are active. The masked adjacency matrix $\mathbb{A}^{\textrm{masked}}$ is given by the element-wise product $\mathbb{A}\circ\mathcal{D}$. The resulting masked adjacency matrix dynamically updates as agents become inactive, allowing the algorithm to focus on interactions between remaining agents. Similarly, a death mask enforces that no interactions or learning updates are computed for agents whose tasks are already completed.

We rely on a graph transformer to encode messages and characterize relationships between different entities in the environment. Our transformer takes the node features $x_{k}$ and edge features $e_{k}$ as input. The graph transformer is based on a Unified Message Passing Model (UniMP) [26] that relies on multi-head dot product attention to selectively prioritize incoming messages from their neighbors based on their relevance. We rely on two layers of this model, with each layer update defined as

where $x_{k}$ are the node features in the graph, $\mathbb{N}(i)$ is the set of nodes which are connected to node $i$, $W_{k}$ are learnable weight matrices and the attention coefficients. $\alpha_{i,j}$ is computed via multi-head dot product attention

At every time step, each agent gets a distance-based reward to an assigned goal, ${\mathcal{R}}_{\mathrm{dist}}(s_{\tau},a^{(i)}_{\tau})$. When an individual agent, $i$, reaches their assigned goal (indicated by $\rho$), it receives a goal-reaching reward $\mathcal{R}_{\mathrm{goal}}(s_{\tau},a^{(i)}_{\tau})$. $\rho$ is 1 if the agent reached the assigned goal and was previously not at the assigned goal; otherwise, 0. We also penalize agents colliding with other agents or obstacles in the environment using a collision penalty $-C$. $\kappa$ is a 0/1 variable that indicates if an agent collided with another agent or an obstacle. The reward for agent $i$ at time step $t$ then becomes

Rewards are shared between all agents active at step $t$. The purpose of sharing rewards between active agents is to encourage synchronous collaboration when possible.

For our algorithm, the traditional DEC-POMDP setup is augmented by the existence of the graph network, transforming the tuple that describes the problem to: $\langle\mathcal{N},\mathcal{S},\mathcal{O},\mathcal{A},R,P,\gamma,\mathcal{G}\rangle$. $g^{(i)}=\mathcal{G}(s;i)$ represents the graph network formed by the entities of the environment with respect to agent $i$. Features of the graph structure are used in the policy gradient learning architecture.

Our learning architecture relies on a centralized critic, which uses the state-action pairs and information from the graph formulation. As shown in Figure 1, the critic receives the full graph embeddings via a global mean pooling operator ($X_{\mathrm{agg}}=\frac{1}{N}\sum\limits_{i=1}^{N}x^{(i)}_{\tau,\mathrm{agg}}$) so that it has a global view of the graph representation. The critic updates parameter $\phi$ to approximate the value function.

We then create an individual actor for each agent. Each agent relies on policy $\pi_{\theta_{i}}^{(i)}(a_{\tau}^{(i)}|o_{\tau}^{(i)},g_{\tau}^{(i)})$, parameterized by $\theta_{i}$ to determine its action $a^{(i)}$ from its local observation $o^{(i)}$ and its local graph network $g^{(i)}$. The resultant policy gradient is:

Algorithm 1 shows the complete training setup for AsynCoMARL. As mentioned in Section 4.1, each agent $i$ has its own interval $\mu$, which determines when it will take a subsequent action. The interval is reset for each environment scenario to avoid overfitting. During a training episode, an agent’s status is marked as active if there have been $\mu$ steps since its last action. If an agent reaches its goal before the end of the episode, its status $d_{i}$ is changed to 0, thus masking its presence in the masked adjacency matrix $A^{\textrm{masked}}$ passed into the centralized critic function.

We rely on the update procedure associated with the MAPPO algorithm [27]. In this algorithm, the advantage estimate is computed using Generalized Advantage Estimation (GAE) method [28], with advantage normalization and value clipping over the complete trajectory. The value function is clipped to avoid large updates, ensuring robust learning. A random mini-batch is sampled from the data buffer, and for each data chunk in the mini-batch, the RNN hidden states for both the policy and the critic are initialized from the first hidden state in the data chunk. The policy and critic parameters are then updated using the Adam optimizer [29], based on the computed loss, which incorporates the policy objective, value loss, and entropy regularization for exploration.

We conduct experiments in two environments, Cooperative Navigation and Rover-Tower, specifically chosen to emulate real-world scenarios with communication constraints. These two environments replicate communications encountered in space missions and planetary rover exploration.

The Cooperative Navigation environment was chosen because it simulates the space environment, a setting where communications are constrained and sporadic but where efficient coordination is important. We use the Cooperative Navigation environment to evaluate the amount of communication used in each approach. In particular, we aim to study how frequently our approach requires communications between agents compared to other baselines.

The Rover-Tower environment was chosen as it simulates a real-world planetary exploration scenario involving a rover and a more capable observing agent. We use the Rover-Tower environment to assess each method’s ability to adapt to more complex communication scenarios. In particular, we aim to study how generalizable each method is to different communication scenarios involving agents with different observation and communication abilities.

Both the Cooperative Navigation and Rover-Tower environments are implemented in the multi-agent particle environment (MPE) framework introduced by [30]. MPE involves a set of agents that have a discrete action space where it can apply a control unit acceleration in the $x-$ and $y-$ directions. We evaluate our proposed model in the following two environments:

Cooperative Navigation: Dolan et al. [4] propose a modified MPE environment that uses the satellite dynamics following the Chlohessy-Wiltshire model [31], where a set of $n$ satellites are moving within a 2D plane. Each satellite is tasked with rendezvousing to an assigned goal location by the end of the episode. The Chlohessy-Wiltshire equations are a set of second-order differential equations whose $x$ and $y$ components are coupled, thereby increasing the difficulty of the simulation dynamics. We use this modified environment to test our experiment due to its relevancy to the problems we are interested in (e.g., environments with limited communications) and because the agents involved will continue along their relative trajectories even when not communicating with one another. In the Cooperative Navigation task, satellites are initialized into an environment defined by the world size (e.g., 2 km). We set the initialization such that each satellite is at least 500 meters away from all other satellites and their intended goals. This constraint ensures that satellites have sufficient space to maneuver, preventing the occurrence of deadlock. For each episode, each satellite is reinitialized at a different location with a new goal, ensuring a variety of maneuvers and enhancing the generalizability to different navigation interactions.
Rover-Tower: Iqbal and Sha [5] develop the Rover-Tower task environment where randomly paired agents communicate information and coordinate. The environment consists of 4 "rovers" and 4 "towers", with each episode pairing rovers and towers randomly. The performance of the pair is penalized on the basis of the distance of the rover to its goal. The task is designed to simulate scientific navigation on another planet where there is limited infrastructure and low visibility. Rovers cannot observe their surroundings and must rely on communication from the towers, which can locate rovers and their destination and can send one of 5 discrete communications to their paired rover. In this scenario, communication is highly restricted. Extended descriptions of both environments are included in the appendix.

We rely on several evaluation metrics to characterize the performance of our algorithm against other baselines. We selected these metrics to assess the agent’s ability to successfully navigate to their goals without collisions.

• •

  Communication frequency $(f_{\mathrm{comm}})$: Average ratio of the number of messages passed between agents over the maximum number of communication opportunities in the episode. $f_{\mathrm{comm}}$ of 1 indicates each agent messaged every other agent at every time step in the episode (lower value indicates the model is more message efficient).

• •

  Success Rate $(S\%)$: Percentage of episodes in which all $n$ agents are able to get to their goals (higher is better).

• •

  Fraction of Episode Completed $(T)$: The fraction of an episode that agents take on average to get to the goal. If the agents do not reach the goal, then the fraction is set to 1. For each episode, the $n$ episode fractions are averaged together; this number is then normalized across all evaluation episodes (lower is better).

• •

  Average Collisions $(\#col)$: the total number of collisions that agents had in an episode, normalized by the number of agents and then normalized by the number of evaluation episodes (lower is better).

While the standard reported metric for MPE environments is the global reward, we have not included it in our example results. Since we introduced a new reward formulation for our model, we found reward comparisons challenging to both interpret and compare, given the different reward functions that the baselines were designed with. Similar to [32], we use the success rate metric to indicate the overall success of the total number of agents.

In the simulation, every policy is trained with 2M steps over 5 random seeds. All results are averaged over 100 testing episodes. Associated hyperparameters for each baseline are included in the appendix. All models are generated by running on a cluster of computer nodes on a Linux operating system. We use Intel Xeon Gold 6248 processors with 384GB of RAM.

In this section, we demonstrate that AsynCoMARL can effectively learn policies for navigation even in settings with less frequent and asynchronous communications.

We compare our methodology against several alternative MARL frameworks that seek to provide limited communication.

• •

  GCS [33]: The authors factorize the joint team policy into a graph generator and graph-based coordinated policy to enable coordinated behavior among agents.

• •

  Actor-Attention-Critic [5]: Actor-Attention-Critic uses a centralized critic with an attention mechanism that dynamically selects which agents to attend to at any time point during training.

• •

  asyncMAPPO [34]: The authors extend multi-agent proximal policy optimization [25] to the asynchronous setting and apply an invariant CNN-based policy to address intra-agent communication.

• •

  TransfQMix [32]: TransfQMix relies on graph transformers to learn encodings over the state of observable entities, and then a multi-layer perceptron to project the q-values of the actions sampled by the individual agents over the q-value of the joint sampled action.

• •

  CACOM [35]: The authors employ a two-stage communication scheme where agents first exchange coarse representations and then use attention mechanisms and learned step size quantization techniques to provide personalized messages for the receivers.

• •

  DGN [11]: DGN relies on graph convolutional networks to model the relational representation and implicitly models the action coordination.

We rely on the following implementations for each baseline and provide links to those implementations here. Note that we used the same hyperparameters as used in their original implementations, assuming that they were optimal.

1. 1.

   asyncMAPPO: https://github.com/yang-xy20/async_mappo/tree/main

2. 2.

   GCS: https://github.com/LXXXXR/GCS_aamas337/tree/master

3. 3.

   DGN: https://github.com/jiechuanjiang/pytorch_DGN

4. 4.

   CACOM: https://github.com/LXXXXR/CACOM/tree/main

5. 5.

   Actor-Attention Critic: https://github.com/shariqiqbal2810/MAAC/tree/master

6. 6.

   TransfQmix: https://github.com/mttga/pymarl_transformers/tree/main

We rely on the following implementations for the two environments we used in our experiments.

1. 1.

   Cooperative Navigation: https://github.com/sydneyid/satellite-cooperative-nav

2. 2.

   Rover-Tower: https://github.com/shariqiqbal2810/MAAC/tree/master

There are $n$ agents and $n$ goals, along with static obstacles in the environment. Each agent is supposed to go to its distinct goal while avoiding collisions with other entities in the environment. Agents start at random locations at the beginning of each episode; the corresponding goals are also randomly distributed. Agents are governed by Clohessy-Wiltshire Equations [31]:

The above equations consider a localized coordinate system centered around one of the satellites, referred to as the target). The target satellite is assumed to have a circular orbit with an orbital rate of $\omega_{n}$. The coordinates are defined such that $x$ is measured radially outward from the target, $y$ is along the orbit track of the target body, and $z$ is along the angular momentum. $u_{x}$, $u_{y}$, and $u_{z}$ represent the acceleration in the x,y, and z- directions. A full derivation of the perturbed equations can be found in [37]. While there exist additional environmental perturbations, such as solar radiative pressure or three-body effects, their impacts are magnitudes smaller [38]. In our method, we only use the $x$ and $y$ to dictate the agent’s motion, as we are assuming all agents act within the same $z$ plane. The experimental setup for each result is reported in Table 5.

In the Rover-Tower environment, there are 4 rovers and 4 towers. Tower agents can send one of 5 discrete communication messages to their paired rover at each time step. Towers cannot move but can communicate with rovers so that rovers can move towards their corresponding goal. Each pair of rover and tower are negatively rewarded by the distance of the rover to its goal at each episode. In our setup, the communication is integrated into the environment (in the tower’s action space and the rovers observation space), rather than being explicitly part of the model.

We rely on the following package versions to support our algorithm:

1. 1.

   gym = 0.26.2

2. 2.

   torch = 1.13.1

3. 3.

   torch-geometric = 2.3.1

4. 4.

   tensorboardX = 2.6.2.2

5. 5.

   wandb = 0.17.4