GCS: Graph-Based Coordination Strategy for Multi-Agent Reinforcement Learning

The acyclicity constraint is an important issue in our problem setting. In this work, we also use the penalty terms like zheng2018dagsnotears ; lachapelle2020gradientdag ; zhu2019dagRL to ensure acyclicity. The result in zheng2018dagsnotears shows that the directed graph $\mathcal{G}$ with binary adjacency matrix $A$ is acyclic if and only if:

where $e^{A\circ A}$ is the matrix exponential, $A\circ A$ guarantees the non-negativity, and $d$ is the number of nodes in the DAGs. The ‘$\operatorname{trace}$’ of a matrix is defined as the sum of the diagonal elements zheng2018dagsnotears . The constraint function $g$ should satisfy that: (a) its derivatives are computable, and (b) $g$ can be the measurement of DAGs.

Moreover, taking the trade-off between efficiency and performance into account, we claim that the maximum depth of graph structure should be adjustable over different tasks. Therefore, we propose an alternative constraint to control the hierarchy of the generated DAGs as follows.

We provide a detailed proof of proposition 2 in Appendix A.1. Here, we define ${c}(A^{k}):=sum({A^{k}})=\sum\nolimits_{i}{\sum\nolimits_{j}{A_{ij}^{k}=0}}$, which is equivalent to $A^{k}=O$. We remark that as long as constraint ${A^{k}}=O$ holds, it can be guaranteed that the maximal length between any two nodes $i$ and $j$ of the DAG does not exceed $k$.

Based on the foregoing, we can optimize $\rho$ parameterized by $\varphi$ with the maximal length $k$ by:

where $W=\rho_{\varphi}(\cdot)$ denotes the weight matrix generated from the graph generator $\rho$. Then the weight matrix is modeled as a Bernoulli distribution, from which the binary adjacency matrix $A$ is sampled. Here, we use the constraints of the weight matrix $g(W)$ and $c(W^{k})$ to approximate those of the adjacency matrix $g(A)$ and $c(A^{k})$ due to the consistency of representing the graph structure. With this approximation, we restate $\eta^{i}$ as:

Fixing graph-based coordinated policy $\bm{\pi}$, we approximate the graph generator $\rho$ as follows. We augment the original problem shown in Equation (8) with a quadratic penalty using the augmented Lagrangian technique bertsekas1997nonlinear :

with the penalty $\xi>0$.

Next, we convert the Equation (10) to an unconstrained Lagrangian function:

We provide a detailed proof of proposition 3 in Appendix A.2.

In proposition 3, we remark that after considering the influence of various decision dependencies on the reinforcement learning tasks, we can obtain the underlying graph structure that makes the best response to MARL tasks.

The graph-based coordinated policy can be obtained from the standard multi-agent actor-critic framework. As for the policy $\pi^{i}$ of the actor, we use the RNN network with the stochastic policy gradient to model the action distributions. The critic used to criticize the joint actions made by the actors is a three-layer feed-forward neural network activated by the ReLU units, denoted as $f_{critic}(o_{t},u_{t})=ReLU(\textit{MLP}\langle o_{t},u_{t}\rangle)$.

As shown in Figure 2, the graph generator $\rho_{\varphi}(\bm{o})$ adopt an encoder-decoder module used to find the ACG. The GAT-based encoder can model the interplay of agents and extract the further latent representations. The MLP-based decoder is used to recover the pairwise relationship between agents to generate an ACG. The graph generator takes the local observations of the agents as input and outputs the ACG $\mathcal{G}$ to obtain the decision dependency for the decision-making process of the graph-based coordinated policy, elaborated as follows.

The graph generator contains two sub-modules. Firstly, we use the graph attention network (GAT) velivckovic2018gat as the attention-based encoder to extract the latent information for the graph structure generation. First, the simple feature is extracted by a multi-layer perceptron (MLP) as an initial step:

Due to the sufficiently expressive power of GAT, we use it to extract the further latent information of the simple feature. We compute the importance coefficients through the attention mechanism:

where $\bm{W}\in R^{d{{}^{\prime}}\times d}$ is a learnable weight matrix, $d{{}^{\prime}}$ and $d$ denote the dimensions of the input vector and the latent vector, respectively, and $k\in{{\{I\}}^{-i}}$ indexes other agents except the agent $i$.

Then the multi-head attention is used to stabilize the learning process of self-attention, and the final latent feature is as follows:

where $M$ is the number of attention heads. $a_{t,m}^{ij}$ are importance coefficients computed by the k-th attention mechanism, and $W^{m}$ is the corresponding weight matrix.

Another sub-module in the graph generator is the decoder that generates a weight matrix used to sample the graph structure. Since the GAT-based encoder has already provided sufficiently expressive features among agents, a single-layer decoder is enough to easily construct the pairwise relationship between the encoder outputs to find a better structure of DAG for the decision policy.

where $\bm{h}_{t}^{l}$ and $\bm{h}_{t}^{r}$ are agents’ higher-level representations from two encoder outputs, $\bm{W_{l}},\bm{W_{r}}\in\mathbb{R}^{d_{h}\times d_{e}},u\in\mathbb{R}^{d_{h}\times 1}$ are trainable parameters, and $d_{h},d_{e}$ are the hidden dimension and encoder output dimension, respectively.

Moreover, the logistic sigmoid function $\sigma(\cdot)$ generates the probability for constructing the Bernoulli distribution from which the binary adjacency matrix $A$ is sampled. The binary adjacency matrix forms a directed graph corresponding to the ACG $\mathcal{G}$. Here, we denote this graph generation process as $\mathcal{G}\sim\rho_{\varphi}(\bm{h})$.

In the graph generator, the attention head in the GAT encoder is set to 8, the stacked attentional layers are set to 4, and the hidden units in the MLP is set to 64. In the graph-based coordinated policy, the actor critic architecture is adopted. The recurrent layer comprised of a GRU with a 64-dimensional hidden state, with a fully-connected layer before and after, is used as the actor. The critic is a two-layer MLP with the ReLu activation.

As an extension of Multi-domain Gaussian Squeeze (MGS) son2019qtran , Collaborative Gaussian Squeeze is a challenging environment for evaluating coordination. In MGS, there exist $K$ domains $[({\mu_{1}},{\sigma_{1}}),...,({\mu_{K}},{\sigma_{K}})]$ in the system. The system contains $N$ agents, and each agent $i$ can take actions $a_{i}$ within range of $\{-10,-9,..,0,1,..,8,9,10\}$. The prior $s_{i}\in[0,0.2]$ given by the environment represents the unit-level resource for each agent $i$. The total amount of resources mobilized by all agents is denoted as $f(\bm{u})=\sum\nolimits_{i}s_{i}\times a_{i}$. The goal is to maximize the joint reward $G(\bm{u})=\sum\nolimits_{k=1}^{K}{f(\bm{u}){e^{-{{(f(\bm{u})-{\mu_{k}})}^{2}}/{\sigma_{k}}^{2}}}}$. In our settings, we modify the original MGS to a collaborative task. We use two domains $[(\mu_{1}=5,\sigma_{1}=1.25),(\mu_{2}=-5,\sigma_{2}=1.25)]$. The computation of the joint reward is as follows:

Figure 3(a) shows the reward curves for this setting. According to the above definition, the reward is maximized when the resources of all the agents reach $\mu_{1}$ or $\mu_{2}$. In this environment, the social welfare depends on the intensity of collaboration.

Cooperative Navigation is a classic scenario implemented in the multi-agent particle world. This scenario has $n$ agents and $n$ landmarks, which are initialized with random locations at the beginning of each episode. The objective of the agents is to cooperate to cover all landmarks by controlling their velocities with directions. The action set $\mathcal{A}$ includes five actions: [up, down, left, right, stop]. Each agent can only observe its velocity, position, and displacement from other agents and the landmarks. The shared reward is the negative sum of displacements between each landmark and its nearest agent. Agents must also avoid collisions, since each agent is penalized with a ‘$-1$’ shared reward for every collision with other agents. We set the length of each episode as 25 time-steps. Therefore, the agents have to learn to navigate toward the landmarks cooperatively to cover all positions quickly and accurately. Figure 3(b) shows a classic scenario in Cooperative Navigation with $N=3$.

GRF is a realistically complicated and dynamic simulation environment without any clearly defined behavior abstraction, which is a suitable testbed for studying multi-agent decision making and coordination. In GRF, we use the Floats wrappers to represent the state. The Floats representation contains a 115-dimensional vector that summarizes the information, such as the ball position and possession, coordinates of all players, and the game state. Each player has 19 actions to control, including the standard move actions and different ball-kicking techniques. The rewards include the $SCORING$ reward $(-1,+1)$ and the $CHECKPOINT$ reward, which is the shaped reward that specifically addresses the sparsity of $SCORING$. Detailed descriptions are shown in Appendix B.

We compared our results with several baselines as follows. VDN and QMIX are state-of-the-art value factorization approaches that follow the regime of centralized training and decentralized execution, belonging to the class of fully decentralized control policies, with which it is difficult to obtain coordinated behaviours. DCG uses fully connected graphs for belief propagation, which only allows the message passing of paired agents. DGN aims at learning abstract representations to make simultaneous decisions.

• •

  VDN sunehag2018vdn : Value Decomposition Network (VDN) imposes the structural constraint of additivity in the factorization, which represents $Q_{tot}$ as a sum of individual Q-values.

• •

  QMIX rashid2018qmix : This was proposed to overcome the limitation that VDN uses the linear decomposition and ignores any extra state information available during training. QMIX enforces $Q_{tot}$ to be monotonic in the individual Q-values $Q^{i}$.

• •

  DCG bohmer2020dcg : Deep Coordination Graph (DCG) factorizes the joint value function of all agents according to a coordination graph into payoffs between pairs of agents, which coordinates the actions between agents explicitly.

• •

  DGN jiang2018dgn : DGN relies on a graph convolutional network to model the relation representations, implicitly modeling the action coordination.

In this game, there are 10 agents, and the maximum episode length is also set to 10. To emphasize the feasibility and effectiveness of our proposed framework, we first conduct the experiment on CGS. We report the average episode rewards over 10 random runs, shown in Figure 4. Our proposed algorithm GCS outperforms the baseline methods by a large margin. It can be seen that our algorithm handles the collaborative problem well; the action coordination graph facilitates behavior learning to promote cooperation. Next, we will verify the effectiveness of our algorithm on more complicated environments.

As shown in Figure 3(b), Cooperative Navigation is a fully cooperative environment in which $N$ agents (circles) must cooperate to reach $N$ landmarks (crosses) with as few collisions as possible. We conduct experiments in Cooperative Navigation with $N=4$ and with $N=6$. Figures 5(a) and 5(b) show the learning curve comparisons for the two cases. We report the average episode reward at a training step, averaged over 10 independent running seeds.

First, our algorithm outperforms most baseline algorithms by giving higher converged rewards. We consider that our performance improvement results from the action coordination graph representing the action dependency for better coordination. Moreover, our algorithm converges fast during training, which is possibly because the hierarchical decision policies can efficiently induce coordination among agents in this cooperative setting. In addition, our algorithm can achieve a lower variance than those baselines, which indicates that the learned action coordination graph can reduce the uncertainty in decision making to facilitate cooperative behaviors among agents.

In contrast, VDN and QMIX take actions simultaneously without considering the action dependency among agents. They are faster during training, but that is of no benefit in inducing cooperation among agents. Additionally, DCG exhibits mediocre performance in this task. We believe that DCG considers only pairwise relationships between agents, which may disturb the overall balance in the system. In this case, DGN shows good performance consistent with ours, which shows that implicit action coordination modeling is also effective in pure cooperative settings.

In order to clearly show the meaningful effect of the action coordination graph in the decision-making process, we give a visualization at a time step in an episode of Cooperative Navigation, as shown in Figure 6. Figure 6(b) shows the topological structure of the learned ACG, and it suggests that the decision dependency of the agent is [3, 4, 2, 1]. The parent sets of the four agents are denoted as $pa(1)=\{4,2\}$, $pa(2)=\{3,4\}$, $pa(3)=\emptyset$, and $pa(4)=\{3\}$, respectively. First, agent 3 decides to move to the bottom-left landmark, then agent 4 takes the best response and decides to move to the bottom-right in order to avoid conflict with agent 3. After agent 2 knows the decisions of the previous two agents, it chooses the closer upper-left as its target instead of the bottom-right. Finally, agent 1 moves after observing the decisions of agents 2 and 4. This visualization shows how agents’ joint actions deriving from the ACG representing the underlying decision dependencies achieves efficiency.

To evaluate our method in complicated and dynamic environments, we conduct several experiments on GRF, as shown in Figure 7. In the 3-vs-2 scenario, three of our players try to score from the edge of the box, and the opponent team contains one defender and one keeper. In the 3-vs-6 scenario, there are six opponent players on the pitch to play against three of our players. In the 5-vs-5 scenario, each team has a keeper, an offensive player, and three defenders. Here, we report the average episode reward at a training step for each scenario, averaged over 10 independent running seeds.

As can be seen in Figure 7, our algorithm always obtains higher rewards than all the baselines in the different scenarios of GRF. This indicates that our method is quite general in complicated and dynamic environments. Moreover, this performance improvement in GRF demonstrates that our approach is good at effectively handling stochasticity and sparse rewards. This is because the learned ACG with the decision dependency is an efficient way to mitigate uncertainty and induce cooperation among agents. Taking the 3-vs-6 scenario for further demonstration, the training curve of QMIX fluctuates and is unstable, indicating this method’s inability to adapt to the dynamically complicated scenario with multiple opponent players. Here, DCG shows a trend of non-convergence, but our algorithm steadily rises to converge and obtains the highest reward, which exhibits the modeling supremacy of our approach for handling complicated tasks.