Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network

An MMDP can be formalized as a tuple $(\mathcal{N},S,A,\tau,\mathcal{U})$. $\mathcal{N}$ is the set of all agents, indexed by $1,....,n$. We use $-i$ to refer to all other agents except $i$. $S$ is the set of all possible states and $A=A_{1}\times...\times A_{n}$, is a set of action spaces where $A_{i}$ is the set of actions for agent $i\in\mathcal{N}$. A joint action $a\in A$ is the combination of individual actions of all agents, $a=\langle a_{1},...,a_{n}\rangle,a_{i}\in A_{i}$. $\tau:S\times A\rightarrow S$ is a transition function that takes a joint action and changes the state according to the deployed environment specifications. $\mathcal{U}=(u_{i},...,u_{n})$ is the global utility function and for every agent $i\in\mathcal{N},u_{i}:S\rightarrow\mathbb{R}$ denotes the utility function to determine the payoff of states in $S$ for that agent. A policy is the task of deciding which action to choose in any state. The policy for the $i$-th agent is denoted by $\pi_{i}:S\rightarrow|A_{i}|$, a distribution over the action space depending on the state the agent is in. The collection of policies of all the agents is called the joint policy $\pi=\langle\pi_{1},...,\pi_{n}\rangle$. If the joint policy is fixed, then all the agents take actions according to that, and the expected reward or the state value $V_{i}^{\pi}(s)$ of an state $s$ for agent $i$ is:

And the expected reward of an action or action value $Q_{i}^{\pi}(s,a)$ of state $s$ for agent $i$ in joint action $a$ is:

Simulation based methods provide a means of policy learning and planning in non-stationary environments where it is possible for the agents to find their best response policy according to their current model of other agents. Monte Carlo Tree Search (MCTS) Kocsis and Szepesvári (2006) is an effective tree searching algorithm that uses simulation data to balance between exploration of new solutions and exploitation of solutions with potential. There are four major steps in the traditional MCTS simulation process. At any given moment, with an already expanded simulation tree, the first step is to start from the root and select a new leaf by going down the tree. While traversing, at every node, scores for its child nodes are calculated and the child with the best score is selected to go further down until a leaf is reached. The score is calculated with,

where $Q(s,a)$ is the action value, $n_{i}$ is the visit count of action $a$ and $N_{i}$ is the total visit count of state $s$. The constant $c$ determines the tradeoff between exploration and exploitation. The next step is to expand the selected leaf and add its child nodes to the tree, called the expansion step. After selection and expansion, the state value of the child nodes are estimated, traditionally with random rollouts. The last step is to carry the newfound information about the expanded node to its ancestors through backpropagation. In an simulated tree, every node in that tree holds information about its expected reward, the number of times that node was visited and the edges to its child nodes.

AlphaZero Silver et al. (2017) takes a different approach with MCTS. In recent studies, it has been shown that generating policies for large state and action spaces can be done with deep neural networks rather than random plays Mnih et al. (2015). Hence, instead of estimating state values with computationally exhausting rollouts, a neural network is used in AlphaZero. The neural network also provides an initial policy for the action space to better guide the tree searching process. This not only makes the process faster, but also helps focus on better solutions.

In multi-agent environments, every agent interacts with a subset of other agents. These interactions can be modeled using a graph structure called Coordination Graph (CG), where the nodes are the agents and interacting agents have edges between them. This structure captures the locality of the dependencies of the agents regarding payoffs. This locality of dependency also determines the cooperative policies of the agents. In Figure 1, the agents create a forest of connected graphs based on their range. The edge creation criteria can be predefined and the generated CG will denote which agents need to communicate.

New models and architectures have been brought to light that use graphs in training models to complete tasks regarding graphical structure, called Graph Neural Networks (GNN). The use of GNNs has been shown to be effective in agent modeling Böhmer et al. (2020)Liu et al. (2020). Graph Convolutional Networks (GCN) is a popular GNN for graph embedding. GCN uses a message passing mechanism that lets the nodes of a graph to aggregate and retain information about their neighbors of different distances. This greatly helps in representation learning of graphical structures and use it to generate practical solutions.

The simulator starts with parameters initial state $s$ and set of all the agents $\mathcal{N}$. The $root$ of the simulation tree is initialized by the initial state $s$ in line 12.

There are four steps in the simulation process and they are repeated multiple times within a fixed amount of time in the while loop in lines 13-22. In the first step, a leaf is reached from the $root$ in line 14 by frequently calculating the scores of the children of a node with Eq. 4 and choosing the child with the highest score. Visit counts and state values of visited nodes, while traversing down the tree, are updated. After choosing a leaf node, its child nodes are created and the leaf is labeled as expanded. The number of possible child nodes is exponential in magnitude with respect to the number of agents. So, in the second step, a small subset of joint actions are sampled from the joint action space with SiCLOP-P in line 17 and the child nodes are created from the set of sampled actions to complete step three and four in lines 18-22. After creating a child node in line 18, its state value is determined in lines 20-21 by calculating penalties and prediction made by SiCLOP-NN. Then the state value is backpropagated to its ancestors in line 22.

At the end of generating the simulation tree, the most visited joint action which denotes the joint action with the most potential, is selected in line 23 and then returned.

The SiCLOP-P component of the algorithm described in lines 25-35, focuses on pruning the joint action space for SiCLOP-S. We propose a novel joint action sampling method inspired by Gibbs sampling to effectively search over the joint policy space. First, we elaborate the sampling process. The predictor SiCLOP-NN generates the best response policy for the agents given a state. Using this predictor, the best response policy of every agent is sampled considering the agent policies as static periodically. The sampling runs for multiple cycle. At cycle $k$, the policy of agent $i$ is updated with the conditional best response policy

In SiCLOP-P, the initial policy is represented by initial joint action selection in line 28. Then, in lines 29-34, a fixed number of joint actions are sampled. The agents are selected periodically in lines 30-32 where the current policy of agent $i$ is replaced by the best response policy. To update the policy, all other agents’ policies are considered to be static. The current adopted policies of the agents are represented by $s^{\prime}$ in line 31. Then the best response policy is updated in line 32 and new action is sampled from that policy in line 33. At the end of the cycles, in line 34, the updated joint action is added to the sampled joint actions and after a fixed number of cycles, the set of sampled joint actions are returned.

The policy generator SiCLOP-P_NN predicts the best response policy for an agent. So, the generated policy for agent $i$ belongs to $BR_{i}(\pi_{-i})$. In Lemma 3.2, we show that SiCLOP-P eventually converges to an equilibrium joint policy for a group of agents.

This shows that with a well trained policy prediction function, the joint policy of every group will converge to a best response policy equilibrium. But in non-stationary environments, there might not a single joint policy equilibrium for the agents to converge to. In that case, the agents continuously create new best response policies to maximise their respective payoffs.

The characteristics of the sampled joint actions change gradually throughout the learning process. In the early stages of learning, the policies of the agents are initialized to be random or according to prior knowledge. The set sampled joint actions represent that initial joint policy. If it is random, then the agents explore different strategies through simulation as a group and develop their policies by finding strategies that lead to better payoffs. At the same time, agents adapt to other agents’ policies and search for best responses. And if it is initialized with handcrafted policy, then the agents evolve by finding counter strategies and updating their existing policies. In both cases, they initially search randomly for new strategies and select the optimal ones in simulation. After repeating the learning process, the agents start to generate more definitive policies, meaning some actions are more preferred than others in their respective policies. This causes the sampling method to sample joint actions that have higher probability of being chosen to maximize rewards for all the agents. Directly predicting the optimal policy can concentrate on single policies and might lose robustness. Rather, simulating a search tree tests different ways of solving a MMDP with a set of best response policies to choose from. This reduces the dependency on single strategies and provides stable robust solutions.

The neural network SiCLOP-NN is a GCN based policy predictor and state value estimator. It has two parts, feature extraction and prediction, as shown in Figure 2. The first part is used for graph embedding as our algorithm focuses on CGs. This feature extraction part consists of multiple GCN layers. The number of GCN layers determine the number of hops of feature information among agents. Two GCN layers would mean that the features of the agents will reach other agents up to two hops away. The second part takes the extracted feature vectors of the agents from the GCN block and generates predicted best response policies and state value estimations for every agent. The state values are aggregated to construct a combined state value. In Figure 2, The red outlined pipeline is policy predictor and the blue outlined pipeline is state value estimator. SiCLOP-NN returns a boltzman probability distribution over the actions to enable weighted sampling of actions in SiCLOP-NN.

SiCLOP-NN takes the local information of the agents and a CG as input. The setting of the local information can be defined, for example, the cells within a limited range of the agent. The CG is also constructed according to predefined rules. In SiCLOP, the input is created in the preprocess function where the local information of the agents are collected into a set in lines 39-40. Then the set is returned with the constructed CG.

The training of the neural network occurs after a fixed number of simulation episodes. In Algorithm 2, a fixed-sized batch $B_{i}$ of state-policy and state-value pairs are randomly selected from the data from recent episodes for every agent. The reason behind selecting data from recent episodes is to train the neural network on recently found policies of the agents. This allows the agents to adapt to new policies faster. Then predictions are made on that batch of states to calculate the cross-entropy error between the real policy and predicted policy for agent $i$. This is calculated using

And the mean squared error between real value and predicted value is calculated with

The neural network is updated to minimize the combined loss.

The neural network trains and predicts on states consisting of any number of agents, with their internal interactions passed as CG into the GCN block. This adds the flexibility of performing predictions on variable number of agents. On the other hand, the input is a collection of observations of the agents dictated by a constraint of locality, i.e. communication range. SiCLOP-NN’s architecture allows making prediction in environments of any size and shape. If the agents are trained in a smaller environment with fewer agents, the learned agents can still apply the learned policies on much larger environments with more agents as their policy prediction only depends on the locality of information.

The first experiment is to observe how the agents improve their policies to get better results. In this experiment, there are $8$ agents in a $15x15$ grid with $15$ obstacles. The agents have maximum 40 steps to complete the tasks. A total of 500 randomly generated problem states were generated and SiCLOP was tested on these 500 solution episodes. Agent policy predictor was trained every 10 episodes. For every step in the episode, the simulator can either run for a maximum amount of time or can expand limited number of simulation nodes. The algorithm was tested on three such settings. To test the scaling up ability of the algorithm, the trained policy predictors were used to test if it can generate good quality solutions on new environments of different sizes and difficulties.

From the results shown in Figure 4, we can see that at the beginning of the experiment, the agents adopt random policies and as the episodes progress they start to adapt to the rules of the environment and converge to a high score around $1.0$. An average score around $1.0$ denotes that the agents are avoiding penalties and succeeding at their given tasks. If the simulation limitations are tighter, the algorithm struggles to find good solutions at the beginning, even yielding to negative scores. But as SiCLOP is allowed to do more simulation before taking actions, it generates better solutions from the start, training the agents on quality solutions. This helps converging to optimal scores much faster, as we can see in Figure 4. It is also shown that, in general, more simulation leads to better score on average.

After training the agents in a $15x15$ grid, the policy predictor is transferred to randomly generated environments. These environments differ in size, number of agents and obstacles. In Table 1, we can see that the average score per agent in these environments are above $1.0$, which implies that the algorithm is capable of adapting to a varying environment and avoid penalties by using knowledge from a different environment. But in case of other planning algorithms, FV-MCTS-MP stores state values of visited states so it cannot use any prior knowledge on new environments and ABC uses the entire state to predict policies, so it can not be applied to an environment of different size. Transfer learning lets the algorithm to train on different environments, enabling agents to initially train in a small environment and then train in larger environments. As seen from the experiment, the prediction model can transfer the knowledge allowing the training in large environments to converge faster. This can save time and computational resources in case of complex environments and large single agent action spaces, making the algorithm applicable to most realistic MMDP problems.

In the second experiment, the cooperation learning and adaptability of the algorithms were tested. SiCLOP and ABC are both model learning algorithms and FV-MCTS-MP is a centralized tree searching algorithm. While ABC adopts a decentralized approach, SiCLOP chooses and learns its policies in a centralized manner. To test the cooperation learning ability of the agents, both algorithms were run on the same 500 episodes with $5$ seconds time limit per action in a $15x15$ grid with $5$ obstacles and $8$ agents. In Figure 5, we can see how the algorithms learn cooperation and adapt to handle $collision$ and $outofbounds$ penalties.

From Figure 5, it is clear that ABC fails to learn how to avoid collisions but can learn to avoid going out of bounds. The penalty of going out of bounds depends on individual agents and collision mostly depends on the interaction of the agents. ABC learns the agent models one at a time and tries every available action. This helps the agent models to learn individual actions effectively. But in case of collision, it is very hard to avoid without communication among agents, causing the ABC agent models to not converge to effective policies. As there is no communication, there is no differentiation between successful and unsuccessful joint actions before the actions are executed. FV-MCTS-MP consistently outperforms ABC in cooperation but due to time limit and small number of iterations, there were no significant increase in performance. SiCLOP outperforms both algorithms as it searches over the joint policy space and needs lesser number of iterations to find cooperative joint policies to avoid the penalties. Throughout the training process, SiCLOP shows a stable performance and converges to almost no average penalty.

Contrary to the collision penalty, all three algorithms manage to adapt to the out of bounds penalty. Before training, FV-MCTS-MP outperforms both SiCLOP and ABC but after training these two perform better. Similar to previous experiment, FV-MCTS-MP’s performance does not get better because of the limitations. The reason behind the similar performances of SiCLOP and ABC is that avoiding out of bounds penalty does not require any cooperation.

In this experiment, SiCLOP was compared to FV-MCTS-MP and ABC by average scores accumulated on gradually increasing size of environments and number of agents. All of the algorithms had $10$ seconds time limit per action. SiCLOP and ABC algorithms were trained on 500 episodes. In Figure 6, we can see the comparison in scores.

The algorithms were tested in three different environments of varying sizes. As the size of the environment gets larger, accumulating a high score becomes harder as there are more obstacles and the objectives are farther. From this experiment, it is clear that as the environment gets larger, ABC is more prone to penalties and accumulates lower average scores. FV-MCTS-MP, with limited time and number of iterations, is unable to find sequence of actions that lead to high score. As the simulation tree of FV-MCTS-MP is wide, it calculates the expected action value over a wide and shallow tree, causing it to have average negative values. So, FV-MCTS-MP prefers not to move the agents to minimize penalties. But, as SiCLOP searches over a deeper search tree, it can find better solutions to the MMDP problems. Also, the size of the environment hardly affects the performance of SiCLOP as the agents only use their local information and interactions. These attributes and better cooperation learning allow SiCLOP to outperform both ABC and FV-MCTS-MP by a significant margin.