PooL: Pheromone-inspired Communication Framework for Large Scale Multi-Agent Reinforcement Learning

DRL methods like Deep Q-Learning (DQN) Mnih et al. (2013) promote the application of reinforcement learning in many fields such as Atari Games Mnih et al. (2015) and Go Silver et al. (2016). But most existing methods of DRL focus on the control of a single agent. As in real life, many scenes contain multi-agent interactions and cooperation. MARL attracts more and more attention. Lowe et al. (2020); Wei et al. (2018); Foerster et al. (2018) are classic MARL algorithms following the framework of CTDE Oliehoek et al. (2008). These algorithms apply modified single-agent reinforcement learning algorithms such as DDPG lillicrap et al. (2016) and Soft Q-Learning Haarnoja et al. (2017) in MARL settings. However, such methods still suffer from the curse of dimensionality because the method still needs to handle all agents’ features while training. Communication is an effective way to solve such problems. CommNet Sukhbaatar et al. (2016) designs a channel for all agents to communicate and extract information from all agents by average pooling. In order to better handle the situation with more agents, Zhao and Ma (2019) uses an information filtering network to select useful information. VAIN Hoshen (2018), ATOC Jiang and Lu (2018) and TarMAC Das et al. (2019) use attention mechanism to select communication targets and information more reasonably. But these methods still assume a centralized controller that can access all communication needed information while training. DGN Jiang et al. (2020) models agents’ interactions with graph structures by Graph Neural Networks (GNN) Zhou et al. (2020). In DGN, agents’ communication is achieved by the convolution of graph nodes(e.g. agents nearby). Graph convolution is able to express more complex agent interactions. But its performance and convergence speed are still confined by the number of agents.

In order to make algorithms scale to many agents, MFQ Luo* et al. (2018) accepts the average action value of nearby agents as extra network input. Zhou et al. (2019) proposes to approximate Q-function with factorized pairwise interactions. Wang et al. (2020) provides another idea that trains agents from few to more with transfer mechanisms.

Swarm Intelligence is widely used in group control problems. Di Caro and Dorigo (1998); Maru et al. (2016) discusses the use of swarm intelligence in adaptive learning of routing tables and network-failure-detection. Bourenane et al. (2007); Xu et al. (2019) utilize pheromones to achieve multi-agent coordination. However, Swarm Intelligence algorithms like Bird Swarm Algorithm and Ant Colony Optimization are rule-based, as a result, they can not learn from environments. Phe-Q Monekosso and Remagnino (2001) first introduces the idea of pheromone mechanisms to reinforcement learning. Matta et al. (2019); Meerza et al. (2019) follow similar ideas and improve the convergence speed of Q-Learning in different environments. Xu et al. (2021) proposes to release the pheromone according to the distance between the agent and the target, and apply the pheromone information to Deep Reinforcement Learning.

Learning-based methods are more and more popular with the success of deep learning in many fields. Most methods mentioned above can not take advantage of deep neural networks. DRL is applied in Xu et al. (2021), but the way Xu et al. (2021) obtains pheromones is artificially designed based on the specific environment. It will be more appropriate if pheromones are generated based on learning algorithms.

The problem for multi-agent coordination in this paper can be considered as a Dec-POMDP game Hansen et al. (2004). It can be well defined by a tuple $M=<I,S,\left\{A_{i}\right\},P,\left\{R_{i}\right\},\left\{\Omega_{i}\right\},O,T,\gamma>$, where $I$ is a set of $n$ agents, $S$ is the state space, $A_{i}$ is the action space for agent $i$, $\bm{\Omega_{i}}$ is the observation space for agent $i$, $T$ is the time horizon for the game and $\gamma$ is the discount factor for individual rewards ${R_{i}}$. When agents took actions $a$ in state $s$, the probability of the environment transitioning to state $s^{\prime}$ is $P(s^{\prime}|s,a)$ and the probability of agents seeing observations $o$ is $O(o|s,a)$.

For agents in the problem above, their goal is to maximize their own expected return $G_{i}=\sum_{t=0}^{T}\gamma^{t}R_{i}^{t}$. The achievement of this goal requires the cooperation of agents belonging to the same team.

Q-Learning is a popular method in single agent reinforcement learning. For agent $i$, Q-Learning uses an action-value function for policy $\pi_{i}$ as $Q^{\pi}_{i}(\Omega_{i},a)=\mathbb{E}\left[R_{i}\mid\Omega^{t}_{i}=\Omega,a^{t}_{i}=a\right]$. Table Q-Learning maintains a Q-table and update Q-values by the following equation:

For every step, the agent selects its action based on the max value of these actions with an epsilon-greedy exploration strategy.

As Q-Table can not handle high dimensional states, Deep Q-Learning (DQN) uses neural network to approximate Q-function. DQN tries to optimize $Q^{*}_{i}$ with back propagation by minimizing the follwing loss function:

In DQN Mnih et al. (2015), there is a replay buffer to store agent information. Data is randomly extracted from the replay buffer to train the model. Target Q-network $\bar{Q}^{*}_{i}$ has the same architecture with ${Q}^{*}_{i}$. $\bar{Q}^{*}_{i}$ copies parameters from ${Q}^{*}_{i}$ frequently. In this way, DQN disrupts the temporal correlation and makes the training of the model more stable.

Figure 2 gives a whole picture of PooL. In the left of Figure 2, a simple grid world is built as a demonstration. From the view of the grey agent, the black-bordered squares represent the real grids of our environment. They are the smallest units to express information in our environment. In order to realize indirect communication based on pheromones, we need to set up a virtual medium, which is represented by the blue-bordered squares. The area near the current agent is divided into three regions from small to large: Influence Domain, Observation Domain, and Perception Domain. Agents handle these three regions with different behaviors. The meanings of the three regions are as follows.

• •

  Influence Domain: the influence range of pheromones released by agents.

• •

  Observation Domain: the partial observation of the agent in the environment.

• •

  Perception Domain: the range of pheromones that agents can perceive.

In nature, generally, animals perceive a much larger range of pheromones than they observe their surrounding environment by eyes. Therefore, in our settings, the Perception Domain is larger than the Observation Domain.

The right part of Figure 2 shows how PooL processes received information and updates pheromones. PooL consists of three components. The first is the information processor (composed of convolution layers and fully connected layers). Its input feature consists of two parts: real-world observations and a summary of pheromone information. Its output is Q-values, which can be transformed into pheromones. The second is the pheromone update mechanism realized through a virtual medium. Thirdly, a pheromone receptor is used to sense the pheromone information from the surrounding by convolution layers and transform the information into the representation features of the current environment. With the increase of the number of agents, the pheromone information extracted by agents can more accurately reflect the situation of the local environment.

In the following sections, we introduce the realization of our framework in three steps. First, a virtual medium is defined to allow the propagation of pheromones. Then, when agents release their pheromones, an update mechanism updates the values of pheromones stored in the virtual medium. Finally, pheromone information is processed by the pheromone receptor and serves as extra input for our reinforcement learning algorithms such as DQN.

In order to achieve pheromone-based indirect communication in the environment, a virtual medium to simulate the mechanism of pheromones in nature needs to be built first. A virtual map can be constructed by dividing the real environment map into $H\times W$ virtual grids. $pos(i)$ denotes a mapping function that can return agent $i$’s current position like $(h,w)$. $Info$ represents the pheromone value stored in the virtual medium whose shape is $(H,W,N_{A})$ ($N_{A}$ is denoted as the number of actions). $N(h,w)$ represents the current number of agents in $(h,w)$. The pheromone released by agent $i$ is denoted as $Phe(i)$. $Phe(i)$ is a vector with $N_{A}$ dimensions.

Following Q-Learning’s idea, pheromones are defined based on the Q-function. As the range of the Q-function’s estimated values changes while training, Q-values are standardized as pheromones. For agent $i$’s Q-values $Q_{i}$, they are transformed by a standardization function $St(Q_{i})$. After standardization, values of pheromones obey the standard normal distribution with the mean of $0.0$ and the standard deviation of $1.0$.

At the beginning of each time step, each agent calculates their Q-values and releases their pheromones on their Influence Domain. We take the mean value of all released pheromones in each grid, and update the value of pheromones in each grid according to the evaporation coefficient $\beta$. For a virtual grid, whose position is $(h,w)$, its pheromone value is updated by the following equation:

$I_{h,w}$ is a set of agents where for every $i\in I_{h,w}$, there is $pos(i)=(h,w)$.

The main difference between our proposed pheromone-based deep Q-Learning and DQN is that our information processor receives inputs from two sources. Except for the partial observation of the current environment, our processor also receives the information processed by the pheromone receptor. For agent $i$, the summary of pheromone information processed by agent $i$ is denoted as $Rec(i,Info)$. The optimization goal for Q-learning can be updated to:

The complete algorithm is described in Algorithm 1. The algorithm can be divided into two parts: calculate and update pheromones released by different agents and choose actions based on information observed by agents.

Compared with DQN, the extra cost of PooL lies in the pheromone update mechanism and pheromone receptor. For pheromone update, it can be done in $O(HW)$ time. For the pheromone receptor, the complexity of its network structure depends on the size of the Perception Domain, and the size of the Perception Domain is generally much smaller than the Observation Domain. As a result, the extra cost of PooL is relatively small compared with DQN.

As shown in Figure 2, we use the structure of multi-layer convolution and multi-layer fully connected layers as our information processor to process observations and summary of pheromone information. The receptor is also a convolution structure and processes pheromone information from the virtual medium. The size of the Influence Domain is $1\times 1$ and the size of the Perception Domain is $3\times 3$. The selection of hyperparameters such as the virtual map size $H,W$, and the evaporation coefficient $\beta$ will be discussed in Section 5.

PooL is mainly evaluated in six multi-agent environments. We first introduce the settings of these six environments, and then show performance of different methods in these six environments.

In this section, we take Battle as an example to discuss the details of our proposed method.