A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment

RL [1] is a sub-branch of machine learning that differs from classical supervised and unsupervised learning in that it learns through interaction. In RL, an agent interacts with an environment, constantly optimizing its policy based on the rewards or punishments received from the environment. RL consists of four main components: agent, state, action, and reward (Figure 2). The goal of RL is to maximize cumulative rewards. The agent must explore and learn through trial and error to find the optimal policy. This process can be modeled as a Markov Decision Process (MDP).

The agent is free to choose deterministic action to perform in a given state or to sample actions from some certain probability distributions. The emphasis here is on "free choice," which refers to the agent’s autonomy in changing action selection. The fixed selection of actions is referred to as a "policy". Specifically, at any time step $t$, the agent observes the current state $s_{t}$ from the environment and then performs action $a_{t}$. This action causes the environment to transition to a new state $s_{t+1}$ based on the transition function $s_{t+1}\sim P(\cdot|s_{t},a_{t})$ and the agent receives a reward signal $r_{t}=R(s_{t},a_{t})$ from the environment. Without loss of generality, we mainly discusses infinite MDP in this paper, where the goal of the agent is to find the optimal policy to maximize cumulative rewards. Mathematically, the above process can be summarized as finding a Markov, time-independent, and stationary policy function $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ that guides the agent to execute appropriate sequential decisions to maximize cumulative rewards, with the optimization objective defined as follows:

where the operator $\mathbb{E}^{\pi}[\cdot]$ computes the expectation over the distribution of sequences $\tau=(s_{0},a_{0},s_{1},a_{1},\ldots)$ generated by the policy $\pi(a_{t}|s_{t})$ and the state transition $P(s_{t+1}|s_{t},a_{t})$.

Based on the objective in Equation 1, the state-action value function, denoted as the $Q$-function, can be defined under the guidance of policy $\pi$. The $Q$-function represents the expected cumulative reward after taking action $a$ based on state $s$ under policy $\pi$. Additionally, the state value function $V$ describes the expected cumulative reward based on state $s$:

Clearly, the relationship between the state-action function $Q$ and the state value function $V$ can be expressed as $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}\left[Q^{\pi}(s,a)\right]$ and $Q^{\pi}(s,a)=\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}\left[R(s,a)+V^{\pi}(s^{\prime})\right]$. Based on the definitions of these two types of value functions, the learning goal of RL in Markov decision processes can be represented as finding the optimal policy $\pi_{*}$ that maximizes the value function: $\pi_{*}=\arg\max_{\pi}V^{\pi}(s),\forall s$.

For the MDP with finite state and action space, there exists at least one deterministic stationary optimal policy that maximizes the cumulative reward. Value based RL methods require the construction and estimation of a value function, which is subsequently used to guide action selection and execution, deriving the corresponding policy. Most of these algorithms do not directly use the $V$-function but require the use of the $Q$-value function to maximize the $Q$-value corresponding to Equation 2. The optimal policy induced by the $Q$-function through greedy action selection can be expressed as $\pi^{*}(a|s)=\mathbbm{1}\{a=\arg\max_{a}Q^{*}(s,a)\}$, where $\mathbbm{1}\{\cdot\}$ is the indicator function. The classical $Q$-learning algorithm, through temporal-difference (TD) learning, updates the agent’s function $\hat{Q}$-value to approximate the optimal function $Q^{*}$ [65], with the update process defined as follows:

In theory, given the Bellman optimality operator $\mathcal{B}^{*}$, the update of $Q$-value function can be defined as [66]:

To establish the optimal $Q$-value function, the $Q$-learning algorithm uses the fixed-point iteration of the Bellman equation to solve the unique solution for $Q^{*}(s,a)=(\mathcal{B}^{*}Q^{*})(s,a)$. In practice, when the world model [67] is unknown, state-action pairs are discretely represented, and all actions can be repetitively sampled in all states, the mentioned $Q$-learning method guarantees convergence to the optimal solution.

However, real-world problems may involve continuous, high-dimensional state-action spaces, making the above assumptions usually inapplicable. In such cases, agents need to learn a parameterized $Q$-value function $Q(s,a|\theta)$, where $\theta$ is the parameter instantiation of the function. To update the $Q$-function, the agent needs to collect samples generated during interaction with the environment in the form of tuples $(s,a,r,s^{\prime})$. Here, the reward $r$ and the state $s^{\prime}$ at the next time step follow feedback from the environment based on the state-action pair $(s,a)$. At each iteration, the current $Q$-value function $Q(s,a|\theta)$ can be updated as follows [68]:

where $\alpha$ is the update rate.

The $Q$-learning approach in Equation 5 can directly use a neural network $Q(s,a|\theta)$ to fit and converge to the optimal $Q^{*}$-value, where the parameters $\theta$ are updated through stochastic gradient descent (or other optimization methods). However, due to the limited generalization and reasoning capabilities of neural networks, $Q$-networks may exhibit unpredictable variations in different locations of the state-action space. Consequently, the contraction mapping property of the Bellman operator in Equation 4 is insufficient to guarantee convergence. Extensive experiments show that these errors propagate with the propagation of online update rules, leading to slow or even unstable convergence. Another disadvantage of using function approximation for $Q$-values is that, due to the action of the $\max$ operator, $Q$-values are often overestimated. Therefore, due to the risks of instability and overestimation, special attention must be paid to ensuring an appropriate learning rate. Deep Q Network (DQN) algorithm proposed by literature [69] addresses these challenges by incorporating two crucial factors: a target $Q$-network and experience replay buffer. Specifically, DQN instantiates a target $Q$-network with parameters $\theta^{-}$ to compute the temporal-difference target. Simultaneously, it uses an experience replay buffer $\mathcal{D}$ to store sampled sequences, ensuring both sample utilization and, during training, mitigating the correlation of sampled data through independent and identically distributed sampling. The optimization objective of DQN can be expressed as:

The target $Q$-network $Q(s,a|\theta^{-})$ is periodically updated to synchronize with the parameters of the $Q$-network. Through these methods and two crucial factors, stable training of the $Q$-network can be achieved.

Furthermore, many recent works have made additional improvements to DQN. In [70], the value function and advantage function (defined as $A(s,a)=Q(s,a)-V(s)$) are decoupled using a neural network structure, enhancing learning performance. In [71], a specific update scheme (a variant of Equation 6) reduces the overestimation of $Q$-values while improving learning stability. Parallel learning [72] or the use of unsupervised auxiliary tasks [73] contributes to faster and more robust learning. In [74], a differentiable memory module can integrate recent experiences by interpolating between Monte Carlo value estimates and off-policy estimates.

Policy gradient based methods [75] are fundamentally designed to directly learn a parameterized optimal policy $\pi_{\theta}$. In comparison to value-based methods, policy-based algorithms have the following characteristics: firstly, these methods can be flexibly applied to continuous action spaces; secondly, they can directly obtain a stochastic policy $\pi_{\theta}(\cdot|s)$. When parameterizing the policy with a neural network, a typical approach is to adjust the parameters in the direction of increasing the cumulative reward: $\theta\leftarrow\theta+\alpha\nabla_{\theta}V^{\pi_{\theta}}(s)$. However, the gradient is also subject to the unknown effects of policy changes on the state distribution. In [1], researchers derived a solution based on policy gradients that does not involve the state distribution:

where $\mu^{\pi_{\theta}}$ is the state occupancy measure under policy $\pi_{\theta}$ [76], and $\nabla_{\theta}\log\pi_{\theta}(a|s)$ is the policy’s score evaluation. When the policy is deterministic, and the action space is continuous, we can further obtain the Deterministic Policy Gradient (DPG) theorem [77]:

One of the most classical applications of policy gradients is the REINFORCE algorithm[78], which uses the cumulative reward $R_{t}=\sum^{T}_{t^{\prime}=t}\gamma^{t^{\prime}-t}r_{t^{\prime}}$ as an estimate for $Q^{\pi_{\theta}}(s_{t},a_{t})$ to update the policy parameters.

The traditional Actor-Critic method consists of two components: the Actor, which adjusts the parameters $\theta$ of the policy $\pi_{\theta}$; and the Critic, which adjusts the parameters $w$ of the state-action value function $Q^{\pi_{\theta}}_{w}$. Based on these two components, the corresponding update methods for the Actor and Critic can be obtained as follows:

The Actor-Critic method combines policy gradient methods with value function approximation. The Actor probabilistically selects actions, while the Critic, based on the action chosen by the Actor and the current state, evaluates the score of that action. The Actor then modifies the probability of choosing actions based on the Critic’s score. The advantage of such methods is that they allow for single-step updates, yielding low-variance solutions in continuous action spaces [79, 80]. However, the drawback is that, during the initial stages of learning, there is significant fluctuation due to the imprecise estimation by the Critic. Figure 3 illustrates the architecture of the Actor-Critic method, and for more details on SARL methods, refer to the review [9].

The Multi-agent system (MAS) evolved from Distributed Artificial Intelligence (DAI) [81]. The research aim is to address real-world problems that are large-scale, complex, real-time, and involve uncertainty. Modeling such problems with a single intelligent agent is often inefficient and contradictory to real-world conditions. MASs exhibit autonomy, distribution, coordination, self-organization, reasoning, and learning capabilities. Developed since the 1970s, research on MASs encompasses both building individual agents’ technologies, such as modeling, reasoning, learning, and planning, and technologies for coordinating the operation of multiple agents, such as interaction, coordination, cooperation, negotiation, scheduling, and conflict resolution. MASs find rapid and extensive applications in areas such as intelligent robotics, traffic control, distributed decision-making, software development, and gaming, becoming a tool for analyzing and simulating complex systems [15].

Depending on task characteristics, MASs can generally be classified into three settings: Fully Cooperative, Fully Competitive, and Mixed. In a Fully Cooperative setting, agents share a common goal, collaborating to accomplish specific tasks, such as the assembly of machine systems. In a Fully Competitive setting, agents’ goals conflict, where one agent’s gain results in another’s loss, like in a sumo wrestling match. Mixed settings involve agents forming multiple groups, with cooperation within groups and competition between them, resembling a soccer match where teammates cooperate while teams compete.

In MASs, different settings lead to variations in policy optimization goals and learned policies. In cooperative settings, agents must consider teammates’ policies, aiming for optimal coordination to avoid interference. In competitive settings, agents consider opponents’ policies, minimizing opponents’ gains while maximizing their own returns, optimizing their policies accordingly.

The symbol $\pi_{i}$, commonly used in RL, is employed here instead of the often-used symbol $\sigma_{i}$ in game theory to represent a probability distribution on the feasible action space. Additionally, in game theory, $-i$ usually denotes the set of all players except player $i$. Consequently, the expected utility of agent $i$ under a mixed policy profile $\bm{\pi}=(\pi_{i},\bm{\pi}_{-i})$ is given by

The form of the expected utility shows that, for agent $i$, the expected utility functions of a two-person constant-sum game and a two-player zero-sum game only differ by a constant $c$. Thus, under the same solution concept, there is no difference in the policies between the two types of games. For rational agents, the goal is often to maximize their utility, meaning maximizing their expected returns. Therefore, we need to introduce the definition of best response.

The best response intuitively means that, given the policies of other agents $\bm{\pi}_{-i}$, the agent $i$’s policy $\pi^{\star}_{i}$ maximizes its utility. Thus, if other agents’ policies are fixed, we can use RL to find the best response. Based on best responses, we can straightforwardly introduce the definition of Nash Equilibrium.

According to the above definition, in a Nash Equilibrium, unilateral deviations by individual agents do not increase their own utility, indicating that each rational agent has no incentive to deviate unilaterally from the equilibrium. Besides Nash Equilibrium, there are many other solution concepts in game theory, such as maximin policy and minimax policy.

The maxmin policy maximizes the agent $i$’s utility in the worst-case scenario (against the most malicious opponent). Therefore, adopting the maxmin policy ensures that the agent’s utility is not lower than the maxmin value. When making decisions solely to maximize their own returns without making any assumptions about other agents, the maxmin policy is a suitable and conservative choice. The "dual" policy to the maxmin policy is the minimax policy.

These two policies are often applied in constant-sum games. However, in cooperative settings, attention is given to how agents in the system choose policies through certain coordination means to achieve mutual benefits. Relevant equilibrium provides a reasonable policy coordination mechanism.

Correlated equilibrium describes a situation where, assuming two agents follow a correlated policy distribution, no agent can unilaterally change its current policy to obtain a higher utility. It should be noted that Nash Equilibrium implies that choices made by each agent are independent, i.e., the actions of agents are not correlated. Nash Equilibrium can be viewed as a special case of correlated equilibrium.

The concepts mentioned above assume that agents make decisions simultaneously. However, in some scenarios, there may be a sequence of decisions. In such cases, agents are defined as leaders and followers, where leaders make decisions first, enjoying a first-mover advantage, and followers make decisions afterward.

Multi-agent Learning (MAL) introduces machine learning techniques into the field of MASs, exploring the design of algorithms for adaptive agent learning and optimization in dynamic environments. The objective of Multi-agent Reinforcement Learning (MARL) (Figure 5) is to train multiple agents in a shared environment using RL methods to accomplish given tasks [56, 19].

Differing from modeling the multi-step decision-making process as a Markov Decision Process (MDP) in SARL, MARL is generally modeled as a Stochastic Game.

At each time step, agent $i\in\mathcal{N}$ is in state $s$, selects action $a^{i}\in\mathcal{A}^{i}$, forms the joint action $\bm{a}=\langle a^{1},\dots,a^{n}\rangle$, and executes it. The environment transitions to the next state $s^{\prime}\sim P(\cdot\mid s,\bm{a})$, and agent $i$ receives its own reward $R^{i}(s,\bm{a})$. Each agent optimizes its policy function $\pi_{i}:\mathcal{S}\rightarrow\Delta(\mathcal{A}^{i})$ to maximize its expected cumulative reward, expressed in the form of a state value function:

where the symbol $-i$ represents all agents except agent $i$. Unlike in SARL where agents only need to consider their own impact on the environment, in MASs, agents mutually influence each other, jointly make decisions, and simultaneously update policies. When the policies of other agents in the system are fixed, agent $i$ can maximize its own payoff function to find the optimal policy $\pi^{\ast}_{i}$ relative to the policies of other agents. In MARL, rationality and convergence are the primary evaluation metrics for learning algorithms.

From the above discussions, it is evident that in the process of MARL, each agent aims to increase its own utility. The learning goal in this case can be to maximize its own Q-function. Therefore, an intuitive learning approach is to construct an independent $Q$-function for each agent and update it according to the $Q$-learning algorithm:

Similar to the SARL algorithm, each agent selects the current action using a greedy policy:

In the above scenario, we assume that each agent can obtain global state information for decision-making. However, constrained by real conditions, individuals in MASs often can only obtain limited local observations. To address this characteristic, we generally model such decision processes as Partially-Observable Stochastic Games (POSG).

In POSG, each agent optimizes its policy $\pi_{i}:\Omega^{i}\rightarrow\Delta(\mathcal{A}^{i})$ to maximize its return. Although POSG is widely applied in real scenarios, theoretical results prove its problem-solving difficulty is NEXP-hard [84]. Fortunately, recent technologies, including centralized MARL training frameworks, utilizing recurrent neural networks to encode historical information [85], and agent communication [20], have alleviated this issue from different perspectives.

In cooperative tasks, POSG can be further modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [84]. The main difference lies in the fact that in Dec-POMDP, the reward functions for each agent are identical. Beyond the challenges of partial observability, MARL faces unique problems compared to SARL. In MARL, the state space becomes larger, and the action space experiences the "curse of dimensionality" problem as the number of agents grows [56]. Additionally, issues such as scalability, credit assignment, heterogeneity, and cooperative exploration have hindered the development of MARL [55].

IQL [86] is a typical algorithm based on DTDE, known for its scalability as it only considers the individual agent. However, due to the lack of consideration for information from other agents, agents often operate in a non-stationary environment, as defined in Section 2.1. To address the inefficiency brought by decentralized learning, optimistic and delayed optimization assign greater update weights to better $Q$ values, alleviating the non-stationarity issue. Techniques like recurrent neural networks can also mitigate these problems.

When updating policies using DQN, direct usage of data from experience replay buffer exacerbates non-stationarity. Importance sampling can be employed to alleviate this issue:

where $\theta_{i}$ is the $Q$ network parameters for agent $i$, $t_{c}$ is the current time, $t_{i}$ is the sample collection time, and $y_{i}^{Q}$ is the temporal difference target, calculated similarly to the single-agent case.

Here, any SARL algorithm can be used to train MASs. However, the complexity of the algorithm grows exponentially with the dimensions of states and actions [87]. This issue can be addressed through policy or value decomposition.

While decomposition methods alleviate the dimension explosion problem, CTCE struggles to evaluate the mutual influences among agents. Recently, attention has been given to the Centralized Training Decentralized Execution (CTDE) framework, especially in complex scenarios.

CTDE has found widespread applications in MASs, particularly excelling in certain complex scenarios. However, when dealing with heterogeneous MASs, CTDE may face challenges. Skill learning or grouping followed by local CTDE training are proposed solutions. Additionally, research has discussed centralized and decentralized aspects from various perspectives [88, 89].

Compared to SARL, real-world scenarios are often better modeled as MARL. However, the presence of multiple agents simultaneously updating their policies introduces more challenges. This section discusses key challenges in MARL, including non-stationarity, scalability, partial observability, and current solutions.

In a single-agent system, the agent only considers its own interaction with the environment, resulting in a fixed transition of the environment. However, in MASs, simultaneous policy updates by multiple agents lead to a dynamic target process, termed non-stationarity.

Non-stationarity indicates that, with simultaneous learning and updates by multiple agents, learning no longer adheres to Markovian properties. This issue is more severe in updates of $Q$ values, particularly methods relying on experience replay, where sampling joint actions in multi-agent environments is challenging. To address non-stationarity, solutions involve modeling opponents (teammates), importance sampling of replay data, centralized training, or incorporating meta-RL considering updates from other teammates.

In real-world scenarios, the number of agents in MASs is often substantial, such as in autonomous driving environments where there may be hundreds or even thousands of agents. Scalability becomes essential but is also a highly challenging aspect. Current methods to address scalability include parameter sharing, where homogeneous agents share neural networks, and heterogeneous agents undergo independent training [24]. Alternatively, techniques such as transfer learning or curriculum learning involve training initially with fewer agents and progressively scaling up to environments with a larger number of agents [60].

Due to sensor limitations and other factors, agents often struggle to obtain global states and can only access partial information. MARL is often modeled as Partially Observable Stochastic Games (POSG), where the environment, from an individual agent’s perspective, no longer adheres to Markovian properties, posing training challenges.

In cooperative MARL, where rewards and returns among agents are identical, all agents possess the same value function. We denote the global action value function as $Q_{\bm{\pi}}(s,\bm{a})$ and the state value function as $V_{\bm{\pi}}(s)$. It’s worth noting that, in MARL, the action value function and state value function of an agent depend on the policies of all agents:

Here, $\theta^{i}$ represents the policy parameters of agent $i$, and the differences in policies among agents mainly manifest in the differences in policy parameters. In the process of policy learning, our objective is to optimize policy parameters to maximize the objective function:

All agents share a common goal, optimizing their policy parameters $\theta^{i}$ to maximize the objective function $J$. Thus, the optimization objective can be expressed as follows:

For a specific agent, by performing gradient ascent, we maximize the objective function:

Here, $\alpha^{i}$ is the learning rate, and the stopping criterion is the convergence of the objective function. In the above equation, direct computation of the gradient is not feasible, and a value network is commonly employed to approximate the policy gradient.

Here, we focus on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm [28] (Figure 8). The algorithm’s pseudocode is provided in Algorithm 1. One one hand, each agent corresponds to a distributed deterministic policy, the Actor module $\pi_{i}(o^{i};\theta^{i})$, and decisions are made by agents based on their respective Actors. On the other hand, each agent is equipped with a centralized Critic network $Q_{i}(s,\bm{a};\phi^{i})$ to guide the updating and optimization of the Actor network. Specifically, MADDPG additionally maintains a target policy $\pi_{i}^{\prime}(o^{i};\theta^{i}_{-})$ for delayed updates to calculate the temporal difference target value for the Critic:

Furthermore, we use the Critic to guide the Actor with the following update:

MADDPG is a typical algorithm based on the CTDE framework. Training the critic network requires access to the environment’s global state information. In environments where global state information is unavailable, usually only the observation information $o_{i}$ of other agents is accessible. As the number of agents increases, training the Critic network becomes more challenging. Techniques such as attention mechanisms [93] can be employed for information aggregation to alleviate the computational cost arising from changes in quantity. Additionally, MADDPG can handle competitive settings. However, due to privacy concerns, it may not be possible to obtain the actions of other agents. In such cases, opponent modeling [59] can be used to estimate the action information of other agents.

While CTDE-based multi-agent policy gradient methods show promising performance in some application scenarios, and methods based on attention mechanisms, such as MAAC [93], can alleviate the curse of dimensionality to some extent, and methods based on counterfactual baselines can address the credit assignment problem to a certain extent. However, in complex cooperative scenarios, such as micro-management in StarCraft [33], these methods are often inefficient. On the contrary, MARL methods based on value decomposition have shown better performance [94]. In the following discussion, we will change the input of the value function from observation $o_{t}^{i}$ to trajectory $\tau_{t}^{i}=(o_{1},a_{1},...,a_{t-1},o_{t})\in\mathcal{T}^{i}$, thus partially alleviating the problems caused by partial observability. It is worth noting that this technique can also be used in policy-gradient-based methods mentioned above to improve coordination performance. Most value decomposition methods are based on the Individual-Global-Max (IGM) principle [83].

Three typical value decomposition methods that satisfy the IGM principle are as follows. VDN [30] represents the joint Q function as the sum of local Q functions:

During optimization, the temporal difference loss is computed for optimization:

where $y_{\text{tot}}=r+\gamma\max_{\bm{a}^{\prime}}Q_{\text{tot}}^{\text{QMIX},-}(\bm{\tau}^{\prime},\bm{a}^{\prime})$ and $Q_{\text{tot}}^{\text{QMIX},-}$ is the target Q network. Based on the expressiveness and simplicity of the mixing network, QMIX achieves superior results in various environments, and its pseudocode is shown in Algorithm 2.

The conditions satisfied by VDN and QMIX are sufficient but unnecessary conditions for the IGM property, leading to some shortcomings in their expressiveness. To further enhance the expressiveness of neural networks, QPLEX [95] proposed a new Duplex Dueling Multi-agent (DDMA) structure:

Here, $\lambda^{i}(\bm{\tau},\bm{a})$ is computed using a multi-head attention mechanism [96], obtaining the credit assignment for different advantage functions. QPLEX, with its expressiveness and network structure, achieves superior results in various environments.

In addition to the aforementioned methods based on policy gradient and value decomposition, some works combine the advantages of both to develop algorithms. One such algorithm is Decomposed Off-policy multi-agent Policy gradients (DOP) [97] (as shown in Figure 10), which introduces the idea of value decomposition into the multi-agent actor-critic framework and learns a centralized but decomposed critic. This framework decomposes the centralized critic into a weighted linear sum of individual critics with local actions as inputs. This decomposition structure not only achieves scalable learning of critics but also brings several benefits. It realizes feasible off-policy evaluation for stochastic policies, alleviates the centralized-decentralized inconsistency problem, and implicitly learns efficient multi-agent reward allocation. Based on this decomposition, DOP develops efficient off-policy multi-agent decomposed policy gradient methods for both discrete and continuous action spaces.

DOP’s specific contributions include: (1) To address the credit assignment problem in MASs and the scalability problem of learning a centralized critic, DOP uses linear value function decomposition to learn a centralized critic, enabling local policy gradients to be derived for each agent based on individual value functions. (2) DOP proposes off-policy MARL algorithms for both stochastic policy gradients and deterministic policy gradients, effectively handling tasks with discrete and continuous action spaces, respectively. (3) Considering the sample efficiency issue of policy gradient algorithms, DOP combines linear value function decomposition with tree-backup techniques, proposing an efficient off-policy critic learning method to improve the sample efficiency of policy learning. In addition to DOP, other work combines the advantages of value decomposition and policy gradient from different perspectives. VDAC [98] directly integrated policy gradient-like methods with value decomposition methods, proposing methods based on linear summation and mixed neural networks. IAC [99], based on MAAC, further optimizes the credit assignment between agents through value decomposition methods, achieving surprising coordination performance in various environments. FACMAC [100] proposed a decomposable joint policy gradient multi-agent algorithm and proposes the continuous environment multi-agent testing environment MAMuJoCo, FOP [101] designed a value decomposition and gradient decomposition method based on maximum entropy, RIIT [102] integrated some common techniques in current MARL into the coordination algorithms and proposes a new open-source algorithm framework PyMARL2 ^***https://github.com/hijkzzz/pymarl2.

RL methods strive to efficiently learn optimal policies, where high-quality training samples are crucial, and exploration plays a key role during the sampling process, constituting a vital aspect of RL [150]. Similar to exploration techniques widely used in single-agent scenarios, exploration in cooperative MARL has garnered attention. Initial algorithms like QMIX and MADDPG, designed for multi-agent tasks, lacked specific exploration policies and underperformed in certain complex scenarios. Subsequent works aim at addressing multi-agent exploration challenges. MAVEN [104] introduced a latent variable to maximize mutual information between this variable and generated trajectories, addressing the exploration limitations of value decomposition methods like QMIX. EITI and EDTI [105] proposed cooperative exploration methods by considering interactions among agents, demonstrating improved coordination in various environments. CMAE [107] uses constrained space selection to encourage exploration in regions with higher value, enhancing cooperative efficiency. EMC [106] extends single-agent, motion-based exploration methods to the multi-agent domain and proposes episodic-memory-based approaches to store high-value trajectories for exploration facilitation. Uneven [108] improves exploration efficiency by simultaneously learning multiple task sets, obtaining a multi-agent generalization feature to address relative overgeneralization in multi-agent scenarios. SMMAE [109] balances individual and team exploration to enhance coordination efficiency, achieving satisfactory results in complex tasks. Recently developed ADER [151] focused on studying the balance between exploration and exploitation in cooperative multi-agent tasks.

Additionally, various methods have explored multi-agent exploration from different perspectives, such as intrinsic reward exploration [152], decentralized multi-agent exploration in distributed scenarios [153, 154], information-based exploration methods [155], cooperative exploration under unknown initial conditions [156], exploration problems in multi-agent multi-armed bandit settings [157], exploration-exploitation balance in competitive multi-agent scenarios [158], structure-based exploration methods [159], optimistic exploration [160], and exploration in reward-sparse scenarios [161]. While these methods have shown promising results in diverse testing environments, developing test environments tailored for various exploration algorithms and proposing comprehensive theories for multi-agent exploration remain worthwhile areas for future research.

Communication plays a crucial role in MASs, enabling agents to share experiences, intentions, observations, and other vital information based on communication technology (Figure 11 [20]). Three factors typically characterize multi-agent communication: when, with which agents, and what type of information is exchanged. Early research focused on integrating communication with existing MARL to improve coordination efficiency. DIAL [110] introduced a simple communication mechanism where agents broadcast information to all teammates, facilitating end-to-end RL training. CommNet [162] proposed an efficient centralized communication structure where hidden layer outputs of all agents are collected and averaged to enhance local observations. VBC [111] adds a variance-based regularizer to eliminate noise components in information, achieving lower communication overhead and better performance than other methods. IC3Net [163], Gated-ACML [164], and I2C [112] learn gating mechanisms to decide when and with which teammates to communicate, reducing information redundancy. SMS [165] models multi-agent communication as a cooperative game, evaluating the importance of each message for decision-making and trimming channels with no positive gain. MASIA [115] introduced the concept of information aggregation, extracting information effectively for decision-making at the information receiver. Additionally, it designed an open-source offline communication dataset, demonstrating the efficiency and rationality of the proposed communication structure in various experiments.

Some works have focused on determining the content of communication. TarMAC [113] generate information by appending signatures, and the receiver use attention mechanisms to calculate the similarity between its local observation and the information signatures from different teammates. DAACMP [166], based on the MADDPG [28] framework, introduced a dual attention mechanism, showing that attention improves the coordination performance of MASs. NDQ [167] uses information-theoretic regularization to design two different information-theoretic regularization terms for optimizing the value function, achieving good coordination performance in locally observable scenarios. TMC [168] applied smoothing and action selection regularizers for concise and robust communication, adding received information as an incentive to individual value functions. MAIC [114] designed a decentralized incentive communication based on teammate modeling, demonstrating excellent communication efficiency in various task scenarios. Recent works, including CroMAC [169], AME[170], $\mathcal{R}$-MACRL [49], and MA3C [171] focus on communication policy robustness during deployment, addressing potential communication interference issues. While existing methods have demonstrated effectiveness in various task scenarios, the natural gap between multi-agent environments and human society poses challenges. Exploring ways to bridge this gap through technologies like large language models [172], promoting human-AI interaction, and conducting interpretable research on communication content are promising areas for future study.

Endowing agents with the ability to infer actions, goals, and beliefs of other agents in the environment is a key focus in MARL, particularly in cooperative tasks [59]. Effective and accurate modeling techniques enable agents to collaborate efficiently (Figure 12). One direct approach is employing Theory of Mind (ToM) from psychology, where ToMnet [116], and subsequent works like ToM2C [173], CTH [174], and [175], incorporate psychological theories into multi-agent tasks, enhancing teammate modeling and improving coordination capabilities. More details and development of ToM can be seen in [176].

Alternatively, some works employ different techniques to model other teammates. OMDDPG [117] uses Variational Autoencoders (VAEs) to infer the behavior of other agents based on local information. LIAM [118] extended this technique to locally observable conditions, developing an efficient teammate modeling technique. LINDA [177] mitigates local observability issues by considering perceived teammate information, assisting in enhancing MAS coordination across diverse tasks. MACC [121] uses local information learning for subtask representation to augment individual policies. MAIC [114] introduced a communication-based method for teammate modeling, extracting crucial information for efficient decision-making. LILI [119] considered agent modeling in robotic scenarios and developed algorithms based on high-level policy representation learning to promote coordination. SILI [178] further considered learning a smooth representation to model other agents based on LILI. Literature [179] empowered robots to understand their own behaviors and develop technologies to better assist humans in achieving rapid value alignment.

Imitation Learning (IL) [187] offers a method for agents to make intelligent decisions by mimicking human experts. In multi-agent scenarios, IL has advanced significantly. For example, [188] proposed to learn a probabilistic cooperative model in latent space from demonstration data. [189] extended this by using a hierarchical structure for high-level intention modeling. [190] employed a connection structure [191] to explicitly model coordination relationships. Multi-agent Inverse RL (IRL) has also been explored, with MAGAIL [122] extending adversarial generative techniques to MASs. MA-AIRL [123] developed a scalable algorithm for high-dimensional state spaces and unknown dynamic environments. [192] learns a reward function based on the representation space to represent the behavior of agents. CoDAIL [124] introduced a distributed adversarial imitation learning method based on correlated policies. DM² [125] recently presented a distributed multi-agent imitation learning method based on distribution matching. MIFQ [193] studied a decomposed multi-agent IRL method, demonstrating excellent performance in various scenarios.

Other research have explored multi-agent imitation learning from different perspectives, including applications in imitatiing driving [194], policy research in multi-agent IRL [195], multi-agent IRL without demonstration data [196], application to mean field game problems [197], design for mulit-agent systems in distributed constrained IRL [198], and asynchronous multi-agent imitation learning [199]. Despite progress in various aspects, there’s a need for research exploring large-scale scenarios, handling heterogeneous or suboptimal demonstration data, and applying multi-agent imitation learning to real-world scenarios like autonomous driving. Investigating these aspects and fostering coordination between AI and domain experts can drive significant advancements in multi-agent policy imitation.

Reinforcement learning, due to its need for continual interaction with the environment for policy optimization, often employs model-free algorithms, where agents use samples from interaction with the environment. However, these algorithms frequently suffer from low sample efficiency. Model-based RL, considered to have higher sample efficiency, typically involves learning the environment’s state transition function (often including the reward function), i.e., the world model. Subsequently, policy optimization is conducted based on the knowledge derived from the world model [67, 200], leading to a significant improvement in sample efficiency.

In recent years, model-based MARL has garnered attention and made progress [35]. Early efforts extended successful techniques in single-agent scenarios to multi-agent settings. For instance, MAMBPO [126] extended the MBPO method [201] to multi-agent tasks, constructing a model-based RL system based on the CTDE paradigm and a world model. This approach moderately enhances the system’s sampling efficiency. CTRL [202] and AORPO [127] further introduced opponent modeling techniques into model learning to acquire adaptive trajectory data, augmenting the original data. MARCO [203] learns a multi-agent world model and employs it to learn a centralized exploration policy to collect more data in high uncertainty areas, enhancing multi-agent policy learning. Some techniques extend widely used single-agent methods, such as Dreamer [204], to multi-agent tasks. MBVD [128] learns an implicit world model based on value decomposition, enabling agents to evaluate state values in latent space, providing them with foresight. MAMBA [129] enhances centralized training in cooperative scenarios using model-based RL. The study discovers that algorithm optimization can be achieved during training through a world model and, during execution, a new corresponding world model can be established through communication. VDFD [130] significantly improves the sample efficiency of MARL by developing techniques that decouple the learning of the world model.

Additionally, researchers explore the learning of world models in MASs from various perspectives, such as collision avoidance in distributed robot control [205], safety and efficiency improvement [206], agent interaction modeling [207], optimization of distributed network systems [208], model-based opponent modeling [120], efficiency improvement in multi-agent communication through model-based learning [209, 210, 211], model-based mean-field MARL [212], efficient multi-agent model learning [213], improving the learning efficiency of offline MARL through model learning [214], and applications of model-based MARL [215]. Although some of these methods have achieved certain results, the challenges posed by the curse of dimensionality with the increasing number of agents and the partial observability introduced by decentralized execution often hinder the development of model-based MARL algorithms. Exploring efficient methods tailored to the characteristics of MASs, such as subset selection [216], is an essential research direction.

Reinforcement learning faces the challenge of the curse of dimensionality in complex scenarios. To address this issue, researchers have proposed hierarchical RL (HRL) [217]. The primary goal of HRL is to decompose complex problems into smaller ones, solve each subproblem independently, and thus achieve the overall goal. In recent years, there have been advancements in multi-agent hierarchical RL [218]. FHM [131] introduced a feudal lord structure into the process of MARL. This method is designed for scenarios where agents have different task objectives but may not be suitable for cooperative tasks with shared objectives. To address the sparse and delayed reward issues in multi-agent tasks, [219] proposed a hierarchical version of the QMIX algorithm and a hierarchical communication structure. This method achieves excellent cooperation performance in various task scenarios but requires manual design of high-level actions. HSD [132] generates corresponding skills through an high-level macro policy, and this method is mainly trained through supervised learning. RODE [133] learns action representations and selects actions hierarchically to enhance cooperation performance in various scenarios. VAST [220] addresses the efficiency issues in large-scale multi-agent cooperation tasks through a hierarchical approach. ALMA [134] fully exploits the task structure in multi-agent tasks and uses high-level subtask decomposition policies and lower-level agent execution policies. HAVEN [135] introduces a bidirectional hierarchical structure for both inter-agent and intra-agent coordination to further improve multi-agent cooperation efficiency.

Apart from the mentioned works, some skill-based approaches in SARL have made progress [221] and have been applied to MARL. [222] considered integrating skills into MASs to improve overall performance. HSL [223] designs a method to extract heterogeneous and sufficiently diverse skills from various tasks for downstream task learning. ODIS [34] focuses on extracting skills from offline datasets. MASD [224] aims to discover skills beneficial for cooperation by maximizing the mutual information between the potential skill distribution and the combination of all agent states and skills. SPC [225] focuses on the automatic curriculum learning process of skill populations in MARL. [226] also explores learning to augment multi-agent skills. While these methods to some extent learn or use skills in MASs, the interpretability of the learned skills is currently lacking. Exploring ways to make the learned skills interpretable, such as bridging the gap between MASs and the human society through natural language [227], is a topic worth researching.

The interaction among agents is a focal point in the study of multi-agent problems. A coordination graph is a method that explicitly characterizes the interaction between agents by decomposing the multi-agent value function into a graph representation [228, 136]. In recent years, this approach has been widely applied in cooperative MARL. In coordination graph-based MARL tasks, nodes generally represent agents, and edges (hyper-edges) represent connected relationships between agents, constructing a reward function on the joint observation action space. A coordination graph can represent a high-order form of value decomposition among multiple agents. Typically, Distributed Constraint Optimization (DCOP) algorithms [229] are employed to find action selections that maximize the value. Agents can exchange information through connected edges over multiple rounds. DCG [137] introduces some deep learning techniques into coordination graphs, expanding them into high-dimensional state-action spaces. In complex task scenarios, such as SMAC, DCG can relieve the relative overgeneralization issues among multiple agents. However, DCG mainly focuses on pre-training on static and dense topological structures, exhibiting poor scalability and requiring dense and inefficient communication methods in dynamic environments.

A core issue in coordination graph problems is how to learn a dynamic and sparse graph structure that satisfies agent action selection. Sparse cooperative Q-function learning [230] attempts to learn sparse graph structures for value function learning. However, the graph structure learned by this method is static and requires substantial prior knowledge. Literature [231] proposed learning a minimal dynamic graph set for each agent, but the computational complexity exponentially increases with the number of agent neighbors. Literature [232] studies the expressiveness of some graph structures but mainly focuses on random topologies and stateless problems. Literature [141] proposed a new coordination graph testing environment, MACO, and introduced the Context-Aware Sparse Deep Coordination Graphs algorithm (CASEC). This algorithm effectively reduces the reward function evaluation error introduced during the graph construction process by learning induced representations and performs well in multiple environments. Subsequent work also enhances coordination from multiple perspectives, such as developing nonlinear coordination graph structures [233], self-organizing polynomial coordination graph structures [234], and so on.

On the other hand, some methods use techniques such as attention mechanisms to find implicit graph structures among multiple agents by cutting unnecessary connection structures through attention mechanisms. For instance, ATOC [140] learns the most essential communication structure through attention mechanisms. DICG [138] learns implicit coordination graph structures through attention mechanisms. MAGIC [139] improves the communication efficiency and teamwork of MASs through graph attention mechanisms. Although these methods can to some extent obtain or improve the interaction topology between agents, currently, these methods generally make progress only in scenarios with a small number of agents. The challenge of obtaining the optimal topology structure in large-scale and strongly interacting scenarios is a future research direction [235].

In addition to the extensively researched topics mentioned above, several other aspects are gradually being explored and uncovered in MARL. These include but are not limited to cooperative interpretability [142], developing multi-agent cooperation algorithms with decision-making order [236, 32, 143, 237], theoretical analysis of multi-agent cooperation [22, 238, 239], multi-agent multi-task (stage) cooperation learning [144], social dilemma problems in MASs [240], asynchronous multi-agent cooperation [241, 242], large-scale multi-agent cooperation [243], delayed reward in multi-agent cooperation [244], multi-agent cooperation in mixed action spaces [145], discovery of causal relationships in multi-agent cooperation [63], curriculum learning in multi-agent cooperation [146], research on teacher-student relationships in cooperative MARL [147], fairness in MASs [148], and entity-based multi-agent cooperation [149], among others.