Cooperative Reward Shaping for Multi-Agent Pathfinding

RL-based planners such as [19, 20, 21, 22], typically cast MAPF as a MARL problem to learn distributed policies for agents from partial observations. This method is particularly effective in environments populated by a large number of agents. Techniques like Imitation Learning (IL) often enhance policy learning during this process. Notably, the PRIMAL algorithm [20] utilizes the Asynchronous Advantage Actor Critic (A3C) algorithm and applies behavior cloning for supervised RL training using experiences from the centralized planner ODrM* [23]. However, the use of a centralized planner limits its efficiency, as solving the MAPF problem can be time-intensive, particularly in complex environments with a large number of agents.

In contrast, Distributed Heuristic Coordination (DHC) [24] and Decision Causal Communication (DCC) [25] algorithms do not require a centralized planner. Although guided by an individual agent’s shortest path, DHC innovatively incorporates all potential shortest path choices into the model’s heuristic input rather than obligating an agent to a specific path. Additionally, DHC collects data from neighboring agents to inform its decisions and employs multi-head attention as a convolution kernel to calculate interactions among agents. DCC is an efficient algorithm that enhances the performance of agents by enabling selective communication with neighbors during both training and execution. Specifically, a neighboring agent is deemed significant only if its presence instigates a change in the decision of the central agent. The central agent only needs to communicate with its significant neighbors.

The reward function significantly impacts the performance of RL algorithms. Researchers persistently focus on designing reward functions to optimize algorithm learning efficiency and agent performance. A previous study [26] analyzes the effect of modifying the reward function in Markov Decision Processes on optimal strategies, indicating that the addition of a transition reward function can boost the learning efficiency. In multi-agent systems, the aim is to encourage cooperation among agents through appropriate reward shaping methods, thereby improving overall system performance. [27] probes the enhancement of cooperative agent behavior within the context of a two-player Stag Hunt game, achieved through the design of reward functions. Introducing a prosocial coefficient, the study validates through experimentation that prosocial reward shaping methods elevate performance in multi-agent systems with static network structures. Moreover, [28] promotes cooperation among agents through the reward of an agent whose actions causally influence the behavior of other agents. The evaluation of causal impacts is achieved through counterfactual reasoning, with each agent simulating alternative actions at each time step and calculating their effect on other agents’ behaviors. Actions that lead to significant changes in the behavior of other agents are deemed influential and are rewarded accordingly.

Several studies, including [29, 17, 30], utilize the Shapley value decomposition method to calculate or redistribute each agent’s cooperative benefits. [30] confirms that if a transferable utility game is a convex game, the MARL reward redistribution, based on Shapley values, falls within the core, thereby securing stable and effective cooperation. Consequently, agents should maintain their partnerships or collaborative groups. This concept is the basis for a proposed cooperative strategy learning algorithm rooted in Shapley value reward redistribution. The effectiveness of this reward shaping method in promoting cooperation among agents, specifically within a basic autonomous driving scenario, is demonstrated in the paper. However, the process of calculating the Shapley value can be intricate and laborious. Ref. [29] aims to alleviate these computational challenges by introducing approximation of marginal contributions and employing Monte Carlo sampling to estimate Shapley values. Coordinated Policy Optimization (CoPO)[18] puts forth the concept of “cooperation coefficient”, which shapes the reward by taking a weighted average of an agent’s individual rewards and the average rewards of its neighboring agents, based on the cooperation coefficient. This approach proves that rewards shaped in this manner fulfill the Individual Global Max (IGM) condition. Findings from traffic simulation experiments further suggest that this method of reward shaping can significantly enhance the overall performance and safety of the system. However, this approach ties an agent’s rewards not merely to its personal actions but also to those of its neighbors. Such dependencies might compromise the stability during the training process and the efficiency of the converged strategy.

Consider a Markov process involving $n$ agents $\{A^{1},\dots,A^{n}\}:=\mathbb{A}$, represented by the tuple $(\mathcal{S},\mathcal{A},O,\mathcal{R},\mathcal{P},\gamma)$. $A^{i}$ represents agent $i$. At each time step $t$, $A^{i}$ chooses an action $a^{i}_{t}$ from its action space $\mathcal{A}^{i}$ based on its state $s^{i}_{t}\in\mathcal{S}^{i}$ and observation $o^{i}\in O$ according to its policy $\pi^{i}$. All $a^{i}_{t}$ form a joint action $\bar{a}_{t}=\{a^{1}_{t},\dots,a^{n}_{t}\}\in(\mathcal{A}^{1}\times\dots\times\mathcal{A}^{n}):=\mathcal{A}$, and all $s^{i}_{t}$ form a joint state $\bar{s}_{t}=\{s^{1}_{t},\dots,s^{n}_{t}\}\in(\mathcal{S}^{1}\times\dots\times\mathcal{S}^{n}):=\mathcal{S}$. For convenience in further discussions, agent’s local observations $o^{i}_{t}$ are treated as a part of the agent’s state $s^{i}_{t}$. Whenever a joint action $\bar{a}_{t}$ is taken, the agents acquire a reward $\bar{r}=\{r^{1}_{t},r^{2}_{t},...,r^{n}_{t}\}\in\mathcal{R}$, which is determined by the local reward function $r^{i}_{t}(\bar{s}_{t},\bar{a}_{t}):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ with respect to the joint state and action. The state transition function $\mathcal{P}(\bar{s}_{t+1}|\bar{s}_{t},\bar{a}_{t}):\mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ characterizes the probability of transition from the current state $\bar{s}_{t}$ to $\bar{s}_{t+1}$ under action $\bar{a}_{t}$. The policy $\pi^{i}(a^{i}_{t}|s^{i}_{t})$ provides the probability of $A^{i}$ taking action $a^{i}_{t}$ in the state $s^{i}_{t}$. $\bar{\pi}$ represents the joint policy for all agents. The action-value function is given by $Q^{i}_{\pi}(s^{i}_{t},a^{i}_{t})=\mathbb{E}_{\tau^{i}\sim\pi^{i}}[\sum_{t=0}^{T}\gamma^{t}r^{i}_{t}]$, where the trajectory $\tau^{i}=[s^{i}_{t+1},a^{i}_{t+1},s^{i}_{t+2},...]$ represents the path taken by the agent $A^{i}$. The state-value function is given by $V^{i}_{\pi}(s)=\sum_{a\in\mathcal{A}^{i}}\pi(a|s^{i}_{t})Q^{i}_{\pi}(s^{i}_{t},a)$ and the discounted cumulative reward is $J^{i}=\mathbb{E}_{s_{0}\sim\rho_{0}}V^{i}_{\pi}(s_{0})$, where $\rho_{0}$ represents the initial state distribution. For cooperative MARL tasks, the objective is to maximize the total cumulative reward $J^{tot}=\sum_{i=1}^{n}J^{i}$ for all agents.

This letter adopts the same definition of the multi-agent path-finding problem as presented in [24, 25]. Consider $n$ agents in an $w\times h$ undirected grid graph $G(V,E)$ with $m$ obstacles $\{B^{1},\dots,B^{m}\}$, where $V=\{v(i,j)|1\leq i\leq w,1\leq j\leq h\}$ is the set of vertices in the graph, and all agents and obstacles located within $V$. All vertices $v(i,j)\in V$ follow the 4-neighborhood rule, that is, $[v(i,j),v(p,q)]\in E$ for all $v(p,q)\in\{v(i,j\pm 1),v(i\pm 1,j)\}\cap V$. Each agent $A^{i}$ has its unique starting vertex $s^{i}$ and goal vertex $g^{i}$, and its position at time $t$ is $x^{i}_{t}\in V$. The position of the obstacle $B^{k}$ is represented as $b^{k}\in V$. At each time step, each agent can execute an action chosen from its action space $\mathcal{A}^{i}=\{\text{``Up'', ``Down'', ``Left'', ``Right'', ``Stop''}\}$. During the execution process, two types of conflict can arise: vertex conflict ($x^{i}_{t}=x^{j}_{t}$ or $x^{i}_{t}=b^{k}$) and edge conflict ($[x^{i}_{t-1},x^{i}_{t}]=[x^{j}_{t},x^{j}_{t-1}]$). If two agents conflict with each other, their positions remain unchanged. The subscript $t$ for all the aforementioned variables can be omitted as long as it does not cause ambiguity. The goal of the MAPF problem is to find a set of non-conflicting paths $\bar{P}=\{P^{1},P^{2},...,P^{n}\}$ for all agents, where the agent’s path $P^{i}=[s^{i},x^{i}_{1},\dots,x^{i}_{t},\dots,g^{i}]$ is an ordered list of $A^{i}$’s position. Incorporating the setup of multi-agent reinforcement learning, we design a reward function for the MAPF task, as detailed in Table I. The design of the reward function basically follows [24, 25], with slight adjustments to increase the reward for the agent moving towards the target to better align with our reward shaping method.

In the MAPF task, the policies trained using IQL often result in scenarios where one agent blocks the path of other agents or collides with them. To enhance cooperation among agents within the IQL framework, a feasible approach is reward shaping. Reward shaping involves meticulously designing the agents’ reward functions to influence their behavior. For example, when a certain type of behavior needs to be encouraged, a higher reward function is typically assigned to that behavior. Thus, to foster cooperation among agents in the MAPF problem, it is necessary to design a metric that accurately evaluates the collaboration of agents’ behavior and incorporate this metric into the agents’ rewards. Consequently, as each agent maximizes its own reward, it will consider the impact of its action on other agents, thereby promoting cooperation among agents and improving overall system efficiency.

Let $I^{i}_{c}(\bar{s}_{t},\bar{a}_{t})$ be a metric that measures the cooperativeness of the action $a^{i}_{t}$ of agent $A^{i}$. To better regulate the behavior of the agent, [27] introduces a cooperation coefficient $\alpha$ and shapes the agent’s reward function in the following form:

where $\alpha$ describes the cooperativeness of the agent. When $\alpha=0$, the agent completely disregards the impact of its actions on other agents, acting entirely selfishly and when $\alpha=1.0$, the agent behaves with complete altruism. For agent $A^{i}$, an intuitive approach to measure the cooperativeness of $A^{i}$’s behavior is to use the average reward of all agents except $A^{i}$:

where $A^{-i}$ denotes the set of all agents except $A^{i}$, and $|A^{-i}|$ represents the number of agents in $A^{-i}$. This reward shaping method is equivalent to the neighborhood reward proposed in [18] when $d_{n}$, the neighborhood radius of the agent, approaches infinity. The physical significance of Eq. (2) is as follows: If the average reward of the agents other than $A^{i}$ is relatively high, it indicates that $A^{i}$’s actions have not harmed the interests of other agents. Hence, $A^{i}$’s behavior can be considered as cooperative. Conversely, if the average reward of the other agents is low, it suggests that $A^{i}$’s behavior exhibits poor cooperation.

However, $I^{i}_{c}(\bar{s}_{t},\bar{a}_{t})$ in Eq. (2) is unstable, which is not only related to $a^{i}_{t}$ but is also strongly correlated with the actions $a^{-i}=\{a^{j}|A^{j}\in\mathbb{A},j\neq i\}$ of other agents. Appendix B-A provides specific examples to illustrate this instability. Within the IQL framework, this instability in the reward function makes learning of Q-values challenging and can even prevent convergence. To address this issue, this letter proposes a new metric to assess the cooperativeness of agent behavior. The specific form of this metric is as follows:

Here $\bar{a}_{t}=\{a^{i}_{t},a^{-i}_{t}\}$. The use of the $\max$ operator in Eq. (3) eliminates the influence of $a^{-i}_{t}$ on $I^{i}_{c}$ while reflecting the impact of $a^{i}_{t}$ on other agents. The term $\max_{a^{-i}}\sum_{j\in A^{-i}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{-i}\})$ represents the maximum reward that all the agents except $A^{i}$ can achieve under the condition of $\bar{s}_{t}$ and $a^{i}_{t}$, whereas the actual value of $\sum_{j\in A^{-i}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{-i}_{t}\})$ is determined by $a^{-i}_{t}$ when $a^{i}_{t}$ is given. Accordingly, it holds true under any circumstances that $\sum_{j\in A^{-i}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{-i}_{t}\})\leq\max_{a^{-i}}\sum_{j\in A^{-i}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{-i}\})$. The complete reward shaping method is then as follows:

Fig. 3 gives an one step example to illustrate how this reward-shaping approach Eq. (4) is calculated and promotes inter-agent cooperation when $\alpha=\frac{1}{2}$. There are two agents, $A^{1}$ and $A^{2}$, along with some obstacles. Without considering $A^{2}$, $A^{1}$ has two optimal actions: “Up” and “Right”. Fig. 3 also illustrates the potential rewards that $A^{2}$ may receive when $A^{1}$ takes different actions. Table II presents the specific reward values for each agent in this example. According to the reward shaping method proposed in Eq. (4), choosing the “Right” action could yield a higher one-step reward, and in such case, $A^{2}$ should also take action“Right”. Appendix B-B provides another analysis of this example from the vantage point of the cumulative rewards of the agent, further elucidating how the reward shaping method Eq. (4) facilitates cooperation among agents.

This section provides a detailed analysis of how reward shaping method Eq. (4) influences agent behavior. For ease of discussion, the subsequent analysis will be conducted from the perspective of $A^{i}$.

In the MAPF problem, the actions of agents are tightly coupled. The action of $A^{i}$ may impact $A^{j}$, and subsequently, the action of $A^{j}$ may also affect $A^{k}$. Thus, the action of $A^{i}$ indirectly affect $A^{k}$. This coupling makes it challenging to analyze the interactions among multiple agents. To mitigate this coupling, we consider all agents in $A^{-i}$ as a single virtual agent $\tilde{A}^{-i}$. The action of $\tilde{A}^{-i}$ is $a^{-i}$ and its reward $r^{-i}$ is their average reward $\frac{1}{|A^{-i}|}\sum_{j\in A^{-i}}r^{j}$. The use of average here ensures that $A^{i}$ and $\tilde{A}^{-i}$ are placed on an equal footing. The virtual agent $\tilde{A}^{-i}$ must satisfy the condition that no collisions occur between the agents constituting $\tilde{A}^{-i}$. It is important to note that the interaction between $A^{i}$ and $\tilde{A}^{-i}$ is not entirely equivalent to the interaction between $A^{i}$ and agents in $A^{-i}$. This is because when considering the agents in $A^{-i}$ as $\tilde{A}^{-i}$, all robots within $\tilde{A}^{-i}$ fully cooperate. In contrast, treating these agents as independent individuals makes it difficult to ensure full cooperation. Nonetheless, $\tilde{A}^{-i}$ can still represent the ideal behavior of agents in $A^{-i}$. Therefore, analyzing the interaction between $A^{i}$ and $\tilde{A}^{-i}$ can still illustrate the impact of reward shaping on agent interactions. Consider the interaction between $A^{i}$ and $\tilde{A}^{-i}$ in the time period $[t_{s},t_{e}]$. $\mathcal{Q}_{\pi}^{i}(\bar{s},a^{i})=\mathbb{E}_{\tau}\left[\sum\limits_{t=t_{s}}^{t_{e}}\gamma^{t-t_{s}}\tilde{r}^{i}(\bar{s}_{t},a^{i}_{t})\bigg{|}\bar{s}_{t_{s}}=\bar{s},a^{i}_{t_{s}}=a^{i}\right]$ and $\mathcal{Q}_{\pi}^{-i}(\bar{s},a^{-i})=\mathbb{E}_{\tau}\left[\sum\limits_{t=t_{s}}^{t_{e}}\gamma^{t-t_{s}}\tilde{r}^{-i}(\bar{s}_{t},a^{-i}_{t})\bigg{|}\bar{s}_{t_{s}}=\bar{s},a^{-i}_{t_{s}}=a^{-i}\right]$ are the cumulative reward of $A^{i}$ and $\tilde{A}^{-i}$. $Q^{tot}(\bar{s},\bar{a})=\mathbb{E}_{\tau}\left[\sum\limits_{t=t_{s}}^{t_{e}}\gamma^{t-t_{s}}(r^{i}+r^{-i})\bigg{|}\bar{s}_{t_{s}}=\bar{s},\bar{a}_{t_{s}}=\bar{a}\right]$ is the cumulative reward for both $A^{i}$ and $\tilde{A}^{-i}$. The optimal policies $\pi^{i}_{*}=\operatorname{argmax}_{\pi}\mathcal{Q}_{\pi}^{i}(\bar{s},a^{i})$, $\pi^{-i}_{*}=\operatorname{argmax}_{\pi}\mathcal{Q}_{\pi}^{-i}(\bar{s},a^{-i})$, and $\bar{\pi}_{*}=\operatorname{argmax}_{\pi}Q^{tot}_{\pi}(\bar{s},\bar{a})$. $A^{i}$ and $\tilde{A}^{-i}$ will select their actions according to $a^{i}_{t}=\operatorname{argmax}_{a^{i}}\mathcal{Q}_{\pi}^{i}(\bar{s}_{t},a^{i})$ and $a^{-i}_{t}=\operatorname{argmax}_{a^{-i}}\mathcal{Q}_{\pi}^{-i}(\bar{s}_{t},a^{-i})$. We hope that $a^{i}_{t}$ and $a^{-i}_{t}$ will collectively maximize $Q^{tot}(\bar{s},\bar{a})=Q^{tot}(\bar{s},\{a^{i}_{t},a^{-i}_{t}\})$. Specifically,

That is, $\mathcal{Q}_{\pi_{*}^{i}}^{i}$, $\mathcal{Q}_{\pi_{*}^{-i}}^{-i}$, and $Q^{tot}_{\bar{\pi}_{*}}(\bar{s},\bar{a})$ satisfy the Individual-Global-Max (IGM) condition. The definition of the IGM condition can be found in [31]. We introduce the following two assumptions to facilitate the analysis.

Here, $\tilde{A}^{-i}$ not reaching destination means $\forall j\in A^{-i}$, $A^{j}$ has not reached its destination. Based on the assumptions, it can be proven that:

Theorem 1 demonstrates that the optimal policy, trained using the reward function given by Eq. (4), maximizes the overall rewards of $A^{i}$ and the virtual agents $\tilde{A}^{-i}$, rather than selfishly maximizing its own cumulative reward. This implies that $A^{i}$ will actively cooperate with other agents, thus improving the overall efficiency of the system. It should be noted that the impact of $\alpha$ on the agent’s behavior is complex. The choice of $\alpha=\frac{1}{2}$ here is a specific result derived from using the virtual agent $\tilde{A}^{-i}$. Although this illustrates how the reward shaping method in Eq. (4) induces cooperative behavior in agents, it does not imply that $\frac{1}{2}$ is the optimal value of $\alpha$ in all scenarios.

Theorem 1 illustrates that the reward shaping method Eq. (4) can promote cooperation among agents in MAPF tasks. However, calculating Eq. (3) requires traversing the joint action space of $A^{-i}$. For the MAPF problem where the action space size for each agent is 5, the joint action space for $n$ agents contains up to $5^{n}$ states, significantly reducing the computational efficiency of the reward shaping method.

Therefore, we must simplify the calculation of Eq. (3). Before approximating Eq. (3), we first define the neighbors of an agent in the grid map as follows: $A^{j}$ is considered a neighbor of $A^{i}$ if the Manhattan distance between them is no more than $d_{n}$, where $d_{n}$ is the neighborhood radius of the agent. If the Manhattan distance between two agents is no more than 2, a conflict may arise between them in a single step. Therefore, we choose $d_{n}=2$, which maximally simplifies interactions between agents while adequately considering potential collisions. Let $\mathcal{N}_{i}$ denote the set of neighboring agents of $A^{i}$, and $|\mathcal{N}_{i}|$ represent the number of neighbors of $A^{i}$. Fig. 4 illustrates the neighbors of $A^{i}$ when $d_{n}=2$.

Considering that in MAPF tasks, the direct interactions among agents are constrained by the distances between them, the interactions between any given agent and all other agents can be simplified to the interactions between the agent and its neighbors. That is:

where $a^{\mathcal{N}_{i}}$ represents the joint actions of all neighbors of $A^{i}$. Next, we approximate $\max_{a^{\mathcal{N}_{i}}}\frac{1}{|\mathcal{N}_{i}|}\sum_{j\in\mathcal{N}_{i}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{\mathcal{N}_{i}}\})$ using $\frac{1}{|\mathcal{N}_{i}|}\sum_{j\in\mathcal{N}_{i}}\max_{a^{j}}r^{j}(\bar{s}_{t},\{a^{i}_{t},a^{j}\})$. This approximation implies that we consider the interaction between agent $A^{i}$ and one of its neighbors $A^{j}$ at a time. The computational complexity of the approximate $I_{c}$ is much lower than the original complexity.

To approximate $\bar{s}_{t}$, we posit that when $A^{i}$ can observe its two-hop neighbors, its observation is considered sufficient, and its state $s^{i}_{t}$ can approximate $\bar{s}_{t}$ to a certain extent, i.e., $\tilde{r}^{i}(\bar{s}_{t},a^{i}_{t})=\tilde{r}^{i}(s^{i}_{t},a^{i}_{t})$. When $d_{n}=2$, the minimum observation range includes all grids within a Manhattan distance of no more than $2\cdot d_{n}=4$ from the agent. For convenience, a $9\times 9$ square is chosen as the agent’s field of view (FOV), consistent with the settings in [24] and covering the agent’s minimum observation range. Consequently, the final reward shaping method is given by:

This reward shaping method possesses the following characteristics:

1. 1.

   $\tilde{r}^{i}$ for $A^{i}$ is explicitly dependent on $a^{i}$ and is explicitly independent of $a^{j}$ for $j\neq i$. This property improves the stability of the agent’s reward function and facilitates the convergence of the network, as shown in Fig. 5.

2. 2.

   This method of reward shaping is applicable to MAPF scenarios where the relationship among agents’ neighbors is time-varying.

3. 3.

   This method, which considers the interaction between just two agents at a time, dramatically reduces the computational difficulty of the reward.

For Eq. (5), it is necessary to adjust $\alpha$ so that the agent can balance its own interests with the interests of other agents. To find an appropriate cooperation coefficient $\alpha$, it is advisable to examine how the policies trained under different $\alpha$ perform in practice and then employ a gradient descent algorithm based on the performance of the policy to optimize $\alpha$. The cumulative reward $J$ of the agent often serves as a performance metric in RL. However, in environments with discrete action and state spaces, the cumulative reward $J$ is non-differentiable with respect to the $\alpha$, presenting a significant challenge for adjusting $\alpha$. Some work employs zero-order optimization of stochastic gradient estimation to handle non-differentiable optimization problems, that is, estimating the gradient by finite differences of function values. We also use differences in the cumulative reward of the agent to estimate the gradient of the cooperation coefficient $\alpha$:

where $\epsilon$ represents the maximum step size of the differential. If the update step length is too small, the updates are halted. To expedite the training process, a method of fine-tuning the network is employed. That is, after each adjustment of $\alpha$, the network is not trained from the initial state. Instead, fine-tuning training is performed based on the optimal policy network obtained previously. This training method improves the speed of the training process.

Following several rounds of network fine-tuning and updates to the cooperation coefficient $\alpha$, a value of $\alpha=0.1675$ and strategy $\pi$ are obtained. Upon obtaining the final $\alpha$ and $\pi$, the CoRS-DHC-trained policy is compared with the original DHC algorithm-trained policy to assess the effect of the reward shaping method on performance improvement. For a fair comparison, the DHC and CoRS-DHC algorithms are tested on maps of different scales ($40\times 40$ and $80\times 80$) with varying agent counts $\{4,8,16,32,64\}$. Recognizing the larger environmental spatial capacity of the $80\times 80$ map, a scenario with 128 agents is also introduced for additional insights. Each experimental scenario includes 200 individual test cases, maintaining a consistent obstacle density of 0.3. The maximum time steps for the $40\times 40$ and $80\times 80$ maps are 256 and 386, respectively.

From the experimental results illustrated in Fig. 6, it is evident that our proposed CoRS-DHC significantly outperforms the existing DHC algorithm in all test sets. In high agent density scenarios, such as $40\times 40$ with 64 agents and $80\times 80$ with 128 agents, our CoRS-DHC improves the pathfinding success rate by more than 20% compared to DHC. Furthermore, as shown in Table III, CoRS effectively reduces the number of steps required for all agents to reach their target spot in various test scenarios. Although the DHC algorithm incorporates heuristic functions and inter-agent communication, it still struggles with cooperative issues among agents. In contrast, our CoRS-DHC promotes inter-agent collaboration through reward shaping, thereby substantially enhancing the performance of the DHC algorithm in high-density scenarios. Notably, this significant improvement was achieved without any changes to the algorithm’s structure or network scale, underscoring the effectiveness of our approach.

Additionally, the policy trained through CoRS-DHC is compared with other advanced MAPF algorithms. The current SOTA algorithm, DCC[25], which is also based on reinforcement learning, is selected as the main comparison object, with the centralized MAPF algorithm ODrM*[6] and PRIMAL[20] (based on RL and IL) serving as references. DCC is an efficient model designed to enhance agent performance by training agents to selectively communicate with their neighbors during both training and execution stages. It introduces a complex decision causal unit to each agent, which determines the appropriate neighbors for communication during these stages. Conversely, the PRIMAL algorithm achieves distributed MAPF by imitating ODrM* and incorporating reinforcement learning algorithms. ODrM* is a centralized algorithm designed to generate optimal paths for multiple agents. It is one of the best centralized MAPF algorithms currently available. We use it as a comparative baseline to show the differences between distributed and centralized algorithms. The experimental results are demonstrated in Fig. 7:

The experimental results indicate that our CoRS-DHC algorithm consistently exceeds the success rate of the DCC algorithm in the majority of scenarios. Additionally, aside from the $40\times 40$ grid with 64 agents, the makespan of the policies trained by the CoRS-DHC algorithm is comparable to or even shorter than that of the DCC algorithm across other scenarios. These results clearly demonstrate that our CoRS-DHC algorithm achieves a performance comparable to that of DCC. However, it should be noted that DCC employs a significantly more complex communication mechanism during both training and execution, while our CoRS algorithm only utilizes simple reward shaping during the training phase. Compared to PRIMAL and DHC, CoRS-DHC exhibits a remarkably superior performance.