Learning Efficient Multi-Agent Cooperative Visual Exploration

In classical visual exploration solutions, an agent first locates its position and re-constructs the 2D map based on its sensory signals, which is formulated as Simultaneous Localization and Mapping (SLAM) (Fuentes-Pacheco et al., 2015). Then a search-based planning algorithm will be adopted to generate valid exploration trajectories. Representative variants include frontier-based methods (Umari & Mukhopadhyay, 2017; Yamauchi, 1997a; Yu et al., 2021b), which always choose navigation targets from the explored region, and sampling-based methods (Li, 2020), which generate goals via a stochastic process. In addition to the expensive search computation for planning, these methods do not involve learning and thus have limited representation capabilities for particularly challenging scenarios. Hence, RL-based methods have been increasingly popular for their training flexibility and strong expressiveness power. Early methods simply train navigation policies in a purely end-to-end fashion (Chen et al., 2019; Jain et al., 2019) while recent works start to incorporate the inductive bias of a spatial map structure into policy representation by introducing a differentiable spatial memory (Henriques & Vedaldi, 2018; Mousavian et al., 2019; Parisotto & Salakhutdinov, 2018), semantic prior knowledge (Liu et al., 2021b) or learning a topological scene graph (Bhatti et al., 2016; Chaplot et al., 2020c; Yang et al., 2018).

The Active Neural SLAM (ANS) method (Chaplot et al., 2020a) is the state-of-the-art framework for single-agent visual exploration, which takes advantage of both planning-based and RL-based techniques via a modular design (details in Sec. 3.2). There are also follow-up enhancements based on the ANS framework, such as improving map reconstruction with occupancy anticipation (Ramakrishnan et al., 2020) and incorporating semantic signals into the reconstructed map for semantic exploration (Chaplot et al., 2020b). Our MAANS can be viewed as a multi-agent extension of ANS with a few multi-agent-specific components.

There have been works extending planning-based visual exploration solutions to the multi-agent setting by introducing handcraft planning heuristics over a shared reconstructed 2D map (Čáp et al., 2013; Cohen, 1996; Desaraju & How, 2011; Hu et al., 2020; Nazif et al., 2010; Patel et al., ; Wurm et al., 2008). However, due to the lack of learning, these methods may have the limited potential of capturing non-trivial multi-agent interactions in challenging domains. By contrast, multi-agent reinforcement learning (MARL) has shown its strong performances in a wide range of domains (Nguyen et al., 2020), so many works have been adopting MARL to solve challenging cooperative problems. Representative works include value decomposition for approximating credit assignment (Rashid et al., 2018; Sunehag et al., 2018), learning intrinsic rewards to tackle sparse rewards (Iqbal & Sha, 2019b; Liu et al., 2021a; Wang* et al., 2020) and curriculum learning (Long et al., 2020; Wang et al., 2020).

However, jointly optimizing multiple policies makes multi-agent RL training remarkably more challenging than its single-agent counterpart. Hence, these end-to-end RL methods either focus on much simplified domains, like grid world or particle world (Wakilpoor et al., 2020), or still produce poor exploration efficiency compared with classical planning-based solutions. Our MAANS framework adopts a modular design and is the first RL-based solution that significantly outperforms classical planning-based baselines in a photo-realistic physical environment.

Finally, we remark that MAANS utilizes a centralized global planner MSP, which assumes perfect communication between agents. There are also works on multi-agent cooperation with limited or constrained communication (Jiang & Lu, 2018; Peng et al., 2017; Sukhbaatar et al., 2016; Foerster et al., 2016; Jain et al., 2019; Wang et al., 2021; Zhu et al., 2021), which are parallel to our focus.

There has been rich literature in deep learning studying representation learning over an arbitrary number of input entities in deep learning (Zhang et al., 2019a; b). In MARL, the self-attention mechanism (Vaswani et al., 2017) has been the most popular policy architecture to tackle varying input sizes (Duan et al., 2017; Jiang et al., 2018; Ryu et al., 2020; Wang et al., 2018) or capture high-order relations (Iqbal & Sha, 2019a; Malysheva et al., 2018; Yang et al., 2018; Zambaldi et al., 2018). A concurrent work (Wang et al., 2021) also considers the zero-shot team-size adaptation in the photo-realistic environment by learning a simple attention-based communication channel between agents. By contrast, our works develop a much expressive network architecture, Spatial-TeamFormer, which adopts a hierarchical self-attention-based architecture to capture both intra-agent and spatial relationships and results in substantially better empirical performance (see Section 5.4.2). Besides, parameter sharing is another commonly used technique in MARL for team-size generalization, which has been also shown to help reduce nonstationarity and accelerate training (Chu & Ye, 2017; Terry et al., 2020). Our work follows this paradigm as well.

We consider a multi-agent coordination indoor active SLAM problem, in which a team of agents needs to cooperatively explore an unknown indoor scene as fast as possible. At each timestep, each agent performs an action among Turn Left, Turn Right and Forward, and then receives an RGB image through a camera and noised pose change through a sensor, which is provided from the Habitat environment. We consider a decision-making setting by assuming perfect communication between agents. The objective of the task is to maximize the accumulative explored area within a limited time horizon.

The ANS framework (Chaplot et al., 2020a) consists of 4 parts: a neural SLAM module, a RL-based global planner, a planning-based local planner and a local policy. The neural SLAM module, which is trained by supervised learning, takes an RGB image, the pose sensory signals, and its past outputs as inputs, and outputs an updated 2D reconstructed map and a current pose estimation. Note that in ANS, the output 2D map only covers a neighboring region of the agent location and always keeps the agent at the egocentric position. For clarification, we call this raw output map from the SLAM module a agent-centric local map.

The global planner in ANS takes in an augmented agent-centric local map, which includes channels indicating explored regions, unexplored regions and obstacles and the history trajectory, as its input, and outputs two real numbers from two Gaussian distributions denoting the coordinate of the long-term goal. This global planner is parameterized as a CNN policy and trained by the PPO algorithm (Schulman et al., 2017). The local planner performs classical planning, i.e., Fast Marching Method (FMM) (Sethian, 1996), over the agent-centric local map towards a given long-term goal, and outputs a trajectory of short-term sub-goals. Finally, the local policy produces actions given an RGB image and a sub-goal and is trained by imitation learning.

With the advantage of the modeling capability of arbitrarily complex strategy in RL, an RL-based global planner which determines the global goals encourages exploration faster. To apply RL training, we model the problem as a decentralized partially observable Markov decision process (Dec-POMDP). Dec-POMDP is parameterized by $\langle S,A,O,R,P,n,\gamma,h\rangle$. $n$ is the number of agents. $S$ is the state space, $A$ is the joint action space. $o^{(i)}=O(s;i)$ is agent $i$’s observations at state $s$. $P(s^{\prime}|s,a)$ defines the transition probability from state $s$ to state $s^{\prime}$ via joint action $a$. $R(s,A)$ is the shared reward function. $\gamma$ is the discount factor. The objective function is $J(\theta)=\mathbb{E}_{a,s}[\sum_{t}\gamma^{t}R(s^{t},a^{t})]$. In this task, the policy $\pi_{\theta}$ generates a global goal for each agent every decision-making step. The shared reward function is defined as the accumulative environment reward every global goal planning step.

Multi-agent Spatial Planner (MSP) is the core component in MAANS, which could perform planning for an arbitrary number of agents. The full workflow of MSP is shown in Fig. 2. MSP first applies a weight-shared CNN feature extractor to extract spatial feature maps from each agent’s local navigation trajectory and then fuses team-wise information with hierarchical transformer-based network architecture, Spatial-TeamFormer. Finally, an action generator will generate a spatial global goal based on the features from Spatial-TeamFormer. Suppose there are a total of $N$ agents and the current decision-making agent has ID $k$. We will describe how agent $k$ generates its long-term goal via the 3 parts in MSP in the following content. Note that due to space constraints, we only present the main ideas while more computation details can be found in Appendix A.4.

We develop a map refiner to ensure all the maps from the neural SLAM module are within the same coordinate system. The workflow is shown as the blue and green part in Fig. 3. The map refiner first composes all the past agent-centric local maps to recover the agent-centric global map. Then, we transform the coordinate system based on the pose estimates to normalize the global maps from all the agents w.r.t. the same coordinate system. Note that when an agent explores the border of the house, the agent-centric local map often covers a large portion of invisible region. As a result, the normalized global map will accordingly contain a large unexplorable boundary surrounding the actual explorable house region. To ensure the feature extractor in MSP concentrates only on the viable part and also induce a more focused spatial action space, we crop the unexplorable boundary of the normalized map and enlarge the house region as our final refined map.

The local planner from ANS plans sub-goals on the agent-centric local map, while in our setting, we can also leverage the information from other agents to plan over a more accurate map. The diagram of map merger is shown in Fig. 3. After obtaining $N$ enlarged global maps via the map refiner, the map merger simply integrates all these maps by applying a max-pooling operator for each pixel location. That is, for each pixel in the merged global map, the probability of it being an obstacle is the maximum value at that pixel over all individual enlarged global maps. We remark that the artificial merged global map is only utilized in the local planner, but not in the global planner MSP. We empirically observe that having a coarse merged map produces better short-term local goal while such an artificial map is not sufficient for accurate global planning. (see Section 5.4.2)

The common training paradigm for visual exploration is multi-task learning, i.e., at each training episode, a random training scene or team size is sampled and all collected samples are aggregated for policy optimization (Chaplot et al., 2020a; Chen et al., 2019). However, we empirically observe that different Habitat scenes and team sizes may lead to drastically different exploration difficulties. During training, gradients from different configurations may negatively impact each other. Similar observations have been also reported in the existing literature (Hessel et al., 2019; Teh et al., 2017). We use policy distillation to tackle this problem. Therefore, we adopt a two-phase distillation-based solution: in the first phase, we train separate policies for representative training scenes with a fixed team size, i.e., we choose $N=2$ in our experiments in the second phase, we learn another policy with $N=2$ agents to distill the collection of pretrained policies over different training scenes and directly measure the generalization ability of this distillation policy to novel scenes and different team sizes. More specifically, for the $i$-th training scene, we first learn a specialized teacher policy $\pi(g,x,y|s,\theta_{i})$ given state $s$ with parameter $\theta_{i}$, where $g$ denotes the region output and $(x,y)$ is the point head output. Then we train another distillation policy $\pi(g,x,y|s,\theta)$ by simply running a dagger-style imitation learning, i.e., randomly rollout trajectories w.r.t. the distillation policy $\pi(s,\theta)$ and imitate the output from the specific teacher policy. Since the region action $g$ is discrete, we adopt a KL-divergence-based loss function while for the continuous point action $(x,y)$, a squared difference loss between the teacher policy and distillation policy is optimized.

We adopt scene data from the Gibson Challenge dataset (Xia et al., 2018b) while the visual signals and dynamics are simulated by the Habitat simulator (Savva et al., 2019). Although Gibson Challenge dataset provides $72$ training and $14$ validation scenes, we discard scenes that are not appropriate for our task, such as scenes that have large disconnected regions or multiple floors so that the agents are not possible to achieve 90% coverage of the entire house. Then we categorize the remaining scenes into $23$ training scenes and $10$ testing scenes. We consider $N=2,3,4$ agents in our experiments. Every RL training is performed with $10^{4}$ training episodes over 3 random seeds. Each evaluation score is expressed in the format of “mean (standard deviation)”, which is averaged over a total of 300 testing episodes, i.e., 100 episodes per random seed. More details are deferred to Appendix E.

We take 3 metrics to examine the exploration efficiency:

1. 1.

   Coverage: Coverage represents the ratio of areas explored by the agents to the entire explorable space in the indoor scene at the end of the episode. Higher Coverage implies more effective exploration.

2. 2.

   Steps: Steps is the number of timesteps used by agents to achieve a coverage ratio of $90\%$ within an episode. Fewer Steps implies faster exploration.

3. 3.

   Mutual Overlap: For effective collaboration, each agent should visit regions different from those explored by its teammates. We report the average overlapping explored area over each pair of agents when the coverage ratio reaches 90%. Mutual Overlap denotes the normalized value of this metric. Lower Mutual Overlap suggests better multi-agent coordination.

We first adapt $3$ single-agent planning-based methods, namely Nearest (Yamauchi, 1997a), Utility (Juliá et al., 2012), and RRT (Umari & Mukhopadhyay, 2017), to our problems by planning on the merged global map. The 3 planning-based baselines are frontier-based, i.e., they choose long-term navigation goals from the boundary between currently explored and unexplored area using different heuristics: Nearests chooses the nearest candidate point; Utility measures a hand-crafted utility function; RRT develops a Rapid-exploring Random Tree and selects the best candidate from the tree through an iterative process. Note that though these are originally single-agent methods and are adapted to multi-agent settings by planning on the merged global map. When choosing global goals, each agent performs computation based on the merged global map, its current position and its past trajectory.

For multi-agent baselines, we compare MAANS with $3$ planning-based methods, namely Voronoi (Hu et al., 2020), APF (Yu et al., 2021b) and WMA-RRT (Nazif et al., 2010). APF (Yu et al., 2021b) computes artificial potential field over clustered frontiers and plans a potential-descending path with maximum information gain. APF introduces resistance force among multiple agents to avoid repetitive exploration. WMA-RRT (Nazif et al., 2010) is a multi-agent variant of RRT, in which agents cooperatively maintain a single tree and follow a formal locking-and-search scheme. Voronoi-based method partitions the map into different parts using a voronoi partition and each agent only searches unexplored area in its own partition.

Finally, we also evaluate the performance of random policies for references. We remark that all the baselines only replace the global planner module with corresponding planning-based methods while utilizing the same neural SLAM, local planner and local policy modules as MAANS for a fair comparison. More implementation details can be found in Appendix B.

We report the training performances of multiple RL variants on 2 selected scenes, Colebrook and Dryville, and measure the Steps and the Mutual Overlap over these 2 scenes.

Due to space constraints, we only present a selected portion of the most competitive results in the main paper and defer the full results to Appendix F.

Fig. 7 demonstrates two 2-agent trials of MAANS and RRT, the most competitive planning-based method, with the same birth place. The merged global map are shown in keep timesteps. As shown in Fig 7, MAANS’s coverage ratio goes up faster than RRT, indicating higher exploration efficiency. At timestep around $90$, MAANS produces global goals successfully allocate the agents towards two distant unexplored area while RRT guides the agents towards the same part of the map. And at timestep around $170$ when MAANS reaches $90\%$ coverage ratio, RRT still stuck in previous explored area though there is obviously another large open space. Notice that at this key timestep RRT selects two frontiers that are marked unexplored but with no actual benefit, which an agent utilizing prior knowledge about room structures would certainly avoid.