Benchmarking Augmentation Methods for Learning Robust Navigation Agents: the Winning Entry of the 2021 iGibson Challenge

PointGoal Navigation (PointNav), as formally defined by Anderson et al. in [3], is a task in which the robot must navigate from a start position to a goal coordinate in the environment. This is a challenging task to perform in a novel scene containing obstacles, especially when the agent only has access to an egocentric camera and an egomotion sensor. Despite these constraints, Wijmans et al. [12] proved that a near-perfect ($>$97% success) PointNav agent could be trained in simulation with deep reinforcement learning. However, this work leveraged a large amount of training scenes; in this work, we focus on how augmentation methods can improve performance in scenarios in which only a small amount of scenes are available, and show that training an agent to reach the same levels of success in such scenarios is not yet solved.

Though there has been a large amount of work on autonomous robot navigation, a very small subset of this research has focused on navigation requiring the robot to interact directly with dynamic elements in the environment. The recent work by Xia et al. [11] introduced a new task, InteractiveNav, in which the robot interacts with displaceable objects in order to reach the goal. This does not require the robot to use a manipulator; the robot can simply push past the objects using its base as it moves. InteractiveNav evaluates how well the robot can balance taking the shortest path to the goal and how much it disrupts the obstacles present in the environment. Their work showed that it is possible to teach an agent to complete this task with deep reinforcement learning, though they supply the agent with the next ten waypoints to the goal position as observations at each step. In this work, we seek to investigate performance for this task using only egocentric vision and an egomotion sensor and how it can be improved using augmentation methods.

SocialNav is another task that requires the robot to maneuver around dynamic elements in the environment, in which the robot must reach the goal without colliding with nearby moving pedestrians. Similar to our work, several recent works have used deep reinforcement learning to avoid colliding with pedestrians, such as SA-CADRL [13] and SARL [14], which use the positions of the pedestrians as observations to their policy, and the work by Pérez-D’Arpino et al. [15] which uses a motion planner that has access to a map of the environment and LiDAR data. In contrast to these works, we aim to complete the task of SocialNav without a motion planner or LiDAR, and investigate how performance can be improved using augmentation methods for visual deep reinforcement learning. We believe this setup is better suited for deployment in novel environments, where accurate localization of pedestrian positions or providing a map to a motion planner may be difficult or infeasible.

Several recent works have studied techniques to improve generalization to novel environments for visual navigation. Ye et al. [16] build upon the work by Wijman et al. [12] through the use of self-supervised auxiliary tasks, which proved to significantly improve sample efficiency. Sax et al. [17] incorporate visual priors about the world to confer performance gains over training end-to-end from scratch without priors. Chaplot et al. [18, 19] improve sample efficiency compared to typical end-to-end agents by having the agent infer information about mapping and localization, which can be supervised using privileged information from the simulator.

Unlike the works mentioned above, we study methods that yield significant improvement for visual navigation without the use of additional auxiliary losses, extra network heads, or additional sensory information (e.g., local occupancy map). In this work, we focus on augmentation methods that can be applied to virtually any visual deep reinforcement learning approach to improve test-time generalization. We study how these methods improve navigation both when used on their own and when combined with each other.

Training an effective visual navigation agent requires a large dataset ($>$70 3D apartment scans) to avoid overfitting [12]. However, it can be difficult to have access to a diverse set of interactive training environments, especially for dynamic environments in which each object is individually interactable and must be arranged within the environment in a realistic manner, such as the synthetic scenes in the iGibson dataset, which contains only 15 scenes. For the 2021 iGibson Challenge, only eight training scenes were available. In such data-constrained scenarios, augmentation methods can be vital in improving the capability of the agent to generalize to novel environments at test-time.

Dynamic Obstacle Augmentation. To improve the test-time generalization of the agent for visual navigation, we introduce several moving pedestrians into the environment (illustrated in Figures 1 and 2). This approach aims to prevent the agent from learning to overfit to the environments it sees during training; with a large number of moving visual distractors present in the scene, it is more difficult for the agent to memorize the layout of the training environment using observations from its camera. Additionally, unlike image augmentation methods that simply perturb the agent’s visual inputs, this method forces the agent to learn a larger variety of paths for a given episode’s pair of start-goal positions. Because the paths that the pedestrians take are newly generated each time the environment is reset, the agent is not able to always take the same path between the same pair of start-goal positions without colliding into a pedestrian.

During training, these moving pedestrians are treated as dynamic obstacles that the agent is not allowed to collide with. If a collision between the agent and a pedestrian occurs, the episode is simply terminated. Between 3 to 12 pedestrians represented by a 3D model of a standing human from the iGibson Challenge [4] is used, which we have found to be sufficiently large (for visual occlusion) while having a small enough radius for the agent to maneuver around in most situations. At the start of an episode, for each pedestrian, two navigable points at least 3 meters apart are randomly sampled from the environment. Throughout the episode, each pedestrian moves back and forth along the shortest obstacle-free path connecting these points. It has the same maximum linear and angular speed as the agent, but its speed is randomly decreased from the maximum value by up to 10% on a per-episode basis, such that different pedestrians within the same episode can be moving at different velocities.

Comparison with Image-based Data Augmentation. Image augmentation is one of the most popular approaches to improving sample efficiency for vision-based learning. Image augmentation aims to inject priors of invariance using random perturbations, such as cropping, erasing, or rotation. In particular, Laskin et al. [7] showed that the Crop and Cutout image augmentations (see Figure 3) are among the most effective for visual deep reinforcement learning performance improvement. Crop extracts a random patch from the original frame; similar to Laskin et al., we crop the frame at a random location such that the resulting image is 8% shorter in height and width (i.e., it maintains the original aspect ratio). Cutout, detailed by Zhong et al. [20], inserts a black rectangle of a random shape, aspect ratio, and location into the original frame. We use the same parameters as Zhong et al., in which the rectangle’s aspect ratio can be between $[0.3,3.33]$, and its scale relative to the original frame can be between $[0.02,0.33]$. In addition, we investigate Crop&Cutout, which combines Crop and Cutout by applying them sequentially.

Observation and Action Spaces. At each step, the policy observes an egocentric (first-person) depth image and its relative distance and heading to the goal point as input. As the goal point is specified relative to the agent’s initial position, and the egomotion sensor indicates the agent’s relative distance and heading from its initial position, they can be used to calculate the agent’s current relative distance and heading to the goal. We choose to use only depth vision rather than depth and RGB vision together, as Wijmans et al. [12] has shown that including RGB can hurt performance relative to using depth alone. The policy outputs a two-dimensional diagonal Gaussian action distribution, from which a pair of actions are sampled (linear and angular velocity). The maximum speeds are 0.5 m/s and 90°/s, and the policy is polled at 10 Hz. If the magnitude of the velocities are below a certain threshold (10% of their maximum values for our experiments), it is perceived as the agent invoking a stop action, which ends the episode.

Reward Function. The reward function used for all our experiments are the same regardless of what augmentation method is used, if any at all:

$\displaystyle r(a_{t},s_{t})\ =\ -\Delta_{d}-w_{1}-w_{2}(I_{back}+I_{col})+w_{3}I_{succ},$

where $-\Delta_{d}$ is the change in geodesic distance to the goal since the previous state, and $I_{back}$, $I_{col}$, and $I_{succ}$ are binary flags indicating whether the agent has moved backwards, collided with the environment, or terminated the episode successfully (see Success Criteria in Subsection III-B), respectively. $w_{1}$ serves as a constant slack penalty incurred at each step to encourage the agent to minimize episode completion time. $w_{1}$, $w_{2}$, and $w_{3}$ are set to 0.002, 0.02, and 10.0. Backward motion is penalized because we have observed that it often leads to poor visual navigation performance. However, we do not prevent it altogether, as backward movement can be helpful for avoiding an incoming obstacle that suddenly appears into the robot’s view.

Success Criteria. For all tasks, episodes are considered successful only if the agent both (1) invokes a stop action that terminates the episode, and (2) the agent is within $0.2$ m of the goal point. Episodes are terminated and deemed unsuccessful after $500$ steps, or if the agent collides with a dynamic pedestrian either for SocialNav or when training with dynamic obstacle augmentation. A collision with a pedestrian is deemed to have occurred if the agent is within 0.3 m of it.

Network Architecture and Training. Our network architecture has two main components; a visual encoder and a recurrent policy. The visual encoder is a ResNet-18 [21] convolutional neural net which extracts visual features from the depth image. The policy is a 2-layer LSTM [22] with a 512-dimensional hidden state. The policy receives the visual features, the relative distance and heading to the goal, the previous action, and its previous LSTM hidden state. In addition to a head that outputs actions, the policy has a critic head that outputs an estimate of the state’s value, which is used for Proximal Policy Optimization (PPO) reinforcement learning as described by Schulman et al. in [23]. We use Decentralized Distributed PPO by Wijmans et al. [12] to train our agents using parallelization, and use the same learning hyperparameters. We train 8 workers per GPU, using 8 GPUs for a total of 64 workers collecting experience in parallel. Each agent is trained for 500M steps (~280 GPU hours, or ~35 hours wall-clock).

All of our agents are trained within Habitat [24], a high-performance photorealistic 3D simulator that provides the virtual robot with first-person visual data. Habitat can reach speeds of up to 3,000 frames per second per simulator instance, while the iGibson simulator used in the iGibson Challenge runs at approximately 100 frames per second [10]. Habitat also readily supports parallelization of simulator instances and gradient updates across multiple GPUs. Its speed and scalability allow us to train many agents very quickly and save a significant amount of time.

In this work, we use both the Gibson-4+ and iGibson datasets (depicted in Figure 5). We use the Gibson-4+ dataset [24] for training and evaluation, which contains 86 high quality 3D scans of real-world indoor spaces curated manually from the larger Gibson dataset [5]. All scenes in the Gibson-4+ dataset have been rated 4 or above on a quality scale from 0 to 5 by a human, and are free of significant reconstruction artifacts such as holes in texture or cracks in floor surfaces. The training split contains 3.6M episodes distributed across the 72 scenes (50k per scene), while the validation split contains 994 episodes across 14 scenes (71 per scene). In order to select checkpoints and evaluate our agents, we randomly select (without replacement) 7 out of the 14 scenes in the validation set to form a val₁ and val₂ set, such that each set contains 497 episodes.

For the 2021 iGibson Challenge, instead of the Gibson-4+ dataset we train on the iGibson dataset of synthetic scenes (8 scenes total) imported into Habitat. We also use the iGibson dataset to evaluate agents trained on the Gibson-4+ dataset for the task of sim-to-sim transfer on InteractiveNav using the iGibson simulator.

In all cases, the agent’s height and radius are modeled after the LoCoBot platform (0.8 m and 0.18 m) [25]. The LoCoBot is depicted on the right of Figure 5.

During evaluation, the agent’s behavior was set to be deterministic for reproducibility, by sampling only the means from the policy’s output Gaussian action distribution. We trained each agent configuration three times using different seeds. For each seed, the checkpoint that yielded the highest success rate on val₁ was selected, and reported evaluation metrics are based on that checkpoint’s performance on val₂. There are no pedestrians present during PointNav evaluation, and three pedestrians for SocialNav evaluation. For InteractiveNav, objects from the YCB dataset [26] are placed on the shortest obstacle-free path from the start to the goal spaced 0.5 m apart.

We use success rate as our primary evaluation metric. Apart from success, to evaluate InteractiveNav performance in iGibson, we use Success weighted by inverse Path Length (SPL) [3], as well as Effort Efficiency and Interactive Navigation Score (INS) [27]. SPL and Effort Efficiency are both scores between 0 and 1, with 1 being the best. SPL measures how short the agent’s path is relative to the length of the shortest path between the start and goal. Effort Efficiency penalizes displacement and forces that are applied to objects in the environment by the robot. INS is simply the average of SPL and Effort Efficiency.

We evaluated the performances of many training configurations, in which both the amount of available training scenes and number of dynamic pedestrians used during training are varied. Table I represents the success rates for PointNav, while Table II shows the success rates for SocialNav. We did not evaluate agents trained with zero pedestrians for SocialNav due to their inability to adapt to the presence of dynamic obstacles at test-time, which led to poor performance.

Finding 1: Dynamic obstacle augmentation improves visual navigation performance, even when the environment is static. Our results indicate that dynamic obstacle augmentation improves performance for both visual navigation tasks, as the best performance is achieved when dynamic pedestrians were added in the training environments. An interesting observation is that dynamic obstacle augmentation improves test-time generalization even for PointNav, where the environment is completely static and there are no dynamic obstacles at test-time.

Finding 2: Dynamic obstacle augmentation is more effective with a small number of training scenes. Both tables also show that the performance gains provided by dynamic obstacle augmentation are more substantial when only a small number (8 or less) of training scenes are available. As the number of available training scenes increases, the gap between the baseline agents and those trained with added dynamic pedestrians decreases. For instance, the difference between the baseline and the best agents is $3.5$% with eight training scenes but the difference is only $0.42$% with $72$ training scenes in Table I. This trend indicates that the limited number of training scenes causes the agent to overfit, but can be mitigated by dynamic obstacle augmentation.

In this section, we compare our technique against image augmentation techniques, Crop and Cutout, as defined in Section III-A. To narrow the scope of our investigation and evaluate how well each technique mitigates overfitting, we limit experiments to agents trained with only eight scenes. When dynamic obstacle augmentation is used, six pedestrians are added during training. Otherwise agents were trained with zero pedestrians for PointNav, or three pedestrians for SocialNav.

Finding 3: Image augmentation can improve test-time success rate as well, yet combining them can decrease performance gains. We find that Crop and Cutout can also improve test-time success when each augmentation is used alone as shown in Figure 6, with the exception of Crop for PointNav. We find that using Cutout alone outperforms dynamic obstacle augmentation by a slight margin for both tasks. We also evaluate the effects of combining the image augmentation techniques. When training with deep reinforcement learning, applying different data augmentation techniques in conjunction does not necessarily improve performance over a single isolated technique. Laskin et al. [7] showed that the best results on the DeepMind Control Suite [8] are achieved when only Crop is used, and performance would only drop when combined with other image augmentation techniques. Similarly, we observe that combining Cutout and Crop (bottom-middle cells) only leads to worse performance than simply using Cutout alone.

Finding 4: Combining dynamic obstacle and image augmentation techniques can further increase success rates. However, as shown in Figure 6, when dynamic obstacle augmentation is combined with either the Crop or Cutout image augmentations, we find that the success rate can improve even further than using either method alone. While combining image augmentation methods hurts performance, combining Dyn with Crop (middle-left cells) yields a higher success rate than using either method alone. This combination achieves the highest success rate for both tasks, tying with Cutout for PointNav and surpassing it for SocialNav. Combining Cutout with Dyn (bottom-left cells) does not affect performance over using Cutout alone, in contrast to the performance decrease that occurs when Crop is added to Cutout.

Though we can evaluate our agents for PointNav and SocialNav in Habitat, emulating the InteractiveNav task in Habitat requires a large amount of engineering and remained out of our reach for this work, despite prolonged effort. Instead, we evaluate the agents trained in Habitat directly in iGibson for InteractiveNav in a zero-shot manner (no adaptation). This is the same approach we used to achieve 1^st place in the iGibson Challenge.

As in the previous subsection, all agents were trained with eight Gibson-4+ scenes, and those trained with dynamic obstacle augmentation were trained with six pedestrians (no pedestrians otherwise). We evaluate our agents in the eight iGibson scenes from the training set for the InteractiveNav task; currently, the evaluation set from the iGibson Challenge is not public.

Finding 5: Dynamic obstacle augumentation is more robust to sim-to-sim transfer from Gibson-4+ to iGibson scenes. As shown in Table III, while all augmentation methods improve performance over using no augmentation at all, dynamic obstacle augmentation did so by the greatest amount (~11% more success). Notably, in contrast to the results in the previous subsection, the results yielded by the image augmentation methods are much less competitive against dynamic obstacle augmentation. In fact, when using Crop or Cutout in conjunction with our method, performance is actually slightly reduced. Whereas Cutout by itself conferred one of the highest gains in performance in the previous subsection, it now provides the least gains over the baseline, beating only Crop&Cutout.

We attribute this to the visual gap between the iGibson scenes and the Gibson-4+ scenes that our agents were trained on; Gibson-4+ scenes are collected from the real world using a 3D camera and may contain some irregularities or artifacts, whereas iGibson scenes are completely synthetic, comprised wholly by 3D computer-aided design (CAD) files made by artists, and thus contain no artifacts and have much more geometric surfaces and edges (see Figures 1 and 5 for comparison).

Because image augmentation methods encourage the agent to infer the removed information from the remaining context available in the visual input, agents trained with these methods are more sensitive to visual changes in that context. In contrast, dynamic obstacle augmentation maintains a substantial performance gain. We conjecture that it allows agents to learn a better spatial understanding of their surroundings by dodging obstacles and recalculating new paths, as opposed to simply teaching the agents to use existing visual context to recover from visual perturbations.

For the agent we submitted to the 2021 iGibson Challenge, we train on the 8 provided iGibson dataset scenes, not on Gibson-4+ scenes using the Habitat simulator. We submitted the same agent to both the InteractiveNav and SocialNav tracks. The results of the top five teams for each track are shown in Table IV. Dynamic obstacle augmentation was the only augmentation used, with 12 pedestrians being present during training. For InteractiveNav, we achieved 4% more success than the 2^nd place team, who used image augmentation and a reward function very similar to ours, except with added terms designed to encourage the robot to seek alternative paths when blocked [28]. Additionally, their agent is trained natively in iGibson, precluding any issues that may arise from sim-to-sim transfer from Habitat to iGibson (which we discuss at the end of this section). For SocialNav, we attain a success rate only 2% lower than the 1^st place team.

Episode visualizations and metrics for causes of episode failure on the challenge scenes were not shown to participants. Upon asking the organizers, we were informed that the large majority of episodes across all entries failed due to timeout (71% for InteractiveNav and 63% for SocialNav). We speculate that the large difference between the success rates on the challenge scenes (around $30$%) and the experiment results seen in subsections V-B and V-C (around $81$% and $60$%), in which only eight training scenes were also used (albeit from Gibson-4+, not iGibson), may be due to the size and layout of the challenge scenes and the sim-to-sim gap from Habitat to the iGibson simulator. Scenes from the iGibson dataset tend to be relatively small (see Figure 5), especially compared to the real-world scans of the Gibson-4+ dataset used to train the agents in subsections V-B and V-C. The size and furniture layout of the challenge scenes may have restricted the robot to narrower paths that caused it to get caught on obstacles more often, or accidentally block a path to the goal with moveable furniture or obstacles (especially for InteractiveNav, where there is also clutter), or make passing an incoming pedestrian along a narrower path much more difficult for SocialNav.

Additionally, we “freeze” the iGibson training scenes in Habitat to maintain high simulation speeds when training our agents. In other words, furniture in the scene such as chairs and lamps could not be moved by the robot in the Habitat simulator, whereas they could be moved in either InteractiveNav or SocialNav upon deployment in the iGibson simulator. Our agents avoid pushing larger objects because these objects cannot move in training, and thus the agents learn to simply try to circumvent them entirely. This may have also contributed to making navigation more difficult for the robot in the iGibson Challenge, as it may not have been able to adapt to cases in which colliding with the environment caused an obstacle to block the path to the goal.