Vision-Language Navigation:
A Survey and Taxonomy

Misra et al. [19] collect a corpus of navigation instructions using crowd sourcing. They used LANI simulator to generate environments randomly and one reference path for each environment. The generated reference paths are near to their neighbor landmarks to elicit instructions. Then, they used Amazon Mechanical Turk to annotate. The experiment environment is simulated and relatively simple. Furthermore, Blukis et al. [70] propose a real-world learning framework, which is similar to LANI, so we will not describe details here.

The task performance is evaluated on a test set $\{(\bar{x}^{(i)},s_{1}^{(i)},s_{g}^{(i)})\}_{i=1}^{M}$, where $\bar{x}^{(i)}$ is an instruction, $s_{1}^{(i)}$ is the start state, and $s_{g}^{(i)}$ is the goal state. Task completion accuracy and the distance of the agent’s final state to $s_{g}^{(i)}$ are evaluated.

Before LANI, Blukis et al. [71] propose a virtual environment navigation task based on Unreal Engine, which uses the AirSim plugin [72] to simulate realistic quadcopter dynamics. The agent is a quad-copter flying between landmarks. This work focus on the problem of mapping, planning and task execution. Language instructions for this task are generated from a pre-defined set of templates. For solving this problem,  Blukis et al. [71] put instructions into a LSTM network to get language embedding at the task beginning. A custom residual network is used to represent the image features at every time step and project features in global reference frame for making actions.

In Misra et al. [19]’s work, this instruction following task is decomposed into two submodules: goal prediction and action generation. They proposed a new language-conditioned image generation network architecture LINGUNET, to make a map from visual input to goals output.

Base on LANI, Blukis et al. [73] propose an approach for mapping natural language instructions and raw observations inputs to continuous control of a quadcopter drone, which used the quadcopter simulator environment from Blukis et al. [71]. For indicating where the agent should visit during navigation and where to stop, they built a model to predict the position-visitation distributions, and then actions are generated from the predicted distributions.

For combing the simulation and reality, Blukis et al. [70] propose a learning framework to map language instructions and images to low-level action outputs. In addition, Supervised Reinforcement Asynchronous Learning (SuReAL) is used in both simulation and reality without the need to fly in real world during training. SuReAL combines supervised learning for predicting next goal and reinforcement learning for continuous action outputs.

More recently, Blukis et al. [74] study the problem of extending to reason about new objects. Due to the lack of sufficient training data, they used a few-shot method trained from extra augmented reality data to ground language instructions to objects. This method can align the objects and their mentions in language instructions.

Action Learning From Realistic Environments and Directives [8](ALFRED) is a indoor task, agents in this task need to receive a language instruction about doing a household and first-person image observations, then generated a sequence of actions to finish it. ALFRED includes 25,743 English instructions, describing 8,055 expert demonstrations, each with an average of 50 steps, resulting in 428,322 image-action pairs.The expert demonstrations are together with both high-level and low-level language instructions in 120 indoor scenes based on AI2-THOR 2.0 simulator [20]. These demonstrations involve partial observability, a long range of actions, in a designated natural language, and irreversible actions.

ALFRED allows users to evaluate both full task and task goal-condition completion. In navigation-only tasks, the model performance can be measured by how far the agent is from the goal. Whether the target conditions of the task have been completed can also be evaluated in the ALFRED task. The evaluation metrics consist of Task Success, Goal-Condition Success and Path Weighted Metrics.

• •

  Task Success. If the object positions and state changes correspond correctly to the task goal-conditions at the end of the action sequence, the task success will be set to 1, and otherwise to 0.

• •

  Goal-Condition Success. This metric is the ratio of goal-conditions completed at the end of an episode to those necessary to have finished a task.

• •

  Path Weighted Metrics. Anderson et al. [75] find that PDDL solver can not find the optimal result, but usually efficient. The path weighted score $p_{s}$ for metric $s$ is given as

  $$p_{s}=s\times\frac{L^{*}}{max(L^{*},\hat{L})}$$

  (1)

  where $\hat{L}$ is the number of actions the model took in the episode, and $L^{*}$ is the number of actions in the expert demonstration. Obviously, a model receives half-credit for taking twice as long as the expert to accomplish a task.

In [8], a Sequence-to-Sequence (Seq2Seq) Model with progress monitoring is introduced. Each visual observation was encoded with a froze ResNet-18 [2]. The step-by-step instructions is combined into a single input sequence with the $<SEP>$ token. Then the sequences are fed into a bi-directional LSTM encoder to produce an encoding sequence. The agent’s actions at each time step are based on the attention mechanism that can identify the related tokens in the language instruction. The agent interacts with the environment by choosing an action and producing dense pixel-wise binary mask to indicate specific objects in the frame.

In [76], during the navigation of the sub-goals, the agent’s field of view is enhanced through multiple perspectives, and the agent is trained to predict its relative spatial relationship with the target position at each time step.

Singh et al. [77] propose a Modular Object-Centric Approach (MOCA) to decouple ALFRED task into two sub-task, i.e., visual perception and action policy generation. The former aiming to predict an interaction mask for the object that the agent interacts with using object-centric mask prediction, while the latter makes a prediction of the current action of the agent.

Embodied Question Answering (EQA) tasks require an agent to ask a question (e.g. ‘What color is the cup?’). The agent usually situated at a random location in an environment (a house or building), and can observe the environment by first-person perspective images. The action space of the agent includes move forward, turn and strafe, etc. After understanding the question, the agent needs to gather useful information in the environment for answering it. Unlike the previous work [78], the agent does not receive any global or structured representation of the environment (map, location, rooms, objects), or of the task (the functional program that generated the question).

EQA is built upon House3D simulator and SUNCG datasets, more details about House3D please refer to section III. In order to ensure the quality of the dataset, the environment inside must be realistic and typical, and there must be no abnormal conditions. Scenes in this task have one kitchen, dining room, living room, and bedroom at least. The EQA v1 dataset contains more than 5000 questions in over 750 environments, involving a total of 45 unique objects in 7 unique room types. There are 1 to 22 related questions in each environment, with an average of 6. Preposition questions are less than other kinds of questions because many frequently occurring spatial relationships are too easy to solve without exploration and cannot be processed by entropy threshold.

The goal of an EQA agent is to answer questions precisely. However, it is important to disentangle success/failure at the intermediate task of navigation from the downstream task of question answering. The evaluation metrics of this task are as following:

• •

  Question Answering Accuracy. The mean rank of the ground-truth answer in the candidate answers list sorted by the model.

• •

  Navigation Accuracy. This metric consists of the distance to the target object at navigation termination $\text{d}_{\text{T}}$, changes in distance to target from initial to final position $\text{d}_{\Delta}$, and the smallest distance to the target at any point in the episode $\text{d}_{\text{min}}$. All these distances are measured in meters along the shortest path to the target.

• •

  The percentage of questions for which an agent either terminates in ($\%\text{r}_{\text{T}}$) or ever enters ($\%\text{r}_{\hookleftarrow}$) the room containing the target objects.

• •

  The percentage of episodes in which agents choose to stop navigation and give a answer before reaching the maximum episode length ($\%\text{stop}$).

Das et al. [25] give a baseline model, which has four modules, i.e., vision, language, navigation, answering and are trained in two stages. First, the navigation and answer module use imitation/supervised learning to independently train the automatically generated navigation expert demonstrations. Then, the navigation architecture uses policy gradients for fine-tuning. This model can map raw sensory input to navigation actions and answers generation.

Anand et al. [79] explore blindfold (question-only) model for EQA which neglects environmental and visual information. Intuitively, the blindfold model is a degraded solution, but it reaches state-of-the-art results on EQA except some rare initialization situations.

Das et al. [80] propose a modular approach of learning strategies for navigating within long-term planning from instructions. For increasing sample efficiency, imitation learning is used to warm-start policies at each level of the hierarchical policy on multiple timescales.

Parvaneh et al. [81] propose a new learning strategy that learns both from observations and generated counterfactual environments. Specifically, the algorithm generating counterfactual observations on the fly for navigation is introduced as the linear combinations of present environments. In addition, the agent’s actions are encouraged to remain stable between initial and counterfactual environments by training objective–effectively removing false features that could mislead the agent.

Furthermore, Wu et al. [82] introduce an easier and practical EQA setting called calibration. They designed a warm-up stage that the agent is asked a few rhetorical questions when it entered into a new environment. The goal is to adapt the agent policy to the new environments. Then they proposed a new model, which contains two modules, designed for the intermediate navigation task, called Navigation Module, and downstream question answering task, called Question Answering Module, respectively.

When using behavioral cloning method to train a recurrent model for navigation, the impact of a novel loss-weighting metric named inflection weighting has grown. Wijmans et al. [83] extend the EQA task in realistic environments from the Matterport3D dataset [65] and propose a model which can surpass the baseline by optimize the infection weighting metric. Moreover, they argue that point clouds can provide more information than traditional RGB images in photo-realistic environment, especially for tasks like embodied navigation and obstacle avoidance.

Yu et al. [84] extend the EQA to Multi-Target version, i.e., Multi-Target EQA. As the questions are more complex, the agent is supposed to navigate to multiple locations and performs some comparative reasoning steps before answering the question. A new model is designed to solve MT-EQA, which consists of four modules: the question-to-program generator, the navigator, the controller, and the VQA module.

While Wu et al. [26] propose the House3D simulator, a Concept-Driven Navigation benchmark task is also proposed (i.e., RoomNav). The goal is defined as ”Go to X”, where X represents a pre-defined room type or object type. This is a semantic concept, and the agent needs to be interpreted from a variety of scenes with different visual appearances. RoomNav consists of 270 houses divided into three splits, i.e., small, large and test with 20, 200, and 50 respectively.

Observation: Three different kinds of visual input signals are utilized for $X_{t}$, including (1) raw pixel values; (2) semantic segmentation mask of the pixel input; (3) depth information, and different combinations of these three. Each concept (language instruction) $I$ is encoded as a one-hot vector representation.

Action Space: This task support continuous and discrete actions, such as agents can move in certain velocities or pre-defined movements.

The episode is regarded as successful if both of the following two criteria are satisfied: (1) the agent is located inside the target room; (2) the agent consecutively sees a designated object category associated with that target room type for at least 2 time steps. An agent sees an object if there are at least 4% of pixels in $X_{t}$ belonging to that object.

In [26], a gated-attention architecture is introduced. Gated-CNN and Gated-LSTM network are used for controlling continuous and discrete actions respectively. The Gated-CNN policy is trained using the Deep Deterministic Policy Gradient (DDPG) [85], while the Gated-LSTM policy is trained using the asynchronous advantage actor-critic algorithm (A3C) [86].

Wu et al. [87] introduce a new memory architecture, Bayesian Relational Memory (BRM), taking the form of a probability relationship graph on semantic entities to improve the generalization ability in unseen environments. BRM supports obtaining some prior knowledge from the training environment, and at the same time, it can update the knowledge in memory after exploration. A BRM agent consists of a module for generating sub-goals and goal-conditioned locomotion module for control

Remote Embodied Visual referring Expression in Real Indoor Environments - REVERIE [11] task requires the intelligent agent to correctly locate the remote target object specified by the concise high-level natural language instructions (which cannot be observed at the starting position). Since the target object and the starting object are in different rooms, the agent needs to navigate to the target location first. Unlike other navigation tasks, agents in this task can execute detection action if agents thinks they has localised the target object and decide to output the region of it. After the detection action, this episode is finished.

The performance of a model is mainly measured by REVERIE success rate, which is the number of successful tasks over the total number of tasks. A task is considered successful if it selects the correct bounding box of the target object from a set of candidates. Since the target object can be observed at different viewpoints or camera views, an episode is successful as long as the agent can identify the target within 3 meters, regardless of from different viewpoints or views. We also measure the navigation performance with four kinds of metrics, including success rate, oracle success rate, success rate weighted by path length, and path length (in meters) [10].

Qi et al. [11] propose a modular model including three parts: a navigator module that decides the action to take for the next step, a pointer module that attempts to localise the target object according to the language guidance, and an interaction module that responses for sending the referring expression comprehension information obtained from the pointer to the navigator to guild it to make more accurate action prediction.

Lin et al. [88] introduce two pre-training tasks, called Scene Grounding task and Object Grounding task, and a new memory-augmented attentive action decoder. The pre-training tasks encourage agents learn where to stop and what to attend to, and the action decoder uses past observations to merge visual and textual information in an effective way.

Talk to car task is proposed by Deruyttere et al. [21], and the dataset for this task is built upon the nuScenes dataset [89]. This dataset includes 11,959 instructions for the 850 videos of nuScenes training set since 3D bounding box annotations for the test set of nuScenes are not available. 55.94% and 44.06% of these commands belong to videos taken respectively in Boston and Singapore. On average a command consist of 11.01 words, 2.32 nouns, 2.29 verbs and 0.62 adjectives. Each video has on average 14.07 commands. The textual annotations are very complete, it not only indicates what to do, such as ‘pick him up’, but how to drive the car, such as ‘turn around’. However, this task did not give a metric to evaluate the performance of the action.

Deruyttere et al. [21] only focus on detecting the referred object, but ignoring the action execution. They only assessed the performance of 7 detection models to find the objects in a command. In the test set, random noun matching approach is used to match the nouns in command with the visual objects.

This task is proposed by Yu et al. [90]. It consists of XWORLD2D and XWORLD3D.

• •

  XWORLD2D. To test the method of interactive grounded language acquisition and generalization,  Yu et al. [22] build a 2D maze-like world, i.e., XWORLD2D, including language related navigation task and question answering task. Specifically, the navigation task requires agent to follow the language instructions to navigate the final destination, and question answering task requires agents to generate one-word answer for questions.

• •

  XWORLD3D. The XWORLD3D is the extension of XWORLD2D where changing the environment from full-observability to partial-observability, and adding two actions Turn left and Turn right to the discrete action set. For increasing the visual variance, at each session the authors randomly rotate each object and scale it randomly within $[0.5,1.0]$. Suppose each map is $X\times Y$ , the maximum time steps of an episode is $3XY$.

In addition, successful episode means the agent acts obey the teacher’s command. The failure episode is triggered whenever the agent hits any object that is not required by the command. The success rate is used to evaluation the model performance.

In [22], an effective baseline method Guided Feature Transformation (GFT) is proposed for language grounding. In order to improve the fusion of visual and language features, the latent sentence embeddings calculated from the language input is regarded as the conversion matrix of visual features. This method is completely differentiable and embedded in the agent’s perception system, which is trained end-to-end by the RL. GFT method can adapt to both 2D and 3D environments without changing any architecture or hyperparameters between the two scenarios.

Chaplot et al. [23] create an environment for language grounding research, where the agent needs to follow natural language instructions and receives positive rewards after successfully completing the task. This environment is built on top of the VizDoom API [24], based on Doom, a classic first person shooting game.

In 3D Doom task, the instruction is a (action, attribute, object) triplet. Each instruction can have multiple attributes, but the number of actions and objects is limited to one each. This environment allows various objects to be generated at different locations on the map. Objects can have various visual attributes. Success rate defined as the proportion of success navigation times in the total number of navigation times. There are two scenarios for evaluation:

• •

  Multi-task Generalization. The agent uses the instructions in the training set for evaluation on an unseen map. The unseen map consists of a combination of invisible objects placed at random locations.

• •

  Zero-shot Task Generalization, where the agent is situated in unseen environment with new combinations of attribute-object pairs which are not seen during the training. The maps in this scenario are also unseen.

The model proposed by [23] uses the Gated-Attention mechanism to combine images and text representations, and utilize standard reinforcement and imitation learning methods to learn strategies for executing natural language instructions. This model is end-to-end and consists of State Processing Module and Policy Learning Module.

This task [91] is based on the interactive environments of AI2-THOR [20]. There are 87 object categories within AI2-THOR that are common among the scenes. However, some of the objects are not visible without interaction. Therefore, only 53 categories based on their visibility at random initialization of the scenes. To test the generalization ability of the method on novel objects, these 53 object categories are split into known and novel sets, and the known set of object categories are used in training. Only the navigation commands of AI2-THOR are used for the task. These actions include: move forward, move back, rotate right, rotate left, and stop.

The evaluation metrics are Success Rate and the Success weighted by Path Length (SPL) metric recently proposed by Anderson et al. [75].

• •

  Success Rate is defined as the ratio of the number of times the agent successfully navigates to the target and the total number of episodes.

• •

  SPL is a better metric which is a function considering both Success Rate and the path length to reach the goal from the starting point. It is defined as $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{L_{i}}{max(P_{i},L_{i})}$, where $N$ is the number of episodes, $S_{i}$ is a binary indicator of success in episode $i$, $P_{i}$ represents path length and $L_{i}$ is the shortest path distance (provided by the environment for evaluation) in episode $i$.

Based on the actor-critic model [86], Yang et al. [91] propose to use Graph Convolutional Networks (GCNs) [92] to incorporate the prior knowledge into a Deep Reinforcement Learning framework. The knowledge of the agents is encoded in a graph. GCNs can encode various structured graphs in an efficient way.

Zang et al. [27] create a new dataset for the problem of following navigation instructions under the behavioral navigation framework of [93]. This dataset contains 100 maps of simulated indoor environments, each with 6 to 65 rooms. It consists of 8066 pairs of free-form natural language instructions and navigation plans for training. This training data is collected from 88 unique simulated environments, 6064 distinct navigation plans in total (2002 plans have two different navigation instructions each, and the rest has one). The dataset contains two different testsets:

• •

  Test-Repeated: uses seen trainset environments and unseen routes, including 1012 pairs of instructions and navigation plans.

• •

  Test-New: uses 12 unseen environments and unseen routes, which is more complex, including 962 pairs of instructions and navigation plans.

An end-to-end deep learning model is proposed by Zang et al. [27] to convert free-form natural language instructions into high-level navigation plans.

Room-to-Room (R2R) is a natural language navigation dataset based on vision in a photo-realistic environment [10]. Environments in this dataset is defined by a navigation graph, where nodes are locations with a self-centered panoramic image, and edges define effective connections for agents navigation. This dataset(spited with train/validation/test) is fully annotated with instructions and paths and each path has 3 different related instructions. The average length of ground-truth path is 4 to 6 edges.

R2R is a task where the inputs are the images from egocentric panoramic camera and instruction. Note that images are updating with the motion of agent, while instruction is given at the beginning. For example, given an instruction “Go towards the front door but before the front door make a left, then through the archway, go to the middle of the middle room and stop.”, an agent is supposed to understand the instruction and follow it from start point to the goal position as soon as possible.

There are a bunch of metrics to evaluate a R2R agent. Typical used metrics are Success Rate, Navigation Error, Path Length and Success weighted by Path Length. The Oracle Navigation Error, Oracle Success Rate can be used to monitor the training process.

• •

  Success Rate (SR) shows how many times the last node of the predicted path is within a certain threshold distance of the last reference path node.

• •

  Navigation Error (NE) measures the distance between the last predicted path node and the last reference path node.

• •

  Path Length (PL) is equal to the total length of the predicted path. It is optimal when it’s equal to the length of the reference path.

• •

  Success weighted by Path Length (SPL) [10] takes both success rate and path length into account and can be calculated in the same way as described in the previous section. A drawback of SPL is that it does not consider the similarity of the intermediate nodes of the predicted path to the reference path. This results in that although SPL score is high, the predicted path does not actually follow the instructions, but just finds the correct target.

• •

  Oracle Navigation Error (ONE) measures the smallest distance from any node in the path to the reference goal node.

• •

  Oracle Success Rate (OSR) measures how often any node in the path is within a certain threshold distance from the goal. The goal is represented by the last node of the reference path.

Due to the inherent limitations of the existing evaluation metrics, the performance of the model cannot be evaluated objectively and comprehensively. There are still many subsequent works to propose some new metrics, which are also introduced here.

Huang et al. [94] propose a discriminator model that can predict how well a given instruction explains the paired path, showing that only a small portion of the augmented data in [95] are high fidelity. This metric can evaluate the instruction-path pair, which improves training efficiency and reduces the training time cost. Ilharco et al. [96] introduce the normalized Dynamic Time Warping (nDTW) metric. nDTW slightly penalizes deviations from the ground-truth path and it is naturally sensitive to the order of the nodes composing each path. Besides, nDTW is suited for both continuous and graph-based evaluations, and can be efficiently calculated. Despite the performance of R2R model has improved rapidly, current research is not clear about the role language understanding in this task, since mainstream evaluation metrics are focused on the completion of the goal, rather than the sequence of actions corresponding to the language instructions. Therefore, Jain et al. [29] analyze the drawbacks of current metrics for the R2R dataset and designed a novel metric named Coverage weighted by Length Score (CLS). CLS measures how closely an agent’s trajectory fits with ground-truth path, not just the completion of the goal. To improve the ability to evaluate language instructions, Zhao et al. [97] propose an instruction-trajectory compatibility model that operates without reference instructions. For system-level evaluations with reference instructions, SPICE metric [98] is better than BLEU [99], ROUGE [100], METOR [101] and CIDEr [102]. In other situations (e.g., selecting individual instructions, or model selection without reference instructions) using a learned instruction-trajectory compatibility model is recommended.

The R2R task is most impressive, and most research works of VLN are based on it. Basic framework of these works are Seq2Seq model, but they focus on different aspects to improve the model performance, therefore we divide them to 7 classes for clear understanding, i.e., Exploration Strategy, Beyond Seq2Seq Architecture, Reinforcement Learning and Imitation Learning, Language Grounding, Data Augmentation, Pre-Traning model and other related research.

Exploration Strategy. In this class, related works focus on finding a effective and fast path from start position to destination. Ma et al. [103] proposes a module to estimate the progress that made by agent towards the goal. Based on that, Ma et al. [104] design two modules for exploration, i.e., Regret Module for moving forward or rolling back and Progress Marker module to help the agent decide which direction to go next by showing directions that are visited and their associated progress estimate. While all current approaches make local action decisions or score entire trajectories using beam search, Ke et al. [105] present the Frontier Aware Search with backTracking (FAST) navigator. When the agent realizes lost itself, FAST navigator can explicit backtrack by using asynchronous search. This can also be plug and play on other models. Huang et al. [106] define two in-domain sub-tasks: Cross-Modal Alignment (CMA) and Next Visual Scene (NVS). The visual and textual representations of the agent learned in certain environment can be transferred to other environments with the help of CMA and NVS. Zhu et al. [107] introduce four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. These auxiliary tasks improve the model performance by giving more semantic information to help the agent to understand environments and tasks. Wang et al. [108] introduce an end-to-end architecture for learning an exploration policy, which enables the agent intelligently interacts with the environment and actively gather information when faced with ambiguous instructions or unconfident navigation decisions. In [109], a learning paradigm using recursive alternate imitation and exploration is proposed to narrow the difference between training and testing stages. Deng et al. [110] introduce the Evolving Graphical Planner (EGP), a method that uses raw images to generate global navigation plan efficiently. It has the advantage of dynamically building a graphical representation and generalizing the action space to allow more flexible decision-making.

Beyond Seq2Seq Architecture. Encouraged by recent progress on attention networks, many transformer-based studies have emerge, and we classify them as Beyond Seq2Seq architecture.

Landi et al. [111] devise Perceive, Transform, and Act (PTA) architecture to use the full history of previous actions for different modalities. Magassouba et al. [112] propose the Cross-modal Masked Path transformer, which encodes linguistic and environment state features to sequentially generate actions similarly to recurrent network approaches. In addition, this method uses feature masking to better model the relationship between the instruction and environment features. Wu et al. [113] and Mao et al. [114] use multi-head attention on visual and textual input to enhance the performance of the model. To capture and utilize the inter-modal and intra-modal relationships among different scenes, its objects, and directional clues, Hong et al. [115] devise language and visual entity relationship graph model. A message-passing algorithm is also proposed to spread information between the language elements and visual entities in the graph, and then the information are combined by the agent to determine the next action. Xia et al. [116] present a novel training paradigm, Learn from Everyone(LEO), which utilizes multiple language instructions (as multiple views) for the same path to resolve natural language ambiguity and improve the model generalization capabilities. Qi et al. [117] distinguish the object and action information from language instructions while most existing methods pay few attentions, and propose a Object-and-Action Aware Model (OAAM) that processes these two different forms of natural language based instruction separately. OAAM enables each process to flexibly match object-centric (action-centric) instructions with their corresponding visual perception (action direction). To capture environment layouts and make long-term planning, Wang et al. [118] present Structured Scene Memory (SSM), which allows the agent to access to its past perception and explores environment layouts. With this expressive and persistent space representation, the agent shows advantages in fine-grained instruction grounding, long-term reasoning, and global decision-making problem.

Reinforcement Learning and Imitation Learning. Many works have found it is beneficial to combine imitation learning and reinforcement learning with VLN models. All of Wang et al. [119], Tan et al. [120], Hong et al. [115], Parvaneh et al. [121] and Wang et al. [108] trained their models with two distinct learning paradigms, i.e., 1) imitation learning, where the agent is forced to mimic the behavior of its teacher. 2) reinforcement learning, which can help the agent explore the state-action space outside the demonstration path. Specifically, in imitation learning, the agent takes the teacher action $a_{t}^{*}$ at each time step to efficiently learn to follow the ground-truth trajectory. In reinforcement learning, the agent samples an action $a_{t}^{s}$ from the action probability $p_{t}$ and learns from the rewards, which allows the agent to explore the environment and improves ability of generalization. Combining IL and RL balances exploitation and exploration when learning to navigate, formally, the following loss function can be used to optimize the model :

For improving generalization ability, Wang et al. [122] integrate model-based and model-free reinforcement learning methods which is a hybrid planned-ahead model and outperforms than baselines.  Wang et al. [119] use Reinforced Cross-Modal Matching (RCM) method with extrinsic and intrinsic rewards. The novel cycle-reconstruction reward is introduced to match the instruction and trajectory globally. In [123], based on SEED RL architecture, Vision And Language Agent Navigation(VALAN) is introduced as a lightweight and extensible framework for learning algorithm research. In order to reduce the impact of reward engineering design, Soft Expert Reward Learning (SERL) model is proposed by Wang et al. [124], which contains two auxiliary modules: Soft Expert Distillation and Self Perceiving. The former activates agent to explore like an expert while the latter enforces the agent to reach the goal as soon as possible. Zhou and Small [125] devise a adversarial inverse reinforcement learning method to learn a language-conditioned policy and reward function. For better adapt to the new environment, variational goal generator is used during training to relabel the trajectory and sample different targets.

Language Grounding. This class of method wants to ground language instructions better through the method of multimodal information fusion, so as to improve the success rate of agent navigation. Wang et al. [119] present a novel cross-modal grounding architecture to ground language on both local visual information and global visual trajectory. Hu et al. [126] propose to decompose the grounding procedure into a set of expert models with access to different modalities (including object detection) and ensemble them at prediction time for utilizing all the available modalities more efficiently. For

For making the agent has a better understanding of the correspondence between text and visual modalities, a cross-modal grounding module composed of two complementary attention mechanisms is designed by Zhang et al. [109]. Kurita and Cho [127] build a neural network to compute the probability distribution over all possible instructions, and use Bayes’ rule to build a language-grounded policy. This method has a better interpretability than the traditional discriminative method by conducting comprehensive experiments. Hong et al. [128] argue that the granularity of the navigation task should be at the level of these sub-instructions, rather than attempting to ground a specific part of the original long and complex instructions without any direct supervision or measuring navigation progress at word level.

Data Augmentation. Using automatically generated navigation instructions as additional training data can enhance the model performance.  Fried et al. [95]propose a speaker-follower framework for data augmentation and reasoning in supervised learning. Hong et al. [128] enrich the benchmark dataset R2R with sub-instructions and their corresponding paths, i.e., Fine-Grained Room-to-Room (FGR2R). By pairing the sub-commands with the corresponding viewpoints in the path, FGR2R can better give the agent sufficient semantic information during the training process. Agarwal et al. [129] present a work-in-progress “speaker” model that generates navigation instructions in two steps. First, hard attention is used to select a series of discrete visual landmarks along the trajectory, and then a language conditioned on these landmarks is generated. Different from traditional data augmentation methods, Parvaneh et al. [121] propose an efficient algorithm to generate counterfactual instances that do not depend on hand-made or the particular field of rules. These counterfactual instances are added to the training, improving the ability of the agent when is tested in new environments. In [130], counterfactual idea has another application. A model agnostic method called Adversarial Path Sampler (APS) is introduced to sample paths to optimize the agent navigation strategies gradually. Yu et al. [131] simultaneously deal with the scarcity of data in the R2R task while removing biases in the dataset through random walk data augmentation. By doing so, they are able to reduce the generalization gap and outperform baselines in navigating unknown environments. An et al. [132] designe a module named Neighbor-View Enhanced Model (NvEM) to adaptively fuse the visual context from the neighbor views at the global level and the local level. Liu et al. [133] propose Random Environmental Mixup (REM) aiming to reduce the performance gap between the seen and unseen environment and improve the overall performance. This method breaks up the environment and the corresponding path, and then recombines them according to certain rules to construct a brand-new environments as training data. Sun et al. [134] point out that depth as a valuable signal source for the navigation has not yet fully explored and thus been ignored in previous research. Hence, they propose a Depth-guided Adaptive Instance Normalization module and Shift Attention (DASA) module to address this issue.

Pre-Traning model. The general feature representation obtained through pre-training model can be applied to various tasks, and has been confirmed in many fields. A strong pre-trained backbone network can be effective in downstream task, like image recognition in CV and question answering in NLP. In particular, Transformer networks Vaswani et al. [135] pre-trained with “masked language model” objective [5] on large language corpus outperforms on majority NLP tasks. Furthermore, Su et al. [136] developed VL-BERT, a pre-trainable generic representation for visual-linguistic tasks. In VLN field, pre-training approaches are used in two aspects:

• •

  Using pre-trained models to solve VLN tasks. In Li et al. [137], the agent is trained with pre-trained language models, (i.e., BERT [135] and GPT-3 [138]) and uses stochastic sampling to generalize well in the unseen environment. The PRE-trained Vision And Language based Navigator (PREVALENT) model proposed by Hao et al. [139] is pre-trained with image-language-action triples, and fine-tuned on the R2R task. Based on the use of the PREVALENT model, the agent can better complete the task in an unseen environment. Since the pre-training model usually has massive parameters, the efficiency of using it will be relatively low. For reducing the scale of pre-train model and improving the efficiency of inference , Huang et al. [140] introduce two lightweight methods, i.e., factorization and parameter sharing based on the PREVALENT[139] model.

• •

  Pre-trained VL or VLN models on VLN tasks. Due to the limitation of training data in VLN task, Majumdar et al. [141] try to use massive web-scraped resources to address this issue. Therefore, VLN-BERT is proposed and proved that pre-training it on image-text pairs from the web before fine-tuning the specific path instruction data significantly improves the performance of VLN task. Hong et al. [142] propose VLN$\circlearrowright$BERT, which is a multi-modal BERT model equipped with a time-aware recurrent function to provide the agent with richer information. Qi et al. [143] propose Object-and-Room Informed Sequential BERT (ORIST) to improve the language grounding performance by encoding visual and instruction inputs at the same fine-grained level, i.e., objects and words. Besides, the trained model can recognize the relative direction of each navigable location and the room type of its current and final navigation target.

Other Related Research. To assess the implications of this work for robotics, Anderson et al. [144] transfer a VLN agent trained in simulation to a physical robot. There is a big difference between high-level discrete action space learned by the agent and robot’s low-level continuous action space. In order to address this issue, they introduce a sub-goal model to identify nearby navigable points and use domain randomization to reduce visual domain differences. With the slow-down performance improvements in VLN tasks, a series of diagnostic experiments are carried out in  [145] to reveal the agent’s key points in the navigation process. The results show that the indoor navigation agent will refer to the object mark and direction mark in the instruction when making a decision. When it comes to visual and verbal alignment, many models claim that they can align object marks with certain visual targets, but there are still doubts about the reliability of this alignment.

A table comparing all models in R2R task. The state-of-the-art method is REM, since it is the effective combination of pre-trained backbone model (VLN$\circlearrowright$BERT) and data augmentation, making REM model the best existing approach on VLN benchmark.

Since all R2R reference paths in the data generation process are the shortest path to the target., there is a certain contradiction between following on instructions and reaching the destination. For properly evaluating consistency, dataset is supposed to be larger and has more diverse reference paths. To address the lack of path variety, Jain et al. [29] propose a data augmentation strategy that generates longer and more tortuous paths without additional human or low-fidelity machine annotation, which is more challenging dataset named Room-for-Room (R4R). Inspired by R4R, Zhu et al. [31] create two datasets of longer navigation tasks, Room-6-Room (R6R) and Room-8-Room (R8R).

In R2R task, the agent can only move on a fixed traversable nodes, rather than freely in the environment. Therefore, Krantz et al. [32] propose Vision-and-Language Navigation in Continuous Environments (VLN-CE) task, where the agent can move freely in the environment rather than transporting between pre-defined navigable nodes. This task setting introduces many challenges that are ignored in previous work.

For solving this problem, Chen et al. [148] propose a modular approach using topological maps, since the conventional end-to-end approaches are struggle in freely traversable environments. They decomposed VLN-CE tasks in two stages: planning and control. During the exploration, topological map representation is built and used on navigation plan stage. The local controller receives the navigation plan and generates low-level discrete action. Wang et al. [149] point out that different robots often equipped with various camera configurations, these differences make it difficult to directly transfer the learned navigation skill between robots. Consequently, a generalization strategy is proposed for visual perception based on meta-learning, which enables the agent to quickly adapt to a new camera configuration through few-shot learning.

Ku et al. [30] introduce Room-Across-Room (RxR), a multilingual extension of R2R task containg English, Hindi, and Telugu. The size, scope and detail of RxR has dramatically expands comparing with R2R dataset, which contains 126K instructions covering 16.5K sampled guide paths and 126K human follower demonstration paths. Every instruction is accompanied by a follower demonstration, including a perspective camera pose trace that shows a play-by-play account of how a human interpreted the instructions given their position and progress through the path.

Yan et al. [33] collect a Cross Lingual R2R dataset, which extends the original benchmark with corresponding Chinese instructions. Then they propose a principled meta-learning method that dynamically utilizes the augmented machine translation data for zero-shot cross-lingual VLN.

Chen et al. [34] propose Touchdown, a dataset for natural language navigation and spatial reasoning using real-life visual observations. They define two tasks (i.e., navigation and spatial description resolution) that require the agent to address a diverse set of reasoning and learning challenges. This is the first large scale outdoor VLN task. In this task, the agent receives a 360°RGB panoramic images when it moves to every navigable point, which is connected with undirected navigation graph. The environment includes $29{,}641$ panoramic images and $61{,}319$ edges from New York City.

Xiang et al. [150] focus on endowing agent with recognizing ability and stopping at the correct location in complicated outdoor environments. They introduce Learning to Stop module to address the issues, which is a simple and model-agnostic module that can be facilely added into other models to improve their navigation performances.

For facilitating the experiment, Zhu et al. [151] divide the original StreetLearn dataset into a small part, i.e., Manh-50, which mainly covers the Manhattan area with 31K training data. In addition, Multimodal Text Style Transfer learning approach is proposed to generate style-modified instructions for external resources and address the data scarcity issue.

For improving the end-to-end outdoor VLN research, Mirowski et al. [37] present StreetLearn task based on Google Street View, which is a egocentric photo-realistic interactive outdoor navigation task.  Mehta et al. [152] are publicly releasing the 29k raw Street View panoramic images needed for Touchdown. Evaluation metrics for this task are Task Completion, Shortest-path distance, Success weighted by Edit Distance, Normalized Dynamic Time Warping and Success weighted Dynamic Time Warping. Due to the length limitation of this paper, details of these metrics can be found in [152].  Mirowski et al. [153] use deep reinforcement learning method to address outdoor city navigation issue. Based on the images and connectivity on Google Street View, a duel pathway navigation model is proposed with interactive environment.

• •

  StreetNav. StreetNav is a extension of StreetLearn proposed by Hermann et al. [36]. The main difference is in StreetNav, driving instructions from Google Maps by randomly sampling start and goal positions are added to dataset, which likes autonomous driving environment.

• •

  Talk to Nav. Vasudevan et al. [154] develop an interactive visual navigation environment based on Google Street View named Talk to Nav dataset with 10,714 routes. They also design an effective model to create large-scale navigational instructions over long-range city environments.

• •

  Street View. Based on Google Street View,  Cirik et al. [155] sample 100k routes in 100 regions in 10 U.S cities to build an outdoor instructions following task environment. They argue outdoor navigation is more challenging task because the outdoor environment is more chaotic and objects are more diverse.

• •

  RUN. Aiming to explaining navigation instructions based on real, dense, and urban map, Paz-Argaman and Tsarfaty [35] propose Realistic Urban Navigation (RUN) task, which has 2,515 instructions annotated by Amazon Mechanical Turk works.

• •

  ARRAMON. Kim et al. [38] propose ARRAMON task, which contains two sub-tasks: object collecting and object referring expression comprehension. During this task, the agent is required to find and collect different target objects one by one through natural language instruction-based navigation in a complex synthetic outdoor environment.

CEREALBAR [39] simulates a scenario, where a leader and a follower collaboratively select cards to earn points. If valid set of card is collected, the players can earn a point. The leader has full observability on the map, and are plausible to plan the path for the follower and gives instructions to direct the way. The follower can only see a first-person view of the environment, and cannot respond to the leader. The performance is measured by correct execution of instructions and the overall game reward.

To solve this task, Suhr et al. [39] introduce a learning approach focusing on recovery from cascading errors between the sequential instructions, and modeling methods to explicitly reason about instructions with multiple goals.

HANNA [41] defines a task for a human to find objects in an indoor environment. In this process, the human can ask a mobile agent via natural language. The task is a high-level command (“find [object(s)]”), modeling the general case when the requester does not need know how to accomplish a task when requesting it. HANNA uses the Matterport3D simulator to photo-realistically emulate a first-person view while navigating in indoor environments.

To address the HANNA problem, Nguyen and III [41] develop a hierarchical decision model via a memory-augmented neural agent. Imitation learning is used to teach the agent to avoid past mistakes and make future progress.

Based on Matterport3D simulator, Nguyen et al. [40] introduces the Vision-based Navigation with Language-based Assistance (VNLA) task, where an agent has visual perception in a photo-realistic indoor environment and is guided to find objects in the environment via instructions.

Furthermore, they construct ASKNAV, a dataset for the VNLA task. After filtering out labels that occur less than five times, 289 object labels and 26 room labels are obtained. The data is defined as a tuple (environment, start pose, goal viewpoints, end-goal). Nguyen et al. [40] develop a general framework with imitation learning to extend the framework for indirect intervention.

de Vries et al. [44] introduces a navigation task for tourist service and constructs a large-scale dialogue dataset named “Talk The Walk”. The tourist is located in a virtual 2D grid environment in New York City and has 360-views of the neighborhood blocks. The guide has an abstracted semantic map of the blocks and the target location is unambiguously shown to the guide from the start. A guide aims to interact with a “tourist”via natural language to help the latter navigate towards the correct location.

de Vries et al. [44] decouples the task of tourist localization from the task and introduces a Masked Attention for Spatial Convolutions (MASC) mechanism to ground the instructions from tourist into the guide’s map.

Robot navigation in real environments are expected to use natural language to communication and understand human’s meaning. To study this challenge, Thomason et al. [42] introduce CVDN, a human to human dialogues situated in simulated, photo-realistic home environments. For training agent to search a goal location in an environment, they define the Navigation from Dialog History (NDH) task. In NDH, an agent is learned to predict navigation actions and find an object given a dialog history between human in previously unseen environments. NDH task can be used for training agents in navigation, question asking, and question answering problems. Thomason et al. [42] formulated a Seq2Seq model to encode an entire dialog history to address the NDH task, whose outputs are actions in the environment. Zhu et al. [156] propose a Cross-modal Memory Network (CMN) to remember the historical information. CMN uses a language memory module to learn latent relationships between the current conversation and a dialog history and uses a visual memory module to learns to associate the current visual views and the previous navigation actions. Roman Roman et al. [157] divide multi-turn NDH task into three sub-tasks: question generation, question answering, and navigation and implement them with three Seq2Seq models, i.e., a Questioner, a Guide and a Navigator. The progress agents are trained with reward signals from navigation actions, question and answer generation.

Base on the R2R task [10], Chi et al. [43] develop an interactive learning framework to allow the agent to ask for users’ help when needed. And they use reinforcement learning with a proposed reward shaping term as the initial model, which enables the agent to ask questions only when necessary.

Banerjee et al. [45] decouple the cooperative communication task into Localization from Dialog History (LDH) and NDH task, where a driver agent must localize in the global map or navigate towards the next target object according to the visual observations and the dialog with the commander. Different from other tasks, the environment and the agent are physical, while the sensor observations are camera RGB images. Moreover, Banerjee et al. [45] evaluate human performance on the LDH task and create an initial, Seq2Seq model for the NDH task.

The problem is mainly reflected in two aspects: one is that when trained VLN model is transferred from seen scenes to unseen scenes, the model performance will drop significantly; besides, when the model is transferred from one task to another VLN task, the performance will also decrease. The possible reason is that the data distributions in different environments vary greatly; hence, the navigation model trained in one environment is difficult to adapt to other environments. Another explanation may be that there is a certain degree of over-fitting in the supervised learning method.

A lot of work [95, 133] has demonstrated that through data augmentation, generating more training data can improve the performance of VLN model. For example,  Wijmans et al. [83] have proved that the point cloud perception can help an EQA agent find its target. However, compared with the real high-quality data, the artificially generated data is still very limited to the model. Unfortunately, collecting real data is still a costly and labor-intensive work, since the methods to collect photo-realistic 3D-scene datasets require scanning real buildings by means of expensive photogrammetry devices [158] such as Matterport 3D scanner, Meshroom or even mobile 3D scanning applications.

In the current VLN datasets, the interaction between the agent and the environment is simplified, only involving basic operations such as opening a microwave oven, moving an object, or reaching a designated location. The movement action that can be taken are restricted to a specific range (such as moving forward, turning left and turning right, or teleporting between navigable point). Although more complex interaction forms such as “bring me the pillow next to the stool” have appeared in some latest research, it is obviously far from the interaction and movement forms of the real environment, because they simplify many physical constraints in the real environment. In the future, a simulator with advanced physics features such as cloth, fluid and soft-body physics will improve the realism of intelligent VLN agent.

Knowledge makes human smarter, for instance, we can summarize experience from daily life as knowledge and use it in the future. Intuitively, knowledge will imporve the model performance in most of AI tasks, which has been proven effective recently [9, 159]. Therefore, how to introduce knowledge into the VLN model and make it work effectively is a problem that researchers need to think about in the future. Some possible ways are as following:

• •

  Algorithmic knowledge. Many algorithms in machine learning can accumulate knowledge through continuous training such as the knowledge about environmental dynamics, spatial representation or physical effects of actions, so that they can better adapt to new environments. Wang et al. [149] used the Model-Agnostic Meta-Learning (MAML) algorithms to improve model generalization in task VLN-CE, which enables the agent to get adaptive quickly to a new camera configuration with a few shots. Another work [160] adopted MAML to learn a self-supervised loss, which can be used in new environments directly without extra training. Besides, lifelong learning [161], transfer learning [106], curriculum learning [31] and external prior knowledge [91, 162] are introduced to help solving VLN tasks.

• •

  External knowledge base. Humans often need the help of commonsense when solving similar problems. For example, with the commonsense knowledge that a toilet is typically found in the washroom, the agent shouldn’t waste time in the living room to find a toilet. Moreover, such knowledge is likely to still hold in previously unseen environments, making it possible to achieving better generalization ability. Some existing pre-training models can play the role of some common sense knowledge bases, such as some image-text pairs, but they are still far from the ability to have common sense reasoning. How to make the agent have common sense knowledge and make it reason on the basis is a very critical problem.

• •

  Cognitive architecture. Humans can perform tasks like VLN excellently, so some principles in cognitive science and human brain can be used for heuristic model building. For example, memory can enhance the reasoning ability of VLN model. In brain areas, such as the hippocampus, have intertwined roles in spatial memory and other cognitive functions. Inspired by this, Hu et al. [163] use fusion adaptive resonance theory (ART) networks to model spatial representation and related memory to enhance the visual navigation model. Other useful cognitive theories like Soar, ACT-R are powerful as well, and no work based on them until now.

It is promising for a physical agent to complete VLN tasks in the real world, since there are many practical application scenarios. For instance, in search-and-rescue tasks, robots might using vision clues to search and language to interact with with human victims [164]. At present, the research on VLN task is still in the simulated environment. The advantage of the simulation platform is that it can easily validate the model, and it is convenient and quick to use. However, some important physical constraints in many real-world cases cannot be characterized well by simulators and have been discarded from consideration. Therefore, there is still a large gap between the simulated environments and the real world, hindering the transfer of progress made on simulation platforms to real world scenarios. Some progress has been made in this direction. For example, Anderson et al. [144] have used a TurtleBot2 mobile robot equipped with a 2D laser scanner and a 360° RGB camera to finish a VLN task. In the future, we believe that building a simulated environment integrated with more realistic physics features, such as tactile perception [165], human-level motor control [166] and audio inputs [167] will improve the performance of VLN model in real world.

Besides, training VLN models usually need intensive computation resources, which makes it prohibitive to deploy such models on portable devices, such as small drones [71]. One direction to solve the problem is to design more lightweight network structure, where operations or layers with low computing load and low memory consumption are included. For example, the model can be compressed by quantizing the network weight or being distilled to a light model to reduce the complexity. Other possible solutions could be model pruning method and Huffman coding for weights of the model.