DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

We collect human annotations on Amazon Mechanical Turk for questions and answers. Each instance in the dataset is a tuple of question type $q\in Q$ asking about a specific property of an object $o\in O$ ($o$ could be empty for questions not related to objects) and a human answer $a\in A$ for the question at the beginning of the task. Figure 2 shows the data collection interface. The hybrid data collection (HDC) process is as follows:

1. 1.

   Each annotator watches a 10 second video clip, which displays the state of the environment right before the task. Annotators also see the original language instructions for the task.

2. 2.

   The annotator then selects one pertinent question (from several predefined questions) that they think may help the agent complete the task. The annotator may also type in a question of their own choosing if none of the provided questions are a good fit.

3. 3.

   The annotator watches a second video clip – this time of an expert agent performing the task.

4. 4.

   The annotator answers their own question and provides feedback (yes/no) on whether asking the question was necessary in the given scenario.

To generate the predefined question choices for the annotator to choose from, we consider three types of questions $Q$, related to the location and appearance of the query object $o$ that needs to be interacted with to finish the task, and the relative direction between the agent’s current position and the target position to guide the navigation:

1. 1.

   Location: where is $o$?

2. 2.

   Appearance: what does $o$ look like?

3. 3.

   Direction: which direction should I turn to?

Given a natural language instruction (e.g. Put the egg in the microwave), we parse it and extract all the nouns (e.g. egg and microwave). We insert each noun into one of the question templates to generate questions (e.g. Where is the egg? and What does the microwave look like?).

We collected human questions and answers for 29,376 sub-goals (e.g., Take the knife to the counter) on Amazon Mechanical Turk via crowd-sourcing. For each sub-goal, we ask two different annotators to provide questions and answers. And we see a modest level of agreement in question selection between different annotators (Fleiss’ $\kappa=0.13$). To ensure the quality of the dataset, we first remove invalid annotations when the annotation time is less than 15 seconds. A worker is compensated $0.25 for each Human Intelligence Task (HIT); the dataset collection cost is $\sim$ $10K. The dataset is gathered over 112 rooms and 80 types of objects. Each human answer contains 6.73 words on average. A lexical complexity analysis on the human answers [28] shows that the number of different words (NDW) in the answers are 7915. The lexical sophistication (the proportion of words not in the 2000 most frequent words in the American National Corpus) is 49%. Example human questions and answers are shown in Figure 3.

In addition to the human answers, we build an oracle that provides templated answers which can be easier for the agent to understand. In Figure 3, we show answers generated by the oracle for some task examples. To create the oracle, we take advantage of the ground-truth states of objects and the agent in the simulated environment: (i) to answer object location questions, we compute the direction of the object relative to the agent, and the receptacle that contains the object; (ii) to answer object appearance questions, we focus on its color and material. Object material is extracted from scene metadata in the underlying simulation environment (AI2-Thor). For object color, we extract object pixel RGB values from images, and map them to color names; (iii) for direction questions, we check the agent’s location at the end of the task in the ground-truth action sequences, and compare it to the agent’s initial location. Based on these metadata, we use language templates to generate answers. Some example templates include:

• •

  Location: The $o$ is to your [direction] in/on the [container].

• •

  Appearance: The $o$ is [color] and made of [material].

• •

  Direction: You should turn [direction] / You don’t need to move.

Each task in ALFRED has a goal, split into multiple sub-goals. Each sub-goal requires the agent to manipulate some objects or move to a target location. Two types of language instructions are given. The high-level task instruction (e.g., Move a knife to the sink) describes the overall goal. The step-by-step instructions (e.g., Pick up the knife) guide the agent to complete each sub-goal. ALFRED exhibits clear patterns in both high-level task structures and sub-goal action sequences: tasks of the same type require very similar sub-goal sequences to complete them, and sub-goals of the same type (especially manipulation sub-goals) have almost fixed action sequences. The limited variety in tasks and sub-goals precludes instructions understanding – allowing models that directly classify the task type from high-level task instructions alone to perform well, without even using the step-by-step instructions [3, 4].

To get rid of the strong patterns in both high-level task structures and sub-goal actions in ALFRED, DialFRED uses data augmentation to increase the number of task types, and focuses on instruction following at the sub-goal level to encourage learning from instructions for each task. We also introduce augmentations on the original ALFRED sub-goal instructions to add ambiguities in language so that the sequence of actions cannot be fully determined by only focusing on the instruction – reasoning based on knowledge of the environment state is needed. We show examples for DialFRED tasks and instructions in Figure 3.

The goal of the questioner is to ask necessary questions to help the performer finish the task. Thus we fine-tune the questioner using reinforcement learning to learn when to ask a question and what question(s) to ask on the validation seen split. The learning system for question asking is modeled as a Markov Decision Process, specified by a tuple $<S,A,T,R>$, where $S$ is the state space (including the language instructions and visual observations), $A=Q\times O$ is the action space of all possible questions (in our cases a combination of question types $Q$ and target object $O$), $T(s^{\prime}|s,a)$ is the transition function encoding how the performer can advance the task given the question and its corresponding answers, and $R(s,a)$ is the reward function encoding the reward for each $(s,a)$ pair. Our questioner model can be viewed as a policy network $\pi_{\theta}(s,a)=p(a|s;\theta)$ mapping each state vector $s$ to a stochastic questioning policy, which can be learned based on the reward function to strike a balance between the number of questions and performance gain.

We adopt the following reward function structure to address this trade-off: reward for task completion $r_{suc}=1.0$, penalty for each step $r_{step}=-0.01$, penalty per question asked $r_{q}=-0.05$, penalty per invalid question $r_{invalid}=-0.1$. Some questions generated by the questioner cannot be answered by the oracle, e.g. the appearance of task-irrelevant objects. Thus we add a penalty for invalid questions. The reward forms the actor critic loss function [32], the gradient of which is used to update the model so that it can learn to ask necessary questions at correct time. The performer model is not updated during questioner fine-tuning.

Inspired by [27], we implemented a questioner based on model confusion. The idea is that if the performer is not confident, the output action distribution would have high entropy; model confusion could be a good heuristic to know when to ask a question. We sort action probabilities in decreasing order; an agent is confused if the minimum difference between the top two actions is less than a threshold $\epsilon$ throughout the action sequences:

where the threshold $\epsilon$ is used to control the degree of confusion for asking questions. In practice, we set the confusion threshold $\epsilon=0.5$ in the experiment.

We evaluate model performance using success rate and path weighted success rate. In DialFRED, task success $s$ is a binary indicator of whether the task goal conditions have been achieved. For example, the task “clean the fork” requires the states of the fork to be changed from “dirty” to “clean”. Task success $s$ is defined as 1 if the goal conditions have been fully achieved, and 0 otherwise. Success rate is calculated via averaging $s$ across all tasks. Path weighted success rate further takes into consideration the number of actions the robot takes. The path weighted success score for a task is calculated by:

where $s$ is the original task success indicator (0 or 1), $\hat{L}$ is the number of steps taken by the agent to complete the task, and $L^{*}$ is the number of steps in the expert demonstration. Intuitively, the more steps the robot took, the lower the score. Again, the path weighted success rate is calculated via averaging $p_{s}$ across all tasks. To understand the trade-off between the number of questions asked and task performance, we measure the number of questions throughout the task.

We implemented 6 baselines and evaluated their performance on our augmented dataset (Table I). In all baselines we use the Episodic Transformer model as the performer. In baselines 2–6, the performer is trained using imitation learning on expert action sequences with language instructions and QAs. In baselines 5–6, the questioner is pre-trained on human dialogues in the training split, and fine-tuned using reinforcement learning on the valid seen split. Baseline details are itemized below:

1. 1.

   In this baseline, no questions are allowed, the performer is trained to predict the action sequences based on the language instruction and visual observations.

2. 2.

   In addition to the instruction, the performer gets all valid QAs at the beginning of the task, including a combination of all three question types (i.e. location, appearance and direction) and objects mentioned in the instruction.

3. 3.

   Questions are sampled randomly based on type: 25% for each of the 3 types and 25% for no question. Given a selected question type, questions and answers for all relevant objects are given to the performer in addition to the instructions.

4. 4.

   When the action sequences generated by the performer satisfy the model confusion criterion (Equation 1), the performer is randomly provided with a valid QA as input.

5. 5.

   The questioner is fine-tuned using reinforcement learning to learn whether to ask a question and what question to ask at the beginning of a task. The performer uses the QA to generate the action sequence to finish the task.

6. 6.

   Similar to baseline 5, the questioner is fine-tuned using reinforcement learning, but now it can ask questions in the middle of the task. Given the instruction and previous QAs, we roll out the performer for 5 steps, following which the questioner is allowed to generate new questions and thus get new answers.

We display (Table I) the task performance (success rate) on validation seen and unseen splits for all baseline models. Comparing the results of baselines 1–3, we see that adding QAs to the instructions improves task performance on both splits. Comparing the results of baselines 2–5, we see that the fine-tuned questioner achieves the best performance, with a smaller number of questions. Comparing the results of two reinforcement learning baselines (5,6), we see that enabling the agent to ask questions in the middle of the task improves performance at the cost of more questions and answers. In addition, the random MC baseline achieves reasonable performance on the valid seen split, but not on the valid unseen split. Since the performer is pre-trained on the training split, which has the same scene as the valid seen split, and is given random combinations of different types of QAs as input, it is not surprising that the random MC baseline does not generalize well to unseen environments.