Virtual, Augmented, and Mixed Reality for Human-Robot Interaction: A Survey and Virtual Design Element Taxonomy

2D Monitor Video Displays: 2D monitor video displays provide a means of overlaying AR synthetic imagery on real images or video feeds, as “window-on-the-world” displays. Due to the inherent 2D nature of traditional monitors, users of these displays are not able to experience depth when viewing interfaces on these displays. Additionally, these displays tether users to computer terminals, and do not allow for a hands-free, free-roam of environments in which AR imagery are being added.

3D Monitor Video Displays: 3D monitor video displays are similar to the above AR 2D monitor video displays, with the difference being that users can see interfaces with depth. Users can see with depth with either glasses-free 3D monitors or with 3D glasses (i.e., anaglyph, polorized, or active shutter).

Tablet Displays: Tablet displays are also similar to AR 2D monitor video displays, although a major difference being users can freely explore their environments due to the portable nature of tablets. Additionally, tablets provide users with touch-based interactions, which is uncommon in the monitor-based displays.

Projectors: Projector-based displays provide AR synthetic imagery to a user’s environment, that users can see without the need of special equipment, such as glasses or head-mounted displays (HMDs), and also allow users to freely roam their environment. Drawbacks to this display type include: the projected imagery is inherently 2D; occlusions (from the environment, users, or robots) disrupt and/or block the AR imagery; and the projected imagery can get washed out in environments that are too bright.

CAVEs: CAVEs, or Cave Automatic Virtual Environments, are VR displays where users are placed in a three to six walled environment. The walls of the environment display 3D imagery when paired with 3D glasses allowing users to be immersed in AV or VR environments. CAVE’s allow for multiple users to experience the VR or AV environments, but unfortunately are expensive, confining, and immovable.

VR HMDs: VR HMDs are 3D stereoscopic displays worn on users’ heads to fully immerse users in VR or AV environments. Recent advances in VR HMD technology have allowed users to not only control the display with head motion, but with hand and body motion as well, permitting users to be hands-free while able to free to walk around synthetic environments naturally with their own body. A downside to these displays is that they enclose users off from the real world without an option of seeing reality from the display while worn.

Optical See-Through HMDs: Optical See-Through HMDs present AR through transparent lenses, allowing users to see virtual imagery overlaid on the real world. Similarly, to VR HMD’s, these displays provide users with hands-free, environmental free-roam experience. However, current technology is restricted to narrow field-of-views to see the virtual imagery. Additionally, the imagery provided by these HMDs are easily washed out in bright lights.

Video Pass-Through HMDs: Video pass-through HMDs are unique in that they can provide AR, AV, and VR experiences to users. By mounting a stereo cameras to the front of a VR HMD, video pass-through HMD’s can pipe video imagery from the real outside world to the dual lens within the HMD. The video feed can be intercepted between camera and lens, allowing for virtual imagery being added to the captured image frames. This process of displaying AR imagery allows these HMD displays to show AR imagery in environments with bright light, such as the outdoors.

Fiducial Markers: Fiducial markers are images that have their optical properties known by a VAM interface beforehand and act as visual reference points for the systems. When a VAM interface’s camera detects a marker, it can determine the marker’s relative pose to that of the camera. These markers are often placed in a robot’s environment, on robots, or on objects within a robots environment. This method of frame rectification is popular due to its high portability and low-cost (Hashimoto et al., 2011; Borrero and Márquez, 2012; Frank et al., 2017; Kobayashi et al., 2007).

Motion Capture Cameras: Motion capture cameras can obtain poses of robots, users, and objects in an environment with high precision by tracking patterns of infrared reflecting markers. Unfortunately, these systems are expensive and immovable, often making interfaces that use this method of frame rectification constrained to laboratory environments (Walker et al., 2018; Hedayati et al., 2018; Walker et al., 2019).

Odometry: Another method of unifying coordinate frames between user and robot is by using odometry (including visual odometry with computer vision algorithms such as SLAM (Mur-Artal et al., 2015)). If agent coordinate frames are initially synced at the initialization of a VAM-HRI interface, odometry can track the relative pose changes agents have undergone over time, which can be used to maintain a rectified coordinate system (Reardon et al., 2018; Gregory et al., 2019).

ICP Alignment: Coordinate frame rectification is also made possible by using point cloud alignment algorithms, such as Iterative Closest Point (ICP) (Segal et al., 2009), which can take in separately collected point clouds and output the transformation between the two perspectives. This method requires all agents to have some overlap between their collected point clouds which makes this method most suitable for initial frame synchronization between multiple agents that is followed by odometry-based frame rectification methods above (Gregory et al., 2019; Reardon et al., 2018).

Machine Learning Image-Based Pose Estimation: Relative poses between agents can also be found through machine learning algorithms, trained to estimate an object’s pose through image frames. Current technology limits this method to known, simple objects; however, as machine learning algorithms strengthen (Xiang et al., 2017) it is feasible to imagine this method becoming more popular for agent frame rectification, especially if it means fiducial markers or motion capture cameras can be eliminated (Bolano et al., 2019).

Direct HMD Teleoperation Interfaces: Direct HMD teleoperation interfaces live-stream stereo video feeds from remote robots to the HMD’s lenses. This process allows users to see from the robot’s ‘eyes’ with immersive 3D stereoscopic vision as if they were embodying the remote robot, especially if user head motions control robot head motions with a one-to-one mapping. These HMD-based interfaces have shown to significantly improve robot teleoperation tasks. Additionally, if virtual imagery is overlaid on the video stream, the interfaces become a AR interface (Higuchi and Rekimoto, 2013). However, there is a significant drawback associated with Direct HMD Interfaces that stems from both miscues from the user’s proprioceptive system and the communication delays inherently found with current network technology. When a remote robot moves and the local user does not move, the user’s proprioceptive system receives conflicting cues (visual cues of movement vs. no body motion detected) causing nausea. The same happens in reverse when a local user turns their head and robot’s head does not immediately turn to match the movement (due to mechanical limitations or communication delays), which creates conflicting proprioceptive cues (no visual cues of movement vs. body motion detected) that also cause nausea.

AV HMD Teleoperation Interfaces: To mitigate the nauseating effects of the Direct HMD Interfaces, two AV HMD Teleoperation Interface paradigms have arisen from research in the VAM-HRI field: Virtual Control Room and Cyber-Physical Interfaces (Lipton et al., 2017). In both interface styles, the state of the user’s eyes are decoupled from the robotic systems state to remove the conflicting proprioceptive system cues. By placing the user in a AV environment, the users’ eyes are represented by virtual cameras in the virtual space that move freely with the users head and body movements. This decoupling method helps mitigate nausea caused by communications/hardware delays and/or imperfect mappings between user head motion and robot head motion.

Direct Manipulation: The paradigm of Direct Manipulation leverages more traditional methods to directly control robots and utilizes 3D translation and/or rotation input from either physical or virtual source, commonly to send teleoperation commands to end effectors or navigational systems. Direct manipulation from physical inputs include body/head tracking and VAM-based 3D controllers (Whitney et al., 2018). Virtual controllers can act as metaphors for existing physical control input devices (i.e., levers, handles, joysticks, etc.) (Hashimoto et al., 2011) or utilize novel designs unconstrained by physics such as floating control spheres (Krupke et al., 2016).

Environment Markup: Under the Environment Markup paradigm, users send commands to robots by adding virtual annotations to a robot’s environment. Examples of such environmentally-anchored annotations include waypoints, trajectories, and planned future poses of manipulable objects. These annotations can take the form of a simple single command or can be chained/combined together to form a series of commands or a singular complex command (Chan et al., 2018; Ishii et al., 2009).

Digital Twins: In contrast to commands sent under the Environment Markup paradigm, Digital Twin command sequencing does not add virtual annotations to a robot’s environment. Instead, command sequences are generated by the manipulation of a digital twin which is a virtual replica (or representation) of a real object or robot. For example, in the case of a robot with a digital twin, a user could press a virtual button on the robot’s digital twin, at which point, the real robot would respond as if its own physical button was pressed. Additionally, robot teleoperation can be achieved by directly manipulating the robot digital twin (arm, body, etc.), after which, the real robot imitates the action taken by the digital twin (e.g., the robot moves itself in the exact trajectory taken by the digital twin or moves its end effector to the final position taken by the digital twin). Real object manipulation is performed in a similar manner, but in this case, a robot mimics a user’s object digital twin manipulations on the associated real object (Sun et al., 2020; Frank et al., 2017; Hashimoto et al., 2011; Krupke et al., 2018).

Visualization Robots improve user understandings of a robot’s current state or future actions but do not afford any two-way interactions with users (i.e., users cannot direct input to the graphical representation). These VDEs typically model a robot’s 3D morphology, in whole or in part. Uses include providing users with a means of understanding a robot’s current state in which case a virtual 3D model of a real robot mimics a physical robot’s joint configurations in real time. Visualization Robots are particularly useful in situations of limited situational awareness when the user cannot directly see the physical robot (or portions of the robot), such as in remote teleoperation tasks (Kot and Novák, 2014) (see Figure 3-A). Visualization Robot VDEs can also provide users with a preview of proposed or planned robot motion by overlaying a 3D robot model onto the environment to allow for better means to assess how a robot will navigate through an environment and whether it will successfully travel to a desired location without collisions. Finally, Visualization Robots can also show the locations of robots that are partially or fully occluded by the environment such as behind a wall or door (Rosen et al., 2020) (see Figures 3-B and 5-D).

Simulated Robots are not linked to a specific instance of a physical robot and are instead independent robot simulations used to evaluate robots in fully virtual settings when using a real robot is not ideal, such as when robot hardware is unavailable, unsafe, has limited battery life, and/or faces physical depreciation when operated repeatedly (as required for sample-inefficient learning algorithms and/or interaction studies). These VDEs are also used for evaluating simulated interactions with autonomous or manually teleoperated robots, e.g., in situations where collocated interaction is infeasible or hazardous for humans (i.e., working near large industrial robots, testing space exploration robots in zero gravity environments, etc.). Simulated Robots are also useful for user training without putting robot hardware at risk of being damaged by inexperienced users and for training real robots through virtual-to-real-world transfer learning techniques (Iuzzolino et al., 2018) that require direct user interaction, such as learning from demonstration (Argall et al., 2009; De Pace et al., 2018; Stilman et al., 2005; Meyer zu Borgsen et al., 2018) (see Figure 3-D).

Robot Digital Twins operate in tandem with real robots and provide users with an immersive virtual robot that can be interacted with in lieu of a real, physical robot. By interacting first with a Robot Digital Twin, users can better predict how their actions will affect the system and offer foresight into the eventual pose and position of the physical robot as it mimics the actions taken by the virtual robot. For instance, mappings between the Robot Digital Twin and real robot include instantaneous duplication (the physical robot moves to match the Robot Digital Twin’s position/attitude immediately), delayed duplication (the physical robot moves to match the Robot Digital Twin’s position/attitude after a set period of time), confirmed duplication (the physical robot matches the Robot Digital Twin when triggered by the user), and more planning-oriented systems, such as using the Robot Digital Twin to denote waypoints or actions for future execution by the physical robot (Krupke et al., 2018; Sun et al., 2020; Walker et al., 2019; Hashimoto et al., 2011) (see Figure 3-C).

Panels & Buttons are virtual control objects that look and act like panels and buttons found in real life and 2D GUIs (e.g., buttons, sliders, switches, etc.) (Li et al., 2019) (see Figure 4-D).

Controllers emulate physical 3D input devices that leverages the 3D capabilities of VAM displays. Controllers often act as metaphors for existing physical control input devices (i.e., levers, handles, joysticks, etc.). However, Controller VDEs have also opened a nascent design space that allows robot designers to create interface input devices that are unconstrained by physics in the form of objects unable to be implemented or on earth or in reality, such as manipulable control toruses (Hashimoto et al., 2011) or floating control spheres (Krupke et al., 2016), allowing for robot interactions that would otherwise be impossible to implement and/or evaluate with traditional non-VAM interfaces (see Figure 5-F).

Simulated Agents simulate, with virtual imagery, physical entities that normally have independent agency (e.g., autonomous robots, humans, animals, etc.). In the case of these simulations, the robot is not able to differentiate between real agents and Simulated Agents. The primary use of these VDEs is to enable robot testing and debugging without requiring the presence of key agents with whom robots would need to interact. For example, a large industrial robot might practice object handovers with a simulated human, without putting any human lives in harm’s way, while the robot’s developers observe and evaluate the mock interactions. Alternatively, a Simulated Agent VDE may be used in tandem with an autonomous Simulated Robot VDE for real humans to interact with, allowing for the testing of autonomous interactions with intelligent robots in situations where a physical robot is unavailable or currently infeasible (Meyer zu Borgsen et al., 2018) (see Figure 3-D).

Simulated Objects use virtual imagery to simulate the presence of physical objects in a robot’s environment. It is important to note that these nonexistent virtual objects hold no association with any real objects in a robot’s setting. These nonexistent objects can be used to simulate obstacles in debugging sessions with real robots (e.g., a virtual wall, table, chair, etc), without robot developers needing to procure physical objects, enabling rapid modification, deletion, and duplication of objects (Borrero and Márquez, 2012) (see Figure 5-G).

Simulated Environments use virtual imagery to synthetically create the presence of environment areas or terrains. These VDEs can be used to evaluate autonomous robot responses to hazardous terrain (e.g., loose gravel, sand, water features, etc.) without endangering robots. Simulated Environments can also be used to evaluate robot interactions in environments that are difficult to find or recreate on Earth such as a lunar Moon with decreased gravity. Finally, Simulated Environments can provide a realistic setting to evaluate interactions (e.g., object hand-offs between user and robot) between Simulated Robots and users (Meyer zu Borgsen et al., 2018) (see Figure 3-D).

Object Digital Twins act as virtual replicas of associated real objects to sequence actions to be taken on their real-world equivalents. These VDEs can allow users to preview actions to be taken on the real object prior to robot execution. For example, a real cup to be moved by a real robot might have a virtual cup overlaid on its current position. A user could then interact with the virtual cup and move it to a new location, fine tune its final placement, and then command a robot to move the real cup to the position of the virtual cup. This interaction pattern can also enable robot action previewing similar to (and potentially in conjunction with) Robot Digital Twins (Krupke et al., 2018; Frank et al., 2017) (see Figure 5-H).

Environment Digital Twins are virtual replicas of real environments rendered as a VAM-based visualization. These environments can be man-made structures/areas or outdoor terrain that are made to be exact replicas of their associated real world environment. As in Object Digital Twins, users could alter the state of the Environment Digital Twin, to have a robot take action on the real environment so its state matches its digital twin. Examples of such systems include interfaces that visualize real satellite terrain data as an Environment Digital Twin VDE to test and/or supervise aerial robot systems scouting a wildland forest fires across the associated real expanse of wilderness (Omidshafiei et al., 2015) (see Figure 4-B).

Cosmetic Alterations alter the color, pattern, or texture of the robot’s physical surfaces. The manipulation of robot surfaces enable new interaction patterns. These VDEs are considered cosmetic with respect to the robot’s morphology and can be combined with additional VDEs to provide a function. For instance, changing the color of a robot arm in a manufacturing context might call attention to a malfunctioning actuator, indicate a hot surface temperature, or discourage touching. In an educational setting, superficial alternations might change the texture of a robot arm to look soft or furry to encourage interaction with children. Additionally, as robots increasingly expand to new consumer domains in the near future, designers could use Cosmetic Alterations to make robots more eye catching in public spaces or enable end-user customization of personal robots in private living spaces to match home decor or personal taste (Avalle et al., 2019) (see Figure 3-E).

Special Effect Alterations add virtual imagery around robots’ physical surfaces to change their appearance indirectly. These effects can take various forms such as a glow effect added around a robot’s body, virtual streamers that render behind a robot’s arm as it moves, virtual flames that spray out of a robot’s end effectors, or virtual light sources that indirectly alter the reflective appearance of a robot’s physical surface. Although we did not come across this VDE during our literature survey, VAM-HRI is still a growing field and we envision this VDE holding value for manipulating human-robot interactions (i.e., adding virtual sparkles to a robot to make it more engaging to children in educational settings).

Body Extensions add virtual parts to a robot without changing its underlying form such as an aerial robot is still recognizable as a UAV even if a virtual arm is added to its chassis, which would not be the case if virtual imagery were overlaid on the aerial robot to make it look like a floating robotic eye (Walker et al., 2018) (see Figure 3-H). Extensions do not necessarily need to be traditional robot parts and might instead appear as human heads, animal limbs, or even imagined parts like magical wings (Cao et al., 2018; Groechel et al., 2019) (see Figure 3-F).

Body Diminishments visually remove, rather than add, portions of a robot (e.g., grippers, heads, arms, wheels) through diminished reality (DR) techniques (Mori et al., 2017). An important use of this VDE is to resolve teleoperation occlusions that occur when a robot arm blocks the line-of-sight between its camera and the object being manipulated (Taylor et al., [n.d.]) (see Figure 3-G).

Form Transformations overlay virtual imagery onto real robots to change the robot’s underlying form and/or make it appear as something other than a robot entirely. Similar to Robot Diminishments, this VDE can change the form of the robot to make it more or less appealing to a targeted user-groups or utilize new communication methods, depending on the designer’s intentions. These form alterations need not be limited to that of new mechanical forms, but can include any form such as that of a human, animal, or fictional character, all of varying degrees of realism (Walker et al., 2018; Zhang et al., 2019) (see Figures 3-H).

Internal Reading VDEs display data returned from internal sensors (e.g., battery levels, robot temperatures, wheel speeds). Making this information more readily available to users may enhance situational awareness and prevent mishaps such as robots running out of battery in the middle of mission-critical tasks or traveling too fast through environmental hazards (Kot and Novák, 2014) (see Figure 3-A).

Internal Readiness VDEs display data about sensors and actuators, such as whether a sensor is ready to collect data or whether an actuator is ready to function, and if not, why (e.g., whether a sensor is disconnected, an actuator is experiencing a fault, etc.). Such information may improve debugging and help prevent robots from operating as black-boxes, with users left wondering why a robot is not functioning as expected (Avalle et al., 2019) (see Figure 3-E).

Robot Pose VDEs convey a robot’s knowledge of its own pose (i.e., configuration and orientation). These VDEs can take different forms such as a textual display of numerical joint angles, a 3D model of a State Visualization Virtual Robot, or a rendering of a virtual axis anchored to a robot’s joints (Kot and Novák, 2014; Nawab et al., 2007) (see Figure 4-A).

Robot Location VDEs convey where a robot is in the environment, e.g., as an occluded robot’s outline with a Spatial Visualization Virtual Robot or a virtual indicator on the other side of a wall or door, as a top-down radar-like display showing user and robot locations, or as an off-screen indicator to direct a user’s attention to a robot outside the current field-of-view (Chandan et al., 2019; Walker et al., 2018) (see Figure 5-D).

External Sensor Purviews are environment-anchored visualizations that show users where a robot’s external sensors (LiDAR, cameras, etc.) are collecting data and/or how and where those sensors are positioned (Hedayati et al., 2018; Kobayashi et al., 2007) (see Figure 4-B).

External Sensor Numerical Readings convey numerical data returned from a robot’s external sensors. These readings can be user– or environment–anchored and can be shown explicitly with digits or as more abstract visualizations such as progress bars, virtual thermometers, or virtual weight scales (Lipton et al., 2017; Cao et al., 2018) (see Figure 4-C). Note that the external sensors need not physically attached to a robot, such as data from motion capture cameras regarding the distance between a robot and a nearby object.

External Sensor Images & Videos allow the user to see remote environments or to see from a robot’s perspective. Images and videos can be presented as either user- or environment-anchored visualizations from cameras on the robot or in the robot’s environment. When stereo cameras are paired with a stereo interface (e.g., an HMD, 3D monitor, or CAVE), users can see the images and videos with depth, granting enhanced immersion and teleoperation capability (Lipton et al., 2017; Kot and Novák, 2014; Li et al., 2019) (see Figure 4-D).

External Sensor 3D Data convey depth information to recreate remote robots’ environments. These reconstructions can take various forms (e.g., point clouds, voxel maps, 3D meshes, etc.) and aim to present sensed depth data in a manner that allows users to perceive remote environments as if they were there in-person. Unlike Images and Video VDEs, the 3D reconstructions utilized by External Sensor 3D Data VDEs enable immersive and free exploration of a remote robot’s environment, without being restricted to the robot’s perspective (Sun et al., 2020; Rosen et al., 2020) (see Figure 3-B). As in the case of the previous two VDEs, the external sensors do not need to be physically attached to the robot and this VDE includes 3D data collected from user-worn HMDs, such as the spatial map generated from a HoloLens.

Sensed Spatial Regions, the first of three region-based VDEs, visualize regions that a robot has identified within its its environment. These visualized regions are produced by a robot through the analysis of sensed environment data, such as exploration frontiers or traversable vs. non-traversable areas depicted through an occupancy grid (unlike 3D reconstructions that do not perform any logical analysis and categorization of environmental regions). Regions can be annotated with information (i.e., estimated information gain if the area were to be explored) (Reardon et al., 2019) (see Figure 4-E).

Robot Inherent Spatial Regions are not informed by data sensed from the environment, but are instead inherent to the robot based on its form or operational mode. For example, these may depict the regions a robot can physically reach or show areas a robot may operate best in, such as the optimal area in which to perform an object handover (Frank et al., 2017) (see Figure 4-F).

User-Defined Spatial Regions are regions defined by user input, rather than by the robot’s sensors or physical constraints (e.g., a user drawing a bounding box on the floor of a robot’s environment with a tablet display). These regions have many potential uses, but are most commonly used to define areas a robot should not enter, in the form of virtual boundaries (Sprute et al., 2019a) (see Figure 4-G).

Entity Labels act as identifiers for entities known by a robotic system. These visualizations enable users to easily reference entities in spoken commands without the need for referring expressions such as enabling users to instruct robots through commands such as “pick up cube B” (Bolano et al., 2019; Sibirtseva et al., 2018) (see Figure 4-H). Additionally, these VDEs allow robots to label points of importance in the environment such as a room’s primary access point (Chandan et al., 2019) (see Figure 5-D)

Entity Attributes convey information a robot knows about an entity’s characteristics, such as whether an entity is heavy, delicate, or dangerous; information known about the entity’s affordances; the current state of an entity (e.g., an entity that’s too hot, in a dangerous location, still drying, sleeping, charging its battery, etc.); or an entity’s geometry and shape (e.g., optimal grasp points or surface normals) (Chan et al., 2018) (see Figure 5-A).

Entity Locations highlight the locations of entities within the robot’s environment through rings, arrows, bounding boxes, etc. This VDE is especially useful when the location of an entity is occluded by walls or containers or outside of the user’s field-of-view (Dima et al., 2020). These can also be used to highlight task- and dialogue-relevant entities, either by allowing robots to passively highlight entities that are of interest to the current task or the subject of the robot’s current attention, or to actively call interlocutors’ attention to entities in the same way that humans typically would through deictic gaze and deictic gesture (Williams et al., 2019a; Sibirtseva et al., 2018; Quintero et al., 2015; Bolano et al., 2019) (see Figures 4-H).

Entity Appearances show what an entity looks like, unlike Entity Location VDEs that communicate the location of an entity. These VDEs are primarily used when an entity is occluded from view and draws analogies to “X-ray vision” by showing users the appearance of real-life entities that are partially or fully visually hidden within the users’ environment (e.g., inside a box, behind a robot arm, or on the other side of a wall or door). This interface feature may be particularly useful in environments, such as warehouses, where objects are stored in sealed containers with contents only knowable through data representations that are exclusively computer-readable (e.g., barcodes) (Ganesan et al., 2018) (see Figure 5-B).

Headings, the first type of VDE in this class, do not show the actual path the robot or its manipulators will take, but simply the direction they are currently traveling in or will be traveling next. These visualizations commonly take the form of arrows pointing in the direction of planned movement; however, they can also take more unique forms (i.e., the utilization of a Form Transformation VDE to provide an eyeless robot with virtual eyes that designers can use to provide gaze cues that communicate future movement intentions (Walker et al., 2018) (see Figure 3-H). Headings may be useful for autonomous robots in crowded, shared spaces with human pedestrians or dynamic obstacles, in which navigational plans need to be recalculated by the robot quickly and frequently. Researchers have also shown how headings can be particularly effective when displayed using projectors, enabling all bystanders to see the intended movements of robots without observers needing to each wear or use specialized hardware (Shrestha et al., 2018) (see Figure 3-C).

Waypoints are environmentally-anchored visualizations of intermediate navigation points. These are typically used to visualize robot intentions but can also be used by robots to suggest spatial destinations indicating where users should move. Waypoints provide another method of previewing robot motion and can either be automatically placed in an environment by a robot trajectory planner or manually placed in the environment by a user. Waypoints are also often combined with other VDEs to show additional information known about each waypoint or what will be performed at each waypoint (Chan et al., 2018; Walker et al., 2018, 2019) (see Figure 3-C).

Callouts are visualizations that communicate where a user should focus their attention. These VDEs use visualizations to attract attention to an object or location, such using a virtual arrow to show where a robot heard a sound or pointing at an object a user should look at (Groechel et al., 2019) (see Figure 3-F).

Spatial Previews use environment-anchored visualizations to show future poses of robots, objects, and other environmental entities. These VDEs can explicitly communicate the expected future position and/or orientation of an entity during a task. A common use for these previews is to depict where a robot will move or where a robot will move an object during a manipulation task. However, robots can also use these VDEs to make requests to users by indicating where a user should place an object. These VDEs can be depicted in various ways, including 2D circles on the ground, complex 3D wireframe or shaded models, combining waypoints with flags indicating orientation at those waypoints, or with one or more Spatial Visualization Virtual Robots (Rosen et al., 2020; Frank et al., 2017) (see Figures 3-B & 5-H).

Trajectories display spatial paths that a robot intends to follow or that it believes an object or agent will follow. These environment-anchored visualizations can show both robot navigation paths and manipulator paths. Trajectories can be visualized in various ways such as lines, splines, or dense stroboscopic Future Robot Pose VDEs in the form of State Visualization Virtual Robots. For example, one common implementation of Trajectory VDEs consists of rendering a trajectory for each wheel on a ground robot, which helps users to anticipate whether or not a wheel will collide with or fall into a terrain hazard. Trajectories can be used to not only indicate the path a robot will take in the future, but also the path a robot (or human) has taken in the past. Trajectory VDEs are typically only used to enhance a user’s view into a robot’s internal model but can also be used to provide new opportunities for control over the robot (e.g., a user directly manipulating the trajectory visualization by grabbing the trajectory line or spline and moving it with their hands). Finally, trajectories can encode data along their paths, such as robot velocities (Walker et al., 2018; Rosen et al., 2020; Leutert et al., 2013; Walker et al., 2019) (see Figure 3-C).

Alteration Previews show the intended permanent modifications a robot plans to make on an object. These rendered previews give the user a chance to verify if the robot’s plan matches that of the user’s task goal(s) prior to execution and cancel or modify them if needed. These modifications typically show how an object will appear during or after the modifications take place (e.g., displaying the proposed path of an etching tool on an object’s surface, where holes will be drilled, how a wall will look after being painted, how a steel bar will look after being bent, etc.). Additionally, these visualizations may communicate what actions will be applied to an object, such as varying pressures applied along the surface on object. This type of VDE is especially useful in circumstances where robot errors arising from command misinterpretation mean the object being acted upon will no longer be usable, potentially wasting hours or days of time and resources to replace the incorrectly modified object (Chan et al., 2018; Leutert et al., 2013) (see Figure 5-A).

Command Options present to users what actions a robot can (or cannot) take in a given state. This may take the form of a robot displaying virtual imagery that indicates what object(s) it can currently pick and/or potential grasp points the robot can utilize (Quintero et al., 2015). In addition to showing user what actions a robot can take, these VDEs also allow a robot to inform users that it is incapable of performing an action it believes the user wishes or might wish it to perform, sometimes with an explanation as to why they are not possible. These VDEs may reduce user frustrations with robotic systems by avoiding situations where a robot silently fails to execute commands and improve user efficiency by aiding users in preemptively realizing that a task will not be performed correctly (or at all) (Arévalo Arboleda et al., 2020) (see Figure 5-C).

Task Status convey beliefs regarding the status of a task currently or previously executed. These VDEs may be represented as traditional textual or numerical visualizations, or as more abstract visual representations, such as progress bars. These visualizations can facilitate human-robot collaborative task planning by helping users quickly and easily understand the current task state a robot is executing, improve debugging by enabling users to compare the state a robot thinks it is in with its actual state, or how much longer a task will take to complete (Bolano et al., 2019; Walker et al., 2018; Ganesan et al., 2018) (see Figure 4-H). Additionally, these VDEs can also convey the status of a concluded task, whether it has resulted in success, failure, or error. These visualizations help users understand what a robot believes the outcome of a task to be (even if incorrect, which may aid in robot debugging). These visualizations can inform a user as to why a task resulted in failure or error, which can often be a mystery to users who would otherwise need to consult complex error logs (Ganesan et al., 2018; De Pace et al., 2018).

Task Instructions enable humans and robots to effectively instruct and guide each other by communicating next steps in collaborative tasks. These instructions can take various forms such as explicit instructions written in text or more abstract instructions that inform a user what to do to accomplish a task such as using an Robot Extension Morphological Alteration VDE to add virtual arms to a robot that point at an object for a user to interact with (Groechel et al., 2019; Ganesan et al., 2018) (see Figures 5-E).