Augmented Reality and Mixed Reality Measurement Under Different Environments: A Survey on Head-Mounted Devices

In AR, users see virtual objects (usually displayed through a handheld device) appearing in the real-world environment. In the AR environment, the virtual objects seen by the users are rendered virtually. Common examples are Nintendo’s Pokémon Go App and the IKEA Mobile App.

The first concept related to AR appeared in 1901 in the novel The Master Key, written by Lyman Frank Baum, which describes a child who experimented with a wearable electronic display to change and overlay data onto a real-world environment. This concept was later implemented by researchers and has became increasingly popular. Many AR-related survey papers have been written to discuss existing AR capabilities, applications, and limitations with technology. Moreover, according to [8], a description of the world’s first HMD using AR was published in 1960 [9].

The first and currently most cited AR-related survey paper was published in 1997 by Dr. Azuma [10], focusing on categories of applications for AR and registration problems. A subsequent (2001) highly-cited survey paper [11] described methods of displaying AR applications such as ”Head-worn displays” (i.e., HMDs), handheld displays, projection displays, as well as challenges for AR displays, such as outdoor environments and positional sensing. Then, in 2010, [8] published a survey paper for the AR-related research linking AR and MR. In this paper, AR and MR were linked using the concept of the reality-virtuality continuum. Similar to [11], this article summarized the display options and application examples for HMDs and also promising new technologies that have made rapid recent progress, such as optical see-through [12] and spatial displays.

Similar to AR, MR also creates virtual objects in a real-world environment using the environment for hybrid rendering methods to generate objects. A current common example is the Microsoft HoloLens [13]. The most significant difference between AR and MR is that the interaction between virtual items and the real environment is added to the MR environment. This difference allows the virtual objects in the MR environment to better simulate and correspond to the physical conditions of the real-world environment. Users can use their own hands to interact with virtual objects directly.

MR was first introduced in 1994 [14, 15], extending AR to see-through and monitor-based displays. These authors used a continuum to present the relation between the real environment, AR, Augmented Virtuality (AV), and the virtual environment. A more complete concept of MR was proposed in [16], which is an extension and mixing of AR and VR. This paper included the concept of space, objects, and users’ activity, which established an important foundation for future MR-related work. A further extension [17] introduced more detailed concepts and specific applications, as well as the possibility of collaboration (i.e., multiple simultaneous users) in MR. Collaboration in MR has since been highly used in research and applications (e.g., Microsoft Mesh: https://www.microsoft.com/en-us/mesh). Similarly, [18] presented collaborative MR works and their own calibration method.

There are also papers describing the concept of MR combined with AR and VR [19, 20, 21]. In addition, there are multiple MR papers focusing on application domains such as education [22, 23, 24] and entertainment [25, 26].

Virtual environments are widely used in various research fields such as computer science and psychology. In [27], the authors focused on distributed virtual environments, which incorporate Internet communication and protocols. Another group targeted potential uses of brain-computer interfaces in interactive virtual environments [28]. A recent survey paper on presence in virtual environments [29] suggests that the sense of presence has a positive correlation with authenticity [30].

In addition to review papers, many individual papers describe research on users in virtual environments. For example, [31] discussed the display system related to the virtual environment, providing an HMD prototype of the virtual environment display system, including hand position tracking sensors.

Later, [32] introduced interaction techniques in the virtual environment, including users’ movement, selection techniques, manipulation (position/rotation), and scaling method. Although this paper was published in 1995, their implementation was presciently close to technology in 2022. In a different approach, [33], mainly focused on cameras (rotation and zooming) for image-based virtual environments using modeling and rendering.

To evaluate interaction techniques in virtual environments, [34] proposed a testbed evaluation method that provided multiple performance metrics for interactions, including object selection and manipulation as well as travel. Lastly, in [35], Immersive Virtual Environment Technology (IVET) used virtual avatars for research in social psychology such as self and group identity.

As described earlier, one of the most widely known tools for using AR, VR, and MR is the HMD, a helmet-shaped device that can move with the user and has screens in front of one or both eyes to display objects in the virtual environment. As of early 2022, the most commonly used HMDs were the NVIS nVisor [36], Google Glass [37], HTC Vive [38], Microsoft HoloLens [13], Oculus Rift [39], and Oculus Quest [3], pictured in Figure 1.

One category of HMD is the Optical See-Through Head-Mounted Device (OST-HMD). This technology uses optical methods to reflect virtual objects and allows the real-world objects to pass through the lens in front of the user’s eyes to achieve the appearance of virtual objects and real-world objects together in the same space. OST-HMDs are particularly useful for AR and MR.

In 1968, Dr. Sutherland published one of the earliest papers [40] regarding a complete design of an HMD. This paper introduced the display system for the HMD and the head position sensor for tracking. Later work [41] described hardware architecture for AR devices, including OST-HMD, virtual retinal system HMD, and video see-through HMD. In a comparison of user navigation performance, [42] found that performance was better with a desktop display than an HMD, although most users were satisfied with the HMD.

There have also been several survey papers reviewing different research areas using HMDs, usually associated with AR and VR. Some review papers have focused on HMD hardware. For example, [43] provided a survey of HMD history such as Active-Matrix Liquid-Crystal-Displays (AM-LCDs), Ferroelectric Liquid Crystal on Silicon (FLCOS) [44], Organic Light Emitting Displays (OLEDs) [45], and Time Multiplex Optical Shutter (TMOS) [46].

There have also been comparisons among different HMDs in user experiments. In [47], the authors compared the distance estimation in multiple HMDs such as Oculus Rift, Nvis, and HTC Vive. Returning to HMD hardware, [48] published a survey paper on state-of-the-art OST-HMD-related techniques for AR and MR, such as Microsoft HoloLens and HoloLens 2 [13]. This survey focused on visual coherence (i.e., blending of virtual and real content) in OST-HMDs in AR and MR research areas. The authors also discussed existing challenges among OST-HMD-related papers, divided into three key areas: spatial realism, temporal realism, and visual realism.

There are a variety of examples of specific applications for HMDs. For example, [49] discussed the social acceptability of disabled/non-disabled people wearing HMDs. [50] analyzed the perception of view transition for dense multi-view content when users are wearing HMDs and transitioning their views. In [51], the authors compared four different kinds of HMDs with conventional optical low-vision aids. Then, [52] used HMDs to investigate the causal agents of head trauma in athletes. Also, in [53], the authors used HMDs to compare user preferences for different virtual navigation instructions that integrate with the real environment in the MR world. Lastly, [54] built a custom HMD to evaluate performance and eye fatigue for context and focal switching in AR.

The above background knowledge demonstrates how interconnected the topics of AR, MR, virtual environments, and HMDs are. Also, many measurement-related papers use the tools mentioned in these topics for experiments. Therefore, we aim to integrate these topics, and organize and discuss papers that use HMDs and environmental factors as the primary measurement goals.

One of the most frequently mentioned environmental factors is the difference between indoor and outdoor environments. For example, [77] mentions that most OST-HMDs add AR light so they do not have to compete against real-world lighting. However, when the OST-HMDs are exposed to direct sunlight in outdoor environments, they lose a significant amount of contrast. The authors recommend that beyond reducing environment light, future OST-HMD works need to consider simulating and making adjustments to the intensive environment lighting in the outdoors. In a comparison of egocentric distance judgments with two different HMDs using indoor and outdoor virtual environments, [88] found greater underestimation in outdoor than indoor environments (see also [47]). In addition, [89] describes how the typical settings of indoor and outdoor environments should be considered; for example, hallways are typical for indoor settings, while lawns are typical for outdoor. In [90], the experimenters tested their HMD-based system in both indoor and outdoor environments to achieve complete 3-D eye-tracking accuracy. Lastly, [91] used a VR device to simulate the MR world and compared the usability of ten different navigation instruction designs.

While the differences between indoor vs. outdoor environments are frequently discussed in the literature, Figure 3a, shows that this factor in the first two time intervals is actually assessed in measurement studies relatively infrequently. Indoor experimental settings are often more convenient and afford more experimental control, especially over variables like luminance. HMD measurement experiments that compare outdoor and indoor environments faced significant challenges in the past. However, since the application of HMDs to the outdoors is a critical challenge, this factor became more prevalent after 2010, and we expect the same trend to continue in future studies.

Another important factor for measurement studies is the difference between two virtual environments or between virtual and real environments. Under the transition of different environments, the virtual objects rendered in front of the user may be displayed differently through HMDs. With different rendering effects, experimenters can do the same measurement across different environments to detect different results.

In [92], the authors developed a Projective Head Mounted Display (PHMD) to build a compound environment of the workstation and surrounding virtual world. In addition, [93, 94] designed an experiment under real and virtual room conditions. The result indicated that the mock HMD in the real-world caused a reliable effect of underestimation. Then, [95] proposed an idea of how to compensate for the difference in space perception between the real-world and the VR environment by asking users to reach the farthest position in each environment.

Figure 3a shows that a large proportion of included papers compare either virtual vs. virtual environment or virtual vs. real environment. One reason to compare two virtual environments is to ensure that results (e.g. differences between two different HMDs) generalize beyond a single virtual environment. When virtual vs. real environments are compared, the main goal is usually to observe that the participants produce different responses and feedback to the same type of display in these two environments. This experiment allows the researchers to understand that the human response to the real environment may not be exactly the same as the response of the virtual environment, and more attention and understanding are needed when designing HMD-related work.

Similarly, recent papers also consider the difference, or lack thereof, for the color of virtual objects and color of the environment. Rendering virtual objects with colors close to the color of environment may make it difficult for participants to see the target objects created by HMDs. This could also impair judgements of distance.

The authors of [96] noted frequent errors in the AR application of OST-HMDs between the input and output colors on display, and applied a complete color reproduction method to solve this problem. In [86], the experimenters used different colors to indicate different target positions for the participants. In addition, [97] introduced an extended color discrimination model to address the issue of the combination of the color displayed by the HMD and the color of the external environment leading to a false perception. By using optical and video see-through HMDs as evaluation devices in the MR world, [61] examined the effect of color parameters on users’ perceptions when using different types of MR devices, showing that users often overestimated color in MR environments. Finally, in recent years, high-fidelity color in MR environments has been an essential task for every OST-HMD. [67] proposed a rendering method to enhance the color contrast between virtual objects and the background of the real environment, which can raise the fidelity degree of color in MR environments and provide a better visual experience.

Figure 3a shows that older papers exploring the effects of color differences are rare, with more appearing recently. This may be explained by the relatively low resolution of older-version HMDs. The increased resolution of new versions of HMDs (for example, the display resolution of HoloLens 2 is 2048x1080 per eye, and the display resolution of Oculus Quest is 1440×1600 per eye) may mean that users are more sensitive to the colors displayed in HMDs. In future measurement-related research in AR/MR, the possible impact of virtual color with the real-world environment is likely to be increasingly important.

The luminance of the environment, both alone and in combination with color, is also an important factor for the perception of virtual objects. Here, luminance indicates the intensity of light in the environment. For example, a dark red background will make bright green easier to notice, or excessively high luminance may prevent users from seeing the target object clearly in a dark environment.

In [98], the authors designed a series of experiments to examine the relation between luminance and distance judgments in HMDs, such as finding a certain threshold of the luminance that affects the result. In addition to luminance affecting distance judgment, [74] proposed that significant increases in luminance while viewing HMDs can cause visual discomfort. This paper evaluated the luminance change and maximum luminance that will cause discomfort to the users. Recent research has focused specifically on environment luminance affecting OST-HMDs, which use additive light models, meaning they lose a significant amount of contrast in high luminance environments (e.g., outdoor on a sunny day with no clouds). In [77], the authors designed a measurement to evaluate perceptual challenges with OST-HMDs and made some suggestions for future improvements.

As shown in Figure 3a, studies have assessed luminance differences as a factor throughout the time periods surveyed, with what appears to be a small increase over time. Common effects of different luminances are changing visibility of virtual objects for users, and high-luminance environments causing user discomfort. As mentioned above, indoor and outdoor luminances are often substantially different. Some related papers used indoor high luminance to simulate outdoor luminance while controlling for other differences between indoor and outdoor environments. Others have recently tried to make HMDs usable in both indoor and outdoor environments.

We also summarize the number of papers for each environmental factor over time and the specific topic areas for environmental factor in Figure 3b. This figure again shows the relatively high proportion of papers with the category of Virtual/Real, across multiple topic areas. By topic area, environmental interaction and questionnaires are frequently covered in papers assessing all four environmental factors. Whereas the topic area of comfort assessment only has three papers, two of which focus on the luminance (magenta bar) conditions that cause participants to experience discomfort while wearing HMDs. Similarly, eye tracking only has a single paper for each of three environmental factors (color, luminance, and in/out) since attention-drawing factors used in these papers are often the changes in color or luminance, these are common environmental factors when designing the measurement of eye tracking.

In this subsection, we examine the relationships among the number of participants recruited for measurement studies, the composition of participant samples, and the category of the studies. For example, many studies are conducted at universities [99], meaning that participants are largely students who are less than 30 years old. We explore the distribution of participant ages among different categories of measurement papers. Similarly, men generally outnumber women among students in computer science and engineering. There are potentially gender differences in the results of the experiments, so we also compare the male and female ratios of participant samples in the surveyed papers. Previous HMD experience may also potentially affect results; we organized papers by whether they assessed HMD experience in their participants and if so, the distribution of experience.

The left panel of Figure 4 shows the “sample size” or total number of participants recruited in each paper. Note in the majority of papers the sample size tends to be limited, between N = 1 to 25. Many factors may affect the number of participants, such as: aims of the study (e.g., qualitative usability vs. quantitative performance), study design (e.g., whether or not there are repeated measures), study duration and resources needed to run the study, and whether participants with particular expertise is required. Nevertheless, in general, larger sample sizes are desirable for quantitative research using inferential statistics.

The Male-to-Female Ratio Total chart in the middle panel of Figure 4 does not support the above assumption that participant samples in most experiments are overwhelmingly male. Although there may be specific areas of research such as male athlete sports-related [52] that will include only male participants, or some papers that do not describe the ratio of men to women in detail, most of the measurement-related papers attempted to roughly balance the ratio of men to women. To ensure that experimental results generalize across genders, it is important to recruit both male and female participants. However, it can be challenging to achieve a male-to-female ratio of exactly 1. We consider an M:F ratio between 0.5 to 1.5 to be roughly balanced; Figure 4 shows that most papers fall within this guideline.

Prior experience with HMDs is also an important participant variable. As shown in the right panel of Figure 4, most papers did not report participant HMD experience (marked as ”N/A”). The small number of papers (14 out of 57) that did report participants’ prior experience with HMDs were roughly balanced as to whether the majority of their sample had experience with HMDs. Depending on the category of the measurement experiment, the HMD experience of the participant who needs to be recruited will be different since the participants’ experience with HMDs may have different progress and outcomes for the experiment. The most common case is in the category of health care, where recruiting experienced participants enables experiments to be performed more quickly and accurately because no additional training is required for them.

As [99] proposed, since most academic papers are written in universities the age distribution of most participants tends to skew younger, corresponding to the typical age of university students. Figure 5 depicts the age range of participants in papers from different topic areas. Most papers provided an age range of participants, but only a small number of papers reported more detail such as the mean age or the source of the participant sample (e.g., university students). In support of [99], we see that the age range between 15-30 in many topic areas is relatively high, likely because recruiting students in universities is convenient and inexpensive when the experiment requires many participants. Also, experiments with novel HMD technology may be more likely to attract students’ attention and make them more willing to participate in the experiment. We do not see evidence in Figure 5 that the age distribution of participants is related to the paper category; most of the researchers recruited 15- to 30-year-old students as participants for each category. However, participant age can often have an impact on experimental results. Much like balancing Male-to-Female ratios, it is important to assess a range of participant ages in order to be able to generalize results. Furthermore, there may be research applications where older users who are relatively unfamiliar with HMDs are the relevant population. Therefore, we recommend that researchers report detailed information about the ages of their participants, and also consider recruiting beyond the typically convenient population of university students.

The method of data collection is a critical aspect of any measurement experiment, and there are a number of potential data sources for studies with HMDs, including sensor data (HMDs or external measuring instruments), user responses, and questionnaires. We collected the data acquisition methods in measurement papers related to HMDs using AR and MR and organized them in this section, and then summarized this information in Figure 6.

Although HMDs are the most common data source in the surveyed papers, since measurements often require participants input, many papers also use user feedback as a data source. User feedback is usually divided into verbal feedback during the experiment, such as verbally telling the experimenter the distance to a target, or written feedback, such as completing questionnaires after the test. Since questionnaire was commonly used in measurement-related papers, we created a separate questionnaire category in Figure 6 (Left). Other standard data sources include external sensors that record data in HMDs or computer software that connects to HMDs to collect data; we collectively call these ”External measuring instruments.” Furthermore, our survey indicated that many recent measurement papers used OST-HMDs as their main measurement data sources. Therefore, we created OST-HMDs as an additional category in Figure 6 (Left).

In addition, 6 (Center) shows the majority of papers directly assessed feedback from participants, which can be used to improve the experiment process, time length, or other environmental factors based on the feedback. Then, about half of the papers had 11 trials or more per participant, shown in Figure 6 (right). Note that 15% of papers did not report the number of trials for each participant, marked as ”N/A.” Only 17% of papers had $>$100 repeated trials per participant, this may be because of the possibility of fatigue and/or discomfort with extended use of an HMD.

As of early 2022, there are many different types of HMDs on the market, including AR, VR, and MR HMDs. Different HMDs use different computer platforms and sensors, and experimenters need to design corresponding measurement experiments according to the capabilities of the device. In this section, we summarize the assessment of different HMDs across time and various topic areas. Since the number of HMDs for AR and MR is relatively small, we also include papers using VR HMDs in this section.

Figure 7a shows that the most widely used HMDs are different depending on the year. In earlier papers, there are many different types of HMDs used, even HMDs built by the researchers themselves. Since about 2004, authors of measurement-related papers began to use one of the most common HMDs during that time, which was a different version of NVIS nVisor [36]. After about 2014, other modern and well-known HMDs, such as Google glass [37], Oculus [39], and HTC Vive [38], began to appear and were widely used by researchers. In addition to general HMDs, OST-HMDs have been frequently used in recent years; such devices include Microsoft HoloLens 1st and 2nd generation [13], which is clearly shown in the rightmost columns of Figure 7a. Researchers’ widespread use of OST-HMDs means that MR has begun to flourish, and more information close to reality can be obtained. In addition, although not mentioned in detail in this figure, a new version of Oculus, Oculus Quest, has been widely used in research in recent years. Oculus Quest is wireless and thus will be more helpful to future research. Since a large number of papers use custom HMDs or rarely used HMDs, we classify them in the ”other” category. For example, in [100], the authors used swimming goggles, video cameras, transparent hot mirrors, and eye trackers to build a custom HMD. There are many papers focusing on technological advances in HMDs, and the use of custom HMDs enables researchers to break away from existing HMDs to achieve their goals. Therefore, the proportion of the category ”Other” in Figure 7a and Figure 7b is relatively high.

Figure 7b shows the HMDs used across different measurement categories. There is no clear relationship between category and device type. There are many experiments of the same type that can be used in similar HMDs; for example, [47] in 2018 used several different HMDs in the topic of distance underestimation, including Oculus Rift, Nvis HMDs, and HTC Vive. We see again a large number of devices falling in the ”other” category in this figure. As described above, many of these are custom HMDs, which were often designed only for a specific research field. For example, in [51] in 2004, the target research area was health care, so the HMDs used needed to match the medical needs to select the appropriate machine.

In this paper, we have categorized a total of 87 papers using HMDs in AR, MR, and VR. The categories included environmental factors and user vs. device measurements. We identified several potential research gaps. Comparison of indoor vs. outdoor was the environmental factors assessed by the fewest papers. This may be largely due to technical limitations with the performance of current HMDs in outdoor environments, but the number continues to increase after 2010. In addition, only one paper evaluated user movement. This indicates almost all of the papers had participants in a stationary standing or sitting position. However, real-world use of AR/MR for tasks such as maintenance, some health care applications (e.g., surgery), or spatial navigation will typically require walking and/or changing body positions (e.g., leaning to look from different perspectives). In the device measurement category, no papers proposed standardized methods for evaluating and comparing technical aspects of HMDs.

In terms of gaps for user experiments, we also found these papers tended to have limited sample sizes. The majority of papers had 25 or fewer participants, although many papers had repeated measures (multiple trials per participant). Surprisingly, there was a clear lack of a gap for the ratio of male to female participants which were often similar. However, most papers did not report whether they assessed prior HMD experience. If HMD use continues to grow as predicted over time, assessing experience with HMDs is likely to become increasingly useful. Additionally, most studies tended to rely on a younger, student population for participant samples, which may not be ideal for all applications or for generalizability of results.

In terms of hardware, the most frequently used HMDs were the NVIS nVisor, Oculus, Google Glass, and Microsoft HoloLens. There were many ”other” devices and seven different HMDs were only used in a single paper. While we categorized HMD hardware, a limitation of the current paper is that we cannot fully categorize the software used by paper due to large software diversity (e.g., custom software versus commercial off the shelf software such as Unity or Unreal Engine versus open source such as ARToolKit). But we still list the software mainly used by mentioned HMD in the Appendix C.

In the future, researchers can find the reference materials they need based on this paper and use our integrated figures to find trends in these research topics in recent years. We hope this paper can be an informative overview to help further develop categories and experiments with environmental-related measurements. Furthermore, this categorization may also be useful for other different types of AR/MR devices, such as mobile AR/MR devices.