Multi-User Augmented Reality with Infrastructure-free Collaborative Localization

Fiducial markers such as ARTags (Mulloni et al., 2011; Klopschitz and Schmalstieg, 2007) and AprilTags (Wang and Olson, 2016a; Kallwies et al., 2020; Zhao et al., 2020) are frequently used in AR systems to provide a reference between the physical environment and virtual objects. While these passive markers can be accurately localized with only a camera and low computational requirements, they are not suitable for real-time tracking of AR users, especially in mobile and NLoS scenarios.

Simultaneous localization and mapping (SLAM) techniques are widely used in robotics and AR systems for identifying and then leveraging features in an environment to track the position of a moving device. These methods use either monocular cameras (Bae et al., 2016; Engel et al., 2014; Mur-Artal et al., 2015; Aider et al., 2005), depth cameras (Bae et al., 2014; Lu and Kambhamettu, 2014; Yuan et al., 2016), or stereo cameras (Bellavia et al., 2013; Brand et al., 2014; You et al., 1999) to extract visual features from the scene and extract the 3D coordinates of the features and the device 6DoF pose. These coordinates, however, are only relative to an origin point, typically the start point. More recently, ARKit (Apple, 2020) by Apple and ARCore (Developers, 2019) by Google have shown persistent AR by providing 6DoF pose estimation with respect to a previously acquired map by combining vision or point-clouds with VIO. However, the biggest challenge facing vision-based localization is that it relies heavily on recognizing known images or clusters of unique feature points in the environment. This results in slow convergence and a high sensitivity to environment dynamics such as displacement of furniture, ambient lighting, and requires rich visual (or depth map) features. Even with advanced hardware platforms such as the Hololens 2 that uses depth sensors, users must often walk around and view several areas of a scene before localization is able to take effect. Due to the limitations of purely vision-based approaches, we advocate combining visual localization approaches with range-based beacons, and mainly focus on relative location of the users with respect to each other.

Beacon-based solutions provide continuous localization using UWB (Olsson et al., 2014), BLE (Shao et al., 2018; Cheung et al., 2006), or ultrasound (Gómez et al., 2013; Lazik et al., 2015; Li et al., 2018) ranging. However, all of these methods rely on pre-installed infrastructure and dense deployments throughout the building, which is not suitable for all AR applications. An alternative approach is to combine vision and ranging mechanisms, where ranging information from on-board radios such as UWB, Bluetooth, or WiFi is used to eliminate the accumulated errors of odometry sensors (Song et al., 2019; Olsson et al., 2014; Liu et al., 2017; Dhekne et al., 2019; Gentner and Ulmschneider, 2017; Wang et al., 2017; Rajagopal et al., 2019). While probably the best systems in terms of performance, the existing solutions still require pre-installed infrastructure or known starting points to link local maps to the physical space. LocAR addresses these limitations by combining relative ranging of the users with local motion information of each users, thus achieving the best of both worlds. One recent example of using VIO with UWB for ranging was introduced by the Apple AirTag platform. AirTag leverages a single moving phone along with UWB ranging (from the Apple U1 chipset) to detect a small battery-powered tag. The system currently only operates in 2D and hence does not work across multiple floors and does not support a network of moving users each localizing each other.

While typical SLAM-based solutions assume a single user, recent developments in AR frameworks, such as Google’s ARCore / Cloud Anchors, Apple’s ARKit and Microsoft’s Spatial Anchors, have enabled multi-user capabilities. In these systems, each AR device individually performs SLAM to capture the visual features of the physical space relative to its local coordinate system. The users then share these visual maps to establish a common coordinate system and estimate the pose of other users. To share these maps between the users, Google ARCore uses a cloud-based architecture, which combines these maps centrally and sends the updated maps to all the users. However, Apple ARKit uses a peer-to-peer architecture, where the host of the AR session shares its current map with the users joining the session. However, any of these techniques impose significant communication overhead (Ran et al., 2019). These maps consist of dense visual features, 3D meshes, or raw point clouds, which are usually large and difficult to transfer. In addition, these map matching approaches assume a significant overlap between all of the users, which becomes unwieldy in terms of network traffic and computation in large areas. On the other hand, in search and rescue environments, where multi-user AR can bring significant added value, users are often purposefully taking disjoint paths through the environment to improve coverage, thus making map matching much more challenging and substantially increasing the convergence time.

Traditional localization systems typically have the goal of estimating the ”absolute” location in a fixed coordinate system that is mapped to the physical space using external systems. In this sense, the idea of ”localization” is inherently tied to the existence of some form of infrastructure from which to base the measurements. However, reliance on infrastructure is infeasible in many AR scenarios, especially in the presence of multiple users. In these cases, it is instead possible to determine the relative locations between users to establish a common coordinate system for multi-user AR applications. For example, to display a virtual overlay of a target on the screen, only the knowledge of the target’s position relative to the display system is required.

The concept of relative localization is first used in sensor network localization for collectively locating stationary (Nagpal et al., 2003; Savvides et al., 2001) and mobile (Moore et al., 2004; Rad et al., 2011; Eren et al., 2004) nodes with respect to each other. These works provide the theoretical foundation for network localization using graph theory (Eren et al., 2004; Popescu et al., 2012; Ferner et al., 2008; Čapkun et al., 2002; Jamali-Rad et al., 2012; Barooah and Hespanha, 2007), information theory (Savic and Zazo, 2013; Ross, 2014; Charrow et al., 2014; Hoffmann et al., 2006), or Bayesian inference methods (Nilsson and Händel, 2013; Carlone et al., 2011). However, the majority of these systems are only evaluated in simulation and do not provide the desired AR performance in terms of latency (real-time operation), accuracy, and degrees of rigid body movement in space (6DoF).

Relative localization has also been explored in robotics for localizing a swarm of drones or multiple robots with respect to each other (Coppola et al., 2018; Carlone et al., 2011; Olsson et al., 2014; Liu et al., 2017). However, all of these systems assume short ranges with all the nodes in Line-of-Site (LOS) and only focus on 2D or 3D localization, instead of 6DoF. In addition, most of these methods require the prior knowledge of the initial position, which suffers from accumulated error over time. Therefore, they can not be directly extended to wide-area AR applications. In this paper, we present a distributed relative localization framework that provides the real-time relative pose estimates of AR users without requiring any pre-existing infrastructure, prior mapping, or known initial position.

We consider an indoor scenario consisting of $N$ mobile users (nodes), all with unknown positions and orientations. Each mobile user has an AR display device, defined as display device, and wants to localize all other users, defined as target devices, with respect to itself, without requiring any a priori knowledge of the physical space or pre-installed infrastructure. The localization framework has to work in real-time on limited compute platforms and needs to scale feasibly with number of devices being tracked. All AR devices are equipped with 2 sensors:

• •

  VIO tracking: currently many smart phones and most AR headsets in the market have built-in VIO. VIO essentially tracks the motion of camera by fusing detected visual feature points with inertial sensor data. The output of VIO is the position and orientation of the device with respect to the reference frame at startup by performing conventional dead-reckoning. Even though VIO provides the camera displacement over time, there is no common origin between multiple users to extract their relative positions. Another challenge of VIO data is the accumulated drift error over time and sensitivity of visual dead-reckoning to environment conditions, such as lighting and motion.

• •

  UWB ranging: Among various wireless technologies that can go through obstacles such as Bluetooth, UWB, WiFi, etc, UWB is the most promising technology to combat multipath propagation in cluttered environments(Rajagopal et al., 2019). As a result, we are seeing appearance of UWB chips on the latest mobile phones, providing peer-to-peer ranging. However, each UWB node is only capable of measuring the distance to neighboring nodes that are in range. Given the mobility of users, we cannot assume that range measurements occur synchronously or with any sort of regularity, resulting in sparse and inadequate data for real-time localization.

• •

  Data Communication: We assume that each user’s device can communicate their state information with any neighbors in a peer-to-peer manner. This requires relatively low data rate exchanges and could leverage the UWB transmissions or use an ad-hoc method like WiFi Direct, Bluetooth or dedicated emergency responder radios. One of the key benefits of our collaborative filtering approach is that devices only needs to exchange data with their neighbors that are replying to UWB messages (not a fully connected network). In our experimental platform, each node communicates using WiFi or LTE from the mobile device, but this could be easily replaced in a production implementation. In our experiments, the state data transmitted on each active neighbor link between neighbors was below 16 kbit/s (assuming 10Hz updates). Our system is also resilient to message drops and reasonable levels of jitter (tens of ms). With message latencies on the order of 100ms, our system appears to perform well and is within common bounds for most single-hop wireless communication systems.

We demonstrate that the synergy between VIO and UWB data allows us to overcome the challenges and limitations of each. This results in a distributed relative localization framework that allows multiple mobile users to create a collaborative AR session. First, LocAR adopts a distributed architecture in favor of scalability by providing each node with peer-to-peer distance measurements to other nodes. Next, to deal with the sparsity of UWB reading, range measurements are combined with local camera VIO traces. The absolute nature of UWB ranging allows us to correct VIO drift over time. Finally, LocAR leverages the presence of multiple users and their mobility to collaboratively estimate the relative position of all users, improving the overall localization accuracy while maintaining low computational overhead. An overview of LocAR framework is depicted in Figure 2. Upon startup of an AR app, the AR session tracks the pose of this device using VIO from the AR API and collects the UWB ranges from neighboring nodes, which are then passed to a particle filter state-estimation solver to extract the location and orientation of other nodes with respect to itself. The next section elaborates on LocAR’s collaborative pose estimation technique and the underlying challenges.

To display a virtual object, three matrices are required:

• •

  $K$: The 3x4 matrix encoding the intrinsic properties of the virtual camera such as resolution, focal length, and centerpoint, which which are assumed to be known

• •

  $D_{O}$: The 4x4 matrix encoding the 6DOF pose of the display relative to some arbitrary origin

• •

  $V_{O}$: The 4x1 vector encoding the 3DOF position of the target object relative to that same origin.

From these, we can calculate the pixel coordinates of the virtual object on the display $[u,v]^{T}$ as:

(1)

$$[u^{\prime},v^{\prime},w^{\prime}]^{T}=K*D_{O}^{-1}*V_{O}$$

(2)

$$[u,v]^{T}=[u^{\prime},v^{\prime}]^{T}/w^{\prime}$$

We can simply combine the latter two matrices as:

(3)

$$V_{D}=D_{O}^{-1}*V_{O},$$

where $V_{D}$ is now the 4x1 vector representing the position of the target object relative to the display. There is no longer a requirement for a fixed origin. The coordinate system is typically chosen with +x pointing towards the right edge of the display, +y pointing towards the top edge of the display, and +z pointing out of the display towards the viewers eyes.

We begin by describing the state-space representation and our error models for VIO and UWB measurements, for a basic particle filter formulation. A particle filter for our state estimation has the following benefits: (i) it is computationally easy to run online, (ii) it allows us to use arbitrary noise models to describe VIO and UWB errors, (iii) it can work with as few as 1-2 beacons in under-defined cases, (iv) as it is agnostic to update rate, it allows handling of asynchronous ranges from the beacons and does not require receiving synchronized ranges to perform trilateration.

In our naive baseline implementation, we run a completely independent particle filter for each device we wish to track. As a result, the computational load scales linearly with the number of devices $N$. However, since the particle filters are run independently, any error that is accumulated due to noise in the display device’s own VIO tracking cannot be mitigated through collaboratively ranging to multiple devices and leveraging the synergistic information that arises.

With unlimited computational resources, it would be possible to jointly model the states of all $N$ moving devices. This way, every range could be used to improve the state estimation of all nodes in the joint distribution. However, sampling from $4N$ dimensional state-space would require a number of samples exponential in $N$ in order to adequately sample the growing dimensionality.

A solution to this problem arises when some state variables $Y^{(i)}$ are always conditionally independent given some other state variable $X$. When this is the case, it is possible to factorize the joint probability distribution and independently track each $Y^{(i)}|X$. This approach, called Rao-Blackwellization (RBPF), is common in the SLAM literature (Thrun, 2002) as a means of estimating a map whose elements are conditionally independent given a user’s location. As illustrated in Figure 3, our formulation of the relative localization problem fits this framework, since UWB provides measurements of device locations that are conditionally independent given the location of the display device.

In the RBPF formulation, a particle filter is used to represent the belief of the display device $D$, where each target device $V_{D}^{(i)}$ can be represented by any probabilistic distribution. We chose to also represent the target device estimates using particle filters. In Section 6.7, we demonstrate the benefit of the collaborative nature of our joint RBPF formulation over the more common naive independent particle filter approach.

Though not the focus of this paper, we realized that many localization researchers have struggled to find a UWB solution that can easily operate in peer-to-peer mode at scale. Unfortunately, most of the freely available reference implementations are designed for fixed infrastructure scenarios that support mobile devices, as opposed to fully peer-to-peer operation. We imagine that phones with UWB hardware could eventually implement this functionality on-board once APIs become available.

In order to perform neighborhood discovery and ad-hoc ranging with UWB, we developed an open-source and easy-to-use firmware image for the MDEK1001 modules from DecaWave. The MDEK1001 is an all in 1 battery- or USB-powered module with an enclosure that pairs a Nordic nRF52832 MCU with a DW1000 chip. The Nordic chip has a 64 MHz Arm Cortex-M4 processor with integrated BLE radio that can be programmed to act like a stand-alone beacon or pair with a mobile phone. Our firmware image exposes a standard serial interface (the AT command set) with the ability to store default parameters to flash memory, making it easy to configure addresses, sleep modes, neighborhood discovery polling rates, and UWB ranging options. Our firmware provides three basic functionalities: (1) Neighborhood discovery using BLE’s GAP discovery protocol, (2) coordinated Double-sided Two-way Ranging (DS-TWR) using UWB and (3) an interface to external systems using either USB serial or a standard BLE GATT server. We design our protocol under the assumption that we have a highly dynamic mesh of nodes with hidden terminals and asymmetric links that change on the order of seconds.

The neighborhood discovery protocol is BLE’s standard device discovery protocol that frequency hops across three channels. We allow users to define a custom advertisement period $T_{BLE}$ (default period of 200 ms) and a configurable signal strength (RSSI) threshold for determining the most recent and closest neighbors. To save power when nodes are idle, we duty-cycle background scanning and disable the UWB radio. A node in the system can announce that it wants to participate in active ranging through its BLE advertisements. This in turn will wake-up all nearby nodes and activate their UWB radios. Figure 5 shows an overview of the BLE and UWB transactions required to perform neighborhood discovery and ranging. Note that the BLE discovery uses three channels and not just a single channel. Once activated, each node initiates a DS-TWR request (detailed in the upper right of the figure and in this application note (Corporation, 2015) ) over UWB at a user-configurable timing interval $T_{UWB}$, with a default value of 100 ms. Each $T_{UWB}$ period the node performs a new DS-TWR request to the next node in its local neighbor list.

If DS-TWR messages are dropped either due to collision or packet corruption, the next polling interval is randomly offset to avoid repeated collisions. We use an exponential random distribution across $T_{UWB}$ similar in nature to Slotted ALOHA. As one would expect, as the number of neighbors increases, the polling rate of each individual neighbor decreases. We provide users with a lookup table for $T_{UWB}$ values needed to support particular maximum node densities within a single collision domain. In Figure 5, you can see that Beacon $B_{1}$ transmits every $T_{UWB}$ to Beacon $B_{2}$ since it has no other neighbors. Beacon $B_{2}$ cycles through 3 total neighbors in its neighborhood list (the neighbor graph shown on the left). After nodes stop transmitting active ranging advertisements for a defined timeout, nodes return to their lower powered duty-cycled listening state. As shown in the bottom line of Figure 5, we also support simultaneously pairing an actively scanning node with a mobile device using a standard BLE GATT server. It is also possible to connect the MDEK1001 to a host device over USB serial or through its built-in RPI header. The default parameters of our firmware support 16-bit addressing (over 30K nodes) with cluster densities of 10 nodes at approximately 1 Hz update rates for each neighbor. Our low power sleep energy is on the order of 10 mW (mostly consumed by background BLE scanning) with an average active ranging energy of 800 mW. In practice, we see BLE neighbor discovery on the order of a 1-2 s with a typical 10-20 s eviction timeout. All source and documentation are available on GitHub.

We developed a prototype of LocAR as a mobile AR application running on iOS. This application provides two main features: (1) it shows the relative location and orientation of other users in the scene in AR (shown in Figure 4-c), and (2) it coordinated ground-truth data collection among mobile users (shown in Figure 4-b). The mobile app collected VIO data using Apple’s ARKit and UWB ranging data using a MDEK1001 module from DecaWave over BLE. All ranging and communication information was shared using MQTT over WiFi, but this could conceptually be replaced by WiFi Direct or some similar peer-to-peer protocol. ARKit captures VIO data at 60 Hz and we collected UWB ranges with a polling rate of 10 Hz. While it is difficult to exactly isolate how much energy VIO consumes on iOS and Android, the Intel T265 stands at a good reference consuming less than 1.5W. This is low enough that it does not significantly impact interactive usage over a few hours during an AR session. As described above, the actual rate at which UWB data was received by each mobile user depends on the distance and number of neighbors around a particular node.

One of the biggest challenges for assessing the performance of a 6DOF localization system at scale is accurately collecting ground-truth poses. We developed a data collection framework that periodically guides users to converge on ”check-in” locations where AprilTags could be used to accurately record pose. We first installed over a dozen 8.5 by 11 in AprilTags (Wang and Olson, 2016b) across the multiple floors of our test building with retro-reflective markers on each corner. We surveyed the corners of each AprilTag using a total station with an estimated accuracy on the order of millimeters. To coordinate synchronized ground-truth readings between different users, we integrated AprilTag decoder into the AR application, in which the users are instructed to move to the nearest AprilTag and wait until all users across the building had a high-confidence ground-truth measurement. Given the known tag location and the pose estimated by the AprilTag decoder (Wang and Olson, 2016b), the application computes the ground-truth location which is then published over MQTT to a central logging service.

Our primary evaluation of LocAR consisted of a deployment across a 30,000 sq ft area spanning 3 floors of an office building, with 3 to 5 users walking in an arbitrary fashion, and 9 static tags deployed for baseline comparison, as shown in Figure 6. We also stress tested LocAR in a diverse set of environments both indoor and outdoor, different lighting conditions, as well as dynamic environments. The snapshots of these environments are shown in Figure 7. In all these experiments, each user carries an iPad or iPhone with a built-in VIO tracking, and a UWB node attached to the back of the device (as shown in Figure 4-a), while the static tags consist of just the UWB platform. As noted before, LocAR does not require any pre-installed infrastructure or static beacons with known location for localization, and here the static tags are only used for our baseline comparison. Unless otherwise specified, all of our presented results only use ranges from mobile nodes.

The experiments consist of both LOS and heavy NLOS situations, with many instances where users are spread across 3 different floors with one or more dry/concrete walls between them. No instructions are provided to users on how to walk or how to hold the tablets. For 7 different experiments and 10-15 min per run, the users walk with different speeds and periodically stand stationary, resulting in a total of about 40 min worth of data per person. This data is divided into an ”evaluation” set, where users are walking normally, and a ”sensitivity analysis” set, where users are walking in pre-defined patterns (evaluated in Section 7). As explained in Section 5.3, the ground-truth was obtained with a number of AprilTags surveyed in a global coordinate frame using a total station. To synchronize the ground-truth measurements between users, the AR application guides users to scan a nearby AprilTag every 5-30 s over the course of each experiment.

The quality of AR performance is sensitive to more than just geometric error. Camera lens parameters, bearing, and distance combine to create the visual error seen by a user. To better capture these effects, we introduce a new AR-specific metric, which we call display-proportional error (DPE), that combines distance, bearing, and the camera field-of-view as a single cohesive benchmark. Figure 8 shows a typical example where 3 virtual cubes are overlaid at a fixed distance from a set of (real) physical orange cones. The cones are located at a distance of 1, 5, and 10 m, respectively, away from the camera. The green cube has no error, the yellow cube is offset by 0.5 m and the red cube is offset by 1 m. Notice that, due to perspective, the cubes that are further from the camera appear closer to the cone, even though their relative error in meters is the same. This simple example highlights why geometric error alone does not do justice to AR localization performance. Instead, display-proportional error computes the AR error as the distance between an object’s true location and its estimated location when projected onto a 2D display, as a proportion of the display’s horizontal size. In the example in Figure 8, the closest yellow box has a DPE of 0.23²²2after accounting for the height that the user is holding the device off the ground. This error corresponds to approximately 1/4 of the screen width, while the farthest yellow box has a DPE of only .03, or about 1/33 of the screen width. In this sense, DPE captures the reprojected error of the estimated 3D locations, and can easily be used to calculate pixel error by simply multiplying by the display’s horizontal resolution.

Therefore, We formalize our error metric definitions as:

• •

  3D geometric error: We calculate the average pair-wise Euclidean distance in 3D between all pairs of mobile nodes in meters.

• •

  Display-Proportional Error:

  (9)

  $$\frac{\epsilon_{xy}}{|dist+\epsilon_{z}|}*\frac{f_{x}}{H_{x}},$$

  Where $\epsilon_{xy}$ is the $xy$ component of the 3D geometric error, $\epsilon_{z}$ is the $z$ component, $dist$ is the true distance between the display and the target object, $f_{x}$ is the camera’s focal length (in pixels), and $H_{x}$ is its horizontal resolution (in pixels)

We compare the performance of LocAR with two baselines:

(1) VIO-Only: a typical infrastructure-free localization method (Bloesch et al., 2015) that uses VIO for pose estimation relative to the start point. This is a common localization method in robotics, but it requires initialization and a priori knowledge of users’ start points. Even though this assumption is not feasible for most multi-user AR applications, it allows us to more easily isolate the performance contributions from VIO and UWB ranging in our system.

(2) UWB-VIO infrastructure-based oracle: an infrastructure-based localization technique, which uses VIO to estimate 6DOF motion and UWB ranging to fixed beacons. We assume each fixed beacon (9 total) has a known global location in order to provide a baseline (Wang et al., 2017). As shown in Figure 4-b, static UWB nodes are placed at a fixed offset from each AprilTag to provide a set of known UWB locations. We consider this technique as our oracle and show that LocAR can achieve performance at nearly the same accuracy without relying on any pre-installed infrastructure.

It should be noted that both of these baselines are originally proposed for absolute localization, either relative to origin or relative to the physical coordinate system, so we obtain the relative localization for comparison with our system using Equation 3.

We evaluate the localization accuracy of LocAR across our evaluation dataset with 5 mobile users, on both single and multiple floors, and with a mixture of LOS and NLOS situations. Figure 10 shows the overall 3D relative localization error and compares it with our two baseline approaches. LocAR achieves a median 3D error of 0.9 m, compared to 2.5 m and 0.8 m in the VIO-Only baseline and UWB-VIO oracle, respectively. We can see that LocAR outperforms VIO-Only baseline by leveraging the UWB ranging and collaborative pose estimation which eliminates drift over time. In addition, LocAR achieves relatively similar accuracy to the UWB-VIO oracle, which relies on pre-installed infrastructure and a priori knowledge of beacons for trilateration that is unnecessary for LocAR.

As mentioned in Section 6.2, the 3D geometric error does not necessarily quantify the localization performance specific to AR application, hence our proposed AR metric, display-proportional error, which captures the relative object displacement error on the screen. Figure 10 compares the AR performance of the three methods using our defined metric, and shows that LocAR can satisfy a high AR quality in 99% of cases with less than 0.5 fractional error on display and a median quality 0.1 fractional error. This means that the users are able to steer in the right direction toward a user 99% of the time across varying ranges and angles.

Next, we evaluate LocAR’s performance as a function of distance. The ground-truth relative distance of users varies between 0.2 m to 27 m including many instances of completely NLOS. Figure 12-a demonstrates the 3D relative error of each sample test (any pair of users at every 5 s interval) grouped by the ground-truth pair-wise distances. As we can see, error in positioning tends to increase slightly with distance, either due to UWB nodes going out of range or inherent VIO drift. However, unlike geometric error, DPE actually improves with distance. This suggests that a visual display showing an overlay with faraway users’ locations would still be effective at portraying those users’ locations.

It can be seen in Figure 12 that there is a linear correlation between error and true distance of nodes and that error stays below 10% the distance on average. This means that at 20 m, the average error is less than 2 m, which is a reasonable amount of error when localizing 2 users in a building with NLOS. This can be better captured with the AR-metric shown in Figure 12 with a decreasing error over extended ranges. As we can see, the trend of geometric error and AR error with respect to true distance is opposite, confirming the AR-specific behavior of localization systems explained in Section 6.2.

In many localization systems, including VIO tracking, error increases with time. Dead reckoning systems have inherent drift that is inevitable, and small errors in individual state estimation accumulate over time. As seen in Figure 14, LocAR does not have this problem, and keeps an almost constant error distribution throughout every experimental run, while VIO has linear drift over time. This success can be attributed to the use of UWB ranging between nodes to keep the drift bounded to within the UWB error.

To evaluate the impact of our collaborative particle filter formulation, we compare the 3D relative error of the naive independent PF and collaborative RBPF, explained in Section 4. To isolate the impact of other parameters, including user mobility, number of users, etc, we perform controlled experiments with a single user and 9 static tags deployed for baseline comparisons. Then we estimate the relative location of static tags with respect to the user for different subsets of tags changing from 1 to 9 randomly selected tags. As seen in Figure 14, collaborative RBPF has a clear advantage over Independent PF. While with 1 static node, they are very similar in 3D relative error of around 0.9 m, independent PF localization remains at this error while the error of collaborative RBPF decreases from 0.85 m to 0.22 m with the addition of static tags.

This was expected as collaborative RBPF takes advantage of other system nodes’ estimates. With this method, the drift of the mobile node is able to be somewhat mitigated by the averaging of noise across measurements to multiple other nodes. As more nodes are able to perform these ”averaging corrections” together, the localization system is able to converge to a more precise estimate than it could with nodes localizing individually. As a side conclusion, we can leverage this feature to further improve the localization performance by deploying some static UWB tags with unknown locations. For example, in a first response operation, the users can deploy some static nodes at random locations as they move around the building to enhance their relative localization performance. Even though LocAR can operate completely infrastructure-free, it can nicely integrate with the infrastructure if one is present.

Real-time convergence and computation are two of the critical requirements of an AR localization system specially in mobile applications. Compared to independent particle filtering, our collaborative formulation achieves higher accuracy at the cost of higher computational overhead. Table 2, however, shows that LocAR can still operate on a wide variety of platforms and in practice, converges in real-time. It should be noted that our implementation is not heavily optimized, and our compute time includes significant system overhead. The key takeaway is that the run-time overhead increases almost linearly with the number of users.

In addition to the multi-story building tests described in Section 6, we also performed a series of tests across several other environments under different conditions, many of which are shown in Figure 7. These environments included:

• •

  A ”busy” office (with furniture being moved and lights being turned on and off to simulate ordinary office commotion)

• •

  A campus cafe with a large atrium and spiral staircase

• •

  A hallway intersection near some elevators inside a brick building

• •

  A dimly lit parking garage with height variation and lots of concrete and metal blocking line of sight.

• •

  An outdoor area between campus buildings

The results of these experiments are shown in Figure 15. We see that performance is consistent across all of these environments, with the parking garage performance suffering slightly due to the heavy NLOS conditions and low light. Note that the performance in all of these environments is slightly better than the primary multi-story building test, which was the most challenging due to its immense scale.

Another factor affecting the performance of LocAR’s localization accuracy is the high dynamics of the environment and mobility of users. To this end, we compared the system performance in 3 different walking scenarios: (1) when users were walking in pairs, which represents the near-best performance as the collaborative algorithm can take advantage of clean ranging estimates between each pair of users walking near each other, (2) normal walking when users randomly move in the space with a usual walking speed, (3) when all the users were performing fast movements such as running, jumping, crawling, etc, for the purpose of stress testing the algorithm. Figure 15-b confirms the expected trend for different walking scenarios, and demonstrates that LocAR is resilient to fast motions and therefore suitable for applications that involves fast motions, such as rescue operations or gaming.

Next, we study LocAR’s localization performance in NLOS scenarios. Previous analyses shows that UWB ranging degrades in complete NLOS (Rajagopal et al., 2019) due to noisy Time of Flight (TOF) estimates that mainly capture multipath reflections instead of the direct distance between nodes. To evaluate this effect, we performed 3 different experiments with different levels of NLOS scenarios. The first experiment includes 5 users that walk mostly in LOS of each other, all on the same floor. We then repeated this experiment, while users were walking in a larger space including both LOS and NLOS conditions. Finally, we performed the experiment while users were spread out across 3 floors with some heavy NLOS conditions such as multiple concrete walls between users, or being apart by more than 1 floor. As we can see in Figure 15-b, the 3D relative localization drops slightly with the increase of NLOS conditions, but we can still maintain a median accuracy of  1 m even in NLOS and extended ranges over 10-20 m.