VisAR: Bringing Interactivity to Static Data Visualizations through Augmented Reality

Augmented reality allows users to have a seamless experience between their world and content others have created. One interesting aspect of AR is that it can breath life into static content. In the AR domain, there have been numerous projects that animate static objects by adding interactivity to the once-inanimate experience. For example, Billinghurst’s MagicBook was a novel AR interface that allowed static components of a physical book to be interactive to the reader [6]. This lead to various follow up studies that experimented with adding interactivity to physical books [10]. Researchers investigated how AR solutions affect user experience while performing tasks. For example, in one study, researchers observed that students using AR-enhanced books were better motivated and more engaged in the material [5] compared to using other methods.

The power of animating static content has also been proven by the widespread usage of the commercial product Layar [2]. Layar provides AR experiences that enhance printed material to be digitally interactive on everyday smartphones. The resulting artifacts showed an 87% click-through rate which is overwhelming compared to single digit click-through rates of other advertisement methods. This shows that when properly designed, using AR for the purpose of adding interactivity to static content has a potential of increasing user engagement.

In the information visualization domain, there have been other attempts to merge AR with information visualization. In the sense that AR inherently couples virtual content with the user’s physical surroundings, White defines situated visualization as visualizations that are coupled with the context of physical surroundings [23]. Our approach is somewhat different from White’s view of situated visualization as we ignore the physical context in which the visualization is placed in and focus on extending the capabilities of the visualization regardless of the context.

In the interaction space, Cordeil et al. coined the concept of spatio-data coordination which defines the mapping between the physical interaction space and the virtual visualization space [8]. When structuring our approach through this concept, we see that bystanders of visualizations cannot interact with the visualizations, having no coordination between their interaction space and the visualization space. By introducing additional elements in the visualization space that have spatio-data coordination with personal interaction spaces, we allow users to directly interact with the visualization and view personalized content in their own display space.

Our work is also inspired by the information-rich virtual environment concept introduced by Bowman et al. [7]. The concept looks into interactions between virtual environments and abstract information. If we consider a synthetic static visualization as a virtual representation of data, our solution provides abstract information about that environment. Users can perform interaction tasks that point from the virtual environment to abstract information; that is, users can retrieve details associated with a synthetic visualization.

Other studies have investigated view management systems that adaptively annotate scenes with virtual content [21, 4]. These systems adjust annotations according to various factors such as the user’s viewpoint or the amount of crowded content on the screen. The results of these studies have direct implications to our system as we rely on annotations on static visualizations.

In this section, we discuss a couple of scenarios that motivate using AR techniques to enable people to gain more insights through interacting with static visualizations.

The analytical workflow starts with a preparation stage which involves accessing the AR application that contains the underlying dataset that was used for creating the visualization. The user then observes a typical static visualization (visualization target) shown on a poster or digital screen through an AR device. The device uses it’s camera and an image-based tracking system such as PTC’s Vuforia [15] to track the visualization as a target. Depending on the number of trackable features, the target can either be a fiducial marker that is carefully positioned on the visualization or simply the entire static visualization itself. Once the AR device recognizes the target, it superimposes virtual content onto the static visualization. For example, in the case of a static scatterplot visualization, the system renders a new set of virtual data points and overlays it onto the static version of the visualization. Users are then able to see and interact with the virtual content that is superimposed on the static visualization. User interaction with the virtual content can be through voice or/and gestures. For example, users can filter a specific set of data points by saying “Filter out cars above $40,000”. Similarly users can see detailed information about a specific data point by simply looking directly at the data point.

To indicate the feasibility of our idea, we implemented a prototype system on the Microsoft Hololens [14] which has accurate tracking and rendering capabilities and an RGB camera that can be used for target recognition. No additional adjustment to the device was required. For the computer vision solution on tracking target images we use PTC’s Vuforia which has support for the Hololens platform. We prepared an example static visualization and defined the entire visualization as the target visualization because it had enough features to qualify as an image target. The visualization system was built using the Unity Editor for Hololens while the example static visualization was built with the D3 library.

We developed VisAR to show the feasibility of using AR devices to bring interactivity to static data visualizations. The current version of VisAR supports two types of visualization techniques (bar chart and scatterplot) and four interaction techniques (details on demand, highlighting, filtering and linking views). We view the current version of VisAR as the early step towards exploring the applications of augmented reality in data visualization. However, generalizing the usage of AR devices for interacting with static visualizations requires support of more sophisticated analytic operations and visualization techniques. For example, how can users perform brushing and linking or zooming on static visualizations using AR technologies? Multiple avenues for future work lie in improving the VisAR interface. We envision expanding VisAR to include other visualization techniques (e.g., linecharts) and interaction tasks (e.g., zooming, panning).

Previous work in the AR community indicates that using AR improves user engagement compared to using non-AR material [5]. However, it is not clear to the visualization community how usage of AR solutions affect user experience during the visual data exploration process. An important avenue for continued research is conducting an in-depth study utilizing both qualitative and quantitative techniques to measure the impact of AR solutions in data visualization compared to non-AR solutions, using various usability (e.g., time and error) and user experience (e.g., engagement [16, 17]) metrics. We hypothesize that using AR solutions increases user engagement in working with data visualizations, but this remains to be formally studied.

Many AR solutions rely on two interaction modalities: speech and gesture. There are advantages in using speech and gesture in visual data exploration since they enable users to express their questions and commands more easily [19]. However, these interaction modalities also bring challenges such as lack of discoverability. How does a user recognize what the possible interactions are? How does a user know what the exact gesture or voice command is for performing a specific interaction? One interesting research avenue is to investigate methods that make possible interactions more discoverable in such systems.

In the current version of VisAR, we are using a high-end AR device (Microsoft Hololens) which currently might not be a feasible option to use for most people. However, with recent advances in smartphone technology, currently available off-the-shelf smartphones are also powerful enough to be used as a personal AR device. By using a smartphone as a hand-held AR device, the current implementation can be ported to have identical capabilities. We recommend using cross-platform AR systems such as Argon [1] to allow users to be able to access the same experience regardless of the underlying device or operating system.

While the benefit of 3D visualizations in immersive AR settings are still yet to be further discussed [3], in this paper we examined a different aspect of AR that takes advantage of 2D visualizations. As AR allows us to add visual interactive components to static scenes, we utilize this capability to add interactivity to static visualizations and enable users to independently address personal data-driven questions. Through the prototype of our solution, we demonstrate that the idea is feasible to be implemented on currently available AR devices. Although further study is needed to measure the advantages of the system, with the abundance of personal AR devices such as smartphones, we believe our solution is not only powerful but also practical.