UrbanVR: An immersive analytics system for context-aware urban design

The view presents 3D visualization of an urban area in VR to enable urban context awareness (T.2.1). We employ Unity3D’s built-in VR rendering framework to generate the view. All rendering features are enabled and adjusted to optimal settings, and our system is able to run at interactive frame rates. Selection, manipulation, and navigation operations are enabled in the view through a gesture interaction system described below. Besides, the view also supports animation, which is integrated to facilitate users’ understanding of shadow and visibility. For shadow analysis in animation mode, the user’s position will be moved to the optimal viewpoint for the building candidate, and the system’s lighting direction will change over time according to the sun movement. For visibility analysis, the user’s position moves along the architect’s defined path while keeping the target building in view.

The view supports visualization of quantitative analysis results (T.2.2), which is mainly made up of a parallel coordinate plot (PCP) as shown in Fig. 2. The PCP consists of seven axes, where the middle three correspond to buildings, orientations, and scales of building candidates, the left two correspond to time and shading values for shadow analysis, and the right two correspond to path and visibility results for visibility analysis, respectively. We bend the time axis into an arc to hint sunrise and sundown over time, and arrange the path axis in according to street topology of the path [39]. Since architects would like to first compare building candidates, we integrate bar charts on the left- and right-most axes to indicate shading and visibility distributions for all orientations and scales of a specific candidate. If the candidate’s orientation and scale are further specified, a red line is added in the PCP to show shading and visibility values at certain time and path position.

Besides PCP, various buttons are integrated in the Analytics View. For the middle three axes, plus and minus buttons are placed at the top and bottom, respectively. Users can click the buttons to select a different design, i.e., to change building candidate, orientation, or scale. For time and path axes, start and stop buttons are placed at the top and bottom to control animation. The view is always placed on top of the development site, and if a design is selected, the view will be moved up to the top of the candidate building.

In our setup, viewpoint optimization is applied to a target, e.g., the development site or building candidate. We adapt a LoD simplification approach that first simplifies LoD3 buildings in an urban scene into LoD1 cuboids [42], and then maximizes the visibility for the target. The target can be represented as $\{CT_{i}\}_{i=1}^{n}$, where $CT_{i}$ represents a cuboid belonging to the target. Target can be occluded by other buildings, and we represent these occluding buildings $\{CO_{i}\}_{i=1}^{m}$. Since field of view (FOV) is fixed by the HMD, we can only change the viewpoint position and view direction. We further simplify the problem by always pointing the view ray to the center of the target. Thus, we define the viewport optimization problem as:

given cuboids of a target $\{CT_{i}\}_{i=1}^{n}$, and cuboids of occluding buildings $\{CO_{i}\}_{i=1}^{m}$, to find an optimal position P for observation that generates maximum view goodness of the target.

To solve the problem, we introduce a constraint to optimize P to the target distance (R1), a constraint to avoid P inside occluding buildings (R2.1), and a constraint to minimize view occlusion of the target (R2.2). We formulate these constraints into different energy terms and assemble the energy terms into an energy function.

Fig. 5 presents an illustration of these energy constraints of one building projected onto the XY plane. We further repeat projections and measurements on XZ and YZ planes, and combine results from all three planes together. Detailed procedures are described as follows.

• 1.

  Camera Target Distance ($E_{1}$). After projection, we can get a list of vertices in the XY plane from $\{CT_{i}\}_{i=1}^{n}$. We extract a closed polygon, i.e., $\alpha$-shape from these vertices, which is the boundary of the target projected onto the XY plane. Maximum diagonal distance of the $\alpha$-shape is calculated and denoted as $d_{\alpha}$. Then we use a point to polygon distance function which measures distance $d$ from projected viewpoint $\textbf{P}_{xy}$ to the $\alpha$-shape. The distance is negative if the point falls in the closed polygon. To allow arbitrary viewing angles, we introduce $\theta$ which measures the elevation angle of the viewpoint over the horizon from the center of the $\alpha$-shape to the viewpoint. Thus, we create the sub-energy term on the XY plane:

  $$E_{1}(\textbf{P}_{xy})=\lambda_{0}e^{(d-d_{min})\times(d-d_{max})}+\lambda_{1}e^{(\theta-\theta_{0})^{2}+(\theta-\theta_{1})^{2}}$$

  (3)

  where $\lambda_{0}$ and $\lambda_{1}$ are the weights for each term. $d_{min}$ and $d_{max}$ are preferred distance range and set to $0.5\times d_{\alpha}$ and $1.5\times d_{\alpha}$, respectively. $\theta_{0}$ and $\theta_{1}$ are preferred view directions meeting the condition $\theta_{0}+\theta_{1}=\pi$, and they are empirically set to $\pi/4$ and $3\pi/4$. $E_{1}(\textbf{P})$ is the sum of all sub-energy terms on each plane.

• 2.

  Camera Obstruction ($E_{2}$). After projecting $\{CO_{i}\}_{i=1}^{m}$ onto the XY plane, we can get $m$ polygons $\{CO_{xy}^{i}\}_{i=1}^{m}$. For each $CO_{xy}^{i}$, we measure its distance to $\textbf{P}_{xy}$, denoted as $d_{i}$. The sub-energy term can be constructed as:

  $$E_{2}(\textbf{P}_{xy},CO_{xy}^{i})=e^{d_{i}}$$

  (4)

  To make sure $E_{2}$ becomes large when P falls in any $CO_{i}$, we model $E_{2}$ as:

  $$E_{2}(\textbf{P})=\sum_{i=1}^{m}{\frac{1}{\sum{E_{2}(\textbf{P}_{\Pi},CO_{\Pi}^{i})}}},\Pi\in\{xy,xz,yz\}$$

  (5)

• 3.

  View Occlusion ($E_{3}$). $\{CT_{i}\}_{i=1}^{n}$ are projected on the XY plane, and $n$ corresponding polygons $\{CT_{xy}^{i}\}_{i=1}^{n}$ are generated. For each $CT_{xy}^{i}$, we measure how much another $C_{xy}^{j}$ can occlude it. Here, $C_{xy}^{j}\in\{CO_{xy}^{i}\}_{i=1}^{m}\cup\{CT_{xy}^{i}\}_{i=1}^{n}-\{CT_{xy}^{i}\}$. The measurement is calculated as follows:

  1. 3..1

     Generate a vector $\textbf{Q}_{xy}^{j}\textbf{Q}_{xy}^{i}$ from center of $C_{xy}^{j}$ (denoted as $\textbf{Q}_{xy}^{j}$) to center of $CT_{xy}^{i}$ (denoted as $\textbf{Q}_{xy}^{i}$), and a second vector $\textbf{Q}_{xy}^{j}\textbf{P}_{xy}$ from $\textbf{Q}_{xy}^{j}$ to $\textbf{P}_{xy}$.

  2. 3..2

     Measure angle $\theta_{1}$ between $\textbf{Q}_{xy}^{j}\textbf{Q}_{xy}^{i}$ and $\textbf{Q}_{xy}^{j}\textbf{P}_{xy}$. Lager $\theta_{1}$ means more likely $C_{xy}^{j}$ may lay in-between $\textbf{P}_{xy}$ and $CT_{xy}^{i}$.

  3. 3..3

     Generate vectors from $\textbf{P}_{xy}$ to all vertices in $CT_{xy}^{i}$, and extract two vectors with maximum and minimum angles, denoted as $\textbf{P}_{xy}\textbf{Q}_{xy}^{i0}$ and $\textbf{P}_{xy}\textbf{Q}_{xy}^{i1}$. These vectors represent up- and low-bound view angles for $CT_{xy}^{i}$, respectively. In the same way, extract $\textbf{P}_{xy}\textbf{Q}_{xy}^{j0}$ and $\textbf{P}_{xy}\textbf{Q}_{xy}^{j1}$ for $C_{xy}^{j}$.

  4. 3..4

     Lastly, we measure intersection angle $\theta_{2}$ between $[\textbf{P}_{xy}\textbf{Q}_{xy}^{i0},\textbf{P}_{xy}\textbf{Q}_{xy}^{i1}]$ and $[\textbf{P}_{xy}\textbf{Q}_{xy}^{j0},\textbf{P}_{xy}\textbf{Q}_{xy}^{j1}]$. Larger $\theta_{2}$ means more occlusions.

  We construct the energy term as:

  $$E_{3}(\textbf{P}_{xy})=\sum_{i=1}^{n}{\sum_{j=1}^{m+n}{\frac{1}{e^{\theta_{1}}+1}\times e^{\theta_{2}}}},\quad wherej\neq i$$

  (6)

  $E_{3}(\textbf{P})$ is the sum of all sub-energy terms on each plane.

With these terms defined, we can model the problem as minimization of the following energy:

where $\omega_{i}$ represents the weight for each term, which are empirically set as $\omega_{1}=1$, $\omega_{2}=100$, $\omega_{3}=10$, respectively. $\omega_{2}$ is the largest to prevent the camera from moving inside a physical object, and $\omega_{3}$ is larger than $\omega_{1}$ to minimize view occlusion meanwhile prevent the camera from moving too far away.

Fig. 6 illustrates the effects of different energy terms on viewport optimization. On the left, we adopt the energy function based on defaults weights for camera target distance (E1) and camera obstruction (E2), whilst the weight for view occlusion (E3) is set to zero. The target building (red outline) is obscured by a surrounding building. In the middle, we adopt the energy function based on default weights for E1 and E3, whilst the weight for E2 is set to zero. The camera is now positioned inside a building that blocks the target building. The full energy function based on E1 + E2 + E3 addresses these issues and leads to optimal viewport as shown on the right. Note that we neglect the result of energy function based on E2 + E3, since the camera will be located in a far distance.

To evaluate the egocentric interactions, we design a within-subjects experiment with 12 conditions: 4 interaction mode $\times$ 3 scene complexity. Each participant participated in each condition. For each condition, a target building colored in translucent cyan is positioned in a development site. Participants are asked to select one from multiple building candidates. The selected candidate has different position, orientation, and scale with the target building. Completion time is recorded and used as the evaluation criteria.

• 1.

  Interaction Mode. The gesture system provides fundamental interactions for UrbanVR, while viewpoint optimization and handle bar are complementary for improving user experience. We would like to first test if gestures alone work, and then verify if the other two can facilitate user exploration. Hence, we design four modes of interaction:

  • 1..1

    C1: Gesture only (Ge).

  • 1..2

    C2: Gesture and viewpoint optimization (Ge+VO).

  • 1..3

    C3: Gesture and handle bar (Ge+HB).

  • 1..4

    C4: Gesture, viewpoint optimization, and handle bar together (Ge+VO+HB).

• 2.

  Scene Complexity. In reality, surroundings of a development site can be sparse or dense. High buildings may occlude users’ view of the site, so more interactions are needed to avoid occlusion. This causes more exploration difficulties. As illustrated in Fig. 7, we design three scenes with different complexities:

  • 2..1

    S1: Simple. The surrounding consists of only two buildings; Fig. 7 (left).

  • 2..2

    S2: Moderate. The site is surrounded by six sparsely located buildings. Two of them are relatively low height on one side, from where the space can be viewed without occlusion; Fig. 7 (middle).

  • 2..3

    S3: Complex. The surrounding consists of eight densely located buildings. All buildings are higher than the target building, thus the space is occluded from almost all viewpoints; Fig. 7 (right).

We recruited 15 participants, 10 males and five females. 12 of them are graduate students, and the others are research staff. The age of the participants range between 20 and 30 years. Two participants played VR games using the HTC Vive controllers. No participant has experience with gesture interactions in VR before the study. Three participants have a background in architecture, while the others have no knowledge about urban design.

UrbanVR system was implemented in Unity3D. The experiments were conducted on a desktop PC with 12 $\times$ Intel(R) Core(TM) i7-6800K CPU @ 3.4GHz, 32GB memory, and a GeForce GTX1080 Ti graphics card. The VR environment was running in a HTC Vive VR HMD, and the hand gestures were captured using a Leap Motion attached to the front of HMD.

The studies are performed in the order of introduction, training, experiment, and questionnaire. First, we present a 5-min introduction about the interactions, followed by a 10-min training session to make sure all participants are familiar with the interactions. Then, the experiment starts, and completion time for each experiment condition is recorded. In the end, feedbacks are collected. The questions include if they had experience with gesture interactions in VR, if they feel dizzy, if they think the gestures are natural, and advices for improvement.

In each experiment condition, the starting viewpoint is positioned on top of the urban scene. Participants are asked to complete the task requiring the following operations:

• 1.

  Navigate to the site. This can be either done through gesture-based map navigation, or viewpoint optimization by selecting the site.

• 2.

  Select a building. Participants can open up a menu through left-hand Open gesture, and eight building candidates will be presented at the left hand position. Then participants can select a building that matches the target through right hand Selection. The selected building will be placed besides the menu.

• 3.

  Manipulate the selected building to match the target. Position, orientation, and size of the selected building differ from the target. Participants need to manipulate the selected building to match the target.

All operations are completed while the participants are sitting in a chair. This process is repeated in total 12 times (4 interaction combinations $\times$ 3 scene complexities) for each participant. The participants are asked to take a break (3 minutes) in every 10 minutes, and take a break (5 minutes) after finishing a task. Sequence of the interaction combination is pseudo-randomly assigned to each participant in order to suppress learning effects gained from previous assignments. Each participant was compensated with SGD $40.00 ($\sim$USD $30) after the study.

We postulate the following hypotheses:

• 1.

  H1. Gesture only (C1) is the slowest interaction technique.

• 2.

  H2. Viewpoint optimization (C2) can facilitate gesture only (C1) interaction, i.e., time(C1) $>$ time(C2).

• 3.

  H3. Handle bar metaphor (C3) can facilitate gesture only (C1) interaction, i.e., time(C1) $>$ time(C3).

• 4.

  H4. All techniques together (C4) is the fastest technique, i.e., time(C2) $>$ time(C4) and time(C3) $>$ time(C4).

15 experiment results are generated for each experiment condition. We first test the results in each condition against normal distribution using Shapiro-Wilk test. The results show that conditions of C2 do not follow normal distribution, while the results in other conditions are following normal distributions under certain probability.

Prerequisites for computing ANOVA are fulfilled for H1, H3 and H4. We perform a two-way ANOVA (3 interaction combinations $\times$ 3 scene complexities) on them. Significant effects of interactions on completion time $(F_{(2,126)}=64.97,p<0.001,\eta^{2}=1.718)$ are observed. Scene complexity also has a significant main effect on completion time $(F_{(2,126)}=59.63,p<0.001,\eta^{2}=1.646)$. We use Kruskal-Wallis test [44] to evaluate H2. The result shows the viewpoint optimization has significant effect on completion time $(H_{(2)}=25.53,p<0.001,\eta^{2}=0.560)$.

We also perform post hoc comparisons of completion times among the interaction combinations. The results are shown in Fig. 8. C1 is on average more than 33s slower than C3 $(t_{(44)}=33.42,\ p<0.001,\ d_{Cohen}=0.882)$, while C3 is on average 23s slower than C4 $(t_{(44)}=22.84,\ p<0.001,\ d_{Cohen}=0.955)$; C1 is on average more than 32s slower than C2 $(t_{(44)}=32.38,\ p<0.001,\ d_{Cohen}=0.772)$, while C2 is on average 24s slower than C4 $(t_{(44)}=23.89,\ p=0.001,\ d_{Cohen}=0.799)$. The results confirmed H1, H2, H3, and H4. Through more detailed probes, we figure out: C1 is slower than C3 $(t_{(14)}=21.07,\ p=0.018,\ d_{Cohen}=0.921),\ (t_{(14)}=23.93,\ p=0.011,\ d_{Cohen}=1.032),\ (t_{(14)}=55.37,\ p<0.001,\ d_{Cohen}=1.698)$, while C3 is slower than C4 $(t_{(14)}=16.73,\ p=0.003,\ d_{Cohen}=1.225),\ (t_{(14)}=23.20,\ p=0.002,\ d_{Cohen}=1.336),\ (t_{(14)}=28.60,\ p=0.002,\ d_{Cohen}=1.399)$; C1 is slower than C2 $(t_{(14)}=19.20,\ p=0.119,\ d_{Cohen}=0.656),\ (t_{(14)}=32.067,\ p=0.035,\ d_{Cohen}=1.284),\ (t_{(14)}=45.87,\ p=0.156,\ d_{Cohen}=1.242)$, while C2 is slower than C4 $(t_{(14)}=18.60,\ p=0.148,\ d_{Cohen}=0.816),\ (t_{(14)}=15.07,\ p=0.170,\ d_{Cohen}=0.765),\ (t_{(14)}=38.00,\ p<0.031,\ d_{Cohen}=1.413)$ for S1, S2, and S3, respectively. We use False Discovery Rate [45] ($\alpha=0.05$) for the correction of above data. The results suggest that for more complicated scenes, handle bar metaphor and viewpoint optimization techniques make interactive VR exploration more efficient.

All participants finished the experiment in 1.5 hours. No one felt dizzy with our system in 10-min tests, but one felt a bit eye dry. All participants agreed that the gestures are natural and easy to use, and viewpoint optimization and handle bar are quite helpful. Three participants especially liked the viewpoint optimization, as they got easily bored and tired when navigating the map using gestures. “Viewpoint optimization is really cool. I think too much navigation just makes users feel bored.” There are also some negative feedbacks. Two participants felt the HMD resolution is not high enough, which reduces immersive feeling. One participant was not able to manipulate the building precisely, even with the handle bar. It took her a long time to complete the task.

The experts agreed that VR is a new technology that is certainly worth exploration for urban design. Both experts appreciated our work in applying immersive analytics for urban design. They know some planning teams are exploring VR and AR technologies. However, these works are more “focusing on demonstrations and lack analysis”, according to EA. In comparison, our system well integrates visualization and analysis. EB also commented that UrbanVR provides “immersive feelings of an urban design, and more importantly, being able to modify the plan and provide quantitative results”.

Both experts had no motion sickness with the visualization. They are familiar with the urban scene studied in the work, yet they did not expect to get much more immersive feelings in VR than desktop 3D visualizations. “The animation is very vivid, especially when navigating on the path” in the visibility analysis, commented by EB. The experts had some difficulty in understanding the PCP in the beginning. EA thought some simple line charts would be more effective. He was persuaded after we explained that there could be too much line charts to represent all visibility and shadow analysis. They liked the idea of bending time-axis into a curve, and arranging path-axis in the same topology with the streets. “These small adjustments give me strong visual cues of reality”, commented by EB. The study clearly reveals that for the development site, shading is determined by building heights, while visibility is mainly affected by building width. The collaborating architect appreciated this finding and praised the bar chart design on the shading and visibility axes.

Both experts got used to the gesture interactions quickly. They tried both building manipulation and map navigation using gestures, and felt the gestures are easy to use in VR. EA especially liked the handle bar metaphor, which “helps a lot when manipulating the buildings”. In the beginning, the experts did not realize that when visualizing the case study, the viewpoint was optimized. After the demonstration, they liked the idea and felt it is necessary, as EA felt “the HMD is too heavy - not suitable for long-time wearing”.

The experts pointed out two major limitations in our system. First, the experts felt the “gestures are not comparable with mouse regarding accuracy”, even though we have provided handle bars for visual feedbacks. Nevertheless, they liked gestural interactions because hand guestures are natural, and they encouraged us to further improve the accuracy. Second, the experts would like to see more analysis features integrated into our system, such as building functionality, street accessibility, transportation mobility, etc. The current analyses are not covering all necessary evaluation criteria they need.