RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual Effects

Volumetric captures or videos refer to the technique of capturing 3D space and subsequently viewing it on a screen with free-viewpoint movement. These techniques have been explored since the 1990s (e.g., Virtualized Reality (Kanade et al., 1997)), but recent research has greatly advanced this domain in both high-quality 3D reconstruction (e.g., Fusion4D (Dou et al., 2016), Montage4D (Du et al., 2018), Relightables (Guo et al., 2019), VolumeDeform (Innmann et al., 2016)) and more accessible volumetric capturing with mobile phones (e.g., Kinect Fusion (Izadi et al., 2011), DepthLab (Du et al., 2020), Polycam (PolyCam, [n.d.])). With recent advances in commercially-available depth cameras like Kinect, volumetric captures have been used in various applications such as telepresence (e.g., Holoportation (Orts-Escolano et al., 2016), JackInSpace (Komiyama et al., 2017), Project Starline (Lawrence et al., 2021), PhotoPortals (Kunert et al., 2014)), remote collaboration (e.g., RemoteFusion (Adcock et al., 2013), Mini-Me (Piumsomboon et al., 2018), On the Shoulder of Giant (Piumsomboon et al., 2019), Virtual Makerspaces (Radu et al., 2021)), remote hands-on instruction (e.g., Loki (Thoravi Kumaravel et al., 2019), BeThere (Sodhi et al., 2013), 3D Helping Hands (Tecchia et al., 2012)), and immersive tutorials for physical tasks (e.g., MobileTutAR (Cao et al., 2022), ProcessAR (Chidambaram et al., 2021), My Tai Chi Coaches (Han et al., 2017)). Past research has utilized static or live 3D reconstructed scenes for remote MR collaboration, facilitating more immersive interactions with remote users (Tait and Billinghurst, 2015; Gao et al., 2016; Teo et al., 2019). Alternatively, live 3D reconstruction has been used to facilitate co-located communications for VR users (e.g., Slice of Light (Wang et al., 2020), Asynchronous Reality (Fender and Holz, 2022)). These captured 3D geometries are also used for anchoring virtual elements (e.g., SnapToReality (Nuernberger et al., 2016), SemanticAdapt (Cheng et al., 2021)), creating virtual contents (e.g., SweepCanvas (Li et al., 2017), Window-Shaping (Huo and Ramani, 2017)), or generating virtual environments (e.g., VRoamer (Cheng et al., 2019), Oasis (Sra et al., 2017, 2016)) by leveraging object detection and semantic segmentation of volumetric scenes (e.g., SemanticPaint (Valentin et al., 2015), ScanNet (Dai et al., 2017)).

More closely related to our work, past work has also explored further blending the virtual and physical worlds by augmenting captured volumetric scenes or the real world. By using VR/MR devices, systems can alternate the captured scene by erasing physical objects (e.g., SceneCtrl (Yue et al., 2017), Diminished Reality (Cheng et al., 2022; Mori et al., 2017)) or replacing them with virtual ones (e.g., RealityCheck (Hartmann et al., 2019), TransforMR (Kari et al., 2021)). Alternatively, previous work has used the depth information to blend virtual augmentation into the real-world with projection mapping (e.g., IllumiRoom (Jones et al., 2013), RoomAlive (Jones et al., 2014), Room2Room (Pejsa et al., 2016), Dyadic Projected SAR (Benko et al., 2014), OptiSpace (Fender et al., 2018)). Systems like Mixed Voxel Reality (Regenbrecht et al., 2017), Remixed Reality (Lindlbauer and Wilson, 2018), and Virtual Reality Annotator (Ribeiro et al., 2018) further advance this approach by augmenting the volumetric scene by leveraging both spatial manipulation (copy, erase, move), temporal modification (record, playback, loop), and volumetric annotation (sketches) with a VR headset and live 3D reconstruction.

While these works partially demonstrated the visual augmenting of captured scenes, supported augmentation techniques remain simple (e.g., appearance change for color or texture). Moreover, since their focus is on the immersive experience of these modified scenes, the authoring aspect of these volumetric scenes and videos is not well explored in the literature. Our focus is rather on the authoring interface, which can support more comprehensive visual augmentation for the volumetric scenes. This is because the current work on authoring tools or video-editing tools for volumetric capture is either focused on static scenes (e.g., DistanciAR (Wang et al., 2021)), timeline manipulation (e.g., 4Dfx (4DViews, [n.d.])), or simple video touch-ups (e.g., DepthKit Studio (DepthKit, [n.d.]), HoloEdit (Studios, [n.d.])). In contrast, RealityEffects enables more expressive visual augmentation for dynamic volumetric scenes by leveraging object-centric augmentation, which we take inspiration from 2D video authoring, as described next.

To design our system feature, we first need to understand and investigate common practices for object-centric augmentation. In this paper, object-centric augmentation refers to “a class of virtual elements 1) that are embedded and spatially integrated with objects in a scene, and 2) whose properties change, respond, and animate based on the behaviors of physical objects in the scene.”. Here, virtual elements can include text, images, visual effects, and visualization; objects can be physical objects, parts of the human body, or environments; and properties can encompass location, orientation, scale, and other visual properties.

While object-centric augmentations are frequently used in many professional videos and several works explore this domain (Goldman et al., 2008), these works lack a taxonomy analysis (Suzuki et al., 2020; Liu et al., 2020; Goldman et al., 2008; Saquib et al., 2022) or focus on more specific domains such as presentations (Liao et al., 2022), storytelling (Saquib et al., 2019) or robotics (Suzuki et al., 2022), leaving a gap in the holistic understanding of possible designs, even for 2D videos and, certainly, for 3D volumetric videos. The goal of this taxonomy analysis is to provide initial insights into object-centric augmentations. We have adapted methods from similar prior research papers (Liao et al., 2022; Xia et al., 2023; Monteiro et al., 2023) to provide an initial and preliminary taxonomy of a representative subset of common practices, recognizing that conducting a systematic visual search of videos is more challenging than conducting a systematic search of research papers.

To collect the video examples, the authors (A1, A2, and A4) manually searched popular video and image search platforms (e.g., YouTube, Pinterest, Vimeo, Behance, and Google Images), primarily relying on visual searches, as these videos are not associated with a specific keyword like “3D visual effects”. After some initial filtering, we started to identify some patterns in the visuals we collected, and with the help of the similar image suggestion feature on Pinterest, we expanded more visual search criteria like annotations, highlights, augmented effects, labels, floating text, floating screen, analysis, visualization, and motion. We also did a reverse search to find the videos, through this process, we first collected 200 videos. Note that there is a much smaller proportion of examples for 3D volumetric videos, none of them are volumetric videos, while most of them feature 3D visual effects. Then, the authors (A1, A2, and A4) filter out by focusing only on the object-centric augmentation (e.g., removing videos that use entirely virtual effects without physical objects or visual effects that are not associated with the physical objects). After the filtering process, we obtained 120 videos that contain object-centric augmentation based on our definition,

We analyzed all 120 collected videos to identify snippets displaying annotations and visual effects, capturing screenshots of object-centric visual effects from each. Through this process, one of the authors (A1) led the collection of screenshots, with assistance from another author (A2), resulting in a total of 336 screenshots, averaging 2.8 screenshots per video. We chose screenshots over full videos as a coding corpus because each video may contain different techniques, and these screenshots served as representative keyframes for our taxonomy analysis. Subsequently, we conducted open coding to identify a preliminary taxonomy of object-centric visual effects. With the 336 collected representative images, author A1 led the initial open coding process to identify a first approximation of the dimensions and categories, then iterated with other authors (A2 and A4) on digital whiteboards (Miro and Google Slides). In this process, the three authors independently reviewed the collected screenshots and refined the taxonomy initially identified by A1. Subsequently, all authors reflected on the initial design space to discuss the consistency and comprehensiveness of the categorization. Finally, after systematic coding by authors A1 and A2, which involved individual tagging for the complete dataset, we reviewed the tagging to resolve discrepancies and obtain final coding results. All authors then reflected on the design space and finalized the categorization by merging, expanding, and removing categories.

We acknowledge several limitations in our current methodologies, including corpus selection and taxonomy analysis. First, our selected videos may not represent a comprehensive and exhaustive corpus. While we aimed to collect as diverse a dataset as possible, the nature of our visual search, rather than a systematic keyword search, limits our ability to claim comprehensive representation. Second, the taxonomy analysis might have benefited from the involvement of the video creators to better capture the design space from their perspectives. Despite these limitations, we believe this taxonomy can help identify common practices and techniques for object-centric augmentation, benefiting both our own and other HCI research.

Text annotation is one of the most common techniques identified. It involves attaching textual labels or descriptions to physical objects. These can be static descriptions, providing information about the object, or dynamic data and parameters, such as speed, distance, or price, akin to embedded data visualization (Willett et al., 2016). The attached objects can be graspable physical items, parts of the human body, or stationary locations like buildings or furniture.

Object highlight is a technique used to visually attract an audience’s attention to a specific object. For example, object highlight techniques include changing the color of the object, highlighting the contour of the object, adding highlighting marks to the object, or changing the opacity of other objects. These object highlights can be applied to either 2D surfaces, such as showing a colored circle on the ground around cars, phones, or pans, or 3D objects, such as displaying a bounding box and sphere or 3D mesh of the target object.

Embedded visuals are 2D images or visual information attached to describe objects, similar to text annotation but through static visuals or animations. Embedded visuals include simple icons to describe the object, 2D images and photos to show the associated information, animation to visually describe the behavior, screens to display the associated website or user interfaces, and charts or graphs to visualize the associated data.

Connected links are lines that indicate the relationship between two elements. These connected lines can be object to virtual elements, linking text annotations or embedded visuals to a specific object to indicate which object is being described. Alternatively, the connected lines can be object to object, explaining the relationship and association between multiple physical objects, such as indicating network communication between multiple IoT devices or visualizing the connection between different body parts like arms or legs. These connected links can dynamically move and animate whenever the physical objects move.

Motion effects are techniques used to visualize the motion of physical objects. Most commonly, motion trajectories are used to show the path a specific object moves, such as illustrating the trajectory of a golf swing, baseball batting swing, and body movement in gymnastics. Alternatively, some videos leverage slow-motion morphing effects or ghost effects to depict the trajectory of the entire body or objects, similar to the famous bullet-time effects in the movie Matrix.

While much less common, we have also observed several other effects, such as particle effects and virtual 3D animation. However, since object-centric augmentation already leverages the dynamic motion of physical objects, simple visual augmentation can significantly make the video more expressive and enrich the viewing experience.

The streaming module utilizes the off-the-shelf Azure Kinect depth camera SDK to capture volumetric point-cloud data. The data feed includes both RGB and Depth data in separate channels, each with a resolution of 640 × 576 and a refresh rate of 30 FPS. Both channels share the same (x,y) coordinate data structure, enabling us to retrieve the depth information for any (x,y) tracking point. The obtained RGB-D data is then passed to the processing module.

With the RGB-D data feed, the application performs 3D scene reconstruction by rendering the 3D point cloud data directly using Three.js, where z = Depth(x,y) and RGB = Color(x,y). We utilize MediaPipe Pose Estimation for body tracking and OpenCV for object tracking. The application calculates the centroid by averaging the (x,y) values, retrieves the depth information with the (x,y) coordinates, and registers the centroid as the attachable object in the authoring interface for further augmentation.

With the attachable object from the processing module, RealityEffects allows users to select objects from pose estimation and color tracking and augment them with object-centric annotations and dynamic visual effects. The object-centric annotations are essentially Three.js coded objects such as static text labels and bounding boxes, with a bloom pass to create glowing highlight effects. The dynamic visual effects are parameterized object motions that allow us to create visualizations from the motion and parameters, such as trajectory, position, distance, and angle. Users can augment the moving object with motion effects like a trajectory (a series of points) and a trailing effect ²²2https://drei.pmnd.rs/?path=/docs/misc-trail--docs, and augment the motion with embedded visualizations using an iframe to create charts and interactive widgets. The augmentation can be applied to a real-time camera data feed, allowing users to review their own performance as it’s being annotated, which unlocks several application scenarios like sport analysis and e-commerce live streaming. Users can also freely move or zoom the camera in 3D space through mouse movements. When using a recorded volumetric video, the system supports simple pause and play functionalities. Since we only use a single Kinect camera, capturing the entire room is challenging. Therefore, we also scan the room with a static 3D scanner (iPad Pro 12-inch with LiDAR camera and 3D Scanner App) and place it as a 3D volumetric background asset (glTF file) only for visual aesthetic purposes in most of the figure and video demonstrations.

First, for the colored physical object, the system tracks the object’s 3D position based on the combination of color tracking and point-cloud information. When the user clicks an object, the system gets the current RGB value of the clicked points in 2D screen. Then, the system captures a similar colors based on an upper and lower threshold range of RGB ($\pm$ 10) to obtain the largest contour in the scene, based on Node OpenCV library. Given the detected object in the 2D scene, the system raycasts to the volumetric scene to obtain the associated point-cloud depth information, which allows us to get the coordinated 3D position in the scene.

For post estimation and body tracking, we simply use MediaPipe to get the estimation of 33 body tracking points. We also tried Kinect-builtin body tracking feature, but the performance was not satisfying because of high latency and low accuracy. Similar to color tracking, the system allows the user to directly select one of the tracking points of the body skeleton, and the system automatically calibrate the 2D coordinates with the depth information. When the user enters the body selection mode, then the system shows the twenty body skeleton points which the user can select. When the user selects a certain skeleton parts, then it becomes highlights and starts tracking in the 3D scene.

For stationary location, the user can simply select a location in the scene and use a ray cast to obtain the stationary 3D position in the physical environment, such as floor or wall. The user can also place it in mid-air by moving the point with a mouse. In this selection, the tracked point is stationary, thus there is no dynamic movement. However, this tracked location can be used as a reference point, such as a distance from a certain location.

First, the user can bind the text label to the associated physical object in the volumetric 3D scene. To place a text annotation, the user specifies the associated object and then clicks the text label button. Then, the system starts showing the 2D text label floating around the tracked object. Since the attached text label is bound to the object, the text label position moves when the object moves. The user can change the text value by typing the name in the menu window. The user can also add a dynamic value by using a variable, based on the JavaScript variables such as Date.now(), or a user-defined variable based on the dynamic parameters, such as position, speed, angle, distance, etc., as we describe in Step 3. While the text is a 2D object, it always moves its orientation to face the camera. The user can also disable this to

The user can also add an object highlight bound to the tracked object. The system supports two basic object highlight options: 1) 3D primitive shapes, such as bounding box, bounding sphere, and bounding cylinder, and 2) 2D highlight shapes such as colored circles or rectangles. To add the object highlights, the user first selects the tracked object and then chooses the object highlight button in the menu. Then, the system lets the user choose the shape of the highlight (default: 3D sphere), then the object highlight is added to the scene. Unlike text annotation, the object highlight is placed in the center of the tracked object. The user can also change the scale, offset, orientation, and color accordingly through a direct manipulation interface.

The user can also add embedded 2D visuals. Informed by the taxonomy analysis, the system supports images, icons, videos, and embedded websites as the associated visual aids. From the technical point of view, all the embedded visual is implemented as embedded iframe in Three.js. Therefore, the image, YouTube video, or website can be embedded as an iframe by specifying the URL or local file. To add the embedded visual, the user can also select the object and enter the embedded visual menu. Then the user can enter the URL or file directory. Once loaded, the added visual elements start following based on the object’s movement. Again, the user can also change the size, orientation, and opacity of these elements. Since the embedded visual is an interactive HTML, the user can also interact with the screen such as buttons or links. By leveraging this feature, we can also embed dynamic graphs and charts by associating the dynamic parameter, as discussed in Step 4. By default, the 2D visual always changes its orientation to face the camera, but the user can also change it by disabling it.

The system can obtain the 3D position of the tracked object by simply getting the current position value. The user can use this dynamic value in text labels or dynamic graphs by using the specific variable. In the system, the user can use this value by using the variable like obj_1.x.

The system also obtains the speed for all the tracked objects, by calculating

where $t_{1}-t_{0}$ is every 0.5 second (every 15 frames with 30 FPS). The user can access this information by using the variable like obj_1.speed.x

When the user selects multiple objects, the system also calculates the distance between the two objects. If the user also wants to show the line between the two points, the user can enter the geometry menu, and then add the connected line between two tracked points. Then, the line geometry automatically changes its length and orientation based on the tracked objects. The endpoint of the line is not the dynamic object but can be also a stationary location such as a specific point in the scene. The user can use the distance information by using the variable like distance_1

When selecting the three tracked objects, then the system also calculates the angle between the two lines. In this case, the angle is calculated given the two 3D vectors $a,b$ through $\theta=cos^{-1}[(a\cdot b)/\|a\|\cdot\|b\|)]$, where $cos^{-1}$ is arc cosine and $a\cdot b$ is a dot product. The user can use the angle data by using the variable like angle_1

In the same way, the user can also obtain the dynamic parameter for the area of three points (the area of a connected triangle) or four points (the area of a connected rectangle). The user can use the area by using the variable like area_1

Dynamic text annotation is the text annotation described in Step 1 but uses the parameterized value in the 3D scene. For example, if the user types the text value as PositionX: ${obj_1.x}$, then the system shows the parameterized value in the text label, which is shown as like PositionX: 34.23.

Similarly, the user can also bind the dynamic parameter to the visual property of the embedded virtual objects, such as scale, rotation, position, opacity, and color. For example, if the user associates the scale of the virtual object with the position of the tracked object, then the embedded virtual object’s size changes in response to the position of the tracked object.

The user can also show the dynamic graph by associating the dynamic value with the charts. As we mentioned, we can embed the interactive 2D data visualization with iframe. We prepare several basic graphs such as line charts, bar charts, or pie charts, based on the Chart.js library. For example, if the user associates the y value of the line graph as an angle of between the arm and body, then the system shows the real-time line chart to show the tracked parameter.

Alternatively, the system can also show the motion effects with several prepared visual effects. For example, the user can show the motion trajectory of the tracked object. To do so, the user selects the motion trajectory option in the menu, then the user selects the object. Then, the system starts the trajectory path of the motion, based on the object location. To implement this, we simply place a small sphere in the position of the tracked object for each frame, then disappear for a certain duration (5 seconds).

Finally, the system also supports the ghost effects by duplicating the tracked object’s geometry. To do this, we simply clone the entire tracked object for every second, so that the user can see the ghost effect.

The study was structured into two sessions: the first aimed at evaluating the prototype’s usability to ascertain its effectiveness, ease of use, and learnability through structured tasks and a survey. Before the first session, we inquired about participants’ experience with 3D graphics software development tools like Unity 3D, Blender, and Unreal Engine. Identifying experienced participants helped in gaining deeper insights during the follow-up conversational interviews. The second session involved an in-depth interview to discuss the system’s benefits, limitations, and potential improvements.

The user study was designed to measure specific usability factors such as learnability, satisfaction, and ease of use. The total duration for the study was between 45 to 60 minutes per participant, structured as follows:

We asked the participants at the beginning of our survey to better learn about their background and demographics towards mixed reality experience, 3D graphics software, video editing experience, and volumetric video experience. The collected demographic information is shown in Figure 9. Non-surprisingly, many of our participants do not have 3D graphics development experience or volumetric video experience. Many of them also mentioned that it was their first time hearing about 3D video or 3D volumetric video.

As shown from the Figure 9, which summarized

The Figure  9 outlines the study results. Overall, the vast majority of participants had positive responses. Participant scores of the overall experience averaged 5.7/7. ”The user experience was fun and interesting. It was pretty intuitive as well”(P9) and ”It is such a fresh and interesting experience for me to play a 3D reality product.” (P13). Some participants also had optimistic views towards the system Seems interesting and could have potential uses for video editing software(P16).

The system was determined to be fairly easy to use, averaging 5.4/7. P8 found ”It was intuitive” and said ”the user panel was accessible”. However, a (P5, P10, P13) found it hard to make selections. However, others(P9, 19) with experience with 3D software found it easy. P13 who was unfamiliar with using software with 3D spaces found it difficult to navigate. P3 put up concerns with certain demographics unfamiliar with 3D software and that it might seem overwhelming. However, they did find the streamlined interface made animation and editing so much faster P3.

The creative freedom of the system was reported to be flexible, with an average score of 5.6/7. Regarding the variety of the features, P1 declared that they could imagine multiple uses for them. Another said The trail feature and ghost effect inspired my creativity P10. Body motion features resonated more and had positive feedback in terms of creative freedom, P19 said the ghosting and trailing effects were features they would personally use for creative reasons.

One declared the tracking and binding features were necessary, as labels and highlights would likely not work without them. (P2) From the questionnaire, highlighting had positive feedback and that the torus seems very useful for highlighting objects in the scene (P8, P11)

When asked about the potential applications and use cases, many participants see the potential for sports analysis (P3, P10) and data visualization (P7, P14). For example, P3 says ”On the note of sports, like for example, in ballet, your position is very important. So if you’re doing plays like, for example, videos where you highlight certain points like knees they have to be at a certain angle.”. As the participants say, our tool allows the user not only to analyze the sports from different angles, but also to measure the trajectory or posture in improvisational and interactive ways. Also, the participants see the future potential of video streaming. For example, one of P4 says that ”I would use this in my stream in some way to incorporate effects. And maybe even have audience members triggering different things in my space” The feedback points to a future when 3D volumetric videos become mainstream. The participants see that tools like RealityEffects allow such a video streaming medium more interactive and engaging.

While the feedback was generally positive, several limitations were noted. P7 mentioned that the tool’s functionality is currently limited, particularly the types of 3D shapes available for highlighting objects, which are restricted to simple geometric forms. Many participants pointed out the need for improvement in tracking accuracy, especially with the color-based tracking system’s susceptibility to slight environmental changes, such as lighting conditions or shadow occlusion. Future work should explore alternative tracking methods for 3D objects, given that volumetric object tracking remains an active research area and advances in this field could significantly enhance object-centric 3D video augmentation.

System limitations also include the requirement for physical interaction or assistance from another person, as stated by P4. P3 elaborated on this by mentioning the extensive setup required for video capture, including the need for an open area, camera rig, and trackable objects. Additionally, the tool currently lacks complex time manipulation features found in traditional video-editing tools. Integrating our features into volumetric video-editing platforms could create richer experiences.

Participants also criticized the quality of 3D capturing. Using only a single Azure Kinect depth camera limits the capture area and fails to record occluded regions. Although we integrated a static 3D scene as a background to mitigate this, the limited capture area restricts applications requiring dynamic movement across larger areas. A potential solution could involve using multiple Kinect cameras, similar to approaches like Remixed Reality (Lindlbauer and Wilson, 2018). While this would allow more immersive visualization, it would also increase computational demands and the complexity of the calibration process. Future work should consider incorporating multiple depth cameras to support activities requiring broader interaction spaces.

Direct manipulation in RealityEffects enables users to feel an immediate connection with the digital content, fostering a sense of control and ownership over the creative process. However, we recognize the importance of considering alternative approaches that could complement or enhance the user experience. Automatic annotation, for example, could offer efficiency benefits by reducing the manual effort required to label and annotate volumetric data. This method could automatically identify and label objects within a scene using advanced machine learning algorithms, which would be particularly useful in complex scenes or for users who require quicker workflows. A suggestion-based interface is another compelling alternative that could blend the strengths of direct manipulation with the efficiencies of automation. By providing users with intelligent suggestions based on context, previous actions, or common patterns, this approach could accelerate the editing process while still allowing users the freedom to make final decisions. Future work could explore these alternatives to support a wider range of user preferences.

Currently, we focus on desktop authoring interfaces due to the complexity of interactions and manipulations involved. However, future investigations could explore opportunities within immersive environments using mixed reality or virtual reality headsets. Such environments would present unique design and technical challenges, such as selecting objects and streaming large amounts of data between the host computer and the headset. Addressing these issues could lead to innovative solutions for immersive augmentation, and we are keen on developing these capabilities for devices like the Hololens.