Viewport Prediction for Volumetric Video Streaming by Exploring Video Saliency and Trajectory Information

Efficiently predicting the viewer’s field of view (FoV) is crucial for immersive videos, such as 360 video and point cloud video. Linear regression (LR) [26, 23] is widely used, along with variations such as weighted LR, KNN-based methods, and the incorporation of confidence values [23, 24, 27] to enhance accuracy. For example, S. Gül et al. [28] employ LR to predict each of the 6DoF, resulting in two-dimensional videos. Y. S. de la Fuente et al. [27] predict future positions by monitoring the angular velocity and acceleration of head movements.

With the advancement of neural networks, these techniques have been employed to predict the viewport. For instance, S. Yoon et al. propose a 6DoF AR algorithm for visual consistency, and X. Hou et al. employ LSTM and MLP models for low-latency 6DoF VR using tracking devices [29, 7].

However, these methods only consider the temporal characteristics of user head motion and do not account for the impact of video content on user attention, which can lead to reduced prediction performance.

By incorporating video content features, viewport prediction can be significantly improved. For instance, C. Fan et al. [25] combine 360-degree video content features with head-mounted display (HMD) sensor-related features for user fixation prediction. Similarly, studies [13, 10, 30] leverage panoramic saliency and head orientation history for panoramic viewport prediction.

However, as shown in Table I, research on viewport prediction of point cloud video considering both video content and user trajectory is limited compared to 360 video or VR video. In this study, we aim to bridge this research gap and maximize the viewport prediction performance by jointly considering point cloud video saliency and the user’s head trajectory.

The detection of visual saliency is a prominent research topic in the field of computer vision, with the goal of quickly locating the most attractive objects or patterns in an image or video clip. This is vital to predicting the user’s visual attention and viewport [31, 13, 32, 30, 6, 33].

Visual saliency detection comes in two categories: images and videos. Deep learning techniques enhance saliency detection in still images, addressing associated challenges through various proposed methods. For instance, Q. Lai et al. [34] propose a weakly supervised visual saliency prediction method using a joint loss module and a deep neural network. M. Zhong et al. [35] develop a computational visual model based on support vector regression to predict user attention to specific categories. G. Ma et al. [36] revisit image salient object detection from a semantic perspective, providing a novel network to learn the relative semantic saliency degree for two input object proposals. In the domain of 3D point clouds, V. F. Figueiredo et al.[37] employ orthographic projections and established saliency detection algorithms to create a 3D point cloud saliency map. In contrast, X. Ding et al. [38] propose a novel approach to detect saliency in point clouds by jointly utilizing local distinctness and global rarity cues supported by psychological evidence.

Detecting video saliency is more intricate, involving spatial and temporal information, with significant research attention. W. Wang et al. [9] introduc an attentive CNN-LSTM architecture, encoding static attention into dynamic saliency representation, leveraging both static and dynamic fixation data. M. Paul et al. [11] propose a novel video summarization framework based on eye tracker data, which calculates motion saliency scores by considering the distance between the viewer’s current focus and the previous focus, facilitating video summarization. However, this approach focuses on extracting salient frames rather than regions within video frames.

Enlightened by these insights, we find it necessary to capture both static and dynamic saliency cues for point cloud video saliency detection. K. Zhang et al. [39] present a spatio-temporal dual-stream network using a cascaded architecture of deep 2D-CNN and shallow 3D-CNN for temporal saliency, excelling in feature extraction but lacking exploration into feature fusion. Z. Wu et al. [40] design three different deep fusion modules, including summation, maximization, and multiplication, to effectively utilize spatiotemporal features. However, these fusion operations cannot discern the differential impact of temporal and spatial saliency features on visual saliency detection in distinct scenarios. Some researchers propose fusing features from different layers or points to obtain more important features [41, 16, 17, 18].

Additionally, the mentioned traditional CNN networks are ineffective for processing unordered inputs like point cloud data. Treating points in 3D space as a volumetric grid is an alternative, but it’s practical only for sparse point clouds, and CNNs are computationally intensive for high-resolution point clouds.

Therefore, towards point-cloud video streaming some studies employ scene flow methods to calculate motion vectors for all points in the current frame, thereby capturing temporal information across multiple frames [42, 43, 44]. Nevertheless, given that the point count in each point cloud frame is approximately $10^{6}$ or more, such an approach demands significant computational resources.

This paper introduces an innovative network for saliency detection in point cloud videos, adept at handling the unordered nature of the data. Our approach emphasizes capturing both static and dynamic saliency cues, making it well-suited for effective point cloud video saliency detection.

Point cloud sampling is often used to reduce the computational complexity of point cloud video analysis. The current two major approaches of point cloud sampling - heuristic and learning-based - have unignorable limitations. Learning-based methods require large amounts of time and memory to train complex neural networks, making them unsuitable for large-scale point cloud videos. Heuristic sampling methods fall into five categories: farthest point sampling (FPS), inverse density importance sampling (IDIS), voxel sampling (VS), random sampling (RS), and geometric sampling (GS). GS considers the curvature of points, RS employs random probability for sampling, and IDIS samples based on inverse density. Therefore, GS, RS, and IDIS cannot guarantee the preservation of inter-frame dynamic components in every sampling outcome. FPS and VS, due to their uniform distribution in spatial sampling, can retain information in point clouds, including inter-frame dynamic regions. Then we conduct an analysis of the time complexity for FPS and VS.

Assuming the original point cloud comprises approximately $N$ points, and our objective is to sample $V$ points, the time complexity for FPS, which selects each $p^{v}$ in the sampling point set $\{p^{1},\ldots,p^{v},\ldots,p^{V}\}$ as the farthest point from the first $v-1$ points, is $O(N^{2})$. For VS, which divides the point cloud into $V$ voxels of equal volume (with time complexity $O(N)$) and computes the centroid of points within each voxel to replace all points in the voxel (with time complexity $O(V*\frac{N}{V})=O(N)$), the final time complexity is $O(N)$.

In summary, FPS exhibit higher computational complexity, making it unsuitable for point cloud videos with large amounts of data. In Section IV, we will provide a detailed introduction to the sampling process of URS and compare the performances of all sampling methods in Sec.VI.

Reducing the number of points in a point cloud video through sampling offers benefits like decreased computation and communication consumption. The URS method is proposed for effective uniformly spaced sampling, preserving temporal information, and KNN is used to aggregate spatial information from neighboring points.

Assuming the total number of each frame is $N$, each frame is cut into $M$ 3D tiles in the spatial dimension for transmission. Let $t_{j,t}$ be the $j$-th 3D tile in frame $t$, the total number of sampling points is $V$ and let $t_{j,t}$ be the $j$-th 3D tile in frame $t$ and $D_{j,t}$ be the sampled data of 3D tile $t_{j,t}$ in frame $t$.

As illustrated in Fig. 3, we establish a mapping relationship between 3D tiles, $t_{j,t}$ and $t_{j,t-1}$, as they share similar spatial locations in their respective frames. This mapping is crucial for the TC layer to analyze changes in global information at their corresponding locations. To ensure the preservation of temporal information, we evenly distribute sampling points within mapping 3D tiles to guarantee similar sampling results. For efficient sampling, we further divide each 3D tile into $N_{c}$ (ranging from 10 to $10^{2}$) smaller cubes and randomly select central points. Around these central points, we search for $N_{n}$ neighbors as sampling points using KNN to consolidate local spatial information, where $N_{n}=\frac{V}{m*N_{c}}$, and $\frac{V}{m}$ is the total number of sampling points in each 3D tile. With the introduced URS method, we can adeptly choose evenly spaced sampling points to retain temporal information. Furthermore, the URS sampling process leverages random sampling, providing an additional boost to sampling efficiency. The overall time complexity of URS is $O(N)$, so VS and URS exhibits lower time complexity compared to FPS. To further compare the fastness of VS and URS, we conduct full experiments in Sec.VI.

The implementation of TC layer requires that sampled data with similar spatial locations have a close mapping relationship. The degree of this closeness can be estimated by the proposed inter-frame mapping strength in Sec.VI, which measures the level of temporal information retention.

The sampling result $\{D_{1,t},D_{2,t},...,D_{j,t},...,D_{m,t}\}$ of frame $t$ obtained by URS is passed to the Fully Connected (FC) layer for feature extraction, to the LDC module for spacial encoding and to the RS layer for down-sampling.

Given the coordinate attributes $p_{i,t}=\{x_{i,t},y_{i,t},z_{i,t}\}$ and color attributes $a_{i,t}=\{r_{i,t},g_{i,t},b_{i,t}\}$ of each point $i$ in $D_{j,t}$, we firstly obtain the initial feature $f_{i,t}$ of point $i$ through the FC layer, the process of feature extraction is simply expressed as:

where $\oplus$ denotes concatenation. Finally, the sampled data $D_{j,t}$ goes through the FC layer to obtain the initial features $F_{j,t}$. Then we further use LDC to fully integrate local coordinate and color discrepancies. The key components of LDC are divided into three parts: Neighborhood Coding, Attention Pooling and Dilated Residual Blocks.

Neighborhood Coding. This Neighborhood Coding unit explicitly embeds the coordinate and color discrepancies between each point $i$ in $D_{j,t}$ and its all neighboring points. In this way, the corresponding point features are always aware of their relative spatial differences, facilitating the network’s ability to learn spatial salient features more effectively.

For the coordinate and color attributes $p_{i,t}^{k}$, $a_{i,t}^{k}$ of neighboring point $k$ around the point $i$, where $a_{i,t}^{k}=\{r_{i,t}^{k},g_{i,t}^{k},b_{i,t}^{k}\}$, is the set of color attributes of neighboring point $k$, we explicitly encode the relative discrepancies as follows:

where the constants $0.299$, $0.587$ and $0.114$ are used in the luminance algorithm[45] to convert RGB values to grayscale $d_{i,t}$, taking into account the sensitivity of the human eye to red, green, and blue. $\|{\cdot}\|$ denotes the coordinate relative Euclidean distance.

The discrepancy encoding $dp_{i,t}^{k}$ between point $i$ and neighboring point $k$ is correspondingly connected with the initial feature $f_{i,t}^{k}$ to obtain the enhanced point feature $\hat{f_{i,t}^{k}}$. Then a set of enhanced features of point $i$ encoded with all $K$ neighboring points $A_{i,t}=\{\hat{f_{i,t}^{1}},\hat{f_{i,t}^{2}},\hat{f_{i,t}^{3}}...\hat{f_{i,t}^{k}},..,\hat{f_{i,t}^{K}}\}$ can be obtained through neighborhood coding.

Attention Pooling. This unit is used to aggregate each feature $\hat{f_{i,t}^{k}}$ in the set of enhanced features $A_{i,t}$ as the spatial discrepancy feature $\hat{f_{i,t}}$ of point $i$. Thus each point $i$ contains the features of each neighboring point, which can compensate for the reduction of both temporal and spatial information, which is caused by the replacement of URS by RS as the downsampling technique in the following network training process.

Given each enhanced feature $\hat{f_{i,t}^{k}}$, we first use shared function to learn its attention score $S_{i,t}^{k}$. Then we use $Softmax$ function to normalize attention score $S_{i,t}^{k}$, and weight enhanced feature by the normalized score to obtain the spatial discrepancy feature $\hat{f_{i,t}}$ as follows:

where $W$ is the shared weight, $\sigma()$ is $Softmax$ function, and $\gamma()$ means Shared MLP function.

Dilated Residual Blocks. The neighborhood coding and attention pooling can spatially link each sampled point to $K$ neighbors. To expand the receptive field of each point, we perform neighborhood coding and attention pooling twice in dilated residual blocks to receive information from up to $K^{2}$ neighbors.

According to the encoding process of the LDC module above, the process of passing the initial features $F_{j,t}$ of the sampled data $D_{j,t}$ through the $c$-th LDC module and RS layer is simplified as:

where $\hat{F_{j,t}^{c-1}}$ denotes the encoded features of $D_{j,t}$ through $(c-1)$-th LDC and RS, which is passed to $c$-th LDC module as input, and the input of $1$st LDC module, $\hat{F_{j,t}^{0}}$, is initialized according to the Eq. 8. $LDC_{c}()$ represents the $c$-th LDC module, ${\theta}_{u}$ is the learnable weight, $R_{c}$ represents the $c$-th RS layer.

The encoded features are decoded by applying MLP layers, Up-Sampling (US) layers, and FC layers. Additionally, to prevent overfitting, a Dropout (DP) layer is used. The output of this decoding process is a set of spatial saliency features, denoted by $F_{S}$.

Based on the mapping relationship of $D_{j,t}$ and $D_{j,t-1}$ extracted from frame $t$ and $t-1$, we propose a TC module to capture temporal information by comparing feature difference between them, as the feature difference can reflect the degree of variation from $D_{j,t}$ to $D_{j,t-1}$.

The general structure of TC is shown in Fig. 4 and is explicitly expressed as follows:

Maxpooling. First, compute the global features of the two corresponding 3D tiles, $D_{j,t}$ and $D_{j,t-1}$, separately:

Comparing Feature Similarity. Compare the similarity of the corresponding global features, obtain the similarity score, and convert it into saliency intensity. It is defined as follows:

where $S_{sim}$ denotes the similarity score calculated between the corresponding global features $Q_{t}^{c}$ and $Q_{t-1}^{c}$. $O_{s}$ is the intensity of temporal saliency obtained by applying the temporal saliency operator, $G()$, on the similarity score $S_{sim}$. The operator is defined as $G(\cdot)=\frac{1}{1+\exp()}+1$. A lower degree of variation and a smaller $O_{s}$ are expected when the similarity score, $S_{sim}$, is higher between the global features $Q_{t}^{c}$ and $Q_{t-1}^{c}$.

Weighting Features. Weighting input feature with saliency intensity. It is defined as follows:

where the temporal change features $C_{t}^{c}$ are obtained by weighting the input features $TC_{t}^{c}$, from frame $t$, with the temporal saliency intensity. The minimum saliency intensity is 1. This means that if the saliency intensity is greater than 1, the saliency intensity will be superimposed after 5 TC layers, otherwise, it will always be 1, without changing the input features. After applying LDC and RS to it, the resulting features are passed as input to the next TC module.

With multiple TC modules, we can efficiently capture the temporal information between the mapping 3D tiles, and then we obtain the temporal salient features $F_{T}$ by decoding using MLP, US and FC layer.

The effect of the number of 3D tiles: We hypothesize that the number of 3D tiles can impact viewport prediction. In our experiments, as shown in Table II, varying the number of 3D tiles (12, 20, and 36) shows a slight increase in MIoU at both the point and tile-levels across different networks. However, the computation time for segmenting the entire point cloud video into individual 3D tiles significantly increases with the number. Therefore, for subsequent experiments, we set the number of 3D tiles to 12.

The efficiency of URS: To evaluate the efficiency of sampling, we select four levels of point cloud sizes ($\sim 10^{3}$ points, $\sim 10^{4}$ points, $\sim 10^{5}$ points, $\sim 10^{6}$ points, and $\sim 10^{7}$ points). We downsample the point clouds to 1/4 of the original number of points using FPS, VS, IDIS, GS, RS, and URS, respectively. We measure the sampling time and memory consumption of each sampling method. Each sampling experiment is independently run 50 times, and the results are averaged for the final experimental results. Additionally, we present the sampling distribution results for each sampling method separately.

The time and memory consumption of all the sampling methods are listed in Fig.6. Experimental results indicate that the sampling time and memory consumption for URS is second only to RS, due to the fact that URS needs to slice the point cloud into 3D tiles and cubes. When the number of sampling points exceeds $10^{5}$, the memory consumption of URS, VS and RS is much smaller than other sampling methods. The confidence intervals for this sampling comparison experiment at the 0.95 confidence level are shown in the Table IV.

As shown in Fig. 6c, we also compare the IFMI of the six sampling methods at the same DaCVV values. When the DaCVV of URS are 0.3 and 0.8, its IFMI values are comparable to VS. In other cases, the IFMI values of URS are slightly lower than VS but higher than or equivalent to FPS, RS, GS and IDIS across all DaCVV values. However, considering time consumption, as shown in Fig. 6a, when the number of points in the point cloud is around $10^{7}$, the time consumption of VS is 80 seconds higher than that of URS (while the number of points in a single frame ranges from $10^{6}$ to $10^{7}$, and a point cloud video sequence contains a total of 300 frames). URS achieves efficient sampling efficiency by trading off a slight loss in temporal information retention.

To observe the sampling effect visually, we take the corresponding 3D tiles $t_{k,t-1}$ and $t_{k,t}$ of consecutive frames $t-1$ and $t$ as sampling inputs and the number of sampling points is set to 12288 to get the result shown as Fig. 7. The sampling result shows that URS can indeed retain temporal information and spatial information. Although FPS and VS retain the temporal information between frames more completely, they consume more sampling time and sampling memory than URS and neglect spatial local information. RS and GS are even less preserving of temporal information than URS.

The results of saliency detection: We compare the saliency prediction performance of STVP-SD with PointNet++, BAFF, and RandLANet. The experimental results are shown in Table III, STVP-SD achieves on-par or better saliency detection performance than state-of-the-art methods. This is because STVP-SD makes full use of the temporal information, spatial coordinates and color information between point cloud video frames. Of the four performance metrics, STVP-SD is lower than the other three networks only in the metric of recall, due to the overly stringent conditions for determining the visible region.

The results of viewport prediction: Table III presents the experimental results, showing that STVP achieved high viewport prediction performance, with a point-level MIoU of 82.09%. The accuracy at the tile-level is even higher, with an MIoU of 88.10%.

To visualize the prediction results, we extract one frame from two video sequences to display the final viewport prediction results. The prediction results are presented in two forms: a point-level prediction (Fig. 8) and a tile-level prediction (Fig. 9). From the point-level prediction results, we can see that although the STVP predictions are slightly different at the edge of the viewport compared to the ground truth, their viewport ranges largely overlap. However, their viewport edges do not resemble this defect, which is perfectly solved by extending the predictions to the tile level. From the tile-level prediction results, we can see that ground truth is almost the same as the predicted result.

The result of ablation: We also compare the performance of all ablation networks, and show the results in Table III. The result indicates that substituting the URS unit with either RS or IDIS leads to a reduction of approximately 23% in the MIoU score of the prediction outcome. This shows that URS plays a fundamental role in preserving the information of the original point cloud video. Replacing or removing it has a large impact on the viewport prediction. The LDC replacement also has a large impact on prediction performance. This is because both spatial color and geometric prominence attract the user’s attention, and LDC increases color discrepancy coding compared to LFA, which can effectively capture spatially color salient regions and fully improve the spatial saliency prediction performance.

We also visualize the prediction results of the ablation experiments in Fig. 10. Since RS does not guarantee that the sampling results of adjacent video frames match, RS++ leads to disconnected frames within the viewport in the prediction results as shown in Fig. 10a and Fig. 10g. Due to the low degree of local spatial information retention in IDIS, IDIS++ also leads to the above situation, as shown in Fig. 10b and Fig. 10h. Meanwhile, since LFA ignores spatially salient information, from Fig. 10c and Fig. 10i we can see that the results of LFA++ have a salient part of the region less than the ground truth, e.g., hair in “Redandblack” and gun in “Soldier”. The temporal detection sub-module is created to capture the dynamic saliency region between frames. Therefore, as shown in Fig. 10d and Fig. 10j, the result of TSD– lacks some dynamic salient regions, e.g., hair in “Redandblack”, neck in “Soldier”, compared with the ground truth.