Explaining Hierarchical Features in Dynamic Point Cloud Processing

Similarly to [7], we use the human Mixamo [11] dataset, consisting of sequences of human bodies while performing various dynamic activities such as dancing or playing sports. The main motivations behind this selection are given in the following: compared to real 3D scenes acquired by LiDAR or mmWave sensors, such synthetic dataset does not suffer from acquisition noise or quantization distortion. This makes the dataset easy to visualize and hence to comprehend, while also isolating the problem of learning features from noisy corrections and other aspects that may arise in a more noisy dataset. Additionally, the synthetic dataset allow us to generate very different movements (from quite simply like walking to highly complicating such as breakdance steps) and study the learning of the highly descriptive features in such cases.

To better understand and explain the hierarchical features learning, different architectures have been implemented, trained and compared to the Classic hierarchical architecture depicted in Figure 2. The key idea is to isolate the multi-scale resolution, from the “deepness” of the architecture, disentangling in such a way the different aspects of the PC processing. The common aspects of all implemented solutions is the high level architecture depicted in Section II: a DE phase (with $L=3$ and downsampling factor in SG of four), a FP phase and a final prediction step. On the other side, the models differentiate in the RNN processing (stacked or parallel), as well as in the sampling modules. Specifically, we implemented the following architectures:

• •

  Shallow hierarchical architecture – Figure 3 a): as in Classic architecture, the PC is sub-sampled in a stacked fashion, leading to a very sparse PC at level $L=3$. Unlike the Classic architecture, the RNN cells are in parallel (instead of stacked), leading to a Shallow network (local features are not grouped and processed at higher levels). This network highlights the effect of multi-scaling (sampling) instead of deep hierarchical learning.

• •

  Single-scale architecture – Figure 3 b): Same architecture as in Classic hierarchical model but without the downsampling modules at each level. Hence, all the levels process the same number points and learn features at the same scale.

• •

  Without-combination architecture – Figure 3 c): this architecture recall the classic deep neural network in which local features are extracted from small neighborhoods (first RNN cell) and processed at higher (subsequent) level. Only the last level feature is then used for the final reconstruction. Note that in the Classic architecture all features learned at different level are used for predicting the final motion vectors.

The above models, as well as the Classic one, are trained using as loss function a combination of the Chamfer distance (CD) [14] and earth’s moving distance (EMD) [14] to measure the distance between predicted $\hat{P}_{t+1}$ and the target PC $P_{t+1}$, as explained in [7]. Those metrics are also used for the evaluation of PC prediction in the experimental results discussion. It is worth noting that the CD distance tends to flatten all scores toward zero values. This is because in the PC the majority of the points are perfectly predicted (all points with no motion or little motion) and most of the errors (high CD scores) are focused in the high motion area. This is the area of strongest interest in PC prediction tasks as we are interested in understanding if the neural network is able to capture such movements. Therefore, we also consider the CD Top 5% metric, which looks at the CD metric of the 5% points with the worst prediction (i.e., points with the farthest distance to their closest point).

Besides the aforementioned metrics, we also visualize the features learned at each level as motion vectors. This is a key aspect to better understand the hierarchical learning process, the goal of this paper. As shown in Figure 2, the motion vectors are obtained by combining features from multiple levels in the FP phase. The final features are then processed by a last fully connected layer and converted into motion vectors. We are interested in visualizing the actual contribution to the final motion vectors from each level. We do this by seeing the motion vectors as the combination of motion vectors produced at each level i.e., $M_{t}=\sum_{l}^{L}M_{t}^{l}$, with $M_{t}$ being the predicted motion vectors and $M_{t}^{l}$ the motion vectors produced by level $l$. Such level contribution $M_{t}^{l}$ is visualized by keeping the features from level $l$ and setting to zero the remaining ones at the input of the FP phase. We then replicate the FP and prediction phase operation with the already trained weights and obtain the motion vector generated by level $l$.

To study hierarchical learning in dynamic PC processing, we study the features learned from the Classic architecture in Figure 2. We visualize those features giving as input PC a running person shown in Figure 4. We investigate how each level of the architecture learns features for a given point of interest, such as a point in the foot (red point in the Figure 4). To do so, we first show in Figure 4 b) the multi-scale neighborhood of the selected point in the foot at each level. Due to the subsequent sampling in the network, the lower level learns features only by looking at points in a small area around the point of interest (top blue square in the figure). On the contrary, the higher level learns features by considering a sparser set of points in a large area (bottom gold square in the figure). In Figure 4 c), we visualize the motion vectors $M_{t}^{l}$ at each level of the Classic hierarchical architecture for given time frame $t$. It can be observed that the lowest level captures small and diverse motions; on the contrary, at the highest level, a single main motion vector is learned for almost all the points in the foot. Similarly to what happens in hierarchical CNNs, we observe that the highest level of the network learns global motions (for example the forward motion of the runner), while lower levels of the architecture learn local movements of the action (small refinement motion that defines the local movement of the foot – which can also go up-backward when the foot goes up). While intuitive, to the best of our knowledge, this visualization is the first one in the literature confirming that the intuition behind hierarchical features in CNNs can be extended to dynamic PCs architectures and dynamic flow: hierarchical features can be explained as local and global motions.

To carry out our deeper understanding of hierarchical learning, now we want to answer the following questions: How does the hierarchical architecture learn global and local motion? (Sections IV-B and IV-C); How does the hierarchical PC architecture differ from CNN? (Section IV-D).

To understand better how global and local motions are learned in the hierarchical architecture, we first investigate the effect of the stacked component on the learned features. To do so, we compare the Classic architecture (in which local features are aggregated and processed at a higher level) to the Shallow one, which learns features without features aggregation. Note that in both architectures there is the multi-scaling effect, with the higher lever observing a much sparser PC (hence processing a more global neighborhood). Looking at the motion vectors $M_{t}^{l}$ learned in the Shallow (Figure 4 d)) architecture, we can note that 1) motion vectors are all at the same magnitude across levels; 2) at higher levels the vectors show highly contrasting movements. This implies that in the Shallow architecture the features lose the global and location motion interpretation. Specifically, we do not have the global motion (runner moving forward) being captured by the higher levels. Similarly, lower levels do not capture only small and refining local motions. This is motivated by the fact that in the Shallow architecture all three levels contribute equally to the output motion. This proves that the features interpretation is not guaranteed by the multi-scale component in the architecture, while it is learned by the stacked aspects of local features being aggregated and processed to learn global motion.

Interestingly, we have also observed that despite the missing interpretation of global and local motion, the Shallow can capture the overall movement of the PC and perform a decent prediction. This is observed in Table I, which shows the prediction error for the different architectures presented in Section III-B. We also add the prediction error for the “Copy-Last-Input” case, in which the prediction is simply the copy of the previous input. This shows that all the methods investigated in this work perform substantially better than “Copy-Last-Input”, hence they are all able to capture some aspects of the body movements. We also visualise in Figure 5 the prediction for three particular sequences: Running, Jumping, and Dancing. We can observe that the Dancing PC is very well predicted from both Shallow and Classic architectures. In Jumping and Running sequences, there is a mild gain from the Classic one (see for example the arm in Jumping) but the Shallow architecture still learns well the overall movement. In short, while the stacked aspect has a big impact on the interpretation of the learned feature, the performance gap is not as substantial.

We are now interested in understanding how much the multi-scale effect is important for capturing the PC motion. We compare the Classic architecture to the Single-scale architecture (i.e., without multi-scale effect) and we show that the multi-scale is essential to achieve a good performance (hence prediction). This is confirmed both numerically and visually. In Table I, the Classic architecture has a better overall prediction accuracy and from Figure 5, it is clear that Single-scale architecture fails in capturing some local movements such as the foot/leg in “Running”, the knee in “Jumping” and the hands in “Dancing”. This is motivated by the following: in complex movement, as in the “Running”, in which the foot performs a forward translation and rotation motion simultaneously (Figure 4 c), the joint-motion can be captured only by looking at points from different scales, the forward movement required a large-scale neighborhood for example. Only by observing the foot neighborhoods at multiple scales both the global forward motion and local rotation motion are captured. This ability to capture complex motion patterns as local and global motions results in more accurate predictions.

We are now interested in answering the second question: How does the hierarchical PC architecture differ from CNN? Both architectures use a hierarchical approach to learn global and local features. However, in CNNs only the last layer features are usually processed to infer the final task, while in the Classic architecture all features from all the levels are combined in an FP propagation phase. We then compare the Classic architecture to the Without-combination one, which does not combine the features from different levels. From Figure 5, we can already observe that the Without-combination architecture fails in predicting key local movements: knee in “Jumping” and leg in “Dancing”. Even if the network can learn hierarchical features, it is not able to merge the local motions in the final reconstruction, in which only the higher level motion is used for the prediction. As a result, the network loses the ability to predict complex motions, resulting in an inferior performance (validated in Table I).