Gaussian Splatting Lucas-Kanade

The task of reconstructing dynamic 3D scenes from monocular video presents a significant challenge, particularly when constrained by limited camera movement. This limitation results in a lack of parallax and epipolar constraints, which are crucial for accurately estimating scene structures Gao et al. (2022); Schönberger & Frahm (2016). Implicit neural representation methods like NeRF have shown promise in mitigating this issue by modeling dynamic scenes using an implicit radiance field Mildenhall et al. (2021a); Chen et al. (2023); Li et al. (2021d). Explicit approaches like Gaussian splatting Kerbl et al. (2023), while more efficient and often improve rendering quality, face additional hurdles. Gaussian splatting relies on an explicit representation of the scene using volumetric Gaussian functions. This necessitates a high degree of accuracy in geometric understanding to ensure proper scene depiction. Consequently, despite advancements in dynamic Gaussian splatting techniques, the requirement for diverse viewing angles to effectively constrain the placement of these Gaussians remains a limiting factor. This inherent limitation restricts their effectiveness for capturing scenes from static viewpoints or dealing with objects exhibiting rapid and complex movements.

Recent Gaussian splatting frameworks for modeling dynamic scenes commonly utilize a canonical Gaussian space as their foundation. This space is typically initialized using Structure from Motion (SfM) Micheletti et al. (2015) point clouds, with which each Gaussian is assigned attributes that describe its position, orientation, and lighting-dependent color. This canonical space acts as a reference point from which deformations are applied to represent the dynamic scene at different time steps. The process of deforming the canonical space to depict the scene at a particular time step is achieved through a warp field, often referred to as a forward warp field due to its function of warping the canonical representation forward to a specific point in time. This warp field is typically modeled using learnable Multi-Layer Perceptrons (MLPs) or offset values, as seen in frameworks like DeformableGS Yang et al. (2023) and Dynamic-GS Luiten et al. (2024). These approaches have laid the groundwork for dynamic Gaussian scene modeling.

Structural cues such as optical flow and monocular depth have been explored to regulate the temporal transitions of Gaussians. However, imposing structural constraints on a generic forward warp field, as seen in DeformableGS Yang et al. (2023), is challenging. Although many studies leverage depth and flow priors, they often fail to regularize motion through the warp field, resulting in sub-optimal trajectories and insufficient motion regularization. Additionally, employing time integration on an auxiliary network to predict the velocity field can introduce computational overhead and lack theoretical constraints. In contrast to this data-driven approach, we demonstrate that it is possible to derive velocity fields directly from a generic warp field by adapting the Lucas-Kanade method with a small motion constraint for Gaussian Splatting. This allows for effective time integration for structural comparisons. By regulating Gaussian motions through analytical velocity fields, we address the limitations of data-driven structural regularizations, enabling continuous learning of the warp field in a time domain without the need for additional learnable parameters.

In real-world captures, our approach outperforms state-of-the-art methods, including other dynamic Gaussian techniques. We demonstrate that our warp field regularization accommodates deformations in complex scenes with minimal camera movement, achieving results that are competitive with NeRF frameworks and enabling Gaussian splatting to model highly dynamic scenes from static cameras. To summarize, our approach offers two key advantages:
Reduced Bias: Methods relying solely on data-driven learning of warp fields can be biased towards the visible time steps, neglecting the overall movement of Gaussians. Our method, applicable to forward warp field techniques, ensures accurate Gaussian trajectories throughout the sequence.
Improved Tractability: The derived analytical solution for warp field velocities provides greater tractability compared to directly supervising the flow field as the difference in estimated deformations between consecutive frames.

Neural Radiance Fields (NeRFs). NeRFs Mildenhall et al. (2021b) have demonstrated exceptional capabilities in synthesizing novel views of static scenes. Their approach uses a fully connected deep network to represent a scene’s radiance and density. Since its introduction, NeRF has been extended to dynamic scenarios by several works Gafni et al. (2021); Park et al. (2021a; b); Pumarola et al. (2021); Tretschk et al. (2021b); Yu et al. (2023). Some of these methods leverage rigid body motion fields Park et al. (2021a; b) while others incorporate translational deformation fields with temporal positional encoding Pumarola et al. (2021); Tretschk et al. (2021b). Despite the advantages of NeRF and its dynamic extensions, they often impose high computational demands for both training and rendering.

Gaussian Splatting. The computational complexity of Neural Radiance Fields (NeRF) has driven the development of alternative 3D scene representation methods. 3D Gaussian Splatting (3DGS) Kerbl et al. (2023) has emerged as a promising solution, leveraging 3D Gaussians to model scenes and offering significant advantages over NeRF in terms of rendering speed and training efficiency.

While initially focused on static scenes, recent works Yang et al. (2023); Wu et al. (2024) have extended 3DGS to the dynamic domain. These approaches introduce a forward warp field that maps canonical Gaussians to their corresponding spacetime locations, enabling the representation of dynamic scene content. DeformableGS Yang et al. (2023) employs an MLP to learn positional, rotational, and scaling offsets for each Gaussian, creating an over-parameterized warp field that captures complex spacetime relationships. 4DGaussians Wu et al. (2024) further refine this approach by leveraging hexplane encoding Fang et al. (2022); Cao & Johnson (2023) to connect adjacent Gaussians. DynamicGS Luiten et al. (2024) incrementally deforms Gaussians along the tracked time frame. Despite these advancements, existing dynamic 3DGS methods struggle with highly dynamic scenes and near-static camera viewpoints. To overcome these limitations, we propose a novel approach that grounds the warp field directly on approximated scene flow.

Flow supervision. Optical flow supervision has been widely adopted for novel view synthesis and 3D reconstruction. Dynamic NeRFs Li et al. (2021d; 2023; a); Tretschk et al. (2021a); Wang et al. (2023); Chen et al. (2023) have explored implicit scene flow representation with an Invertible Neural Network, semi-explicit representation from time integrating a learned velocity field, and explicit analytical derivation from the deformable warp field. Flow-sup NeRF Wang et al. (2023) pioneered the use of analytical flow supervision in NeRF, laying the foundation for more precise and physically grounded modeling of dynamic scenes.

Several Gaussian Splatting works have begun exploring flow supervision. Motion-aware GS Guo et al. (2024) applies flow supervision on the cross-dimensionally matched Gaussians from adjacent frames, without explicitly accounting for the flow contributions of Gaussians to each queried pixel. Gao et al.Gao et al. (2024) on avatar rendering apply flow supervision for adjacent frames after rendering the optical flow map with $\alpha$-blending Wallace (1981). However, while these approaches introduce data-driven flow supervision, they do not effectively regularize the motion through the warp field, leading to suboptimal trajectories and insufficient motion regularization. This limitation makes them suitable only for enforcing short-term geometric consistency, while their performance deteriorates for out-of-distribution time steps Aoki et al. (2019); Li et al. (2021b). In the context of dynamic Gaussian Splatting, we propose to regularize the deformable warp field through analytical time integration to ensure consistent geometric relationships across time steps.

Dynamic Gaussian Splatting. Dynamic 3D Gaussian Splatting frameworks follow a similar optimization pipeline as static Gaussian Splatting. Each Gaussian is defined by parameters for its mean $\bm{\mu}$, covariance $\Sigma$, and spherical harmonics $SH$ color coefficients. To project Gaussians from 3D to 2D, we calculate the view space covariance matrix as follows:

$$\Sigma^{\prime}=JV\Sigma V^{T}J^{T},$$

(1)

where $J$ is the Jacobian of the projective transformation and $V$ is the viewing transform. To represent a scene’s radiance, the covariance $\Sigma$ is decomposed into scaling matrices $S$ and rotation matrices $R$ as: $\Sigma=RSS^{T}R^{T}$. This decomposition supports differential optimization for dynamic Gaussian Splatting. The color $C$ of a pixel $\bm{u}$ on the image plane is derived using a point-based volume rendering equation by taking samples of density $\sigma$, transmittance $T$, and color $c$ along the ray with interval $\delta_{i}$ as:

$$C(\bm{u})=\sum_{i=1}^{N}T_{i}\left(1-\exp(-\sigma_{i}\delta_{i})\right)c_{i}\quad\text{with}\quad T_{i}=\exp\left(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\right).$$

(2)

Adaptive density control manages the density of 3D Gaussians in the optimization, allowing dynamic variation over iterations. For more details, refer to Kerbl et al. (2023).

Dynamic Gaussian Splatting frameworks utilize a canonical space and learn warp fields, denoted as $\mathcal{F}$, which transform canonical Gaussians $\mathcal{G}$ into their deformed states at time $t$. The offset in Gaussian parameters due to this transformation is expressed as:

$$\delta\mathcal{G}=\mathcal{F}_{\theta}(\mathcal{G},t).$$

(3)

Typically, these deformed geometries are mapped through a neural network Yang et al. (2023); Luiten et al. (2024); Wu et al. (2024), with optional color mapping applied as part of the transformation process. Following deformation, a novel-view image $\hat{I}$ is rendered using differential rasterization with a view matrix $V$ and target time $t$. While MLP-based deformation mapping often suffers from overfitting to training views, leading to degraded novel-view reconstruction, we address this by enforcing analytical constraints on the warp field. This ensures Gaussian motions adhere to the expected scene flow field, mitigating the issues of overfitting and SfM-initialized point cloud limitations.

Warp field expression with twist increments. Warp fields, previously defined for point cloud registration tasks Aoki et al. (2019), can be used to model rigid transformations between sets of Gaussian means. Let $\bm{\mu}_{\tau}\in\mathbb{R}^{N_{1}\times 3}$ and $\bm{\mu}_{s}\in\mathbb{R}^{N_{2}\times 3}$ represent the target and source Gaussian means, respectively, where $N_{1}$ and $N_{2}$ denote the number of Gaussians in each set. The warp field, which defines the rigid transformation aligning the source Gaussian set to the target set in the special Euclidean group $SE(3)$, can be expressed as $\mathcal{W}(\bm{\xi})=\exp\left(\sum_{\mu=1}^{6}\bm{\xi}_{\mu}P_{\mu}\right)$.

Here, $\bm{\xi}\in\mathbb{R}^{6}$ are the twist parameters, and $P$ are the generator matrices of the group. The warp field is optimized by minimizing the difference between the transformed source Gaussian means and the target means:

$$\underset{\bm{\xi}}{\text{argmin}}\left\|\phi(\mathcal{W}(\bm{\xi})\cdot\bm{\mu}_{s})-\phi(\bm{\mu}_{\tau})\right\|^{2}_{2},$$

(4)

where $\phi:\mathbb{R}^{N\times 3}\rightarrow\mathbb{R}^{K}$ is an encoding function that either explicitly extracts geometric features such as edges and normals or implicitly encodes the Gaussian means into feature vectors. The notation $(\cdot)$ represents the application of the warp field transformation.

This optimization can be solved iteratively given the twist parameter Jacobian matrix $J$, denoting the changes in the warp field concerning twist parameters, that can be further decomposed into a product of warp field gradient and the encoding functional gradient as:

$$J=\frac{(\mathcal{W}^{-1}(\bm{\xi})\cdot\bm{\mu}_{\tau})}{\partial\bm{\xi}^{T}}\cdot\frac{\partial\phi(\mathcal{W}^{-1}(\bm{\xi})\cdot\bm{\mu}_{\tau})}{\partial(\mathcal{W}^{-1}(\bm{\xi})\cdot\bm{\mu}_{\tau})^{T}}.$$

(5)

The optimization process iteratively learns the twist increment $\Delta\bm{\xi}$ that best aligns the encoded source Gaussian means with the target Gaussian means.

We investigate the problem of fitting deformable Gaussian Splatting from a monocular video sequence. To initialize the process, we employ structure-from-motion methods to set the canonical Gaussians and camera parameters. For a given set of 3D Gaussians $\mathcal{G}$ at time $t$, the warp field deforms these Gaussians from their canonical positions to their dynamic locations at time $t$. Our goal is to find the optimal network parameters $\theta$ for the forward warp field $\mathcal{W}_{c\rightarrow}(\mathcal{G},t)$. This optimization aims to ensure that both the derived scene flow $\hat{\mathcal{T}}_{t\rightarrow t+1}$ and the rasterized image $\hat{\mathcal{I}_{t}}$ closely match the expected flow field ${\mathcal{T}}_{t\rightarrow t+1}$ and the reference image $\mathcal{I}_{t}$:

$$\mathscr{L}=\mathscr{L}_{color}+\mathscr{L}_{motion}=|\hat{\mathcal{I}_{t}}-\mathcal{I}_{t}|+\left\|\hat{\mathcal{T}}_{t\rightarrow t+1}-\mathcal{T}_{t\rightarrow t+1}\right\|.$$

(6)

The scene flow component can be further divided into optical flow and depth estimations, corresponding to in-plane and out-of-plane motions. Consider a Gaussian mean $\bm{\mu}=\bm{\mu}(t)$ moving in the scene. Its instantaneous scene flow is $\frac{d\bm{\mu}}{dt}$. If the corresponding pixel location on the image plane is $\bm{u}_{i}=\bm{u}_{i}(t)$, the optical flow can be expressed as the projection of the scene flow to the image plane, with $\frac{\partial\bm{\mu}}{\partial\bm{u}_{i}}$ as the instantaneous camera projective relationship:

$$\frac{d\bm{u}_{i}}{dt}=\frac{d\bm{u}_{i}}{d\bm{\mu}}\frac{d\bm{\mu}}{dt}.$$

(7)

Deriving an approximation to the scene flow needs the expression reversed. We can represent the relationship between $\bm{\mu}$ and $\bm{u}_{i}$ with dependencies on time as $\bm{\mu}=\bm{\mu}(\bm{u}_{i}(t);t)$ Vedula et al. (2005). By differentiating this mapping with respect to time, we obtain:

$$\frac{d\bm{\mu}}{dt}=\frac{\partial\bm{\mu}}{\partial\bm{u}_{i}}\frac{d\bm{u}_{i}}{dt}+\frac{d\bm{\mu}}{dt}\Big{|}_{\bm{u}_{i}}.$$

(8)

This equation describes the motion of a point in 3D space while decomposing the motion of a Gaussian in the world into two components. The in-plane component $\frac{d\bm{u}_{i}}{dt}$ represents the instantaneous optical flow, while the out-of-plane term reflects the motion of $\bm{\mu}$ along the ray corresponding to $\bm{u}_{i}$. This decoupling allows us to regularize the optical flow field and the depth field separately from the estimated scene flow.

The scene flow regularization can thus be re-expressed as a combination of optical flow loss applied over the interval of $(t\rightarrow\Delta t)$ and depth loss of Gaussians at $(t+\Delta t)$, where the next step Gaussians are analytically derived:

$$\mathscr{L}_{motion}(t)=\left\|\hat{\mathcal{T}}_{t\rightarrow t+1}-\mathcal{T}_{t\rightarrow t+1}\right\|=\alpha\mathscr{L}_{flow}+\beta\mathscr{L}_{depth}.$$

(9)

Decoupling the motions alleviates the constraints on the supervision dataset and adds flexibility to the optimization process. A visualization of the regularization pipeline is provided in Fig. 2. Further implementation details of the losses are available in Appendix A.1.

Sensitivity to SfM Initialization: Our approach is sensitive to the quality of the Structure-from-Motion (SfM) Gaussian space initialization. Accurate initialization is crucial for 3D Gaussian Splatting, particularly without other scene priors. As shown in Fig. 5.a, reconstruction quality deteriorates when dynamic regions have insufficient initialized points.

Camera Parameter Sensitivity: Our method is sensitive to inaccuracies in bundle-adjusted camera intrinsic and extrinsic parameters. Errors in these parameters lead to inaccurate rasterization projections, causing noisy geometry or failed reconstruction, as shown in Fig. 5.b.

Computational Complexity: The pseudo-inverse operation for velocity field computation requires $(N\times d)$ square matrix decompositions, where $N$ is the number of Gaussians, and $d$ is the number of Gaussian parameters. Potential mitigations include optimized implementations or pre-filtering to remove static Gaussians.

This paper presents a method for incorporating scene flow regularization into deformable Gaussian Splatting. We derive an analytical scene flow representation, drawing upon the theoretical foundations of rigid transformation warp fields in point cloud registration. Our approach significantly enhances the structural fidelity of the underlying dynamic Gaussian geometry, enabling reconstructions of scenes with rapid motions.

Comparison with other Deformable Gaussian Splatting variants demonstrates that regularizing on the warp-field derived scene flow produces more accurate dynamic object geometries and improved motion separation due to its continuous-time regularization. Additionally, comparison with flow-supervised NeRFs reveals the advantages of using Gaussians’ explicit scene representations.

While our method exhibits limitations as discussed in the previous section, the accurate geometries learned from dynamic scenes open up promising possibilities for future applications, including 3D tracking from casual hand-held device captures and in-the-wild dynamic scene rendering.

This work was supported by Fujitsu.