Adaptively Meshed Video Stabilization

Traditional video stabilization methods can be classified into three types, including 2D, 3D and 2.5D methods. 2D methods aim to stabilize videos mainly containing planar motion. Through image matching technologies, such as feature matching, 2D methods usually model the camera motion with inter-frame transformation matrix sequences [6], [7]. Under the assumption of static scenes, many smoothing techniques are implemented to smooth the obtained transformation matrices, such as Gaussian low-pass filtering [13], Particle filtering [14] and Regularization [15], then stabilized frames are generated with the smoothed transform matrices. Grundmann et al. [16] proposed a linear programming framework to calculate a global homography through minimizing the first, second and third derivatives of the resulting camera path. [17] extended the method of [16] by replacing the global homography with a homography array to reduce the rolling shutter distortion. Joshi et al. proposed a hyperlapse method [1], which optimally selects some frames from the input frames, ensures that the selected frames can best match a desired speed-up rate and also result in the smoothest possible camera motion, and stabilizes the video according to these selected frames. Meanwhile, Zhang et al. [9] introduced the “as-similar-as-possible” constraint to make the motion estimation more robust. 2D methods are computationally efficient and robust against planar camera motions, but may fail in the existence of strong parallax.

Parallax can be well handled by 3D methods. By detecting a set of feature points and tracking them at each frame, 3D methods reconstruct the 3D locations of these feature points and the 3D camera motion by the Structure-from-Motion (SFM) technique [11]. Thus these 3D methods are more complicated than 2D methods. Buehler et al. [18] got the smoothed locations of feature points by limiting the speed of the projected feature points to be constant. Liu et al. [2] developed a 3D content-preserving warping technique. In [19], video stabilization is solved with an additional depth sensor, such as a Kinect camera. Moreover, some plane constraints were introduced to improve video stabilization performance [20], [21]. Due to the implementation of SFM, 3D methods usually stabilize videos much slower than 2D methods and may be fragile in the case of planar motions.

2.5D methods combine the advantages of 2D methods and 3D methods. Liu et al. [22] extracted robust feature trajectories from the input frames and smoothed the camera path with subspace constraints. They also handled parallax by the content preserving warping [23]. As this subspace property may not hold for dynamic scenes where cameras move quickly, Goldstein et al. introduced epipolar constraints in [24] and employed time-view reprojection for foreground feature trajectories. In [25], Liu et al. designed a specific optical flow by enforcing strong spatial coherence and further extended it to meshflow [26] for sparse motion representation. However, these methods may fail when large foreground objects exist. So Ling et al. [27] took both foreground and background feature trajectories to estimate the camera jitters through solving an optimization problem. In [28], Zhao et al. implemented fast video stabilization assisted by foreground feature trajectories. Kon et al. [12] generated virtual trajectories by augmenting incomplete trajectories using a low-rank matrix completion scheme when the number of the estimated trajectories is insufficient. In [29], a novel method combines trajectory smoothing and frame warping into a single optimization framework and can conveniently increase the strength of the trajectory smoothing.

Recently convolutional neural networks(CNNs) were implemented for video stabilization. Wang et al. [10] built a novel video stabilization dataset, called DeepStab dataset which includes 61 pairs of stable and unstable videos, and designed a network for multi-grid warping transformation learning, which can achieve comparable performance as traditional methods in regular videos and is more robust for low-quality videos. In [30], adversarial networks are taken to estimate an affine matrix, with which steady frames can be generated. Huang et al. [31] designed a novel network which processes each unsteady frame progressively in a multi-scale manner from low resolution to high resolution, and then outputs an affine transformation to stabilize the frame. In [32], a framework with frame interpolation techniques is utilized to generate stabilized frames. Generally speaking, CNN-based methods have great potential to robustly stabilize various videos, but may be limited by the lack of training video datasets.

We adopt the KLT tracker of [33] to detect feature points at each frame and track them to generate feature trajectories. To avoid all feature trajectories gathering in some small regions, we first divide each video frame into 10 by 10 grids and detect 200 corners with a global threshold. For grids which have no detected corners, we will decrease their local thresholds to ensure that at least one corner can be detected in each grid. Then these corners are tracked to generate feature trajectories. Thus there will be more feature trajectories in the grids with rich contents while less corners in grids with poor contents. As [29], the corner detection procedure is executed only when the number of tracked feature points decreases to a certain level. However, in our method, much fewer corners are detected (1000 corners in [29]) since we directly produce the triangle mesh according to the positions of feature trajectories in each frame. We keep all trajectories whose length is longer than 3 frames.

Suppose there are $N$ feature trajectories in a shaky video, which are denoted as $\{P_{i}\}_{i=1}^{N}$. These feature trajectories may come from either the background or foreground objects. Denote the first and last frames, in which trajectory $i$($i\in[1,N]$) appears, as $s_{i}$ and $e_{i}$. Denote the feature point of trajectory $i$ at frame $t$ ($s_{i}\leq t\leq e_{i}$) as $P_{i,t}$. The feature points of all feature trajectories which appear at frame $t$ are grouped into a set,

Then we generate a triangle mesh of frame $t$ with $M_{t}$ through standard constrained Delaunay triangulation[34], which is illustrated in Fig. 2(b). Suppose there are $K_{t}$ triangles in the triangle mesh of frame $t$, then the set of triangles of frame $t$ is denoted as $Q_{t}=\{Q_{t,1},Q_{t,2},...,Q_{t,K_{t}}\}$. The stabilized views of $M_{t}$ and $Q_{t}$ are represented as $\widetilde{M}_{t}$ and $\widetilde{Q}_{t}$, respectively.

Our method estimates $\{\widetilde{P}_{i}\}_{i=1}^{N}$, i.e., the stabilized views of feature trajectories $\{P_{i}\}_{i=1}^{N}$, through solving an optimization problem under the following three constraints.

1. 1.

   For a given feature trajectory $P_{i}$, its stabilized view $\widetilde{P}_{i}$ should vary slowly.

2. 2.

   At frame $t$, each stabilized triangle $\widetilde{Q}_{t}$ is geometrically similar to its original triangle $Q_{t}$.

3. 3.

   At frame $t$, the transformations between $\widetilde{Q}_{t}$ and its adjacent triangles should be similar to the corresponding transformations between $Q_{t}$ and its adjacent triangles.

As shown in Fig. 2(b), all triangles are constructed with feature trajectories and we can calculate their stabilized views and warp these regions according to the corresponding transformations. At each frame, our method also evenly sets $10$(other number should also be fine) control points on each of four frame edges and produces $36$ control points which are denoted as $E_{t}=\{E_{t,1},...,E_{t,36}\}$. Then we divide frame $t$ with the new set $\{M_{t},E_{t}\}$ through Delaunay triangulation to produce a triangle mesh. The obtained triangle mesh is represented as $\{Q_{t,1},...,Q_{t,K_{t}},B_{t,1},...,B_{t,L_{t}}\}$, where $Q_{t}=\{Q_{t,1},...,Q_{t,K_{t}}\}$, called inner triangles, are $K_{t}$ triangles whose vertices are made up of only $M_{t}$, and $B_{t}=\{B_{t,1},...,B_{t,L_{t}}\}$, called outer triangles, are $L_{t}$ triangles whose vertices contain at least one control point. In Fig. 2(c), inner triangles are presented in red and outer triangles in green. Then a two-stage optimization problem is solved to calculate $\widetilde{M_{t}}$ and $\widetilde{E}_{t}$, i.e., the stabilized views of $M_{t}$ and $E_{t}$. At the first stage, a global optimization problem is solved to estimate the stabilized views of feature trajectories $\{P_{i}\}_{i=1}^{N}$, i.e., $\{\widetilde{P}_{i}\}_{i=1}^{N}$, which form $\widetilde{M}_{t}$ as (1). At the second stage, a frame-by-frame optimization problem is solved to produce $\widetilde{E}_{t}$. Finally, each frame is warped through the transformations between the triangles generated by $\{\widetilde{M}_{t},\widetilde{E}_{t}\}$ and the ones by $\{M_{t},E_{t}\}$.

Based on the three constraints in Section III-A, the stabilized view of $\{P_{i}\}_{i=1}^{N}$ is obtained through solving the following optimization problem,

where

$\mathcal{O}_{Smooth}$ is the smoothing term to smooth feature trajectories, $\mathcal{O}_{InterSim}$ is the inter-frame similarity transformation term to ensure that the warped frame closely follows the similarity transformation of its original frame, $\mathcal{O}_{IntraSim}$ is the intra-frame similarity transformation term to ensure that the similarity transformation between adjacent regions of a stabilized frame is close to the corresponding transformation of its original frame, and $\mathcal{O}_{Reg}$ is the regularization term to avoid too much warping. The details of these four terms are provided below.

Given the stabilized views of feature trajectories, $\{\widetilde{P}_{i}\}_{i=1}^{N}$, we have the stabilized views of $M_{t}$ and $Q_{t}$ at each frame, i.e., $\widetilde{M}_{t}$ and $\widetilde{Q}_{t}$. At each frame, we estimate the stabilized view of control points $E_{t}$, i.e., $\widetilde{E}_{t}$, by solving an optimization problem. Then the stabilized view of $B_{t}$, $\widetilde{B}_{t}=\{\widetilde{B}_{t,1},\widetilde{B}_{t,2},...,\widetilde{B}_{t,L_{t}}\}$, can be computed from $\widetilde{M}_{t}$ and $\widetilde{E}_{t}$. Finally, each frame is warped according to the transformations between $\{Q_{t},B_{t}\}$ and $\{\widetilde{Q}_{t},\widetilde{B}_{t}\}$. There are three types of triangles in $B_{t}$, triangles with $1$, $2$ or $3$ control points, which are shown in Fig. 5. We design an optimization problem to calculate only the stabilized views of control points in $B_{t}$ as follows,

where

and the definitions of $\mathcal{O}_{InterSim}$ and $\mathcal{O}_{IntraSim}$ are given below.

To enhance the robustness of our method, we further introduce two adaptive weighting mechanisms to adjust the weighting parameters of the optimization in (III-B). They aim to improve the spatial and temporal adaptability of our method.

To verify the performance of our method, we collected 36 typical videos from public datasets¹¹1 http://liushuaicheng.org/SIGGRAPH2013/database.html,
http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/. . These videos can be classified into three categories, including “ Regular”, “Large foreground” and “Parallax”, which are shown in Fig. 9. “Regular” category contains several regular videos recording some common scenes. “Large foreground” category contains several videos where large foreground objects occupy a large part of video frames and cause serious discontinuous depth variation. “Parallax” means there is strong parallax in the captured scenes of the videos. The last two categories are more challenging and their experiments confirm that our method can achieve much better performance than previous works.

For quantitative comparison, we consider two evaluation metrics: Stability score and SSIM.

Stability score: This metric is taken from [35]. As shown in Fig. 10, the red dotted curve represents a feature trajectory, and the blue line directly connects the start point and the end point of the feature trajectory. Then the stability score is defined as the length of the blue line divided by that of the red dotted curve. The range of stability score is (0,1], and a larger value means a more stable result. Following [29], feature trajectories are extracted by the KLT tracker and are divided into segments of the length of 40 frames. We ignore segments whose lengths are less than 40 frames. The final stability score of a stabilized video is defined as the average stability score of all segments.

SSIM: SSIM (structural similarity) [36] is a widely-used index to measure the video stabilization performance. This index combines luminance, contrast and structure comparison to calculate the similarity of two given images. We calculate the SSIM of every two adjacent frames in a video and average them to produce the final SSIM value of a video.

Here we first perform ablation study to evaluate the effectiveness of terms of our proposed optimization framework in (2). To analyze the importance of each term in (III-B), we remove one term of (III-B) at a time and then evaluate the stabilization performance. The quantitative and qualitative results are shown in Fig. 11 and Table I.

Among all terms in (III-B), we do not check the influence of the regularization term $\mathcal{O}_{Reg}$ as it is a basic term to make sure that the stabilized trajectories stay close to their original ones and are free of drift. For the other terms, we remove one term from the performance index by setting its weight to zero, i.e., one of $\alpha$, $\beta$, $\gamma$ and $\varepsilon$ is set to $0$. Table I shows the quantitative results of the ablation study where both the stability score and SSIM are calculated. Fig. 11 shows the visualization results of the ablation study. In Table I, we find that without $\mathcal{O}_{Smooth1}$, the stability performance is seriously degraded, which confirms that the first-order derivative term is the main one to guarantee the low-frequency characteristics of the stabilized videos. The stability score and SSIM also degrade when the second-order derivative term is removed from the optimization function. According to the analysis in Section III-D1, removing the second-order derivative term means that feature trajectories are smoothed only with the first-order derivative term and the camera motion is enforced to be static. Thus fast camera movement may yield artifacts and even collapse, which is demonstrated in Fig. 11. $\mathcal{O}_{InterSim}$ also significantly influences the performance. As shown in Fig. 11, due to the lack of the local shape preserving constraint, serious distortion may happen when parallax or large foreground objects exist, which also cause the non-smoothing of stabilized videos. Table I also shows that when $\mathcal{O}_{IntraSim}$ is removed, the stabilization performance becomes worse, which illustrates the importance of this term in keeping the similar transformation in the adjacent regions. $\mathcal{O}_{IntraSim}$ can also avoid the trajectory mis-matching and improve the stabilization results.

We compare our method with several state-of-the-art methods, including Subspace [22], Meshflow [26], Deep learning [10], Effective [29] and Global [37]. Subspace is implemented in Adobe Premiere stabilizer. The codes of Meshflow [26], Deep learning [10] and Effective [29] were found at https://github.com/sudheerachary/Mesh-Flow-Video-Stabilization, https://github.com/cxjyxxme/deep-online-video-stabilization-deploy and https://github.com/705062791/TVCG-Video-Stabilization-via-joint-Trajectory-Smoothing-and-frame-warping. Thanks to the authors of[37], who provided us with the binary implementation of Global.

Here we first compare our method with these methods through some qualitative results, which are shown in Fig. 12 and Fig. 13. Fig. 12 shows feature trajectories of an example shaky video and the corresponding stabilized videos. The Deep learning method stabilizes videos without using future frames. Therefore, compared with other methods, its stabilization effect is the worst. Although the Effective method reduces most of high-frequency shakiness, we can still catch many visible low-frequency footstep motions. Therefore, its trajectories are not good enough. The results of the Subspace, Meshflow and Global methods eliminate almost all high-frequency shakiness, and generate similar stabilization effects. The results of the proposed method show significant improvement, which is confirmed by the smoothness of the stabilized trajectories.

In addition to stability, noticeable visual distortion should also be avoided in stabilized videos. Fig. 13 shows some example frames in several stabilized videos by different methods. The Subspace method performs motion compensation through smoothing long feature trajectories with low-rank constraints. As shown by Fig. 13(b), when there are not enough long feature trajectories, for example, in the case of fast moving foreground objects, Subspace may fail and result in serious distortion. The Meshflow and Global methods discard foreground feature trajectories and estimate the camera motion with only the background feature trajectories. However, separating foreground trajectories from background ones is a difficult task, especially in large foreground scenes or strong parallax. So these methods may suffer performance degradation in challenging videos, as shown by Fig. 13(b) and 13(c). The Deep learning method stabilizes videos without future frames and regresses one homography for each of fixed grids. The inaccurate estimation and weak constraint for local shape preserving may incur noticeable distortion, which is illustrated in Fig. 13(e). The Effective method estimates the camera motion based on all feature trajectories. However, this method divides frames into fixed mesh grids of the same size and extract a similar number of feature trajectories in each mesh grid, which is not reasonable for complicated scenes, such as large foreground and parallax. Fig. 13(a) and 13(d) show that this method may cause distortion in regions with rich contents or large depth variation.

The quantitative comparison on all test videos is also performed. Table II and Table III show Stability score and SSIM generated by different methods on all test videos. In terms of Stability score, the proposed method outperforms all other methods on most videos in the “Regular” category. Only in two videos, the Subspace method gets the highest Stability score, which is slightly better than that of the proposed method. However, the Subspace method gets worse results than some other methods in other videos. More significant performance improvement of the proposed method can be observed in the other two complicated categories, “Large foreground” and “Parallax”. Our method gets much higher Stability scores on most videos. Note that the compared methods, except the Subspace method, take fixed meshes in stabilizing videos. For the scenes containing large foreground objects or parallax, these methods may suffer serious performance degradation while our method can get much better results with the adaptively adjusted mesh. Moreover, the proposed method takes both background feature trajectories and foreground feature trajectories to smooth the camera motion and may be hardly disturbed by the active motion of the foreground objects. Thus the proposed method achieves higher Stability scores in most videos than the methods which discard foreground feature trajectories. So our method is the most robust among all methods. Similar SSIM comparison results can also be observed in Table III. The proposed method gets much better stability score and SSIM in most videos than other methods. Since stability score measures the stability of local feature trajectories and SSIM measures the global similarity of adjacent frames in the stabilized videos, the results confirm the effectiveness of our method. For more thorough comparison, a supplemental demo is provided at http://home.ustc.edu.cn/~zmd1992/content-aware-stabilization.html.