Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation

The necessities of motion interpolation techniques have existed since the early days of computer animation. A key advantage of computer animation over traditionally drawn animation is its ability to automatically evaluate a smooth motion from a keyframe representation. The widely accepted method for applying interpolation to motion keyframes is through the use of function curves (F-Curves), by which various mathematical functions can be defined between intervals. The common functions used today for keyframed motion representation are often based on Bézier spline curves [37, 18], though any function can technically be employed for this purpose. In addition, inverse kinematics-based constraints [34] have been used jointly with Markov chain methods [23] and decision graphs [21] to generate constrained keyframe or motion sequence transitions in interactive applications such as 3D video games.

More recently, deep learning methods have enabled effective motion completion from sparse keyframe representations. By formulating motion interpolation as a motion synthesis problem with keyframe constraints, a neural network-assisted keyframed animation approach is emerging as a more effective solution to the current approaches. Recurrent neural networks have been able to derive realistic motion details from motion keyframe compositions [48, 22, 15] and real-time control schemes [40]. In addition, sequence masking approaches such as BERT-based and autoencoder-based methods [19, 10, 26, 31, 29, 7] have enabled full keyframe sequence analysis for completing motions with a more comprehensive context. However, these methods are severely affected by the tendency of transformer-based networks that toward a trivial local minimum, given an initial LERP starting point. This limitation in transformers has been thoroughly observed and documented as the result of early gradient instabilities in attention weights [42, 45, 24, 2].

Learned pose manifolds have become a prominent approach to synthesis plausible human poses, which restrain pose attributes to a specified space [41, 32]. These methods mainly focus on dense poses from complete motion sequences. Our work extends this concept for an incomplete and sparse scenario, by using the keyframes as constraints to derive an implicit motion manifold.

Inter-frame video interpolation is a similar task to motion interpolation, due to its common goal of predicting transitions between frames on the temporal axis. Like motion in-painting methods, temporal transformers for video interpolation use blended inputs as masks for continuous attributes [28]. Alternatively, when interpolating with an individual interval, the mask tokens can be copied from the previous keyframe, with positional encoding being the sole difference between input tokens [27]. While the feature extraction process of video data can take full advantage of the global context afforded by using transformers, other mechanisms such as convolutions can be integrated with a transformer for video in-painting as well [36]. For skeleton-based motion data, existing studies suggest that graph convolutional networks can improve analytic and synthetic performance [30, 26, 13, 8]; however, the application of pure convolutional approaches for data synthesis is only valid when the interval length between keyframes is constant.

Various mask-based machine learning techniques have been proposed to estimate missing values of incomplete sequences based on their known values. The use of masked token is well known to be highly effective for producing pre-trained linguistic transformer models [42, 9, 25, 38]. However, due to aforementioned limitations on token representations for continuous attributes, adapting transformer-based masked data modelling for computer vision tasks has largely focused on masking schemes. Discretised tokens for visual data is a proposed workaround [44, 3]; however, the information loss of tokenisation renders this technique unfeasible for precise mediums like motion data. Masked auto-encoders assign mask tokens to an encoded latent space [16, 44], and adopt them to masked sequences at the decoder level. This allows the transformer encoder to learn solely from known tokens; however, the monolithic definition of masked tokens results in a discontinuous sequence representation in latent space.

Additionally, masked convolutional networks have been a consistently effective approach for image-based data. The popular U-Net structure of convolutional models [33] involves a data bottleneck, which elicits behaviours for ignoring masked image regions [17, 39, 46]. However, the pooling technique of convolutional models restrains the learned features to a fixed scale, and struggles to extrapolate global features with increased data scales, e.g. higher image resolutions or longer videos. Conversely, transformer-based methods are particularly effective with long-range feature learning, and as such, we believe them to be the more suitable for keyframe-guided sequential motion learning.

Our pose representations are comprised of seven elements per joint: $P_{t}\in\mathbb{R}^{J\times 3}$ values for global 3D positions produced through forward kinematics (FK), and $q_{t}\in\mathbb{R}^{J\times 4}$ values for unit quaternion representations of local rotations, that is, $x_{t}=[P_{t},q_{t}]\in\mathbb{R}^{J\times 7}$, where $J$ represents the number of joints. In motion capture, local 3D positions for each joint are generally constant, determined by its positional offsets from its parent joint, and do not require explicit representation for the input. The sole exception to this is the root joint, where the global position is the local position.

In Stage-I, we encode the keyframe sequence $K$ into a learned keyframe context token representation $\Phi^{\text{key}}(K)$. It is used as a feature map for both the intermediate token generation and motion synthesis stages. First, we project the pose data $x_{t}$, $t\in T_{K}$, of each keyframe into a pose embedding vector $x^{\prime}_{t}\in\mathbb{R}^{d}$ using a linear layer, where $d$ is the token embedding dimension. Next, we adopt sinusoidal positional encoding (PE) $PE_{pos}$ [42] of $n$-dimension to $x^{\prime}_{t}$. Unlike natural language processing (NLP) approaches, we do not add PE to our token representations, but instead concatenate a fixed-length PE for two key reasons:

• •

  PE acts as a sliding binary vector, and thus can represent $2^{n}$ positions using $n$ elements. For our task, we set $n$ to 16, allowing our PE vector to produce $2^{16}$ positions, which is sufficient for our purposes.

• •

  Additive PE introduces minor disruptions in token representations. While the discrete data of NLP can benefit from slight variations in token representations, it acts as a hindrance in continuous data due to the necessity of precision.

Our Stage-I transformer $\Phi^{\text{key}}$ includes a number of layers based on multi-head self-attentions, each followed by GELU-activated feed-forward networks (FFNs). We replace all instances of LayerNorm the transformer encoders with RMSNorm [47], which does not involve a re-centering step. This avoids the token-level re-centering present in LayerNorm, which we observed to be detrimental for our regression tasks. We believe that this is due to feature biases that are present in and crucial to continuous attributes, e.g. a pose with a high root position values will result in all joints having similarly high position values. Therefore, we introduce a sequence level re-centering scheme for normalisation at the transformer input-level only, re-centreing the root positions of the input based on the mean root positions of all keyframes in the input sequence.

In Stage-II, our intermediate token generation transformer $\Phi^{\text{imd}}$ aims to learn motion manifolds navigable by temporal indices $t\in\mathbb{N}$ by the guidance of the keyframing context from $\Phi^{\text{key}}(K)$. In detail, we accomplish this through a keyframe context-guided attention mechanism, where the attention derives key and value mappings exclusively from linear transformations of $\Phi^{\text{key}}(K)$. For each intermediate token, the query is simply a sinusoidal embedding of the token’s temporal position.

We constrain our manifold implicitly using two mechanisms. Firstly, the intermediate tokens are entirely sourced as a product of $\Phi^{\text{key}}(K)$ value transformations, which inherently limits the range of latent representations in the manifold. Secondly, the 1D convolutional layers in Stage-III entail feature dependencies between temporally adjacent tokens, which enables the disparately obtained $\Phi^{\text{key}}$ and $\Phi^{\text{imd}}$ tokens to converge towards coordination.

In Stage-III, a transformer $\Phi^{\text{syn}}$ is introduced to take the intermediate token representations $m_{t}$ and estimate the complete motion sequence $\hat{Y}$. Since the primary role of keyframe context tokens $\Phi^{\text{key}}(K)$ is to derive the intermediate token embeddings, an additional projection is performed using a single FFN $\texttt{FFN}(\Phi^{\text{key}}(K))$ to reformulate the keyframing context tokens into keyframe tokens that adhere to the motion manifold. The intermediate tokens obtained from $\Phi^{\text{imd}}$ are ready to be used directly. To this end, we can construct the resulting motion manifold $\hat{M}$ with a token sequence $\{\hat{m}_{0},\hat{m}_{1},...,\hat{m}_{N-1}\}$ as follows:

where $\Phi^{\text{key}}(K)_{t}$ is the context token representation for the keyframe of index $t$ in the sequence.

Before applying the transformer $\Phi^{\text{syn}}$, we feed the tokens of $\hat{M}$ through a 1D convolutional layer of kernel size 3. For each layer of $\Phi^{\text{syn}}$, a self-attention function is followed by a FFN. The output token sequence of $\Phi^{\text{syn}}$ is fed through another 1D convolutional layer before the final linear projection. The final output $\hat{Y}_{t}$ consists of a root position estimation $\hat{p}_{t,0}\in\mathbb{R}^{3}$ of $p_{t,0}$ and local quaternions $\hat{q}_{t}\in\mathbb{R}^{J\times 4}$. Note that $p_{t,0}$ and $q_{t}$ jointly can be used to compute the global position $P_{t}$ and rotation $Q_{t}$ information by FK.

The stages in our method are trained jointly in an end-to-end manner using a set of loss functions. To determine the loss for a motion sequence of length $N$, we use $\ell_{1}$ distance for the following features obtained from $\hat{Y}$ and $Y$:

• •

  Local/root position loss: With predicted and real coordinate values of the root joint at the $t$-th frame as $\hat{p}_{t,0}\in\mathbb{R}^{3}$ and $p_{t,0}\in\mathbb{R}^{3}$ respectively,

  $$\footnotesize L_{root}=\frac{1}{N}\sum_{t=0}^{N-1}||\hat{p}_{t,0}-p_{t,0}||_{1}.$$

  (2)

• •

  Local rotation loss: With predicted (pre-normalised) and expected (unit-normalised) quaternion values of all joints at frame $t$ as $\hat{q}_{t,j}\in\mathbb{R}^{4}$ and $q_{t,j}\in\mathbb{R}^{4}$ respectively,

  $$\footnotesize L_{quat}=\frac{1}{NJ}\sum_{t=0}^{N-1}\sum_{j=0}^{J-1}||\hat{q}_{t,j}-q_{t,j}||_{1}.$$

  (3)

• •

  Global position loss: With predicted and real FK-derived coordinate values of all joints at frame $t$ as $\hat{P}_{t}\in\mathbb{R}^{J\times 3}$ and $P_{t}\in\mathbb{R}^{J\times 3}$ respectively,

  $$\footnotesize L_{FK_{p}}=\frac{1}{NJ}\sum_{t=0}^{N-1}||\hat{P}_{t}-P_{t}||_{1}.$$

  (4)

• •

  Global rotation loss: With predicted and real FK-derived quaternion values of all joints at frame $t$ as $\hat{Q}_{t}\in\mathbb{R}^{J\times 4}$ and $Q_{t}\in\mathbb{R}^{J\times 4}$ respectively,

  $$\footnotesize L_{FK_{q}}=\frac{1}{NJ}\sum_{t=0}^{N-1}||\hat{Q}_{t}-Q_{t}||_{1}.$$

  (5)

Although quaternions are unit-normalised in practice, we found that calculating $L_{quat}$ with non-normalised predictions resulted in improved gradient stability during the training process and, in turn, greater training convergence.

In summary, our training loss function $L$ is as follows:

where $\alpha_{l}$ and $\alpha_{g}$ are local and global feature loss scaling parameters, respectively. The accuracy of local attributes is best prioritised over that of global attributes, since normalised quaternions remain in use for deriving global features, which lead to gradient instability [15, 10].

We benchmark our method in the motion interpolation task against both the state-of-the-art RNN-based network [15] and BERT-based network [10] in motion transition generation. To evaluate the effectiveness of each model, they are to complete the following motion interpolation task:

• •

  Input: The model is provided with the keyframes $K$ of an $N$-frame ground truth motion. While keyframes can be defined for any combination of frames, we place each keyframe evenly every 5, 15, or 30 frames for consistency, starting with the first frame, e.g., $K=\{x_{0},x_{5},x_{10},...,x_{|K|}\}$ for 5-frame intervals.

• •

  Expected output: Each model is to output an $N$-frame motion $\hat{Y}=\{\hat{y}_{0},\hat{y}_{1},...,\hat{y}_{N-1}\}$ given the keyframes $K$. This output is compared with the ground truth with the L2P and L2Q metrics used in state-of-the-art comparisons [15, 10]. These metrics measure the average $\ell_{2}$ errors of all positional and rotational attributes respectively for each pose. In addition, we apply the Normalised Power Spectrum Similarity (NPSS) [12] metric to measure visual similarities between the estimated and actual motion outputs.

We source our motions for both training and evaluation from the Ubisoft La Forge Animation (LaFAN1) dataset [15] and the Carnegie-Mellon University Motion Capture (CMU Mocap) dataset ²²2Dataset available at: http://mocap.cs.cmu.edu/. The CMU Mocap motions are resampled from their original 120 frames per second (FPS) down to 30 FPS considering the computational cost, and to match the frame rate of the LaFAN1 dataset. While LaFAN1 focuses largely on motions with visibly dynamic details such as locomotion and sports actions, CMU Mocap provides a larger variety of motions, many of which exhibit more minute details and ambiguous motion trajectories. From a data-level perspective, this suggests that root position accuracy is more important in the LaFAN1 dataset compared to the CMU Mocap dataset. We employ both datasets to demonstrate our method’s ability to adapt with different levels of motion dynamics.

For our model, the positional data of each motion sample is recentred around the $XYZ$ means of the keyframed root positions. Given that unit quaternion values are restricted to a range of $[-1,1]$, while positional values can be of any scalar value, we rescale positional data such that $L_{\text{root}}\approx L_{\text{quat}}$ with initial model weights. During training, we randomly select between $[\lfloor\frac{|Y|}{24}\rfloor,\lfloor\frac{|Y|}{4}\rfloor]$ keyframes for each sampled motion $Y$, with the first and last frames being stipulated as keyframes. Each motion sample batch has a random length of $|Y|\in[72,144]$.

We implement our method with 8 layers for each transformer, with a token embedding size of $d=512$, and an FFN size of $4\times d$. Each multi-head attention layer is split between 8 attention heads. Since our method relies on coordination between Stage-I and Stage-II, the training stability of deeper models benefits greatly from larger batch sizes. For our 8-layer setting, we found that a batch size of 64 motions per epoch is sufficient for convergence.

We train our model using the Adam optimiser [20] for 50,000 epochs with a scheduled learning rate. Specifically, we employ both a warm-up and decay strategy for our learning rate using the following strategy [42]:

where $e$ represents the number of current training epoch. In addition, we linearly scale $\alpha_{g}\in[0,1]$ for 1,000 epochs after warm-up, in order to avoid conflicting gradients caused by ambiguous quaternion polarity when backpropagating through FK functions, i.e. $FK(Q)=FK(-Q)$ for any set of joint rotation quaternions $Q$. We set $\alpha_{l}=1$ throughout the training process.

We benchmark the performance of our method against state-of-the-art models by evaluating their L2P, L2Q, and NPSS metrics with testing datasets. Table 1 compares the performance of our architecture against the BERT-based motion in-painting transformer [10], the encoder-decoder-based $\Delta$-interpolator [31], the RNN-based approach $\text{TG}_{\text{complete}}$ [15], and the masked auto-encoder (MAE) architecture [16].

The quantitative performance of our model is greatly improved over all existing methods, as well as LERP, in a large majority of keyframing scenarios. The performance improvement of our method compared to LERP increases with the length of keyframe intervals, as using learning based methods provides the opportunity to reconstruct non-linear motion details. Note that for a short keyframe interval, a linear estimation ($f(x+\Delta x)=f(x)+\hat{f}^{\prime}(x)\Delta x+o(\Delta x)$) of a continuous motion (function) can be relatively accurate, which explains why similar performance is found between LERP and our method for the 5-frame interval setting. However, other existing methods are significantly worse than LERP. Our model notably outperforms both $\text{TG}_{\text{complete}}$ and MAE with its single token mask in every scenario. Thus, a clear motion interpolation improvement can be observed from our decoupling strategy with motion manifold technique, compared to the RNN model.

The BERT-based method [10] exhibits a clear disadvantage in its performance due to its reliance on LERP-based input mask tokens. By deriving the mask token embeddings from a sub-optimal estimation, self-attention mechanisms tend to converge towards reproducing the input token rather than composing more realistic poses, as it is close to a trivial local minimum to learn. Consequently, such models never learn to fully consider the keyframe tokens as their main source of information. Figure 3 highlights the near-identical latent manifolds of LERP output and BERT-based evaluation. We observe a similar behaviour with the $\Delta$-interpolator model, where LERP-based transformations are applied as a post-processing step [31]. While its $\Delta$-mode strategy allows the model to perform marginal improvements over LERP more frequently, it is still heavily reliant on the performance of LERP, which does not bode well with longer keyframe intervals. To this end, we can deduce that the realistic interpolation can be difficult with LERP-reliant solutions. Conversely, our manifold learning approach fully considers the continuous joint positions and rotations of the input keyframes, and is able to converge upon a significantly more optimal solution.

Table 2 documents an ablation study for each our architecture’s contributions. Major improvements to the architecture’s L2P, L2Q, and NPSS performance can be observed for the inclusion of manifold self-attention in Stage III, the replacement of LayerNorm with our sequence-level re-centering normalisation scheme, and concatenation over addition of PE. It should be noted that the Stages I and II only variant of our model (i.e., (a) in Table 2) is structurally identical to the $\Delta$-interpolator model with $\Delta$-mode disabled [31]. In addition, we demonstrate the use of $\Phi_{\text{key}}$ token representations over separately trained keyframe embeddings in Stage-III, which leads to improved convergibility of deeper architecture settings. We further demonstrate the importance of such deeper settings, which provide a significant boost to our model’s evaluation accuracy.

Figure 4 visualises the latent motion manifolds in 2D and 3D spaces for our method, LERP and ground truth using t-SNE analysis. The manifold of our method is obtained from the inputs of Stage-III, and LERP and ground truth are from the pose data. It can be observed that the lower-dimensional curves (i.e., manifolds) represent the motion of higher-dimensional in a smooth manner, and ours is very close to the one associated with the ground truth, compared with LERP. This indicates the superiority of our method to derive a high-quality latent motion space. Figure 3 illustrates an example of hopping motion interpolated by different methods with quantitative metrics. The ground truth data is reduced to a motion manifold with t-SNE. The offset of the manifold from each interpolation method compared to the ground truth one is obtained by $\ell_{1}$ distance for visualization. Particularly, the offset values are enlarged for observation purpose. Our method gains the best motion manifold with the least offset from the ground truth manifold.

Figure 5 dissects the L2P and L2Q improvements of our method into individual joints. We can clearly observe that the main improvements of our method over LERP exist within the global positions and rotations of foot joints, whereas improvements are more widely spread compared to the MAE and RNN-based methods. On average, our approach brings a L2P and L2Q benefit to all joints.

The inference time of different motion interpolation methods was evaluated in our experiments, as the visual latency of keyframe adjustments is important for efficient animation work, as well as real-time applications. We implemented these methods with PyTorch on an AMD Ryzen 9 3950X processor and an NVIDIA GeForce RTX 3090 GPU. Table 3 shows that the parallelism provided by the transformer is highly beneficial to our method when interpolating complete motion sequences. Our method shows a similar order of time complexity to the LERP method, being consistently around $2.5\times$ LERP inference time, and significantly faster than the RNN-based approach. In addition, our method is stable in terms of the inference time for different sequence lengths, whilst the RNN-based approach shows a significant increasing latency from 31-frame sequence to 121-frame sequence.

Though our model is designed for sparse keyframe interpolation, we can additionally perform motion completion as it can be defined as a specifically constructed keyframe set. Table 4 compares the performance of our model against linear interpolation and the state-of-the-art models for motion completion. With the benefit of the motion context, our model outperforms LERP, but not to the efficacy of the RNN-based [15] and transformer-based [10, 31] models.

One limitation of our approach is that its maximum motion length is limited by the length of its training samples. Unlike most transformer-based solutions that can trivially employ relative positional encodings [5, 35], our method relies on continuous positional vectors, such as sinusoidal encodings, for its manifold representations, and thus cannot employ the same model to accept inputs of arbitrary length. Further research for a compatible relative position representation would allow our approach to be seamlessly applied in keyframed animation workflows for longer sequences.