Bidirectionally Deformable Motion Modulation For
Video-based Human Pose Transfer

The major challenge of video-based pose transfer is to maintain the spatio-temporally misaligned characteristics of $I_{s}$ while synthesizing unseen content according to the target poses. In this subsection, we introduce a new modulation mechanism – Deformable Motion Modulation (DMM) to synthesize continuous frame sequences by modulating the affine transform of $I_{s}$ with an augmentation of spatio-temporal sampling locations. It aims to estimate local geometric transformations on an initially aligned feature space so that it can enhance the smoothness of the propagated features in forward and backward branches. We design the proposed DMM with three components, namely motion offset, motion mask and style weight, inspired by the success of Deformable Convolution Network (DCN)  [5, 51] and StyleGANv2  [14, 15]. As depicted in Figure 2, we parametrize them as the output of coarsely warped features $f_{i-1}$ in the forward branch or $b_{i+1}$ in the backward branch, the output results generated from previous layer ${\widetilde{x}}_{i}^{l-1}$ at time $i$, and the source style code from $I_{s}$. We firstly initialize the standard convolution as

$${\widetilde{f}}_{i}\ \left(p\right)=\sum_{k=1}^{K}{w_{k}\cdot f_{i}\ \left(p\right)}+bias\ ,$$

(1)

where $K$ is a set of the sampling location of a kernel, $y\left(p\right)$ is the convoluted result of input $x$ at position $p$ with the sampled weight $w_{k}$. To equip convolution with modulation and irregular receptive field of view, we formulate our proposed DMM as

$${\widetilde{f}}_{i}\ \left(p\right)=\sum_{k=1}^{K}{\ w_{k}^{\prime\prime}\cdot\ m_{i\rightarrow i-1}(p)\cdot f_{i}\ \left(p+p_{k}+o_{i\rightarrow i-1}(p)\right)},$$

(2)

where $p_{k}$ is the pre-defined kernel offset depending on $K$, $o_{i\rightarrow i-1}\in\mathbb{R}^{2K}$ and $m_{i\rightarrow i-1}\in\mathbb{R}^{K}$ are both learnable shift offsets and a non-negative modulation scalar for a kernel at $p$ location regressed by the geometric relationship between the propagated features $f_{i}$ or $b_{i}$ and the previous generation layer ${\widetilde{x}}_{i}^{l-1}$, $w_{k}^{\prime\prime}$ is the stylized weights modulated by the incoming statistics of style code extracted from $I_{s}$. More specifically, $w_{k}^{\prime\prime}$ is responsible for manipulating the style transfer accompanied with the motion mask $m_{i\rightarrow i-1}$ so that a long-range spatio-temporal correspondence of the sequence can be captured at the current time. This goal can be achieved by computing the weights with demodulation  [15], which is expressed as

$$w_{jhk}^{\prime}=A_{j}\cdot\ w_{jhk},$$

(3)

$$w_{jhk}^{\prime\prime}=\nicefrac{{w_{jhk}^{\prime}}}{{\sqrt{\sum_{jk}{{w_{jhk}^{\prime}}^{2}+\epsilon}}}},$$

(4)

where $w_{jhk}$ represents the weights of $j$-th input feature and $h$-th output feature map on $k$-th sampling kernel location, i.e., $w_{k}\subset w_{jhk}$, $A_{j}$ is the $j$-th scalar from the source style vector, $w_{jhk}^{\prime}$ is computed for estimating the affine transformation based on the statistics of incoming style code, $\epsilon$ is a small number to prevent computation from numerical error. The demodulation can well preserve the semantic details of the source image while it is able to interpolate unseen content by considering the forward and backward propagation features. The augmented spatio-temporal sampling offsets can also produce dynamic receptive fields of view to track the semantics of interest so that it can synthesize a sequence of good-looking and smooth video frames.

It has been a challenge to produce stable and smooth videos simply by relying on current pose to generate the target person image due to discontinuous noisy poses extracted by some third-party human skeleton extractors  [2, 21, 10]. We introduce a simple bidirectional propagation mechanism to interpolate the probability of missing structural guidance from both forward and backward propagation.

Mesh Flow. We define the transformation flow as $F_{i\rightarrow s}\in\mathbb{R}^{H\times W\times 2}$ between $P_{s}$ and $P_{i}$, where $H$ and $W$ are height and width of the generated image resolution, $P_{s}$ and $P_{i}$ are the source image and target pose at time $i$. Following previous work  [22, 25], we apply SPIN  [18] as the 3D human pose and shape estimator to predict parametric representations by inferencing RGB images into the implicit differentiable model SMPL  [26]. The SMPL representation consists of three major elements, including a weak perspective camera vector $C\in\mathbb{R}^{3}$, a pose vector $\theta\in\mathbb{R}^{72}$ and a shape vector $\beta\in\mathbb{R}^{10}$. It parametrizes a triangulated mesh to produce the explicit pose representation by computing the corresponding $SMPL\left(\theta,\beta\right)\in\mathbb{R}^{6890\times 3}$. By utilizing Neural Mesh Renderer (NMR)  [16] and the $SMPL$, we can obtain the corresponding visible vertices of triangulated faces $V\in\mathbb{R}^{13776\times 3\times 2}$ between the source and target mesh, and the weight index map of source mesh $W\in\mathbb{R}^{H\times W\times 3}$. Therefore, we can compute the $F_{i\rightarrow s}$ by matching the correspondence of source $W$ and $V$. The detailed computation is demonstrated in the Supplementary - Mesh Flow Computation.

Bidirectional propagation. Once we obtain the transformation flow $F_{i\rightarrow s}$, we perform feature propagation to extract the latent temporal information in a recurrent manner. We leverage a bidirectional propagation mechanism to manipulate the coarsely spatial-aligned sequence before feeding it into the generator. As shown in Figure 3, the pre-warped frames are formulated as

$$x_{i-N:i+N}=warp\left(F_{i\rightarrow s}\left(P_{s},P_{1:M}\right),I_{s}\right),$$

(5)

where $N=M/2$. We use a shared 2D CNN encoder to independently extract features of $x_{i-N:i+N}$ in both forward branch $\mathcal{F}$ and backward branch $\mathcal{B}$, respectively. With the recurrent propagation, the extracted features at time $i$ are encapsulated with the spatio-temporal information across the entire input sequence in the feature space. The temporal forward features and backward features computed at time $i$ are represented as

$$f_{i}=conv\left(\mathcal{F}\left(x_{i}\right)\circledcirc f_{i-1}\right),$$

(6)

$$b_{i}=conv\left(\mathcal{B}\left(x_{i}\right)\circledcirc b_{i+1}\right),$$

(7)

where $\mathcal{F}\left(x_{i}\right)$ indicates the feature maps of the forward encoder $\mathcal{F}$, $\mathcal{B}\left(x_{i}\right)$ is also used for backward encoder $\mathcal{B}$, and $\circledcirc$ denotes concatenation operator. With the recurrent features from forward and backward propagations, the model can expand the field of view across the whole input sequence so that a more robust spatio-temporal consistency is captured during the generation process. Moreover, the outliers of input noisy pose at time $i$ can also be interpolated by the warped features from $x_{i-N:i-1}$ to $x_{i+1:i+N}$. With the assistance of Equation  2, the probability of estimating generative result can be formulated as

$$q\left(x_{i}|I_{s}\right)=\prod_{i-N}^{i}q\left(f_{i}|f_{i-1}\right)+\prod_{i}^{i+N}q\left(b_{i}|b_{i+1}\right).$$

(8)

The combinations of $q\left(x_{i}|I_{s}\right)$ can dramatically provide positive gain to the network in synthesizing new content by feature interpolation.

Following similar training strategies in current pose transfer frameworks  [35, 25], the final objective loss function in our model is composed of six terms including a spatial adversarial loss $\mathcal{L}_{adv}$, a spatio-temporal adversarial loss $\mathcal{L}_{temp}$, an appearance loss $\mathcal{L}_{l1}$, a perceptual loss $\mathcal{L}_{per}$ a style loss $\mathcal{L}_{gram}$, and a contextual loss $\mathcal{L}_{cx}$ as follows:

$$\begin{split}\mathcal{L}_{full}=\lambda_{adv}\mathcal{L}_{adv}+\lambda_{temp}\mathcal{L}_{temp}+{\lambda_{l1}\mathcal{L}}_{l1}\\ +{\lambda_{per}\mathcal{L}}_{per}+{\lambda_{gram}\mathcal{L}}_{gram}+{\lambda_{cx}\mathcal{L}}_{cx}\ ,\end{split}$$

(9)

where $\lambda_{adv}$, $\lambda_{temp}$, $\lambda_{1}$, $\lambda_{per}$, $\lambda_{gram}$, and $\lambda_{cx}$ are the hyperparameters to optimize the convergence of the network.

Spatial adversarial loss. We utilize the traditional generative adversarial loss  [9, 29] $\mathcal{L}_{adv}$ to mimic the distribution of the training set with a convolutional discriminator $D_{s}$. It is formulated as:

$$\mathcal{L}_{adv}=\mathbb{E}\left[\log{\left(D_{s}\left(I_{s},\ I_{i}\right)\right)}+\log{\left(1-D_{s}\left(I_{s},\ {\hat{I}}_{i}\right)\right)}\right],$$

(10)

where $(I_{s},I_{i})\in\mathbb{I}_{real}$, ${\hat{I}}_{i}\in\mathbb{I}_{fake}$ , and $i\in 1\ldots M$ indicate samples from the distribution of real person image, generated person image, the numbers of an input patch.

Temporal adversarial loss. Similar to $\mathcal{L}_{adv}$, the temporal adversarial loss $\mathcal{L}_{temp}$ optimizes the temporal consistency in time and feature channels of a mini patch with a 3D CNN discriminator $D_{t}$.

Appearance loss. To enforce discriminatively pixel-level supervision, we employ a pixel-wise L1 loss to provide guidance on synthesizing photo-realistic appearance compared to the ground-truth image.

Perceptual loss. To minimize the distance in feature-level space, we apply a standard perceptual loss  [12]. It computes the L1 difference of a selected layer $\ell=Conv1\_2$ from a VGG-19  [38] model $\theta_{\ell}\left(\cdot\right)$ pre-trained in ImageNet  [6]. It is defined as

$\displaystyle\mathcal{L}_{per}=\sum_{C_{\ell}H_{\ell}W_{\ell}}\|{\theta_{\ell}({\hat{I}}_{i})-\theta_{\ell}(I_{i})|}\|_{1},$

(11)

where $C_{\ell}$ is the number of channels, $H_{\ell}$ and $W_{\ell}$ are the height and width of the feature maps in a particular layer $\ell$ respectively.

Style loss. Similar to the perceptual loss to minimize the L1 distance in feature-level space, we further calculate the Gram matrix of some activated feature maps at the selected layers to maximize the similarities.

$\displaystyle\mathcal{L}_{gram}=\sum_{C_{\ell}H_{\ell}W_{\ell}}\|{Gram(\theta_{\ell}({\hat{I}}_{i}))-Gram(\theta_{\ell}(I_{i})|)}\|_{1},$

(12)

where the used layers are the same as in perceptual loss.

Contextual loss. To maximize the similarities between two non-aligned images in context space, we utilize the contextual loss  [30] to allow spatial alignment according to contextual correspondence during the deformation process.

$\displaystyle\mathcal{L}_{cx}=-\sum_{C_{\ell}H_{\ell}W_{\ell}}log\left[CX\left(\delta_{\ell}\left({\hat{I}}_{i}\right),\delta_{\ell}\left(I_{t}\right)\right)\right],$

(13)

where $\ell=relu\left\{3\_2,4\_2\right\}$ layers from a pre-trained VGG-19 model $\theta\left(\cdot\right)$, the $CX\left(\cdot\right)$ function is the similarity measurement defined in  [30].

Dataset. We conducted experiments on two publicly available high-resolution video datasets for video-based human pose transfer, including FashionVideo  [47] and iPER  [24]. Both are collected from a human-centric manner with diverse garments, poses, viewpoints, and occlusion scenarios. The FashionVideo consists of 600 videos with around 350 frames per video. It is partitioned into 500 videos for training and 100 videos for testing. It is collected from a static camera and a clean white background. The iPER dataset contains 206 videos with roughly 1100 frames each. There are 164 videos for training and 42 videos for testing purposes. Different focal lengths and genders are included to capture various poses and views in some indoor or natural backgrounds.

Evaluation metrics. To evaluate structural similarity, the SSIM  [44] index is used to achieve this goal by applying covariance and mean. The PSNR computes the power of maximum value and its mean squared error. The L1 distance represents the pixel-wise fidelity. We also employ two supervised perceptual metrics including Fréchet Inception Distance (FID)  [11] and Learned Perceptual Image Patch Similarity (LPIPS)  [50]. The FID is used to measure the distribution disparity between the generated images and the training images by computing the perceptual distances. The LPIPS is targeted on evaluating the Wasserstein-2 distance between the distributions of the generated samples and real samples. To measure the temporal coherence, we utilize Fréchet Video Distance (FVD)  [41] to extract features on time and feature space by a pre-trained I3D  [3] network. It considers a distribution over the entire video, thereby avoiding the drawbacks of frame-level metrics. The term “FVD-Train128f” denotes the protocol of computing the FVD on randomly selected consecutive $128$ frames for a sequence on training set and generated images with $50$ iterations, likewise for “FVD-Test128f” on testing set.

Training strategy. We implement the proposed method with the public framework PyTorch. We adopt the Adam  [17] optimizer with momentum $\beta_{1}=0.5$ and $\beta_{2}=0.999$ to train our model for $50,000$ iterations in total. The learning rate is set to ${10}^{-4}$. To keep the original aspect ratio of the images, we resize the video frames to $256\times 256$ by thumbnail approach. The negative slope of LeakyReLU  [32] is set to $0.2$. The weighting hyperparameters $\lambda_{adv}$, $\mathcal{L}_{temp}$, $\lambda_{1}$, $\lambda_{per}$, $\mathcal{L}_{gram}$, and $\lambda_{cx}$ are set to $5$, $5$, $2$, $500$, $0.5$, and $0.1$. All models are trained and tested on a server with four NVIDIA GeForce RTX 2080 Ti GPUs with 11GB memory for each.

To demonstrate the superiority of the proposed model, we compare our model with several state-of-the-art approaches including GFLA  [35], Impersonator++  [25], DPTN  [48], and NTED  [34].

Quantitative Comparison. As shown in Table 1, our model achieves the best results on most evaluation metrics. The large margin of enhancement on FVD score indicates the best performance of our method in terms of spatio-temporal consistency. It represents the merits of the proposed bidirectionally deformable motion modulation in modulating long-range motion sequences with minimum discontinuity. Our model can achieve the best results on those image-based perceptual measuring metrics in the challenging FashionVideo dataset. For some images with natural backgrounds like those in iPER dataset, our model is also able to get highly competitive performance. It quantifies that our model has a better style transfer and video synthesis ability against current methods.

Qualitative Comparison. Apart from quantitative comparison, we also conduct a comprehensive qualitative measurement to compare the perceptually visual quality with the state-of-the-arts. We illustrate some generated results with various poses in Figure 4. We demonstrate a wide variety of viewpoints including front view, left side of body, right side of body, and back view on the Fashion dataset (row 1 – row 4). These results can highlight the superiority of our method from transferring person facial characteristics and complex texture on the garments in different points of view. To evaluate the synthesis quality in natural background, we present some generated results on uncommon gestures in iPER dataset (row 5 – row 8). It shows that our method can confidently handle arbitrary poses, shapes and backgrounds with minimum generated artifacts compared with others. It is benefited from the irregular field of view constructed by the deformable motion offset so that the multi-scale features can be effectively activated.

As a video-based solution, our method can generate temporally coherent sequences conditioned on some noisy poses without pre-processing, as shown in Figure 5. In general, the majority of structural guidance is hampered due to statistical outliers, especially in some occluded scenarios. It leads to an uncompleted shape and artifacts on the generated images, even though recurrent neural networks are applied in  [24, 35, 25]. With the proposed bidirectional modulation mechanism, our method can synthesize smooth sequences with high-fidelity transferring effects.

Deformable Motion Modulation. The proposed DMM is used to synthesize continuous frame sequences by modulating the 1D style code of source image with an augmentation of spatio-temporal sampling locations. We use a sum operation to fuse the features extracted from bidirectional propagation. Compared to the model w/o DMM, our model achieves superior results for all evaluation metrics in Table 2. The lack of style modulation mechanism leads to failure in style transferring results, in spite of the simple appearance style from the source image, as shown in Figure 6 (a). Moreover, based on the result of the model w/o DCN, we observe that there is a positive gain in synthesizing new content if the receptive field of view during convolution is expanded. It can achieve higher FID and LPIPS scores for image-based perception. The enhancement on FVD demonstrates the importance of capturing temporal information from adjacent frames. Furthermore, we compare the results with the model w/o Style Weight. The modulated style weight is an important component to perform affine transformation decomposed from style codes to structural poses. As depicted in Figure 6 (b), the generated images are not with style consistence due to lack of a generalization on style transfer. It verifies that our proposed DMM can provide benefits to the fusion of style statistics so that it can minimize the distribution between real-world images and the synthesized.

Forward / Backward Propagation. The proposed bidirectional propagation mechanism is used to interpolate the probability of missing structural guidance from both forward and backward propagation flows in order to enhance the temporal consistency. The results of evaluation metrics in Table 2 report that they both have an effective contribution in generating realistic images and maintaining temporal coherence between adjacent frames. In addition, the qualitative results in Figure 6 (c-d) demonstrate that the forward and backward propagation can preserve more details on structural shape and appearance details. Both comparisons on different measurements verify the efficacy of the proposed bidirectional propagation flow.

The proposed DMM uses geometric kernel offset to transform regular receptive field of view to some irregular shapes  [5, 51]. To investigate the effectiveness of the proposed deformable motion modulation, we illustrate some visualizations on the DMM module in feature space.

Region of Interest. The region of interest for DMM is to highlight the global area with effective motion offsets and motion mask. As demonstrated in Figure 7, we plot the kernel offsets as a kind of optical flow by following  [1] so that we can observe the activated regions of interest in each propagation branch. The visualizations for the motion mask also highlight the activated magnitude along with the motion offsets. It is reasonable that the motion offsets and masks are not aligned for both forward and backward branches because they are designed to capture the temporal information in two different sequences. Based on the global shape on the offset regions and masks in both forward and backward propagation, we can clearly point out the human body shape with a predictable movement. The regions with more semantic information are with higher density. The activated regions provide geometric guidance for the network to modulate the style code extracted from the source image.

Activated Unit. The success of the proposed DMM relies on the augmentation of spatio-temporal sampling locations. We visualize the behavior of the deformable filters in Figure 8. The activation units are highlighted with green points and red points for the corresponding augmented sampling locations. It is obvious that the proposed semantics on the sampling locations are dependent on the activated units. It is certified that the proposed DMM module can produce a dynamic receptive field of view to keep track of interested semantics so that it can synthesize a sequence of high-quality and smooth video frames.