One Style is All You Need to Generate a Video

Our generator consists of three distinct networks: a synthesis network $\mathbf{G}$, a temporal style generator $\mathbf{F_{t}}$, and a conditional embedding $\mathbf{F_{c}}$ as shown in Fig. 1. The synthesis network $\mathbf{G}$, which is based on StyleGAN2, is inherently agnostic to temporal cues when generating images. To ensure the generation of temporally coherent video frames, we introduce a specific temporal embedding, which interfaces with $\mathbf{G}$ to guide the synthesis process using a latent trajectory. This trajectory is derived from the network $\mathbf{F_{t}}$, which comprises a 4-layer Perceptron (MLP) that maps a random vector $z_{m}$ to a $k$-dimensional motion style vector $m$. At this stage, the vector $m$ does not encompass any temporal context. An auxiliary network time2vec produces sinusoidal bases that are scaled by $m$ to finally output the temporal style vectors $w^{t}_{m}$.
Time2vec: Our proposed $k$ dimensional time embedding consists of $k-1$ sinusoidal bases and a linear term as seen in Eq. 1, where the parameters $w_{j}$ and $\phi_{j}$ are trainable.

$$v_{j}(t)=\mathcal{F}(\omega_{j}t+\phi_{j}),$$

(1)

where $\mathcal{F}$ is the identity function when $j=0$ and the sine function for $1\leq j\leq k-1$. The linear term $v_{0}(t)$ represents the time direction. The time $t$ does not need to be discrete as the time2vec embedding is continuous. This allows us to generate videos with arbitrary frame rates. However, during the training we use integer valued time-points. We note that StyleGAN-V’s time representation lacks the linear term, which might explain why its generation is plagued by unnatural repetitive motion despite its elaborate interpolation scheme. By restricting the dynamics to a fixed set of sinusoidal functions, we avoid over-fitting to the training data, and make the model more robust and generalizable to unseen data. Moreover, since sinusoidal functions are periodic, they can naturally capture cyclic patterns in the data (e.g. lip movement, hand waving). We obtain a temporal style vector as an input to $\mathbf{G}$ using the following set of equations:

	$\displaystyle m$	$\displaystyle=\mathbf{F_{t}}(z_{m}),$		(2)
	$\displaystyle w^{t}_{m}$	$\displaystyle=m*v(t),$		(3)
	$\displaystyle w^{t+1}_{m}$	$\displaystyle=m*v(t+1).$		(4)

The product of motion style $m$ and temporal embedding vector $v(t)$ is a temporal style vector $w^{t}_{m}$. Note that a single $m$ is used to compute temporal styles for consecutive frames. Additionally, $\mathbf{F_{c}}$ encodes the action and actor embeddings and outputs a content style vector $w_{c}$. It defines the general appearance of the actor along with the nature of the action. To generate a frame at time $t$, both temporal and content styles $[w_{c},w^{t}_{m}]$ are concatenated and injected to the synthesis blocks. During the training, we generate three consecutive frames for each video element of the batch. The triplets share the same vector $m$ while their temporal embeddings are generated from their respective time points. During the inference, a single vector $m$ is enough to generate a long duration video. We leverage this ability of $m$ to encapsulate the entire dynamics of a sequence to compute a temporal GAN inversion, as described in Section 5.4. A basic structure of the generator network is shown in Figure 1. To ensure the smooth integration of action-id embeddings, we employ a ramp function [28] that linearly scales the vectors derived from the action-id embedding with a factor ranging from 0 to 1, in a scheduled manner.

Unlike StyleGAN-V, we choose to stay closer to the StyleGAN’s original principle, which is to allow variations in input only through the style vectors. Furthermore, our time embedding fundamentally differs from StyleGAN-V’s in its design. StyleGAN-V requires multiple randomly sampled vectors to compute wave parameters which ultimately defines the motion of a generated video. In contrast, our wave parameters are independently learned and are fixed during inference. Our latent vector $m$ interacts with the waves only as an amplitude scaling factor. Hence, our time representation is simpler and manipulable. We leverage these advantages in Section 5.4 to perform GAN-inversion of the motion style using off-the-self methods, which cannot be achieved with StyleGAN-V.

Shuffle discriminator: Consistency in content over time is a crucial aspect of video generation. Although the time2vec module in $\mathbf{G}$ provides temporal bases to guide motion learning, it does not ensure consistency in content across the sequence. In order to address this, we design a 2D-CNN based discriminator $\mathbf{D_{s}}$ (as seen in Figure 1) that evaluates whether the frame features are consistent or not. During the training of $\mathbf{D_{s}}$, each batch element consists of two frames. For the fake adversarial example, pairs of frames are shuffled among the batch to contain two different contents. In contrast, for the real example, the pairs are consecutive frames drawn from real videos. The feature maps of the pairs undergo a series of 2D convolutions, are flattened, and then concatenated into a single vector before passing through a fully connected layer. During the training of $\mathbf{G}$, a batch of unshuffled fake pairs is input to $\mathbf{D_{s}}$.

As shown in Figure 1, two additional linear layers ($d_{action}(.),d_{actor}(.)$) are present at the level of $d_{t}(.)$, which produce actor and action representations. Dot products are computed between the corresponding embedded vector and the feature vector. The final output of the discriminator is the weighted sum of the three dot products. We use the same ramp-up function to scale $d_{action}(.)$ as in the generator [28].

We have used three publicly available video datasets with their labels: MEAD [37], RAVDESS [20] and UTD-MHAD [3]. Our MEAD training set contains 30 individuals talking while expressing 8 different emotions ($18883$ videos). We train our network only with the sequences where generic sentences are being recited. We set aside the emotion specific dialogues as unseen test sequences. For the training, we chose $128\times 128$ image dimension and between $60-170$ frames as the dataset contains videos of variable length.

The RAVDESS dataset contains 24 talking faces also with 8 different emotions (not same categories as MEAD). To create a test set, we exclude sequences of 7 different emotions for four individuals. Though the dataset set contains only two dialogues, compared to over 20 dialogues in MEAD, RAVDESS contains more variation in head movements of the actors.

UTD-MHAD contains $754$ videos of 8 individuals performing 27 different actions. The video frame size is $128\times 128$ with variable video length ($33-81$) as provided in the dataset. We created a test set by excluding videos of each action sequence performed by few selected target actor from the training set. Thus, we train the network to learn motion and content independently.

For the conditional video generation, we choose ImaGINator [39] as our baseline. Though it requires a conditional input image to generate videos, it is free of any additional representation like pose or motion maps. We adapted its network to output $128\times 128\times 32$ size image (originally $64\times 64\times 32$). We trained it on MEAD and UTD-MHAD datasets for up to $5$K epochs.

To demonstrate that our generator does not falter in video quality, we choose MOCOGAN-HD [33] and StyleGAN-V [30] as our baselines in unconditional setting as they both use StyleGAN2’s image synthesizer. For MOCOGAN-HD, we first trained a StyleGAN2 network on MEAD dataset with $256^{2}$ image size for upto 150K iterations. Then the MOCOGAN-HD network was trained with the hyper-parameters set as suggested in the author’s implementation. For StyleGAN-V, we trained on with image of dimension $256^{2}$, with a batch size of 64 and with up to 25000K images according to the author’s implementation.

We trained our method on a single Nvidia’s A100 GPU with $80$GB VRAM. The training image size was $128^{2}$ with a batch size of $16$ triplet frames. The hyperparameters for the generator, discriminators and the optimizers were kept the same as suggested in [17]. The transition factor $\lambda$ of action-id vectors in both generator and discriminator started at $4000$ iterations and ended at $6000$ iterations, which was set empirically. We trained our model on all datasets for up to $120k$ iterations which took about 2 weeks. Our method can generate longer videos with diverse motion types and arbitrary frame rates.

Table 1 reports the FVD scores of the generated videos by all the methods. Our conditional method (Ours(C)) scores the best which is in agreement with the videos provided in the supplementary data. Few frames of the generated video samples are depicted in Figure 2. Motion artifacts are strongly present in MOCOGAN-HD and ImaGINator’s output. Though StyleGAN-V generates long duration videos, it suffers from erratic, repeated motion. Our methods (both conditional Ours(C) and unconditional Ours(UC)) produce far better results. We have reported FVD score computed over 64 frames only for StyleGAN-V and our method as other baselines are incapable of long duration video generation.

To assess the preservation of the actor’s identity, we computed the ArcFace[6] similarity between the frames of the generated videos. ArcFace computes the cosine similarity between the feature vector of the first and the successive frames obtained from face recognition network. As seen in Table 1, our methods preserve the appearance of the actor throughout the sequence while MOCOGAN-HD is not consistent generating the same face over the sequence. The authors of [30] also made this observation.

The FVD score is widely used to evaluate video quality. However, as it is a comparison of distributions of representations in a high dimensional space, it may not accurately characterize the true quality of the video. The same can be said about the ArcFace score. Furthermore, these metrics can be influenced by factors such as spatial resolution, video length, etc. To complement these metrics, a human evaluation was conducted to assess the realism of the generated videos. To conduct the human evaluation, we generated 10 sets of videos, each consisting of 6 videos with 32 frames (1 real video and 5 generated videos using the proposed methods and the baselines). We asked 25 university students and researchers to watch 3 randomly selected sets and rank the 6 videos based on their perceived realism. The ranking distributions of the survey is presented in Figure 3. Notably, videos generated with Ours(C) and Ours(UC) models consistently ranked higher than those generated using the baseline methods. This demonstrates that our method produces more realistic videos compared to existing approaches.

While we demonstrated that the video quality is improved, the aforementioned metric cannot assess the preservation of temporal semantics across different sequences. We then propose a new metric named LiA (for Lips Area) to evaluate our ability to reproduce the semantic of talking-face videos while changing the content such as the actor-id or action-id (emotion). LiA value computes the polygonal area of the lips detected using face landmark detectors [19]. A LiA signal is then obtained by computing LiA value sequentially for each frames of a generated or a real video. Though there are other factors such as eye brows and head orientation that contribute to the overall dynamics of a talking face, we focus on lip motion as it appears to be the most dynamic part of the face on this dataset. We generated $100$ different sequences using different content styles and the same temporal style for the baseline methods. The average correlation coefficient $\bar{\text{r}}_{t}$ of the LiA signals of the generated videos by all the methods are reported in Table 1. We observed that even for the same temporal style, the ImaGINator produced different motion pattern depending on the starting input frame. MOCOGAN-HD has relatively low $\bar{\text{r}}_{t}$ score as the face unusually distorts over time.

We generate videos of unseen actor-action combination only present in the test set. Figure 4 shows a few selected frames of real videos, and generated videos by ImaGINator, and our conditional method. Our method is able to successfully transfer a learnt action to an actor who was never seen performing this action in the training set. To evaluate our method in this dataset, we additionally train an action recognition model using the implementation of [8]. We train the model on skeletal key points extracted from the video frames of our training set which contains 27 different actions. The trained model was able to achieve (77%, 100%) top-1 and top-3 accuracies on the real test cases. In our generated case, it was able to achieve (68.5%, 93.5%) top-1 and top-3 accuracies. We present the confusion matrices for 27 different classes in the supplementary data (in Appendix Section 4). Our model not only generated high-quality videos, as shown in Table 2, but also accurately captured many actions. On the other hand, ImaGINator performed poorly on this dataset, with evidence of mode collapse in the type of motion despite the conditioning during inference. We have included the generated videos in the supplementary data.

A talking face video can be generated with a random temporal style, however it is a challenging task to map a real motion to a learned temporal representation. However, thanks to our simple temporal representation $m$, it is possible to extract a temporal style of real lip movement, without the need of any motion computation or landmark point detection, directly by GAN-inversion. To the best of our knowledge, it is the first time a GAN-inversion for temporal styles is proposed to recover a reusable dynamic representation.
In the following experiments, we invert the temporal style of unseen videos from test cases of MEAD and RAVDESS dataset. For the MEAD dataset, we recover the motion from real video of actors pronouncing sentences which were excluded from the training set. We assume that the excluded dialogues carry unseen lip motions, and recovering such motion should demonstrate the flexibility of our temporal representation. In these experiments, conditional labels are set to the known actor and emotion of the real input sequence and only $m$ is recovered by optimization. To this end, we minimize the sum of the LPIPS loss [46] and the MSE between $N$ real and generated frames:

$\displaystyle m^{*}$

$\displaystyle=\underset{m}{\mathrm{arg\,min}}\sum_{t=0}^{N}\mathcal{L}(I(t),G([w_{c},w^{t}_{m}])),$

(5)

where, $I(t)$ is the real video frame at time $t$, $\mathcal{L}$ is the summation of the two losses, and $m^{*}$ is the optimized motion style vector. Figure 5(a) shows an example of LiA signals for real and inverted videos using our model trained with $k=64,128,256$. The higher the number of sinusoidal bases used in the generator, the more faithful the recovered motion is to the real video. We performed the inversion and LiA signal analysis for $39$ different emotion specific sentences excluded from the training set (see Appendix Section 6 for the complete list), and report the average correlation to be $0.6$, $0.79$ and $0.91$ for $k=64,128,256$ respectively. The evaluation was done for input videos with $120$ frames because of the memory limitation. This implies that a single vector $m^{*}$ can faithfully represent the dynamic of at least up to $120$ frames. Furthermore, in Figure 5(b), the inversion is able to recover the large movement of head in the RAVDESS dataset. The facial structure is further improved using pivotal tuning [25] where we adjust generator’s weight by fixing the previously optimized $m^{*}$. Thus the recovered motion in the form of $m^{*}$ can then be transferred to another actor. We believe this is a novel way for re-enactment between different actors and actions.

Because our content and motion spaces are highly disentangled, it is possible to edit the attributes of the videos over time. We choreograph a sequence where actors change their expression over time by a linear interpolation in the action embedding space (see supplementary videos). The interpolation does not interfere with the general motion.