TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator

We aim to generate a realistic fake video $V_{F}=(I_{1},I_{2},\cdots,I_{2^{n}})$ that matches with the given text description $t$ using a recurrent unit $R$ and a generator $G$ where $n\in\mathbb{N}$ and $I_{i}$ is each frame of video $V_{F}$. At this stage, we only focus on the text-to-image generation task without considering the image sequence. Then, our goal is simplified to the generation of the single realistic image $I_{1}$ from $t$.

To train the model with text, we must first transform the text string into an encoded feature vector. We adopted the pre-trained skip-thoughts vector network [24] to encode text $t$ into a $4,800$-dimensional vector. Since the encoded vector is high-dimensional, we used principal component analysis (PCA) to derive meaningful features and reduce its dimensionality. We defined this embedded vector as $\phi_{t}$.

We start with a single GRU cell $R$ and a generator $G$ to create one frame. Given the embedded text vector $\phi_{t}$, the recurrent unit $R$ outputs the vector $v_{1}=R(z_{0},(\phi_{t},z_{1}))$, where $z_{0}$ and $z_{1}$ are random noises from $\mathcal{N}(0,1)$. The noise $z_{0}$ is a initial hidden state, and $(\phi_{t},z_{1})$ is an input of $R$. This $v_{1}$ from $R$ is the source input vector for $G$, and it creates the resulting image $I_{1}=G(v_{1})$ with the same size as the real frame. To ensure that $I_{1}$ matches with the provided text description $t$ and follows the distribution of real data, we trained the generator $G$ and the image discriminator $D_{I}$ adversarially using a GAN framework with slightly modified losses consisting of real, wrong, and fake pair as similar to those used by Reed et al. [11]. Overall losses will be explained in Section III-E.

When training the image discriminator $D_{I}$, the real image $X_{i}$ is randomly selected from $2^{n}$ frames of the real video $V_{R}$. Therefore, $G$ and $R$ aim not only to create the corresponding image for the given text but also to model the image distribution of the frames in the given video dataset. After the training of the text-to-image generation, $G$ can generate various images if some appropriate input vector is received. Therefore, if $R$ provides a meaningful sequence of vectors, $G$ can easily generate consecutive frames, which leads to realistic video generation. $R$ could act as an instructor that teaches $G$ to synthesize consecutive frames. This initial stage is the basis of the whole training process and plays a significant role in generating a series of frames.

The evolutionary generation stage begins after the completion of the initial stage training described in the above section. We now create a series of consecutive frames based on the model trained in the previous stage. Since $G$ has the ability to generate various frames and $R$ is a recurrent unit that can output a series of meaningful vectors, the extension of the text-to-image generation can lead to successful consecutive frames generation.

This stage consists of the process from step $1$ to $n$, and the goal of each step $m\in\{1,2,\cdots n\}$ is to generate $2^{m}$ consecutive frames stably. As the step proceeds, the number of created frames increases, and we can finally reach the video-level generation we desired. In each step $m$, $2^{m}$ images can be obtained by iterative operations of $R$ and $G$ learned in the previous stage. Let’s look at an example of the generation in step $1$. After the text-to-image generation, we forward $R$ once more with $v_{1}$ as a hidden state and the ($\phi_{t}$, $z_{2}$) as an input where $z_{2}$ is randomly sampled noise from $\mathcal{N}(0,1)$. Then the next latent vector $v_{2}=R(v_{1},(\phi_{t},z_{2}))$ is obtained from $R$. This vector $v_{2}$ is again delivered through the same $G$ to create another image, $I_{2}$.

Unlike the text-to-image generation stage, the temporal consistency of the generated frames should be managed in this stage. At each step $m$, $m^{th}$ step-discriminator $D_{S_{m}}$ is newly added to discriminate the sequence of $2^{m}$ frames. $D_{S_{m}}$ receives the fake input $(I_{1},I_{2},\cdots,I_{2^{m}},\phi_{t})$ and the real input $(X_{1}$ $,X_{2},\cdots,X_{2^{m}},\phi_{t})$ where $(X_{1},X_{2},\cdots,X_{2^{m}})$ are $2^{m}$ randomly selected connected frames from the real video $V_{R}$. Since the real input has temporal information (images are concatenated with original order), the fake input should be generated to have temporal information correctly. After this training step converges, we move on to the further step $m+1$. Then, $D_{S_{m}}$ is removed, and training proceeds with a new step-discriminator $D_{S_{m+1}}$.

To summarize, we use different step-discriminator $D_{S_{1}},\cdots,D_{S_{n}}$ on individual $1,2,\cdots,n$ steps. For $m\in\{1,2,\cdots,n\}$, $R$ repeats $2^{m}$ times, and $G$ generates a sequence of $2^{m}$ images from input vector $v_{1},v_{2},\cdots,v_{2^{m}}$. $D_{S_{m}}$ is initialized at the beginning of step $m$ to discriminate the $2^{m}$ images, and it is only used for the step $m$. Meanwhile, $D_{I}$ is used through all steps $1,2,\cdots,n$ to maintain high-quality image output. This step-by-step training converges quickly and stably. When all the training finishes, we can easily generate the images by forwarding single $R$ and $G$ for $2^{n}$ times, and it forms the video with $2^{n}$ lengths. The detailed training procedure and algorithm is described in Section III-E and Algorithm 1.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.99]CVPR_step_D.pdf Illustration of the first convolution layer of step-discriminators when step changes. At the beginning of step $m$ of evolutionary generation stage, step-discriminator $D_{S_{m}}$ is initialized using $D_{S_{{m-1}}}$.

Each step-discriminator $D_{S}\in\{D_{S_{1}},\cdots,D_{S_{n}}\}$ is newly added at the beginning of every step of the evolutionary generation stage. Since $D_{I}$ is already trained through the steps, there could be an imbalance in the training progress of discriminators $D_{I}$ and $D_{S}$. Therefore, simply applying the adversarial learning of $D_{I}$ and $D_{S}$ can even disrupt the learning of $G$. To prevent this, we present an advanced scheme which initializes the $D_{S}$ to a better state rather than noise.

For all $m\in\{1,2,\cdots,n\}$, the weight of $D_{S_{m}}$ is initialized with the previous step-discriminator $D_{S_{m-1}}$. Image discriminator $D_{I}$ is used for initialization of the first step-discriminator $D_{S_{1}}$. We designed all the step-discriminators to have the same architecture except for the number of input channels in the first layer. The only difference is that $D_{S_{m}}$ receives $2^{m}$ images as input, while $D_{S_{{m-1}}}$ receives $2^{m-1}$ images. Let $F_{m}^{l,k}$ be the weights connected to the $k^{th}$ input channel of the $l^{th}$ layer of step-discriminator $D_{S_{m}}$. Then, our step-discriminator initialization can be defined as:

All layers except the first layer are initialized with same weight of the step-discriminator $D_{S_{m-1}}$. However, there is a slight variation only in the first layer, which is that $F_{m}^{1,2i-1}$ and $F_{m}^{1,2i}$ are initialized to $F_{m-1}^{1,i}/2$ for all $i=1,\cdots,2^{m-1}$. This is illustrated in Figure III-B. After the evolution in step $m-1$ to step $m$, the appearance of the generated images within each step should be similar, but the number of the resulting images are twice as long. To retain the output of the discriminator when the step changes, we divide the weight value by $2$. We believe that this initialization technique can assist the training of our framework to maintain stability even with sudden step changes. The effect of this method will be discussed further in the results section.

One of the main failures in training GAN is mode collapse. To mitigate this phenomenon, independent samples pairing (ISP) is applied in our training process. Our model produces two output images ${I_{k}}^{a}$ and ${I_{k}}^{b}$ with one fixed text description $t$ and two different randomly sampled input noises $z_{k}^{a}$ and $z_{k}^{b}\in\mathcal{N}(0,1)$ when generating the $k^{th}$ frame ($k=1,2,\cdots,2^{n}$). These two independently generated images are paired by concatenation in the channel dimension and form the fake pair. To make the generator create various examples corresponding to the same text $t$, we train the discriminator to distinguish between $({I_{k}}^{a},{I_{k}}^{b},\phi_{t})$ and $(X^{a},X^{b},\phi_{t})$, where $X^{a}$ and $X^{b}$ are two real dissimilar images associated with the same text description. Since $X^{a}$ and $X^{b}$ are dissimilar, if a mode collapsed generator generates very similar $I_{k}^{a}$ and $I_{k}^{b}$, it will be easily detected as fake by the discriminator. Thus, the generator attempts to create different images with the same $t$ to deceive the discriminator. This independent samples pairing technique is only used for image discriminator $D_{I}$.

We retain the adversarial training framework of the generator and discriminator. Unlike in a conventional GAN, however, we include a slight perturbation by having two branches on the discriminator similar to  [8]. At first, the discriminator passes several convolution layers to acquire a high-level feature map. Then, one branch calculates the text-image match loss by concatenation with $\phi_{t}$, and the other branch performs patch discrimination without text concatenation. The operations of these two independent branches ensure that the image matches well with the text while improving the image quality. We use three kinds of text-image losses to train the model for text matching, the same as those used by Reed et al. [11]. The losses are obtained by a real pair ($X$, $\phi_{t}$), fake pair ($I$, $\phi_{t}$), and wrong pair ($X$, $\hat{\phi_{t}}$), where $\hat{\phi_{t}}$ is one of the embedded text vector that is not identical to $\phi_{t}$.

The overall loss for $G$, $R$, $D_{I}$, and $D_{S}$ are as follows:

where

and

$X^{i}$ and $I^{i}$ are randomly selected frames from real and fake videos, respectively. $D_{I}(I^{i})$ is an image loss without text conditioning, and $D_{I}(I^{i},\phi_{t})$ is the pair loss with text conditioning. At evolutionary step $m$, $D_{S}$ represents $D_{S_{m}}$. Here, $X^{i,S}$ is a $2^{m}$ consecutive images set randomly selected from the real video, and $I^{i,S}$ is generated fake images set with $2^{m}$ images. They are concatenated in channel dimensions. $L_{S}$ is only used at the evolutionary generation stage, and $L_{I}$ is used through all processes. Our overall training algorithm is described in Algorithm 1.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=1]access_kth.pdf Qualitative results of the models trained on KTH dataset. (a) Our generation results. (b) Comparative results with previous works.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=1]Paper_MUG.pdf Qualitative results for MUG dataset. The images above and below are parts of generated video frames created from the same text description. Generated video frames are well matched with given text description.

The KTH Action dataset [25] contains six types of human actions: walking, jogging, running, boxing, hand waving, and hand clapping. Of these, we use the jogging class. In each clip, a man is jogging from left to right or right to left on two backgrounds. We extracted 16 frames from each video sequence and reshaped it into $128\times 128$. For training, a total of 200 videos (3,200 frames) are used. Our qualitative results are shown in Figure 1. We can see that the person in the generated video is moving exactly as described in the text description while maintaining the image quality with high resolution.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=1]Paper_Kinetics_edit_ECCV.pdf Example results of text-to-video generation trained on Kinetics dataset.

MUG is a human facial expression database [27]. There are tens of people in the dataset, and each person shows seven types of facial expressions: ‘happy’, ‘disgust’, ‘sad’, ‘neutral’, ‘surprise’, ‘fear’, and ‘anger’. We reshaped and used $128\times 128$ resolution frames, and the models are trained on a total of $1,030$ videos. Result images on the MUG dataset and their given captions are shown in Figure 1. We observe that our model generates sharp images corresponding to the given text.

Kinetics is a large-scale, human-focused video dataset from YouTube [29]. The dataset comprises thousands of video URLs covering $600$ human action classes. We used six classes from this dataset: ‘snow bike’, ‘swimming’, ‘sailing’, ‘golf’, ‘kite surfing’, and ‘water ski’, which are similar to those used in previous works  [21]. We reshaped and used a $128\times 128$ frame size, and every model is trained on a total of $3,032$ videos. For Kinetics, the recent text-to-video generation method proposed by Li et al.(T2V) [21] is also compared qualitatively.

The generated qualitative results are shown in Figure IV-A. From the results, we can easily see that our method produces a much higher-quality video. Our generated images are much clearer because of their higher resolution, and they can also capture more distinctive features of given text, apart from the resolution differences.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.99]Figure_NN.pdf Ablation study on nearest neighbors. Left images are generated samples, and right images are corresponding nearest neighbors in training dataset.

To address that our model does not simply memorize the dataset, we present the nearest neighbor image in Figure IV. We can observe that our generated results are different from the nearest neighbors in the training set.

When we aim to create $2^{n}$ frames of video, our network starts with generating a single frame ($n=0$) and gradually double the number of images to create ($n=1,2,3,4$). To show the advantage of our step-by-step evolutionary generation framework, we perform an ablation study with various cases. Several steps are omitted in the comparison experiment, but total training time in all cases is same for fair comparison.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.99]Abl_init_new_2.pdf Ablation study on step discriminator initialization. Training loss is used as a measure to demonstrate the effectiveness of step-discriminator initialization compared to random initialization. The points where the blue line rises abruptly indicates the time when the step changes.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.99]access_abl_isp.pdf Ablation study on Independent Samples Pairing. Two different video clips are independently generated from one text input ‘swimming in the swimming pool’ with different noises. The left (with ISP) generated a completely different but appropriate video following the text description. However, without ISP, two samples are identically generated even with the different input noise.

In Sec III-C, we described our step discriminator initialization as our training strategy to enhance the network. In this experiment, we record the training loss of $D_{S}$ for the cases with and without our initialization. The results are shown in Figure V. With random noisy initialization, $D_{S}$ shows an unstable loss graph at the beginning of every step. Since random initialization does not utilize the previous learning, the loss rises rapidly when the discriminator is newly added. The step-discriminator initialization indicates that $D_{S}$ is not affected by step change. This means that the model can reliably handle the generation of a larger number of images owing to the suggested initialization for the step-discriminator.

We employ Independent Samples Pairing to prevent the mode-collapse of the generator. The effects of ISP can be visualized in Figure V. Without ISP, the generator often produces identical outputs when the same input text with different noise are given. However, we verify that our network generates various videos when the same text description and different random noise are given.