POS: A Prompts Optimization Suite for Augmenting Text-to-Video Generation

Video Retrieval seeks to find a video $v*$ that aligns with the text prompt in semantics. We first prepare a pool of $N$ text-video pairs $\mathcal{X}=\{S_{i},V_{i}\}_{i=1}^{N}$, where $S$ and $V$ represent text and video, respectively. Given a text prompt $\mathcal{C}$, we initially estimate the similarity between the text prompt and the text within $\mathcal{X}$ and then select the video accordingly:

where $\text{sim}(\cdot,\cdot)$ is the cosine similarity, and $E_{t}$ can be a off-the-shelf language model like [27, 3], to extract the text feature. The reasons we anchor on the text-similarity for the video selection are three-fold: first, text similarity is a more reliable clue than the text-video relevance; second, the computation efficiency for text feature extraction is superior; third, due to the available text-video dataset like WebVid-10M [1], InvernVid [40], collecting the text-video pairs is not effort-intensive as well.

Guided Noise Inversion module takes the searched $v*$ as the inversion source to approximate the optimal noise. Formally, we follow the efficient latent diffusion architecture [28], where the raw video $v*$ is first encoded via the encoder of VQ-VAE [26] and then the inversion is recurrently acted on the latent feature with $T$ steps: $\epsilon_{\text{inv}}=\text{INV}(\mathcal{E}(v*),t,\varnothing)|_{t=0}^{T-1}$, where $\epsilon_{\text{inv}}$ is the inverted noise, $\varnothing$ represents the empty text and $\mathcal{E}$ is the VQ-VAE encoder. With the video as the inversion source, $\epsilon_{\text{inv}}$ can also inherit the coherence of $v*$, thereby benefiting the temporal consistency as well.

Video Synthesize with Improved Noise. Given the improved noise $\epsilon_{\text{mix}}$ and text prompt $\mathcal{C}$, the latent code $z_{0}$ of the video is calculated by the following formula:

where $T$ is the total DDIM sampling steps.

Reference-Guided Rewriting aims to provide descriptions to guide the rewriting, allowing the LLMs to “imagine” reasonable textual details. In particular, we first use the Sentence-BERT model [27] to encode the text prompt. Consider a text prompt $\mathcal{C}$, its top-$k$ references are selected using the cosine similarity:

Denoising with Hybrid Semantics. Reference sentences provide valuable guidance for the reasonable details compensation, however, we found this strategy cannot perfectly maintain the original semantics of the text prompt due to the excellent association ability of LLMs. To remedy this issue, we propose to introduce the original text prompt into the denoising process. Specifically, we apply the rewritten sentence as the condition in the early stage to boost the content quality, while the original text prompt is employed in the latter denoising steps to pull the semantics close to the original prompt. As a result, we evolve the video synthesis of Eq. 5 as follows:

where $m=\lfloor T\times\gamma\rfloor,\gamma\in(0,1).$ $m$ indicates how many steps of rewritten text is performed, $\lfloor\cdot\rfloor$ is the floor operation. Following the latent diffusion model [28], we finally synthesize the video by feeding the final latent feature $z_{0}$ into the VQ-VAE decoder: $\hat{v}=\mathcal{D}(z_{0})$.

ONA vs Inversion in Video Editing. Video Editing (VE) approaches often invert the video for editing to initialize the noise  [42, 17, 24]. ONA differs from VE methods in many aspects. (1) Different goals. VE inverts the video to preserve more information about the original video, thereby better supporting the subsequent editing. While ONA uses the inversion to approach the potential optimal noise, which is the core idea of this paper. (2) Video diversity. Video Editing (VE) does not require much consideration of diversity. While our focus is on the general text-to-video generation, consequently, ONA has to maintain the diversity, which is accomplished via the Gaussian mixture. (3) Inversion source. The video for editing is the straightforward inversion source in VE. In contrast, the video for the optimal noise is not available during inference, therefore, ONA resorts to searching a neighbor video as the inversion source.

SPR vs Text Prompt-Enhancement. Prior research has employed LLMs to enhance text prompts for improving text-to-vision generation [47, 5, 12, 21]. However, there approaches either introduce extra training burden [5], or simply utilize the frozen LLMs to provide additional textual details [21], but rarely study how to better unlock the potential of LLMs when rewriting and pay rare attention to the alignment between the original text and the videos condition on the rewritten text [47, 12]. While SPR is optimization-free, and RGR in our SPR assists LLMs by showing several sentences as references, and DHS in SPR presents a hybrid denoising strategy to keep the semantics of the final video consistent with the original text. These features set our SPR apart from existing text prompt-rewriting methods.

Datasets & Metrics. We follow the previous works [11, 30, 4, 2, 39] and adopt the widely-used MSR-VTT [43] and UCF101 [32] datasets for performance evaluation. MSR-VTT provides 2,990 video clips for testing, each accompanied by around 20 captions. To evaluate the performance, we randomly select one caption per video to create a set of 2,990 text-video evaluation pairs. FID [8]¹¹1FID: https://github.com/GaParmar/clean-fid and CLIP-FID [15]²²2CLIP-FID: https://github.com/GaParmar/clean-fid is used to assess the video quality, along with CLIP-SIM [41]³³3CLIPSIM: https://github.com/openai/CLIP metric to measure semantic consistency between videos and text prompts. UCF101 contains 13,320 video clips of 101 human action categories. We evaluate performance using 3,783 test videos. As there are no captions, we use the text prompts from PYoCo [4] for video generation. IS [29] and FVD [35, 44] serve as the evaluation criteria. Video IS [30, 13] uses C3D[34] pretrained on UCF101 as the feature extractor⁴⁴4IS: https://github.com/pfnet-research/tgan2, while FVD utilizes I3D [36] pretrained on Kinetics-400 [14] for video feature encoding⁵⁵5FVD: https://github.com/wilson1yan/VideoGPT.

Resolution. Following previous works [30, 46, 4, 6], we generate videos of size $16\times 256\times 256$ for performance evaluation. This choice is motivated by two main factors: First, the training dataset WebVid-10M [1] for most text-to-video models is dominated by 360P videos; Second, both the MSR-VTT and UCF101 datasets consist of videos with a resolution of around 240P. Consequently, $256\times 256$ ensures a consistent and comparable evaluation setting.

Sampling Strategy. For FID and CLIP-FID, we sample 14,950 images from 2,990 MSR-VTT videos and generated videos respectively. Concretely, we sample 5 images for each video in both real and generated videos. For the real video, sample an image every 12 frames. For the generated video, sample an image every 4 frames. Regarding CLIP-SMI, we calculate the clip similarity between each image of the generated video and the corresponding prompt and then average it. All 3,783 UCF101 test videos are used to calculate FVD and IS. For the real video, we sample an image every 5 frames, for a total of 16 frames, to calculate FVD.

Training. Both SCVideo and the Noise Prediction Network employ the same UNet architecture and identical loss functions. The difference lies in the initialization of network parameters. SCVideo is initialized using the pretrained weights of SD 1.4, and the newly introduced weights and bias are initialized with identity matrix and zero, respectively. NPNet is initialized from the parameters of a well-trained SCVideo. Additionally, we only optimize the 3D temporal convolution and all the attention modules of the former, whereas the latter undergoes training for all parameters. SCVideo is trained with 4 Nvidia A100-80G GPUS with batch size 5, and the total number of training steps per GPU is 80K. The learning rate is set as $3\times 10^{-5}$ with $10^{-2}$weight decay. We train our model using AdamW optimizer, and the hyperparameters are configured as $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999 and $\epsilon$ = $10^{-8}$. In addition, a cosine schedule with a linear warmup of 1000 steps is adopted during the training process. NPNet is trained with 2 Nvidia A100-80G GPUS with batch size 5, and the total number of training steps per GPU is 40K. Other hyperparameter settings are the same as SCVideo.

Can POS Benefit Image Generation? To answer this question, we augment the Stable Diffusion (SD) v1.5 [28] and Stable Diffusion-XL (SD-XL) [23] with our POS and evaluate their performance on MS-COCO test set [16]. We employ the test set of Flickr30k [45] as the candidate pool instead of the MS-COCO training set to verify the generalization of POS, and $\eta$ and $\gamma$ are set as 0.05 and 0.4, the other hyperparameters remain the same as video experiments.

Size of Candidate Pool. To investigate the effect of candidate pool size, we randomly sample 10K, 100K, 1M, and 10M samples from the Webvid dataset to study the performance trend. From Table LABEL:tab:poolDis, we can observe that the larger the retrieval pool typically results in better performance. For example, enlarging the pool size from 10K to 10M can promote the FID on MSR-VTT from 43.236 to 42.072. Notably, although our default set is $N=$100K, reducing the scale to 10K does not severely hinder the performance. We take 100K to pursue a better trade-off between the scale of the candidate pool and the performance.

Proportion of Guided Noise in ONA. Hyperparameter $\eta$ in Eq. 4 determines the final composition of the noise, this part discusses the performance impact of how to hybridze the two types of noises by varying $\eta$. The results are reported in Figure 6, We can observe a clear tendency from the figure, that using the pure random noise ($\eta=0$) or guided noise ($\eta=\infty$) can not yield satisfactory performance. Instead, an appropriate fusion of the two noise types is a more effective manner, we can harvest the best performance when $\eta=0.5$.

Number of Dominated Steps of Rewritten Text in DHS. The hyperparameter $\gamma$ controls how many steps of the rewritten text are performed in DHS. We vary $\gamma$ in this part and study its effect on performance, and do not equip the ONA to observe a clearer trend. As shown in Table LABEL:tab:diffStep, the rewritten sentence ($\gamma=1$) can boost the quality compared to the raw prompt. Specifically, compared with the baseline($\gamma=0$ without rewritten text), applying the rewritten sentence reduces FID from 48.331 to 43.531 and promotes the IS score from 23.224 to 25.648. However, the CLIP-SIM drops from 0.283 to 0.275, indicating the semantic consistency is deteriorated. Our DHS strategy can remedy the issue, for example, introducing the original text in the last 90% denoising steps ($\gamma=0.1$) can boost the CLIP-SIM from 0.275 to 0.284.

Number of Reference Text for Rewriting. By default, we employ 5 reference sentences for the text rewriting, this subsection examines the impact of varying the number of references for text rewriting. Table LABEL:tab:numTextDis presents the performance results across different reference numbers, highlighting the positive impact of incorporating references (we focus solely on the rewriting strategy, excluding the ONA and DHS techniques). For instance, the FID score is significantly improved from 45.435 without references to 43.531 with five references. Despite consistent improvements in other metrics, CLIP-SIM shows a decline due to the introduction of new content or objects. However, this issue can be effectively mitigated by our DHS mechanism, as evidenced in Table 1 and Table LABEL:tab:diffStep.

Effect of Large Language Models (LLMs). ChatGPT is taken as the rewriting engine due to its excellent performance, we also present a discussion regarding the LLMs to further validate our methods. Table LABEL:tab:llms compares the results of using ChatGPT (GPT-3.5-Turbo) and Llama2-7B in SPR. The findings indicate that ChatGPT consistently outperforms Llama2-7B across five key metrics, exhibiting a substantial advantage. Moreover, ChatGPT also shows a higher level of intelligence in our practice, as it does not need regularization to filter out irrelevant content.