TVG: A Training-free Transition Video Generation Method
with Diffusion Models

Diffusion models (Sohl-Dickstein et al. 2015; Ho, Jain, and Abbeel 2020; Rombach et al. 2022) are a class of probabilistic generative models which learns a data distribution by denoising noisy. The forward process of diffuses the data samples with a Markov process contains $T$ timesteps: The forward process is gradually adding noise to the data sample $x_{0}$ to $x_{t}$ with a Markov process contains $T$ timesteps:

	$\displaystyle q(\boldsymbol{x}_{t}|\boldsymbol{x}_{t-1})=\mathcal{N}\bigl{(}\boldsymbol{x}_{t};\sqrt{1-\beta_{t}}\boldsymbol{x}_{t-1},\beta_{t}\boldsymbol{I}),$		(1a)
	$\displaystyle q(\boldsymbol{x}_{t}|\boldsymbol{x}_{0})=\mathcal{N}\bigl{(}\boldsymbol{x}_{t};\sqrt{\overline{\alpha}_{t}}\boldsymbol{x}_{0},(1-\overline{\alpha}_{t})I\bigl{)},$		(1b)

where $\beta_{t}$ is a predefined variance schedule, $\overline{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$, is derived from the variance noise schedule, and $\alpha_{t}=1-\beta_{t}$. The reverse process begins from the noise and transitions towards the original data $q(\boldsymbol{x}_{0})$. The reverse denoising process obtains less noisy data $x_{t-1}$ from the noisy input $x_{t}$ at each timestep:

$\displaystyle p_{{{\theta}_{l}}}(\boldsymbol{x}_{t-1}|\boldsymbol{x}_{t})=\mathcal{N}\bigl{(}\boldsymbol{x}_{t};\frac{1}{\sqrt{\alpha_{t}}}(\boldsymbol{x}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}}_{t}}\epsilon_{{{\theta}_{l}}}(\boldsymbol{x}_{t},t)),\sigma^{2}_{t})$

(2)

where $\epsilon_{{\theta}_{l}}(\boldsymbol{x}_{t},t)$ is the denoising function to estimate the noise added at each step and ${{\theta}_{l}}$ is the learnable network parameters.

Given two input images, $x_{0}$ and $x_{S}$, the goal of transition video generation is to produce a video sequence $X_{t}=\{x_{i}\}_{i=1}^{S}$, starting with $x_{0}$ and ending with $x_{S}$. The intermediate frames, $x_{1}$ to $x_{S-1}$, should exhibit smooth transitions that blend the characteristics of both input images, forming a coherent sequence. This process can be formalized as a conditional distribution $p_{{\theta}_{l}}(X_{t}\mid x_{0},x_{S})$, modeled by a conditional diffusion process.

In video generation, Latent Diffusion Models (LDMs) (Rombach et al. 2022; Zhou et al. 2022; An et al. 2023) are commonly used to reduce computational complexity by fitting the conditional distribution within the latent space, where denoising occurs. The initial and final frames, $x_{0}$ and $x_{S}$, anchor the sequence, with intermediate frames initialized by adding Gaussian noise. These frames are encoded into a latent representation $z_{t}\in\mathbb{R}^{S\times C\times H\times W}$, where $S$, $C$, $H$, and $W$ represent the number of frames, channels, height, and width, respectively. A backward denoising process, such as DDIM (Song, Meng, and Ermon 2021), is applied to obtain the clean latent representation $z_{0}$, which is then decoded into the video sequence $X_{t}$. The reverse process typically incorporates additional denoising conditions to enhance the quality of the generated video, such as text prompt latents, $c_{\text{txt}}$, and the latent of the initial frame, $c_{\text{img}}$.

Our base model, DynamiCrafter (Xing et al. 2023), further leverages FPS as a denoising condition. It uses CLIP (Radford et al. 2021) to encode text prompts and the initial frame, while employing a frame-wise Variational Auto-Encoder (VAE) architecture for the video encoder and decoder. The denoising process is carried out using a U-Net architecture.

A Gaussian process ($\mathcal{GP}$) (Williams and Rasmussen 2006) is a collection of random variables, any finite number of which have a joint Gaussian distribution. Consider a data set with inputs and targets consisting of $N$ data-points as $\{x_{i},y_{i}\}_{i=1}^{N}$, where the inputs are denoted by $\boldsymbol{X}$ =$\{x_{1},...,x_{N}\}$ , and targets by $\boldsymbol{Y}$ =$\{y_{1},...,y_{N}\}$. The goal of $\mathcal{GP}$ is to learn an unknown function $f$ that maps elements from input space to a target space, which can provide predictive distributions on test points $\boldsymbol{X^{*}}$ =$\{x^{*}_{1},...,x^{*}_{M}\}$. Assuming the GP prior on $f(x)$ with some additive Gaussian white noise with variance $\sigma^{2}$:

$\displaystyle y_{i}=f(x_{i})+\epsilon_{i},\epsilon_{i}\sim\mathcal{N}(0,\sigma^{2})$

(3)

A Gaussian process regression ($\mathcal{GPR}$) formulates the functional form of $f$ by drawing random variables from a multi-variate normal distribution given by $[f(x_{1}),f(x_{2}),...,f(x_{n})]\sim\mathcal{N}(\mu,\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X}})$, with the mean of $\mu$ and the covariance matrix of $\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X}}$. $\mu_{i}=\mu(x_{i})$, where $\mu(\cdot)$ is a mean function of the $\mathcal{GP}$; $(\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X}})_{ij}=k(x_{i},x_{j})$, where $k(\cdot)$ is a kernel function of the $\mathcal{GP}$. The conditional distribution at any unseen points $\boldsymbol{X}^{*}$ is given by:

$\displaystyle\boldsymbol{f^{*}}|\boldsymbol{X^{*}},\boldsymbol{X},\boldsymbol{Y}\sim\mathcal{N}(\mu_{\boldsymbol{f^{*}}},\Sigma_{\boldsymbol{f^{*}}})$

(4)

where

$\displaystyle\mu_{\boldsymbol{f^{*}}}=\mu\boldsymbol{X^{*}}+\boldsymbol{K}_{\boldsymbol{X^{*}},\boldsymbol{X}}[\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X}}+\sigma^{2}\boldsymbol{I}]^{-1}\boldsymbol{Y}$

(5)

$\displaystyle\Sigma_{\boldsymbol{f^{*}}}=\boldsymbol{K}_{\boldsymbol{X^{*}},\boldsymbol{X^{*}}}-\boldsymbol{K}_{\boldsymbol{X^{*}},\boldsymbol{X}}[\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X}}+\sigma^{2}\boldsymbol{I}]^{-1}\boldsymbol{K}_{\boldsymbol{X},\boldsymbol{X^{*}}}$

(6)

In our experiments, we found that DynamiCrafter often produced abrupt transitions when using input images with significantly different content, a result of conditional image leakage. To mitigate this, we applied linear interpolation to blend the two input frames $x_{0}$ and $x_{S}$:

$\displaystyle x_{b}=\beta\cdot x_{0}+(1-\beta)\cdot x_{S}.$

(7)

Setting $\beta$ to 0.5 balances information from both frames, creating a blended image $x_{b}$ that integrates details from both inputs. Other values would favor one frame over the other, which is not suitable for our video transition task. This image is then processed by CLIP to extract feature embeddings, serving as the condition $c_{img}$. This approach leverages the pre-trained model without requiring additional training. In contrast, separate extraction and fusion of features using CLIP can create a semantic gap, complicating their integration in diffusion models and often leading to errors. Addressing this gap usually requires retraining a feature projection module, as done in StoryDiffusion (Zhou et al. 2024), but our method avoids this by fully utilizing the pre-trained model.

Traditional methods typically employ a single prompt for the entire video, limiting frame-wise control. To improve this, inspired by (Zhang et al. 2024a; Jang et al. 2024), we propose Temporal-Level Prompts Control. We begin by obtaining prompts $t_{0}$ and $t_{S}$ for the initial and final frames, then use CLIP to extract their semantic features, $F^{0}_{\text{txt}}$ and $F^{S}_{\text{txt}}$. These features are interpolated using spherical linear interpolation (SLERP).

$\displaystyle F^{s}_{\text{txt}}=\frac{\sin{((1-\alpha)\theta)}}{\sin{\theta}}\cdot F^{0}_{\text{txt}}+\frac{\sin{(\alpha\theta)}}{\sin{\theta}}\cdot F^{S}_{\text{txt}},$

(8)

where $\alpha\in[0,1]$ is the interpolation parameter, and $\theta=\cos^{-1}\left(\frac{F^{0}_{\text{txt}}\cdot F^{S}_{\text{txt}}}{\|F^{0}_{\text{txt}}\|\|F^{S}_{\text{txt}}\|}\right)$ is the angle between the two vectors. We sample $S$ weights uniformly between 0.9 and 0.1, corresponding to the temporal progression from the initial to the final frame, creating $S$ interpolated semantic features. These features form $c_{\text{txt}}\in\mathbb{R}^{S\times L\times D}$, ensuring smooth transitions and minimizing semantic gaps, allowing for seamless integration into the diffusion model and reducing the likelihood of abrupt content changes in the video.

The denoising U-Net employs temporal-level and spatial-level Transformer modules, integrating self-attention and cross-attention to capture spatial relationships within frames and temporal relationships between frames. However, the spatial modules overlook inter-frame dependencies, while the temporal modules impose only limited inter-frame constraints through attention mechanisms. This results in poor controllability, leading to error propagation across frames and degrading the quality of transition videos.

To address these issues, we incorporate $\mathcal{GPR}$ into the latent space of the U-Net within video generation models. This integration with attention modules explicitly constrains inter-frame relationships, enhancing the overall video quality.

Given a temporal-level latent vector $z\in\mathbb{R}^{S\times N\times P}$, where $N$ represents the vector size and $P$ denotes the number of channels, we compute the attention mechanism with $\mathcal{GPR}$ for both temporal and spatial attention as follows:

$\displaystyle z^{\prime}=\text{Attention}(z)+\gamma\cdot z+(1-\gamma)\cdot\mathcal{GPR}(z),$

(9)

where $\gamma\in[0,1]$ serves as a balancing factor, controlling the trade-off between different feature contributions. We hypothesize that the transition content between two frames in a video can be modeled as a smooth stochastic process. The objective is to derive a distribution function for this process, with the first frame as the observation input and the last frame as the prediction target.

In the latent space, the features of the first and last frames are extracted from the input images, while intermediate frames are generated by denoising Gaussian noise. This allows us to construct a stochastic process distribution function based on the initial and final frames, predicting intermediate frames to ensure coherent transitions that accurately reflect the relationship between the start and end frames.

To implement this, we leverage $\mathcal{GPR}$ to model the distribution function, utilizing its capability to flexibly capture the underlying data distribution and smoothly interpolate between frames while maintaining consistency with the input frames. Specifically, the first frame of $z$ is defined as $X\in\mathbb{R}^{N\times P},$ where $X$ consists of $N$ data points serving as input to $\mathcal{GPR}$, as outlined in the Preliminary section. Similarly, the last frame of $z$ is defined as $Y\in\mathbb{R}^{N\times P},$ representing the target with $N$ data points. We employ the Radial Basis Function as the kernel $k(\cdot)$ to formulate the distribution function $\boldsymbol{f^{*}}.$ The remaining frames of $z$ are used as the input $\boldsymbol{X^{*}}$ to the function, as shown in Equation 4. The mean $\mu_{\boldsymbol{f^{*}}}$ obtained from the function, as shown in Equation 5, is then used to refine the corresponding frame features.

This approach effectively maps features into a Gaussian space, aligning inter-frame relationships with a stationary random distribution, thereby enhancing both correlation and smoothness between frames. However, due to inherent differences between the original model’s feature distribution and the Gaussian distribution, a complete substitution of the original features would necessitate further training to achieve optimal performance. To reduce computational costs and avoid additional training, we integrate these features as a penalty term within the original attention mechanism, rather than fully replacing the original features. Despite this conservative approach, it significantly enhances inter-frame information exchange and smooths transitions between frames, while maintaining compatibility with the feature distribution of the pre-trained model.

In practice, we observed significant quality differences in transition videos generated from two images, $x_{0}$ and $x_{S}$, depending on whether the sequence was processed in original or reverse order. As shown in Figure 3, this issue is linked to conditional image leakage, where the transition video tends to resemble the initial image more closely. To address the shortcomings in the latter part of the video, particularly the deviations from the final frame and the lack of smoothness, we propose a Frequency-aware Bidirectional Fusion (FBiF) approach to improve overall generation quality.

As illustrated in Figure 2, we generate a forward transition video, then repeat the process by reversing the order of the input images and prompts fed into the video generation model. In the latent space, we apply $\mathcal{GPR}$ to map features into a unified Gaussian space, aligning the feature distributions of both the forward and reversed videos. Let $z_{g}=\mathcal{GPR}(z)$, where $z_{g}\in\mathbb{R}^{S\times N\times P}$; the reversed features are similarly defined as $z_{g}^{rev}\in\mathbb{R}^{S\times N\times P}$. Inspired by (Pan, Cai, and Zhuang 2022), we consider the average-pooled (avgpool) features to represent low-frequency components, while the max-pooled (maxpool) features capture high-frequency components. Since our goal is to emphasize the primary concepts in the video content rather than merely enhancing details, we employ a frequency-aware fusion method, combining the two feature sets along the temporal sequence as follows:

	$\displaystyle z_{fused}^{s}=$	$\displaystyle\ \lambda_{s}\cdot\text{avgpool}(z_{g}^{s})+(1-\lambda_{s})\cdot\text{avgpool}(\text{reverse}(z_{g}^{rev})^{s})$		(10)
		$\displaystyle+\lambda_{freq}\cdot\text{maxpool}(z_{g}^{s})+\lambda_{freq}\cdot\text{maxpool}(\text{reverse}(z_{g}^{rev})^{s}).$

We implemented our Transition Video Generation approach using the pre-trained DynamiCrafter model (Xing et al. 2023). Most hyperparameters were kept at default settings, except for the FPS, which was set to 16 to enhance visual dynamics. The hyperparameter $\gamma$ was set to 0.9, $\lambda_{freq}$ was 0.1. We used a DDIM (Song, Meng, and Ermon 2021) schedule with 10 steps during denoising. All experiments ran on a single NVIDIA A800 GPU.

For quantitative evaluation, we tested the MorphBench (Zhang et al. 2024a) and TC-Bench-I2V (Feng et al. 2024) datasets, generating 16 frames per video following established protocols (Chen et al. 2024; Zhang et al. 2024a; Feng et al. 2024). The video resolution was set to $512\times 512$ for MorphBench and $320\times 512$ for TC-Bench-I2V. For qualitative evaluation, we collected 10 image pairs from the Internet for comparison with other models and commercial products.

To demonstrate the effectiveness of our algorithm, we have presented some of the results generated by our model in Figure 4. It can be seen that for different scenarios, our model exhibits strong video-transition capabilities, with the video maintaining dynamic elements while the content gradually transitions. Additionally, we compared our algorithm with several commercial products as shown, some of which were even unable to produce transition videos as shown in Figure 1. For more visualization results, we recommend readers refer to the supplementary materials.

Evaluation on TC-Bench-I2V: We evaluated our model on the TC-Bench-I2V dataset using the Transition Completion Ratio (TCR, $\uparrow$) and TC-Score ($\uparrow$) (Feng et al. 2024). These metrics utilize GPT-4 to provide assessments closely aligned with human perception. TCR measures the percentage of videos that successfully match the provided prompts, while TC-Score offers a comprehensive evaluation considering factors such as transition completion, object consistency, and handling of additional objects, reflecting user experiences more realistically.

Table 1 presents a comparison of our algorithm with three state-of-the-art methods across three video transition scenarios: Attribute Transition (Attribute), involving changes in color, shape, material, and texture; Object Relation Change (Object), focusing on object interactions like passing or striking; and Background Shifts (Background), such as transitions from day to night.

Our results demonstrate consistent superior performance across most metrics and scenarios. Notably, our method ranks second in TCR for the background scenario, likely due to the lower dynamic demands, which favor DiffMorpher, a frame-by-frame generation-based approach. However, in dynamic scenarios like Object Relation Change, our algorithm significantly outperforms others, aligning more closely with user experience expectations. Figure 5 illustrates comparative examples of generated results, with samples displayed every two frames.

Evaluation on MorphBench: Following previous work (Zhang et al. 2024a), we evaluated the fidelity and smoothness of video content using traditional metrics, specifically Frechet Inception Distance (FID, $\downarrow$) (Heusel et al. 2017) and Perceptual Path Length (PPL, $\downarrow$) (Karras et al. 2020). The results on the MorphBench dataset, summarized in Table 2, show that while our approach does not outperform all existing methods, it remains competitive with current diffusion-based models (Song, Meng, and Ermon 2021; Wang and Golland 2023; Zhang et al. 2024a; Chen et al. 2024; Xing et al. 2023). Notably, even traditional techniques like Warp&Blend surpass various generative algorithms in certain metrics, highlighting that superior quantitative scores do not necessarily translate to better visual effects in video transitions.

Our primary objective is to develop an algorithm that seamlessly transitions between the final frame of one video and the first frame of another, enhancing continuity and fluidity. Although dynamic videos, which are favored for their rich visual appeal (Roggeveen et al. 2015; Steuer et al. 1995; Coyle and Thorson 2001), often introduce greater frame variation that can reduce quantitative performance, the enhanced user experience remains paramount.

Figure 6 illustrates content generated from a pair of images in the MorphBench dataset, using identical prompts across different methods. Our method and DiffMorpher employ separate prompts for each image, while SEINE and DynamiCrafter merge the descriptions into a single prompt, transitioning from input A to input B without introducing additional content. Figure 7 presents the perceptual loss (Zhang et al. 2018) between consecutive frames and final PPL, starting with input A at frame index 0 and ending with input B at frame index 17, covering both input images and generated frames. DiffMorpher achieves the lowest PPL, indicating superior quantitative performance. This is primarily because only a small portion of the generated frames retain perceptual similarity to input A, with the rest transitioning gradually toward input B in a low-dynamic manner, resulting in final frames that closely resemble the target. Although perceptual loss is relatively high during the initial dynamic phase, it remains low thereafter, enhancing DiffMorpher’s quantitative metrics. However, this approach results in a near-static user experience, making it difficult to effectively engage the audience.

In contrast, other methods, including ours, prioritize transforming image characteristics while maintaining high dynamic variability, leading to fluctuating PPL values throughout the sequence and impacting the final quantitative results. While videos with high dynamic variability often appeal more to users, SEINE and DynamiCrafter demonstrate more abrupt transitions. SEINE fully transitions to input B midway through the sequence with minimal change afterward, while DynamiCrafter adopts the style of input B only in the final quarter, lacking the smoothness needed for a seamless transition. In contrast, our method excels at gradually blending specific regions into the style of input B while preserving natural motion, achieving both high dynamic performance and smooth, visually coherent transitions throughout the entire sequence, underscoring the superiority of our approach.

Another factor contributing to the significant perceptual loss between frames in highly dynamic content is the limitation of existing open-source video generation models, which struggle to produce longer video sequences. Typically, these models generate around 16 frames, and content beyond this length often becomes unstable. As our method is training-free, it currently cannot be extended to generate videos with more frames. Ensuring a high dynamic range under this frame limitation inevitably impacts the quality and smoothness of the video. However, this impact is acceptable under current circumstances. If more advanced open-source models for generating longer videos become available, our method should achieve further performance improvements.

Human Evaluation: To assess the real-world user experience of our algorithm, we conducted a Human Evaluation study using 22 curated videos: 9 from MorphBench, 9 from TC-Bench, and 4 from our own dataset. 15 volunteers participated, yielding a total of 990 votes. Participants compared pairs of videos and voted on which appeared more natural and smooth across three scenarios: Ours vs. DiffMorpher, Ours vs. SEINE, and Ours vs. DynamiCrafter. The results, summarized in Table 3, indicate a strong preference for our algorithm, highlighting its effectiveness in generating smoother, more natural transitions.