Research map / Video / Video generation
Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVS, a training-free encapsulated composition of a T2I model (epiCRealism) and a T2V model (AnimateDiff-Lightning) that refines video frames with a T2V temporal-only prior injected via DDIM inversion and selective feature injection, improving imaging quality and motion smoothness with 1.6x-4.5x speedup.
Problem
Text-to-video diffusion models struggle to jointly achieve high imaging quality and motion smoothness; naively applying T2I models frame-by-frame causes flicker, while explicit flow/correspondence consistency methods break on model-generated videos.
Approach
Encapsulate a T2V temporal-only stabilization block (T2I+[T2V]+T2I) inside the T2I denoising trajectory, bridging latents via predicted clean latents at a shared timestep, and use DDIM inversion with selective feature injection (K, V injected, Q blended) so that only the temporal prior of the T2V model is transferred.
Key results
- EVS achieves overall best quality on VBench subsets, e.g., on VC2 videos Imaging AP 5.46 and DOVER 73.20 with MS 0.9881 and SC 0.9808, versus TokenFlow MS 0.9696.
- EVS reports 1.6x-4.5x inference speedup over compared composition baselines (e.g., 120.86s vs. 546.94s for TokenFlow on VC2).
- Ablation shows T2I+[T2V]+T2I balances consistency (MS 0.9881) and imaging quality (AP 5.46) better than T2I+T2V or T2V+T2I.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply EVS composition of T2I and T2V | VideoCrafter-2.0 videos · VBench Overall Consistency | Overall score: 0.6435 -> 0.8545 | Helped |
| Apply EVS composition of T2I and T2V | AnimateDiff-V2 videos · VBench Overall Consistency | Overall score: 0.5806 -> 0.8243 | Helped |
| Use EVS instead of baseline composition methods | VideoCrafter-2.0 videos · VBench Overall Consistency | Inference time: 1.6x-4.5x speedup | Helped |
| Use frame-wise T2I only instead of EVS (ablation) | VideoCrafter-2.0 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7567 | Hurt |
| Use T2V only instead of EVS (ablation) | VideoCrafter-2.0 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7655 | Hurt |
| Apply T2I, then T2V instead of encapsulating T2V within T2I (ablation) | VideoCrafter-2.0 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7707 | Hurt |
| Apply T2V, then T2I instead of encapsulating T2V within T2I (ablation) | VideoCrafter-2.0 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.8474 | Hurt |
| Use frame-wise T2I only instead of EVS (ablation) | AnimateDiff-V2 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7567 | Hurt |
| Use T2V only instead of EVS (ablation) | AnimateDiff-V2 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7655 | Hurt |
| Apply T2I, then T2V instead of encapsulating T2V within T2I (ablation) | AnimateDiff-V2 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.7707 | Hurt |
| Apply T2V, then T2I instead of encapsulating T2V within T2I (ablation) | AnimateDiff-V2 videos · VBench Overall Consistency | Overall score: 0.8545 -> 0.8474 | Hurt |
| Increase T2I insertion timestep tT2V (ablation) | EVS · VBench video generation | Imaging quality | Helped |
Surprising findings
- Pure T2V-based refinement gives the highest consistency but T2I-based refinement gives the highest imaging quality; the basic two-stage compositions tend to favor whichever model is applied later.
Limitations
- Relies on hand-tuned hyperparameters (tI, tV, tT2V, nV) and an empirical choice of SFI layers and blending rate gamma.
- Evaluated on a single 186-video subset of VBench with two base T2V models.
- DDIM inversion with SFI is only optional for challenging cases, so the full pipeline is not always used.
Experimental setup
Datasets: VBench Overall Consistency subset (videos generated by VideoCrafter-2.0 and AnimateDiff-V2)
Benchmarks: VBench
Metrics: Motion Smoothness Subject Consistency DOVER Aesthetic Predictor V2.5 inference time PSNR
Baselines: Rerender-A-Video FRESCO TokenFlow BIVDiff AnyV2V
Builds on: SDEdit DDIM inversion TokenFlow Rerender-A-Video BIVDiff
Closest papers by meaning
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.91)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.90)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.89)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.