Research map / Video / Video generation
VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free, plug-and-play VideoElevator decomposes each T2V sampling step into temporal motion refining (temporal low-pass filter, T2V-based SDEdit, DDIM inversion) and spatial quality elevating (inflated T2I), improving ZeroScope, AnimateDiff, and LaVie with foundational and personalized text-to-image models.
Problem
Text-to-video diffusion models lag behind text-to-image models in frame quality and text alignment because video training data is limited and low quality, and low-quality content accumulates over the T2V sampling chain.
Approach
Decompose every sampling step so that a T2V model refines temporal motion (with a temporal-only low-pass frequency filter, SDEdit, and DDIM inversion into the T2I noise distribution), and then an inflated T2I model performs the spatial denoising step, letting any T2V and T2I sharing the same autoencoder be combined without training.
Key results
- On VBench, ZeroScope+SDv2.1-base raises aesthetic score from 0.561 to 0.618 and frame quality from 0.517 to 0.593 versus ZeroScope alone.
- On VideoCreation with personalized T2I, ZeroScope+SD-LoRA raises domain consistency from 0.700 to 0.871 and aesthetic score from 0.578 to 0.671 over ZeroScope.
- In human evaluation, raters preferred VideoElevator over ZeroScope for text alignment (81%) and frame quality (88%), and over AnimateDiff for frame quality (86%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate SDv1.5 with LaVie using VideoElevator | LaVie · VBench | Frame quality: 0.681 -> 0.706 | Helped |
| Integrate SD-LoRA using VideoElevator | ZeroScope · VideoCreation | Domain consistency: 0.700 -> 0.871 | Helped |
| Remove LPFF (ablation) | VideoElevator with ZeroScope and personalized T2I · VideoCreation | Frame consistency: 0.991 -> 0.982 | Hurt |
| Add spatial LPFF to temporal LPFF (ablation) | VideoElevator with ZeroScope and personalized T2I · VideoCreation | Frame quality: 0.684 -> 0.668 | Hurt |
| Replace DDIM inversion with same noise (ablation) | VideoElevator with ZeroScope and personalized T2I · VideoCreation | Frame consistency: 0.991 -> 0.999 | Mixed |
| Replace DDIM inversion with random noise (ablation) | VideoElevator with ZeroScope and personalized T2I · VideoCreation | Frame consistency: 0.991 -> 0.969 | Hurt |
| Increase ZeroScope sampling steps from 50 to 100 (ablation) | ZeroScope · VideoCreation | Frame consistency: 0.981 -> 0.983 | Helped |
| Use VideoElevator instead of conventional ZeroScope at 100 steps (ablation) | ZeroScope · VideoCreation | Frame consistency: 0.983 -> 0.991 | Helped |
Surprising findings
- Using the same noise for all frames during inversion yields higher frame consistency (0.999) but makes all frames identical, so it is not usable.
- Applying the low-pass filter spatially as well as temporally degrades frame quality and aesthetic score, so only the temporal filter is used.
Limitations
- Uses 100 sampling steps versus 50 for baselines, increasing inference cost.
- Evaluation of personalized T2I is limited to 100 VideoCreation prompts and a small human study on three pipelines.
Experimental setup
Datasets: VBench VideoCreation
Benchmarks: VBench VideoCreation
Metrics: frame consistency prompt consistency frame quality (CLIP-IQA) aesthetic score domain similarity user preference
Baselines: ZeroScope AnimateDiff LaVie AnimateDiff XL Stable Diffusion LoRA personalized AnimateDiff
Builds on: Latent Diffusion Model DDIM SDEdit AnimateDiff Tune-A-Video
Closest papers by meaning
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.92)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.91)
- VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.89)
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.