Research map / Video / Video generation
TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TI2V-Zero, a tuning-free zero-shot method that turns a frozen pretrained text-to-video diffusion model (ModelScopeT2V) into a text-conditioned image-to-video generator via repeat-and-slide frame-by-frame sampling, DDPM inversion, and resampling.
Problem
Text-conditioned image-to-video (TI2V) generation normally requires costly training on video-text data and specialized architectures for image and text conditioning; the paper asks whether a pretrained T2V diffusion model can accept an image condition with no training.
Approach
Guide the reverse denoising of a frozen T2V diffusion model by replacing the first K frame latents with noised copies of the given image (repeated and slid frame-by-frame), with DDPM-inversion initial noise and resampling to keep temporal consistency and visual details.
Key results
- On MUG, TI2V-Zero with resampling reaches FVD 180.09 versus 1094.72 for DynamiCrafter.
- On UCF101, TI2V-Zero with resampling reaches FVD 477.19 versus 589.59 for DynamiCrafter.
- Ablation on MUG shows DDPM inversion lowers FVD from 1656.37 to 339.89, and resampling further lowers it to 180.09 with 4 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TI2V-Zero instead of DynamiCrafter | TI2V generation · MUG | FVD: 1094.72 -> 180.09 | Helped |
| Use TI2V-Zero instead of DynamiCrafter | TI2V generation · UCF101 | FVD: 589.59 -> 477.19 | Helped |
| Add DDPM-based inversion (ablation) | TI2V-Zero on MUG · MUG | FVD: 1656.37 -> 339.89 | Helped |
| Increase DDIM sampling steps from 10 to 50 (ablation) | TI2V-Zero on MUG · MUG | FVD: 339.89 -> 463.55 | Hurt |
| Add 2 resampling steps (ablation) | TI2V-Zero on MUG · MUG | FVD: 339.89 -> 207.62 | Helped |
| Increase resampling steps from 2 to 4 (ablation) | TI2V-Zero on MUG · MUG | FVD: 207.62 -> 180.09 | Helped |
| Generate from synthesized rather than real starting frames (ablation) | TI2V-Zero on UCF101 · UCF101 | FVD: 477.19 vs. 366.41 | Mixed |
Surprising findings
- The naive replacing-based baseline that injects the image latent into all frames fails to keep temporal consistency with the given first frame, motivating the frame-by-frame repeat-and-slide design.
- Starting from synthesized frames (TI2V-Zero-Fake) yields better FVD against real videos than starting from real frames, attributed to in-distribution backbone-generated frames.
Limitations
- Generation quality is bounded by the capabilities of the pretrained T2V backbone.
- Outputs sometimes are blurry or show flickering artifacts.
- Slow inference since the full diffusion process runs for each new frame.
Experimental setup
Datasets: MUG UCF101 OPEN (self-constructed, 1,000 Stable Diffusion images and 10 ChatGPT-generated prompts)
Benchmarks: MUG UCF101 OPEN
Metrics: FVD sFVD tFVD
Baselines: DynamiCrafter Replacing-based baseline ModelScopeT2V
Builds on: ModelScopeT2V DDPM classifier-free guidance DDIM RePaint-style resampling
Closest papers by meaning
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.90)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation (AAAI 2025 · similarity 0.89)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.