Research map / Video / Video generation
Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies conditional image leakage in image-to-video diffusion models and proposes Analytic-Init and TimeNoise to reduce over-reliance on the conditional image, yielding higher motion scores.
Problem
Image-to-video diffusion models tend to generate videos with less motion than expected because they over-rely on the conditional image at large time steps (conditional image leakage).
Approach
An inference-time strategy that starts sampling from an earlier time step with an analytically optimal Gaussian initial noise (minimizing KL to the true marginal), plus a training-time time-dependent logit-normal noise on the conditional image (TimeNoise) that raises noise levels at large time steps to disrupt leakage.
Key results
- On UCF101, DynamiCrafter motion score rises from 50.96 to 71.04 with Analytic-Init, and FVD drops from 363.8 to 316.3.
- VideoCrafter1 motion score rises from 63.36 to 139.04 with Analytic-Init while FVD improves from 353.9 to 341.6.
- Analytic-Init achieves the best user ranking (1.77) among noise initialization methods versus FreeInit (1.95), at lower inference time (22.6s vs 49.6s per video).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Analytic-Init to DynamiCrafter | DynamiCrafter · UCF101 | Motion Score: 50.96 -> 71.04 | Helped |
| Add TimeNoise and Analytic-Init | DynamiCrafter · UCF101 | FVD: 363.8 -> 332.1 | Helped |
| Add TimeNoise and Analytic-Init | DynamiCrafter · UCF101 | Motion Score: 50.96 -> 73.92 | Helped |
| Add Analytic-Init to VideoCrafter1 | VideoCrafter1 · UCF101 | Motion Score: 63.36 -> 139.04 | Helped |
| Add TimeNoise and Analytic-Init | VideoCrafter1 · UCF101 | FVD: 353.9 -> 443.7 | Hurt |
| Add TimeNoise and Analytic-Init | VideoCrafter1 · UCF101 | Motion Score: 63.36 -> 66.7 | Helped |
| Add TimeNoise and Analytic-Init | SVD · UCF101 | FVD: 388.3 -> 272.4 | Helped |
| Add TimeNoise and Analytic-Init | SVD · UCF101 | Motion Score: 16.64 -> 21.44 | Helped |
| Add Analytic-Init (ablation) | DynamiCrafter naive-tune · UCF101 | FVD: 382.5 -> 342.9 | Helped |
| Combine inference and training strategies rather than using TimeNoise alone (ablation) | DynamiCrafter · UCF101 | FVD: 334.9 -> 332.1 | Helped |
| Combine inference and training strategies rather than using TimeNoise alone (ablation) | VideoCrafter1 · UCF101 | FVD: 452.2 -> 443.7 | Helped |
| Combine inference and training strategies rather than using TimeNoise alone (ablation) | SVD · UCF101 | FVD: 272.2 -> 272.4 | No clear effect |
Surprising findings
- Increasing noise levels via larger P_mean in the EDM framework (as in SVD) reduces motion in generated videos, contrary to the expectation that more noise bridges the training-inference gap.
- Naive fine-tuning with TimeNoise-like constant noise leads to poor image alignment, and excessive TimeNoise on VideoCrafter1 disrupts image alignment.
Limitations
- Balancing the time-dependent noise distribution against image integrity requires tuning; no definitive noise distribution for scratch-trained models.
- Method validated only on existing pretrained I2V-DMs fine-tuned on WebVid-2M, not on models trained from scratch.
- Potential for misuse in creating misleading fake videos.
Experimental setup
Datasets: WebVid-2M UCF101
Benchmarks: UCF101 ImageBench
Metrics: FVD IS motion score motion score error user preference rate
Baselines: VideoCrafter1 DynamiCrafter SVD FrameInit FreeInit Progressive Noise CDM PIA
Builds on: Analytic-DPM EDM CDM DynamiCrafter VideoCrafter1
Closest papers by meaning
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.91)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.90)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.90)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.