Research map / Video / Video generation
Video Interpolation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIDIM, a cascaded pixel-space video diffusion model that interpolates between a start and end frame with parameter-free frame conditioning and classifier-free guidance, outperforming prior VFI methods on large, ambiguous motion.
Problem
Video frame interpolation methods fail when motion between the start and end frames is large, nonlinear, or ambiguous, producing blurry regression-based outputs.
Approach
Use a cascaded (base + spatial super-resolution) video diffusion model conditioned on start and end frames by concatenating them along the frame axis with fake maximum-SNR noise levels, adding no extra parameters, and applying classifier-free guidance on the conditioning frames; the whole clip is jointly denoised.
Key results
- On Davis-7 (all 7 frames), VIDIM achieves FVD 199.32 vs. 214.8 for FILM and 234.5 for AMT, while RIFE has the best FID (23.98 vs. VIDIM 28.06).
- In a 31-participant user study on Davis-7, VIDIM was preferred 87.9% of the time over AMT, RIFE, FILM and LDMVFI.
- Scaling the base model from 441M to 1.6B parameters is reported as essential to avoid severe artifacts; the cascade with large models achieves the lowest FID among VIDIM variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VIDIM instead of AMT | video interpolation · Davis-7 | FVD: 234.5 -> 199.32 | Helped |
| use VIDIM instead of LDMVFI | video interpolation · UCF101-7 | FVD: 316.3 -> 278 | Helped |
| use VIDIM instead of AMT | video interpolation · Davis-7 | PSNR: 21.09 -> 19.62 | Hurt |
| use VIDIM instead of RIFE | video interpolation · UCF101-7 | FID: 18.72 -> 34.48 | Hurt |
| add explicit conditioning on the original high-resolution start and end frames during training (ablation) | VIDIM super-resolution model · video interpolation | FID | Helped |
| add classifier-free guidance on the conditioning frames (ablation) | VIDIM · video interpolation | sample quality | Helped |
| increase the base model from medium to large (ablation) | VIDIM cascade · Davis-7 | FID: 22.59 -> 28.06 | Hurt |
| increase the super-resolution model from medium to large (ablation) | VIDIM cascade with large base model · Davis-7 | FID: 38.78 -> 28.06 | Helped |
| increase the super-resolution model from medium to large (ablation) | VIDIM cascade with large base model · UCF101-7 | FID: 40.44 -> 34.48 | Helped |
| use reconstruction guidance with the inpainting baseline instead of explicit frame conditioning (ablation) | super-resolution inpainting baseline · video interpolation | FID: 60.11 vs. VIDIM FID scores | Hurt |
Surprising findings
- RIFE achieves the best FID over all 7 frames on Davis-7 and UCF101-7 yet has the worst FVD, showing FID misses temporal consistency.
- Medium super-resolution model beats the large one on isolated reconstruction metrics, though the large SR model is needed for low FID in the full cascade.
Limitations
- Fixed-resolution, pixel-space cascade; generating at arbitrary aspect ratios remains open.
- Reconstruction metrics (PSNR/SSIM) favor blurrier outputs, and VIDIM underperforms baselines on those for UCF101-7.
- Outputs may differ from ground truth under ambiguous motion, and CFG weights above about 4.0 cause color artifacts.
Experimental setup
Datasets: WebVid internal video datasets DAVIS UCF101
Benchmarks: Davis-7 UCF101-7
Metrics: PSNR SSIM LPIPS FID FVD human preference
Baselines: AMT RIFE FILM LDMVFI Inpainting baseline (Ho et al. video diffusion)
Builds on: Imagen Video Video Diffusion Models FiLM classifier-free guidance cosine log-SNR schedule
Closest papers by meaning
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (ICLR 2025 · similarity 0.90)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.90)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.90)
- IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis (ICLR 2025 · similarity 0.88)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.88)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.