Research map / Video / Video generation
LDMVFI: Video Frame Interpolation with Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDMVFI, the first latent diffusion model for video frame interpolation, framed as conditional generation with a VFI-specific autoencoder (VQ-FIGAN) that uses frame-aided decoding and deformable-kernel synthesis.
Problem
Existing VFI methods trained with L1/L2/VGG losses score well on PSNR but poorly on perceptual quality, especially for dynamic textures with complex motion; diffusion models had not been effectively applied to VFI at high resolution.
Approach
Formulate VFI as conditional image generation in a latent space: a denoising U-Net (with MaxViT attention) generates the target frame latent conditioned on input frame latents, and a VQ-FIGAN decoder uses neighboring-frame feature pyramids and predicts deformable interpolation kernels to synthesize the output frame.
Key results
- LDMVFI achieves the best LPIPS, FloLPIPS and FID on VFITex (0.150 / 0.207 / 32.316 vs. next best ~0.188 LPIPS for XVFI), roughly 20% gain over the second-best method.
- On UCF-101 LDMVFI obtains LPIPS 0.026 and FID 26.301, best among compared methods, while its PSNR (32.186) is below several baselines.
- In a 20-participant 2AFC user study on 22 BVI-HFR full HD videos, LDMVFI was preferred over ST-MFNet, IFRNet and BMBC with statistically significant advantage at 95% confidence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LDMVFI instead of the strongest listed baseline | VFI · Middlebury | LPIPS: 0.020 -> 0.019 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · UCF-101 | LPIPS: 0.032 -> 0.026 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · DAVIS | LPIPS: 0.114 -> 0.107 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · VFITex | LPIPS: 0.188 -> 0.150 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · SNU-FILM-Easy | LPIPS: 0.019 -> 0.014 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · SNU-FILM-Medium | LPIPS: 0.033 -> 0.028 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · SNU-FILM-Hard | LPIPS: 0.065 -> 0.060 | Helped |
| Use LDMVFI instead of the strongest listed baseline | VFI · SNU-FILM-Extreme | LPIPS: 0.136 -> 0.123 | Helped |
| Add neighbor-frame features during decoding and use MaxCA blocks (ablation) | LDMVFI · Middlebury | LPIPS: 0.077 -> 0.019 | Helped |
| Predict deformable kernels instead of decoding the frame directly (ablation) | LDMVFI · Middlebury | LPIPS: 0.028 -> 0.019 | Helped |
| Set downsampling factor f=8 instead of f=32 (ablation) | LDMVFI · Middlebury | LPIPS: 0.019 -> 0.018 | Helped |
| Set downsampling factor f=16 instead of f=32 (ablation) | LDMVFI · Middlebury | LPIPS: 0.019 -> 0.017 | Helped |
Surprising findings
- Removing the diffusion process (V6) substantially degrades LPIPS and FloLPIPS, while the diffusion-based MCVD baseline performs poorly overall, indicating naive diffusion formulations are insufficient for VFI.
- LDMVFI achieves lower PSNR than most baselines on several test sets (e.g., 25.541 vs. 28.287 for ST-MFNet on DAVIS), showing PSNR is not aligned with the perceptual gains.
Limitations
- Very slow inference (8.48 s per 480p frame) due to iterative DDIM denoising
- Large parameter count (439M) inherited from generic image U-Net design
- Reported FID on Middlebury and UCF-101 is unreliable due to very few test frames
Experimental setup
Datasets: Vimeo90k-septuplet BVI-DVC
Benchmarks: Middlebury UCF-101 DAVIS VFITex SNU-FILM BVI-HFR
Metrics: LPIPS FloLPIPS FID PSNR SSIM user preference ratio (2AFC)
Baselines: BMBC AdaCoF CDFI XVFI ABME IFRNet VFIformer ST-MFNet FLAVR MCVD
Builds on: latent diffusion models VQGAN denoising diffusion probabilistic models MaxViT AdaCoF
Closest papers by meaning
- Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation (CVPR 2025 · similarity 0.89)
- Disentangled Motion Modeling for Video Frame Interpolation (AAAI 2025 · similarity 0.88)
- Perception-Oriented Video Frame Interpolation via Asymmetric Blending (CVPR 2024 · similarity 0.87)
- VFIMamba: Video Frame Interpolation with State Space Models (NeurIPS 2024 · similarity 0.86)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.86)
- CV-VAE: A Compatible Video VAE for Latent Generative Video Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.