Research map / Video / Video generation
Video Diffusion Models Are Strong Video Inpainter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FFF-VDI, which fine-tunes Stable Video Diffusion for video inpainting by propagating noise latents from future frames into the first frame's latent and using DDIM inversion to reduce hallucination.
Problem
Propagation-based video inpainting relies on optical flow accuracy and accumulates noise over time, while diffusion-based inpainting suffers from temporal inconsistency when masks move and object hallucination for large masks.
Approach
Fill the first frame's noise latent by propagating future frames' noise latents via optical flow (only into the first frame), refine with a deformable noise alignment module that learns offsets without flow, and use DDIM-inverted noise at inference; then fine-tune an image-to-video diffusion model (SVD) on this conditioning.
Key results
- On YouTube-VOS, FFF-VDI reaches 35.06 PSNR, 0.9812 SSIM, 0.031 VFID versus ProPainter's 34.43, 0.9735, 0.042.
- On DAVIS, FFF-VDI reaches 35.03 PSNR and 0.075 VFID versus ProPainter's 34.47 and 0.098.
- Ablation: adding the LP module raises YouTube-VOS PSNR from 32.21 to 34.03 over the per-frame-style baseline; adding DNA further lifts it to 35.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FFF-VDI instead of ProPainter | Video inpainting · YouTube-VOS | VFID: 0.042 -> 0.031 | Helped |
| Use FFF-VDI instead of ProPainter | Video inpainting · DAVIS | PSNR: 34.47 -> 35.03 | Helped |
| Add latent propagation (LP) to per-frame diffusion (ablation) | FFF-VDI · YouTube-VOS | PSNR: 32.21 -> 34.03 | Helped |
| Add latent propagation (LP) to per-frame diffusion (ablation) | FFF-VDI · DAVIS | PSNR: 30.59 -> 34.11 | Helped |
| Add DNA to LP (ablation) | FFF-VDI · YouTube-VOS | E_warp*: 1.142 -> 0.940 | Helped |
| Add DNA to LP (ablation) | FFF-VDI · DAVIS | VFID: 0.089 -> 0.074 | Helped |
| Add DDIM inversion to LP and DNA (ablation) | FFF-VDI · YouTube-VOS | PSNR: 35.01 -> 35.06 | Helped |
| Add DDIM inversion to LP and DNA (ablation) | FFF-VDI · DAVIS | PSNR: 35.10 -> 35.03 | Hurt |
| Add DDIM inversion to LP and DNA (ablation) | FFF-VDI · YouTube-VOS | VFID: 0.031 -> 0.031 | No clear effect |
Surprising findings
- DDIM inversion yields only a marginal quantitative gain (35.01 to 35.06 PSNR on YouTube-VOS) despite its clear qualitative effect on hallucination.
Limitations
- Relies on optical flow (RAFT and ProPainter's flow completion) for the first-frame latent propagation, so flow errors still matter.
- Quantitative gains from DDIM inversion are small, and the effect is mainly qualitative.
- Evaluated mainly with random masks and ProPainter's test masks; text-guided or very long videos are not studied.
Experimental setup
Datasets: YouTube-VOS
Benchmarks: YouTube-VOS DAVIS
Metrics: PSNR SSIM VFID E_warp
Baselines: FGVC STTN TSAM FuseFormer ISVI FGT E2FGVI ProPainter Stable Diffusion (per-frame)
Builds on: Stable Video Diffusion Video LDM ProPainter DDIM inversion E2FGVI
Closest papers by meaning
- AVID: Any-Length Video Inpainting with Diffusion Model (CVPR 2024 · similarity 0.90)
- HomoGen: Enhanced Video Inpainting via Homography Propagation and Diffusion (CVPR 2025 · similarity 0.89)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.89)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.88)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.88)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.