Research map / Video / Video generation
Elevating Flow-Guided Video Inpainting with Reference Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RGVI, a flow-guided video inpainting framework that combines one-shot pixel pulling with Stable Diffusion reference generation on a key frame, and introduces the HQVI benchmark built with alpha matte composition.
Problem
Video inpainting requires propagating observable content across frames while generating non-observable content; existing flow-based methods suffer from sub-pixel loss or re-sampling artifacts in pixel propagation and lack strong generation for large missing areas and high resolution.
Approach
Decouples propagation and generation: a one-shot pixel pulling method warps optical flows (not colors) via recurrent grid warping and pulls colors once, with a forward/backward verification step; a key frame chosen by connection count is filled by Stable Diffusion and propagated to all frames, with occlusion handled by an additional positive mask.
Key results
- On HQVI at 240x432, RGVI reaches LPIPS 0.0335 and VFID 0.1825 versus ProPainter's 0.0388 and 0.2128, while RGVI without reference attains the highest PSNR (31.60).
- At 1200x2160 on HQVI, RGVI with reference reaches PSNR 30.10 and VFID 0.0058 versus 29.81 and 0.0101 for the no-reference variant.
- In a 10-participant user study on DAVIS 2016, RGVI obtained the best average rank (1.59) versus ProPainter (1.90) and FuseFormer (2.52).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace recurrent internal pixel propagation with one-shot pulling (ablation) | RGVI · HQVI 240×432, no reference generation | LPIPS: 0.0595 -> 0.0390 | Helped |
| replace recurrent internal pixel propagation with one-shot pulling (ablation) | RGVI · HQVI 240×432, no reference generation | PSNR: 31.43 -> 31.60 | Helped |
| replace recurrent internal and reference propagation with one-shot pulling (ablation) | RGVI · HQVI 240×432 | LPIPS: 0.0558 -> 0.0335 | Helped |
| replace recurrent internal and reference propagation with one-shot pulling (ablation) | RGVI · HQVI 240×432 | PSNR: 30.17 -> 30.66 | Helped |
| add reference generation (ablation) | RGVI · HQVI 240×432 | LPIPS: 0.0390 -> 0.0335 | Helped |
| add reference generation (ablation) | RGVI · HQVI 240×432 | PSNR: 31.60 -> 30.66 | Hurt |
| add reference generation (ablation) | RGVI · HQVI 1200×2160 | PSNR: 29.81 -> 30.10 | Helped |
| add the occluding-object mask (ablation) | RGVI · HQVI 240×432 videos containing occluding objects | PSNR: 35.13 -> 37.31 | Helped |
| add the occluding-object mask (ablation) | RGVI · HQVI 240×432 videos containing occluding objects | LPIPS: 0.0137 -> 0.0102 | Helped |
Surprising findings
- The variant without reference generation achieves higher PSNR and SSIM than the full model on HQVI at 240p and 480p, while the full model wins on perceptual metrics, which the authors attribute to blurry textures scoring higher on traditional metrics.
Limitations
- Relies on optical flow, so flow completion errors cause structural misalignment.
- Generated reference content can sometimes look unnatural.
- Multi-key-frame fallback is needed in rare cases where a single reference frame is insufficient.
Experimental setup
Datasets: YouTube-VOS 2018 VideoMatte240K Pexels videos DAVIS 2016
Benchmarks: HQVI DAVIS 2016 YouTube-VOS 2018 (YTVI)
Metrics: PSNR SSIM LPIPS VFID user rank
Baselines: STTN FGVC FuseFormer E2FGVI ProPainter
Builds on: ProPainter FGVC E2FGVI latent diffusion model RAFT
Closest papers by meaning
- Infinite-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation (AAAI 2025 · similarity 0.88)
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.88)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.87)
- HomoGen: Enhanced Video Inpainting via Homography Propagation and Diffusion (CVPR 2025 · similarity 0.87)
- AVID: Any-Length Video Inpainting with Diffusion Model (CVPR 2024 · similarity 0.87)
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.