Research map / Video / Video generation
HomoGen: Enhanced Video Inpainting via Homography Propagation and Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HomoGen, a two-stage video inpainting method that propagates pixels via homography registration and feeds them as priors into a Stable Diffusion-based video diffusion model with content-adaptive control.
Problem
Video inpainting suffers from blur and distortion from flow-based pixel propagation and error accumulation in video transformers when filling corrupted regions in videos.
Approach
Replace optical-flow warping with homography-based propagation (no consistency check) and use a video diffusion model that treats propagated pixels as conditions, injecting them via learned spatiotemporal adaptive de-normalization and step-dependent scaling so denoising filters artifacts.
Key results
- HomoGen reaches 34.81 PSNR and 0.032 VFID on YouTube-VOS, versus 34.46 PSNR and 0.042 VFID for ProPainter.
- VFID improves by 23.81% over the state of the art on YouTube-VOS, 21.43% on DAVIS and 17.86% on RORD.
- In a 15-participant user study, volunteers preferred HomoGen over ProPainter across spatial coherence, temporal consistency, visual quality and overall impression, with 49-50% preference for HomoGen per criterion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HomoGen instead of ProPainter | Video inpainting · YouTube-VOS test | VFID: 0.042 -> 0.032 | Helped |
| use HomoGen instead of ProPainter | Video inpainting · DAVIS test | VFID: 0.098 -> 0.077 | Helped |
| use HomoGen instead of ProPainter | Video inpainting · RORD test | VFID: 0.028 -> 0.023 | Helped |
| remove homography-based pixel propagation (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 33.16 | Hurt |
| replace homography priors with flow priors and add flow consistency check (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 34.05 | Hurt |
| replace homography priors with flow priors and omit flow consistency check (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 34.23 | Hurt |
| replace PCVDM with a video transformer (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 33.88 | Hurt |
| use flow priors with consistency check and a video transformer (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 34.46 | Hurt |
| remove text prompts (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 34.07 | Hurt |
| remove GoP-wise CLIP embeddings (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 34.62 | Hurt |
| replace content-adaptive controls with prior pasting (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 31.74 | Hurt |
| replace content-adaptive controls with prior concatenation (ablation) | HomoGen on YouTube-VOS · YouTube-VOS | PSNR: 34.81 -> 32.86 | Hurt |
Surprising findings
- Removing the flow consistency check and retaining all propagated pixels improves results when the priors feed a diffusion model, since diffusion models tolerate artifacts better than video transformers.
Limitations
- Generating identity-preserved content in videos with scene alternations remains challenging.
- Efficiently processing high-resolution videos remains challenging.
- Homography-based propagation slightly increases warping error compared with flow-based propagation.
Experimental setup
Datasets: Shutterstock internal video dataset (5M sequences) YouTube-VOS DAVIS RORD
Benchmarks: YouTube-VOS DAVIS RORD
Metrics: PSNR SSIM VFID Ewarp
Baselines: DFVI CPNet FGVC STTN FuseFormer FGT ISVI E2FGVI ProPainter
Builds on: Stable Diffusion inpainting ControlNet ProPainter latent diffusion models
Closest papers by meaning
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.89)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.87)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.87)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.87)
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.87)
- Improving Progressive Generation with Decomposable Flow Matching (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.