Research map / Video / Video generation
Keyframe-Guided Creative Video Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoRepainter, a two-stage creative video inpainting method that edits a keyframe with image inpainting tools, then propagates the edit through an adapted image-to-video diffusion model (SVD) using symmetric mask conditioning and LoRA.
Problem
Video inpainting for creative editing must balance temporal/visual coherence with user control and diversity; text-guided video inpainting models have limited visual quality and controllability.
Approach
Decouple still-content creation (keyframe inpainting with state-of-the-art image models) from motion propagation by repurposing an image-to-video diffusion model via a symmetric mask condition (black- and white-filled masked-image latents) that avoids mask-downsampling ambiguity, plus LoRA on attention layers and content-agnostic mask augmentations.
Key results
- Inpainting on DAVIS: consistency 96.01 and background PSNR 31.46 vs best baseline CoCoCo 93.94 / 28.33 (I2VGen-XL 29.51 PSNR).
- Editing evaluation: consistency 97.31, background PSNR 32.29, MSE 3.77 vs AnyV2V 96.28 / 18.38 / 173.96.
- Training only ~27.9M of 1525.9M UNet parameters (<2%) via LoRA plus input conv; symmetric mask condition improves background fidelity (MSE 11.59 vs 22.23 without).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · DAVIS video inpainting | CLIP consistency: 93.94 -> 96.01 | Helped |
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · DAVIS video inpainting | PSNR (background): 29.51 -> 31.46 | Helped |
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · DAVIS video inpainting | MSE (background) ×104: 13.09 -> 8.85 | Helped |
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · In-the-wild video editing | CLIP consistency: 96.28 -> 97.31 | Helped |
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · In-the-wild video editing | PSNR (background): 31.99 -> 32.29 | Helped |
| Use VideoRepainter instead of the strongest baseline | VideoRepainter · In-the-wild video editing | MSE (background) ×104: 6.66 -> 3.77 | Helped |
| Remove symmetric mask conditioning (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | PSNR (background): 30.78 -> 27.52 | Hurt |
| Add full-mask augmentation (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | PSNR (background): 28.28 -> 30.78 | Helped |
| Add grid-mask augmentation (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | PSNR (background): 28.56 -> 30.78 | Helped |
| Add temporal Bezier mask augmentation (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | CLIP consistency: 94.87 -> 96.73 | Helped |
| Add temporal mask variants (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | CLIP consistency: 96.52 -> 96.73 | Helped |
| Train all model parameters instead of only attention LoRA adapters (ablation) | VideoRepainter · Ablation evaluation, svd-base, 14 frames | PSNR (background): 30.78 -> 28.90 | Hurt |
Surprising findings
- Full-model fine-tuning yielded lower background fidelity than the parameter-efficient adapter, attributed to overfitting on limited data.
Limitations
- Performance bounded by the underlying I2V backbone; out-of-domain keyframe edits (e.g., anime character in realistic street) show little or no motion.
- Assumes a user-provided mask rather than instruction-based editing.
Experimental setup
Datasets: self-collected watermark-free video dataset (~300K videos) DAVIS
Benchmarks: DAVIS in-the-wild editing samples (20)
Metrics: CLIP consistency score background PSNR background MSE aesthetic score
Baselines: CoCoCo I2VGen-XL ModelScope AnyV2V
Builds on: Stable Video Diffusion LoRA EDM MultiDiffusion
Closest papers by meaning
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility (AAAI 2025 · similarity 0.88)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.87)
- AVID: Any-Length Video Inpainting with Diffusion Model (CVPR 2024 · similarity 0.87)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.87)
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.87)
- Infinite-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.