Research map / Video / Video generation
AVID: Any-Length Video Inpainting with Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AVID, a text-guided video inpainting diffusion model that adds motion modules, a ControlNet-style structure guidance module, and a zero-shot Temporal MultiDiffusion sampler with middle-frame attention guidance for arbitrary-length videos.
Problem
Text-guided video inpainting must keep edits temporally consistent, support different inpainting types with different structural fidelity needs, and handle variable video lengths beyond the training clip length.
Approach
Inflate a text-guided image inpainting LDM with AnimateDiff-style motion modules, add a ControlNet-style HED structure guidance branch with an adjustable scale, and extend to arbitrary length via overlapping-window MultiDiffusion sampling plus attention to the middle frame's keys and values.
Key results
- Achieves temporal consistency (TC) of 97.2 on uncropping, 96.5 on object swap, and 96.3 on re-texturing, the highest among compared methods.
- User study preference: 69.1% (uncropping), 74.5% (object swap), 73.2% (re-texturing) favor AVID over per-frame LDM, Text2Video-Zero, and VideoComposer.
- Background preservation L1 of 42.3, 41.1, 40.7 (×10^-3) is lowest (best) for uncropping, object swap, and re-texturing respectively, except VideoComposer's 55.7/71.0/64.5 being worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVID instead of the strongest listed baseline | Uncropping · Uncropping | TC: 96.4 -> 97.2 | Helped |
| Use AVID instead of the strongest listed baseline | Object swap · Object swap | BP (×10−3): 71.0 -> 41.1 | Helped |
| Use AVID instead of the strongest listed baseline | Object swap · Object swap | TA: 31.5 -> 31.5 | No clear effect |
| Use AVID instead of the strongest listed baseline | Object swap · Object swap | TC: 96.5 -> 96.5 | No clear effect |
| Use AVID instead of the strongest listed baseline | Re-texturing · Re-texturing | BP (×10−3): 64.5 -> 40.7 | Helped |
| Use AVID instead of the strongest listed baseline | Re-texturing · Re-texturing | TA: 32.1 -> 32.0 | Hurt |
| Use AVID instead of the strongest listed baseline | Re-texturing · Re-texturing | TC: 95.5 -> 96.3 | Helped |
| Add structure guidance for re-texturing (ablation) | AVID · Re-texturing | structural fidelity | Helped |
| Increase structure guidance for object swap (ablation) | AVID · Object swap | text instruction match | Hurt |
| Use Temporal MultiDiffusion instead of separate non-overlapping clips (ablation) | AVID · Long videos | temporal smoothness | Helped |
| Add middle-frame attention guidance (ablation) | AVID · Long videos | identity consistency | Helped |
| Set middle-frame attention guidance weight to 1.0 (ablation) | AVID · Long videos | generation quality | Hurt |
Surprising findings
- Applying the fixed-length model directly to longer videos causes drastic quality degradation, motivating the zero-shot pipeline.
Limitations
- Performance is limited by the learned motion module quality, especially for complicated actions.
- Structure guidance scale must be hand-tuned per editing sub-task.
- CLIP-based metrics may not correlate well with human perception.
Experimental setup
Datasets: Shutterstock video dataset (watermark-removed) 125 evaluation videos (unseen in training)
Benchmarks: 125 in-house evaluation videos across uncropping, object swap, re-texturing
Metrics: background preservation (L1) CLIP text-video alignment CLIP-image temporal consistency user study preference
Baselines: Per-frame LDM inpainting Text2Video-Zero VideoComposer
Builds on: Stable Diffusion inpainting (LDM) AnimateDiff ControlNet MultiDiffusion XMem
Closest papers by meaning
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.90)
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility (AAAI 2025 · similarity 0.88)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.88)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.87)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.87)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.