Research map / Video / Video generation
CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoCoCo, a text-guided video inpainting model with a motion capture module (damped global and textual cross attention), instance-aware region selection for training masks, and a task-vector strategy to plug personalized T2I models into the inpainting model.
Problem
Text-guided video inpainting lacks effective methods; existing approaches (e.g., AVID) suffer from weak text-video alignment, motion inconsistency, and poor compatibility with personalized text-to-image models.
Approach
Adds a damped global attention layer and a textual cross-attention layer to AnimateDiff-style temporal motion blocks, replaces random training masks with instance-aware masks grounded via Grounding DINO with TokenSpan, and merges personalized T2I models into the inpainting model via task-vector arithmetic.
Key results
- CoCoCo achieves CLIP score 24.9, background preservation L1 6.20 (lowest), and temporal consistency 97.2 (highest) versus AnimateDiffV3 (25.3, 7.2, 96.7) and VideoComposer (19.8, 21.7, 96.5).
- In a user study, CoCoCo is preferred for visual quality at 72.9% versus 14.6% for AnimateDiffV3 and 12.5% for VideoComposer.
- Ablation: instance-aware region selection raises CLIP score from 21.6 to 23.6, and the motion capture block lowers BP from 8.6 to 6.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instance-aware region selection instead of random mask selection (ablation) | CoCoCo motion block with two temporal attentions · WebVid-10M validation (1,000 videos) | CLIP score: 21.6 to 23.6 | Helped |
| Add the motion capture block (ablation) | CoCoCo with instance-aware region selection · WebVid-10M validation (1,000 videos) | background preservation: 8.6 to 6.2 | Helped |
| Add instance-aware region selection and the motion capture block (ablation) | CoCoCo with random masks and two temporal attentions · WebVid-10M validation (1,000 videos) | temporal consistency: 96.3 to 97.2 | Helped |
Surprising findings
- Zero-shot AnimateDiffV3 and VideoCrafter2 obtain higher CLIP scores than CoCoCo (25.3 and 26.2 vs 24.9), though CoCoCo has better background preservation and consistency.
Limitations
- Compared against AVID only qualitatively in appendix because AVID was not open-source at submission time
- Transformation strategy requires tuning task-vector mixing coefficients alpha and beta, which vary across T2I and inpainting models
Experimental setup
Datasets: WebVid-10M
Benchmarks: WebVid-10M validation subset (1000 videos)
Metrics: CLIP score background preservation L1 temporal consistency (CLIP feature cosine similarity) user study preference
Baselines: AnimateDiffV3 VideoCrafter2 VideoComposer AVID
Builds on: AnimateDiff AVID Stable Diffusion Inpainting Grounding DINO task vectors
Closest papers by meaning
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.88)
- AVID: Any-Length Video Inpainting with Diffusion Model (CVPR 2024 · similarity 0.88)
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.87)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.86)
- Towards Language-Driven Video Inpainting via Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.