Research map / Video / Video generation
Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a five-stage decoupled-guidance pipeline that generates separate background and foreground guidance for video background replacement, releasing the ~140K-pair Sparkle dataset, the 458-video Sparkle-Bench, and a Sparkle-tuned Kiwi-Edit.
Problem
Instruction-guided video background replacement requires new temporally consistent scenes while preserving foreground-background interactions, but existing datasets (e.g., OpenVE-3M) produce static, unnatural backgrounds and misaligned prompts due to foreground-only guidance.
Approach
Generate background-only videos by animating foreground-removed, prompt-compatible background images with an I2V model, track the foreground with a multi-anchor voting method (BAIT using SAM3), and regenerate the full video from decoupled foreground and background Canny edges, with EditScore filtering after each modification.
Key results
- Kiwi-Sparkle reaches 3.29 overall on OpenVE-Bench versus 2.58 for Kiwi-Edit (+28%) and 2.74 for UniVideo.
- On Sparkle-Bench, Kiwi-Sparkle scores 3.81 overall versus 2.74 for the strongest baseline, Lucy-Edit-1.1.
- Data quality on the OpenVE-3M subset rises from 3.09 to 3.71 average score (+20%) under the same judge protocol.
Surprising findings
- Copy-and-paste synthesis degrades quality severely; the decoupled regeneration gives a 115% visual quality gain over it in the Time theme.
- Foreground-only guidance produces structural collapse and loss of lighting control, showing background guidance is needed even with better foreground tracking.
Limitations
- Restricted to fixed-camera source videos, so camera-moving videos are excluded.
- Time theme (light and shadow adjustment) remains the hardest, with room for improvement.
- Evaluation relies on a single VLM judge (Gemini-2.5-Pro) and the benchmark is built from videos that failed the final pipeline check.
Experimental setup
Datasets: OpenVE-3M Sparkle
Benchmarks: OpenVE-Bench Sparkle-Bench
Metrics: Instruction Compliance Consistency & Detail Fidelity Visual Quality & Stability Foreground Integrity Foreground Motion Consistency Background Dynamics Background Visual Quality Overall score
Baselines: Kiwi-Edit Lucy-Edit-1.1 UniVideo OmniVideo2 DITTO ICVE InsViE OpenVE-Edit Runway Aleph
Builds on: OpenVE-3M Kiwi-Edit Wan2.2-Fun-A14B-Control EditScore SAM3
Closest papers by meaning
- VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.86)
- Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (NeurIPS 2025 · similarity 0.86)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.85)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.84)
- Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis (CVPR 2024 · similarity 0.84)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.