Research map / Video / Video generation
Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ground-A-Video performs training-free, zero-shot multi-attribute video editing by combining grounding (bounding-box + caption) cross-frame gated attention, modulated cross-attention over per-frame null-text embeddings, depth-conditioned inflated ControlNet, and optical-flow-guided inverted latent smoothing on Stable Diffusion.
Problem
Multi-attribute video editing with text-to-image diffusion models suffers from neglected edits, edits on wrong elements, mixed edits, and failure to preserve unchanged regions, along with temporal inconsistency when adding grounding conditions.
Approach
Integrates spatially-discrete grounding (boxes + captions) applied via cross-frame gated attention with spatially-continuous depth ControlNet, plus modulated cross-attention to make per-frame null-text embeddings interact across frames and optical-flow smoothing of DDIM-inverted latents, all without training on video.
Key results
- Ground-A-Video achieves CLIP text-align 0.837 and frame consistency 0.970, the best among TAV, CAV, CV, and Gen-1 baselines.
- In a 28-participant user study on a 1-5 scale, Ground-A-Video scores 4.13 edit accuracy and 4.24 preserve accuracy versus 2.25-2.99 and 2.02-3.13 for baselines.
- Ablation removing groundings drops text-align from 0.837 to 0.802, showing grounding guidance contributes most to edit accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ground-A-Video instead of the strongest baseline, Gen-1 | DAVIS multi-attribute video editing · DAVIS multi-attribute video editing | Text-Align: 0.833 -> 0.837 | Helped |
| Use Ground-A-Video instead of the strongest baseline, ControlVideo | DAVIS multi-attribute video editing · DAVIS multi-attribute video editing | Frame-Con (CLIP): 0.963 -> 0.970 | Helped |
| Use Ground-A-Video instead of the strongest baseline, Tune-A-Video | DAVIS multi-attribute video editing · DAVIS multi-attribute video editing | Edit-Acc: 2.99 -> 4.13 | Helped |
| Use Ground-A-Video instead of the strongest baseline, Tune-A-Video | DAVIS multi-attribute video editing · DAVIS multi-attribute video editing | Preserve-Acc: 3.13 -> 4.24 | Helped |
| Use Ground-A-Video instead of the strongest baseline, Tune-A-Video | DAVIS multi-attribute video editing · DAVIS multi-attribute video editing | Frame-Con (user study): 3.05 -> 4.01 | Helped |
| Add Modulated Cross-Attention (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Text-Align: 0.835 -> 0.837 | Helped |
| Add Modulated Cross-Attention (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Frame-Con (CLIP): 0.967 -> 0.970 | Helped |
| Add groundings (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Text-Align: 0.802 -> 0.837 | Helped |
| Add groundings (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Frame-Con (CLIP): 0.960 -> 0.970 | Helped |
| Use Cross-Frame Gated Attention instead of no gated attention (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Text-Align: 0.829 -> 0.837 | Helped |
| Use Cross-Frame Gated Attention instead of no gated attention (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Frame-Con (CLIP): 0.956 -> 0.970 | Helped |
| Add ControlNet (ablation) | Ground-A-Video · DAVIS multi-attribute video editing | Frame-Con (CLIP): 0.948 -> 0.970 | Helped |
Surprising findings
- Frame-independent grounding guidance can yield worse results than no grounding at all, so grounding must be applied cross-frame.
Limitations
- Misleading groundings (incorrect bounding boxes) lead to inaccurate editing.
- Evaluation is on a small subset of 20 DAVIS videos with 8-frame clips.
- Structural flexibility from ControlNet must be tuned via ControlNet Scale.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: CLIP text alignment CLIP frame consistency user study edit accuracy user study preserve accuracy user study frame consistency
Baselines: Tune-A-Video Control-A-Video ControlVideo Gen-1 Tune-A-Video-ControlNet
Builds on: Stable Diffusion GLIGEN ControlNet null-text inversion Tune-A-Video
Closest papers by meaning
- A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing (CVPR 2024 · similarity 0.87)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.86)
- VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.86)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.86)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.86)
- FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.