Research map / Video / Video generation
VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoGrain, a zero-shot, training-free method that modulates space-time cross- and self-attention in Stable Diffusion to enable class-, instance-, and part-level multi-grained video editing.
Problem
Multi-grained video editing (class-, instance-, and part-level) suffers from semantic misalignment of text-to-region control and feature coupling across instances of the same class in diffusion models.
Approach
Modulate space-time cross-attention to amplify each local prompt's attention within its region and suppress outside attention, and modulate spatial-temporal self-attention to increase intra-region focus and reduce inter-region interaction, using layout masks from clustering and SAM-Track, with no parameter tuning.
Key results
- VideoGrain reports CLIP-T 36.56, CLIP-F 98.63, Warp-Err 1.42 and Q-edit 25.75, the best among compared methods (DMT Q-edit 16.63).
- In human evaluation with 20 participants, VideoGrain scores 88.4 Edit-Accuracy and 83.0 Overall, versus 64.5 Overall for the next best (DMT).
- Ablation: cross-attention modulation raises CLIP-T from 33.59 to 36.09 and self-attention modulation lowers Warp-Err from 2.53 to 1.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | CLIP-F: 97.71 -> 98.63 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | CLIP-T: 35.09 -> 36.56 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | Warp-Err: 2.05 -> 1.42 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | Q-edit: 16.63 -> 25.75 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | Edit-Acc: 69.0 -> 88.4 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | Temp-Con: 79.4 -> 85.0 | Helped |
| Use VideoGrain with ST-Layout Attn | Stable Diffusion v1.5 · 76 video-text pairs | Overall: 64.5 -> 83.0 | Helped |
| Add cross-attention modulation (ablation) | Baseline video editing method · Ablation study | CLIP-T: 33.59 -> 36.09 | Helped |
| Add cross-attention modulation (ablation) | Baseline video editing method · Ablation study | Q-edit: 8.70 -> 14.26 | Helped |
| Add self-attention modulation after cross-attention modulation (ablation) | Baseline + Cross Modulation · Ablation study | Warp-Err: 2.53 -> 1.42 | Helped |
| Add self-attention modulation after cross-attention modulation (ablation) | Baseline + Cross Modulation · Ablation study | Q-edit: 14.26 -> 25.75 | Helped |
Surprising findings
- Instance-level editing with ground-truth-like grounding (Ground-A-Video, SAM-Track masks) still fails on multi-grained edits, suggesting layout guidance alone does not guarantee successful edits.
Limitations
- Quality is bounded by the Stable Diffusion base model; artifacts appear when the image prior is weak.
- Struggles with large shape deformations and significant appearance changes since it is based on a T2I model.
- Flickering in multi-object or heavily overlapping pose scenarios, and ControlNet may be needed for complex motion.
Experimental setup
Datasets: DAVIS TGVE Internet stock videos (iStock, Pexels)
Benchmarks: DAVIS TGVE 76 self-collected video-text pairs
Metrics: CLIP-T CLIP-F Warp-Err Q-edit Edit-Accuracy Temporal Consistency Overall quality (human)
Baselines: FateZero ControlVideo TokenFlow Ground-A-Video DMT FLATTEN VideoP2P
Builds on: Tune-A-Video Prompt-to-Prompt FateZero DDIM inversion ControlNet SAM-Track
Closest papers by meaning
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.89)
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.88)
- Trajectory attention for fine-grained video motion control (ICLR 2025 · similarity 0.88)
- Re-Attentional Controllable Video Diffusion Editing (AAAI 2025 · similarity 0.87)
- FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation (CVPR 2024 · similarity 0.87)
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.