Research map / Video / Video generation
PEEKABOO: Interactive Video Generation via Masked-Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free masked spatio-temporal attention module (PEEKABOO) that adds bounding-box-based spatial and temporal control to off-the-shelf UNet text-to-video diffusion models with negligible latency, plus new benchmarks SSv2-ST and IMC.
Problem
Text-to-video diffusion models produce coherent videos from text but offer no interactive spatio-temporal control over where and how objects appear, and existing control methods require retraining or adapters on grounded data.
Approach
Convert the spatial, cross, and temporal attention layers of a frozen 3D UNet into masked attention so foreground tokens and pixels attend only to their own region (and background to background) for a fixed number of early denoising steps, then allow free generation; this gives zero-training control with no extra diffusion steps.
Key results
- PEEKABOO improves mIoU over ModelScope by up to 3.8x and over ZeroScope by up to 2.9x across benchmarks, with about 1.03x latency.
- On MSR-VTT, ModelScope with PEEKABOO reaches FVD 609 versus 868 for the base ModelScope when using GPT-4-generated boxes.
- Ablations on IMC show removing the cross-attention mask causes the largest AP50 drop, followed by temporal and spatial masks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PEEKABOO masked attention | ModelScope · DAVIS16, LaSOT, ssv2-ST, IMC | mIoU: 19.6 -> 26.0; 4.0 -> 14.6; 12.0 -> 33.2; 9.6 -> 36.1 | Helped |
| Add PEEKABOO masked attention | ModelScope · DAVIS16, LaSOT, ssv2-ST, IMC | AP50: 5.7 -> 16.6; 0.7 -> 10.2; 6.6 -> 35.8; 2.4 -> 33.3 | Helped |
| Add PEEKABOO masked attention | ModelScope · MSR-VTT | FVD: 868 -> 609 | Helped |
| Add PEEKABOO masked attention | ModelScope · DAVIS16, LaSOT, ssv2-ST, IMC | latency: 1× -> 1.03× | Hurt |
| Remove cross-attention masking (ablation) | PEEKABOO on ZeroScope and ModelScope · IMC | AP50 | Hurt |
| Remove temporal-attention masking (ablation) | PEEKABOO on ZeroScope and ModelScope · IMC | AP50 | Hurt |
| Remove spatial-attention masking (ablation) | PEEKABOO on ZeroScope and ModelScope · IMC | AP50 | Hurt |
Surprising findings
- Removing the temporal attention mask hurts AP50 more than removing the spatial attention mask, which the authors attribute to loss of temporal control.
Limitations
- Relies on bounding-box inputs and an off-the-shelf detector (OWL-ViT) for evaluation, so measured control depends on detector accuracy.
- Evaluated only on UNet-based diffusion models; applicability to transformer-based video models is not shown.
- Benchmarks are small (295 SSv2 prompts, 102 IMC pairs, 40 DAVIS pairs).
Experimental setup
Datasets: Something-Something v2 LaSOT DAVIS-16 MSR-VTT
Benchmarks: SSv2-ST IMC LaSOT DAVIS-16 MSR-VTT
Metrics: mIoU AP50 Coverage Centroid Distance FVD
Baselines: ModelScope ZeroScope LLM-VD CogVideo MagicVideo
Builds on: Mask2Former Latent Diffusion Models MasaCtrl Make-A-Video
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.89)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.88)
- TrackGo: A Flexible and Efficient Method for Controllable Video Generation (AAAI 2025 · similarity 0.88)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.87)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.