Research map / Video / Video generation
VideoTetris: Towards Compositional Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoTetris, a training-free spatio-temporal cross-attention composition method for compositional text-to-video generation, plus a ControlNet-style auto-regressive long-video model trained on a filtered, recaptioned Panda-70M subset with reference frame attention.
Problem
Text-to-video diffusion models struggle to follow compositional prompts with multiple objects, spatial relations, and dynamically changing object counts across long, multi-prompt videos.
Approach
Decompose prompts temporally (per frame) and spatially (per sub-object with region masks), then compose the cross-attention maps spatially and temporally in a training-free manner; for long videos, add an optical-flow-filtered, LLM-recaptioned training set and a reference frame attention block for object consistency.
Key results
- On compositional short video generation, VideoTetris reaches VBLIP-VQA 0.5563 and VUnidet 0.2350, the best among compared models (VideoCrafter2 0.4510 / 0.1719).
- On long progressive-prompt videos, VideoTetris reaches VBLIP-VQA 0.4839 and VUnidet 0.2137 versus FreeNoise 0.4372 / 0.1823 and StreamingT2V 0.2412 / 0.1367.
- FreeNoise achieves higher CLIP-SIM (0.9706 vs 0.9521 for VideoTetris), which the authors attribute to near-static motion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full spatio-temporal decomposition instead of Decomposing in [1] (ablation) | VideoTetris · Compositional prompts | VBLIP-VQA: 0.5203 -> 0.5563 | Helped |
| Use the full spatio-temporal decomposition instead of Decomposing in [3] (ablation) | VideoTetris · Compositional prompts | VBLIP-VQA: 0.4982 -> 0.5563 | Helped |
| Use the full composition instead of Composing in [2] (ablation) | VideoTetris · Compositional prompts | VBLIP-VQA: 0.5112 -> 0.5563 | Helped |
| Add Reference Frame Attention (ablation) | VideoTetris · Long progressive compositional prompts | MAWE: 10.21 -> 9.98 | Helped |
| Add Reference Frame Attention (ablation) | VideoTetris · Long progressive compositional prompts | CLIP: 33.50 -> 34.80 | Helped |
| Add Reference Frame Attention (ablation) | VideoTetris · Long progressive compositional prompts | AE: 7.21 -> 8.07 | Helped |
| Add Reference Frame Attention (ablation) | VideoTetris · Long progressive compositional prompts | CLIP-SIM: 0.92 -> 0.96 | Helped |
Surprising findings
- FreeNoise obtains a higher CLIP-SIM than VideoTetris, which the authors explain as a consequence of its minimal motion rather than better quality.
Limitations
- Long-video generation via ControlNet incurs huge computation cost and overly strong control, causing excess transition frames.
- Object consistency and continuously changing object positions are hard to maintain during transitions.
- The method still produces bad cases for long videos due to limitations of current long-video models.
Experimental setup
Datasets: Panda-70M
Benchmarks: custom compositional T2V prompts (100 ChatGPT-generated prompts) StreamingT2V test prompts
Metrics: VBLIP-VQA VUnidet CLIP-SIM MAWE CLIP AE
Baselines: ModelScope AnimateDiff VideoCrafter2 Gen-2 Pika LVD FreeNoise StreamingT2V
Builds on: VideoCrafter2 StreamingT2V Panda-70M FreeNoise ControlNet
Closest papers by meaning
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.91)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.91)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.90)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.90)
- Compositional 3D-aware Video Generation with LLM Director (NeurIPS 2024 · similarity 0.90)
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.