Research map / Video / Video generation
Compositional text-to-video diffusion
Generates videos from text with improved motion, composition and extendable length, plus benchmarks.
159 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Video generation, keyword cluster “video generation · video diffusion · text video”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 4 tracked benchmarks.
36 papers in 2023–24 and 120 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 15.8% to 10.7%.
Open questions the data points to
- MSR-VTT (CLIPSIM): best went from 0.312 (up to 2024) to 0.327 in 2025 — Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · 2 citations)
- Papers report DynamiCrafter on UCF101 (FVD) anywhere from 288.01 to 589.59: comparisons against it depend on whose number you take.
- Papers report ModelScopeT2V on MSR-VTT (FID) anywhere from 11.09 to 14.89: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Video Generation From Text (AAAI 2018 · 161 citations)
- 2022 · Make It Move: Controllable Image-to-Video Generation With Text Descriptions (CVPR 2022 · 60 citations)
- 2023 · Make-A-Video: Text-to-Video Generation without Text-Video Data (ICLR 2023 · 311 citations)
- 2024 · VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (CVPR 2024 · 223 citations)
- 2025 · StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · 34 citations)
- 2026 · VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning (AAAI 2026 · 5 citations)
Earliest papers here
- Video Generation From Text (AAAI 2018 · 161 citations)
- Middle-Out Decoding (NeurIPS 2018 · 0 citations)
- Make It Move: Controllable Image-to-Video Generation With Text Descriptions (CVPR 2022 · 60 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
UCF-101 · FVD (lower is better)
7 papers, 2023–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 626 | CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (ICLR 2023 · 117 citations) |
| 2024 | 260.7 | Seer: Language Instructed Video Prediction with Latent Diffusion Models (ICLR 2024 · 6 citations) |
MSR-VTT · CLIPSIM (higher is better)
6 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.305 | Make-A-Video: Text-to-Video Generation without Text-Video Data (ICLR 2023 · 311 citations) |
| 2024 | 0.312 | VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · 22 citations) |
| 2025 | 0.327 | Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · 2 citations) |
MSR-VTT · FVD (lower is better)
4 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 213 | VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · 22 citations) |
UCF-101 · IS (higher is better)
4 papers, 2023–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 50.46 | CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (ICLR 2023 · 117 citations) |
What has been tried
81 papers here report 837 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 5%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train spatial and temporal modules together instead of training only temporal modules | perturbed T2V models · motion quality | 87% | Helped | VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (CVPR 2024 · 223 citations) |
| add the Temporal Adapter | WebVid validation (1K samples) · FVD | 1470.1 -> 530.2 | Helped | SimDA: Simple Diffusion Adapter for Efficient Video Generation (CVPR 2024 · 45 citations) |
| add fine visual embeddings via Attention Injection to coarse embeddings | VideoBooth ablation subset · CLIP-Image | 75.4366 -> 76.1631 | Helped | VideoBooth: Diffusion-based Video Generation with Image Prompts (CVPR 2024 · 43 citations) |
| Add randomized blending to merge overlapping enhanced video chunks | Overlapping video chunks · temporal consistency | Helped | StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · 34 citations) | |
| replace no frame interaction with fully cross-frame interaction | Canny Edge, cross-frame mechanism ablation · Frame Consistency (%) | 89.94 -> 95.36 | Helped | ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · 34 citations) |
| Use Vlogger's planned scene-by-scene generation process instead of no autoregression | Vlog generation · CLIP-I | 0.5683 -> 0.6294 | Helped | Vlogger: Make Your Dream A Vlog (CVPR 2024 · 27 citations) |
| add LFQ to MAGVIT | ImageNet 128×128 · FID | 2.65 -> 2.48 | Helped | Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · 22 citations) |
| Add T2I pretraining to T2V-only training | MSR-VTT · CLIPSIM | 0.244 -> 0.247 | Helped | VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · 22 citations) |
| Add temporal coherence loss | Text-to-video generation · Frame consistency | 89.71 -> 91.06 | Helped | A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · 21 citations) |
| Add quality regression and LoRA fine-tuning components | AIGVQA-DB — Static Quality · SRCC | 0.864 -> 0.916 | Helped | AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (CVPR 2025 · 18 citations) |
| Add DDPM-based inversion | MUG · FVD | 1656.37 -> 339.89 | Helped | TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · 17 citations) |
| Add the Global Facial Extractor (low-frequency reference-face and key-point signal) | IPT2V evaluation · FaceSim-Arc | 0.05 -> 0.73 | Helped | Identity-Preserving Text-to-Video Generation by Frequency Decomposition (CVPR 2025 · 13 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| DynamiCrafter | UCF101 | FVD | 288.01 – 589.59 | 2 papers, e.g. CVPR 2025 and CVPR 2024 |
| ModelScopeT2V | MSR-VTT | FID | 11.09 – 14.89 | 2 papers, e.g. CVPR 2024 and NeurIPS 2024 |
What reviewers will expect
Datasets and baselines named in the 56 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WebVid-10M · 7 of 56 papers
- Panda-70M · 6 of 56 papers
- UCF101 · 4 of 56 papers
- OpenVid-1M · 4 of 56 papers
- VidProM · 4 of 56 papers
- VBench · 2 of 56 papers
- T2VQA-DB · 2 of 56 papers
- VBench prompts · 2 of 56 papers
Baselines
- VideoCrafter2 · 8 of 56 papers
- CogVideoX-5B · 7 of 56 papers
- AnimateDiff · 4 of 56 papers
- LaVie · 4 of 56 papers
- Pika · 4 of 56 papers
- FreeNoise · 3 of 56 papers
- CLIPScore · 3 of 56 papers
- I2VGen-XL · 3 of 56 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 156 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 10 papers
- Google · 9 papers
- Peking University · 8 papers
- Shanghai Jiao Tong University · 7 papers
- University of California, Berkeley · 7 papers
- The Chinese University of Hong Kong · 7 papers
- University of Science and Technology of China · 6 papers
- Zhejiang University · 6 papers
- University of Rochester · 5 papers
- ByteDance · 5 papers
Use it
Download BibTeX of 27 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient video diffusion models (283 papers)
- Trajectory-guided motion control (184 papers)
- Autoregressive long video generation (156 papers)
- World-model video generation (110 papers)
Elsewhere in the topic
- Text-to-motion diffusion (140 papers)
- Diffusion-based human image animation (126 papers)
- Diffusion-based video editing (92 papers)
- Talking face generation (81 papers)
- Video-to-audio synthesis (76 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).