Research map / Video / Video generation
Video-to-audio synthesis
Generates synchronized soundtracks from video with latent diffusion and text conditioning.
76 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Video generation, keyword cluster “audio · editing · talking”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 58 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 4.8% to 5.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Predicting Latent Narrative Mood Using Audio and Physiologic Data (AAAI 2017 · 31 citations)
- 2020 · Audeo: Audio Generation for a Silent Performance Video (NeurIPS 2020 · 31 citations)
- 2021 · Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning (ICLR 2021 · 53 citations)
- 2022 · Audio-Driven Neural Gesture Reenactment With Video Motion Graphs (CVPR 2022 · 25 citations)
- 2023 · MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (CVPR 2023 · 148 citations)
- 2024 · Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · 35 citations)
- 2025 · MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · 29 citations)
- 2026 · DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing (ICML 2026 · 0 citations)
Earliest papers here
- Predicting Latent Narrative Mood Using Audio and Physiologic Data (AAAI 2017 · 31 citations)
- Audeo: Audio Generation for a Silent Performance Video (NeurIPS 2020 · 31 citations)
- Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning (ICLR 2021 · 53 citations)
What has been tried
37 papers here report 407 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 3%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Average all audio components into a single conditioning vector rather than using temporal windows | VGGSound · AV-Align | 0.35 -> 0.29 | Hurt | Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · 35 citations) |
| jointly train on audio-visual-text and audio-text data | VGGSound test · FD_PaSST | 77.38 -> 70.19 | Helped | MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · 29 citations) |
| remove long-term modeling | V2M-bench · KL | 0.738 -> 0.898 | Hurt | VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling (CVPR 2025 · 18 citations) |
| combine CLIP and I3D Flow visual features | MV100K · MCC | 0.343/0.419 | Helped | V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · 18 citations) |
| Add negative prompting | VGGSound-sync text-control evaluation · CLAP accuracy | 85.5 -> 93.2 | Helped | Video-Guided Foley Sound Generation with Multimodal Controls (CVPR 2025 · 16 citations) |
| replace estimated video energy with ground-truth audio energy | VGGSound · FD | 15.24 -> 13.93 | Helped | Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · 14 citations) |
| Add the LLM converter to audio-relevant visual conditioning | VGGSound test · KLD | 2.43 -> 2.25 | Helped | Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · 11 citations) |
| use reflow for 1-step sampling | VGGSound test, 1 step · Acc (%) | 43.18 -> 94.96 | Helped | Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching (NeurIPS 2024 · 8 citations) |
| Replace ViT with ViViT for visual feature extraction | Ablation test set · KLD | 1.07 -> 1.25 | Hurt | GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions (AAAI 2025 · 7 citations) |
| add the BPS network | AIST++ · BAS | 0.22 -> 0.33 | Helped | DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · 6 citations) |
| add teacher guidance loss | VinTAGe-Bench · FID | 19.48 -> 16.43 | Helped | VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · 5 citations) |
| Add R2-DiT refinement | Combined test set · Hand-C | 0.8579 -> 0.8832 | Helped | AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- VGGSound · 11 of 27 papers
- Greatest Hits · 4 of 27 papers
- AIST++ · 3 of 27 papers
- SymMV · 2 of 27 papers
- TikTok · 2 of 27 papers
- AVSync15 · 2 of 27 papers
- Landscape · 2 of 27 papers
- AudioSet · 2 of 27 papers
Baselines
- Diff-Foley · 9 of 27 papers
- SpecVQGAN · 6 of 27 papers
- Im2Wav · 5 of 27 papers
- V2A-Mapper · 4 of 27 papers
- FoleyCrafter · 4 of 27 papers
- CoDi · 3 of 27 papers
- Seeing&Hearing · 3 of 27 papers
- Frieren · 3 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 69 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 4 papers
- Alibaba Group · 4 papers
- Adobe · 3 papers
- University of Washington · 3 papers
- Peking University · 3 papers
- Tsinghua University · 3 papers
- University of Michigan · 2 papers
- Renmin University of China · 2 papers
- Baidu · 2 papers
- Hong Kong University of Science and Technology · 2 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Diffusion-based human image animation (126 papers)
- Diffusion-based video editing (92 papers)
- Talking face generation (81 papers)
- Text-to-motion diffusion (140 papers)
Elsewhere in the topic
- Efficient video diffusion models (283 papers)
- Trajectory-guided motion control (184 papers)
- Compositional text-to-video diffusion (159 papers)
- Autoregressive long video generation (156 papers)
- World-model video generation (110 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).