Research map / Image generation / Autoregressive and latent generation
Controllable music generation
Generate symbolic and audio music with hierarchical, autoregressive, and diffusion models.
42 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Autoregressive and latent generation, keyword cluster “speech · music · audio”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
12 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.5% to 2.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · DeepBach: a Steerable Model for Bach Chorales Generation (ICML 2017 · 125 citations)
- 2018 · The challenge of realistic music generation: modelling raw audio at scale (NeurIPS 2018 · 116 citations)
- 2019 · Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (ICLR 2019 · 149 citations)
- 2020 · An Iterative Polishing Framework Based on Quality Aware Masked Language Model for Chinese Poetry Generation (AAAI 2020 · 22 citations)
- 2021 · Symbolic Music Generation with Transformer-GANs (AAAI 2021 · 64 citations)
- 2022 · A Brand New Dance Partner: Music-Conditioned Pluralistic Dancing Controlled by Multiple Dance Genres (CVPR 2022 · 55 citations)
- 2023 · EDGE: Editable Dance Generation From Music (CVPR 2023 · 238 citations)
- 2024 · MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · 14 citations)
- 2025 · JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · 6 citations)
- 2026 · Steering Autoregressive Music Generation with Recursive Feature Machines (ICLR 2026 · 1 citation)
Earliest papers here
- DeepBach: a Steerable Model for Bach Chorales Generation (ICML 2017 · 125 citations)
- The challenge of realistic music generation: modelling raw audio at scale (NeurIPS 2018 · 116 citations)
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (ICLR 2019 · 149 citations)
What has been tried
34 papers here report 356 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 6%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add image conditioning through the visual synapse to text conditioning | Extended MusicCaps · FAD | 3.07 -> 1.12 | Helped | MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · 14 citations) |
| add Hokkien speech pre-training | Gezi Opera · naturalness MOS | 4.08 ± 0.03 -> 4.18 ± 0.03 | Helped | FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis (AAAI 2024 · 9 citations) |
| Add individual timestep vectors for each track | Ablation study · CLAP (Melody) | 0.28 -> 0.32 | Helped | JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · 6 citations) |
| Add force-smoothing training | Segment-wise editing · PER (%) | 25.68 -> 20.82 | Helped | SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · 4 citations) |
| increase the SCG sample count per step from 4 to 16 | Note density · loss | 0.318 ± 0.770 -> 0.131 ± 0.325 | Helped | Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion (ICML 2024 · 4 citations) |
| add learned musical controls | Chinese pop songs test set · PD (%) | 86.35 -> 97.98 | Helped | CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · 3 citations) |
| train with multiple tasks rather than a single task | Lyrics-to-song and most evaluated tasks · performance | Helped | SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · 2 citations) | |
| Add JEP-AGA | music infilling · objective distribution overlap | PDp: 0.74 -> 0.83; CDf: 0.80 -> 0.88 | Helped | Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling (AAAI 2025 · 1 citation) |
| add accompaniment conditioning | RapBank vocals mixed with accompaniment · MOS-M | 3.19 -> 3.84 | Helped | Drop the Beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation (AAAI 2025 · 1 citation) |
| Replace SongSong's rhythm-based lyric-melody alignment with GSV-style speech synthesis architecture | SongCi clips, ablation evaluation · FAD_vgg | 7.22 -> 20.30 | Hurt | SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away (AAAI 2025 · 1 citation) |
| Add predicted lyric length | MetaMIDI test set · SDD | 7.34±5.09 -> 4.58±4.59 | Helped | S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation (AAAI 2025 · 1 citation) |
| remove high-quality fine-tuning | MusicCaps test · FAD | 1.71 -> 2.01 | Hurt | SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 14 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Pop909 · 2 of 14 papers
- Slakh2100 · 2 of 14 papers
- MetaMIDI · 2 of 14 papers
Baselines
- SongComposer · 2 of 14 papers
- Suno · 2 of 14 papers
- Ground Truth · 2 of 14 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 31 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 5 papers
- The Hong Kong Polytechnic University · 2 papers
- Microsoft · 2 papers
- Tencent · 2 papers
- Tsinghua University · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Shenzhen Research Institute of Big Data · 2 papers
- Stanford University · 2 papers
- New York University · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Neural text-to-speech synthesis (72 papers)
- Diffusion audio synthesis (52 papers)
Elsewhere in the topic
- Classical generative image synthesis (147 papers)
- High-resolution diffusion synthesis (140 papers)
- Retrieval and matching with generative features (140 papers)
- Generative latent space tooling (129 papers)
- Generalizable fake image detection (119 papers)
- Latent diffusion with learned tokenizers (114 papers)
- Synthetic training data from generators (111 papers)
- Compositional scene generation (105 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).