Research map / Image generation / Autoregressive and latent generation
Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a four-level hierarchical symbolic music language and a cascaded diffusion model (one 2D-UNet per level, top-down, each conditioned on upper levels) that generates whole pop songs with verse-chorus structure.
Problem
Whole-song symbolic music generation with well-formed long-term structure (verse-chorus form, phrases, cadences) remains unsolved; end-to-end models scale poorly with song length.
Approach
Define a computable compositional hierarchy (Form, Reduced Lead Sheet, Lead Sheet, Accompaniment) rendered as image-like piano rolls, and generate each level with a cascaded diffusion model conditioned on upper-level background, autoregressive segment retrieval, and optional pre-trained latent external controls.
Key results
- ILS for chord (1.32) and accompaniment texture (1.19) exceeds baselines Polyff.+ph.l. (0.52, 0.61) and TFxl(REMI)+ph.l. (0.68, 0.74).
- Survey with 57 participants: the model consistently outperforms baselines on music quality and whole-song structural metrics, especially accompaniment.
- Degree-of-copying analysis shows the model's similarity to training set (0.6530) close to the test set (0.6567), indicating low copying risk.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cascaded diffusion instead of Polyffusion with phrase labels (ablation) | Whole-song generation of symbolic music · POP909 full songs | ILS chord: 0.52± 0.06 -> 1.32± 0.05 | Helped |
| Use cascaded diffusion instead of Polyffusion with phrase labels (ablation) | Whole-song generation of symbolic music · POP909 full songs | ILS accompaniment texture: 0.61± 0.04 -> 1.19± 0.06 | Helped |
| Use cascaded diffusion instead of Transformer-XL with phrase labels (ablation) | Whole-song generation of symbolic music · POP909 full songs | ILS chord: 0.68± 0.06 -> 1.32± 0.05 | Helped |
| Use cascaded diffusion instead of Transformer-XL with phrase labels (ablation) | Whole-song generation of symbolic music · POP909 full songs | ILS accompaniment texture: 0.74± 0.04 -> 1.19± 0.06 | Helped |
| Add external chord condition | Cascaded diffusion, Reduced Lead Sheet generation · Chord control | Rule-based distance: 2.09± 0.80 -> 1.73± 1.02 | Helped |
| Add external chord condition | Cascaded diffusion, Reduced Lead Sheet generation · Chord control | Latent similarity: 0.37± 0.09 -> 0.48± 0.14 | Helped |
| Add external rhythm condition | Cascaded diffusion, Lead Sheet generation · Rhythm control | Rule-based distance: 2.27± 0.53 -> 1.10± 0.74 | Helped |
| Add external rhythm condition | Cascaded diffusion, Lead Sheet generation · Rhythm control | Latent similarity: 0.14± 0.23 -> 0.75± 0.16 | Helped |
| Add external texture condition | Cascaded diffusion, Accompaniment generation · Texture control | Rule-based distance: 3.94± 1.46 -> 0.87± 0.80 | Helped |
| Add external texture condition | Cascaded diffusion, Accompaniment generation · Texture control | Latent similarity: 0.02± 0.11 -> 0.89± 0.06 | Helped |
| Use cascaded diffusion rather than Polyffusion with phrase labels (ablation) | Whole-song music generation · POP909 generated pieces | Latent pitch similarity to training set: 0.8639± 0.0487 -> 0.8743± 0.0491 | Mixed |
| Use cascaded diffusion rather than phrase-labeled baselines | Whole-song music generation · POP909 generated pieces | Rule-based melody similarity to training set: 0.6567± 0.1141 -> 0.6530± 0.1321 | No clear effect |
Limitations
- Evaluation of whole-song samples limited to 32 measures in the subjective survey due to participant burden.
- Focused on single-track pop songs (lead sheet and piano accompaniment); multi-track not addressed.
- Higher-level Form language is hard to evaluate directly, so experiments focus on lower levels.
Experimental setup
Datasets: POP909
Benchmarks: POP909 test split
Metrics: Inter-Phrase Latent Similarity (ILS) subjective 5-point ratings (creativity, naturalness, musical control efficacy (rule-based distance, latent similarity) degree of copying similarity
Baselines: Polyffusion + phrase label control (Polyff.+ph.l.) Transformer-XL REMI + phrase label control (TFxl(REMI)+ph.l.) Polyffusion TFxl(REMI)
Builds on: Polyffusion Mousai classifier-free guidance REMI EC2-VAE
Closest papers by meaning
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.90)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.89)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.89)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.88)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.88)
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.