Research map / Image generation / Autoregressive and latent generation
SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SongBloom, a unified autoregressive-diffusion model that interleaves AR sketch (MuQ-embedding token) generation with patch-wise continuous-latent diffusion to produce full-length lyric-to-song generation up to 150 seconds.
Problem
Lyric-to-song generation with coherent long-form structure, accurate lyric alignment, and high-fidelity vocals and accompaniment; prior NAR diffusion models struggle with phoneme-frame alignment and AR codec-token models have low musicality and fidelity.
Approach
Decouples high-level sketch planning from acoustic synthesis and interleaves them patch-by-patch in one jointly optimized model: an LM generates SSL-based sketch tokens and hidden vectors conditioned on prior acoustic latents, and a patch-wise DiT diffuses continuous acoustic latents conditioned on the sketch as chain-of-thought-like guidance.
Key results
- SongBloom-full-ft achieves PER 5.49% and MCC 0.86 with FAD 3.20, beating Suno-v4.5 on PER-related and FAD metrics (Suno-v4.5 PER 24.67%, FAD 3.39).
- Subjective MOS: SongBloom-full-ft gets vocal musicality 3.91 and correctness 3.42, comparable to Suno-v4.5 (3.87 and 2.95).
- Ablation on SongBloom-tiny: removing sketch tokens raises PER from 9.44% to 109.76%, and the RTF of SongBloom-full is 1.649 versus 13.724 for YuE-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SongBloom-full for 1,000 steps on synthesized verse-chorus data | SongBloom-full · Lyric-to-song evaluation | PER (%): 6.75 -> 5.49 | Helped |
| Fine-tune SongBloom-full for 1,000 steps on synthesized verse-chorus data | SongBloom-full · Lyric-to-song evaluation | MCC: 0.88 -> 0.86 | Hurt |
| Fine-tune SongBloom-full for 1,000 steps on synthesized verse-chorus data | SongBloom-full · Lyric-to-song evaluation | FAD: 3.43 -> 3.20 | Helped |
| Fine-tune SongBloom-full for 1,000 steps on synthesized verse-chorus data | SongBloom-full · Lyric-to-song evaluation | SER (%): 17.67 -> 14.50 | Helped |
| Fine-tune SongBloom-full for 1,000 steps on synthesized verse-chorus data | SongBloom-full · Lyric-to-song evaluation | MUS vocal: 3.59 ± 0.25 -> 3.91 ± 0.24 | Helped |
| Add sketch tokens as a diffusion condition (ablation) | SongBloom-tiny with H+C conditions · SongBloom-tiny diffusion-condition ablation | PER (%): 109.76 -> 9.44 | Helped |
| Add sketch tokens as a diffusion condition (ablation) | SongBloom-tiny with H+C conditions · SongBloom-tiny diffusion-condition ablation | MUS | Helped |
| Add acoustic context from the previous patch (ablation) | SongBloom-tiny with H-only diffusion condition · SongBloom-tiny diffusion-condition ablation | FAD: 7.58 -> 5.60 | Helped |
| Add acoustic context from the previous patch (ablation) | SongBloom-tiny with H-only diffusion condition · SongBloom-tiny diffusion-condition ablation | PER (%): 11.35 -> 10.49 | Helped |
| Use pitch sketches instead of no sketch (ablation) | SongBloom-tiny · SongBloom-tiny sketch representation ablation | PER (%): 109.76 -> 103.22 | Helped |
| Add chroma features to pitch sketches (ablation) | SongBloom-tiny · SongBloom-tiny sketch representation ablation | PER (%): 103.22 -> 68.47 | Helped |
| Use SSL embedding sketches instead of pitch + chroma (ablation) | SongBloom-tiny · SongBloom-tiny sketch representation ablation | PER (%): 68.47 -> 9.44 | Helped |
Surprising findings
- Without sketch tokens the model cannot learn phoneme-audio alignment, yielding PER above 100%.
- Suno-v4.5 achieves the lowest structural error rate (10.43%) but much higher PER than SongBloom, attributed to redundant chorus repetition.
Limitations
- SSL-based sketch representation lacks interpretability and symbolic control
- No RL-based preference alignment (DPO/PPO) yet applied
- Evaluation relies on a small 20-sample human study and an in-house test set
Experimental setup
Datasets: 100K-hour in-house song dataset (Chinese and English), cleaned with Demucs, WhisperX, and a structure analyzer GPT-4o generated test lyrics synthesized verse-chorus data for fine-tuning
Benchmarks: in-house 20-sample test set with GPT-4o lyrics and 10s reference clips
Metrics: PER MuLan cycle consistency FAD SER Audiobox-Aesthetic (CE, CU, PC, PQ) RTF MOS (MUS, QLT, CRR, CST)
Baselines: Suno-v4.5 Udio-v1.5 Haimian Mureka-O1 ACE-step YuE-7B DiffRhythm-full SongEditor
Builds on: MusicLM DiT rectified flow matching stable-audio-vae MuQ
Closest papers by meaning
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.90)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.89)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.88)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.88)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.88)
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.