Research map / Image generation / Autoregressive and latent generation
SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SongGen, a single-stage autoregressive transformer over X-Codec tokens that generates songs (vocals plus accompaniment) from lyrics, text descriptions and an optional 3-second voice prompt, with mixed and dual-track output modes.
Problem
Text-to-song generation is hard due to domain complexity and scarce open paired data (audio, lyrics, captions); existing multi-stage pipelines accumulate errors and are cumbersome to train and run.
Approach
A single autoregressive transformer decoder jointly models vocal and accompaniment audio codec tokens, with an auxiliary vocal-token prediction loss (Mixed Pro) for mixed output and an interleaved vocal/accompaniment token pattern (Interleaving A-V) for dual-track output, conditioned via cross-attention on VoiceBPE lyrics, MERT voice features and FLAN-T5 text.
Key results
- Mixed Pro achieves FAD 1.71 and PER 40.58 versus 2.18 and 38.80 for the multi-stage baseline, with higher aesthetics scores (e.g., PQ 7.19 vs 6.69).
- In human MOS, Mixed Pro gets OVL 3.96 and HAM 4.01 versus 3.39 and 2.97 for the multi-stage baseline; Interleaving (A-V) reaches VQ 4.15.
- Mixed Pro generates 30-second samples in 18.04 s on an A800 versus 42.85 s for the multi-stage baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use single-stage generation instead of the multi-stage pipeline | SongGen Mixed pro · MusicCaps test | FAD: 2.18 -> 1.71 | Helped |
| use single-stage generation instead of the multi-stage pipeline | SongGen Mixed pro · MusicCaps test | PER: 38.80 -> 40.58 | Hurt |
| add an auxiliary vocal token prediction target | SongGen Mixed mode · MusicCaps test | PER: 51.84 -> 40.58 | Helped |
| add an auxiliary vocal token prediction target | SongGen Mixed mode · MusicCaps human evaluation | VQ: 3.55 -> 4.07 | Helped |
| use Interleaving (A-V) rather than Parallel (A-V) | SongGen dual-track · MusicCaps test | FAD: 2.31 -> 1.87 | Helped |
| place accompaniment before vocals rather than vocals before accompaniment | SongGen dual-track · MusicCaps test | FAD: 1.96 -> 1.87 | Helped |
| remove high-quality fine-tuning (ablation) | SongGen Mixed pro · MusicCaps test | FAD: 1.71 -> 2.01 | Hurt |
| remove high-quality fine-tuning (ablation) | SongGen Mixed pro · MusicCaps test | PER: 40.58 -> 43.68 | Hurt |
| remove curriculum learning for codebook loss weights (ablation) | SongGen Mixed pro · MusicCaps test | FAD: 1.71 -> 2.35 | Hurt |
| remove curriculum learning for codebook loss weights (ablation) | SongGen Mixed pro · MusicCaps test | PER: 40.58 -> 55.71 | Hurt |
Surprising findings
- Placing accompaniment before vocals (A-V) gives better results than vocals before accompaniment, for both parallel and interleaving patterns.
- Fine-tuning Parler-tts (a TTS model) on the training data performs poorly for text-to-song, showing song generation is harder than speech synthesis.
Limitations
- Generates only up to 30-second songs, insufficient for complete song structure.
- Uses X-Codec at 16 kHz sampling rate, limiting fidelity.
- Trained on only about 2K hours of publicly sourced data, leaving gaps versus ground truth and Suno.
Experimental setup
Datasets: Million Song Dataset Free Music Archive MTG-Jamendo MusicCaps LP-MusicCaps-MSD
Benchmarks: MusicCaps (326 English-voiced test clips)
Metrics: FAD KL CLAP CLaMP3 PER SECS CE CU PC PQ
Baselines: Parler-tts Multi-Stage pipeline Suno Ground Truth
Builds on: MusicGen X-Codec Parler-tts MERT
Closest papers by meaning
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.92)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.90)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.90)
- CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · similarity 0.89)
- SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training (AAAI 2025 · similarity 0.89)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.