Research map / Image generation / Autoregressive and latent generation
SongCreator: Lyrics-based Universal Song Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SongCreator, a dual-sequence language model with bidirectional cross-attention and task-specific attention masks that generates songs (vocals plus accompaniment) from lyrics and supports multiple song editing and conditioning tasks.
Problem
Generating complete songs with both vocals and accompaniment from lyrics remains hard, and existing methods model vocals and accompaniment jointly, handle only one task, or cannot edit songs.
Approach
A dual-sequence LM with separate vocal and accompaniment decoders linked by a bidirectional cross-attention layer, a song decoder over their embeddings, and a family of self-attention and cross-attention masks that unify generation, continuation and editing tasks in one model.
Key results
- Lyrics-to-song FAD 2.14 vs 2.31 for MusicGen and 6.47 for MusicLM; musicality 4.25 vs 4.30 ground truth.
- Vocals-to-song and accompaniment-to-song are competitive with SingSong, with FAD 1.88 and 1.24 respectively, though SingSong was preferred 54.1% to 30% in a preference test.
- Lyrics-to-song is preferred over Jukebox 60% of the time in an AB preference test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with multiple tasks rather than a single task (ablation) | SongCreator · Lyrics-to-song and most evaluated tasks | performance | Helped |
| add bidirectional cross-attention between vocal and accompaniment decoders (ablation) | SongCreator DSLM · Lyrics-to-song | human preference | Helped |
| add bidirectional cross-attention between vocal and accompaniment decoders (ablation) | SongCreator DSLM · Lyrics-to-vocals | human preference | Helped |
| add non-causal self-attention masks during training (ablation) | SongCreator DSLM · Lyrics-to-song, vocals-to-song, and accompaniment-to-song | human preference | Helped |
| use the BR BCA mask instead of A2V, V2A, or None (ablation) | SongCreator · Lyrics-to-song | human preference | Helped |
| use the BR BCA mask instead of the None mask (ablation) | SongCreator · Lyrics-to-song | human preference | Helped |
| use the A2V BCA mask instead of the BR mask (ablation) | SongCreator · Accompaniment-to-song | human preference | Helped |
| consider the vocal-accompaniment relationship instead of generating vocals with the vocal decoder alone (ablation) | SongCreator · Lyrics-to-vocals | Musicality: 3.68 -> 3.98 | Helped |
| use multi-task SongCreator instead of SongCreator (Single) (ablation) | SongCreator · Lyrics-to-song, no prompt | Musicality: 3.85\pm 0.06 -> 4.25\pm 0.05 | Helped |
Surprising findings
- SingSong was preferred over SongCreator (54.1% vs 30%) in the vocals-to-song preference test, which the authors attribute to SingSong's 46k-hour training set.
- Removing the bidirectional cross-attention also degraded lyrics-to-vocals, suggesting vocal-accompaniment relations help vocal generation.
Limitations
- Cannot control genre or style through text descriptions.
- Vocal clarity is limited because accompaniment interference hampers BEST-RQ encoding of vocals.
- Generates only up to 30 seconds, so full song structures are not supported.
Experimental setup
Datasets: In-house song-with-lyrics corpus (8,500 hours) Manually constructed 30-example song editing set
Benchmarks: In-house lyrics-to-song, lyrics-to-vocals, vocals-to-song, accompaniment-to-song, music continuation, and editing test s
Metrics: FAD MCD SECS MOS musicality MOS quality MOS harmony MOS naturalness AB preference
Baselines: MusicLM MusicGen GPT VALL-E SingSong SingSong (Diffusion) AudioLM Jukebox SongCreator (Single)
Builds on: BEST-RQ AudioLM UniLM GLM Demucs
Closest papers by meaning
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.92)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.89)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.88)
- CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · similarity 0.88)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.87)
- S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.