Research map / Image generation / Autoregressive and latent generation
CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSL-L2M, a song-level lyric-to-melody Transformer decoder with in-attention conditioning on lyric, tag, statistical and VQ-VAE musical controls, trained on a new 10,170-song aligned Chinese pop dataset.
Problem
Lyric-to-melody generation at full-song level suffers from weak lyric-melody correlation learning, limited controllability, strict syllable-note alignment handling, and poor musical structure.
Approach
Introduces REMI-Aligned representation with syllable- and sentence-level lyric-melody alignment tokens, and feeds multi-granularity lyric controls (sentence semantic embeddings, POS, tone) and musical controls (human tags, sentence-level statistics, VQ-VAE learned features) into an autoregressive Transformer decoder via in-attention.
Key results
- CSL-L2M achieves PD 86.35%, DD 93.50%, MD 1.27 versus TeleMelody (40.02%, 49.82%, 2.93) and SongComposer (33.37%, 44.98%, 3.12) on objective metrics.
- Subjective quality score 3.94 for CSL-L2M versus 2.72 (TeleMelody) and 2.47 (SongComposer) on a 1-5 scale with 70 participants.
- Spearman rho for pitch mean control reaches 0.98, with low cross-attribute correlation (e.g., 0.09 between Align and PM), indicating independent controllability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned musical controls (ablation) | CSL-L2M · Chinese pop songs test set | PD (%): 86.35 -> 97.98 | Helped |
| add duration-related controls (ablation) | CSL-L2M · Chinese pop songs test set | DD (%): 86.41 -> 93.50 | Helped |
| add pitch-related controls alongside duration controls (ablation) | CSL-L2M · Chinese pop songs test set | PD (%): 66.07 -> 86.35 | Helped |
| add note density and alignment controls alongside duration and pitch controls (ablation) | CSL-L2M · Chinese pop songs test set | DD (%): 63.59 -> 93.50 | Helped |
| add musical controls (ablation) | CSL-L2M · Chinese pop songs test set | PD (%): 49.20 -> 86.35 | Helped |
| use CSL-L2M instead of SongComposer | Chinese pop songs test set · Chinese pop songs test set | Quality: 2.47 -> 3.94 | Helped |
| use CSL-L2M instead of TeleMelody | Chinese pop songs test set · Chinese pop songs test set | Structure: 2.45 -> 4.20 | Helped |
Surprising findings
- Removing all musical controls drops PD from 86.35% to 49.20%, showing musical controls help melody modeling beyond user control.
Limitations
- Learned VQ-VAE musical controls require existing melodies, so the full version is not used for main comparisons.
- Evaluation is on a private Chinese pop dataset, limiting generality to other languages.
- Subjective evaluation uses only 10 songs.
Experimental setup
Datasets: 10,170 Chinese pop songs with aligned lyrics and melodies (new dataset)
Benchmarks: own 10,170-song Chinese pop test set
Metrics: Pitch Distribution Similarity (PD) Duration Distribution Similarity (DD) Melody Distance (MD) subjective harmony, rhythm, structure, emotion, quality (1-5 Spearman rank correlation for controllability
Baselines: TeleMelody SongComposer
Builds on: REMI in-attention (MuseMorphose) SongMASS TeleMelody
Closest papers by meaning
- SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training (AAAI 2025 · similarity 0.92)
- S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation (AAAI 2025 · similarity 0.90)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.89)
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.88)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.88)
- Text2midi: Generating Symbolic Music from Captions (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.