Research map / Image generation / Autoregressive and latent generation
S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S²MILE, an end-to-end music-to-lyric generator that combines hierarchical song-level and sentence-level music encoders with a lyric length predictor and a LoRA-tuned Mistral-7B decoder.
Problem
Generating singable lyrics directly from full music (not just melody) that are both structurally aligned (syllable/length format) and semantically consistent with the music's inherent meaning, without rigid hand-crafted rules or scarce expert-annotated note-syllable data.
Approach
A hierarchical music information extractor (Q-Former over HTS-AT audio for song-level semantics; T5 encoder pretrained on MetaMIDI notes for sentence-level structure) plus a note-based lyric length predictor, whose outputs are injected as prompts into a LoRA-tuned Mistral-7B lyric generator; evaluated with CLAP and CLaMP semantic similarity.
Key results
- S²MILE achieves CLaMP score 0.203 and CLAP score 0.057, the best among compared systems (SongMASS 0.163 / 0.042; SongComposer 0.145 / 0.023).
- Human evaluation with 17 musically trained participants gives S²MILE the highest meaningfulness (3.75) and coherence (3.60) among generated systems.
- Adding note similarity and instrument contrastive losses reduces the sentence-level extractor's hybrid MSE-CE pretraining loss from 2.4 to 1.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add predicted lyric length (ablation) | S²MILE · MetaMIDI test set | SDD: 7.34±5.09 -> 4.58±4.59 | Helped |
| Add song-level information (ablation) | S²MILE · MetaMIDI test set | CLAP Score: 0.041±0.101 -> 0.057±0.101 | Helped |
| Add sentence-level information (ablation) | S²MILE · MetaMIDI test set | SDD: 7.71±4.83 -> 4.58±4.59 | Helped |
| Add predicted lyric length (ablation) | S²MILE · MetaMIDI test set | CLAP Score: 0.058±0.101 -> 0.057±0.101 | No clear effect |
| Add note similarity loss and instrument contrastive loss during sentence-level extractor pretraining (ablation) | S²MILE sentence-level extractor · MetaMIDI | hybrid MSE-CE loss: 2.4 to 1.4 | Helped |
Surprising findings
- Removing the lyric length predictor slightly improves text quality (PPL and Dist metrics) while hurting structural alignment, suggesting a trade-off.
- SongMASS, despite using closed-source syllable-aligned data, falls well behind S²MILE on most metrics except format.
Limitations
- Generates lyrics sentence by sentence rather than complete song lyrics in one pass.
- Length predictor may over-focus on length at the cost of text quality.
- Evaluation relies on a small subjective study (17 participants, 50 pairs) and CLAP/CLaMP proxies.
Experimental setup
Datasets: MetaMIDI time-stamped lyrics from online sources
Benchmarks: MetaMIDI-derived test split (8:1:1)
Metrics: Dist-1 Dist-2 Dist-N PPL INT SDD SDD-S SDD-N CLAP score CLaMP score
Baselines: SongMASS GPT-3.5 w/ N GPT-3.5 w/ KN SongComposer
Builds on: BLIP-2 Q-Former HTS-AT T5 Mistral SongMASS LoRA
Closest papers by meaning
- SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training (AAAI 2025 · similarity 0.90)
- CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · similarity 0.90)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.88)
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.87)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.87)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.