Research map / Large language models / Language, safety and interpretability
SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SongGLM, a GLM-based lyric-to-melody model with a unified word/note token representation using 2D (word and phrase) alignment encoding, pre-trained with harmonized n-gram, phrase, and long-span blank infilling on a 206,884-song English lyric-melody dataset.
Problem
Lyric-to-melody generation struggles with accurate one-to-multiple lyric-note alignment and with lyric-melody harmony, where prior methods rely on intermediate templates or strict rules that limit diversity.
Approach
A unified symbolic song representation with word-level and phrase-level alignment IDs (2D alignment encoding), plus multi-task GLM-style blank-infilling pre-training whose n-grams are selected by syllable-stress/melodic-peak and syllable-stress/rhythm-skeleton relationships.
Key results
- SongGLM-small reaches D_A 94.32% and MD 4.17 versus TeleMelody's MD 6.36 and ReLyMe-in-TeleMelody's 5.90 on the objective metrics.
- SongGLM-base achieves the best objective scores, with D_A 96.83% and D_P 96.50%.
- In a 10-participant listening test, SongGLM-base obtains an overall score of 7.79 versus 6.95 for ReLyMe-in-TeleMelody.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 2D alignment encoding (ablation) | SongGLM_small with CLM pre-training · Lyric-to-melody test set | D_A (%): 83.57 $\pm$ 0.89 -> 90.20 $\pm$ 0.66 | Helped |
| Add the syllable stress–melodic peak relationship (SMR) to harmonized n-gram sampling (ablation) | SongGLM_small · Lyric-to-melody test set | D_P (%): 91.77 $\pm$ 0.78 -> 94.11 $\pm$ 0.67 | Helped |
| Add the syllable stress–rhythm skeleton relationship (SRR) to harmonized n-gram sampling (ablation) | SongGLM_small · Lyric-to-melody test set | D_IOI (%): 89.04 $\pm$ 1.10 -> 91.88 $\pm$ 1.05 | Helped |
| Use multi-task pre-training instead of harmonized n-gram-only, phrase-only, or long-span-only pre-training (ablation) | SongGLM_small · Lyric-to-melody test set | D_P (%): Harmonized N-gram: 94.11 $\pm$ 0.67; Phrase: 94.09 $\pm$ 0.66; Long: 94.24 $\pm$ 0.63; multi-task SongGLM_small: 96.48 $ | Mixed |
| Use harmonized n-gram sampling instead of random span sampling (ablation) | SongGLM_small · Lyric-to-melody test set | D_P (%): 92.84 $\pm$ 0.65 -> 94.11 $\pm$ 0.67 | Helped |
| Add pre-training and 2D alignment encoding instead of training from scratch without either (ablation) | SongGLM_small · Lyric-to-melody test set | D_A (%): 84.15 $\pm$ 0.69 -> 94.32 $\pm$ 0.64 | Helped |
Limitations
- Evaluated only on English lyrics and a dataset built from MIDI songs
- Subjective evaluation uses only 10 participants and 15 samples per model
- Relies on the CMU Pronouncing Dictionary, restricting words to those in it
Experimental setup
Datasets: MelodyNet CMU Pronouncing Dictionary
Benchmarks: lyric-melody paired test set
Metrics: D_A alignment distribution similarity D_P pitch distribution similarity D_D duration distribution similarity D_IOI distribution similarity melody distance (DTW) subjective 10-point listening scores
Baselines: SongMASS TeleMelody ReLyMe (in SongMASS) ReLyMe (in TeleMelody) Scratch CLM
Builds on: GLM OctupleMIDI MASS ReLyMe
Closest papers by meaning
- CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · similarity 0.92)
- S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation (AAAI 2025 · similarity 0.90)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.89)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.89)
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.87)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.