Research map / Large language models / Alignment and preferences
MusicRL: Aligning Music Generation to Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes MusicLM with RL (REINFORCE with KL) using MuLan text-adherence, a learned acoustic quality reward, and a user-preference reward model trained on 300k AITK pairwise preferences, with the combined MusicRL-RU performing best in human side-by-side ratings.
Problem
Text-to-music models trained with next-token prediction lack a notion of musicality and text adherence, and music appreciation is subjective, so rater-based or automatic metrics only partially capture human preferences.
Approach
Apply KL-regularized REINFORCE to a MusicLM autoregressive stage using rewards from MuLan text-audio similarity, a quality MOS predictor, and a Bradley-Terry reward model trained on large-scale implicit user pairwise preferences collected via a public web interface, then combine them sequentially.
Key results
- MusicRL-R is preferred over MusicLM with an 83% win rate (win/(win+loss)).
- MusicRL-U achieves a 74% win rate against MusicLM, with the user preference reward model reaching 60% accuracy on held-out pairs.
- MusicRL-RU, the sequential combination, achieves an 87% win rate against MusicLM and 66% against MusicRL-R.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune MusicLM on combined MuLan and quality rewards | MusicLM · text-to-music generation | human preference win rate against MusicLM: 83% | Helped |
| fine-tune MusicLM on user preference reward | MusicLM · text-to-music generation | human preference win rate against MusicLM: 74% | Helped |
| add user preference reward fine-tuning after MuLan and quality reward fine-tuning | MusicRL-R · text-to-music generation | human preference win rate against MusicLM: 87% | Helped |
| add user preference reward fine-tuning after automatic reward fine-tuning | MusicRL-R · text-to-music generation | human preference win rate against MusicRL-R: 66% | Helped |
| combine automatic reward fine-tuning with user preference reward fine-tuning | MusicRL-RU · text-to-music generation | human preference win rate against MusicRL-U: 62% | Helped |
| fine-tune on quality reward alone | MusicLM · MusicLM RL fine-tuning | quality score (MOS): 3.5 MOS to 4.6 MOS | Helped |
| fine-tune on MuLan reward alone | MusicLM · MusicLM RL fine-tuning | MuLan score: 0.58 to 0.71 | Helped |
| fine-tune on combined MuLan and quality rewards | MusicLM · MusicLM RL fine-tuning | quality score (MOS): 3.5 MOS to 4.4 MOS | Helped |
| fine-tune on combined MuLan and quality rewards | MusicLM · MusicLM RL fine-tuning | MuLan score: 0.58 to 0.71 | Helped |
| fine-tune on user preference reward | MusicLM · MusicLM RL fine-tuning | quality score (MOS): 3.5 MOS to 4 MOS | Helped |
| fine-tune on user preference reward | MusicLM · MusicLM RL fine-tuning | MuLan score: 0.58 to 0.55 | Hurt |
| drop text tokens from reward model input (ablation) | User preference reward model · user preference pairwise evaluation | accuracy | No clear effect |
Surprising findings
- Text adherence measured by MuLan barely predicts user preference (matching the top-MuLan clip only 51.6% of the time), suggesting users do not primarily reward prompt adherence.
- The reward model was not better than chance when trained from scratch, so it had to be initialized from the MusicLM checkpoint.
Limitations
- Population gap between general users providing feedback and selected raters assessing results.
- User preference data was collected on a different MusicLM version than the one finetuned (off-policy).
- Reward model accuracy is modest (60%), and internal team agreement with dataset preferences was also 60%.
Experimental setup
Datasets: MusicCaps AITK user preference data LaMDA-generated music descriptions
Benchmarks: internal 101-prompt qualitative side-by-side evaluation
Metrics: win rate win/(win+loss) MOS MuLan score quality score reward model accuracy
Baselines: MusicLM
Builds on: MusicLM AudioLM SoundStorm MuLan REINFORCE
Closest papers by meaning
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction (ICML 2026 · similarity 0.91)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.88)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.88)
- LLark: A Multimodal Instruction-Following Language Model for Music (ICML 2024 · similarity 0.87)
- Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling (AAAI 2025 · similarity 0.87)
- MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.