Research map / Large language models / Agents, code and math
MuPT: A Generative Symbolic Music Pretrained Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a family of decoder-only Transformers (190M to 4.23B) on ABC notation with a synchronized multi-track reordering (SMT-ABC) and proposes the SMS Law, a modified scaling law for repeated symbolic music data.
Problem
Symbolic music generation with LLMs struggles with bar misalignment across tracks when ABC notation is serialized track by track, and existing scaling laws fit poorly for small, repeated music datasets.
Approach
Reorder multi-track ABC so that bars with the same index across all tracks are grouped together with a delimiter token (SMT-ABC), and fit an extended Chinchilla-plus-data-constrained scaling law (SMS Law) with an extra ND term, a GELU-gated overfitting term, and a continuous effective-data formulation.
Key results
- SMS Law achieves train R^2 0.9780 and test R^2 0.9612 (Huber 0.0028 on test), beating Chinchilla (test R^2 -0.0933) and Data-Constrained law.
- In human AB tests (N=150), listeners preferred MuPT over GPT-4 in 79% of cases (115/35, p=6.6641e-8) and over MMT (110/40, p=4.2249e-6).
- MuPT achieves intra-texture similarity 0.4288 and repetition rate 44.3%, versus ground truth 0.3729/43.5% and GPT-4 0.3614/16.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMS Law instead of the Chinchilla law | Symbolic music scaling-law fitting · MuPT test scaling-law fit | R²: -0.0933 -> 0.9612 | Helped |
| Use SMS Law instead of the Data-Constrained law | Symbolic music scaling-law fitting · MuPT test scaling-law fit | Huber loss: 0.0071 -> 0.0028 | Helped |
| Use MuPT rather than GPT-4 | First-bar conditioned music generation · First-bar conditioned generation | Texture similarity: 0.3614 -> 0.4288 | Helped |
| Use MuPT rather than GPT-4 | First-bar conditioned music generation · First-bar conditioned generation | Repetition Det. Rate (%): 16.9 -> 44.3 | Helped |
| Use MuPT rather than MMT | First-bar conditioned music generation · First-bar conditioned generation | Texture similarity: 0.1767 -> 0.4288 | Helped |
| Use MuPT rather than GPT-4 in paired listening comparisons | MuPT · MuPT paired listening evaluation | Listener wins: 115/35 | Helped |
| Use MuPT rather than MMT in paired listening comparisons | MuPT · MuPT paired listening evaluation | Listener wins: 110/40 | Helped |
| Use GELU instead of ReLU in the overfitting regularization term (ablation) | SMS Law · SMS Law test scaling-law fit | Huber loss: 0.0095 -> 0.0085 | Helped |
| Use GELU instead of LeakyReLU in the overfitting regularization term (ablation) | SMS Law · SMS Law test scaling-law fit | Huber loss: 0.0095 -> 0.0085 | Helped |
| Use GELU instead of Tanh in the overfitting regularization term (ablation) | SMS Law · SMS Law test scaling-law fit | Huber loss: 0.0094 -> 0.0085 | Helped |
| Use GELU instead of SELU in the overfitting regularization term (ablation) | SMS Law · SMS Law test scaling-law fit | Huber loss: 0.010 -> 0.0085 | Helped |
| Use GELU instead of Sigmoid in the overfitting regularization term (ablation) | SMS Law · SMS Law test scaling-law fit | Huber loss: 0.0700 -> 0.0085 | Helped |
Surprising findings
- Chinchilla Law gives negative test R^2 (-0.0933) on the symbolic music data, indicating poor generalization for repeated small data.
- Human listeners rated MuPT statistically indistinguishable from human-composed ground truth (81/69, p=0.4237).
Limitations
- Synchronized reordering skips about 0.01% of samples with unequal track lengths.
- Human evaluation relies on pairwise AB tests with a limited number of comparisons (N=150).
- Scaling law fit uses a limited set of model sizes (190M to 1.07B) and includes ad hoc terms.
Experimental setup
Datasets: ABC text corpus (33.6B tokens)
Benchmarks: validation loss scaling fits repetition rate intra-texture similarity human AB listening test
Metrics: R^2 Huber loss repetition detection rate intra-similarity pairwise win rate Wilcoxon signed-rank p-value
Baselines: GPT-4 Multitrack Music Transformer (MMT) Chinchilla Law Data-Constrained Law ChatMusician
Builds on: Chinchilla scaling law Data-Constrained scaling law LLaMA2 ChatMusician
Closest papers by meaning
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.90)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.89)
- Text2midi: Generating Symbolic Music from Captions (AAAI 2025 · similarity 0.89)
- Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.89)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.89)
- An Independence-promoting Loss for Music Generation with Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.