Research map / Label-efficient and robust learning / Self-supervised and contrastive
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MERT, a masked-prediction self-supervised music audio model that uses RVQ-VAE (EnCodec) acoustic teacher targets plus a CQT musical reconstruction loss, scaled to 95M and 330M parameters and evaluated on 14 MIR tasks under probing.
Problem
Open, general-purpose, and computationally affordable pre-trained models for acoustic music understanding are lacking; speech SSL recipes do not capture pitch and harmony in music, and scaling acoustic transformers is unstable.
Approach
HuBERT-style masked language modelling for music where the pseudo-labels come from multiple teachers: an EnCodec RVQ-VAE acoustic teacher (8 codebooks) predicted with cross-entropy, and a CQT spectrogram reconstructed with MSE as a musical teacher, combined with in-batch noise mixup and Pre-LN plus attention relaxation for stable scaling.
Key results
- MERT-330M (RVQ-VAE) reaches an average score of 64.7 across 14 tasks, matching the combined previous SOTA average of 64.5 and setting new SOTA on 4 metrics.
- MERT-95M (RVQ-VAE) with musical CQT loss and in-batch mixup reaches MTT ROC 91.0, GTZAN genre accuracy 78.6, and emotion R2 60.0/76.4 in the ablation.
- MERT uses 1.9% (95M) and 6.6% (330M) of Jukebox-5B's parameters while matching or beating it on most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a CQT musical teacher to K-means Logmel+Chroma acoustic targets (ablation) | MERT-95M · Giantsteps key detection | refined accuracy: 55.1 -> 65.0 | Helped |
| Add a CQT musical teacher to K-means Logmel+Chroma acoustic targets (ablation) | MERT-95M · Downstream tasks subset | average score: 62.1 -> 67.3 | Helped |
| Increase MFCC K-means target classes from 100 to 500 (ablation) | MERT-95M with MFCC acoustic teacher · Downstream tasks subset | average score: 49.4 -> 51.3 | Helped |
| Replace the MFCC acoustic teacher with Logmel+Chroma K-means targets (ablation) | MERT-95M · Giantsteps key detection | refined accuracy: 15.6 -> 55.1 | Helped |
| Use RVQ-VAE codebook 7 only instead of all codebooks (ablation) | MERT-95M with CQT teacher · Downstream tasks subset | average score: 66.9 -> 57.6 | Hurt |
| Use RVQ-VAE codebook 0 only instead of all codebooks (ablation) | MERT-95M with CQT teacher · Downstream tasks subset | average score: 66.9 -> 60.5 | Hurt |
| Randomly access one RVQ-VAE codebook per batch instead of predicting all together (ablation) | MERT-95M with CQT teacher · Downstream tasks subset | average score: 66.9 -> 65.8 | No clear effect |
| Set the musical loss weight to 1 rather than omit the musical loss (ablation) | MERT-95M K-means Logmel+Chroma · Giantsteps key detection | refined accuracy: 55.1 -> 65.0 | Helped |
| Increase musical loss weight from 1 to 2 (ablation) | MERT-95M K-means Logmel+Chroma · GTZAN genre classification | accuracy: 78.6 -> 66.9 | Hurt |
| Increase musical loss weight from 1 to 5 (ablation) | MERT-95M K-means Logmel+Chroma · GTZAN genre classification | accuracy: 78.6 -> 70.3 | Hurt |
| Add in-batch noise mixup with probability 0.5 (ablation) | MERT-95M RVQ-VAE · Emomusic valence | R2: 53.2 -> 60.0 | Helped |
| Add in-batch noise mixup with probability 0.5 (ablation) | MERT-95M K-means Logmel+Chroma · Downstream tasks subset | average score: 67.3 -> 65.2 | Hurt |
Surprising findings
- Higher transformer layers do not necessarily show stronger genre clustering, suggesting genre may not be the most abstract label or that top layers specialise to the MLM objective.
- Scaling to 330M gave an inverse-scaling effect on some specific tasks, and fp16 training remained prone to gradient explosions.
Limitations
- Trained only on 5-second clips, limiting performance on tasks needing long musical context.
- Training remains unstable in fp16 at larger batch and model sizes; gradient explosions persist.
- Evaluated only under a frozen probing protocol with a shallow MLP head, not full fine-tuning.
Experimental setup
Datasets: MagnaTagATune MTG-Jamendo Giantsteps Giantsteps-MTG-keys GTZAN MTG-Genre Emomusic Nsynth MTG-instrument VocalSet GTZAN Rhythm MUSDB18
Benchmarks: MagnaTagATune MTG-Jamendo GiantSteps key detection GTZAN genre MTG-Genre Emomusic NSynth MTG-instrument VocalSet GTZAN Rhythm MUSDB18
Metrics: ROC-AUC AP accuracy refined accuracy R2 F1 SDR
Baselines: MusiCNN CLMR MULE Jukebox-5B JukeMIR HuBERT-base music data2vec-base music
Builds on: HuBERT EnCodec data2vec DeepNorm attention relaxation
Closest papers by meaning
- SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes (ICLR 2025 · similarity 0.89)
- BEATs: Audio Pre-Training with Acoustic Tokenizers (ICML 2023 · similarity 0.87)
- LLark: A Multimodal Instruction-Following Language Model for Music (ICML 2024 · similarity 0.86)
- SALMONN: Towards Generic Hearing Abilities for Large Language Models (ICLR 2024 · similarity 0.85)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.85)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.