Research map / Probabilistic and scientific modeling / Variational and generative sampling
An Independence-promoting Loss for Music Generation with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an MMD-based independence-promoting loss in an RKHS that regularizes the EnCodec tokenizer so that the factorized (parallel/delay) decoding of MusicGen-style language models better approximates the joint token distribution.
Problem
Multi-codebook audio tokens force language models either to model the joint distribution (costly, many autoregressive steps) or the product of codebook marginals (fast but exact only if codebooks are independent). The paper targets reducing inter-codebook statistical dependence in the codec latent s
Approach
Use a batch-based unbiased MMD estimator in a multi-scale Gaussian RKHS as a differentiable proxy for total correlation, computed between joint codes and shuffled (approximately factorized) codes, added as an auxiliary loss to EnCodec training; adapt it to the delay decoding pattern so it targets the time-delayed token structure used by the language model.
Key results
- On MusicCaps, MusicGen-MMD (300M) reaches FAD_vgg 2.98, FAD_MERT 1.45 and KL 1.18, beating MusicGen (3.60, 1.57, 1.22) and the diffusion baselines, with MUSHRA-style OVRL 74.75 vs 62.54 for MusicGen.
- On the held-out test set, the MMD loss with delay adaptation lowers FAD_vgg from 0.95 (delay) to 0.59, outperforming the flatten joint-distribution model (0.69) while keeping the frame rate.
- Total correlation between codebooks drops from 5.1e-2 (RVQ) to 4.8e-2 with MMD on EnCodec; the gain in reconstruction is not reported as significant, and the effect is smaller than the FAD gain suggests.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add independence-promoting MMD loss to the tokenizer | MusicGen · MusicCaps | FAD_vgg: 3.60 -> 2.98 | Helped |
| Add independence-promoting MMD loss to the tokenizer | MusicGen · MusicCaps | OVRL: 62.54 ± 3.68 -> 74.75 ± 3.68 | Helped |
| Adapt MMD optimization to the delay decoding strategy (ablation) | MusicGen with delay decoding · Held-out test set | FAD_vgg: 0.90 -> 0.59 | Helped |
| Adapt MMD optimization to the delay decoding strategy (ablation) | MusicGen with delay decoding · Held-out test set | KL: 0.45 -> 0.46 | Hurt |
| Replace the multi-scale Gaussian kernel with the squared inverse kernel (ablation) | EnCodec tokenizer · Internal dataset | MSMelSpec: 0.107 -> 0.127 | Hurt |
| Replace RVQ with PVQ plus hierarchical dropout (ablation) | MusicGen tokenizer · Held-out test set | FAD_vgg: 0.97 -> 1.26 | Hurt |
| Replace RVQ with PVQ (ablation) | MusicGen tokenizer · Held-out test set | FAD_vgg: 0.97 -> 1.66 | Hurt |
Surprising findings
- The MMD loss improves generation FAD_vgg more than it changes total correlation, with the decoding-matched (delay) variant giving the largest gain.
- Flattened joint-distribution modelling underperforms the factorized delay model with MMD on FAD_vgg, attributed to training difficulty.
Limitations
- The Gaussian kernel bandwidths are fixed rather than learned, and kernel choice was only partially explored.
- Large batch sizes (1280 samples per GPU via macro-batching) are needed for low-variance MMD estimates, raising memory cost.
- Subjective MUSHRA test used only 11-12 annotators, and the MTurk results show only marginal differences between methods.
Experimental setup
Datasets: internal music dataset ShutterStock music Pond5 music MusicCaps FMA-Pop
Benchmarks: MusicCaps
Metrics: FAD (CLAP-LAION, MERT-4, VGGish) KL divergence (PaSST) CLAP score MUSHRA-style OVRL MTurk OVRL and REL total correlation MSSpec loss
Baselines: MusicGen AudioLDM AudioLDM2-Music Mustango MusicLM Noise2Music UniAudio
Builds on: EnCodec MusicGen maximum mean discrepancy delay pattern
Closest papers by meaning
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.87)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.87)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.87)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.