Research map / Probabilistic and scientific modeling / Molecules and proteins
MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MusER, a VQ-VAE-based model whose latent sequence dimensions are regularized to encode individual musical elements, enabling element-level disentanglement, two-level decoding, and element transfer for emotional symbolic music generation.
Problem
Emotional symbolic music generation lacks explicit modeling of how individual musical elements (pitch, duration, velocity, etc.) contribute to emotion, and no fine-grained element-level control over emotion exists.
Approach
A musical element-based regularization loss on VQ-VAE latent vector sequences: a transformer dimensionality-reduction model maps each element's l-D latent sub-vectors to 1-D, and a pairwise sign-based MAE loss forces the latent variation patterns to match element variation patterns; a two-level decoder (global decoder plus per-element decoders) then reconstructs elements from disentangled latents.
Key results
- MusER (Trans_CA_Dec_G+eps) is not statistically different from real data (p>=0.05) on 3 of 6 objective metrics, with best overall p-value profile vs. CP Transformer and Transformer GAN.
- Velocity gives the highest silhouette score for arousal (Q1 vs Q4: 0.222; Q2 vs Q3: 0.341), while valence is hard to distinguish from any single element.
- In element transfer, replacing velocity latents from Q4 with Q1 latents led 90% of listeners to perceive an arousal change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MusER instead of CP Transformer | Emotional music generation · EMOPIA | B-PR: 27.09±8.52 -> 33.02±9.62 | Helped |
| Use MusER instead of Transformer GAN | Emotional music generation · EMOPIA | PR: 50.73 -> 53.47±11.50 | Hurt |
| Use MusER instead of Transformer GAN | Emotional music generation · EMOPIA | POLY: 4.43 -> 4.63±1.04 | Mixed |
| Use transformer-based DR instead of mean DR (ablation) | MusER · EMOPIA | PR: 50.58±11.09 -> 53.47±11.50 | Helped |
| Use cross-attention instead of concatenation to feed latent vectors to the decoder (ablation) | MusER · EMOPIA | NPC: 8.99±1.52 -> 8.57±1.47 | Helped |
| Add element decoders to the global decoder (ablation) | MusER · EMOPIA | NPC: 9.58±1.55 -> 8.57±1.47 | Helped |
| Add the global decoder to the element decoders (ablation) | MusER · EMOPIA | B-NPC: 4.94±0.88 -> 4.43±0.73 | Helped |
| Add MED regularization (ablation) | MusER · EMOPIA | B-PR: 29.15±8.61 -> 33.02±9.62 | Helped |
| Add MED regularization (ablation) | MusER · EMOPIA | PR: 49.26±10.56 -> 53.47±11.50 | Mixed |
| Transfer velocity latent vectors from Q1 into Q4 music | MusER · EMOPIA Q4 music | velocity distribution | Helped |
| Transfer pitch, duration, and velocity latent vectors from Q1 into Q3 music | MusER · EMOPIA Q3 music | arousal perception: 90% | Helped |
Surprising findings
- Valence is hard to distinguish using any single musical element, with best silhouette scores near 0 for Q1 vs Q2 and Q3 vs Q4.
- MusER scores lower than Transformer GAN on pitch range and lower on bar-level polyphony, and transferred music scored below non-transferred music in the listening test.
Limitations
- Parameter count mainly from the MED module, making MusER larger than baselines.
- Valence is not clearly distinguishable from individual elements, and transferred music is judged less favorably than non-transferred music.
- Arbitrary pitch transfer often leads to poor listenability.
Experimental setup
Datasets: Pop1k7 EMOPIA
Benchmarks: EMOPIA
Metrics: Pitch Range Number of Pitch Classes Polyphony bar-level PR/NPC/POLY p-value vs real data silhouette coefficient listening test ratings (humanness, richness, overall) emotion classification accuracy
Baselines: CP Transformer Transformer GAN
Builds on: VQ-VAE Compound Word Transformer linear transformer attribute regularization
Closest papers by meaning
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.88)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.87)
- Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling (AAAI 2025 · similarity 0.86)
- MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners (ICML 2025 · similarity 0.86)
- MID-FiLD: MIDI Dataset for Fine-Level Dynamics (AAAI 2024 · similarity 0.86)
- LLark: A Multimodal Instruction-Following Language Model for Music (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.