Research map / Image generation / Autoregressive and latent generation
MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGE-LDM, a class-agnostic three-track latent diffusion model (mixture, submixture, source) that unifies music generation, source imputation, and text-queried source extraction via latent inpainting, trained jointly on Slakh2100, MUSDB18, and MoisesDB.
Problem
Existing multi-track music diffusion models rely on fixed instrument classes and an additive waveform mixing assumption that breaks in nonlinear latent spaces, limiting joint generation and separation across heterogeneous datasets.
Approach
Model mixture, submixture, and source latents jointly in a single Stable Audio-style DiT, assign independent per-track diffusion timesteps (with zeros for observed tracks) so tasks become latent inpainting, and use CLAP text/audio embeddings so any stem can be specified by free-form language.
Key results
- T1 achieves FAD 0.47 on S_A total generation, the lowest among compared models (MSG-LD 1.38), but generalizes worse to S_Full (1.79 vs MSG-LD 1.55).
- The full-data model T4 reaches FAD 2.78 on MUSDB18 and 1.47 on MoisesDB, best among the models in those columns.
- Adaptive timestep inpainting (T1) outperforms RePaint variants on Log-Mel L1 source extraction across most stems (average 2.48 vs 2.99-3.56 on S_B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on the full Slakh2100 set S_A + S_B instead of S_A alone | MGE-LDM · Slakh2100 S_Full test | FAD: 1.79 -> 0.63 | Helped |
| Train on S_A + S_B instead of S_A alone | MGE-LDM · Slakh2100 S_A test | FAD: 0.47 -> 3.14 | Hurt |
| Train on S_A + S_B + MUSDB18 + MoisesDB instead of MUSDB18 + MoisesDB | MGE-LDM · MUSDB18 test | FAD: 2.87 -> 2.78 | Helped |
| Train on S_A + S_B + MUSDB18 + MoisesDB instead of MUSDB18 + MoisesDB | MGE-LDM · MoisesDB test | FAD: 1.59 -> 1.47 | Helped |
| Condition mixture generation on the four-instrument text prompt | MGE-LDM T2/T3/T4 · Slakh2100 S_A test | FAD: 3.14 -> 2.24; 8.80 -> 3.96; 6.83 -> 5.05 | Helped |
| Use track-specific adaptive timestep conditioning instead of canonical single-sample inpainting (ablation) | MGE-LDM · Slakh2100 S_A test, drums extraction | Log-Mel L1 distance: 3.20 -> 2.71 | Helped |
| Use track-specific adaptive timestep conditioning instead of RePaint T=25 U=10 (ablation) | MGE-LDM · Slakh2100 S_A test, drums extraction | Log-Mel L1 distance: 2.28 -> 2.71 | Hurt |
| Increase RePaint resampling while reducing reverse timesteps (T=250 U=1 to T=25 U=10) (ablation) | RePaint-based inpainting · Slakh2100 S_A test, drums extraction | Log-Mel L1 distance: 3.20 -> 2.28 | Helped |
| Increase resampling count at T=250 from U=1 to U=4 (ablation) | RePaint-based inpainting · Slakh2100 S_A test, drums extraction | Log-Mel L1 distance: 3.20 -> 2.34 | Helped |
| Expand MGE-LDM T1 training from S_A to S_A + S_B | MGE-LDM · Slakh2100 S_A test, one source imputed (bass) | sub-FAD: 1.02 -> 2.11 | Hurt |
| Train on MUSDB18 + MoisesDB instead of S_A + S_B | MGE-LDM · Slakh2100 S_B test, organ imputed | sub-FAD: 1.03 -> 0.15 | Helped |
| Train on S_A + S_B + MUSDB18 + MoisesDB instead of S_A + S_B | MGE-LDM · Slakh2100 S_B test, pipe imputed | sub-FAD: 3.66 -> 0.20 | Helped |
Surprising findings
- Fixed-class baseline MSG-LD beats the proposed model on partial generation for most single-source imputation tasks on S_A.
- RePaint configurations with more total denoising steps (T=250, U=2/4) underperform T=25, U=10, suggesting resampling count matters more than trajectory length.
Limitations
- Experiments only at 16 kHz mono audio, limiting fidelity and omitting spatial cues.
- Continuous latent space does not enforce additivity, contributing to hallucinated sources during extraction.
- Requires multi-stem supervision during training; MUSDB18-only training overfits.
Experimental setup
Datasets: Slakh2100 MUSDB18 MoisesDB
Benchmarks: Slakh2100 test (S_A, S_Full) MUSDB18 test MoisesDB test
Metrics: FAD (VGGish) sub-FAD Log-Mel L1 distance
Baselines: MSDM MSG-LD HDemucs AudioSep
Builds on: Stable Audio v-objective RePaint TD-Paint DDIM
Closest papers by meaning
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.92)
- Presto! Distilling Steps and Layers for Accelerating Music Generation (ICLR 2025 · similarity 0.87)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.87)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.86)
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.86)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.