Research map / Image generation / Autoregressive and latent generation
Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-source diffusion model trained on joint distribution of stems that performs total generation, partial generation (source imputation), and source separation with a new Dirac-likelihood posterior sampler.
Problem
Music generation and source separation are usually handled by separate models; mixture generators lose source information, while separators cannot generate coherent mixtures or stems.
Approach
Learn the joint score of contextual stems with a denoising diffusion model, then solve generation, imputation, and separation at inference by conditioning the prior; separation uses a Dirac delta likelihood that enforces the sum constraint (limit of Gaussian likelihood with gamma to 0).
Key results
- Total generation: MSDM FAD 6.55 vs mixture model 6.67, with comparable listening-test quality (6.51 vs 6.15).
- Source separation on Slakh2100: MSDM Dirac with correction reaches 16.48 dB average SI-SDRi vs Demucs 16.11 dB and Demucs+Gibbs (512 steps) 17.73 dB; ISDM Dirac with correction reaches 17.27 dB.
- Dirac likelihood outperforms the Gaussian likelihood for both MSDM and ISDM separators (e.g., MSDM 14.92 vs 12.90 dB without correction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MSDM instead of a mixture-trained model | total generation · Slakh2100 | FAD: 6.67 -> 6.55 | Helped |
| use MSDM instead of a mixture-trained model | total generation · Slakh2100 | coherence: 5.67 ± 2.60 -> 6.35 ± 2.36 | Helped |
| use MSDM instead of a mixture-trained model | total generation · Slakh2100 | quality: 6.15 ± 2.47 -> 6.51 ± 2.19 | No clear effect |
| use Dirac likelihood instead of Gaussian likelihood | source separation · Slakh2100 test set, all stems | SI-SDR_I (dB): 13.67 -> 16.30 | Helped |
| add correction step (ablation) | ISDM Dirac · Slakh2100 test set, all stems | SI-SDR_I (dB): 16.30 -> 17.27 | Helped |
| add correction step (ablation) | MSDM Dirac · Slakh2100 test set, all stems | SI-SDR_I (dB): 14.92 -> 16.48 | Helped |
| add correction step (ablation) | ISDM Gaussian · Slakh2100 test set, all stems | SI-SDR_I (dB): 13.67 -> 14.58 | Helped |
| add correction step (ablation) | MSDM Gaussian · Slakh2100 test set, all stems | SI-SDR_I (dB): 12.90 -> 14.54 | Helped |
| replace contextual MSDM prior with independent per-source priors | Dirac source separator · Slakh2100 test set, all stems | SI-SDR_I (dB): 14.92 -> 16.30 | Helped |
| increase sampling stochasticity with S_churn (ablation) | source separators · Slakh2100 test-set hyperparameter search | SI-SDR_I (dB) | Helped |
| use Dirac likelihood instead of Gaussian likelihood (ablation) | MSDM and ISDM source separators · Slakh2100 hyperparameter search | SI-SDR_I (dB) | Helped |
| generate a partial accompaniment conditioned on fixed sources | MSDM · Slakh2100 partial generation | quality | Helped |
Surprising findings
- Trained on only 6s/MUSDB18-HQ-style data, the model transfers poorly out of distribution (MSDM SI-SDRi around -0.88 dB on MUSDB tested directly).
Limitations
- Performance is constrained by the amount of available contextual training data (145h Slakh2100).
- Source relationship assumed additive; extension to other functional relations left to future work.
- Separation inference is much slower than Demucs.
Experimental setup
Datasets: Slakh2100 MUSDB18-HQ
Benchmarks: Slakh2100 MUSDB18-HQ
Metrics: FAD (VGGish) sub-FAD subjective quality and coherence/density (1-10 listening tes SI-SDR improvement (SI-SDRi)
Baselines: Demucs Demucs + Gibbs NCSN-BASIS Gaussian likelihood mixture diffusion model
Builds on: Moûsai NCSN-BASIS Karras et al. EDM noise schedule and Euler sampler Demucs
Closest papers by meaning
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.92)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.90)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.89)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.88)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.88)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.