Research map / Image generation / Autoregressive and latent generation
Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes open-source audio diffusion models (Stable Audio Open Small) into streaming interactive music generators via routing/attention masks enabling KV caching, plus RL-free ARC-Forcing rollout post-training, deployed as a generative delay on a consumer laptop.
Problem
Interactive streaming music generation is dominated by large discrete-AR models (LMMs) requiring heavy compute; bidirectional diffusion models are not streamable and lack KV caching, and rollout-based error accumulation is unaddressed.
Approach
A routing mask separating clean context from noisy target frames plus encoder-decoder or block-causal attention masks lets clean context be KV-cached over diffusion steps (and time), and ARC-Forcing combines Self-Forcing rollouts with an ARC relativistic adversarial discriminator (no RL or reward model) to reduce error accumulation and enable few-step sampling.
Key results
- Text-conditioned LMDM (ED)+AF with 8 steps reaches FD 29.00 and CLAP 0.32 in primed setting, versus Magenta RealTime FD 72.14 (unprimed setting in table), with TTFF 0.03s versus about 4s.
- ARC-Forcing reduces per-window metric degradation over 2-minute generations, with error accumulation strongly mitigated for both Enc-Dec and Block-Causal LMDMs.
- Sketch-conditioned LMDM with ARC-Forcing achieves delay under 1 second by distilling to 8 steps and running ONNX-exported models on a consumer laptop; the paper reports qualitative musician study with three instrumentalists.
Surprising findings
- Block-Causal LMDMs, despite strictly better asymptotic complexity, were empirically slower in wall-clock time than Enc-Dec, attributed to suboptimal implementation.
- Text-prompt following regressed toward a generic EDM sound during live ONNX deployment, not seen in offline evaluation.
Limitations
- Output quality trails large frontier models such as Suno and genre bias toward EDM from MTG-Jamendo.
- LMDMs are more responsive to past audio context than to text conditions.
- Latency (~1s delay) limits flexibility; sub-second blocks need better causal codecs.
Experimental setup
Datasets: MTG-Jamendo Song Describer Dataset Slakh FSD50k humpback whale songs MUSDB18 MusicCaps
Benchmarks: Song Describer Dataset MUSDB18 MusicCaps Slakh
Metrics: FD-OpenL3 KL-PaSST CLAP score D-NFE TTFF CoCoLA Mel Rhythm Dynamics
Baselines: Magenta RealTime Stable Audio Open MusicGen-Large Live Music Models
Builds on: Self-Forcing ARC (Adversarial Relativistic Contrastive) Stable Audio Open Small CFG++ Live Music Models
Closest papers by meaning
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.90)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.90)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.89)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.89)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.88)
- Presto! Distilling Steps and Layers for Accelerating Music Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.