Research map / Image generation / Fast diffusion sampling
Presto! Distilling Steps and Layers for Accelerating Music Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presto! accelerates EDM-style text-to-music diffusion transformers by combining a new score-based distribution matching distillation for step reduction (Presto-S), a variance-aware budget-conditioned layer dropping method (Presto-L), and their layer-then-step combination (Presto-LS), reaching 10-18x speedup.
Problem
Diffusion-based text-to-music generation is slow due to many sampling steps and costly per-step DiT computation, and prior distillation for TTA/TTM is limited to short or lower-quality audio and has not used GAN-based or layer-based distillation.
Approach
Extends DMD2 adversarial distillation to continuous-time EDM models with continuous noise-level inputs, disentangled noise distributions per loss term, and a least-squares discriminator on partially downsampled features; improves layer distillation by shifting the drop schedule to preserve final-layer variance, adding budget conditioning and a self-teacher loss; and combines them by doing layer di
Key results
- Presto-LS reaches 230/435 ms latency for 32s mono/stereo audio on an A100 40GB, about 15x faster than Stable Audio Open, with MMD 0.73 and recall 0.14 at 4 NFE.
- Presto-S (4 steps) achieves FAD 0.22 and MMD 0.83 with CLAP 33.13, the best among distilled methods and close to the base model's quality.
- Presto-L improves over the base DPM-2S sampler on all metrics (FAD 0.18 vs 0.24, MMD 0.61 vs 0.82) while speeding up by about 27%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use training rather than inference noise distribution for the DMD loss (ablation) | Presto-S · Song Describer | FAD: 0.37 -> 0.27 | Helped |
| Use training rather than inference noise distribution for the DMD loss (ablation) | Presto-S · Song Describer | CLAP: 27.45 -> 33.12 | Helped |
| Use training rather than inference noise distribution for the GAN loss (ablation) | Presto-S · Song Describer | FAD: 0.27 -> 0.23 | Helped |
| Use training rather than inference noise distribution for the DSM loss (ablation) | Presto-S · Song Describer | FAD: 0.23 -> 0.22 | Helped |
| Replace non-saturating GAN with least-squares GAN (ablation) | Presto-S · Song Describer | FAD: 0.24 -> 0.22 | Helped |
| Replace non-saturating GAN with least-squares GAN (ablation) | Presto-S · Song Describer | CLAP: 31.48 -> 33.13 | Helped |
| Use continuous rather than discrete generator noise-level inputs (ablation) | Presto-S · Song Describer | quality and text relevance | Helped |
| Shift layer dropping to preserve the final DiT layer and add explicit budget conditioning (ablation) | Presto-L · Song Describer | FAD: 0.24 -> 0.18 | Helped |
| Use Presto-L layer distillation rather than ASE layer dropping (ablation) | DPM-2S sampling · Song Describer | FAD: 0.25 -> 0.18 | Helped |
| Distill the model with Presto-S rather than DMD-GAN | 4-step generation · Song Describer | FAD: 0.29 -> 0.22 | Helped |
| Combine layer distillation with step distillation (ablation) | Presto-LS · Song Describer | MMD: 0.83 -> 0.73 | Helped |
| Initialize real and fake score models from the layer-distilled model instead of the original score model (ablation) | Presto-LS · step and layer distillation | stability | Hurt |
Surprising findings
- Layer dropping with the Presto-L schedule improved quality and text relevance over the full base model, not just speed.
- Naively combining step and layer distillation (or starting layer distillation from a step-distilled checkpoint) led to instability and generator collapse, requiring a layer-then-step order with full-capacity score models.
Limitations
- Authors do not release the model, citing ethical concerns about competing with creators.
- Presto-LS gains over step-only distillation are modest because the layer-drop schedule had to be reduced for stability.
- Discriminator design and placement inside the DiT is left for future work.
Experimental setup
Datasets: 3.6K hour licensed instrumental music dataset Song Describer (no vocals)
Benchmarks: Song Describer (no vocals)
Metrics: FAD MMD CLAP Score Density Recall Coverage RTF
Baselines: Consistency Models (CM) SoundCTM DITTO-CTM DMD-GAN ASE MusicGen Stable Audio Open
Builds on: DMD2 EDM ASE DiT Consistency Models
Closest papers by meaning
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.90)
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.88)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.88)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.87)
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.87)
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.