Research map / Image generation / Autoregressive and latent generation
Fast Timing-Conditioned Latent Audio Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stable Audio: a latent diffusion model with a fully-convolutional VAE and timing-conditioned embeddings that generates variable-length, up-to-95-sec 44.1kHz stereo music and sound from text in about 8 seconds on an A100.
Problem
Generating long-form, variable-length 44.1kHz stereo music and sound effects from text is computationally expensive, and standard audio diffusion models are trained on fixed-length chunks, so they cannot control output duration.
Approach
Latent diffusion over a heavily downsampled (1024x) fully-convolutional VAE latent, conditioned on CLAP text features plus per-second timing embeddings (seconds_start and seconds_total) so the model fills a 95 s window and pads the rest with silence, enabling variable-length outputs.
Key results
- On MusicCaps, Stable Audio reaches FD_openl3 108.69, KL_passt 0.80 and CLAP_score 0.46, the best among the compared models, while generating 95 s stereo at 44.1kHz in 8 s on an A100.
- On AudioCaps, Stable Audio gets FD_openl3 103.66 versus 186.53 for AudioGen-medium and 170.31 for AudioLDM2-large, but its KL_passt (2.89) is worse than AudioGen-medium (1.42).
- Human study (19 users) gives Stable Audio audio quality 3.0 and text alignment 2.9 on MusicCaps, with 94.7% stereo correctness and 92.1% intro presence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · MusicCaps | FD_openl3: 123.30 -> 118.09 | Helped |
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · MusicCaps | KL_passt: 1.09 -> 0.97 | Helped |
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · MusicCaps | CLAP_score: 0.43 -> 0.44 | Helped |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · MusicCaps | FD_openl3: 126.93 -> 118.09 | Helped |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · MusicCaps | KL_passt: 1.06 -> 0.97 | Helped |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · MusicCaps | CLAP_score: 0.41 -> 0.44 | Helped |
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · AudioCaps | FD_openl3: 119.29 -> 114.25 | Helped |
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · AudioCaps | KL_passt: 2.73 -> 2.57 | Helped |
| Use CLAP_ours instead of CLAP_LAION text embeddings (ablation) | Stable Audio · AudioCaps | CLAP_score: 0.19 -> 0.16 | Hurt |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · AudioCaps | FD_openl3: 119.28 -> 114.25 | Helped |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · AudioCaps | KL_passt: 2.69 -> 2.57 | Helped |
| Use CLAP_ours instead of T5 text embeddings (ablation) | Stable Audio · AudioCaps | CLAP_score: 0.11 -> 0.16 | Helped |
Surprising findings
- On AudioCaps, Stable Audio's text alignment and KL_passt are worse than AudioGen-medium and AudioLDM2-48kHz, and its stereo correctness is only 57%.
- A privately trained CLAP text encoder slightly outperforms the open-source CLAP and T5 embeddings, with comparable performance overall.
Limitations
- Text alignment and KL on sound effects (AudioCaps) lag behind some baselines, possibly due to few sound effects in training data.
- Timing conditioning is less accurate around 40-60 sec durations, likely due to less training data at those lengths.
- Stereo correctness is low on AudioCaps prompts that do not require spatial movement.
Experimental setup
Datasets: AudioSparx (806,284 audios, 19,500 hours) MusicCaps AudioCaps
Benchmarks: MusicCaps AudioCaps
Metrics: FD_openl3 KL_passt CLAP_score audio quality MOS text alignment MOS musicality MOS stereo correctness musical structure
Baselines: AudioLDM2-music AudioLDM2-large AudioLDM2-48kHz MusicGen-small MusicGen-large MusicGen-large-stereo AudioGen-medium
Builds on: latent diffusion models Descript Audio Codec CLAP DPM-Solver++ Moûsai
Closest papers by meaning
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.89)
- Presto! Distilling Steps and Layers for Accelerating Music Generation (ICLR 2025 · similarity 0.88)
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.88)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.88)
- IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling (ICML 2025 · similarity 0.88)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.