Research map / Image generation / Fast diffusion sampling
SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoundCTM, a consistency trajectory model distilled from LDM text-to-sound teachers that gives high-quality 1-step generation and improved multi-step generation with deterministic sampling, scaled to a 1B DiT for 44.1kHz full-band audio.
Problem
Diffusion-based text-to-sound models are slow, and existing consistency-distillation T2S models give poor 1-step quality and change semantic content under multi-step stochastic sampling, which hinders trial-and-refinement sound creation.
Approach
Adapts Consistency Trajectory Models to T2S with a teacher-network feature distance (teacher's intermediate features used as the distillation metric), distillation of classifier-free guided trajectories with omega as a student condition, and nu-sampling that jointly uses conditional and unconditional student jumps.
Key results
- At 16kHz, SoundCTM 1-step achieves FAD_vgg 2.08 and KL_passt 1.26 on AudioCaps, best among 1-step distillation models, and 16-step improves FAD_vgg to 1.38.
- At 44.1kHz, SoundCTM-DiT-1B 1-step generation runs in 0.06 s on H100 and reaches FD_openl3 84.07 while 16-step reaches FD_openl3 72.24 and KL_passt 1.27, matching or beating the teacher's 72.34 / 1.74 at 40 steps on KL.
- In a subjective MUSHRA test, SoundCTM-DiT-1B 16-step scored OVAL 72.0 and REL 77.0, above the teacher (66.9 / 69.4) and the 1-step model (63.2 / 69.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase deterministic sampling from 1 to 16 steps | SoundCTM on AudioCaps, 16 kHz · AudioCaps test, 16 kHz | FAD_vgg: 2.08 -> 1.38 | Helped |
| Increase deterministic sampling from 1 to 16 steps | SoundCTM-DiT-1B · AudioCaps test, 44.1 kHz | KL_passt: 1.74 -> 1.27 | Helped |
| Increase sampling from 1 to 16 steps | SoundCTM-DiT-1B · AudioCaps test, 44.1 kHz | OVAL: 63.2 ± 1.46 -> 72.0 ± 1.30 | Helped |
| Use the teacher network feature distance instead of L2 distance (ablation) | SoundCTM on AudioCaps, 16 kHz · AudioCaps test, 16 kHz | FAD_vgg: 2.43 -> 2.17 | Helped |
| Add the DSM loss (ablation) | SoundCTM-DiT-1B · AudioCaps test, full-band, 16 steps | FD_openl3: 108.7 -> 65.6 | Helped |
| Add the DSM loss (ablation) | SoundCTM-DiT-1B · AudioCaps test, full-band, 16 steps | CLAP: 0.23 -> 0.42 | Helped |
| Use the teacher feature distance instead of omitting it (ablation) | SoundCTM-DiT-1B · AudioCaps test, full-band, 16 steps | FD_openl3: 69.2 -> 65.6 | Helped |
| Use the teacher feature distance instead of omitting it (ablation) | SoundCTM-DiT-1B · AudioCaps test, full-band, 16 steps | CLAP: 0.41 -> 0.42 | Helped |
| Use the teacher feature distance instead of omitting it (ablation) | SoundCTM on AudioCaps, 16 kHz · AudioCaps test, 16 kHz, 16 steps | FAD_vgg: 1.62 -> 1.37 | Helped |
| Add GAN loss (ablation) | SoundCTM on AudioCaps, 16 kHz · AudioCaps test, 16 kHz, 2 steps | FAD_vgg: 2.43 -> 2.91 | Hurt |
| Add GAN loss (ablation) | SoundCTM training · Training | Memory consumption: 46782 MiB -> 64492 MiB | Hurt |
| Increase ν for 8-step sampling | SoundCTM-DiT-1B · AudioCaps test, full-band, 8 steps | Sample quality | Helped |
Surprising findings
- Using the teacher network as the feature extractor for the distillation distance outperformed l2 distance in latent space at both 0-time and s-time.
- ConsistencyTTA showed no quality improvement trade-off with more steps, suggesting error accumulation under multi-step stochastic sampling.
Limitations
- Single-step quality does not surpass the teacher; adding GAN loss did not help in preliminary experiments and raised memory use.
- Experiments are limited to AudioCaps, and Stable Audio Open results may not be comparable because it was not trained on AudioCaps.
- Feature-distance design relies on having a teacher diffusion model available.
Experimental setup
Datasets: AudioCaps AudioSet MUSDB18-HQ
Benchmarks: AudioCaps
Metrics: FAD_vgg KL_passt IS_passt CLAP score FD_openl3 inference speed OVAL REL LSD CLAP-LPAPS
Baselines: ConsistencyTTA AudioLCM TANGO AudioLDM 2 Stable Audio Open AudioLDM2-48kHz DITTO-2
Builds on: Consistency Trajectory Models Consistency Models TANGO EDM LPIPS
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.89)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.89)
- Customized Condition Controllable Generation for Video Soundtrack (CVPR 2025 · similarity 0.88)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.88)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.88)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.