Research map / Image generation / Autoregressive and latent generation
ETTA: Elucidating the Design Space of Text-to-Audio Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large-scale empirical study of diffusion and flow-matching text-to-audio design choices, yielding AF-Synthetic (1.35M synthetic captions), an improved DiT (ETTA-DiT) and the ETTA model, which is SOTA among publicly-trained TTA models on AudioCaps and MusicCaps.
Problem
Text-to-audio (TTA) design space is complex and under-understood: effects of data, architecture, training objective, and sampling on benchmark performance are confounded and unclear.
Approach
Systematically sweeps data, DiT architecture/implementation, model size, training objective (v-diffusion vs OT-CFM with logit-normal timestep sampling), and samplers (Euler/Heun, NFE, CFG) on a Stable Audio Open-based latent diffusion setup, and combines the best findings with a new large synthetic caption dataset (AF-Synthetic) captioned by Audio Flamingo and CLAP-filtered.
Key results
- On MusicCaps, ETTA reaches FD_P 10.06 and KL_S 0.84 versus Stable Audio Open's 36.42 and 1.32, with IS_P 3.32 vs 2.93.
- On AudioCaps, ETTA achieves FD_P 13.12 and IS_P 14.36, improving over Stable Audio Open (38.27 FD_P, 12.09 IS_P) with a single public-data model.
- In creative caption subjective evaluation, ETTA gets REL 4.05 vs 3.95 for Stable Audio Open, with p<0.05 by Wilcoxon signed-rank test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on AF-Synthetic instead of the baseline data (ablation) | Stable Audio Open · MusicCaps | FD_P: 39.96 -> 26.22 | Helped |
| Replace the baseline DiT implementation with ETTA-DiT (ablation) | ETTA trained on AF-Synthetic · MusicCaps | FD_P: 26.22 -> 20.48 | Helped |
| Use OT-CFM with uniform timestep sampling (ablation) | ETTA-DiT trained on AF-Synthetic · MusicCaps | FD_P: 20.48 -> 22.16 | Hurt |
| Use logit-normal instead of uniform timestep sampling (ablation) | ETTA trained with OT-CFM · MusicCaps | FD_P: 22.16 -> 21.59 | Helped |
| Train on AudioCaps instead of AF-Synthetic (ablation) | ETTA · MusicCaps | FD_P: 21.59 -> 76.14 | Hurt |
| Train on TangoPromptBank instead of AF-Synthetic (ablation) | ETTA · MusicCaps | CL_M: 0.44 -> 0.38 | Hurt |
| Increase DiT depth from 4 to 24 (ablation) | ETTA · AudioCaps | FD_P: 36.46 -> 28.46 | Helped |
| Increase DiT depth from 24 to 36 (ablation) | ETTA · AudioCaps | FD_P: 28.46 -> 27.08 | Helped |
| Increase DiT width from 384 to 1536 (ablation) | ETTA · AudioCaps | FD_P: 35.97 -> 28.46 | Helped |
| Increase convolutional feed-forward kernel size from 1 to 3 (ablation) | ETTA · AudioCaps | FD_P: 28.46 -> 28.72 | Hurt |
| Set rope_base to 512 instead of 16384 (ablation) | ETTA · AudioCaps | FD_P: 13.01 -> 12.64 | Helped |
| Set dropout to 0.0 instead of 0.1 (ablation) | ETTA · MusicCaps | FD_P: 12.15 -> 11.75 | Helped |
Surprising findings
- Solely switching the DiT implementation (ETTA-DiT) to AdaLN-based conditioning gave worse results in the vanilla stable-audio-tools setup, though it improved results in the authors' improved implementation.
- Data quality matters more than quantity: AF-AudioSet (0.16M captions) is competitive with AF-Synthetic (1.35M) on MusicCaps, and TangoPromptBank at similar size scores much worse on KL and CLAP.
Limitations
- Focuses on systematic ablation rather than novel methods; findings tied to the Stable Audio Open-based setup.
- Fine-tuning on AudioCaps (FT-AC-100k) shows noticeable degradation for music generation.
- Evaluation relies on objective metrics that may not reflect human perception; authors note need for better evaluation benchmarks.
Experimental setup
Datasets: AF-Synthetic AudioCaps AF-AudioSet TangoPromptBank AudioSet VGGSound WavCaps Laion-630K HiFi-TTS FSD50k ESC-50
Benchmarks: AudioCaps MusicCaps
Metrics: FD_PANNs FD_OpenL3 KL_PaSST KL_PANNs IS_PANNs CLAP score OVL REL
Baselines: Stable Audio Open AudioLDM2 AudioLDM2-large TANGO-AF TANGO2 Make-An-Audio 2 GenAU-L Audiobox Sound MusicGen-medium Jen-1 QA-MDT FluxMusic
Builds on: Stable Audio Open stable-audio-tools Diffusion Transformer (DiT) optimal transport conditional flow matching classifier-free guidance Audio Flamingo
Closest papers by meaning
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.90)
- Creative Text-to-Audio Generation via Synthesizer Programming (ICML 2024 · similarity 0.89)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.89)
- Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.89)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.88)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.