Research map / Image generation / Autoregressive and latent generation
AudioGen: Textually Guided Audio Generation
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, Yossi Adi
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mixing augmentation (fusing audio pairs and concatenating captions) (ablation) | AudioGen-base · AudioCaps test | REL (text relevance): 63.23 -> 66.01 | Improved |
| add mixing augmentation (ablation) | AudioGen-base · AudioCaps test | FAD: 2.84 -> 3.13 | Hurt |
| add mixing augmentation (ablation) | AudioGen-base · AudioCaps test | KL: 2.14 -> 2.09 | Improved |
| add mixing augmentation (ablation) | AudioGen-base · AudioCaps test | OVL: 70.85 -> 71.68 (within CI) | No clear effect |
| scale up ALM to 1B parameters and T5 to large (mix augmentation) | AudioGen · AudioCaps test | FAD: 3.13 -> 1.82 (vs DiffSound 7.39) | Improved |
| use AudioGen-base instead of DiffSound (diffusion over VQ-VAE codes) | AudioGen · AudioCaps test | KL: 2.57 -> 2.09 (base); 1.69 (large) | Improved |
| multi-stream modeling with 2 streams (x64 DSF) (ablation) | AudioGen-base · AudioCaps test | FAD: 3.13 -> 10.35; speed-up x2.0 | Hurt |
| multi-stream modeling with 4 streams (x128 DSF) (ablation) | AudioGen-base · AudioCaps test | KL: 2.09 -> 2.36; speed-up x5.1 | Hurt |
| multi-stream modeling with 2 streams (x64 DSF) (ablation) | AudioGen-large · AudioCaps test | FAD: 1.82 -> 6.89; speed-up x2.3 | Hurt |
| use T5-large instead of T5-base text encoder (base ALM) (ablation) | AudioGen · AudioCaps test | KL: 2.09 -> 1.91 | Improved |
| use ALM-large instead of ALM-base (T5-base) (ablation) | AudioGen · AudioCaps test | FAD: 3.13 -> 2.27 | Improved |
| apply classifier-free guidance (remove it, gamma=1 vs 3.0) (ablation) | AudioGen · AudioCaps test | FAD: FAD minimum at gamma=3.0; removing CFG gives poor performance | Improved |
Closest papers by meaning
- AudioGenX: Explainability on Text-to-Audio Generative Models (AAAI 2025 · similarity 0.91)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.91)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.89)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
