Research map / Image generation / Autoregressive and latent generation
Taming Data and Transformers for Audio Generation
Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Guha Balakrishnan, Vicente Ordóñez
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLAP embedding (ablation) | AutoCap · AudioCaps test | CIDEr: 80.7 -> 83.2 | Improved |
| add stage 2 BART fine-tuning (ablation) | AutoCap · AudioCaps test | CIDEr: 75.6 -> 83.2 | Improved |
| add stage 1 pretraining (ablation) | AutoCap · AudioCaps test | CIDEr: 59.6 -> 83.2 | Improved |
| unfreeze BART word embeddings in stage 2 (ablation) | AutoCap · AudioCaps test | CIDEr: 83.2 -> 82.6 | Hurt |
| add metadata (text) to audio input | AutoCap · AudioCaps test | CIDEr / SPICE: CIDEr 80.4 -> 83.2; SPICE 19.0 -> 18.2 | Mixed |
| replace UNet backbone with FIT (ablation) | GenAU-S · OOD evaluation | IS: 13.55 -> 15.76 | Improved |
| replace DiT backbone with FIT (ablation) | GenAU-S · OOD evaluation | IS: 13.96 -> 15.76 | Improved |
| recaption with AutoCap instead of WavCaps captions (ablation) | GenAU-S · OOD evaluation | IS: 11.83 -> 15.76 | Improved |
| scale training data from AudioCaps to AutoReCap (ablation) | GenAU-L · OOD evaluation | IS: 12.14 -> 18.98 | Improved |
| scale model from GenAU-S to GenAU-L (ablation) | GenAU · OOD evaluation | IS: 15.76 -> 18.98 | Improved |
| train on full 10-second AutoReCap-XL subset | GenAu-Large · AudioCaps test | IS / FAD: IS 11.75 -> 12.56; FAD 1.21 -> 1.71 | Mixed |
Closest papers by meaning
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.88)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.85)
- UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.85)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.85)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
