Research map / Image generation / Autoregressive and latent generation
Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthio augments small audio classification datasets with synthetic audio from a Stable Audio T2A model trained on Sound-VECaps, aligned via DPO on the small dataset and prompted with LLM-generated diverse captions filtered by CLAP and self-reflection.
Problem
Small-scale audio classification datasets are scarce; traditional augmentations do not capture real-world diversity, and naive synthetic data from text-to-audio models is inconsistent acoustically with the target data and lacks compositional diversity.
Approach
Align a T2A diffusion model to the small target dataset with DPO (real clips as winners, generated clips as losers), then prompt it with LLM-generated diverse captions (MixCap) filtered by CLAP and refined via self-reflection.
Key results
- Synthio outperforms all baselines by 0.1%-39% across 10 datasets and 4 small-data settings (n=50,100,200,500) when augmenting with a T2A model trained only on weakly captioned AudioSet-derived data.
- At n=50 on ESC-50, Synthio reaches 49.50% vs 22.25% gold-only (+122%); on USD8K it reaches 76.12% vs 55.09%.
- Removing DPO alignment causes the largest average ablation drop (about 4.5%), and synthetic-only training with Synthio beats Vanilla Syn. Aug. on GTZAN, VS, TUT and MSDB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment limited labeled data with Synthio synthetic audio | AST audio classification · ESC-50 (n=100) | accuracy: 77.25 -> 83.35 | Helped |
| augment limited labeled data with Synthio synthetic audio | AST audio classification · USD8K (n=100) | accuracy: 79.73 -> 85.00 | Helped |
| augment limited labeled data with Synthio synthetic audio | AST audio classification · GTZAN (n=100) | accuracy: 71.05 -> 71.20 | Helped |
| augment limited labeled data with Synthio synthetic audio | AST audio classification · Medley (n=100) | accuracy: 65.79 -> 71.23 | Helped |
| augment limited labeled data with Synthio synthetic audio | AST audio classification · DCASE (n=100) | accuracy: 16.93 -> 19.38 | Helped |
| add DPO alignment (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 66.75 -> 83.35 | Helped |
| add MixCap caption generation (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 73.50 -> 83.35 | Helped |
| add iterative self-reflection (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 77.65 -> 83.35 | Helped |
| use MixCap and self-reflection instead of template captions (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 78.00 -> 83.35 | Helped |
| use DPO instead of ERM fine-tuning (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 73.20 -> 83.35 | Helped |
| add CLAP filtering (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 82.55 -> 83.35 | Helped |
| change CLAP threshold from 0.85 to 0.3 (ablation) | Synthio audio augmentation · ESC-50 (n=100) | accuracy: 83.35 -> 82.55 | No clear effect |
Surprising findings
- On full (non-downsampled) datasets, traditional augmentations are much more competitive and Synthio gives no gain over Vanilla Syn. Aug. (identical numbers on USD8K, GTZAN, Medley).
- Naive LLM-caption augmentation without MixCap sometimes underperforms template captions, e.g., ESC-50 at n=50 (36.80 vs 41.25 for template captions).
Limitations
- Performance depends on the T2A model's capability and training data quality; a T2A model trained on AudioCaps alone performs much worse.
- Captions generated by LLMs may carry LLM biases into training.
- Synthio is computationally more intensive than traditional augmentation methods.
Experimental setup
Datasets: Sound-VECaps ESC-50 UrbanSound8K GTZAN Medley-solos-DB MUSDB18 NSynth Instruments TUT Urban DCASE Task 4 Vocal Sounds FSD50K AudioCaps
Benchmarks: ESC-50 UrbanSound8K GTZAN Medley-solos-DB TUT Urban NSynth Instruments Vocal Sounds MUSDB18 DCASE Task 4 FSD50K AudioCaps
Metrics: accuracy F1-macro METEOR CIDEr SPICE SPIDEr CLAP similarity FAD Inception-style spectral feature distributions
Baselines: SpecAugment Audiomentations Pitch Shifting Random Noise Retrieval Vanilla Synthetic Augmentation Vanilla Syn. Aug. + LLM Caps
Builds on: DPO Diffusion-DPO Stable Audio AST CLAP
Closest papers by meaning
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- Contrastive Learning from Synthetic Audio Doppelgängers (ICLR 2025 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.87)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.86)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.