Research map / Vision-language and multimodal / Human and embodied modeling
Creative Text-to-Audio Generation via Synthesizer Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTAG, which generates abstract text-to-audio by optimizing 78 parameters of a virtual modular synthesizer (Voice in SynthAX) against LAION-CLAP similarity using evolutionary strategies, and compares it to AudioGen and AudioLDM.
Problem
Neural text-to-audio models produce acoustically literal sounds in large, uninterpretable latent spaces that are hard to inspect or tweak, leaving creative abstraction under-explored.
Approach
Instead of training a neural generator, iteratively optimize the parameters of a differentiable-free modular synthesizer (78 parameters) with gradient-free evolution strategies to maximize CLAP similarity to a text prompt, yielding interpretable, abstract sounds.
Key results
- On AudioSet-50 classification, CTAG Top-1 accuracy is 26.2% vs 17.4% for AudioLDM and 51.6% for AudioGen; on ESC-50 CTAG reaches 16.4% Top-1 vs 23.0% AudioLDM and 54.0% AudioGen.
- In a 10-participant user study, CTAG sounds were identified more accurately than AudioLDM (odds ratio 2.72) and similar to AudioGen (odds ratio 0.85), and were rated more artistic (3.54 vs 2.32 for AudioGen and 2.90 for AudioLDM).
- LES (Learned Evolution Strategies) was the strongest optimizer in ablations, and the 2-second duration and Voice (78-parameter) architecture gave the best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize synthesizer parameters for the text prompt with CLAP | CTAG · AudioSet-50 | CLAP score: 0.249 ± 0.160 -> 0.573 ± 0.126 | Helped |
| Optimize synthesizer parameters for the text prompt with CLAP | CTAG · ESC-50 | CLAP score: 0.277 ± 0.180 -> 0.585 ± 0.130 | Helped |
| Use a simple template prompt instead of the class label (ablation) | CTAG · AudioSet-50 | Top-1 accuracy (%): 26.2 -> 25.2 | Hurt |
| Use an LLM-generated caption prompt instead of the class label (ablation) | CTAG · AudioSet-50 | Top-1 accuracy (%): 26.2 -> 23.6 | Hurt |
| Use a simple template prompt instead of the class label (ablation) | CTAG · ESC-50 | Top-1 accuracy (%): 16.4 -> 11.4 | Hurt |
| Use an LLM-generated caption prompt instead of the class label (ablation) | CTAG · ESC-50 | Top-1 accuracy (%): 16.4 -> 13.8 | Hurt |
| Generate abstract sounds with CTAG rather than AudioLDM | CTAG vs AudioLDM · User study | Accuracy: 34.0 -> 56.0; odds ratio = 2.72 | Helped |
| Generate abstract sounds with CTAG rather than AudioGen | CTAG vs AudioGen · User study | Accuracy: 59.5 -> 56.0; odds ratio = 0.85 | No clear effect |
| Use CTAG instead of AudioGen | CTAG vs AudioGen · User study | Artistic interpretation: 2.32 -> 3.54; contrast = 1.22 | Helped |
| Use CTAG instead of AudioLDM | CTAG vs AudioLDM · User study | Artistic interpretation: 2.90 -> 3.54; contrast = 0.65 | Helped |
| Use the LES optimizer rather than the other tested optimization algorithms (ablation) | CTAG · ESC-50 | CLAP score and classification accuracy | Helped |
| Use 2-second sounds rather than 1-, 3-, or 4-second sounds (ablation) | CTAG · ESC-50 | CLAP score and classification accuracy | Helped |
Surprising findings
- CTAG achieves higher CLAP scores (0.573 on AudioSet-50) than real ESC-50 recordings (0.416), showing that optimizing the objective exceeds real-data alignment while classifiers recognize its sounds less often.
Limitations
- Each prompt requires iterative optimization from random initialization, so generation is slower than feed-forward models.
- Limited to brief, non-mixture sounds that the synthesizer can model.
- Classifiers recognize CTAG sounds less often on ESC-50, reflecting a distribution shift from real audio.
Experimental setup
Datasets: ESC-50 AudioSet
Benchmarks: ESC-50 AudioSet-50
Metrics: Top-1 accuracy Top-5 accuracy CLAP score spectral complexity flux HFC spectral rolloff spectral centroid MP3 compression ratio user identification accuracy
Baselines: AudioGen AudioLDM
Builds on: SynthAX torchsynth LAION-CLAP Evosax CMA-ES
Closest papers by meaning
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.89)
- Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.88)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.86)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.86)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.86)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.