Research map / Image generation / Text-to-image diffusion
Audio Generation with Multiple Conditional Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds timestamp, pitch-contour, and energy-contour conditions to a frozen pre-trained text-to-audio diffusion model (Tango) via a trainable control encoder and Fusion-Net, with a new consolidated AudioCondition dataset and controllability metrics.
Problem
Text-only conditioning of text-to-audio models gives limited control over the temporal order, pitch, and energy of generated audio, and no datasets or metrics exist for multi-condition audio generation.
Approach
Freeze a pre-trained TTA model (Tango) and train a control condition encoder (frame-level FLAN-T5 label embeddings for timestamps, CWT-quantized pitch and log-energy contours) plus a Fusion-Net inserted between U-Net layers, with one shared model handling all three control types.
Key results
- Ours reaches Eb 29.07% and At 47.11% on temporal order, versus 5.20% Eb and 45.11% At for Tango.
- Ours achieves energy MAE of 0.200 versus 0.553 for Tango and pitch DTW of 10.79 versus 36.11 for Tango.
- Multi-condition Ours (1076M params) matches or slightly beats three single-condition models (3177M params total) on most metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add timestamp, pitch-contour, and energy-contour controls to text conditioning | Tango · AudioCondition test | Eb: 5.20 -> 29.07 | Helped |
| add timestamp, pitch-contour, and energy-contour controls to text conditioning | Tango · AudioCondition test | DTW: 36.11 -> 10.79 | Helped |
| add timestamp, pitch-contour, and energy-contour controls to text conditioning | Tango · AudioCondition test | MAE: 0.553 -> 0.200 | Helped |
| train one model on multiple control conditions instead of separate single-condition models (ablation) | Ours · AudioCondition test | MAE: 0.267 -> 0.200 | Helped |
| replace the timestamp encoder with GLIGEN bounding-box processing (ablation) | Ours-Sin · AudioCondition test | Eb: 29.11 -> 4.79 | Hurt |
| replace the timestamp encoder with text-only Tango-Timestamp conditioning (ablation) | Ours-Sin · AudioCondition test | Eb: 29.11 -> 9.45 | Hurt |
| replace Fusion-Net with summation (ablation) | Ours · AudioCondition test | DTW: 10.79 -> 24.72 | Hurt |
| set classifier-free guidance scale to 1 rather than 5 (ablation) | Ours · AudioCondition test | Eb: 29.07 -> 11.40 | Hurt |
| set classifier-free guidance scale to 3 rather than 5 (ablation) | Ours · AudioCondition test | Eb: 29.07 -> 27.90 | Hurt |
| increase classifier-free guidance scale from 5 to 10 (ablation) | Ours · AudioCondition test | Eb: 29.07 -> 25.76 | Hurt |
| use 10 inference steps rather than 200 (ablation) | Ours · AudioCondition test | Eb: 29.07 -> 21.54 | Hurt |
| use 50 inference steps rather than 200 (ablation) | Ours · AudioCondition test | Eb: 29.07 -> 30.02 | Helped |
Surprising findings
- Removing classifier-free guidance (scale 1) hurts timestamp control severely (Eb 11.40%), while larger guidance scales beyond 5 reduce controllability.
Limitations
- SED-based evaluation is limited to the 10 sound classes supported by PB-SED, so most classes lack automatic metrics.
- Subjective evaluation uses only 20 clips from the test set.
- Pitch control does not have a clearly optimal inference step setting for all metrics.
Experimental setup
Datasets: AudioCondition AudioSetStrong AudioSet WavCaps AudioCaps
Benchmarks: AudioCondition test set
Metrics: event-based F1 (Eb) clip-level macro F1 (At) pitch standard deviation pitch skewness pitch kurtosis pitch DTW energy MAE subjective expert listening
Baselines: AudioLDM Tango CCTA Tango-Timestamp Ours-Box Ours-Add Ours-Sin
Builds on: Tango GLIGEN AudioLDM classifier-free guidance
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.89)
- MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners (ICML 2025 · similarity 0.88)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.86)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.86)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.86)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.