Research map / Image generation / Autoregressive and latent generation
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds BEWO-1M, a simulation-based and GPT-assisted stereo audio dataset with spatial captions, and SpatialSonic, a one-stage latent diffusion model that generates spatially controlled stereo audio from text or images via azimuth state matrices.
Problem
Stereo audio generation with explicit spatial control (direction, motion, multiple sources) is hard due to scarce spatially-annotated stereo data and unstable generative models that produce random, indistinct spatial audio.
Approach
Simulates stereo audio with spatial attributes (direction, distance, motion, room size) induced by GPT-4/GPT-4o and converts them into Gaussian coarse or discrete fine azimuth state matrices fused with text/image embeddings in a latent diffusion transformer; image conditioning uses region-aware encoding from a detection-based captioning pre-training.
Key results
- On the BEWO-1M T2A SS-set, SpatialSonic reduces FSAD to 0.17 versus 0.63 for Stable-audio-open and 0.97 for Make-An-Audio2, with MOS-Direction 3.84.
- On the I2A COCO2017 set, SpatialSonic improves FSAD by 1.56 over the best baseline and reaches CLIP score 5.618 versus 4.737.
- On the AudioCaps 1-C T2A benchmark, SpatialSonic obtains IS 13.79 and CLAP 0.672, competitive with the best baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpatialSonic instead of Stable-audio-open | T2A spatial audio generation · BEWO-1M M-set | FSAD: 0.53 -> 0.16 | Helped |
| Use SpatialSonic instead of Stable-audio-open | T2A spatial audio generation · BEWO-1M RW-set | FSAD: 0.60 -> 0.28 | Helped |
| Use SpatialSonic instead of S&H† | V2A spatial audio generation · COCO-2017 V2A | FSAD: 2.08 -> 0.52 | Helped |
| Add coarse and fine azimuth guidance and provide matrix S at inference (ablation) | SpatialSonic · BEWO-1M M-set | FSAD: 0.53 -> 0.16 | Helped |
| Add coarse and fine azimuth guidance and provide matrix S at inference (ablation) | SpatialSonic · BEWO-1M RW-set | FSAD: 0.60 -> 0.28 | Helped |
| Add coarse and fine azimuth guidance and provide matrix S at inference (ablation) | SpatialSonic · I2A | FSAD: 1.41 -> 0.52 | Helped |
| Use a coarse azimuth matrix instead of a fine matrix (ablation) | SpatialSonic · BEWO-1M M-set | FSAD: 0.32 -> 0.16 | Helped |
| Use a coarse azimuth matrix instead of a fine matrix (ablation) | SpatialSonic · BEWO-1M RW-set | FSAD: 0.48 -> 0.28 | Helped |
| Use a fine azimuth matrix instead of a coarse matrix (ablation) | SpatialSonic · I2A | FSAD: 0.96 -> 0.52 | Helped |
| Use T5 instead of CLAP as text encoder (ablation) | SpatialSonic · BEWO-1M M-set (20% training data) | FSAD: 0.56 -> 0.24 | Helped |
| Use T5 instead of CLAP as text encoder (ablation) | SpatialSonic · Audiocaps (20% BEWO-1M training data) | CLAP: 2.33 -> 2.35 | Hurt |
| Use T5+CLAP instead of T5 as text encoder (ablation) | SpatialSonic · BEWO-1M M-set (20% training data) | FSAD: 0.24 -> 0.40 | Hurt |
Surprising findings
- T5 converges slower than CLAP but gives clearly better spatial accuracy (FSAD 0.24 vs 0.56 on M-set at 20% data).
- Fine-grained azimuth matrices hurt T2A (too strict guidance, even for noise), while coarse Gaussian matrices work better for T2A and fine matrices help I2A.
Limitations
- Image encoder's limited size restricts understanding of diverse classes and open-world scenes.
- Simulation ignores head shadow effects; head adaptation left to future fine-tuning.
- Subjective MOS evaluation uses only 15 experts and real-world subset is small (about 200 clips).
Experimental setup
Datasets: AudioCaps WavCaps FSD50K ESC50 VGGSound COCO2017 FairPlay STARSS23 SimBinaural
Benchmarks: AudioCaps test set BEWO-1M test subsets (SS, DS, SD, M, RW) COCO2017 I2A test subset interactive I2A set
Metrics: FD IS KL FAD CLAP score OVL REL GCC MSE CRW MSE FSAD
Baselines: AudioLDM2 Make-An-Audio 2 Stable-audio-open See2sound Seeing-and-hearing AudioGen-L UniAudio Make-An-Audio AudioLDM TANGO TANGO 2
Builds on: Stable Audio AudioLDM 2 T5 CLIP v-prediction diffusion
Closest papers by meaning
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.91)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.91)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.89)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.88)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- ViSAGe: Video-to-Spatial Audio Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.