Research map / Video / Video generation
Audible World Models: Spatially Aware Sound Generation for 3D Worlds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Audible World Models, a training-free agentic pipeline that builds a panoramic 3D world proxy, synthesizes label-conditioned dry audio, anchors sources to reconstructed geometry, and renders listener-dependent binaural audio via GSound geometric propagation.
Problem
Generated 3D worlds are usually silent or get soundtracks from text or rendered video, which lack explicit source placement and listener-motion-dependent spatial behavior.
Approach
Treat sound as part of the generated world state: VLM agents parse foreground/background audible labels from a panorama, generate dry audio per label, place sources on reconstructed geometry, predict acoustic parameters, and render binaural audio with geometric acoustic propagation (GSound) along the listener trajectory.
Key results
- Isolated-source DoA MAE of 6.91° with 96.1% accuracy within 15°; mixture DoA MAE 17.44° vs 45.6° for See-2-Sound.
- Human study (20 scenes, 20 participants): mean MOS 4.145 vs 2.498 for strongest baseline; Rank-1 preference 79.5% vs 12.5%.
- Semantic: CLAP 0.3172 vs 0.3366 for Stable Audio 1.0 (best), but best IB-Text (0.2199) and caption score (0.2760).
Surprising findings
- Removing the acoustic simulation keeps CLAP highest (0.328) while collapsing spatial metrics, showing global audio-text similarity does not capture spatial consistency.
- Stable Audio 1.0 obtains the highest CLAP score, so the method does not win every semantic metric.
Limitations
- Static geometry and fixed source placements; no dynamic world construction or object motion inference.
- Offline pipeline, not real-time; construction cost remains.
- DoA references test consistency with the constructed world, not real-world source positions or material accuracy.
Experimental setup
Benchmarks: 80 generated scenes (semantic) isolated-source DoA mixture DoA VLM duration-matched 5s comparison human study on 20 scenes
Metrics: CLAP ImageBind text/image caption score DoA MAE F1@15 F1@45 ILD correlation MOS Rank-1 Top-2
Baselines: Stable Audio 1.0 MMAudio SEE-2-SOUND OmniAudio TangoFlux
Builds on: HunyuanWorld Stable Audio 1.0 GSound SAM2 MoGe-2
Closest papers by meaning
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.91)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.88)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.87)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.87)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.87)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.