Research map / Vision-language and multimodal / Visual question answering
The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TWNM, a FOA-conditioned spatial audio-language model with slot-regularized spatial encoder, hybrid projector, progressive training ending in SAPO, evaluated on a 1,000-question simulated ASA benchmark.
Problem
Large audio-language models process audio mostly as a monophonic semantic stream and lack a clear task interface and supervision for spatial audio scene understanding (where sources are, how attributes bind, multi-object relations, and physically plausible scene-level reasoning).
Approach
Formalizes audio scene analysis as three levels (atomic perception, relational integration, cognitive reasoning); feeds FOA multichannel input through a slot-based spatial encoder fused with Whisper semantic features into a Qwen2.5-7B decoder, trained with SFT then SAPO using metadata-derived correctness rewards.
Key results
- TWNM-SAPO reaches 70.8% overall accuracy on the 1,000-question ASA benchmark under exact MCQA scoring, versus 58.85% for Qwen2.5-Omni-7B (exact avg.) and 62.30% for Gemini-3.1-Pro Preview.
- TWNM-SAPO reaches 66.37% on spatial-family questions versus 50.36% for the best listed external reference (GPT-audio).
- On the STARSS23 QA diagnostic, TWNM-SAPO reaches 44.00% overall versus 40.20% for Qwen2.5-Omni-7B; Gaussian-noise input drops it to 25.80%.
Surprising findings
- BAT-SFT stays near chance (25.20% overall, 21.51% on L2) even after binaural fine-tuning under direct ASA prompting.
- SAPO improves L3 scene-level reasoning but regresses on several low-level and binding-heavy families such as Location from ID and Speech.
Limitations
- Main benchmark is simulated with pyroomacoustics shoebox rooms; held-out split does not separate dry-clip identities across render corpora.
- External baselines are non-equivalent diagnostics (different inputs, audit protocols) rather than strict same-input baselines.
- STARSS23 diagnostic uses corpus-level overlap with Stage-1 adaptation data and is limited to static short segments.
Experimental setup
Datasets: FSD50K Clotho LibriTTS-R SongDescriber STARSS23
Benchmarks: TWNM ASA benchmark STARSS23 QA diagnostic
Metrics: accuracy count accuracy DOA MAE distance MAE
Baselines: Qwen2-Audio-7B Qwen2.5-Omni-7B Step-Audio-R1.1 Gemini-3.1-Pro Preview BAT-SFT EINV2 Sp-ACCDOA Qwen3-Omni-Flash GPT-audio
Builds on: Audio Flamingo 3 Whisper SAPO LoRA Sp-ACCDOA
Closest papers by meaning
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.94)
- BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.92)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.92)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.92)
- SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.92)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.