Research map / Vision-language and multimodal / Multimodal LLMs

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HLR-AVSceneQA, a simulator-built spatial audio-visual QA benchmark (hear-localize-reason hierarchy), and HLR-LLM, Qwen2.5-Omni extended with a binaural spatial audio branch trained via a three-stage curriculum ending in CoT tuning.

Problem

Existing audio-visual LLMs treat audio as monaural semantic content and cannot reason about where sounds originate relative to the visual scene; existing spatial audio-visual benchmarks cover only fragmented subproblems and are trained with direct-answer supervision.

Approach

Extend a strong monaural AV-LLM with a parallel binaural spatial audio encoder/projector (Spatial-AST tokens) and train with a curriculum whose final stage uses templated chain-of-thought rationales derived from simulator metadata, evaluated on a unified hear-localize-reason benchmark.

Key results

• HLR-LLM reaches 78.21% egocentric direction and 52.03% allocentric direction accuracy, versus 64.27% and 42.71% for dual-channel Qwen2.5-Omni.
• HLR-LLM achieves 61.50% proximity accuracy and 80.87% correspondence vs 55.64% and 72.59% for dual-channel Qwen2.5-Omni.
• On STARSS23 fine-tuning with HLR-AVSceneQA pretraining, HLR-LLM improves sound recognition from 44.59% to 53.93% and direction from 16.13% to 29.92%.

Surprising findings

• CoT tuning degrades scene captioning metrics and far-field ASR, a trade-off with structured reasoning.
• Removing vision still yields relatively strong hidden-source accuracy (66.62%), suggesting binaural cues carry useful evidence for non-visible source configurations.

Limitations

• Benchmark is simulator-based and cannot fully capture real-world acoustic and visual complexity.
• Focuses on indoor static scenes; no moving sources or dynamic agents.
• Fine-grained direction ambiguity near bin boundaries and multi-source overlap remain failure modes.

Experimental setup

Datasets: HLR-AVSceneQA Spatial audio-only QA (BAT) VGGSound LibriSpeech HM3DSem Objaverse-XL STARSS23

Benchmarks: HLR-AVSceneQA STARSS23

Metrics: accuracy word error rate CIDEr-D METEOR SPIDEr

Baselines: MiniCPM-o-v2.6 VideoLLaMA2 Video-SALMONN 2 Qwen2.5-Omni Qwen3-Omni SAVVY GPT-Realtime-Mini Gemini-2.5-Flash-Lite Hear You Are LLM Dual-channel Qwen2.5-Omni

Builds on: Qwen2.5-Omni BAT Hear You Are LLM SoundSpaces 2.0 chain-of-thought prompting

Closest papers by meaning

• The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.94)
• SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.93)
• BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.91)
• OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.91)
• Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation (CVPR 2025 · similarity 0.90)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.90)