Research map / Vision-language and multimodal / Multimodal LLMs
SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAVVY-Bench, a benchmark for 3D spatial reasoning over dynamic, spatial-audio egocentric videos, and SAVVY, a training-free pipeline that builds a dynamic global map from AV-LLM snapshots, segmentation, and spatial-audio cues to boost AV-LLM accuracy.
Problem
Existing AV-LLMs and spatial benchmarks focus on static or 2D scenes and downmix spatial audio to mono, so they cannot reason about direction and distance of moving sound sources in 3D dynamic audio-visual scenes.
Approach
A training-free two-stage plugin: estimate egocentric (azimuth, distance) tracks of target/reference/facing objects from AV-LLM snapshots, text-guided segmentation with monocular depth, and SRP-PHAT/CDR spatial audio; then aggregate them into a 2D global map and apply a coordinate transform to answer egocentric or allocentric queries.
Key results
- SAVVY on Gemini-2.5-pro improves egocentric direction by +9.5% and allocentric direction by +12.3% accuracy.
- Abstract/intro reports +7.1% overall QA accuracy over the best AV-LLM (Gemini-2.5 Pro).
- Human accuracy is 78.7% overall, 20.7% above SAVVY, which is the best method; most open-source AV-LLMs score under 5% t-mIoU on temporal grounding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAVVY spatial-reasoning pipeline to Gemini-2.5 Pro | Gemini-2.5 Pro · SAVVY-Bench overall | accuracy: +7.1% | Helped |
| Add SAVVY spatial-reasoning pipeline | Gemini-2.5 Pro · SAVVY-Bench egocentric direction | accuracy: +9.5% | Helped |
| Add SAVVY spatial-reasoning pipeline | Gemini-2.5 Pro · SAVVY-Bench allocentric direction | accuracy: +12.3% | Helped |
| Add SAVVY spatial-reasoning pipeline | Gemini-2.5 Pro · SAVVY-Bench egocentric distance | distance accuracy | Helped |
| Add SAVVY spatial-reasoning pipeline | Gemini-2.5 Pro · SAVVY-Bench allocentric distance | distance accuracy | Helped |
| Use SD and Audio tracks without Seg instead of all track types (ablation) | SAVVY · SAVVY-Bench allocentric QA | accuracy: +12.7% | Helped |
| Use SD and Audio tracks without Seg instead of all track types (ablation) | SAVVY · SAVVY-Bench allocentric QA | distance accuracy: +4.0% | Helped |
| Add SD tracks to Audio-only tracking (ablation) | Audio-only tracking · SAVVY-Bench localization | localization accuracy: +7.8% | Helped |
| Use Seg-based tracks alone (ablation) | SAVVY · SAVVY-Bench localization | loc_acc: 72.5% | Helped |
| Add Seg tracks to SD+Audio (ablation) | SD+Audio · SAVVY-Bench allocentric QA | accuracy | Hurt |
| Combine SD, Seg, and Audio tracks (ablation) | SAVVY · SAVVY-Bench egocentric direction | accuracy: +3.5% | Helped |
| Combine SD, Seg, and Audio tracks (ablation) | SAVVY · SAVVY-Bench egocentric distance | distance accuracy: +8.3% | Helped |
Surprising findings
- Most open-source AV-LLMs perform at or below chance on allocentric relative direction questions.
- Gemini-2.5-pro outperforms Gemini-2.5-flash on temporal grounding by 24.8%, but most open-source models achieve under 5% temporal mIoU.
Limitations
- Relies on monaural-downmixed input for AV-LLMs and on handcrafted spatial audio pipeline that is training-free.
- Open-source AV-LLMs struggle with temporal grounding and object referral at 7B scale.
- Segmentation module adds noise to allocentric static estimates.
Experimental setup
Datasets: Aria Everyday Activities (AEA)
Benchmarks: SAVVY-Bench
Metrics: accuracy average relative accuracy for distance t-mIoU referral accuracy localization accuracy
Baselines: Gemini-2.5-pro Gemini-2.5-flash VideoLLaMA2 LongVALE EgoGPT Video-SALMONN Ola MiniCPM-o-2.6
Builds on: SRP-PHAT CLIPSeg SAM2 Kalman filter DBSCAN
Closest papers by meaning
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.93)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.92)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.90)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.90)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.90)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.