Research map / Large language models / Agents, code and math
BAT: Learning to Reason about Spatial Sounds with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds BAT, an LLM (LLaMA-2 7B with LLaMA-adapter v2) that takes binaural spatial audio via a new Spatial-AST encoder and answers spatial sound perception and reasoning questions, trained on a simulated SpatialSoundQA dataset.
Problem
LLMs cannot perceive or reason about in-the-wild spatial (binaural, reverberant 3-D) audio, such as sound event identity, direction, distance, and relations between multiple sources.
Approach
Fuse a binaural spatial audio encoder (Spatial-AST, using mel-spectrograms plus interaural phase difference and three [CLS] tokens for class, distance, and direction) with LLaMA-2 7B via a projection module, trained with a perception-to-reasoning curriculum on simulated SoundSpaces 2.0 QA data.
Key results
- BAT with binaural input and prompt achieves 76.89% average binary accuracy on spatial sound reasoning (type E) in SpatialSoundQA.
- Spatial-AST two-stage binaural model with IPD reaches 50.03 mAP, 17.94° MAE, and 32.54% DER on its encoder tasks.
- Monaural BAT baseline (LTU-like) reaches only 54.53% average reasoning binary accuracy, near random.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the three-stage perception-to-reasoning curriculum instead of one-stage training (ablation) | Bat · SpatialSoundQA Type E | Average binary accuracy: 73.55 -> 76.89 | Helped |
| Remove question types C and D from one-stage training (ablation) | Bat · SpatialSoundQA Type E | Average binary accuracy: 73.55 -> 54.68 | Hurt |
| Train on source-separation tasks before the final reasoning stage (ablation) | Bat · SpatialSoundQA Type E | Average binary accuracy: 54.68 -> 71.93 | Helped |
| Use binaural rather than monaural audio (ablation) | Bat · SpatialSoundQA Type E | Average binary accuracy: 54.53 -> 76.89 | Helped |
| Include both CLS and audio tokens rather than CLS tokens only (ablation) | One-stage Bat · SpatialSoundQA Type E | Average binary accuracy: 66.62 -> 73.55 | Helped |
| Include both CLS and audio tokens rather than audio tokens only (ablation) | One-stage Bat · SpatialSoundQA Type E | Average binary accuracy: 70.33 -> 73.55 | Helped |
| Increase learnable query size from 32 to 64 (ablation) | Bat · SpatialSoundQA Type E | Average binary accuracy: 72.45 -> 76.89 | Helped |
| Increase the detection loss weight lambda_1 (ablation) | Spatial-AST · Spatial-AST encoder | mAP and MAE | Mixed |
| Remove reverberation (ablation) | Spatial-AST · Monaural encoder | mAP: 51.13% -> 52.64% | Helped |
| Add audio normalization (ablation) | Spatial-AST · Monaural encoder | mAP: 51.13% -> 51.39% | Helped |
| Add audio normalization (ablation) | Spatial-AST · Monaural encoder | DER: 39.57% -> 26.87% | Helped |
| Include IPD features (ablation) | Spatial-AST · Binaural single-task detection | mAP: 51.70 -> 52.26 | Helped |
Surprising findings
- Removing sound-source-separation question types (C and D) from one-stage training drops reasoning binary accuracy to near chance (54.68%).
- Monaural input performs comparably or better than binaural on sound event detection and distance prediction for the encoder.
Limitations
- Handles at most two simultaneous sound sources.
- Limited to binaural audio and simulated environments, with an unaddressed sim2real gap.
- Reasoning questions are binary Yes/No, and the model is not open-ended.
Experimental setup
Datasets: AudioSet SoundSpaces 2.0 (simulated binaural RIRs over Matterport3D) SpatialSoundQA
Benchmarks: SpatialSoundQA
Metrics: mAP accuracy distance error rate mean angular error binary accuracy
Baselines: AudioMAE SELDnet Monaural BAT (LTU-like) Random
Builds on: LLaMA-adapter v2 AudioMAE SoundSpaces 2.0 LTU
Closest papers by meaning
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.92)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.91)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.91)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.89)
- SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.89)
- JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.