Research map / Vision-language and multimodal / Multimodal LLMs

JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends the Qwen2.5-Omni audio-visual LLM to 3D by adding RGB-D with depth-based positional encodings and first-order ambisonics audio with a learned Neural Intensity Vector, trained on a 61K synthetic SpatialSceneQA benchmark.

Problem

Audio-visual LLMs operate on 2D RGB video and monaural audio, which prevents reliable sound source localization and 3D spatial reasoning in physical environments.

Approach

Replace the STFT-based classical intensity vector with a learnable 1D-CNN encoder over raw FOA waveforms (Neural IV), combine it with depth-projected 3D positional encodings for RGB features, and adapt an AV-LLM via LoRA for end-to-end DoA estimation, 3D grounding, and speaker matching.

Key results

• JAEGER (Neural IV) reaches 2.21° median DoA error for single source and 4.11° for overlapping sources, versus 19.09° for BAT on overlapping sources.
• Visual grounding achieves 0.32 3D IoU and 0.16 m median visual offset, against 0.00 IoU for N3D-VLM and 0.01 for Qwen3-VL-8B.
• Two-speaker audio-visual reasoning accuracy is 99.2% with Neural IV, while zero-shot Qwen2.5-Omni reaches 44.0%.

Surprising findings

• Removing the FOA encoder drops reasoning accuracy to near-random (about 43-48%) even after fine-tuning.
• Under harder 4-6 candidate settings, Neural IV accuracy falls to 72.5% with six candidates, below Classical IV's 80.0%.

Limitations

• Evaluation is primarily simulation-based; transfer to real acoustics, sensors, and calibration is not established.
• Real-world STARSS23 experiment uses only 753 clips and is a sanity check.
• Harder-setting experiments use only 160 training samples per candidate setting.

Experimental setup

Datasets: SpatialSceneQA LibriSpeech HM3D STARSS23

Benchmarks: SpatialSceneQA STARSS23

Metrics: median angular error 3D IoU visual offset accuracy

Baselines: BAT Qwen2.5-Omni Qwen3-VL-8B N3D-VLM SimpleFuse JAEGER (Classical IV)

Builds on: Qwen2.5-Omni LoRA N3D-VLM data2vec SoundSpaces 2.0

Closest papers by meaning

• SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.89)
• OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.89)
• The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.89)
• Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.89)
• BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.88)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.88)