Research map / Vision-language and multimodal / Cross-modal learning
Hear you are: Teaching LLMs Spatial Reasoning with Vision and Spatial Sound
Hyeonggon Ryu, Joon Son Chung, David Harwath
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q8 DoA (multiple matching objects) | DoA accuracy: 46.37 -> 64.27 | Improved |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q1 non-matching | accuracy: 50.75 -> 61.67 | Improved |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q4 invisible audio | DoA accuracy: 16.71 -> 41.18 | Improved |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q6 sounding accuracy | accuracy: 52.33 -> 72.33 | Improved |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q1 sound classification | accuracy: 51.01 -> 52.69 | No clear effect |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q8 class | accuracy: 75.33 -> 70.27 | Hurt |
| use binaural instead of monaural audio (ablation) | Hear You Are LLM · Q3 visible audio | 3-field accuracy: 66.52 -> 69.73 | No clear effect |
| add image input to binaural audio (ablation) | Hear You Are LLM · Q1 coming-from | accuracy: 26.40 -> 69.64 | Improved |
| add image input to binaural audio (ablation) | Hear You Are LLM · Q8 | avg DoA error (deg): 32.32 -> 23.78 | Improved |
| evaluate model trained on R+B+Q with image or audio removed at test time (ablation) | Hear You Are LLM · Q5 and Q6 | sounding accuracy | Hurt |
Closest papers by meaning
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.93)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.92)
- BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.92)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.90)
- SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.90)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
