Research map / Vision-language and multimodal / Multimodal LLMs
Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Xianzheng Ma, TAO SUN, Shuai Chen, Yash Sanjay Bhalgat, Jindong Gu, Angel X Chang, Iro Armeni, Iro Laina, Songyou Peng, Victor Adrian Prisacariu
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that blind (text-only) fine-tuned LLMs match 3D-LLMs on SQA3D, builds Real-3DQA by filtering 3D-independent questions, adds a viewpoint-rotation score, and proposes 3D-reweighted fine-tuning (3DR-FT) to boost 3D reliance.
Problem
Existing 3D situated QA benchmarks such as SQA3D can be solved largely through textual shortcuts, so reported gains of 3D-LLMs may not reflect genuine 3D spatial reasoning.
Approach
Filter benchmark questions that both a 3D-LLM and its blind (text-only) fine-tuned counterpart answer correctly, add a viewpoint-rotation consistency metric (VRS), and reweight the training loss per token by the blind model's surprisal relative to the 3D model to push reliance on 3D inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D-reweighted fine-tuning instead of supervised fine-tuning | LEO · Real-3DQA | EM_R: 19.1 -> 29.3 | Improved |
| add 3D-reweighted fine-tuning instead of supervised fine-tuning | Chat-Scene · Real-3DQA | EM_R: 22.1 -> 33.9 | Improved |
| add 3D-reweighted fine-tuning instead of supervised fine-tuning | LEO · Real-ScanQA | EM_R: 6.1 -> 13.9 | Improved |
| add 3D-reweighted fine-tuning instead of supervised fine-tuning | Chat-Scene · SQA3D | EM_R: 57.2 -> 48.9 | Hurt |
| train with 3D input instead of blind (text-only) fine-tuning (ablation) | LEO · Real-3DQA | EM_R: 13.6 -> 19.1 | Improved |
| train with 3D input instead of blind (text-only) fine-tuning (ablation) | Chat-Scene · Real-3DQA | EM_R: 14.4 -> 22.1 | Improved |
| add 3D input instead of removing it at inference (ablation) | LEO · SQA3D | EM: 32.4 -> 49.4 | Improved |
| use correct 3D input instead of shuffled 3D input at inference (ablation) | LEO · SQA3D | EM: 44.2 -> 49.4 | Improved |
| add situation description to input at inference (ablation) | LEO · SQA3D | EM: 49.3 -> 49.4 | No clear effect |
| add the question to input at inference (ablation) | LEO · SQA3D | EM: 0.2 -> 49.4 | Improved |
| evaluate on Real-3DQA (filtered, debiased) instead of SQA3D | GPT4Scene · Real-3DQA vs SQA3D | EM: 60.6 -> 33.1 | Hurt |
| evaluate on Real-3DQA (filtered, debiased) instead of SQA3D | LEO · Real-3DQA vs SQA3D | EM: 49.4 -> 14.3 | Hurt |
Surprising findings
- A blind fine-tuned language model without any 3D input can match or surpass state-of-the-art 3D-LLMs on SQA3D.
- 3DR-FT lowers Chat-Scene's SQA3D score from 57.2 to 48.9 while raising Real-3DQA performance, because many degraded questions were shortcut-solvable.
Limitations
- Rotation augmentation covers only four angles (0°, 90°, 180°, 270°) and about 750 base questions (3,000 variants).
- Filtering depends on the specific 3D-LLMs and blind models chosen, and GPT-4o-mini filtering is conservative.
- Many rotation-augmented items required manual correction (about 5% of GPT outputs invalid).
Experimental setup
Datasets: SQA3D ScanQA
Benchmarks: SQA3D Real-3DQA Real-ScanQA
Metrics: EM EM_R (refined exact match) VRS (Viewpoint-Rotation Score)
Baselines: 3D-LLM Chat-3D v2 LEO Chat-Scene GPT4Scene
Builds on: SQA3D LEO Chat-Scene GPT-4o-mini
Closest papers by meaning
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.93)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.92)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
- Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.91)
- SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models (CVPR 2025 · similarity 0.91)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
