AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Do 3D Large Language Models Really Understand 3D Spatial Relationships?

Xianzheng Ma, TAO SUN, Shuai Chen, Yash Sanjay Bhalgat, Jindong Gu, Angel X Chang, Iro Armeni, Iro Laina, Songyou Peng, Victor Adrian Prisacariu

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 5 citations (OpenAlex)

Read the paper

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that blind (text-only) fine-tuned LLMs match 3D-LLMs on SQA3D, builds Real-3DQA by filtering 3D-independent questions, adds a viewpoint-rotation score, and proposes 3D-reweighted fine-tuning (3DR-FT) to boost 3D reliance.

Problem

Existing 3D situated QA benchmarks such as SQA3D can be solved largely through textual shortcuts, so reported gains of 3D-LLMs may not reflect genuine 3D spatial reasoning.

Approach

Filter benchmark questions that both a 3D-LLM and its blind (text-only) fine-tuned counterpart answer correctly, add a viewpoint-rotation consistency metric (VRS), and reweight the training loss per token by the blind model's surprisal relative to the 3D model to push reliance on 3D inputs.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add 3D-reweighted fine-tuning instead of supervised fine-tuningLEO · Real-3DQAEM_R: 19.1 -> 29.3Improved
add 3D-reweighted fine-tuning instead of supervised fine-tuningChat-Scene · Real-3DQAEM_R: 22.1 -> 33.9Improved
add 3D-reweighted fine-tuning instead of supervised fine-tuningLEO · Real-ScanQAEM_R: 6.1 -> 13.9Improved
add 3D-reweighted fine-tuning instead of supervised fine-tuningChat-Scene · SQA3DEM_R: 57.2 -> 48.9Hurt
train with 3D input instead of blind (text-only) fine-tuning (ablation)LEO · Real-3DQAEM_R: 13.6 -> 19.1Improved
train with 3D input instead of blind (text-only) fine-tuning (ablation)Chat-Scene · Real-3DQAEM_R: 14.4 -> 22.1Improved
add 3D input instead of removing it at inference (ablation)LEO · SQA3DEM: 32.4 -> 49.4Improved
use correct 3D input instead of shuffled 3D input at inference (ablation)LEO · SQA3DEM: 44.2 -> 49.4Improved
add situation description to input at inference (ablation)LEO · SQA3DEM: 49.3 -> 49.4No clear effect
add the question to input at inference (ablation)LEO · SQA3DEM: 0.2 -> 49.4Improved
evaluate on Real-3DQA (filtered, debiased) instead of SQA3DGPT4Scene · Real-3DQA vs SQA3DEM: 60.6 -> 33.1Hurt
evaluate on Real-3DQA (filtered, debiased) instead of SQA3DLEO · Real-3DQA vs SQA3DEM: 49.4 -> 14.3Hurt

Surprising findings

Limitations

Experimental setup

Datasets: SQA3D ScanQA

Benchmarks: SQA3D Real-3DQA Real-ScanQA

Metrics: EM EM_R (refined exact match) VRS (Viewpoint-Rotation Score)

Baselines: 3D-LLM Chat-3D v2 LEO Chat-Scene GPT4Scene

Builds on: SQA3D LEO Chat-Scene GPT-4o-mini

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy