Research map / 3D vision / 3D generation
SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Spartun3D, an LLM-generated situated 3D spatial QA and captioning dataset (~133k examples), and Spartun3D-LLM, which adds a situated spatial alignment loss to LEO to align object features with situated text descriptions.
Problem
Current 3D-based LLMs struggle with situated understanding from an agent's egocentric perspective because existing 3D datasets are global-view and human-annotated, and their architectures lack explicit alignment between 3D spatial representations and language.
Approach
Generate situated 3D data with GPT-4o from a situated scene graph (agent standing point, orientation, rotation angles, direction, distance, passby objects), and add an object-text alignment loss with spatial self-attention that pulls object features toward CLIP embeddings of situated object descriptions on top of LEO.
Key results
- On SQA3D zero-shot with Mask3D proposals, Spartun3D-LLM reaches EM 36.2 vs LEO's 12.4.
- On Spartun3D situated QA with Vicuna-7B initialized from LEO, Spartun3D-LLM* reaches EM 56.9 on attributes and relations and CIDEr 254.7 on affordance.
- Zero-shot navigation accuracy on MP3D ObjNav is 20.3% for Spartun3D-LLM versus 0% for LEO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train LEO on Spartun3D instead of the original LEO training data | LEO · SQA3D zero-shot, Mask3D | exact match: 12.4 -> 34.7 | Helped |
| add the situated spatial alignment module (ablation) | LEO+Spartun3D · SQA3D zero-shot, Mask3D | exact match: 34.7 -> 36.2 | Helped |
| add the situated spatial alignment module (ablation) | LEO*+Spartun3D · SQA3D fine-tune, Mask3D | exact match: 53.5 -> 54.8 | Helped |
| add the situated spatial alignment module (ablation) | LEO+Spartun3D · Spartun3D captioning | sentence similarity: 76.3 -> 78.1 | Helped |
| add the situated spatial alignment module (ablation) | LEO*+Spartun3D · Spartun3D affordance | CIDEr: 211.3 -> 254.7 | Helped |
| initialize from instruction-tuned LEO weights | Spartun3D-LLM · Spartun3D attributes and relations | exact match: 53.7 -> 56.9 | Helped |
| replace CLIP text encoder with BERT (ablation) | Spartun3D-LLM · Spartun3D attributes and relations | exact match: 56.9 -> 53.5 | Hurt |
| replace CLIP text encoder with BLIP (ablation) | Spartun3D-LLM · Spartun3D attributes and relations | exact match: 56.9 -> 54.2 | Hurt |
| use the Spa-prompt with calculated qualitative spatial relations instead of the Cord-prompt with raw coordinates (ablation) | GPT-4o dataset generation · Spartun3D human evaluation | percentage of examples with high human scores | Helped |
| scale up the Spartun3D training dataset (ablation) | LEO on SQA3D · SQA3D | performance | Helped |
Surprising findings
- Coordinate-based prompts produced poor-quality generated data; qualitative spatial relations (distance, direction) prompts were needed for high human scores.
- LEO was biased toward answering 'left' about 97% of the time on 'which direction' questions, and training on Spartun3D largely mitigated this bias.
Limitations
- Synthetic data contains errors (semantic errors, common-sense violations, spatial inconsistencies, visual-textual misalignment) in about 26 of 200 sampled examples.
- Dataset depends on human-annotated 3RScan object labels and affordances, which can bias GPT-4o generations.
- Navigation evaluation is limited to 5 scenes (~1000 examples) and zero-shot setting.
Experimental setup
Datasets: Spartun3D SQA3D 3RScan ScanNet MP3D ObjNav Scan2Cap ScanQA
Benchmarks: Spartun3D SQA3D MP3D ObjNav Scan2Cap ScanQA
Metrics: CIDEr METEOR BLEU-4 ROUGE-L sentence similarity exact match accuracy
Baselines: LEO LEO+Spartun3D 3D-Vista 3D-LLM
Builds on: LEO SQA3D 3RScan LoRA CLIP
Closest papers by meaning
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.90)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.89)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)
- SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.