Research map / 3D vision / 3D generation
Empowering Large Language Models with 3D Situation Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds View2Cap, an automatically generated egocentric 3D situation dataset from RGB-D scan trajectories with VLM captions and QA pairs, and adds a situation grounding module that predicts observer position and orientation from anchor objects, on top of LEO.
Problem
Existing 3D LLMs describe scenes from a global viewpoint and ignore the egocentric observer's situation, so left/right and other spatial descriptions are ambiguous; situation-aware 3D-text data is also scarce and costly to annotate with 3D labels.
Approach
Use camera trajectories of RGB-D scans as egocentric situations, generate region captions and QA pairs from frames with a 2D VLM (LLaVA-OneVision), and ground situations via object anchors that predict position offsets and discretized rotation bins.
Key results
- On Scan2Cap, the method reaches 75.2 CIDEr versus LEO's 72.4, and 56.3 EM@1 on SQA3D test-style setting versus LEO's 55.3 (per Table 2 reported values).
- Situation grounding reaches Acc@1.0m 36.9 and Acc@30° 28.5, improving over SIG3D reproduction (35.2 and 26.3).
- On the ViewQA task, adding SG and View2Cap raises EM from 39.3 (LEO) to 42.0 and EM-R from 44.1 to 46.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use View2Cap and Situation Grounding for 3D scene understanding | 3D LLM · Scan2Cap (val) | CIDEr: 72.4 -> 75.2 | Helped |
| use View2Cap and Situation Grounding for 3D scene understanding | 3D LLM · SQA3D (test) | EM@1: 50.0 (52.4) -> 54.0 (56.0) | Helped |
| add the anchor mechanism (ablation) | LEO w. SG · Situation grounding (SQA3D) | Acc@1.0m: 30.4 -> 32.2 | Helped |
| use discrete rotation bins (ablation) | LEO w. SG with anchors · Situation grounding (SQA3D) | Acc@15°: 16.9 -> 21.6 | Helped |
| pretrain on View2Cap captioning data (ablation) | situation grounding model · Situation grounding (SQA3D) | Acc@0.5m: 13.6 -> 17.4 | Helped |
| add the Situation Grounding module (ablation) | LEO · ViewQA | EM: 39.3 -> 40.2 | Helped |
| add the Situation Grounding module (ablation) | LEO · SQA3D | EM: 62.8 -> 50.8 | Hurt |
| train with View2Cap data (ablation) | LEO with SG module · ViewQA | EM: 40.2 -> 42.0 | Helped |
| train with View2Cap data (ablation) | LEO with SG module · SQA3D | EM: 50.8 -> 54.0 | Helped |
| train with View2Cap data (ablation) | LEO with SG module · ScanRefer | Acc@0.25: 38.3 -> 42.8 | Helped |
| train with both simple and detailed captioning data instead of either alone (ablation) | situation captioning model · Situation captioning | CIDEr: 31.3 -> 36.0 | Helped |
Surprising findings
- The GPT-4 caption score for View2Cap is 3.09 before refinement, suggesting VLM-from-2D captions are a viable supervision source for 3D models.
Limitations
- Evaluation of situation captioning and QA relies on the generated ViewQA data and GPT-4 scoring rather than human-verified labels.
- Relies on ground-truth-like object annotations for anchors and on EmbodiedScan labels for verification.
- Gains on SQA3D EM@1 are modest relative to some baselines and the grounding evaluation is limited to SQA3D.
Experimental setup
Datasets: View2Cap SQA3D ScanNet 3RScan Matterport3D EmbodiedScan ScanRefer Scan2Cap ScanQA
Benchmarks: Scan2Cap ScanQA SQA3D ScanRefer ViewQA
Metrics: CIDEr BLEU-4 METEOR ROUGE-L Sentence-Sim EM@1 EM-R Acc@0.5m Acc@1.0m Acc@15°
Baselines: Scan2Cap 3DJCG Vote2Cap-DETR ScanQA SIG3D 3D-VisTA 3D-LLM LL3DA LEO SceneVerse
Builds on: LEO LLaVA-OneVision SQA3D EmbodiedScan 3D-VisTA
Closest papers by meaning
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- Situational Awareness Matters in 3D Vision Language Reasoning (CVPR 2024 · similarity 0.89)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.89)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.89)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.89)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.