Research map / Vision-language and multimodal / Multimodal LLMs
Multi-modal Situated Reasoning in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MSQA, a 251K-pair interleaved text-image-point-cloud situated QA dataset generated with scene graphs and LLMs, plus the MSNN next-step navigation benchmark and an MSR3D baseline built on LEO.
Problem
Existing 3D situated reasoning datasets are small, costly to collect, limited to single-modality (text) situation descriptions that cause referral ambiguity, and narrow in task scope.
Approach
Scalably generate situated QA pairs by sampling viewpoints in real 3D scenes, adjusting scene graphs to be situated, and prompting LLMs; use interleaved text-image-point-cloud inputs and an MSR3D model that normalizes the scene to the agent's pose.
Key results
- On MSQA, fine-tuned MSR3D with point cloud text-only input reaches 56.48 overall correctness, versus 50.65 for zero-shot GPT-3.5 and 7.84 for zero-shot LEO.
- Interleaved T+I input (54.13) slightly underperforms text-only (56.48) for MSR3D, with a larger drop when images appear only in questions (43.58 vs 56.41).
- GPT-based scoring correlates with human scores at Pearson 0.941, versus 0.708 for exact match and 0.19 for BLEU-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LEO on MSQA | LEO · MSQA | correctness score: 7.84 -> 55.86 | Helped |
| Use MSR3D situation modeling with point cloud input rather than LEO FT | 3D situated question answering · MSQA | correctness score: 55.86 -> 56.48 | Helped |
| Add interleaved text-image input instead of text-only input (ablation) | MSR3D · MSQA | correctness score: 56.48 -> 54.13 | Hurt |
| Use point cloud instead of textual scene descriptions (ablation) | MSR3D · MSQA | correctness score: 50.05 -> 56.48 | Helped |
| Add 3D scene input (ablation) | MSR3D · MSQA | correctness score: 22.92 -> 56.48 | Helped |
| Add the situation component (ablation) | MSR3D · MSQA | correctness score: 51.20 -> 56.48 | Helped |
| Use interleaved text-image input when images appear only in the situation (ablation) | MSR3D · MSQA subset: situation has image, question has no image | correctness score: 55.54 -> 56.48 | Helped |
| Use interleaved text-image input when images appear only in the question (ablation) | MSR3D · MSQA subset: situation has no image, question has image | correctness score: 56.41 -> 43.58 | Hurt |
| Use point cloud rather than textual scene descriptions (ablation) | MSR3D · MSQA counting, overall | correctness score: 35.82 -> 33.46 | Hurt |
| Use situation transformation rather than agent-as-object modeling (ablation) | MSR3D · SQA3D | EM-refined: 51.92 -> 52.94 | Helped |
| Use situation transformation rather than embedding modeling (ablation) | MSR3D · SQA3D | EM-refined: 50.72 -> 52.94 | Helped |
| Use situation transformation rather than cross-attention modeling (ablation) | MSR3D · SQA3D | EM-refined: 51.21 -> 52.94 | Helped |
Surprising findings
- Interleaved image input slightly hurts performance versus text-only input, especially when images appear only in the question.
- Textual scene descriptions beat point clouds on counting for small counts but collapse for counts of 3 or more.
Limitations
- LLM-generated data contains unnatural situations and biases, such as distant objects chosen for descriptions.
- Scene-graph generation is rule-based and the pipeline does not yet use all available 3D assets.
- Evaluation is limited to QA and single-step action prediction rather than object grounding or other tasks.
Experimental setup
Datasets: MSQA MSNN ScanNet 3RScan ARKitScenes SQA3D
Benchmarks: MSQA MSNN SQA3D
Metrics: GPT-score correctness accuracy Pearson correlation with human scores
Baselines: GPT-3.5 GPT-4o Claude-3.5-Sonnet LEO
Builds on: LEO SQA3D OpenEQA Vicuna LoRA
Closest papers by meaning
- Situational Awareness Matters in 3D Vision Language Reasoning (CVPR 2024 · similarity 0.91)
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.90)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.90)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.90)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)
- Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.