Research map / Vision-language and multimodal / Multimodal LLMs

SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SceneCOT, a grounded chain-of-thought framework for 3D scene QA that decomposes reasoning into task recognition, region localization, entity grounding and grounded reasoning, plus the 185K-instance SceneCOT-185K dataset.

Problem

3D LLMs struggle with explainable, grounded reasoning in complex 3D scenes, and answers often lack coherence with object grounding (poor grounding-QA coherence).

Approach

Decompose 3D reasoning into explicit CoT steps with special tokens that invoke a symbolic engine and grounding modules (PQ3D), injecting object probabilities, 3D/polar locations, or object image tokens as visual clues into an LLaVA-1.5 (Vicuna-7B) reasoner trained with LoRA.

Key results

• On MSQA (v2.1), SceneCOT reaches 55.6 overall, with 47.9 on Counting and 47.2 on Spatial, vs. 56.6 for retrained Chat-Scene.
• On Beacon3D, SceneCOT achieves 58.9 case-level and 23.2 object-level GPT-score, and the highest good coherence (GC 34.7 vs. 20.4 for SceneVerse).
• Zero-shot grounding F1@50 on MSQA, SQA3D, and ScanQA is 52.1, 51.6, and 40.8, well above Chat-Scene (15.9, 25.9, 35.9).

Surprising findings

• Chat-Scene's grounding performance was hard to reproduce, with 8-GPU attempts remaining low, which the authors attribute to sensitivity to training settings.
• SceneCOT lags on Existence and Attribute versus LEO and MSR3D, attributed to grounding errors.

Limitations

• Limited to tasks predefined in MSQA and ScanNet scenes; does not address long-horizon embodied task planning.
• Substantially higher inference latency due to multi-module calls and long reasoning traces.
• Thought design struggles on Spatial Relationship; performance bounded by grounding and semantic errors.

Experimental setup

Datasets: MSQA Nr3D Beacon3D ScanNet SceneVerse

Benchmarks: MSQA Beacon3D Nr3D SQA3D ScanQA

Metrics: GPT-score GPT-score by object grounding-QA coherence (GC) F1@50 Top-1 grounding accuracy EM-R

Baselines: GPT-4o LEO MSR3D Chat-Scene SplatTalk PQ3D SceneVerse LLaVA-3D

Builds on: LLaVA-1.5 PQ3D Chain-of-Thought prompting Mask3D

Closest papers by meaning

• SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding (NeurIPS 2026 · similarity 0.90)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
• Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.89)
• SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.89)
• Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.88)
• Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.88)