Research map / 3D vision / 3D generation

SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding

NeurIPS 2026 · 3D vision › 3D generation · approach: Open-vocabulary 3D scene understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SceneScaffold, which turns the visual bottleneck of a 3D-LMM into an active scene-state constructor (entity, scene-frame, relation, and global summary states plus residual tokens) under a fixed 100-token budget, and reports gains over 3D-LLaVA on grounding, QA, and dense captioning.

Problem

Object-centric homogeneous visual compression in 3D large multimodal models under-represents spatial scene organization, forcing the LLM to recover spatial relations from a flat token sequence and reducing reliability on relation-intensive and spatially ambiguous 3D queries.

Approach

Replace passive feature compression with role-aware scene-state construction: entity states read top-K object-core superpoints, scene-frame states read geometric boundary and grid-region anchors, relation states are induced from entity and frame states, and a global summary is pooled, all serialized with role embeddings and trained with role-preserving losses.

Key results

• SceneScaffold reaches 47.0 mIoU on ScanRefer and 47.9 mIoU on Multi3DRefer, improving over 3D-LLaVA by 3.7 and 5.2 mIoU.
• On ScanQA SceneScaffold scores 95.8 CIDEr (vs 92.6 for 3D-LLaVA) and on SQA3D 55.5 EM (vs 54.5).
• On the Rel.+Amb. diagnostic subsets, gains over 3D-LLaVA are 3.0 mIoU on ScanRefer and 2.8 mIoU on Multi3DRefer.

Surprising findings

• Replacing any state component with extra residual detail tokens does not recover full performance, suggesting gains come from role-factored organization rather than token count.
• Removing the relation states slightly improves SQA3D EM (56.4 vs 55.5), which the paper does not explain in detail.

Limitations

• Gains over the strongest prior point-cloud generalist are modest on several tasks (e.g. Scan2Cap, ScanQA CIDEr below NDTokenizer3D).
• Diagnostic Rel.+Amb. subset is defined by keyword heuristics and the relation/frame regularizers are coarse geometric labels.

Experimental setup

Datasets: ScanRefer Multi3DRefer ScanQA SQA3D Scan2Cap

Benchmarks: ScanRefer Multi3DRefer ScanQA SQA3D Scan2Cap

Metrics: mIoU CIDEr BLEU-4 METEOR ROUGE-L EM EM-R

Baselines: 3D-LLaVA NDTokenizer3D Chat-Scene LEO Scene-LLM LSceneLLM Grounded 3D-LLM LL3DA

Builds on: 3D-LLaVA LLaVA LoRA

Closest papers by meaning

• Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.90)
• SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes (ICLR 2026 · similarity 0.90)
• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.90)
• LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · similarity 0.89)
• The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.89)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers (NeurIPS 2024 · similarity 0.89)