Research map / Vision-language and multimodal / Human and embodied modeling
MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MMScan, a large real-scanned 3D scene dataset with top-down, VLM-initialized and human-corrected hierarchical grounded language annotations, plus visual grounding and QA benchmarks and training gains for grounding and 3D-LLMs.
Problem
Existing 3D-language datasets focus on object-level annotations and inter-object spatial relations, lacking hierarchical region- and object-level grounded descriptions needed for training and benchmarking 3D-LLMs.
Approach
A top-down annotation logic (region to object, single-target to inter-target, spatial and attribute aspects) where VLMs (CogVLM, InternVL-Chat, GPT-4) initialize captions and humans correct them, and the resulting meta-annotations are converted into grounding/QA benchmark samples and grounded scene captions.
Key results
- On the MMScan VG benchmark (IoU@0.25), ViL3DRef achieves the best gTop-1 of 26.34 and EmbodiedScan the highest AP_box of 15.18, far below ScanRefer's reported 48.1% accuracy on its own benchmark.
- Fine-tuned LLaVA-3D reaches 61.6% overall accuracy on the MMScan QA benchmark, versus 23.0 to 31.9 for zero-shot 3D-LLMs.
- Training EmbodiedScan with MMScan co-training plus human-fixed data raises AP25 from 37.27 to 44.44, and LLaVA-3D caption tuning improves ScanQA EM@1 from 23.1 to 24.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train EmbodiedScan grounding by co-training on MMScan captions with Human Fix | EmbodiedScan grounding model · EmbodiedScan grounding, overall | AP_25: 37.27 -> 44.44 | Helped |
| add Human Fix to co-training on MMScan captions (ablation) | EmbodiedScan grounding model · EmbodiedScan grounding, overall | AP_25: 42.96 -> 44.44 | Helped |
| tune LLaVA-3D with all MMScan captions instead of no captions | LLaVA-3D · ScanQA val | EM@1: 23.1 -> 24.7 | Helped |
| tune LLaVA-3D with all MMScan captions instead of no captions | LLaVA-3D · SQA3D test | EM@1: 51.6 -> 54.1 | Helped |
| tune LLaVA-3D with MMScan QA samples instead of no added samples | LLaVA-3D · ScanQA val | EM@1: 23.1 -> 27.9 | Helped |
| tune LLaVA-3D with MMScan QA samples instead of no added samples | LLaVA-3D · SQA3D test | EM@1: 51.6 -> 54.9 | Helped |
| tune LLaVA-3D with MMScan visual grounding samples instead of no added samples | LLaVA-3D · ScanQA val | EM@1: 23.1 -> 24.1 | Helped |
| replace reconstructed points with projected points (ablation) | EmbodiedScan grounding model · MMScan visual grounding, overall | gtop-1_25: 19.66 -> 13.31 | Hurt |
| add multi-view RGB images to projected points (ablation) | EmbodiedScan grounding model · MMScan visual grounding, overall | gtop-1_25: 13.31 -> 16.62 | Helped |
| replace reconstructed points with projected points plus multi-view RGB images (ablation) | EmbodiedScan grounding model · MMScan visual grounding, overall | gtop-1_25: 19.66 -> 16.62 | Hurt |
Surprising findings
- Zero-shot 3D-LLMs perform much worse than expected on MMScan QA, with a ranking different from prior benchmarks.
- Human involvement improves annotation quality, but its effect on grounding training gains is smaller than expected.
Limitations
- Still relies on human annotators for correction, limiting scalability.
- Scene diversity is limited (built on existing real-scanned data).
- Trained 3D-LLMs can hallucinate.
Experimental setup
Datasets: EmbodiedScan ScanRefer ScanQA SQA3D
Benchmarks: MMScan visual grounding benchmark MMScan question answering benchmark MMScan 3D captioning benchmark ScanQA SQA3D
Metrics: AP gTop-k AP_sample AP_box AR GPT-4 accuracy SimCSE Sentence-BERT BLEU-1 BLEU-4
Baselines: ScanRefer BUTD-DETR ViL3DRef ReGround3D MVT 3D-VisTA EmbodiedScan 3D-LLM Chat3D-v2 LL3DA LEO LLaVA-3D
Builds on: EmbodiedScan Sr3D LLaVA-3D Grounded 3D-LLM
Closest papers by meaning
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.90)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.90)
- Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers (NeurIPS 2024 · similarity 0.90)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.90)
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.