Research map / Point clouds / Point cloud classification
Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Situat3DChange, a situated 3D change QA/description/rearrangement dataset built on 3RScan with human and GPT-4 generated text, plus SCReasoner, a token-efficient 3D MLLM that compares two point clouds via a Mamba-based selective projector and star-operation fusion.
Problem
Existing 3D situated reasoning datasets use static scenes, while 3D change datasets are synthetic and lack situational context; MLLMs also prepend all tokens from two highly similar point clouds, which is redundant and fails to highlight changes.
Approach
A situation-aware change dataset with egocentric and allocentric spatial information, paired with SCReasoner, which selects informative tokens from the previous scene using Mamba and fuses them with current-scene tokens via star operation, adding no extra tokens to the LLM decoder.
Key results
- SCReasoner (mamba*) reaches GPT score 13.9 on change description and 30.7 on rearrangement, versus LEO's 12.7 and 30.1 and InternVL2's 8.2 and 16.3 after fine-tuning.
- On the QA task, fine-tuned InternVL2-7B averages 54.0 versus 53.8 for SCReasoner (mamba*) and 51.9 for LEO.
- Over 5 seeds, SCReasoner improves QA, description, and rearrangement over LEO (53.42 vs 52.76, 13.63 vs 12.73, 30.75 vs 30.17).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LEO's paired-scene token input with SCReasoner's selective comparison projector and fusion | LEO · Change description | GPT: +1.2% | Helped |
| Replace LEO's paired-scene token input with SCReasoner's selective comparison projector and fusion | LEO · Rearrangement instruction | GPT: +0.6% | Helped |
| Use Mamba rather than a linear projection for token selection (ablation) | SCReasoner · Change description | GPT: 13.4 -> 13.9 | Helped |
| Use Mamba rather than a linear projection for token selection (ablation) | SCReasoner · Rearrangement instruction | GPT: 30.3 -> 30.7 | Helped |
| Use element-wise multiplication rather than addition for fusion with Mamba selection (ablation) | SCReasoner · Change description | GPT: 13.3 -> 13.9 | Helped |
| Use element-wise multiplication rather than addition for fusion with Mamba selection (ablation) | SCReasoner · Rearrangement instruction | GPT: 30.5 -> 30.7 | Helped |
| Fine-tune on S3C+SN rather than S3C alone | SCReasoner · Scan2Cap (val) | CIDEr: 72.4 -> 83.4 | Helped |
| Fine-tune on S3C+SN rather than S3C alone | SCReasoner · Situat3DChange | Average QA: 53.8 -> 52.5 | Hurt |
| Add ScanNet benchmarks (SN) to S3C fine-tuning | SCReasoner · Change description | GPT: 13.9 -> 16.8 | Helped |
| Use panorama rather than stitched multi-view input (ablation) | one-shot InternVL · Situat3DChange | QA: 35.5 -> 35.7 | No clear effect |
| Add one-shot chain-of-thought prompting (ablation) | InternVL · Change description | GPT: 3.8 -> 5.3 | Helped |
| Add one-shot chain-of-thought prompting (ablation) | Janus · Rearrangement instruction | GPT: 4.7 -> 4.3 | Hurt |
Surprising findings
- Fine-tuning only on S3C causes catastrophic forgetting on ScanNet benchmarks such as Scan2Cap and ScanQA.
- Zero-shot and one-shot 2D MLLMs perform poorly on long-form tasks, with GPT scores mostly below 5 and near 0 for several models.
Limitations
- Human change annotations are limited, with each change annotated by only one annotator.
- The 3RScan test labels are unavailable, so only hold-out evaluation on the validation split is supported.
- The human-annotated data is not adapted to other domains, making the pipeline hard to transfer.
Experimental setup
Datasets: 3RScan 3DSSG ScanNet benchmarks (Scan2Cap, ScanQA, SQA3D) MMMU
Benchmarks: Situat3DChange (S3C) Scan2Cap ScanQA SQA3D
Metrics: CIDEr BLEU-4 METEOR ROUGE sentence similarity GPT score accuracy EM@1 revised REL for distance
Baselines: LEO InternVL2-7B DeepSeek-VL-7B DeepSeek-VL2-1.3B Janus-7B Qwen2.5-VL-7B LLaVA-NeXT-7B
Builds on: LEO MSQA 3RScan 3DSSG Mamba InternVL2
Closest papers by meaning
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.89)
- Empowering Large Language Models with 3D Situation Awareness (CVPR 2025 · similarity 0.88)
- From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.88)
- SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model (ICLR 2025 · similarity 0.88)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.87)
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.