Research map / 3D vision / 3D generation
MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills 2D MLLM reasoning segmentation (LISA) into a 3D sparse-conv network via attention-filtered multi-view pseudo-masks, a token-for-query alignment loss, and a spatial consistency loss, achieving label-free 3D reasoning segmentation.
Problem
3D reasoning segmentation requires costly 3D-text annotations, and naively transferring 2D MLLM reasoning segmentation to multi-view 3D yields hallucinated objects in invisible views and cross-view inconsistency.
Approach
Use frozen 2D MLLM+SAM (LISA) to generate multi-view pseudo-masks and [SEG] embeddings, weight views by token attention to filter unreliable ones, then train a 3D network with a token-for-query semantic alignment loss (BCE) and a cosine-based spatial consistency loss against 2D features, with no 3D labels.
Key results
- In the label-free setting, VideoLISA (MLLM-For3D) reaches 44.18 Acc@0.25 and 32.90 mIoU on Reason3D ScanNet and 41.00 Acc@0.25 on Instruct3D, while Reason3D-style baselines with labels reach 43.21 and 18.35 respectively.
- On Intent3D, label-free VideoLISA reaches 50.89 AP-style Acc@0.25 vs IntentNet's 41.90 AP@0.25 (+9.0), and 41.56 Acc@0.50 vs 25.36.
- On VG-w/o-ON, label-free VideoLISA reaches 29.50 Acc@0.25 and 24.28 mIoU, outperforming EDA (26.50 Acc@0.25) and other baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Token-for-Query semantic alignment (ablation) | MLLM-For3D (LISA-7B) · Instruct3D (val) | Acc@0.25: 33.92 -> 39.10 | Helped |
| add spatial consistency (ablation) | MLLM-For3D (LISA-7B) · Instruct3D (val) | Acc@0.25: 34.53 -> 39.10 | Helped |
| increase view count from 2 to 4 (ablation) | MLLM-For3D (LISA-7B) · Instruct3D (val) | Acc@0.25: 33.21 -> 39.10 | Helped |
| increase view count from 4 to 8 (ablation) | MLLM-For3D (LISA-7B) · Instruct3D (val) | Acc@0.25: 39.10 -> 39.08 | Hurt |
| replace LISA-7B with LISA-13B (ablation) | MLLM-For3D · Instruct3D (val) | Acc@0.25: 39.10 -> 40.90 | Helped |
| use MLLM-For3D with VideoLISA instead of Reason3D | 3D reasoning segmentation · Instruct3D (ScanNet++) | Acc@0.25: 18.35 -> 41.00 | Helped |
| use MLLM-For3D with VideoLISA instead of Reason3D | 3D reasoning segmentation · Reason3D (ScanNet) | mIoU: 31.20 -> 32.90 | Helped |
| use MLLM-For3D with VideoLISA instead of Reason3D | 3D reasoning segmentation · Reason3D (Matterport3D) | Acc@0.50: 17.43 -> 27.99 | Helped |
| use MLLM-For3D with VideoLISA instead of IntentNet | 3D intention grounding · VG-w/o-ON (val) | mIoU: 18.92 -> 24.28 | Helped |
| use MLLM-For3D with VideoLISA instead of Reason3D | 3D intention grounding · Intent3D (val) | Acc@0.25: 61.71 -> 50.89 | Hurt |
Surprising findings
- Naively unprojecting LISA-7B 2D masks to 3D yields a projection-only baseline that performs far worse, indicating 2D reasoning models lack spatial reasoning when applied to multi-view 3D.
- Increasing views beyond 4 (to 8) does not consistently help and can degrade performance due to occlusions and hallucinated [SEG] tokens.
Limitations
- Computationally demanding because it requires multiple 2D MLLM inferences per scene.
- Pseudo-label uncertainty is not explicitly modeled.
- Performance with more views degrades, and gains are sensitive to view selection.
Experimental setup
Datasets: Reason3D ScanNetV2 Matterport3D Instruct3D ScanNet++ v1 Intent3D VG-w/o-ON ScanRefer
Benchmarks: Reason3D Instruct3D Intent3D VG-w/o-ON
Metrics: Acc@0.25 Acc@0.50 mIoU AP@0.25 AP@0.50
Baselines: Reason3D Intent3D SegPoint IntentNet EDA ChatScene TGNN 3D-STMN
Builds on: LISA VideoLISA Reason3D MinkowskiNet
Closest papers by meaning
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.92)
- Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.90)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.90)
- Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.90)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.