Research map / 3D vision / 3D generation
Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inst3D-LMM, an instance-aware 3D large multi-modal model that fuses multi-view 2D CLIP features into instance-level 3D tokens (MCMF) and adds spatial-relation scene tokens (3D-ISR), trained end-to-end on multi-task 3D instruction data.
Problem
Existing 3D LMMs encode 3D point and 2D image features separately, neglecting 2D-3D semantic interaction and inter-object spatial relations, which hurts accuracy and token efficiency for multi-task 3D scene understanding.
Approach
Instance-level tokens are produced by cross-attending 3D instance features to per-instance multi-view CLIP features (summarized by learnable [CLS] tokens), and scene-level tokens are produced by spatial-condition self-attention over pairwise distance and angle features among instances; both are fed to an LLM trained jointly on multiple 3D tasks.
Key results
- On ScanRefer, Inst3D-LMM reaches 57.8% Acc@0.25, +2.3 over Chat-Scene (55.5%).
- On ScanQA, Inst3D-LMM achieves 88.6 CIDEr versus 87.7 for Chat-Scene.
- On Scan2Cap@0.50, Inst3D-LMM reaches 38.3 B-4 and 79.7 CIDEr, beating 3D-VisTA and LL3DA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Inst3D-LMM instead of Chat-Scene | 3D visual grounding · ScanRefer | Acc@0.25: 55.5 -> 57.8 | Helped |
| use Inst3D-LMM instead of Chat-Scene | 3D visual grounding · Multi3DRefer | F1@0.25: 57.1 -> 58.3 | Helped |
| use Inst3D-LMM instead of Chat-Scene | 3D question answering · ScanQA | CIDER: 87.7 -> 88.6 | Helped |
| use Inst3D-LMM instead of LL3DA | 3D dense captioning · Scan2Cap@0.50 | B-4: 35.9 -> 38.3 | Helped |
| add multi-view 2D features (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 36.0 -> 57.8 | Helped |
| use MCMF instead of direct concatenation for projection (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 38.8 -> 57.8 | Helped |
| use MCMF instead of parallel projection and combination (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 37.6 -> 57.8 | Helped |
| use MCMF instead of vanilla cross-attention (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 39.2 -> 57.8 | Helped |
| add 3D-ISR spatial relation modeling (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 35.6 -> 48.3 | Helped |
| use 3D-ISR instead of ViL3DRel spatial relation modeling (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 36.1 -> 48.3 | Helped |
| use 3D-ISR instead of 3D-LLM spatial relation modeling (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 39.2 -> 48.3 | Helped |
| use 3D-ISR instead of Chat-3D-v2 relation-aware tokens (ablation) | Inst3D-LMM · ScanRefer | Acc@0.25: 40.8 -> 48.3 | Helped |
Surprising findings
- Vicuna-13B did not uniformly improve over Vicuna-7B: it scored lower on ScanRefer Acc@0.25 (56.0 vs 57.8) and ScanQA EM (52.0 vs 51.6 is higher but CIDEr lower at 83.1 vs 88.6).
Limitations
- Relies on frozen pre-trained 3D instance segmentation (Mask3D) and 2D foundation models, so performance is bounded by their quality.
- Generalist model underperforms some closed-set specialist methods on grounding (e.g., CORE-3DVG, ConcreteNet on ScanRefer).
Experimental setup
Datasets: ScanNetV2 ScanRefer Multi3DRefer ScanQA Scan2Cap ScanNet200
Benchmarks: ScanRefer Multi3DRefer ScanQA Scan2Cap
Metrics: Acc@0.25 Acc@0.50 F1@0.25 F1@0.50 BLEU-4 METEOR ROUGE CIDEr EM
Baselines: Chat-Scene Chat-3D v2 LL3DA Grounded 3D-LLM 3D-LLM 3D-VisTA ReGround3D 3DMIT
Builds on: Chat-Scene ViL3DRel Mask3D SAM CLIP
Closest papers by meaning
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.90)
- LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · similarity 0.89)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.89)
- A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
- SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.