Research map / 3D vision / 3D generation
Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Chat-Scene represents a 3D scene as object-level embeddings (frozen Uni3D and DINOv2 features) paired with learnable object identifier tokens, letting a Vicuna-7B LLM reference and ground objects in a unified QA format trained jointly on five 3D scene-language benchmarks.
Problem
Existing 3D multimodal LLMs lack general object referencing and grounding abilities, and scene-language data is too scarce to train location tokens or scene-level representations effectively.
Approach
Decompose the scene into Mask3D object proposals, assign each a learnable object identifier token, and represent each object by fused 3D (Uni3D) and multi-view 2D (DINOv2) object-centric features, so the LLM references objects by identifier within a unified question-answering format trained jointly without task-specific heads.
Key results
- Chat-Scene reaches 55.5 Acc@0.25 and 50.2 Acc@0.5 on ScanRefer, surpassing prior SOTA by 3.7 points at Acc@0.5.
- On Multi3DRefer, achieves 57.1 F1@0.25 and 52.4 F1@0.5, a 14.0-point gain at F1@0.5 over M3DRef-CLIP.
- On ScanQA, reaches CIDEr 87.7 and B-4 14.3, and on SQA3D EM 54.6 and EM-R 57.5, improving over prior LLM-based models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learnable object identifier tokens instead of plain-text identifiers (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 47.2 -> 50.2 | Helped |
| Use learnable object identifier tokens instead of fixed Gaussian embeddings (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 46.1 -> 50.2 | Helped |
| Use learnable identifiers rather than plain text identifiers (ablation) | Chat-Scene · ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, SQA3D | token cost: 6$N$ -> 3$N$ | Helped |
| Replace 3D-only object features with multi-view 2D features (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 41.2 -> 45.8 | Helped |
| Use multi-view rather than single-view 2D features (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 32.9 -> 45.8 | Helped |
| Add multi-view 2D features as separate object tokens alongside 3D features (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 41.2 -> 50.2 | Helped |
| Early-fuse 3D and multi-view 2D features instead of keeping them as separate tokens (ablation) | Chat-Scene · ScanRefer validation | Acc@0.5: 50.2 -> 46.9 | Hurt |
| Use Chat-Scene rather than ConcreteNet | 3D scene grounding · ScanRefer validation | Acc@0.5: 46.53 -> 50.23 | Helped |
| Use Chat-Scene rather than M3DRef-CLIP | 3D multi-object grounding · Multi3DRefer validation, ALL | F1@0.5: 38.4 -> 52.4 | Helped |
| Use Chat-Scene rather than LEO | 3D dense captioning · Scan2Cap validation | CIDEr@0.5: 68.4 -> 77.19 | Helped |
| Use Chat-Scene rather than LEO | 3D question answering · ScanQA validation | CIDEr: 80.0 -> 87.70 | Helped |
| Use Chat-Scene rather than Scene-LLM | 3D situated question answering · SQA3D test, Avg. | EM: 54.2 -> 54.57 | Helped |
Surprising findings
- A 2D-only multi-view DINOv2 feature outperforms the 3D Uni3D feature alone, suggesting semantic 2D cues matter more than spatial 3D cues in this setup.
- Learnable identifier tokens outperform plain-text identifiers despite cutting token cost from 6N to 3N.
Limitations
- Relies on frozen pre-trained detectors and encoders (Mask3D, Uni3D, DINOv2) that sometimes produce errors.
- Scene-language data scarcity limits spatial relation coverage and causes misclassification of rare classes.
- Hallucination risk because training data does not cover all scenarios.
Experimental setup
Datasets: ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D ScanNet
Benchmarks: ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D
Metrics: Acc@0.25 Acc@0.5 F1@0.25 F1@0.5 CIDEr@0.5 BLEU-4@0.5 CIDEr BLEU-4 EM EM-R
Baselines: ScanRefer ScanQA 3DJCG 3D-VLP M3DRef-CLIP 3D-VisTA ConcreteNet Vote2Cap-DETR++ LAMM Chat-3D 3D-LLM LL3DA
Builds on: Mask3D Uni3D DINOv2 Vicuna LoRA
Closest papers by meaning
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.90)
- SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding (NeurIPS 2026 · similarity 0.89)
- LangSplat: 3D Language Gaussian Splatting (CVPR 2024 · similarity 0.88)
- LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · similarity 0.88)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.88)
- SceneForge: Enhancing 3D-text alignment with Structured Scene Compositions (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.