Research map / Vision-language and multimodal / Multimodal LLMs
3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 3D-LLaVA, a point-cloud-only 3D LMM whose Omni Superpoint Transformer serves as visual token selector, visual prompt encoder, and mask decoder, pretrained with instance segmentation and 2D-to-3D distillation.
Problem
Existing 3D LMMs rely on extra prompt encoders, offline multi-view feature extraction, or task-specific heads, which complicates pipelines; they also lack integrated referring segmentation alongside dialogue and captioning.
Approach
A single versatile visual connector (Omni Superpoint Transformer) that selects top-K visual tokens, encodes visual prompts into the visual token space by reusing the OST, and decodes 3D masks from [SEG] hidden states, pretrained with hybrid instance segmentation and 2D-to-3D CLIP feature distillation.
Key results
- Achieves 92.6 CIDEr on ScanQA (val), improving the prior best (Chat-Scene 87.7) by 4.9 points.
- Reaches 43.3% mIoU on ScanRefer and 42.7% mIoU on Multi3DRefer, surpassing SegPoint by 1.6 and 6.6 points.
- Scores 78.8 CIDEr@0.5 on Scan2Cap and 54.5% EM on SQA3D test, roughly on par with Chat-Scene (54.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OST rather than coordinate projection for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | CiDEr: 68.7 -> 78.8 | Helped |
| Use OST rather than coordinate projection for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | BLEU-4: 33.9 -> 36.9 | Helped |
| Use OST rather than coordinate projection for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | METEOR: 26.7 -> 27.1 | Helped |
| Use OST rather than coordinate projection for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | Rouge-L: 55.1 -> 57.7 | Helped |
| Use OST rather than pooling for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | CiDEr: 76.8 -> 78.8 | Helped |
| Use OST rather than pooling for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | BLEU-4: 36.6 -> 36.9 | Helped |
| Use OST rather than pooling for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | METEOR: 26.9 -> 27.1 | Helped |
| Use OST rather than pooling for visual prompt encoding (ablation) | 3D-LLaVA · Scan2Cap | Rouge-L: 57.5 -> 57.7 | Helped |
| Increase visual tokens from 50 to 100 (ablation) | 3D-LLaVA · ScanQA | CiDEr: 91.1 -> 92.6 | Helped |
| Increase visual tokens from 50 to 100 (ablation) | 3D-LLaVA · Scan2Cap | CiDEr: 74.9 -> 78.8 | Helped |
| Increase visual tokens from 50 to 100 (ablation) | 3D-LLaVA · ScanQA | BLEU-4: 15.9 -> 17.1 | Helped |
| Increase visual tokens from 50 to 100 (ablation) | 3D-LLaVA · Scan2Cap | BLEU-4: 35.9 -> 36.9 | Helped |
Surprising findings
- Box-level grounding by converting masks to boxes gives 51.2% Acc@0.25 but only 40.6% Acc@0.5, trailing Chat-Scene on both thresholds.
- Increasing visual tokens from 100 to 200 gives no substantial gain on ScanQA and Scan2Cap, and 400 tokens slightly degrades performance.
Limitations
- Lags behind Chat-Scene on box-level grounding Acc@0.5 (40.6 vs 50.2).
- Relies on scarce 3D annotated data, which the authors identify as the main obstacle.
- Uses only point clouds, so it lacks 2D image information that competitors such as Chat-Scene exploit.
Experimental setup
Datasets: ScanNet ScanNet200 ScanRefer Nr3D Multi3DRefer ScanQA SQA3D Scan2Cap
Benchmarks: ScanRefer Multi3DRefer ScanQA SQA3D Scan2Cap
Metrics: mIoU CIDEr BLEU-4 METEOR ROUGE-L EM EM-R Acc@0.25 Acc@0.5
Baselines: Chat-Scene Grounded 3D-LLM SegPoint LEO Scene-LLM LL3DA 3D-LLM 3D-VisTA Vote2Cap-DETR 3D-STMN
Builds on: LLaVA-1.5 Vicuna-1.5 OpenScene LoRA Mask3D
Closest papers by meaning
- LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning (CVPR 2024 · similarity 0.91)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.89)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.89)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.88)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.88)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.