Research map / Vision-language and multimodal / Multimodal LLMs
LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D Capabilities
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, Xihui Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D position encoding to 2D patches (3D patches) (ablation) | LLaVA-3D · MMScan QA | accuracy: 42.1 -> 55.4 (+13.3) | Improved |
| add 3D position encoding to 2D patches (3D patches) (ablation) | LLaVA-3D · Scan2Cap | CIDEr: 29.7 -> 84.1 (+54.4) | Improved |
| add 3D position encoding to 2D patches (ablation) | LLaVA-3D · ScanQA | metric: 29.4 -> 29.8 (+0.4) | No clear effect |
| add 3D position encoding to 2D patches (ablation) | LLaVA-3D · SQA3D | accuracy: 59.8 -> 60.1 (+0.3) | No clear effect |
| add grounding decoder instead of directly outputting text coordinates (ablation) | LLaVA-3D · ScanRefer | Acc@0.25: 7.8 -> 50.1 | Improved |
| add grounding decoder instead of special-token coordinate output (ablation) | LLaVA-3D · ScanRefer | Acc@0.25: 8.2 -> 50.1 | Improved |
| use CLIP only instead of SAM + CLIP for 3D scene features (ablation) | 3D LMM from Vicuna-7B · ScanRefer-related benchmarks | inference time: 900s -> 0.2s | Improved |
| use CLIP only instead of SAM + CLIP features (ablation) | 3D LMM from Vicuna-7B · ScanQA | metric: 22.1 -> 23.4 | Improved |
| use Pooling + MLP instead of Q-Former connector (ablation) | 3D LMM from Vicuna-7B · ScanQA | metric: 21.9 -> 22.1 | No clear effect |
| build from 2D LMM LLaVA-1.5 instead of LLM Vicuna (ablation) | LLaVA-3D · ScanRefer | Acc@0.25: 43.8 -> 47.9 | Improved |
| use stronger 2D base (InternVL2.5-7B instead of LLaVA-1.5-7B) (ablation) | LLaVA-3D · ScanQA | metric: 27.0 -> 29.1 | Improved |
| use video LMM base (LLaVA-Video-7B instead of InternVL2.5-7B) (ablation) | LLaVA-3D · SQA3D | accuracy: 58.8 -> 60.1 | Improved |
Later papers that compared with this method
1 later accepted paper reports LLaVA-3D as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding (ECCV 2026 · better on 4 of 5 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer (CVPR 2025 · similarity 0.93)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.91)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.90)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.89)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
