Research map / Vision-language and multimodal / Multimodal LLMs
Language-Image Models with 3D Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LV3D, a 2D+3D multi-turn QA pretraining mix, and CUBE-LLM, a LLaVA-1.5-style MLLM (Vicuna-7B + DINOv2) pretrained on it, showing 3D grounding and reasoning gains from data scaling plus visual chain-of-thought and specialist box prompting.
Problem
Existing multimodal LLMs ground and reason only in 2D image coordinates; extending them to 3D grounding and reasoning (e.g., for driving scenes) requires data and task design that has not been explored.
Approach
Unify heterogeneous 2D and 3D datasets into multi-turn QA with decomposed 2D-to-3D token sequences (point, box, depth, size, orientation) so that a standard next-token MLLM learns 3D perception through data scaling alone, with visual chain-of-thought and candidate-box prompting at inference.
Key results
- On Talk2Car, CUBE-LLM with VCoT and CenterPoint box prompts reaches 71.4 BEV AP_A vs 50.1 for MSSG (+21.3).
- On DriveLM-QA (baseline split), overall score 50.1 vs 32.4 for the DriveLM baseline.
- On refCOCO/+/g, average 87.0 across splits, the best among listed generalist MLLMs (Qwen-VL 85.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LV3D pretraining to LLaVA-1.5 (ablation) | Cube-LLM · DriveLM-Grounding | BEV AP_A: 33.2 -> 66.0 | Helped |
| Add 2D-only LV3D pretraining (ablation) | Cube-LLM · DriveLM-Grounding | BEV AP_A: 39.6 -> 50.5 | Helped |
| Add outdoor 3D data to 2D-only LV3D pretraining (ablation) | Cube-LLM · DriveLM-Grounding | BEV AP_A: 50.5 -> 66.0 | Helped |
| Scale pretraining from LV3D-small to full LV3D (ablation) | Cube-LLM · Objectron | mAP cls+loc 3D: 36.1 -> 45.4 | Helped |
| Scale pretraining from LV3D-small to full LV3D (ablation) | Cube-LLM · ArkitScenes | mAP cls+loc 3D: 28.3 -> 31.8 | Helped |
| Scale pretraining from LV3D-small to full LV3D (ablation) | Cube-LLM · SUN-RGBD | mAP cls+loc 3D: 25.5 -> 28.8 | Helped |
| Add visual chain-of-thought prompting (ablation) | Cube-LLM · Talk2Car | BEV AP_A: 43.6 -> 46.3 | Helped |
| Add CenterPoint candidate boxes as visual prompts (ablation) | Cube-LLM · Talk2Car | BEV AP_A: 46.3 -> 71.4 | Helped |
| Add CenterPoint candidate boxes as visual prompts (ablation) | Cube-LLM · Talk2Car | BEV AP_B: 30.1 -> 61.2 | Helped |
| Add CenterPoint candidate boxes as visual prompts (ablation) | Cube-LLM · Talk2Car | 3D AP_A: 34.7 -> 64.1 | Helped |
| Add CenterPoint candidate boxes as visual prompts (ablation) | Cube-LLM · Talk2Car | 3D AP_B: 18.2 -> 39.8 | Helped |
| Add refCOCO and COCO 2D data to LLaVA pretraining (ablation) | Cube-LLM · Talk2Car | BEV AP_A: 19.7 -> 27.0 | Helped |
Surprising findings
- Camera-only CUBE-LLM (no LiDAR) already beats the camera+LiDAR Talk2Car-3D baseline on BEV AP_A by 15.7 points.
- Replacing CLIP with DINOv2 causes minimal degradation on standard VLM benchmarks while improving 3D tasks.
Limitations
- Single-frame input only; no video or temporal dynamics, so moving vs stationary objects are often wrong.
- No visual token resampling, limiting scaling to higher resolutions.
- Semantic mismatch errors on similar objects (e.g., silver vs white sedan) and inaccurate depth.
Experimental setup
Datasets: LLaVA data refCOCO/+/g GRIT AS (Ferret/All-Seeing) COCO Objects365 SUN-RGBD Hypersim ArkitScenes Objectron KITTI NuScenes
Benchmarks: Talk2Car DriveLM-Grounding DriveLM-QA refCOCO/+/g VQAv2 GQA VizWiz ScienceQA-Image POPE Objectron ArkitScenes SUN-RGBD
Metrics: AP_0.5 BEV AP 3D AP accuracy ChatGPT score BLEU_1 ROUGE_L CIDEr Overall accuracy@IoU
Baselines: LLaVA-1.5 DriveLM baseline MSSG Talk2Car-3D Ferret Shikra Qwen-VL MiniGPT-v2 MDETR UniTAB
Builds on: LLaVA-1.5 Omni3D DriveLM DINOv2
Closest papers by meaning
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.92)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.91)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.91)
- 3D Question Answering via only 2D Vision-Language Models (ICML 2025 · similarity 0.90)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.