Research map / 3D vision / 3D generation
Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VG LLM, which adds latent 3D geometry features from a frozen-pretrained VGGT encoder (fused by addition at patch level) to Qwen2.5-VL so MLLMs can reason about 3D scenes from RGB video alone.
Problem
MLLMs struggle with 3D geometry and spatial reasoning from video frames, and existing 3D-aware video LLMs depend on explicit 3D inputs (point clouds, BEV maps, depth) that are hard to obtain.
Approach
Inject latent 3D visual geometry features from a pretrained multi-view 3D reconstruction encoder (VGGT encoder plus fusion decoder) into MLLM visual tokens by patch-level addition after spatial merging, so the model gets 3D priors from RGB video without any explicit 3D input.
Key results
- On VSI-Bench, VG LLM-4B averages 47.3% (above Gemini-1.5-Pro's 45.4%) and VG LLM-8B reaches 50.7%, a new state of the art in the paper's comparison.
- On 3D video object detection (6-frame), the 4B model raises average F1 from 25.7% to 36.4% over Qwen2.5-VL-3B; the 8B model improves F1 by 11.2 points over Qwen2.5-VL-7B.
- On CV-Bench, VG LLM-8B reaches 81.7% average, with 91.1% on the 3D subset, the highest among compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VGGT visual geometry features | Qwen2.5-VL on 3D video object detection · 3D video detection, 4-frame | F1@0.25: 30.0 -> 38.2; 32.5 -> 41.2 | Helped |
| Add VGGT visual geometry features | Qwen2.5-VL on 3D video object detection · 3D video detection, 6-frame | F1@0.25: 25.7 -> 36.4; 29.6 -> 40.8 | Helped |
| Add VGGT visual geometry features (ablation) | Qwen2.5-VL on VSI-Bench · VSI-Bench | average accuracy: 49.8 -> 50.7; 56.6 -> 58.2; 59.8 -> 62.2 | Helped |
| Use one or three cross-attention layers instead of direct addition (ablation) | VG LLM feature fusion · ScanRefer | Acc@0.25: 36.4 -> 33.7; 36.4 -> 34.4 | Hurt |
| Concatenate 2D and 3D tokens and project with an MLP instead of direct addition (ablation) | VG LLM feature fusion · ScanRefer | Acc@0.25: 36.4 -> 27.7 | Hurt |
| Add predicted camera information instead of latent geometry features (ablation) | Qwen2.5-VL-3B · ScanRefer | Acc@0.25: 31.9 -> 32.1 | No clear effect |
| Add predicted depth information instead of latent geometry features (ablation) | Qwen2.5-VL-3B · ScanRefer | Acc@0.25: 31.9 -> 32.3 | No clear effect |
| Add predicted point information instead of latent geometry features (ablation) | Qwen2.5-VL-3B · Scan2Cap | CIDEr@0.5: 58.0 -> 67.7 | Helped |
| Add predicted point information instead of latent geometry features (ablation) | Qwen2.5-VL-3B · 3D video detection | F1@0.25: 30.0 -> 30.4 | Helped |
| Fine-tune on S1 mixed spatial dataset (ablation) | Qwen2.5-VL-7B · VSI-Bench | average accuracy: 29.5 -> 49.8 | Helped |
| Add visual geometry features | VG LLM on Scan2Cap · Scan2Cap | METEOR@0.5: 30.2 -> 28.9 | Hurt |
| Add spatial reasoning fine-tuning and geometry enhancement | VG LLM-4B versus Qwen2.5-VL-3B · generic multimodal benchmarks | accuracy: BLINK: 46.6 -> 48.9; TempCompass$_{\text{MC}}$: 62.2 -> 63.9; Video-MME$_{\text{w/o\ sub.}}$: 60.1 -> 57.4 | Mixed |
Surprising findings
- Adding VGGT's predicted point maps helps 3D dense captioning, but predicted camera or depth information gives no clear benefit.
- Fine-tuning on 3D tasks did not clearly improve Video-MME with subtitles for the 4B model, and the 4B model underperforms its baseline on Video-MME without subtitles (57.4 vs 60.1).
Limitations
- Capacity limited by 3B/7B backbones; scaling not explored.
- Only supervised fine-tuning, no reinforcement learning.
- VGGT is trained on ScanNet-like datasets, so evaluation on ScanNet-based benchmarks may overestimate its performance.
Experimental setup
Datasets: SPAR-7M LLaVA-Video-178K (LLaVA-Hound split) ScanRefer Scan2Cap EmbodiedScan ScanNet VLM-3R
Benchmarks: ScanRefer Scan2Cap 3D video object detection (EmbodiedScan-based) VSI-Bench CV-Bench Video-MME BLINK TempCompass NextQA
Metrics: Acc@0.25 Acc@0.5 CIDEr@0.5 BLEU-4@0.5 precision recall F1 accuracy mean relative accuracy
Baselines: SPAR-8B Video-3D LLM GPT4Scene SAT-LLaVA-Video-7B Gemini-1.5-Pro LLaVA-3D Qwen2.5-VL-3B Qwen2.5-VL-7B
Builds on: Qwen2.5-VL VGGT Video-3D LLM SPAR LLaVA-Video-178K
Closest papers by meaning
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.94)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.93)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.93)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.92)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.