Research map / Vision-language and multimodal / Multimodal LLMs

VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VLM-3R, a video LMM that fuses implicit metric-scale 3D tokens from the CUT3R reconstruction model (spatial and view tokens) into LLaVA-NeXT-Video via cross-attention, trained on 200K spatial QA pairs, and a new temporal benchmark VSTI-Bench.

Problem

Video LMMs lack deep visual-spatial and temporal 3D understanding; existing geometry-augmented methods need depth sensors or pre-built 3D maps, or use normalized-scale geometry that hinders metric reasoning.

Approach

Use a feed-forward metric-scale 3D reconstruction encoder (CUT3R) to produce implicit spatial tokens and camera view tokens from monocular video, fuse them with CLIP/SigLIP visual tokens via a residual cross-attention block (Spatial-Visual-View Fusion), and instruction-tune with auto-generated 3D QA data.

Key results

• VLM-3R (7B) achieves 60.9 average on VSI-Bench, ranking first among open-source models, with Relative Direction 80.5 and Absolute Distance 49.4.
• On VSTI-Bench, VLM-3R averages 58.8, above Gemini-2.5 Pro (42.4) and GPT-4o (38.2).
• Ablation: removing spatial tokens drops VSI-Bench average from 60.90 to 59.46, and explicit point-cloud fusion gives 57.87.

Surprising findings

• Mixing 30k general LLaVA-Video samples only restores Video-MME from 59.9 to 62.1, still below the base LLaVA-NeXT-Video's 62.7.
• On OpenEQA, VLM-3R's overall score (61.7) is slightly below its base model LLaVA-NeXT-Video (62.4) despite better spatial-question accuracy.

Limitations

• Dataset and benchmark focus on static indoor scenes, so dynamic environments are not covered.
• Performance is bounded by the accuracy of monocular end-to-end 3D reconstruction.
• Object size reasoning slightly drops relative to the baseline, suggesting limits in monocular reconstruction quality.

Experimental setup

Datasets: ScanNet ScanNet++ ARKitScenes VSI-Bench LLaVA-Video ScanQA SQA3D

Benchmarks: VSI-Bench VSTI-Bench OST-Bench Video-MME VQAv2 ScanQA SQA3D OpenEQA

Metrics: accuracy mean relative accuracy CIDEr BLEU-4 EM

Baselines: Gemini-2.5 Pro GPT-4o LLaVA-NeXT-Video-7B LLaVA-OneVision-7B Qwen2.5-VL-7B Spatial-MLLM VG-LLM Video-3D LLM LLaVA-3D

Builds on: LLaVA-NeXT-Video CUT3R VSI-Bench LoRA

Closest papers by meaning

• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.92)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.92)
• HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.92)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.92)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.92)