Research map / Vision-language and multimodal / Multimodal LLMs

Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Vid-LLM is a compact video-only 3D multimodal LLM that couples a feed-forward geometry reconstruction branch with a 3D vision-language branch via a Cross-Task Adapter with bridge tokens, trained with dual-teacher distillation and then joint optimization.

Problem

3D multimodal LLMs typically require 3D inputs (point clouds, depth, poses), limiting scalability; the paper targets 3D VL reasoning (QA, dense captioning, grounding) from monocular video alone.

Approach

A Cross-Task Adapter with learnable bridge tokens lets geometry and semantic streams share information, a metric depth model recovers real scale, and 3D patch tokens back-projected from predicted depth and poses are fed to the LLM, trained with a two-stage dual-teacher distillation then joint optimization.

Key results

• On ScanQA, Vid-LLM reaches CIDEr 101.9 among video-based models vs 94.7 for 3DRS-VGGT (3DRS^dagger), and SQA3D EM@1 57.3.
• On ScanRefer, Vid-LLM achieves Acc@0.25 63.2 and Acc@0.5 56.4, exceeding the best 3D-based 3DRS (62.9 / 56.1).
• Among joint reconstruction-reasoning models, Vid-LLM runs at 1.6 s/scene versus 2.7 s/scene for VGGT+LLaVA-3D, with ScanNet reconstruction F-score 0.582 vs 0.580 for Uni3DR2.

Surprising findings

• The concatenation baseline with metric alignment achieves a higher ScanNet reconstruction score (0.591) yet underperforms Vid-LLM on all 3D VL tasks, suggesting good reconstruction alone does not ensure reasoning quality.
• Training LLaVA-3D on VGGT-predicted geometry slightly decreased 3D VL accuracy compared with the non-fine-tuned concatenation variant.

Limitations

• Performance depends on reconstruction quality; incomplete geometry from limited viewpoints, reflective surfaces, or oblique views degrades counting and grounding.
• Grounding relies on Mask3D instance proposals, and no code, data, or weights are released yet (code promised upon acceptance).

Experimental setup

Datasets: ScanQA SQA3D Scan2Cap ScanRefer Multi3DRefer Nr3D Sr3D ScanNet

Benchmarks: ScanQA SQA3D Scan2Cap ScanRefer Multi3DRefer Nr3D Sr3D ScanNet Co3Dv2 RealEstate10K NYU Depth v2

Metrics: CIDEr BLEU-4 METEOR ROUGE EM@1 Acc@0.25 Acc@0.5 grounding accuracy F-score AbsRel

Baselines: 3DRS Video-3D LLM LLaVA-3D ChatScene LEO VGLLM Uni3DR2 3DRS (VGGT input) VILA-40B LLaVA-Video-7B

Builds on: VGGT LLaVA-3D DINOv2 Uni3DR2 VGLLM

Closest papers by meaning

• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.93)
• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.92)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.92)
• MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.90)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.90)