Research map / 3D vision / 3D generation
Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video-3D LLM treats 3D indoor scans as videos, injects sinusoidal 3D position encodings from back-projected global coordinates into video frame features, and uses greedy maximum-coverage frame sampling to train a multi-task 3D scene understanding model built on LLaVA-Video 7B.
Problem
MLLMs trained mostly on 2D image data struggle with 3D scene understanding tasks such as visual grounding, dense captioning, and 3D question answering, and existing 3D LLMs rely on point cloud or voxel representations that disconnect from pretrained 2D/video priors.
Approach
Represent a 3D scene as a video of RGB frames with per-patch global 3D coordinates encoded by sinusoidal position encoding and added to visual embeddings, plus a greedy maximum-coverage frame selection that maximizes voxel coverage under a frame budget.
Key results
- Video-3D LLM (Uniform, 32 frames) reaches 58.1% Acc@0.25 on ScanRefer and 83.8 C@0.5 on Scan2Cap, outperforming LLaVA-3D (54.1 and 79.2).
- Video-3D LLM achieves 58.6% EM on SQA3D and 30.1% EM on ScanQA, versus 55.6 and 27.0 for LLaVA-3D, using 26% of LLaVA-3D's 3D training data.
- Maximum coverage sampling with 8 frames improves ScanRefer Acc@0.25 by 4.54 points over uniform sampling; the adaptive MC* variant averages about 18 frames and matches 32-frame uniform performance at 527 ms inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use maximum coverage rather than uniform sampling with 8 frames (ablation) | Video-3D LLM · ScanRefer | Acc@0.25: 48.93 -> 53.47 | Helped |
| Use maximum coverage rather than uniform sampling with 8 frames (ablation) | Video-3D LLM · Scan2Cap | C@0.5: 68.82 -> 73.08 | Helped |
| Increase frame count from 8 to 32 under uniform sampling (ablation) | Video-3D LLM · ScanRefer | Acc@0.25: 48.93 -> 58.11 | Helped |
| Add 3D position encoding (ablation) | Video-3D LLM · Scan2Cap | C@0.5: 31.03 -> 83.76 | Helped |
| Replace sinusoidal position encoding with MLP encoding (ablation) | Video-3D LLM · ScanRefer | Acc@0.25: 58.11 -> 59.63 | Helped |
| Aggregate patch coordinates by average rather than center (ablation) | Video-3D LLM · Scan2Cap | C@0.5: 80.88 -> 83.76 | Helped |
| Use minimum and maximum patch coordinates rather than center coordinates (ablation) | Video-3D LLM · Scan2Cap | C@0.5: 80.88 -> 82.75 | Helped |
| Reduce grounding patch size from 27 to 14 (ablation) | Video-3D LLM on ScanRefer · ScanRefer | Acc@0.5: 48.93 -> 50.08 | Helped |
| Use InfoNCE rather than BCE loss for grounding (ablation) | Video-3D LLM on ScanRefer · ScanRefer | Acc@0.25: 51.63 -> 56.44 | Helped |
| Represent 3D scenes as videos rather than voxels (ablation) | Video-3D LLM · Scan2Cap | C@0.5: 54.9 -> 83.8 | Helped |
| Represent 3D scenes as videos rather than voxels (ablation) | Video-3D LLM · ScanQA | EM: 28.1 -> 30.1 | Helped |
| Use maximum coverage sampling with adaptive frame count instead of 32-frame uniform sampling (ablation) | Video-3D LLM · ScanQA | EM: 30.09 -> 30.35 | Mixed |
Surprising findings
- Zero-shot LLaVA-Video-7B already scores 48.5 EM on SQA3D, suggesting current 3D QA benchmarks may not sufficiently test spatial reasoning.
Limitations
- Visual grounding and dense captioning rely on Mask3D-generated object proposals at inference, and the model classifies among proposals rather than directly regressing boxes.
- Compared with the voxel baseline, gains on ScanQA are modest, and the approach depends on depth and camera pose availability for back-projection.
Experimental setup
Datasets: ScanNet ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D
Benchmarks: ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D
Metrics: Acc@0.25 Acc@0.5 F1@0.25 F1@0.5 CIDEr@0.5IoU BLEU-4@0.5IoU CIDEr EM
Baselines: ChatScene LLaVA-3D LEO Chat-3D v2 Grounded 3D-LLM PQ3D 3D-VisTA LLaVA-Video-7B Oryx-34B SceneLLM LL3DA 3D-LLM
Builds on: LLaVA-Video Mask3D InfoNCE LLaVA-3D
Closest papers by meaning
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.94)
- MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.92)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.92)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.91)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.91)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.