Research map / Vision-language and multimodal / Multimodal LLMs

MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Probes the 3D awareness of MLLMs via multi-view voxel correspondence and distills features from 3D foundation models (VGGT, FLARE) into MLLM visual tokens (3DRS) to improve 3D scene understanding.

Problem

MLLMs for 3D scene understanding are pretrained only on 2D data, so their visual representations may lack 3D awareness; it is unclear how 3D feature quality relates to downstream performance and how to improve it.

Approach

Measure 3D awareness via cross-view feature similarity within voxels, observe that higher correspondence scores track better downstream results, and align MLLM per-pixel visual features to features from pretrained 3D foundation models (VGGT/FLARE) with an offline-precomputed cosine distillation loss added to the cross-entropy objective.

Key results

• 3DRS with LLaVA-NeXT-Video reaches 62.9 Acc@0.25 on ScanRefer (vs 58.1 baseline) and 104.8 CIDEr on ScanQA (vs 102.1), surpassing Video-3D LLM.
• Gains are consistent across three MLLMs: LLaVA-OneVision w/ 3DRS improves ScanRefer Acc@0.25 from 57.3 to 61.8; Qwen2-VL from 57.0 to 60.1.
• 3D foundation model teachers (VGGT, FLARE) outperform 2D teachers (DINOv2, MAE, SigLIP2); supervising only the last layer works best.

Measured effects reported in the paper

Surprising findings

• Supervising additional intermediate layers degrades performance, which the authors attribute to over-constraining geometric features and weakening semantic representations.
• The choice of distillation loss (Euclidean, cosine, or combined) has limited influence on results.

Limitations

• Performance is upper-bounded by the quality of the teacher 3D foundation model; teacher errors can propagate to the student.
• Finetuning dataset is small relative to MLLM pretraining data; 3D awareness is not integrated into pretraining.
• For Qwen2-VL only the projector and LLM were finetuned due to GPU memory constraints.

Experimental setup

Datasets: ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D VSI-Bench pretraining data from VG-LLM ScanNet

Benchmarks: ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D VSI-Bench

Metrics: Acc@0.25 Acc@0.5 F1@0.25 F1@0.5 CIDEr BLEU-4 EM

Baselines: Video-3D LLM LLaVA-3D Chat-Scene Inst3D-LLM Grounded 3D-LLM PQ3D 3D-VisTA GPT-4o Gemini-1.5 Pro LLaVA-Video-72B LLaVA-OneVision-72B

Builds on: Video-3D LLM VGGT FLARE LLaVA-NeXT-Video LLaVA-OneVision

Closest papers by meaning

• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.93)
• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.92)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.92)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.92)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.91)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.91)