Research map / Vision-language and multimodal / Human and embodied modeling
Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes seven frozen visual foundation encoders (image, video, diffusion, 3D) on four 3D scene understanding tasks (VQA, visual grounding, segmentation, registration) using a unified multi-view 3D projection framework.
Problem
Unclear which visual foundation model encodings are best for complex 3D scene understanding, compared with their image-based counterparts.
Approach
A unified probing framework that projects features from image, video, diffusion, and 3D encoders onto 3D points via multi-view projection and mean pooling, then trains only shallow heads on frozen encoders, enabling a systematic cross-modality comparison across scene-level tasks.
Key results
- DINOv2 achieves the best ScanQA CIDEr (73.2) and SQA3D CIDEr (129.1) among evaluated encoders, with V-JEPA and SVD close behind.
- StableVideoDiffusion reaches 54.7 overall and 46.5 multiple-category accuracy on ScanRefer, the top result among evaluated encoders.
- DINOv2 achieves the best semantic segmentation mIoU of 62.8 on ScanNet, while StableVideoDiffusion achieves 0.06 RTE and 96.8% RR@0.05m on partial scene registration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DINOv2 features instead of the other evaluated encoders | Lexicon3D vision-language reasoning · ScanQA | CIDEr: 71.4 -> 73.2 | Helped |
| use DINOv2 features instead of the other evaluated encoders | Lexicon3D vision-language reasoning · SQA3D | CIDEr: 127.7 -> 129.1 | Helped |
| use video encoders, especially StableVideoDiffusion, rather than image or 3D encoders | 3D object grounding · ScanRefer | Overall accuracy: 52.0 -> 54.7 | Helped |
| use video encoders to distinguish objects among multiple objects of the same semantic category | 3D object grounding · ScanRefer | Multiple accuracy: 43.4 -> 46.5 | Helped |
| use image encoders rather than video or 3D encoders | 3D semantic segmentation · ScanNet | mIoU: 30.4 -> 62.8 | Helped |
| use generation-pretrained diffusion features rather than discriminative encoder features | partial scene registration · ScanNet | RTE: 0.14 -> 0.06 | Helped |
| use keyframe sampling rather than clip sampling for temporal downsampling (ablation) | V-JEPA segmentation · ScanNet | performance | Helped |
| concatenate features from multiple encoders (ablation) | 3D semantic segmentation · ScanNet | segmentation performance | Helped |
| combine StableDiffusion and Swin3D features rather than using either alone (ablation) | 3D semantic segmentation · ScanNet | segmentation performance | Helped |
| use StableDiffusion noise levels below 100 steps rather than 200 steps (ablation) | StableDiffusion feature extraction · ScanQA | CIDEr: 66.6 -> 68.5 | Helped |
| extract StableDiffusion features from decoder layer 1 rather than layer 0 (ablation) | StableDiffusion feature extraction · ScanQA | CIDEr: 65.9 -> 68.2 | Helped |
Surprising findings
- Language-pretrained encoders (LSeg, CLIP) do not perform well on LLM-based VQA despite their language alignment.
- CLIP achieves only 3.4 mIoU on ScanNet segmentation, and V-JEPA only 8.1 mIoU, far below DINOv2.
Limitations
- Model capacities and training data are not strictly comparable across the public checkpoints evaluated.
- Evaluation is limited to indoor scenes; outdoor scenarios are not studied.
- Only frozen-encoder linear or shallow probing is used, without finetuning the encoders.
Experimental setup
Datasets: ScanQA SQA3D ScanRefer ScanNet 3D-Language Structured3D
Benchmarks: ScanQA SQA3D ScanRefer ScanNet semantic segmentation partial scene registration on ScanNet
Metrics: BLEU-1 BLEU-4 METEOR ROUGE CIDEr EM-1 accuracy (unique/multiple/overall) Acc mAcc mIoU
Baselines: 3D-LLM M3DRef GrowSP Uni3D SAM
Builds on: DINOv2 CLIP V-JEPA Swin3D 3D-LLM REGTR
Closest papers by meaning
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.91)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.90)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding (NeurIPS 2026 · similarity 0.90)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.90)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.