Research map / 3D vision / 3D generation
Probing the 3D Awareness of Visual Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes frozen visual foundation models for 3D awareness via single-image depth/surface-normal probes and zero-shot multiview correspondence, finding self-supervised DINOv2 best and all models struggling with large viewpoint changes.
Problem
Whether visual foundation models trained on 2D images represent 3D structure, evaluated as single-view surface geometry and cross-view consistency.
Approach
Evaluate 3D awareness of frozen representations using trainable dense multiscale probes for depth and surface normals and training-free correspondence matching across views, rather than fine-tuning, across models with different supervision types.
Key results
- DINOv2 performs best on depth and surface normal probes, with StableDiffusion a close second, while CLIP performs very poorly.
- Single-view task performance is strongly correlated across tasks (correlation coefficients above 0.82), while multiview correspondence correlations are much lower.
- Models match accurately for small viewpoint changes but performance drops sharply for large viewpoint changes, with StableDiffusion and SAM dropping most sharply.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use discriminative self-supervised pretraining rather than language supervision | Visual foundation models · Single-image depth and surface normal estimation, scenes and objects | Probe performance | Helped |
| Use DINOv2 representations rather than other evaluated model representations | Frozen visual foundation model representations · Single-image depth and surface normal estimation | Probe performance | Helped |
| Use Stable Diffusion representations rather than most other evaluated representations | Frozen visual foundation model representations · Single-image depth and surface normal estimation | Probe performance | Helped |
| Use vision-language pretraining rather than discriminative self-supervised pretraining | Visual foundation models · Single-image depth and surface normal estimation | Probe performance | Hurt |
| Use dense supervision, including depth supervision, rather than self-supervised or text-conditioned generation training | Visual foundation models · Single-image depth and surface normal estimation | Probe performance | Hurt |
| Increase viewpoint variation between views | Visual foundation models · NAVI and ScanNet geometric correspondence | Correspondence recall | Hurt |
| Use Stable Diffusion or SAM features instead of features with more gradual viewpoint-related performance drops | Geometric correspondence estimation · Object correspondence across viewpoint changes | Correspondence recall | Hurt |
| Increase viewpoint variation | Stable Diffusion features · SPair-71k chairs semantic correspondence | Keypoint correspondence accuracy | Hurt |
| Use semantic correspondence performance as an indicator of 3D consistency | Visual foundation model evaluation · Semantic correspondence versus geometric correspondence | Correlation with multiview performance | Hurt |
Surprising findings
- Discriminative self-supervised models outperform generative and depth-supervised models on single-view 3D probes.
- Semantic correspondence performance correlates more with single-view tasks than with geometric multiview correspondence, suggesting semantic correspondence is not a good measure of 3D consistency.
Limitations
- Pretrained checkpoints differ in training data and compute scale, so comparisons are not fully controlled.
- Only minimal probing approaches were used; other probing techniques were not explored.
- Only two basic aspects of 3D understanding (single-view geometry and correspondence) were evaluated.
Experimental setup
Datasets: NYUv2 NAVI ScanNet SPair-71k ImageNet-22k
Benchmarks: NYUv2 NAVI ScanNet SPair-71k
Metrics: RMSE recall at depth thresholds angular error correspondence recall Pearson correlation
Baselines: DeiT III MAE iBOT DINO DINOv2 CLIP SigLIP StableDiffusion SAM MiDaS
Builds on: DPT AdaBins DINOv2 keypoint-free correspondence estimation (LoFTR)
Closest papers by meaning
- Feat2GS: Probing Visual Foundation Models with Gaussian Splatting (CVPR 2025 · similarity 0.91)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.90)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.88)
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.88)
- Can 4D Foundation Models Remember? (NeurIPS 2026 · similarity 0.87)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.