Research map / 3D vision / 3D generation
A General Protocol to Probe Large Vision Models for 3D Physical Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes frozen large vision models (Stable Diffusion, DINOv1/v2, OpenCLIP, VQGAN) with linear SVMs on region-pair features to test how well they encode 3D physical properties (geometry, material, support, shadow, occlusion, depth) on real images.
Problem
Whether large-scale pre-trained vision models implicitly encode 3D physical properties of scenes (geometry, material, support relations, lighting/shadows, occlusion, depth) learned only from 2D images.
Approach
A general, lightweight linear-probing protocol: for each property, build region-pair yes/no questions from existing real-image annotated datasets, pool features per region, and grid-search layer, diffusion time step and SVM C on validation data to measure how well each frozen model encodes the property.
Key results
- Stable Diffusion features (time step 60-360, decoder layer D3) reach test AUC 96.3 on same plane, 99.6 on depth and 95.4 on shadow, while being weaker on material (83.6) and occlusion (84.8).
- DINOv2 matches or beats Stable Diffusion on support relation (92.8 vs 92.1) and shadow (95.8 vs 95.4), and outperforms OpenCLIP, DINOv1 and VQGAN on all seven properties in the comparison table except where Stable Diffusion is best.
- Injecting probed Stable Diffusion features into iDisc for surface normal estimation, trained only 1000 of 45000 iterations, gives mean angular error 13.8 vs 14.6 for iDisc.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Stable Diffusion features instead of DINOv2 features | Linear probing protocol · ScanNetv2 — Same Plane | ROC AUC (%): 94.5 -> 96.3 | Helped |
| Use Stable Diffusion features instead of DINOv2 features | Linear probing protocol · ScanNetv2 — Perpendicular Plane | ROC AUC (%): 82.1 -> 86.0 | Helped |
| Use Stable Diffusion features instead of DINOv2 features | Linear probing protocol · DMS — Material | ROC AUC (%): 83.5 -> 83.6 | Helped |
| Use DINOv2 features instead of Stable Diffusion features | Linear probing protocol · NYUv2 — Support Relation | ROC AUC (%): 92.1 -> 92.8 | Helped |
| Use DINOv2 features instead of Stable Diffusion features | Linear probing protocol · SOBA — Shadow | ROC AUC (%): 95.4 -> 95.8 | Helped |
| Use Stable Diffusion features instead of DINOv2 features | Linear probing protocol · Separated COCO — Occlusion | ROC AUC (%): 84.4 -> 84.8 | Helped |
| Use DINOv2 features instead of Stable Diffusion features | Linear probing protocol · NYUv2 — Depth | ROC AUC (%): 99.6 -> 99.7 | Helped |
Surprising findings
- Stable Diffusion's best-performing U-Net layer is always the decoder layer D3 and the best time steps are usually before 400, so the most informative features come from a middle decoder layer at low noise.
Limitations
- Only linear probes are used, so failures on material and occlusion may reflect probe capacity rather than absent information.
- Evaluation is limited to binary region-pair questions on a few datasets of real images with modest image counts (400 train images per property).
Experimental setup
Datasets: ScanNetv2 DMS NYUv2 SOBA Separated COCO
Benchmarks: ScanNetv2 DMS NYUv2 SOBA Separated COCO
Metrics: ROC AUC
Baselines: OpenCLIP DINOv1 DINOv2 VQGAN iDisc
Builds on: DIFT DINOv2 DINO CLIP VQGAN
Closest papers by meaning
- Large Spatial Model: End-to-end Unposed Images to Semantic 3D (NeurIPS 2024 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (CVPR 2024 · similarity 0.86)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.86)
- One at a Time: Progressive Multi-Step Volumetric Probability Learning for Reliable 3D Scene Perception (AAAI 2024 · similarity 0.86)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.