Research map / 3D vision / Depth and stereo
Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a perspective-only metric depth model (iDisc) on ERP-converted patches with pitch-aware augmentation, FoV alignment and multi-resolution training, achieving zero-shot metric depth on 360° and fisheye datasets.
Problem
Zero-shot metric depth estimation degrades sharply on large field-of-view cameras (fisheye, 360°) when models are trained on perspective images, and existing metric depth foundation models fail to generalize to them.
Approach
Convert perspective training images into equirectangular (ERP) patches via grid sampling with gnomonic geometry, using pitch-angle-controlled patch latitude to simulate high-distortion large-FoV regions, plus FoV alignment and multi-resolution augmentation, so a perspective-trained model generalizes to any camera.
Key results
- On Matterport3D, DAC reaches δ1 0.7727 vs 0.5287 for iDisc and 0.4381 for Metric3Dv2 (Mix 16M), a large zero-shot gain.
- On ScanNet++ fisheye, DAC achieves δ1 0.8517 vs 0.6489 for Metric3Dv2 (Indoor 670K), and the paper reports up to ~50% δ1 improvement over the next-best method.
- On outdoor KITTI360, DAC gives only marginal gains over iDisc (δ1 0.7858 vs 0.7833), with the authors attributing this to limited pitch variance in outdoor training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pitch-aware ERP conversion (ablation) | DAC trained on HM3D · Pano3D-GV2 | δ1: 0.4911 -> 0.7251 | Helped |
| add pitch-aware ERP conversion (ablation) | DAC trained on HM3D · ScanNet++ | δ1: 0.4711 -> 0.6539 | Helped |
| add 10° pitch augmentation (ablation) | DAC trained on HM3D · Pano3D-GV2 | δ1: 0.6912 -> 0.7251 | Helped |
| add 10° pitch augmentation (ablation) | DAC trained on HM3D · ScanNet++ | δ1: 0.6539 -> 0.6741 | Hurt |
| add FoV alignment (ablation) | DAC trained on HM3D · Pano3D-GV2 | δ1: 0.4075 -> 0.7251 | Helped |
| add FoV alignment (ablation) | DAC trained on HM3D · ScanNet++ | δ1: 0.5428 -> 0.6539 | Helped |
| add multi-resolution training (ablation) | DAC trained on HM3D · Pano3D-GV2 | δ1: 0.5128 -> 0.7251 | Helped |
| add multi-resolution training (ablation) | DAC trained on HM3D · ScanNet++ | δ1: 0.5504 -> 0.6539 | Helped |
Surprising findings
- Removing pitch-aware ERP conversion sharply drops DAC performance on Pano3D-GV2 (δ1 0.7251 to 0.4911), showing simulated pitch-based distortion is key to generalization.
- UniDepth, despite network-based spherical conversion, struggles on large-FoV cameras, while the geometry-based pipeline generalizes better.
Limitations
- Outdoor gains over iDisc are marginal, limited by low camera pitch variance in outdoor training data.
- Fisheye inputs must be converted to ERP and a virtual focal length is used for evaluation, which is a heuristic for metric scaling.
- Evaluation on the main large-FoV benchmarks is trained separately per domain (indoor vs outdoor) rather than jointly.
Experimental setup
Datasets: HM3D-tiny Taskonomy-tiny Hypersim DDAD LYFT Matterport3D Pano3D-GV2 ScanNet++ KITTI360 NYUv2 KITTI
Benchmarks: Matterport3D Pano3D-GV2 ScanNet++ KITTI360
Metrics: δ1 δ2 δ3 Abs Rel RMSE log10
Baselines: Metric3Dv2 UniDepth iDisc
Builds on: iDisc Metric3D Gnomonic projection SIlog loss
Closest papers by meaning
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.89)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.87)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.87)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.87)
- Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues (NeurIPS 2025 · similarity 0.86)
- Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.