Research map / 3D vision / Depth and stereo
Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prompts a pretrained DPT-based depth foundation model (Depth Anything v2) with low-cost iPhone LiDAR via a multi-scale prompt fusion block, trained with simulated LiDAR, Zip-NeRF pseudo GT, and an edge-aware depth loss, achieving accurate metric depth up to 4K.
Problem
Metric depth estimation from monocular images remains inaccurate and scale-ambiguous even for depth foundation models, and existing LiDAR-guided methods are limited by scarce training data with both LiDAR input and precise GT depth.
Approach
Treat metric depth estimation as prompting a depth foundation model with a metric prompt (low-cost LiDAR), fused at multiple DPT decoder scales via zero-initialized prompt fusion blocks, and train with a scalable data pipeline (LiDAR simulation on synthetic data, Zip-NeRF pseudo GT plus an edge-aware gradient loss on real data).
Key results
- On ARKitScenes at 1440x1920, the model reports L1 0.0138 and RMSE 0.0316 with LiDAR prompting, versus 0.0402 L1 for Metric3D v2 with LiDAR alignment (non-zero-shot comparison as stated in the table).
- On ScanNet++ the model reaches AbsRel 0.0175, delta0.5 0.9781 and TSDF F-score 0.7619, outperforming the best listed post-aligned baseline (MSPF, F-score 0.7133).
- The zero-shot synthetic-trained model (Ourssyn) beats non-zero-shot baselines on ScanNet++, and the model runs at 20.4 FPS at 768x1024 on an A100 with 2.0 FPS at 2160x3840.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add low-cost LiDAR prompting (ablation) | Prompt Depth Anything · ARKitScenes | L1: 0.0605 -> 0.0138 | Helped |
| Add low-cost LiDAR prompting (ablation) | Prompt Depth Anything · ScanNet++ | L1: 0.0505 -> 0.0250 | Helped |
| Initialize from a pretrained depth foundation model (ablation) | Prompt Depth Anything · ARKitScenes | L1: 0.0194 -> 0.0132 | Helped |
| Replace prompt fusion with AdaLN prompting (ablation) | Prompt Depth Anything · ARKitScenes | L1: 0.0132 -> 0.0197 | Hurt |
| Replace prompt fusion with cross-attention prompting (ablation) | Prompt Depth Anything · ScanNet++ | L1: 0.0115 -> 0.0443 | Hurt |
| Replace prompt fusion with ControlNet prompting (ablation) | Prompt Depth Anything · ScanNet++ | L1: 0.0115 -> 0.0206 | Hurt |
| Add ARKitScenes training data to synthetic training data (ablation) | Prompt Depth Anything · ARKitScenes | L1: 0.0163 -> 0.0134 | Helped |
| Add ScanNet++ annotated GT training data (ablation) | Prompt Depth Anything · ScanNet++ | L1: 0.0121 -> 0.0114 | Helped |
| Use ScanNet++ pseudo GT directly for supervision (ablation) | Prompt Depth Anything · ScanNet++ | F-score: 0.7341 -> 0.6505 | Hurt |
| Add edge-aware depth loss using pseudo GT gradients and FARO annotated depth (ablation) | Prompt Depth Anything · ScanNet++ | F-score: 0.6505 -> 0.7619 | Helped |
Surprising findings
- Naive downsampling of depth to simulate LiDAR makes the model learn depth super-resolution rather than correcting LiDAR noise; sparse anchor interpolation was needed.
- Direct supervision with pseudo GT depth from Zip-NeRF decreased performance, and only the edge-aware gradient loss made it helpful.
Limitations
- iPhone LiDAR cannot handle long-range depth because far-object returns are very noisy.
- Temporal flickering of LiDAR input leads to flickering depth predictions.
- Trained mainly on indoor data; the outdoor case relies on replacing the prompt with vehicle LiDAR.
Experimental setup
Datasets: HyperSim ARKitScenes ScanNet++
Benchmarks: ARKitScenes ScanNet++
Metrics: L1 RMSE AbsRel delta 0.5 TSDF Acc TSDF Comp Precision Recall F-score
Baselines: Metric3D v2 ZoeDepth DepthPro Depth Anything v2 Depth Anything v1 Marigold Lotus BPNet Depth Prompting MSPF
Builds on: Depth Anything v2 DPT Zip-NeRF ControlNet
Closest papers by meaning
- Depth Prompting for Sensor-Agnostic Depth Estimation (CVPR 2024 · similarity 0.89)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.87)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.87)
- Metric Depth Estimation from Arbitrarily Degraded Low-Resolution Depth Prompts (NeurIPS 2026 · similarity 0.86)
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.86)
- Flexible Depth Completion for Sparse and Varying Point Densities (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.