Research map / 3D vision / Depth and stereo
BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BetterDepth, a depth-conditioned latent diffusion refiner built on a pre-trained feed-forward MDE model (Depth Anything), trained with global pre-alignment and local patch masking on small synthetic data to combine zero-shot robustness with fine detail.
Problem
Zero-shot affine-invariant monocular depth estimation: feed-forward models are robust but over-smooth details, while diffusion-based MDE captures details but generalizes poorly due to scarce, less diverse synthetic depth labels.
Approach
Use a pre-trained MDE model's depth as conditioning for a Stable Diffusion-based latent U-Net refiner; align the conditioning to ground truth globally via least-squares scale/shift and mask out locally dissimilar latent patches in the diffusion loss so the refiner stays faithful to the prior while adding details.
Key results
- BetterDepth (74K) reaches average rank 1.8 on zero-shot benchmarks, with 4.2% AbsRel and 98.0% delta1 on NYUv2 and 7.5% AbsRel on KITTI, outperforming Depth Anything and Marigold.
- BetterDepth-400 (400 training samples) already compares favorably to prior art, e.g., 4.6% AbsRel on NYUv2.
- Plug-and-play refinement improves MiDaS and DPT without retraining, e.g., MiDaS KITTI AbsRel drops from 18.3 to 15.1 and DPT NYUv2 AbsRel from 9.1 to 7.9.
- On Middlebury 2014, BetterDepth achieves 50.35% edge recall vs 23.75% for Marigold and 16.12% for Depth Anything.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth conditioning to the diffusion refiner (ablation) | BetterDepth · NYUv2 | AbsRel: 6.1 -> 5.2 | Helped |
| Add global pre-alignment to depth conditioning (ablation) | BetterDepth with depth conditioning · NYUv2 | AbsRel: 5.2 -> 4.7 | Helped |
| Add local patch masking (ablation) | BetterDepth with global pre-alignment · NYUv2 | AbsRel: 4.7 -> 4.2 | Helped |
| Add local patch masking (ablation) | BetterDepth with global pre-alignment · KITTI | AbsRel: 7.9 -> 7.5 | Helped |
| Apply BetterDepth refinement to MiDaS without retraining | MiDaS · NYUv2 | AbsRel: 9.5 -> 8.4 | Helped |
| Apply BetterDepth refinement to MiDaS without retraining | MiDaS · DIODE | AbsRel: 26.6 -> 26.6 | No clear effect |
| Apply BetterDepth refinement to DPT without retraining | DPT · NYUv2 | AbsRel: 9.1 -> 7.9 | Helped |
| Use mean rather than median test-time ensembling (ablation) | BetterDepth · KITTI | AbsRel: 7.5 -> 7.4 | Helped |
| Use mean rather than median test-time ensembling (ablation) | BetterDepth · NYUv2 | δ1: 98.0 -> 98.1 | Helped |
| Train with geometric and image priors together rather than geometric prior alone (ablation) | BetterDepth · KITTI | AbsRel: 8.0 -> 7.5 | Helped |
| Reduce diffusion denoising from 50 steps to 2 steps (ablation) | BetterDepth · KITTI | δ1: 95.2 -> 92.5 | Hurt |
Surprising findings
- Naively conditioning the diffusion model on the depth prior (model #2) does not beat Depth Anything zero-shot; global pre-alignment is needed to make the conditioning effective.
- Only 400 training samples are needed to achieve competitive zero-shot performance.
Limitations
- null
Experimental setup
Datasets: Hypersim Virtual KITTI NYUv2 KITTI ETH3D ScanNet DIODE Middlebury 2014
Benchmarks: NYUv2 KITTI ETH3D ScanNet DIODE Middlebury 2014
Metrics: AbsRel delta1 DBE_comp DBE_acc edge precision edge recall
Baselines: Depth Anything Marigold GeoWizard DepthFM DiverseDepth MiDaS LeReS Omnidata HDN DPT Depth Anything V2
Builds on: Marigold Depth Anything Stable Diffusion DPT
Closest papers by meaning
- ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation (CVPR 2024 · similarity 0.88)
- SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillation (CVPR 2025 · similarity 0.88)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.88)
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.88)
- Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation (NeurIPS 2024 · similarity 0.87)
- MonSter: Marry Monodepth to Stereo Unleashes Power (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.