Research map / 3D vision / Depth and stereo
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2 on synthetic RGB-D data (Hypersim and Virtual KITTI) to produce a zero-shot affine-invariant monocular depth estimator, with test-time ensembling.
Problem
Monocular depth estimators generalize poorly to unfamiliar scenes and domains; the question is whether priors in pretrained generative image diffusion models can yield more generalizable affine-invariant depth estimation.
Approach
Repurpose a pretrained latent diffusion image generator (Stable Diffusion v2) into an image-conditioned depth estimator by fine-tuning only the denoising U-Net on synthetic depth with image latents concatenated to the depth latent, plus annealed multi-resolution noise and a scale-shift test-time ensembling scheme.
Key results
- Marigold (w/ ensemble) reaches AbsRel 5.5 on NYUv2 and 9.9 on KITTI, and has the best average rank (1.4) across five zero-shot benchmarks, versus 3.2 for the next best (HDN).
- Trained only on synthetic data (74K samples), Marigold outperforms prior affine-invariant estimators in most cases, with over 20% gains in specific cases.
- Fine-tuning takes about 2.5 days on a single RTX 4090; ensembling 10 predictions reduces NYUv2 AbsRel by about 8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-resolution rather than Gaussian training noise (ablation) | Marigold · NYUv2 ablation split | AbsRel (%): 7.7 -> 5.8 | Helped |
| use multi-resolution rather than Gaussian training noise (ablation) | Marigold · KITTI ablation split | AbsRel (%): 14.2 -> 12.1 | Helped |
| anneal multi-resolution training noise (ablation) | Marigold · NYUv2 ablation split | AbsRel (%): 5.8 -> 5.6 | Helped |
| anneal multi-resolution training noise (ablation) | Marigold · KITTI ablation split | AbsRel (%): 12.1 -> 11.3 | Helped |
| add Hypersim training data to Virtual KITTI (ablation) | Marigold · NYUv2 ablation split | AbsRel (%): 13.9 -> 5.6 | Helped |
| add Virtual KITTI training data to Hypersim (ablation) | Marigold · KITTI ablation split | AbsRel (%): 13.7 -> 11.3 | Helped |
| increase test-time ensemble size from 1 to 10 predictions (ablation) | Marigold · NYUv2 | AbsRel (%): ≈8% | Helped |
| increase test-time ensemble size from 1 to 20 predictions (ablation) | Marigold · NYUv2 | AbsRel (%): ≈9.5% | Helped |
| increase the number of inference denoising steps (ablation) | Marigold · NYUv2 and KITTI | depth prediction performance | Helped |
| train using Hypersim only rather than Hypersim plus Virtual KITTI (ablation) | Marigold · KITTI ablation split | AbsRel (%): 11.3 -> 13.7 | Hurt |
Surprising findings
- Only 10 denoising steps are needed for saturation, fewer than the 50 usually needed for diffusion image generators.
- Initializing inference with standard Gaussian noise works better than the multi-resolution noise used in training.
Limitations
- Inference is slow due to iterative denoising and ensembling of multiple runs.
- Stochastic generative nature can yield inconsistent predictions for similar inputs.
- Weaker handling of distant scene parts.
Experimental setup
Datasets: Hypersim Virtual KITTI NYUv2 KITTI ETH3D ScanNet DIODE
Benchmarks: NYUv2 KITTI ETH3D ScanNet DIODE
Metrics: AbsRel delta1 accuracy
Baselines: DiverseDepth MiDaS LeReS Omnidata HDN DPT
Builds on: Stable Diffusion DDPM DDIM multi-resolution noise
Closest papers by meaning
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.90)
- Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues (NeurIPS 2025 · similarity 0.89)
- Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation (NeurIPS 2025 · similarity 0.89)
- MonoDiff: Monocular 3D Object Detection and Pose Estimation with Diffusion Models (CVPR 2024 · similarity 0.88)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.88)
- BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.