Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns the pre-trained Marigold-LCM diffusion depth model into a zero-shot metric depth estimator at inference time by optimizing its noise latent and metric scale/offset against a defocus-blur forward model using an AIF/blurred image pair.
Problem
Monocular metric depth estimation (MMDE) methods generalize poorly to out-of-distribution data, and relative-depth models lack metric scale; the paper seeks zero-shot metric depth without retraining.
Approach
Frame metric depth as an inverse problem under a disc-PSF defocus image formation model, using two DSLR images (small and large aperture) and optimize Marigold's initial noise latent plus sigmoid-bounded scale and offset via gradients from the blur reconstruction loss, with no retraining.
Key results
- On the self-collected dataset, the Disc-PSF method reaches RMSE 0.273, REL 0.125 and delta1 0.879 averaged over 7 scenes, beating MLPro (RMSE 0.468), Metric3D (0.459) and UniDepth (0.644).
- Using a Gaussian PSF instead of the disc PSF degrades RMSE to 0.528, showing the value of a physically matched forward model.
- Using a single blurred image without the AIF image raises RMSE to 1.36 on the toys scene versus 0.346 for the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add defocus optimization with the Disc PSF rather than using a Gaussian PSF (ablation) | Marigold-LCM metric depth estimation · Collected real dataset (7 scenes) | RMSE: 0.528 -> 0.273 | Helped |
| Use a single f/16 blurred image as both input and loss reference instead of the AIF-plus-blurred-image pair (ablation) | Marigold-LCM on toys · toys | RMSE: 0.346 -> 1.36 | Hurt |
| Add optimization of the noise latent to the affine-parameter optimization (ablation) | Marigold-LCM metric depth estimation · Collected real dataset | RMSE: 0.297 -> 0.273 | Helped |
| Add optimization of the noise latent to the affine-parameter optimization (ablation) | Marigold-LCM metric depth estimation · Collected real dataset | δ1: 0.743 -> 0.879 | Helped |
| Use 2, 3, or 4 sampling steps instead of 1 (ablation) | Marigold-LCM with defocus optimization · Sampling-step ablation | RMSE (m): 0.2345; 0.2352; 0.2501; 0.2410 | Mixed |
| Change the blurred-image aperture from f/8 to f/4, f/11, f/13, or f/16 (ablation) | Marigold-LCM with defocus optimization · Real dataset aperture ablation | RMSE (m): 0.2345; 0.2722; 0.2335; 0.3682; 0.4172 | Mixed |
| Vary the initial noise latent across runs (ablation) | Marigold-LCM with defocus optimization · stairs and shoerack | RMSE (m): stairs: 0.27 (0.01); shoerack: 0.25 (0.02) | No clear effect |
| Vary the initial α and β values (ablation) | Marigold-LCM with defocus optimization · Initialization sensitivity experiment | δ1 | Mixed |
Surprising findings
- Optimizing with a Gaussian PSF, a common approximation, performs worse than the uncorrected MMDE baselines on average.
- Increasing diffusion sampling steps raised runtime without consistent metric gains and increased texture-depth coupling.
Limitations
- Inference is significantly slower than feed-forward MMDE methods (minutes per scene).
- Performance is limited when the initial Marigold prediction is badly wrong in some regions (e.g., shoe rack).
- Forward model loses accuracy at occlusion boundaries and is best suited to indoor scenes with small depth ranges.
Experimental setup
Datasets: NYU-v2
Benchmarks: self-collected 7-scene real dataset (DSLR + RealSense D435) NYU-v2 (simulated blur)
Metrics: RMSE REL log10 delta1 delta2 delta3
Baselines: MLPro UniDepth Metric3D Ours - Gaussian PSF
Builds on: Marigold Stable Diffusion v2 latent consistency model depth from defocus
Closest papers by meaning
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.89)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.88)
- SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillation (CVPR 2025 · similarity 0.87)
- Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation (NeurIPS 2024 · similarity 0.87)
- Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior (AAAI 2025 · similarity 0.87)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.