Research map / Image generation / Fast diffusion sampling
ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECoDepth, a Stable-Diffusion-based monocular depth estimator conditioned on a ViT class-probability embedding (CIDE module) instead of BLIP/CLIP pseudo-caption embeddings, reaching new SOTA on NYU Depth v2 and KITTI.
Problem
Single image depth estimation relies on contextual priors, and prior diffusion-based methods condition on pseudo-caption text embeddings that emphasize salient objects and miss smaller scene content; the paper asks whether ViT embeddings are a better conditioning signal.
Approach
Condition a latent diffusion UNet on a learned linear combination of learnable embeddings, weighted by the 1000-dim ImageNet class-logit vector from a frozen ViT (the CIDE module), and regress depth from multi-scale UNet decoder features.
Key results
- NYU Depth v2 Abs Rel 0.059 and RMSE 0.218, vs. VPD 0.069 and 0.254 (14% Abs Rel improvement).
- KITTI Sq Rel 0.139, vs. GEDepth 0.142 (2% improvement).
- Zero-shot transfer from NYU-only training gives mean relative improvement over NeWCRFs of 20.5%, 23.1%, 81.3%, 25.2% on SUN RGB-D, iBims-1, DIODE, HyperSim, versus ZoeDepth's 16.0%, 18.5%, 45.0%, 8.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition the diffusion backbone on ViT-derived scene embeddings | ECoDepth · NYU Depth v2 | Abs Rel: 0.069 -> 0.059 | Helped |
| Condition the diffusion backbone on ViT-derived scene embeddings | ECoDepth · KITTI | Sq Rel: 0.142 -> 0.139 | Helped |
| Train on NYU Depth v2 and transfer zero-shot | ECoDepth · SUN RGB-D | mRIθ: 20.5% vs 0.0% | Helped |
| Train on NYU Depth v2 and transfer zero-shot | ECoDepth · iBims-1 | mRIθ: 23.1% vs 0.0% | Helped |
| Train on NYU Depth v2 and transfer zero-shot | ECoDepth · DIODE Indoor | mRIθ: 81.3% vs 0.0% | Helped |
| Train on NYU Depth v2 and transfer zero-shot | ECoDepth · HyperSim | mRIθ: 25.2% vs 0.0% | Helped |
| Replace ViT scene embeddings with one-hot scene-label embeddings (ablation) | ECoDepth · NYU Depth v2 | RMSE: 0.218 -> 0.244 | Hurt |
| Replace ViT scene embeddings with one-hot scene-label embeddings (ablation) | ECoDepth · NYU Depth v2 | Abs Rel: 0.059 -> 0.067 | Hurt |
| Replace ViT scene embeddings with one-hot scene-label embeddings (ablation) | ECoDepth · NYU Depth v2 | δ1: 0.978 -> 0.968 | Hurt |
Surprising findings
- A one-hot scene-label embedding performs nearly as well as the CLIP caption embedding (RMSE 0.244 vs 0.225 for text caption), suggesting caption information content resembles a label.
Limitations
- VPD cannot be trained on KITTI because it needs per-image text labels, so KITTI comparison is limited to non-VPD methods.
- KITTI gains over prior SOTA are marginal (2% on Sq Rel, comparable on other metrics).
Experimental setup
Datasets: NYU Depth v2 KITTI
Benchmarks: NYU Depth v2 KITTI Sun-RGBD iBims-1 DIODE Indoor HyperSim
Metrics: Abs Rel RMSE log10 Sq Rel delta1 delta2 delta3 mRI
Baselines: VPD TADP ZoeDepth NeWCRFs GEDepth MIM AiT DDP PixelFormer IEBins URCDC
Builds on: VPD Stable Diffusion ZoEDepth TADP
Closest papers by meaning
- BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation (NeurIPS 2024 · similarity 0.88)
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.87)
- BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model (NeurIPS 2024 · similarity 0.86)
- WorDepth: Variational Language Prior for Monocular Depth Estimation (CVPR 2024 · similarity 0.86)
- Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation (NeurIPS 2025 · similarity 0.86)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.