Research map / Label-efficient and robust learning / Self-supervised and contrastive
Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jasmine adapts Stable Diffusion v2 into a self-supervised monocular depth estimator, using mix-batch image reconstruction to preserve SD priors and a Scale-Shift GRU to bridge SSI-to-SI depth mismatch.
Problem
Self-supervised monocular depth estimation suffers from blur and artifacts due to reprojection losses (occlusions, texture-less regions, illumination changes), and SD-based depth models have required high-precision supervision, so SD priors were not usable in self-supervised training.
Approach
Train SD's U-Net in a self-supervised setting with a mix-batch surrogate task that alternates depth prediction with image reconstruction (KITTI + Hypersim images, photometric loss) to anchor SD latent priors, and a Scale-Shift GRU that iteratively converts SSI depth into SI depth while filtering noisy gradients.
Key results
- On KITTI (Eigen, 1024x320), Jasmine reaches AbsRel 0.090, SqRel 0.581, RMSE 3.944, a1 0.919, outperforming MonoViT (AbsRel 0.096) and other self-supervised methods.
- Zero-shot on CityScape, Jasmine achieves AbsRel 0.123 vs. MonoViT 0.140 and Mono-ViFI 0.134 (self-supervised baselines).
- Ablation: removing the SD prior causes AbsRel to rise to 0.516 from 0.090; disabling SSG raises AbsRel to 0.129 and removing MIR raises it to 0.132.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Stable Diffusion prior (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.516 -> 0.090 | Helped |
| Add MIR and SSG (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.175 -> 0.090 | Helped |
| Add SSG (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.129 -> 0.090 | Helped |
| Add MIR (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.132 -> 0.090 | Helped |
| Add the Scale-Shift Transformer to SSG (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.098 -> 0.090 | Helped |
| Use MIR instead of direct image reconstruction (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.129 -> 0.090 | Helped |
| Mix KITTI and Hypersim reconstruction images instead of using Hypersim only (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.106 -> 0.090 | Helped |
| Use photometric rather than latent-space reconstruction supervision (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.095 -> 0.090 | Helped |
| Use KITTI and ETH3D as auxiliary reconstruction images (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.090 -> 0.090 | No clear effect |
| Use KITTI and Virtual KITTI instead of KITTI and Hypersim as auxiliary data (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.090 -> 0.094 | Hurt |
| Use downsampled Hypersim images at Hy/4 (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.090 -> 0.091 | No clear effect |
| Use downsampled Hypersim images at Hy/1.6 (ablation) | Jasmine · KITTI Eigen raw test | AbsRel: 0.090 -> 0.090 | No clear effect |
Surprising findings
- Synthetic auxiliary images are not essential: real-world KITTI+ETH3D auxiliary data gives comparable results, and fewer than 1k auxiliary samples remain effective.
- Domain divergence of auxiliary data matters more than its image quality or scale for the surrogate task.
Limitations
- Substantially more compute (2.83T MACs) than prior self-supervised methods such as Monodepth2 and MonoViT.
- Relies on a pretrained self-supervised teacher (MonoViT) pseudo-label loss to stabilize training, with a performance upper bound.
- Evaluation on zero-shot benchmarks compares against a limited set of self-supervised baselines because many SD-based models lack open code.
Experimental setup
Datasets: KITTI Hypersim Virtual KITTI ETH3D DrivingStereo CityScape
Benchmarks: KITTI DrivingStereo CityScape KITTI improved GT
Metrics: AbsRel SqRel RMSE RMSElog a1 a2 a3
Baselines: Marigold E2E FT Lotus Monodepth2 HR-Depth R-MSFM6 DevNet DepthSegNet SD-SSMDE MonoViT LiteMono DaCCN
Builds on: Stable Diffusion E2E FT Lotus Monodepth2 MonoViT Marigold
Closest papers by meaning
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (CVPR 2024 · similarity 0.89)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.87)
- ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation (CVPR 2024 · similarity 0.86)
- BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation (NeurIPS 2024 · similarity 0.86)
- Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior (AAAI 2025 · similarity 0.86)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.