Research map / Label-efficient and robust learning / Self-supervised and contrastive
Synthetic-to-Real Self-supervised Robust Depth Estimation via Learning with Motion and Structure Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a synthetic-to-real self-supervised depth estimation framework for daytime, rain, and nighttime that transfers cost-volume motion-structure knowledge from a frozen daytime Manydepth model, then adapts to real adverse data via consistency reweighting and a daytime depth-distribution prior.
Problem
Self-supervised monocular depth estimation under diverse outdoor conditions (day, night, rain) is hard because unified representations are difficult to learn and labeled real adverse data is scarce; prior synthetic-only or direct photometric adaptation methods are suboptimal.
Approach
Distill daytime motion-structure knowledge via cost volumes (rather than features or pseudo-depth alone) into a synthetic-adverse model, then adapt to real adverse data using a consistency-reweighted pseudo-label loss between daytime and synthetic model predictions plus a KL-based alignment of differentiable depth histograms to the daytime depth distribution.
Key results
- On nuScenes single-frame, the method gives average gains of 4.8% AbsRel, 16.8% SqRel and 4.2% RMSE over md4all and DM-MDE on day/night/rain.
- On Robotcar (day/night) single-frame, gains of 7.8% AbsRel, 6.5% SqRel and 2.7% RMSE over recent methods.
- Zero-shot on DrivingStereo rain and fog, AbsRel improves 3.0% over the state of the art.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cost-volume learning (Lcv) during synthetic adaptation instead of depth supervision only (ablation) | Manydepth-based depth estimator · nuScenes-night | AbsRel: 0.1865 -> 0.1809 | Helped |
| add cost-volume learning (Lcv) during synthetic adaptation instead of depth supervision only (ablation) | Manydepth-based depth estimator · nuScenes-rain | AbsRel: 0.1310 -> 0.1290 | Helped |
| add direct daytime feature learning (Lfeat) (ablation) | Manydepth-based depth estimator · nuScenes-night | AbsRel: 0.1865 -> 0.1911 | Hurt |
| add direct daytime feature learning (Lfeat) (ablation) | Manydepth-based depth estimator · nuScenes-rain | AbsRel: 0.1310 -> 0.1335 | Hurt |
| add real adaptation without consistency reweighting or structure prior (ablation) | SA + Lcv · nuScenes-night | AbsRel: 0.1809 -> 0.1781 | Helped |
| add real adaptation without consistency reweighting or structure prior (ablation) | SA + Lcv · nuScenes-rain | AbsRel: 0.1290 -> 0.1288 | Helped |
| add consistency reweighting (CR) (ablation) | SA + Lcv + RA · nuScenes-night | AbsRel: 0.1781 -> 0.1743 | Helped |
| add consistency reweighting (CR) (ablation) | SA + Lcv + RA · nuScenes-rain | AbsRel: 0.1288 -> 0.1256 | Helped |
| add structure prior constraint (LKLdis) (ablation) | SA + Lcv + RA + CR · nuScenes-night | AbsRel: 0.1743 -> 0.1713 | Helped |
| add structure prior constraint (LKLdis) (ablation) | SA + Lcv + RA + CR · nuScenes-rain | AbsRel: 0.1256 -> 0.1266 | Hurt |
| apply synthetic-to-real adaptation with motion and structure priors | Ours-Single versus strongest reported baseline · Robotcar-night | AbsRel: 0.1219 -> 0.1103 | Helped |
| apply synthetic-to-real adaptation with motion and structure priors | Ours-Single versus strongest reported baseline · DrivingStereo-fog, zero-shot | AbsRel: 0.1259 -> 0.1281 | Hurt |
Surprising findings
- Naive feature distillation (Lfeat) performs significantly worse than the synthetic-only baseline, while cost-volume distillation helps.
- Applying real adaptation with pseudo-labels alone does not reliably improve real-data performance without reweighting and the prior.
Limitations
- Relies on synthetic adverse images from pretrained augmentation models and a frozen daytime teacher, so the pipeline inherits their synthetic-real gap.
- Evaluated only on driving datasets (nuScenes, Robotcar, DrivingStereo) with limited condition types.
- Multi-frame results on Robotcar and DrivingStereo are not reported because those test sets are single-frame.
Experimental setup
Datasets: nuScenes Robotcar DrivingStereo synthetic adverse images from pretrained augmentation models
Benchmarks: nuScenes Robotcar DrivingStereo
Metrics: AbsRel SqRel RMSE delta1
Baselines: Monodepth2 R4Dyn RNW md4all-AD md4all-DD robust-depth DM-MDE Manydepth WSGD DeFeatNet ADIDS
Builds on: Manydepth md4all Monodepth2 DM-MDE
Closest papers by meaning
- Learning Depth from Past Selves: Self-Evolution Contrast for Robust Depth Estimation (AAAI 2026 · similarity 0.88)
- From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior (CVPR 2024 · similarity 0.87)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.87)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.86)
- PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation (AAAI 2024 · similarity 0.86)
- Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.