Research map / 3D vision / Depth and stereo
Vision-Language Embodiment for Monocular Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes embodied depth priors from camera geometry and planar assumptions, plus CLIP-based text VAE with image-conditioned sampler, for monocular depth estimation on KITTI and DDAD.
Problem
Monocular depth estimation from a single image is ill-posed and scale-ambiguous; existing supervised models underuse intrinsic camera information and language priors on object scale.
Approach
Computes a dense 'embodied scene depth' from camera intrinsics/height under a planar ground assumption, fuses it with RGB features via cross-attention, and uses a text VAE over captions augmented with depth descriptions as a latent prior, sampled by an image-conditioned sampler.
Key results
- On KITTI Eigen split the method reports AbsRel 0.0251 and RMSE 1.654, versus 0.044 AbsRel for AFNet and 0.048 for ECoDepth.
- On DDAD the method reports AbsRel 0.145 and RMSE 8.673, versus 0.149 AbsRel for BinsFormer and 8.989 RMSE for iDisc.
- Ablation on KITTI shows the full model (ESD+EFF+TD+DD) lowers AbsRel from 0.050 baseline to 0.0251.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Embodied Scene Depth (ablation) | Ours · KITTI | AbsRel: 0.050 -> 0.0310 | Helped |
| Add Embodied Feature Fusion (ablation) | Ours with Embodied Scene Depth · KITTI | AbsRel: 0.0310 -> 0.0314 | Hurt |
| Add text description (ablation) | Ours with Embodied Scene Depth and Embodied Feature Fusion · KITTI | AbsRel: 0.0314 -> 0.0258 | Helped |
| Add depth description (ablation) | Ours with text description · KITTI | AbsRel: 0.0258 -> 0.0251 | Helped |
| Use the proposed method instead of Metric3D v2 | Monocular depth estimation · KITTI Eigen split | AbsRel: 0.039 -> 0.0251 | Helped |
| Use the proposed method instead of BinsFormer | Monocular depth estimation · DDAD | RMSE: 8.989 -> 8.673 | Helped |
Surprising findings
- Embodied road depth on KITTI shows 99.33% of pixels within 10% error, suggesting near-LiDAR accuracy on flat road surfaces.
- Segmentation model choice has little effect on embodied ground depth accuracy.
Limitations
- Planar assumption degrades on non-flat surfaces such as sidewalks, rail tracks and parking lots.
- Evaluated only on driving datasets (KITTI, DDAD), relying on road-scene geometry.
Experimental setup
Datasets: KITTI DDAD
Benchmarks: KITTI Eigen split DDAD
Metrics: AbsRel Sq Rel RMSE RMSE log delta<1.25
Baselines: ECoDepth MiM AFNet Depth Anything Metric3D v2 UniDepth BTS AdaBins iDisc WorDepth
Builds on: WorDepth CLIP Restormer ExpansionNet-v2 Telea inpainting
Closest papers by meaning
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
- On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation (CVPR 2024 · similarity 0.88)
- DepthCues: Evaluating Monocular Depth Perception in Large Vision Models (CVPR 2025 · similarity 0.86)
- Depth Prompting for Sensor-Agnostic Depth Estimation (CVPR 2024 · similarity 0.86)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.86)
- RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.