Research map / 3D vision / Depth and stereo
WorDepth: Variational Language Prior for Monocular Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WorDepth, a text-conditioned variational (VAE) prior for monocular depth estimation that uses CLIP caption embeddings and an image-based conditional sampler to produce metric-scale depth maps.
Problem
Monocular depth estimation from a single image is ill-posed, especially scale ambiguity; the paper asks whether text captions describing the scene can serve as a prior to ground depth to metric scale.
Approach
Encode the caption with a frozen CLIP text encoder into a mean and standard deviation of a latent distribution over plausible depth maps (text-VAE), and use an image-based conditional sampler (Swin-L) to select a specific latent sample per patch, trained by alternating optimization between text-VAE and sampler.
Key results
- On NYU Depth V2, WorDepth reaches delta<1.25 of 0.932 and Abs Rel of 0.088, versus 0.910 and 0.098 for the Swin-L baseline (about 10% relative Abs Rel reduction).
- On KITTI Eigen split, WorDepth achieves delta<1.25 of 0.979 and Abs Rel of 0.049, improving over the Swin-L baseline (0.969, 0.054).
- In zero-shot transfer from NYU Depth V2 to SUN-RGBD, WorDepth gets delta<1.25 of 0.833 vs 0.803 for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the language variational prior and image-based conditional sampler | Swin-L depth estimator · NYU Depth V2 | Abs Rel: 0.098 -> 0.088 | Helped |
| add the language variational prior and image-based conditional sampler | Swin-L depth estimator · KITTI Eigen Split | Abs Rel: 0.054 -> 0.049 | Helped |
| add the language variational prior and image-based conditional sampler | depth estimator trained on NYU Depth V2 · SUN-RGBD zero-shot | Abs Rel: 0.141 -> 0.123 | Helped |
| set text-VAE optimization probability p to 0% rather than 1% (ablation) | WorDepth · NYU Depth V2 | Abs Rel: 0.088 -> 0.091 | Hurt |
| set text-VAE optimization probability p to 10% rather than 1% (ablation) | WorDepth · NYU Depth V2 | Abs Rel: 0.088 -> 0.090 | Hurt |
| set text-VAE optimization probability p to 50% rather than 1% (ablation) | WorDepth · NYU Depth V2 | Abs Rel: 0.088 -> 0.163 | Hurt |
| set text-VAE optimization probability p to 90% rather than 1% (ablation) | WorDepth · NYU Depth V2 | Abs Rel: 0.088 -> 0.211 | Hurt |
| set text-VAE optimization probability p to 100% rather than 1% (ablation) | WorDepth · NYU Depth V2 | Abs Rel: 0.088 -> 0.225 | Hurt |
Surprising findings
- Sampling with probability 100% on the text-VAE branch (no image conditioning) still yields reasonable caption-to-depth results, and performance degrades sharply when the text-VAE is updated more often (p>=10%) under a fixed training budget.
- Gains on KITTI in the delta<1.25 metric are smaller than on NYU, which the authors attribute to vaguer object-size priors for outdoor captions such as 'car'.
Limitations
- Performance depends on caption quality; inaccurate or vague captions can misguide depth predictions.
- Captions can be maliciously crafted to steer predictions incorrectly.
- Gains are smaller on outdoor KITTI where object sizes vary widely within the same caption category.
Experimental setup
Datasets: NYU Depth V2 KITTI SUN-RGBD
Benchmarks: NYU Depth V2 KITTI SUN-RGBD
Metrics: Abs Rel RMSE log10 RMSElog delta<1.25 accuracy
Baselines: DepthCLIP CLIPMDE GeoNet DORN TransDepth AdaBins NeWCRFs DepthFormer ASTransformer Swin-L baseline
Builds on: CLIP AdaBins VAE (reparameterization trick) scale-invariant loss
Closest papers by meaning
- RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions (NeurIPS 2024 · similarity 0.88)
- On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation (CVPR 2024 · similarity 0.86)
- ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation (CVPR 2024 · similarity 0.86)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.86)
- Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces (CVPR 2025 · similarity 0.85)
- UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.