Research map / 3D vision / Depth and stereo
RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RSA, which predicts a global scale and shift from a text caption (frozen CLIP text encoder + MLP heads) to convert relative inverse depth from frozen monocular depth models into metric depth.
Problem
Monocular depth models predict scale-ambiguous relative depth, and existing relative-to-metric alignment either needs ground truth at test time or dataset-specific metric decoders that fail to generalize across domains.
Approach
Use language descriptions of objects in a scene (structured from panoptic segmentation or natural captions from LLaVA) to predict the scale and shift of a global linear transformation that maps inverse relative depth to metric depth, with the depth model frozen and no ground truth needed at test time.
Key results
- On NYUv2 with Depth Anything, RSA (NYUv2,KITTI) reaches Abs Rel 0.147 vs 0.169 for image-based scaling and 0.058 for ground-truth linear fit.
- Zero-shot on DDAD with Depth Anything, RSA (NYUv2,KITTI,VOID) achieves Abs Rel 0.198 vs 0.243 for image-based scaling.
- Zero-shot on SUN-RGBD with Depth Anything, RSA (NYUv2,KITTI,VOID) achieves Abs Rel 0.203 vs 0.279 for image-based scaling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace image features with language descriptions for scale and shift prediction | RSA relative-to-metric alignment · NYUv2; DPT | Abs Rel: 0.097 vs 0.097 | No clear effect |
| Replace image features with language descriptions for scale and shift prediction | RSA relative-to-metric alignment · VOID; DPT trained on NYUv2,KITTI,VOID | Abs Rel: 0.266 -> 0.248 | Helped |
| Train scale prediction on NYUv2,KITTI,VOID rather than using image scaling | DepthAnything relative-to-metric alignment · SUN-RGBD zero-shot | Abs Rel: 0.279 -> 0.203 | Helped |
| Train scale prediction on NYUv2,KITTI,VOID rather than using image scaling | DPT relative-to-metric alignment · DDAD zero-shot | Abs Rel: 0.191 -> 0.165 | Helped |
| Include background classes from panoptic segmentation in captions instead of foreground objects only (ablation) | RSA with DPT · KITTI | Abs Rel: 0.068 -> 0.061 | Helped |
| Include background classes from panoptic segmentation in captions instead of foreground objects only (ablation) | RSA with DPT · NYUv2 | Abs Rel: 0.101 -> 0.100 | Helped |
| Add object-instance counts to captions (ablation) | RSA with DPT and panoptic segmentation captions · NYUv2 | Abs Rel: 0.102 -> 0.100 | Helped |
| Add object-instance counts to captions (ablation) | RSA with DPT and panoptic segmentation captions · KITTI | Abs Rel: 0.063 -> 0.061 | Helped |
Surprising findings
- Captions listing only foreground object types and counts work well indoors but poorly outdoors; including background classes from segmentation substantially helps outdoor scale prediction.
- Single-dataset RSA models are only marginally better than the unified multi-dataset model, suggesting a narrow domain gap for language.
Limitations
- Assumes a single global scale and shift per image, which may be insufficient when relative depth is inaccurate or contains outliers.
- Vulnerable to malicious or uninformative captions that steer predictions incorrectly.
Experimental setup
Datasets: NYUv2 KITTI VOID SUN-RGBD DDAD
Benchmarks: NYUv2 KITTI VOID SUN-RGBD DDAD
Metrics: Abs Rel RMSE log10 RMSE log delta threshold accuracy
Baselines: ZoeDepth DistDepth ZeroDepth DPT MiDaS Depth Anything AdaBins DepthFormer NeWCRFs Monodepth2 Image-based scaling Median scaling
Builds on: DPT Depth Anything CLIP MaskDINO LLaVA v1.6
Closest papers by meaning
- DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images (NeurIPS 2026 · similarity 0.89)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.89)
- WorDepth: Variational Language Prior for Monocular Depth Estimation (CVPR 2024 · similarity 0.88)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.88)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.88)
- SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.