Research map / Vision-language and multimodal / Human and embodied modeling
On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically probes how language guidance (scene, caption, spatial-relationship, activity sentences) affects VPD-style language-guided monocular depth estimation, finding scene-level bias, poor robustness to masking and distribution shift versus vision-only depth models.
Problem
Whether natural-language priors actually help language-guided monocular depth estimation, and how robust and generalizable these methods are to low-level spatial sentences, object masking, and dataset/scene shift.
Approach
Generates object-centric 3D spatial-relationship sentences (depth, horizontal, vertical) from NYUv2 ground-truth depth and segmentation, injects them as extra language guidance into VPD, and benchmarks against vision-only depth estimators under masking and distribution shift.
Key results
- Adding low-level spatial sentences to scene-level guidance raises RMSE from 0.382 to 0.440 in supervised VPD training (Table 1).
- In zero-shot VPD, RMSE rises from 0.252 (scene-level) to 1.223 (vertical-only sentences) (Table 2).
- Under scene distribution shift VPD has a 107.48% RMSE increase versus original, the largest among compared models; on Sun RGB-D VPD's RMSE is about 20% higher than MIM-Depth (0.442 vs 0.367).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add low-level spatial sentences to scene-level sentences during training (ablation) | VPD · NYUv2 supervised test | RMSE: 0.382 -> 0.440 | Hurt |
| replace scene-level sentences with low-level spatial sentences during training (ablation) | VPD · NYUv2 supervised test | RMSE: 0.382 -> 0.424 | Hurt |
| add captions to scene-level guidance (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 0.279; 0.252 -> 0.347 | Hurt |
| add depth-relation sentences to scene-level guidance and captions (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 0.311; 0.252 -> 0.399 | Hurt |
| add horizontal and vertical relations to scene-level guidance, captions, and depth sentences (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 0.403; 0.252 -> 0.443 | Hurt |
| replace scene-level guidance with captions (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 0.474 | Hurt |
| replace scene-level guidance with depth-relation sentences (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 1.045 | Hurt |
| replace scene-level guidance with vertical-relation sentences (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 1.223 | Hurt |
| replace scene-level guidance with horizontal-relation sentences (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.252 -> 1.148 | Hurt |
| replace scene-level descriptions with activity-level descriptions (ablation) | VPD · NYUv2 zero-shot test | RMSE: 0.254 -> 0.297 | Hurt |
| mask objects in the input image (ablation) | VPD with scene-level and depth guidance · NYUv2 masked-object test | ΔRMSE: 0.093 vs 0.007 | Hurt |
| mask objects in the input image and use only depth-relation guidance (ablation) | VPD · NYUv2 masked-object test | ΔRMSE: 0.794 vs 0.007 | Hurt |
Surprising findings
- Low-level spatial sentences degrade depth estimation relative to scene-level prompts, even though they explicitly encode depth information.
- CLIPScore barely differs between correct and relationship-swapped sentences, and sometimes favors incorrect ones.
Limitations
- Experiments are built on a single language-guided method (VPD), limiting generality to other language-guided depth estimators.
- Sentences are generated from NYUv2 ground-truth annotations and templates, which may not reflect natural human descriptions.
- Zero-shot results use the open-source VPD checkpoint without retraining on the new sentence types.
Experimental setup
Datasets: NYUv2 Sun RGB-D
Benchmarks: NYUv2 Sun RGB-D
Metrics: RMSE Abs. REL Log10 delta1 delta2 delta3
Baselines: AdaBins MIM-Depth IDisc
Builds on: VPD Stable Diffusion CLIP AdaBins
Closest papers by meaning
- DepthLM: Metric Depth from Vision Language Models (ICLR 2026 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- SOVGaussian: Sparse-View 3D Gaussian Splatting for Open-Vocabulary Scene Understanding (AAAI 2025 · similarity 0.88)
- Vision-Language Embodiment for Monocular Depth Estimation (CVPR 2025 · similarity 0.88)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.88)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.