Research map / Vision-language and multimodal / Multimodal LLMs

DepthLM: Metric Depth from Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Turns standard VLMs into pixel-level metric depth estimators via marker-based pixel reference, intrinsic-conditioned focal-length unification, and sparse-label text SFT, matching expert pure-vision depth models.

Problem

VLMs perform poorly at 3D understanding such as pixel-level metric depth estimation, lagging far behind pure vision expert models.

Approach

Render visual markers on images instead of text coordinates for pixel reference, unify focal length by resizing images to resolve camera ambiguity, and fine-tune with plain text SFT using one labeled pixel per image, with no dense head or regression loss.

Key results

• DepthLM 7B reaches 0.838 average delta1 across 8 datasets, versus 0.370 for GPT-5 and 0.219 for Qwen2.5-VL-72B.
• DepthLM 3B (0.824 average) surpasses Depth Pro and Metric3Dv2 on several datasets; the 7B model exceeds 0.9 delta1 on iBims1 and NYUv2.
• Multi-task DepthLM 7B averages 0.804 delta1 on distance, speed, time, two-point distance and pose, outperforming GPT-5 (0.210) by over 3.8x.

Surprising findings

• A single labeled pixel per training image already achieves over 0.8 delta1, suggesting image diversity matters more than label density.
• SFT matches GRPO accuracy at equal sample counts while being 8-16x faster per sample, so RL is not needed here.

Limitations

• Focuses on the simplest VLM design choices; no investigation of architecture changes or finer strategies.
• Excludes low-quality and synthetic datasets from training, leaving performance gains from more data unexplored.
• Accuracy still below pure vision models on some datasets such as ETH3D and outdoor sets like DDAD relative to UniDepthV2.

Experimental setup

Datasets: Argoverse2 Waymo NuScenes ScanNet++ Taskonomy HM3d Matterport3d DDAD sunRGBD iBims1 NYUv2 ETH3D

Benchmarks: DepthLMBench Argoverse2 DDAD NuScenes ETH3D ScanNet++ sunRGBD iBims1 NYUv2

Metrics: delta1 accuracy AbsRel relative accuracy improvement

Baselines: GPT-5 Gemini-2.5-Pro GPT-o3 Qwen2.5-VL Seed1.5-VL SpatialRGPT SpaceLLaVA Depth Pro Metric3Dv2 UniDepthV2 ZoeDepth

Builds on: GRPO Qwen2.5-VL Depth Pro Metric3D

Closest papers by meaning

• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.89)
• Leveraging VLM-Based Pipelines to Annotate 3D Objects (ICML 2024 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
• On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation (CVPR 2024 · similarity 0.88)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.88)