Research map / Vision-language and multimodal / Multimodal LLMs

WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WalkGPT, a pixel-grounded LVLM that answers pedestrian accessibility queries with segmentation masks and text-based relative distances, trained on a new 41k-sample PAVE dataset built from SANPO.

Problem

Existing LVLMs lack explicit grounding and depth reasoning needed for accessibility-aware pedestrian navigation, leading to hallucinated objects and unreliable distance estimates; no large pedestrian-view dataset with joint QA and grounding annotations existed.

Approach

Combines a SAM-based pixel encoder with an LLM via a Multi-Scale Query Projector (MSQP) for multi-scale image tokens and a Calibrated Text Projector (CTP) with a Region Alignment (InfoNCE) loss for mapping SEG-token embeddings to the mask decoder, with depth expressed as language tokens rather than a depth head.

Key results

• WalkGPT (13B) reaches 20.16 mIoU and 48.95% Depth Acc. on PAVE, versus 18.10 mIoU and 39.00% for PixelLM-FT.
• WalkGPT achieves 76.2% on RefCOCO val and 72.6% on RefCOCOg val, outperforming LISA and PixelLM by roughly 3-4 points on those splits.
• WalkGPT lowers CHAIR_i hallucination to 18.49 versus 22.16 for LLaVA-1.5 and raises object coverage to 83.66.

Surprising findings

• Zero-shot grounded LVLMs fail to produce any depth estimates even with added distance tokens.
• Removing the distance token barely changes segmentation mIoU (20.16 to 20.01), while depth accuracy drops sharply.

Limitations

• Dataset artifacts such as road reflections and transparent fences cause misinterpretation of accessibility.
• Depth is only expressed as text and not dense or metric-depth estimation; Swin-UNETR has higher mIoU (20.60) than WalkGPT on PAVE.
• Evaluation is on 6 held-out sessions (about 600 frames) and cross-domain generalization is untested.

Experimental setup

Datasets: PAVE SANPO ADE20K RefCOCO RefCOCO+ RefCOCOg

Benchmarks: PAVE RefCOCO RefCOCO+ RefCOCOg

Metrics: CIDEr METEOR AP50 mIoU Recall Depth Acc. AbsRel CHAIR_i Cover

Baselines: GLAMM LISA PixelLM GSVA OMG-LLaVA Sa2VA MCN VLT CRIS LAVT ReLA U-Net

Builds on: LISA SAM GLAMM LLaVA

Closest papers by meaning

• mmWalk: Towards Multi-modal Multi-view Walking Assistance (NeurIPS 2025 · similarity 0.86)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.85)
• DialogueVPR: Towards Conversational Visual Place Recognition (CVPR 2026 · similarity 0.85)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.85)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.85)
• Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.85)