Research map / Vision-language and multimodal / Multimodal LLMs

VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VLMs frequently lose a selected path under nearby similar distractors in controlled circuit and spiral tracing tasks, with failures driven by local competition that scaling, reasoning, and prompting do not fix.

Problem

VLMs score well on multimodal benchmarks but may lack robust control over basic visual operations such as line tracing; prior work conflates crossings, long paths and shortcuts, obscuring the actual failure mechanism.

Approach

Isolates line tracing in crossing-free controlled tasks (modified circuit connections and a new abstract Swirl dataset) and varies the similarity of nearby distractor paths, then uses causal masking and attention/representation probes to show that local competition from similar distractors drives path-switching errors.

Key results

• Gemini3-Flash is the best model on Circuit Connections at 71.0%, while Claude Sonnet 4.5 reaches only 6.2%; on Swirl at High density, GPT-5.4 gets 20.3% and Claude Sonnet 4.5 and Qwen3-VL-235B get 0%.
• In Qwen3-VL-8B masking tests on the Shared segment condition, masking the distractor red dot raises accuracy from 76.5% to 90.0%, while masking the queried-path red dot drops it to 2.0%.
• Adjacent-wire or adjacent-turn jumps account for most errors across models (e.g., 71.4% to 98.2% of error cases on circuit task), and reasoning models still show 71-84% adjacent-turn jump errors.

Surprising findings

• Explicit step-by-step tracing instructions lower accuracy across all four models and difficulty levels.
• Reasoning mostly substitutes global heuristics (angular matching, ring decomposition) for tracing, and GPT-5.4 uses about 3,443 tokens at High density.

Limitations

• Controlled synthetic tasks abstract away many properties of natural visual tasks.
• Closed-source model results may change across API versions; internal analyses only possible for open models.
• Failure-mode interpretations rely on interventions and error patterns rather than full mechanistic identification.

Experimental setup

Datasets: Circuit Connections (modified from smooth visual search work, 15 images / 107 wires) Swirl (synthetic, 100 images per rotation-density level; 200 per level for reasoning analysis) HANDLOOM tangled cables metro maps

Benchmarks: Circuit Connections (modified) Swirl HANDLOOM tangled cables metro maps

Metrics: exact-match accuracy on ordered dot sequence adjacent-jump error rate attention margin representation cosine margin reasoning token length

Baselines: Claude Sonnet 4.5 GPT-5.4 Gemini3-Flash Qwen3-VL-235B

Builds on: smooth visual search (circuit connection task) TraversalBench HANDLOOM Pathfinder

Closest papers by meaning

• Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.87)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.87)
• Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.87)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.86)
• InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.86)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)