Research map / Vision-language and multimodal / Multimodal LLMs
Evaluating Vision-Language Models as Evaluators in Path Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PATHEVAL, a benchmark testing whether VLMs can act as plan evaluators by choosing the better of two rendered paths under natural-language scenarios, finding that visual low-level perception is the bottleneck.
Problem
LLMs and VLMs are weak end-to-end planners, but may still serve as plan evaluators; it is unclear whether VLMs can perceive fine-grained path properties (e.g., clearance) from images well enough to judge which of two paths better satisfies a natural-language scenario.
Approach
A controllable benchmark of RRT-generated path pairs in 2D/3D maze-like environments, mapped to 15 commonsense scenarios with required optimization descriptors, used to decompose VLM failure into attribute abstraction, low-level perception, and information integration; it further probes vision encoders with linear probes and fine-tuning.
Key results
- Most open VLMs score near random (about 0.50) on PATHEVAL 2D, while GPT-4o reaches 0.665 (2D) and 0.624 (3D); Qwen2-VL-7b reaches 0.502 without descriptors.
- Providing verbalized descriptor values raises Qwen2-VL-7b to 0.731 (2D) and GPT-4o to 0.860 (2D), indicating the vision component is the bottleneck.
- Fine-tuning LLaVA-NeXT-7b with frozen vision encoder fails on image-only input (0.52), while task-specific discriminative fine-tuning of CLIP/DINO encoders improves path-pair separability in linear probes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide verbalized path descriptor values along with the images | GPT-4o on PATHEVAL 2D · PATHEVAL 2D | accuracy: 0.665 -> 0.860 | Helped |
| Provide verbalized path descriptor values along with the images | Qwen2-VL-7b on PATHEVAL 2D · PATHEVAL 2D | accuracy: 0.502 -> 0.731 | Helped |
| Provide verbalized path descriptor values along with the images | GPT-4o on PATHEVAL 3D · PATHEVAL 3D | accuracy: 0.624 -> 0.895 | Helped |
| Provide verbalized path descriptor values along with the images | Qwen2-VL-7b on PATHEVAL 3D · PATHEVAL 3D | accuracy: 0.511 -> 0.742 | Helped |
| Fine-tune on PATHEVAL training pairs using images only (ablation) | LLaVA-NeXT-7b · PATHEVAL 2D | accuracy: 0.51 -> 0.52 | No clear effect |
| Add textual descriptor values to the fine-tuning inputs (ablation) | LLaVA-NeXT-7b · PATHEVAL 2D | accuracy: 0.51 -> 0.96 | Helped |
| Use random IDs as target labels during image-only fine-tuning (ablation) | LLaVA-NeXT-7b · PATHEVAL 2D | accuracy: 0.48 | No clear effect |
| Fine-tune the vision encoder for path-pair discrimination rather than keeping it frozen (ablation) | CLIP ViT-B/32 · Path-pair identity probe (2D) | accuracy: 0.510 -> 0.783 | Helped |
| Fine-tune the vision encoder for path-pair discrimination rather than keeping it frozen (ablation) | CLIP ViT-L/14-336 · Path-pair identity probe (2D) | accuracy: 0.498 -> 0.749 | Helped |
| Fine-tune the vision encoder for path-pair discrimination rather than keeping it frozen (ablation) | OpenCLIP ViT-B/32 · Path-pair identity probe (2D) | accuracy: 0.540 -> 0.743 | Helped |
| Fine-tune the vision encoder for path-pair discrimination rather than keeping it frozen (ablation) | SigLIP base · Path-pair identity probe (2D) | accuracy: 0.529 -> 0.731 | Helped |
| Fine-tune the vision encoder for path-pair discrimination rather than keeping it frozen (ablation) | DINO ViT-16 · Path-pair identity probe (2D) | accuracy: 0.495 -> 0.763 | Helped |
Surprising findings
- GPT-4o favors the shortest path even when the scenario requires maximizing length, suggesting overuse of commonsense priors.
- Fine-tuning the full VLM end-to-end does not fix the vision bottleneck; encoders must be adapted with a discriminative objective.
Limitations
- Benchmark is limited to synthetic 2D/3D RRT-generated paths in fixed obstacle layouts (rings, waves, mazes, random).
- Human annotators were tested on only 60 sampled examples, and 3D cases are hard even for humans.
- Fine-tuning experiments use only LLaVA-NeXT-7b and a single setup.
Experimental setup
Datasets: PATHEVAL
Benchmarks: PATHEVAL
Metrics: accuracy per-descriptor perception accuracy average cosine similarity
Baselines: GPT-4o-mini LLaVA-NeXT-13b Qwen2-VL-7b LLaVA-OneVision-7b Intern-VL2-8b Intern-VL2-40b LLaMA-3.2-11b Random baseline
Builds on: RRT-Connect LLaVA-NeXT
Closest papers by meaning
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.90)
- World-aware Planning Narratives Enhance Large Vision-Language Model Planner (NeurIPS 2025 · similarity 0.90)
- From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving (NeurIPS 2026 · similarity 0.90)
- Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · similarity 0.90)
- VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.88)
- Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.