Research map / Large language models / Agents, code and math

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

NeurIPS 2026 · Large language models › Agents, code and math · approach: LLM-driven embodied and driving agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DriveHierarchy, a four-rank hierarchical benchmark (perceptual grounding, contextual memory, mental reasoning, closed-loop execution) for VLM driving, combining open-loop QA from public datasets with 100 CARLA-SUMO closed-loop scenarios, and evaluates 15 VLMs plus a benchmark-guided LoRA SFT case study.

Problem

Existing VLM driving benchmarks either assess open-loop understanding or closed-loop driving, but offer little structure for diagnosing which capabilities underlie driving competence and how they relate to closed-loop behavior.

Approach

Organize VLM driving evaluation into a hierarchy of four ranks spanning open-loop perception, memory, and reasoning plus closed-loop execution, use correlation-aware subtask weighting, and use the capability deficits to select SFT targets whose gains transfer to closed-loop driving.

Key results

• Qwen2.5-VL(72B) achieves the best weighted open-loop score of 54.47, with Qwen3-VL(32B) at 49.64 and Gemma-3-it(27B) at 49.53.
• Gemma-3-it(27B) gets the highest closed-loop average of 12.619, ahead of Qwen2.5-VL(72B) at 9.366; the benchmark-guided SFT of Qwen3-VL(8B) on all weak groups raises closed-loop score from 0.902 to 8.021.
• Spearman correlation of open-loop ranks with closed-loop R4 is 0.664 for R1, 0.596 for R2 and 0.418 for R3.

Surprising findings

• Driving-specialized VLMs (DA-DriveLM 4B, ReasonDrive 7B) score substantially lower than general-purpose VLMs in overall open-loop coverage.
• Targeted SFT on one capability group often improves non-target abilities (e.g., R1_2_B supervision raised R1_2_A by +35.78) rather than causing degradation.

Limitations

• Closed-loop results come from simulation and do not directly proxy real-world deployment.
• Hierarchy is broad but not exhaustive, with limited higher-level reasoning task families.
• Single-run deterministic scores; SFT results established on a single base model; composite weighting is a modeling choice.

Experimental setup

Datasets: NuScenes NuPlan NAVSIM DriveLM DriveBench LingoQA DRAMA

Benchmarks: DriveHierarchy open-loop (R1-R3) DriveHierarchy closed-loop R4 (CARLA-SUMO, 100 scenarios)

Metrics: exact-match accuracy log-ratio distance score IoU sequence inversion score route-safety-efficiency product score

Baselines: Qwen3-VL InternVL3.5 Gemma-3 Qwen2.5-VL DA-DriveLM ReasonDrive ZwZ MiniCPM-V-4.5 Pixtral InternVL3

Builds on: CARLA SUMO DriveLM NAVSIM LingoQA

Closest papers by meaning

• Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.90)
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.89)
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.89)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.89)
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving (NeurIPS 2024 · similarity 0.89)
• DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.89)