Research map / Vision-language and multimodal / Multimodal LLMs

More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds DriveMind, a nuPlan-based driving VQA dataset with plan-aligned CoT, and uses SFT/GRPO ablations plus a perturbation-based causal probe to argue that VLM driving agents' planning relies on textual priors rather than the generated reasoning.

Problem

Whether planning in VLM driving agents is causally mediated by their generated chain-of-thought reasoning, or whether planning instead relies on shortcut textual priors (ego state, history) while CoT is a non-causal byproduct.

Approach

Controlled information ablations (removing vision, CoT, or priors) on a modular nuPlan VQA dataset, combined with sequence-level attention analysis and a training-free perturbation probe (lateral offset and direction inversion of ego history), to test a Reasoning-Planning Decoupling Hypothesis.

Key results

• Plan_NoV agent (no visual input, no CoT) matches CoT agent: open-loop 1s score 98.95 vs 98.91 for CoT.
• Removing all priors drops CoT agent open-loop 1s score from 98.91 to 72.40 (CoT_NoPri).
• Attention to textual priors in planning rises to 26.97% in final layer while image tokens fall to 1.82%.

Surprising findings

• Blind planner without vision performs on par with the full CoT agent.
• GRPO on the SFT model does not improve and may weaken the reasoning-planning link (CoT_grpo weaker than CoT).

Limitations

• Fine-tuning only on 7B-scale models; larger models not explored.
• Causal probe evaluated via MFD/RID and manual inspection; scale of qualitative inspection is limited.
• CoT generated by GPT-4.1 with access to ground-truth future trajectory, which may bias reasoning quality.

Experimental setup

Datasets: nuPlan DriveMind

Benchmarks: nuPlan Challenge test set

Metrics: open-loop score closed-loop score ADE FDE collision ratio MFD RID attention proportion

Baselines: OmniDrive CoT (Qwen2.5-VL SFT) Plan (no CoT) Base (no fine-tuning) GPT-4o zero-shot

Builds on: GRPO OmniDrive DriveLM DriveVLM DriveCoT

Closest papers by meaning

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.92)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.91)
• $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.90)
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.89)
• Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.89)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)