Research map / Vision-language and multimodal / Multimodal LLMs

$AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AutoDrive-P³, a Qwen2.5-VL-3B driving VLM trained with SFT on a new P³-CoT perception-prediction-planning chain-of-thought dataset and a hierarchical GRPO (P³-GRPO) that rewards all three stages, with detailed and fast thinking modes.

Problem

VLM-based end-to-end driving either outputs plans without chain-of-thought over perception and prediction, or handles perception, prediction and planning as separate QA tasks without synergy; prior GRPO use supervises only planning outputs.

Approach

A unified perception-prediction-planning CoT dataset plus a GRPO variant whose reward has per-stage terms (perception IoU/F1, prediction label-weighted IoU, planning L2 or PDMS), so all three stages receive reinforcement signals in one chain.

Key results

• On NAVSIMv1, Ours-Detailed reaches 90.6 PDMS, above the best listed baseline WoTE at 88.3.
• On NAVSIMv2, Ours-Detailed reaches 86.2 EPDMS (False) and 89.9 (True), versus 84.7/88.2 for DiffusionDrive.
• On nuScenes, the average collision rate drops to 0.06% versus 0.10-0.12% for the best VLM and non-VLM baselines, with average L2 of 0.33 m.

Surprising findings

• Planning-only GRPO left the 3s collision rate improved but P3-GRPO gave much larger perception and prediction gains, which the authors read as evidence of stage synergy.
• Unbalanced reward weights (1:1:1:7) reduced perception and prediction scores and slightly worsened planning L2.

Limitations

• The model hallucinates during reasoning, as the authors acknowledge.
• RL is run only in offline simulators with no real-world interaction.
• The P3-CoT dataset sizes and statistics are inconsistent across the paper (e.g. 25303 vs 24403 frames for nuScenes), and manual verification covers only sampled CoT outputs.

Experimental setup

Datasets: nuScenes NAVSIM P3-CoT

Benchmarks: nuScenes NAVSIMv1 NAVSIMv2

Metrics: L2 collision rate PDMS EPDMS perception score prediction score

Baselines: UniAD ST-P3 VAD OmniDrive OpenDriveVLA AutoVLA AutoDrive-R² DiffusionDrive WoTE DriveVLM OpenEMMA GPT-Driver

Builds on: GRPO DriveLM AutoVLA Vision-R1

Closest papers by meaning

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.95)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.92)
• From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving (NeurIPS 2026 · similarity 0.91)
• Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.91)
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.91)