Research map / Vision-language and multimodal / Multimodal LLMs

DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DriveAgent-R1, a 3B VLM driving agent that uses tool-based active perception and hybrid text/tool thinking, trained with SFT followed by cascaded RL (MP-GRPO and adaptive mode selection).

Problem

Existing VLM-based driving planners rely on passive perception with text-only chain-of-thought, processing fixed multi-view inputs and unable to actively seek visual evidence under uncertainty for high-level behavior planning.

Approach

A vision toolkit (Retrieve View, RoI Inspection, Depth Estimation, 3D Object Detection) lets the agent actively query visual evidence mid-reasoning; a hybrid-thinking mechanism lets it choose between text-only and tool-augmented reasoning, trained by Mode-Partitioned GRPO and an adaptive mode selection RL stage with a contrastive tool reward.

Key results

• DriveAgent-R1 (3B) reaches 51.34% first-frame joint accuracy with tools on Drive-Internal test, a +6.07 gain from tool use, versus GPT-5's 56.48%.
• On nuScenes test, DriveAgent-R1 achieves 47.10% sequence average joint accuracy with tools, above GPT-5's 45.14%.
• On nuScenes open-loop planning, DriveAgent-R1 reports average ADE 0.28m versus DriveVLM-Dual's 0.31m, with a comparable collision rate.

Surprising findings

• Tool access degrades Qwen2.5-VL-3B and -7B accuracy, while GPT-4.1 and Gemini-2.5-Flash improve, suggesting effective tool use requires capability or specialized training.
• Adaptive mode reduces latency versus forced tool mode (6.74s vs 7.91s), and DriveAgent-R1 is 56.48% first-frame vs GPT-5 on Drive-Internal, where GPT-5 remains higher with tools.

Limitations

• Over-reliance on new visual evidence from tools can cause errors when acquired views are misinterpreted (failure cases in appendix).
• Weak foundational understanding of complex road topologies and traffic rules, per the authors.
• Drive-Internal dataset is proprietary, limiting reproducibility.

Experimental setup

Datasets: Drive-Internal nuScenes DriveAlign-VQA DriveBench

Benchmarks: Drive-Internal test nuScenes validation subset DriveBench nuScenes open-loop planning

Metrics: first-frame joint accuracy sequence average joint accuracy mode selection accuracy ADE collision rate Perception/Prediction/Planning/Behavior scores inference latency

Baselines: Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-72B Doubao-Seed-1.6 Gemini-2.5-Flash GPT-4.1 GPT-5 DriveLM Dolphins UniAD VAD-Base EMMA

Builds on: GRPO Drive-R1 Qwen2.5-VL Depth Anything V2

Closest papers by meaning

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.93)
• $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.91)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.90)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.90)
• MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving (CVPR 2026 · similarity 0.90)