Research map / Vision-language and multimodal / Robot action and navigation
AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
AutoVLA unifies chain-of-thought reasoning and physical action-token trajectory generation in one autoregressive Qwen2.5-VL-3B model, trained with SFT for fast/slow thinking modes and GRPO-based RFT that penalizes unnecessary reasoning.
Problem
Existing VLA models for end-to-end driving produce physically infeasible actions or rely on complex intermediate modules, and use fixed reasoning strategies that are inflexible and slow on simple scenarios.
Approach
Discretize trajectories into a K-disk clustered codebook of 2048 physical action tokens added to a pretrained VLM, train with SFT on mixed fast (action-only) and slow (CoT) data, then apply GRPO with PDMS/ADE rewards and a CoT length penalty so the model learns adaptive reasoning.
Key results
- On NAVSIM, AutoVLA Post-RFT reaches 89.11 PDMS vs 80.54 one-shot (10.6% gain) with 66.8% runtime reduction; best-of-6 reaches 92.12 PDMS.
- On Bench2Drive closed-loop, AutoVLA achieves 78.84 driving score and 57.73% success rate, above Orion (77.74, 54.62).
- Physical action tokenization gives 80.54 PDMS vs 71.31 for text waypoint output, with runtime 3.95 s vs 7.65 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GRPO reinforcement fine-tuning to the full-data CoT SFT model | AutoVLA · NAVSIM (nuPlan) test | PDMS: +10.6% | Helped |
| Apply GRPO reinforcement fine-tuning with a CoT length penalty | AutoVLA · NAVSIM test scenarios | Runtime: 66.8% reduction | Helped |
| Add CoT reasoning supervision instead of action-only supervision (ablation) | AutoVLA · nuScenes validation | L2 distance: 0.41 -> 0.40 (ST-P3 Avg.); 0.71 -> 0.70 (UniAD Avg.) | Mixed |
| Add CoT reasoning supervision instead of action-only supervision (ablation) | AutoVLA · nuScenes validation | Collision rate: 0.18 -> 0.20 (ST-P3 Avg.); 0.35 -> 0.31 (UniAD Avg.) | Mixed |
| Replace text waypoint output with physical action tokens (ablation) | AutoVLA · NAVSIM (nuPlan) test | PDM Score: 71.31 -> 80.54 | Helped |
| Replace text waypoint output with physical action tokens (ablation) | AutoVLA · NAVSIM (nuPlan) test | Runtime: 7.65 -> 3.95 | Helped |
| Replace FAST (DCT) tokenization with K-disk tokenization (ablation) | AutoVLA · NAVSIM (nuPlan) test | PDMS: 67.63 -> 80.54 | Helped |
| Increase the K-disk codebook size from 2048 to 4096 (ablation) | AutoVLA action tokenizer · Waypoint reconstruction | ADE(m): 0.0182 -> 0.0141 | Helped |
| Increase the K-disk codebook size from 2048 to 4096 (ablation) | AutoVLA action tokenizer · Waypoint reconstruction | Codebook Usage: 100.0% -> 91.46% | Hurt |
| Reduce the K-disk codebook size from 2048 to 256 (ablation) | AutoVLA action tokenizer · Waypoint reconstruction | ADE(m): 0.0182 -> 0.0687 | Hurt |
| Add CoT reasoning supervision rather than action-only training (ablation) | AutoVLA · nuPlan with fewer than 50k training samples | PDMS | No clear effect |
| Increase the amount of training data (ablation) | AutoVLA · nuPlan and nuScenes | Planning performance | Helped |
Surprising findings
- With fewer than 50k training samples on nuPlan, CoT reasoning did not outperform action-only supervision in PDMS and collision score.
- On nuScenes, reasoning gave no clear quantitative gain, attributed to many scenarios being straightforward.
Limitations
- Remains highly GPU-dependent, requiring significant memory and compute; inference near 1 Hz.
- Reasoning gains are not consistent on simpler datasets such as nuScenes and small-data regimes on nuPlan.
Experimental setup
Datasets: nuPlan (NAVSIM) nuScenes Waymo E2E CARLA-Garage DriveLM
Benchmarks: NAVSIM (nuPlan) navtest PDMS nuScenes planning L2 and collision Waymo E2E RFS Bench2Drive closed-loop CARLA
Metrics: PDMS L2 distance collision rate RFS driving score success rate runtime
Baselines: Ego Status MLP TransFuser DRAMA Hydra-MDP Centaur TrajHF ST-P3 VAD UniAD EMMA OpenEMMA OpenDriveVLA-3B
Builds on: Qwen2.5-VL GRPO DeepSeek-R1 FAST tokenization DriveLM
Closest papers by meaning
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.93)
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.92)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.92)
- NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoning (CVPR 2026 · similarity 0.91)
- OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.91)
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.