Research map / Vision-language and multimodal / Robot action and navigation
Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Impromptu VLA dataset of ~80K curated driving clips on four unstructured corner-case categories with planning-oriented Q&A and trajectories, and shows that fine-tuning Qwen2.5-VL on it improves NeuroNCAP and nuScenes open-loop results.
Problem
Vision-Language-Action models for autonomous driving struggle in unstructured corner-case scenarios, largely because targeted benchmarks and training data are scarce.
Approach
A taxonomy of four unstructured driving categories and a VLM-centric curation pipeline (Qwen2.5-VL 72B CoT annotation plus human verification) that turns clips from eight open driving datasets into multi-task planning Q&A with trajectories, used as an intermediate fine-tuning stage before nuScenes.
Key results
- On NeuroNCAP, Base+Impromptu+nuScenes (3B) raises the average NeuroNCAP score from 1.77 to 2.15 and lowers the average collision rate from 72.5% to 65.5%.
- On nuScenes open-loop, the 3B model with Impromptu pre-training reaches an average L2 of 0.30m, close to EMMA+ (0.29m).
- On the Impromptu validation set, 3B Q&A accuracy for dynamic objects rises from 0.20 to 0.92 and the average L2 error falls from 6.62m to 0.69m.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on Impromptu VLA before nuScenes (ablation) | Qwen2.5-VL 3B · NeuroNCAP | average NeuroNCAP score: 1.77 -> 2.15 | Helped |
| Fine-tune on Impromptu VLA before nuScenes (ablation) | Qwen2.5-VL 3B · NeuroNCAP | average collision rate (%): 72.5 -> 65.5 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 3B · Impromptu VLA validation | V.R.U. accuracy: 0.87 -> 0.91 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 3B · Impromptu VLA validation | traffic light accuracy: 0.95 -> 0.96 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 3B · Impromptu VLA validation | dynamic object accuracy: 0.20 -> 0.92 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 3B · Impromptu VLA validation | meta-planning accuracy: 0.56 -> 0.84 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 3B · Impromptu VLA validation | average trajectory L2 error (m): 6.62 -> 0.69 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 7B · Impromptu VLA validation | V.R.U. accuracy: 0.86 -> 0.91 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 7B · Impromptu VLA validation | traffic light accuracy: 0.92 -> 0.97 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 7B · Impromptu VLA validation | dynamic object accuracy: 0.22 -> 0.92 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 7B · Impromptu VLA validation | meta-planning accuracy: 0.55 -> 0.83 | Helped |
| Fine-tune on Impromptu VLA (ablation) | Qwen2.5-VL 7B · Impromptu VLA validation | average trajectory L2 error (m): 5.74 -> 0.54 | Helped |
Surprising findings
- The base model's average L2 error on the Impromptu validation set (6.62m for 3B) is much larger than after fine-tuning (0.69m), and the paper does not discuss why the gap is so large.
Limitations
- Annotations rely mainly on Qwen2.5-VL, which may introduce model-specific biases.
- Hyperparameters were set from general practice without thorough tuning.
- The 'Road With Unclear Boundaries' category was too rare in the nuScenes validation subset to compute an F1 score.
Experimental setup
Datasets: Mapillary ONCE NAVSIM nuScenes Waymo Argoverse-V2 KITTI IDD
Benchmarks: NeuroNCAP nuScenes Impromptu VLA validation set
Metrics: NeuroNCAP score collision rate L2 error (1s/2s/3s/avg) Q&A accuracy
Baselines: UniAD VAD SparseDrive BridgeAD EMMA+
Builds on: Qwen2.5-VL NAVSIM Senna EMMA
Closest papers by meaning
- FIVE-VLA: Fast and EffectIVE Closed-Loop Autonomous Driving with Recurrent Action Memory (NeurIPS 2026 · similarity 0.93)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.92)
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model (NeurIPS 2025 · similarity 0.90)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.