Research map / Vision-language and multimodal / Robot action and navigation

Generative Scenario Rollouts for End-to-End Autonomous Driving

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GeRo, a plug-and-play VLA framework that pretrains latent ego/agent tokens and then performs language-conditioned autoregressive scenario rollouts refined with consistency losses and GRPO rewards, reaching SOTA on Bench2Drive closed-loop and nuScenes open-loop.

Problem

VLA driving planners rely on imitation of sparse trajectory annotations and underuse generative capability, with weak language-action alignment and brittleness in long-tail scenarios.

Approach

Jointly learn compact latent ego/agent tokens with planning, motion, and VQA supervision, then autoregressively roll out future latent tokens and ego-action answers conditioned on scenario descriptions, stabilized by a token-consistency loss and GRPO rewards for collision, TTC, and language alignment.

Key results

• GeRo (Qwen2.5VL) reaches 79.60 driving score (+15.7 over Qwen2.5VL's 63.90) and 57.80% success rate (+26.2).
• GeRo (ORION) reaches 81.90 driving score and 60.10% success rate on Bench2Drive, above ORION's 77.74 and 54.62%.
• On nuScenes open-loop, GeRo (ORION) reduces average L2 from 0.69 to 0.27 and collision rate from 0.44 to 0.12; zero-shot GeRo (Qwen2.5VL) L2 is 0.45.

Surprising findings

• Zero-shot transfer from Bench2Drive to nuScenes still reduces L2 error by roughly 50% relative to baselines.

Limitations

• Rewards and rollouts depend on pseudo-labels from the pretrained model when ground truth is unavailable
• Additional scenario descriptions and QA pairs are generated with an unspecified LLM, and the paper gives limited detail on them

Experimental setup

Datasets: Bench2Drive nuScenes ChatB2D DriveLM-nuScenes

Benchmarks: Bench2Drive nuScenes

Metrics: Driving Score Success Rate Efficiency Comfortness Multi-Ability Avg L2 collision rate

Baselines: Qwen2.5VL ORION ORION (Qwen2.5VL) TCP-traj ThinkTwice DriveAdapter UniAD-Base UniAD-Tiny VAD GenAD MomAD DriveTransformer-Large

Builds on: GRPO ORION Qwen2.5-VL UniAD DriveLM

Closest papers by meaning

• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.89)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
• Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments (CVPR 2025 · similarity 0.88)