Research map / Vision-language and multimodal / Robot action and navigation

FIVE-VLA: Fast and EffectIVE Closed-Loop Autonomous Driving with Recurrent Action Memory

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FIVE-VLA, a 641M-parameter VLA for driving built on FastVLM with single-pass action prediction and a Recurrent Action Memory (RAM) that conditions on previous action tokens, reporting SOTA on Bench2Drive/Fail2Drive and faster inference.

Problem

Existing vision-language-action driving models are large, generate text (CoT or waypoint prefixes) with slow autoregressive decoding, use many vision tokens for high-resolution images, and lack temporal memory that does not disrupt the pretrained VLM.

Approach

Use FastViTHD (FastVLM) to encode 448x896 images into 98 tokens, skip text generation by appending the constant "Waypoints:" tokens to the input, and add RAM: a tiny two-block memory that conditions learnable action queries on the previous step's output action tokens, operating in action space instead of vision space.

Key results

• FIVE-VLA reaches 77.27% SR and 90.95 DS on Bench2Drive vs. 67.27% SR and 85.07 DS for SimLingo, with 641M parameters.
• On NVIDIA Physical AI AV single-view, FIVE-VLA lowers NC-V to 0.78% vs 0.87% for SimLingo using 98 vs 512 vision tokens.
• Throughput is 3.89 fps on T4 (about 8x SimLingo) and 29.96 fps on A100 (about 30x ORION).

Surprising findings

• Adding explicit past ego-waypoint history improves open-loop ADE/FDE but degrades closed-loop SR (73.03% to 64.39%), while RAM improves closed-loop results.
• Vision-domain temporal memories (Flex, QT-Former, sliding window) did not improve over the no-memory baseline under the same fine-tuning budget.

Limitations

• Training with data streams without RAM degrades performance due to reduced batch diversity, forcing a two-stage training procedure.
• Robustness analysis shows wrong-action memory injection strongly harms trajectory error (ADE up about 145% at p=0.5).

Experimental setup

Datasets: SimLingo dataset NVIDIA Physical AI AV Bench2Drive Fail2Drive

Benchmarks: Bench2Drive Fail2Drive NVIDIA Physical AI AV

Metrics: driving score success rate mean ability harmonic mean ADE FDE NC-V TTC-V fps

Baselines: SimLingo ORION AutoVLA BridgeDrive Transfuser++ HiP-AD Flex QT-Former Sliding Window

Builds on: FastVLM SimLingo Transformer cross-attention memory LoRA

Closest papers by meaning

• Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.93)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.90)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.88)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
• ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)