Research map / Vision-language and multimodal / Robot action and navigation

Less Language, More Latents: Annotation-Efficient VLAs for Driving

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LADA, a three-stage latent-action pipeline that learns a 25-entry VQ codebook of driving intents from unlabelled trajectories, grounds language via a translator trained on ~5% HLC/CF labels, and trains a SimLingo-based driving VLA on latent actions, matching or beating fully supervised SimLingo on Bench2Drive.

Problem

Driving vision-language-action models need language-annotated frames (high-level commands, counterfactuals, CoT, VQA) that are scarce and expensive, while expert trajectories are abundant.

Approach

Learn a discrete latent action codebook from observation-trajectory pairs with a VQ bottleneck (plus a light HLC alignment loss), map language to codes with a translator trained on a 5% labelled subset using counterfactual action dreaming, and train the driving VLA on observation-latent pairs for all frames.

Key results

• LADA reaches Driving Score 87.98 and Success Rate 70.46% on Bench2Drive using ~5% HLC and CF labels, vs. 86.08 / 65.78 for full-language SimLingo.
• Full-HLC (100% HLC labels) reaches 82.85 DS / 61.36 SR; LADA exceeds it by about 5 DS and 9 SR.
• Removing counterfactual action dreaming lowers LADA to 85.27 DS / 65.00 SR.

Surprising findings

• Using about 1% HLC labels still roughly matches the Full-HLC baseline, and increasing to 20% gives little extra benefit.
• Simple IDM-style pseudo-labelling from the same 5% labels performs worse than full-HLC supervision, showing classifier label noise hurts.

Limitations

• Experiments only in the CARLA simulator; real-world transfer is unvalidated.
• Counterfactual supervision relies on privileged simulator state.
• Discarding VQA and CoT annotations sacrifices interpretability and explanation ability.

Experimental setup

Datasets: SimLingo CARLA expert demonstration corpus

Benchmarks: Bench2Drive

Metrics: Driving Score Success Rate ability scores (merging, overtaking, emergency brake, give w codebook classification accuracy

Baselines: SimLingo (Full-language) Full-HLC 5-HLC IDM intent classifier Anchor k-means Heuristic route labeller

Builds on: SimLingo LAPO LAOM LAPA VQ-VAE

Closest papers by meaning

• Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.92)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.92)
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• FIVE-VLA: Fast and EffectIVE Closed-Loop Autonomous Driving with Recurrent Action Memory (NeurIPS 2026 · similarity 0.90)
• DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.90)
• Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.90)