Research map / Vision-language and multimodal / Robot action and navigation
SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimLingo, a camera-only vision-language-action driving model built on InternVL2-1B that unifies closed-loop driving, VQA/commentary, and language-action alignment via a new Action Dreaming dataset.
Problem
Closed-loop autonomous driving with VLMs has lagged behind specialized driving models, and VQA-based language understanding is disconnected from the driving action space, so answers can contradict behavior.
Approach
A camera-only VLM (InternVL2-1B) fine-tuned with LoRA to output disentangled path and speed waypoints alongside language, trained on Action Dreaming data where simulated alternative futures are paired with instructions, forcing the model to act on language instructions.
Key results
- SimLingo-BASE reaches 6.87 driving score on CARLA Leaderboard 2.0 Sensor track, 4.6x over CaRINA hybrid and 33% over TF++.
- SimLingo achieves 85.07 driving score and 67.27% success rate on Bench2Drive, versus 64.22 DS for the prior best DriveAdapter.
- On Action Dreaming, success rate rises from 24.52% (without dream data) to 81.13% average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use disentangled path and speed waypoints instead of entangled waypoints (ablation) | SimLingo-BASE · CARLA Leaderboard 2.0 | Driving Score: +39.9 percent | Helped |
| Use a CLIP-pretrained vision encoder instead of a ViT trained from scratch (ablation) | SimLingo-BASE · CARLA Leaderboard 2.0 sensor track | Driving Score: 0.45 vs 6.87 | Helped |
| Use a CLIP-pretrained ViT instead of ImageNet-pretrained ResNet-34 (ablation) | SimLingo-BASE · CARLA Leaderboard 2.0 sensor track | Driving Score: 2.71 compared to 6.87 | Helped |
| Add Action Dreaming data to training (ablation) | SimLingo · Action Dreaming | Success Rate: 24.52 -> 81.13 | Helped |
| Add Action Dreaming data to the training mixture (ablation) | SimLingo · Bench2Drive | Driving Score: 84.55±0.42 -> 85.07±0.95 | Helped |
| Remove VQA and Commentary training (ablation) | SimLingo · Bench2Drive | Driving Score: 84.55±0.42 vs 85.07±0.95 | No clear effect |
| Remove Commentary, VQA, and Action Dreaming training (ablation) | SimLingo · Bench2Drive | Driving Score: 84.67±1.52 vs 85.07±0.95 | No clear effect |
| Use language command instead of GPS target points (ablation) | SimLingo · Bench2Drive | Driving Score: 85.07±0.95 -> 86.08±1.76 | No clear effect |
| Condition driving actions on generated Commentary (Chain-of-Thought) (ablation) | SimLingo · Bench2Drive | Driving performance | No clear effect |
Surprising findings
- Pure vision-language training (VQA and commentary) did not change driving performance, and chain-of-thought commentary showed no statistically significant driving gains.
- Using only a language command for navigation (HLC) gives driving performance within variance of GPS target points (86.08 vs 85.07 DS).
Limitations
- Only the lightweight SimLingo-BASE was evaluated on the official Leaderboard 2.0; the full SimLingo was tested only on Bench2Drive.
- Chain-of-thought commentary conditioning did not show statistically significant driving improvements.
- Evaluation is only in simulation; VLM inference latency for real-time driving is not addressed.
Experimental setup
Datasets: CARLA driving data collected with PDM-lite (3.1M samples) DriveLM VQA labels Action Dreaming instruction-action pairs Commentary labels (newly generated)
Benchmarks: CARLA Leaderboard 2.0 Bench2Drive DriveLM-hard VQA Commentary benchmark Action Dreaming
Metrics: Driving Score Route Completion Infraction Score Success Rate Efficiency Comfortness GPT score SPICE
Baselines: TransFuser++ CaRINA hybrid Kyber-E2E TCP-traj w/o distillation UniAD-Base VAD InternVL2-1B InternVL2-2B InternVL2-4B
Builds on: InternVL2 TransFuser++ DriveLM PDM-lite TCP
Closest papers by meaning
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.88)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.88)
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.88)
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · similarity 0.87)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.87)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.