Research map / Large language models / Agents, code and math
LMDrive: Closed-Loop End-to-End Driving with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LMDrive, a frozen-LLM multi-modal (camera+LiDAR) closed-loop driving agent that follows natural-language navigation and notice instructions, with a 64K-clip CARLA dataset and the LangAuto benchmark.
Problem
Existing autonomous driving systems rely on fixed-format inputs and cannot follow natural-language instructions, and LLM-based driving work has been limited to open-loop evaluation, ignoring cumulative error and temporal consistency.
Approach
A pre-trained vision encoder (multi-view camera + LiDAR, BEV tokens) is pre-trained on perception tasks and frozen, then its tokens are compressed via a Q-Former and fed with language instructions into a frozen LLM whose outputs are waypoints and completion flags, converted to controls by PID; only Q-Former and adapters are trained.
Key results
- LLaVA-v1.5 backbone achieves driving score 36.2 on LangAuto, versus 10.7 for random init. and 33.5 for Vicuna.
- Removing visual pre-training of the vision encoder drops driving score from 36.2 to 16.9.
- On LangAuto-Tiny, LLaVA-v1.5 reaches driving score 66.5 and route completion 77.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Q-Former for visual-token reduction (ablation) | LMDrive · LangAuto | DS: 31.7±3.5 -> 36.2±2.3 | Helped |
| add BEV tokens to the LLM input (ablation) | LMDrive · LangAuto | IS: 0.72±0.03 -> 0.81±0.03 | Helped |
| pre-train the vision encoder (ablation) | LMDrive · LangAuto | DS: 16.9±5.1 -> 36.2±2.3 | Helped |
| use LLaVA-v1.5 instead of Vicuna-v1.5 as the LLM backbone | LMDrive · LangAuto | DS: 34.0±3.8 -> 36.2±2.3 | Helped |
| provide notice instructions during adversarial events | LMDrive · LangAuto-Notice | IS: 0.81 -> 0.87 | Helped |
| merge consecutive navigation instructions into one long instruction | LMDrive (LLaVA-v1.5) · LangAuto-Sequential | DS: 36.2 -> 34.0 | Hurt |
Surprising findings
- A randomly initialized 7B LLM nearly fails to drive (driving score 10.7), showing pretrained LLM weights matter for instruction-following driving.
- Removing the Q-Former or BEV tokens lowers performance, with removing BEV tokens mainly increasing infractions.
Limitations
- Evaluated only in the CARLA simulator, not on real vehicles.
- Relies on privileged rule-based expert data and a simulator-generated instruction set.
- Misleading-instruction handling and notice following may trade off against conservative behavior.
Experimental setup
Datasets: LMDrive instruction-following CARLA dataset (64K clips) CARLA expert driving data (~3M frames)
Benchmarks: LangAuto LangAuto-Short LangAuto-Tiny LangAuto-Notice LangAuto-Sequential
Metrics: driving score route completion infraction score
Baselines: Random Init. LLaMA LLaMA2 Vicuna Vicuna-v1.5 LLaVA-v1.5
Builds on: BLIP-2 LLaVA InterFuser LBC PointPillars
Closest papers by meaning
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · similarity 0.88)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.88)
- Feedback-Guided Autonomous Driving (CVPR 2024 · similarity 0.87)
- SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment (CVPR 2025 · similarity 0.87)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.86)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.