Research map / Large language models / Agents, code and math
LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LaMPilot, a framework where LLMs write code (language model programs) over AD functional primitives to follow user driving instructions, plus LaMPilot-Bench, a simulator-based benchmark of 4,900 scenarios evaluated on safety and efficiency.
Problem
Autonomous driving systems struggle to interpret and execute spontaneous natural-language user instructions (e.g., "overtake the car ahead"), and there is no benchmark for evaluating LLM-based agents on instruction-following in driving.
Approach
Use LLMs to generate code policies (language model programs) that call classical planning/control API primitives, executed at low frequency in a simulator, instead of having LLMs output low-level controls; paired with a benchmark of instructions, initial states, and programmatic safety/efficiency scoring.
Key results
- GPT-4 Turbo with human feedback and retrieval reaches 87.3% completion; GPT-4 with human feedback achieves the highest driving score of 64.0.
- Rule-based IDM achieves 20.4% completion and driving score 17.3 without using instructions.
- Few-shot CoT raises Llama 2 completion from 20.4% to 63.3%, and GPT-4 driving score from 28.5 to 55.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add three human-written code exemplars (0-shot to 3-shot) (ablation) | Llama 2 · LaMPilot-Bench | Completion: 20.4% -> 63.3% (+42.9) | Helped |
| Add three human-written code exemplars (0-shot to 3-shot) (ablation) | PaLM 2 · LaMPilot-Bench | Completion: 35.7% -> 71.4% (+35.7) | Helped |
| Add three human-written code exemplars (0-shot to 3-shot) (ablation) | ChatGPT · LaMPilot-Bench | Completion: 40.8% -> 83.7% (+42.9) | Helped |
| Add three human-written code exemplars (0-shot to 3-shot) (ablation) | GPT-4 · LaMPilot-Bench | Completion: 72.4% -> 84.7% (+12.3) | Helped |
| Add three human-written code exemplars (0-shot to 3-shot) (ablation) | GPT-4 Turbo · LaMPilot-Bench | Completion: 74.5% -> 80.6% (+6.1) | Helped |
| Add human feedback and retrieved code examples (ablation) | Llama 2 · LaMPilot-Bench | Driving Score: 39.6 -> 21.4 (-18.2) | Hurt |
| Add human feedback and retrieved code examples (ablation) | PaLM 2 · LaMPilot-Bench | Driving Score: 36.6 -> 35.6 (-1.0) | Hurt |
| Add human feedback and retrieved code examples (ablation) | ChatGPT · LaMPilot-Bench | Driving Score: 41.2 -> 54.2 (+13.0) | Helped |
| Add human feedback and retrieved code examples (ablation) | GPT-4 · LaMPilot-Bench | Driving Score: 55.9 -> 64.0 (+8.1) | Helped |
| Add human feedback and retrieved code examples (ablation) | GPT-4 Turbo · LaMPilot-Bench | Driving Score: 52.6 -> 60.5 (+7.9) | Helped |
Surprising findings
- Zero-shot LLM policies increase collision rates (1%-4%) relative to rule-based baselines despite higher completion.
- Human feedback lowered Llama 2 completion (63.3% to 32.7%) and PaLM 2 driving score slightly.
Limitations
- Collision rates remain notable even with the best LLMs, indicating safety gaps.
- Framework is text-only and does not use visual perception.
- Benchmark is built on the HighwayEnv simulator with custom intersections, limiting realism.
Experimental setup
Datasets: LaMPilot-Bench
Benchmarks: LaMPilot-Bench
Metrics: collision rate task completion rate TTC score speed variance score time efficiency score driving score
Baselines: IDM MOBIL Human Driver Llama 2 zero-shot PaLM 2 ChatGPT GPT-4 GPT-4 Turbo
Builds on: Code as Policies Voyager Chain-of-Thought prompting HighwayEnv Retrieval-Augmented Generation
Closest papers by meaning
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.89)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.88)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.88)
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models (CVPR 2024 · similarity 0.88)
- nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.